A Hybrid Machine Learning and Rule-Based Recommendation Framework: A Probabilistic Approach for Reliable Customer Next-Transaction Prediction

Abstract

This study presents a hybrid recommendation framework that combines machine learning-based probabilistic prediction with rule-based decision logic for forecasting customers' next transactions in digital banking. It aims to help in customer retention and personalization, as well as cross-sell strategies in settings where the behavior of the user is heterogeneous and transaction data are largely imbalanced. The machine learning component estimates behavioral consistency probabilities, whereas the rule-based component translates these probabilities into personalized product recommendations according to customer segmentation and business constraints. The principal methodological contribution is the proposed inverse transform probability mechanism, which reinterprets predicted probabilities as behavioral consistency scores across active, dormant, and new customer segments. This conversion makes it possible to have a uniform probabilistic understanding of the states of the users while maintaining the economic significance of the recommendation decisions. The model performance was evaluated using the area under the precision-recall curve (PR-AUC) because it is a good performance metric for imbalanced classification problems. The empirical results demonstrate good predictive ability and temporal robustness, achieving an average PR-AUC of 0.82 over the evaluation period. Furthermore, the probabilistic framework is in line with the concept of expected monetary value (EMV) under a binomial state-price interpretation and links the prediction of behavior to the financial decision-making process. The proposed framework provides an interpretable probabilistic methodology for next-transaction prediction under realistic synthetic banking scenarios and establishes a foundation for future validation using real banking transaction data. The results obtained on synthetic transaction data suggest that the inclusion of behavioral consistency in probabilistic models can enhance the stability and practicality of recommending systems for financial institutions.