From Prediction to Action: Explainable Churn Modelling with Calibration and Threshold-Aware Decision Support

Abstract

The problem of predicting customer churn plays an important role in the field of telecommunication business decisions. However, the existing literature mainly focuses on achieving better classification accuracy without paying enough attention to calibration, interpretability, and the business value of threshold choice. The goal of this study is to introduce a churn prediction framework that incorporates optimization of the classifier, its calibration, explainability, and business-oriented choice of the threshold. The open-source Telco Customer Churn dataset was used to compare four models like Logistic Regression, Random Forest, Extreme Gradient Boosting (XGBoost), and Light Gradient Boosting Machine (LightGBM). The process of hyperparameter tuning is carried out by means of randomized search and stratified cross-validation. Further experiments include repeated stratified cross-validation and hold-out validation of the models' results. Random Forest showed the best performance in terms of mean cross-validation, while LightGBM demonstrated the best F1-score and Matthew’s correlation coefficient on the hold-out set. Due to the lack of statistical significance of the difference between these algorithms, LightGBM was chosen for further use. Isotonic calibration decreases the Brier score from 0.1609 to 0.1371. The threshold optimization shows that choosing the decision boundary equal to 0.30 improves the balance between precision and recall compared to the usual 0.50. Furthermore, the cost-sensitive analysis demonstrates that the optimal value of thresholds decreases as the cost of missing out on a true churner rises. SHapley Additive exPlanations (SHAP) values and permutation-based feature importance were calculated to identify the four most important predictors of churn, including contract type, customer tenure, internet service type, and billing variables. In this research, a decision support framework is developed by calibrating probabilities and tuning decision thresholds.