Optimizing the Collection Process in Credit Risk Management: A Comparison of Machine Learning Techniques for Predicting Payment Probability at Different Stages of Arrears

1. Introduction

1.1. Problem Statement

In mass credit management, scoring models have proven to be the most valuable tool for the past two decades. By analyzing historical data, these models provide predictions of future behavior, which help control portfolios with greater accuracy and less uncertainty. A scoring model considers numerous variables simultaneously, which helps to establish a pattern and group members together based on their likelihood of experiencing an event. These models work best when dealing with large volumes of data with relatively homogeneous values. It is important to note that scoring models are designed to identify patterns and groupings, rather than to provide precise predictions for individual cases. (see Figure 2)

Due to its ease of interpretation, logistic regression is the favorite in this kind of problems. However, with the increase in available information, the growth over time of debtors, and more lenders, obtaining a score with a high predictive capacity becomes a much more complex task.

To measure which model has greater predictive power, the Kolmogorov-Smirnoff statistic is used [18], which measures how different or how far apart two probability distributions are. A model is said to have greater predictive power if its KS value is closer to 1 (see Figure 3).

1.2. Scoring Models for Credit Collection

Effective collections management is a critical aspect of managing large credit portfolios. It affects not only customer interactions, but also collections operations. Without access to agile and effective tools, negative reactions from customers to their payment obligations are likely to occur. In the credit world, it is commonly understood that as the age of arrears increases, the chances of recovering funds decrease. It is therefore imperative to develop effective strategies to prevent this situation from occurring.

Typically, collections management relies on portfolio segmentation and customer contact channels to determine the appropriate actions based on the number of days in arrears, the type of product and the likelihood of payment. This approach may involve a variety of methods such as phone calls, field visits, text messages, emails or letters to encourage customers to meet their obligations and ensure a positive outcome. (see Figure 1)

When implementing scoring models for collections, it is crucial to consider the number of days that have passed since the loan was due, to determine whether it is still recoverable. Hence, it is recommended to segment the portfolio into 30 or 15-day arrears ranges, based on the loan disbursement terms. Therefore, the following arrears segments are possible:

• 0 - No Arrears segment
• 1 - 30 segment
• 31 - 60 segment
• 61 - 90 segment
• 91 - 120 segment
• More than 120 segment

Where More than 120 is considered as a loss segment.

When arrears increase, it is crucial to differentiate between each segment and design a scoring model to distinguish reliable payers from those who default on payments. It is essential to keep in mind that as arrears increase, the number of individuals decreases, which may impact the model’s ability to predict. However, thankfully, we can measure the scoring model’s ability to discriminate using the Kolmogorov-Smirnoff (KS) metric.

1.3. Literature Review

This section reviews some efforts to predict the probability of client default to make informed lending decisions. We explore various modeling techniques, including logistic regression, XGBoost, and artificial neural networks, and evaluate their performance using the Kolmogorov-Smirnov (KS) statistic.

1.3.1. Credit Scoring with Logistic Regression

Logistic regression is a frequently utilized statistical method in the credit rating industry because of its capability to model the likelihood of a binary event, such as a credit default [23]. This model is renowned for its interpretability and predictive accuracy. Recent research has shown that logistic regression has achieved remarkably high accuracy in forecasting credit risk, reaching up to 99% in certain instances [22].

1.3.2. Credit Scoring with XGBoost

XGBoost, a decision tree-based machine learning algorithm, is widely acclaimed for its exceptional performance in classification and regression tasks. This model excels at uncovering intricate patterns in data and has demonstrated successful applications in credit risk prediction [4]. However, there have been instances where XGBoost has underestimated credit risk, indicating the necessity for further refinements and validations [27].

1.3.3. Credit Scoring with Artificial Neural Networks

Artificial neural networks are adept at capturing non-linear and complex relationships between variables, making them well-suited for predicting credit defaults [25] [8]. In comparative studies, neural networks have demonstrated performance comparable to logistic regression, achieving an accuracy of 71% in training and 72% in testing [25].

1.3.4. Performance Evaluation

The Kolmogorov-Smirnov (KS) statistic is a metric utilized to evaluate the predictive performance of scoring models. It measures the difference between the cumulative distributions of good and bad payers’ scores, indicating the degree of distinction between the two sets of scores. Recent studies have incorporated KS alongside other metrics like the area under the ROC curve and the GINI test to evaluate and compare the effectiveness of various models [23] [20].

1.3.5. Comparison of Models

In comparing models for credit risk prediction, logistic regression, neural networks, and XGBoost have all been identified as suitable techniques. Each method has its own strengths and weaknesses. Logistic regression is highly interpretable and exhibits high accuracy in predicting credit extension [23] [22]. XGBoost, on the other hand, delivers high performance and can handle large amounts of data, although it may require additional adjustments to prevent underestimating risk [6]. While neural networks are complex, they can capture non-linear relationships and have demonstrated performance comparable to logistic regression [25] [8].

In sum, selecting the most suitable modeling technique for credit scoring and collections depends on various factors such as interpretability, data volume, and predictive capability. Viable techniques include logistic regression, XGBoost, and artificial neural networks, each with their distinct advantages and limitations. Evaluating performance using metrics like the Kolmogorov-Smirnov statistic is essential for assessing the efficacy of each model in predicting credit risk.

This study, unlike previous studies, explores the effectiveness of conventional logistic regression compared to two machine learning techniques, Extreme Gradient Boosting (XGBoost) and artificial neural networks (ANNs), in predicting the ability to pay at different delinquency levels of a large retail loan portfolio. Using the KS metric, the performance of each model at each arrear segment is measured.

Figure 2. Credit Scoring Scheme.

Figure 2. Credit Scoring Scheme.

Currently, efforts are focused on feature engineering to find better predictors to maximize the discrimination of logistic regression-based models. The objective is often to estimate the probability that a customer will pay off his debt, determine his risk level in case of loan approval, or assess the cost-effectiveness of offering several products to the same person.

In this paper, we focus on maximizing the predictive power of the model by testing different supervised learning techniques in the recovery phase of loans at different arrears segments.

Figure 3. Representation of the separation or divergence of two probability distributions.

Figure 3. Representation of the separation or divergence of two probability distributions.

2. Methodology

2.0.1. Kolmogorov-Smirnov Tests

The K-S test, or Kolmogorov-Smirnov test, is a non-parametric method utilized to assess the similarity of two distinct continuous distributions. It evaluates the hypothesis of whether or not they are identical. The KS statistic is computed by employing the cumulative empirical distribution function [1].

$$\widehat{F}\left(x\right)={\displaystyle \frac{1}{n}}\sum _{i=1}^n\left\{\begin{array}{c}\begin{array}{cc}1,\hfill & \mathrm{if}{x}_i\le x\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\hfill \end{array}\right.$$

(1)

Consider two samples $x_s$ and $y_s$ of size $n_1$ and $n_2$ respectively, with cumulative distribution functions $F_1$ and $F_2$ of a continuous random variable X. The KS test is used to test hypotheses:

$$\left\{\begin{array}{c}\begin{array}{cc}{H}_0:\hfill & F_1\left(x\right)=F_2\left(x\right),\forall x\hfill \\ H_1:\hfill & F_1\left(x\right)\ne F_2\left(x\right)\hfill \end{array}\hfill \end{array}\right.$$

(2)

Based on the use of the empirical cumulative distribution function (1), the KS statistic is used to test the null hypothesis $H_0$. Its value is obtained using the following expression:

$$KS=\underset{x}{max}|\widehat{F_1}\left(x\right)-\widehat{F_2}\left(x\right)|$$

(3)

The notation $\widehat{F_1}$ represents the empirical accumulation function of $x_s$ and $\widehat{F_2}$ represents the empirical distribution function of $y_s$. If the KS statistic is greater than the critical value $K{S}_{\alpha }$ for a given significance level $\alpha$, we reject the null hypothesis $H_0$. In [18], you can find a table of critical values for different sample sizes.

Then, the KS statistic is a measure of divergence between the distributions of two variables. It is the maximum distance between $F_1$ and $F_2$, and its value ranges between 0 and 1. Values close to 0 indicate that the distributions of $x_s$ and $y_s$ are identical, while values close to 1 indicate that the distributions of $x_s$ and $y_s$ differ. Therefore, the KS statistic is useful for distinguishing the differences between two distributions.

In our current project, we aim to determine the classification technique that achieves the highest KS value. We will compare the results of logistic regression, Extreme Gradient Boosting, and Artificial Neural Networks. The predicted score values for individuals who are categorized as good customers with 0 will be represented by $x_s$, while the predicted score values for individuals who are categorized as bad customers with 1 will be represented by $y_s$.

2.1. Data Sample Selection

When building a scoring model, it is important to collect as much information as possible that has been generated during the life of the loan. At the collection stage, it is especially essential to have data on payment behavior at the lending institution, information on the social and demographic profile of the customer, and data on collection performance. Sometimes it may also be useful to include information on payment behaviour at other financial institutions.

For our study, we have focused on loans granted directly to consumers in the retail sector, which includes credit for items such as televisions, computers, technology, household appliances, and other consumer goods. To explain the information needed to develop the scoring models, consider Figure 4. The period of time prior to the observation point is called the ‘behavioural window’, which cannot be longer than 36 months according to the provision of the superintendence of banks and insurance of Ecuador.

Figure 4. Historic Data Selection.

Figure 4. Historic Data Selection.

Normally, a 36-month history is used when scoring models are created. However, during the collection phase, using such a long history can be counterproductive. This is because the collection stage is much more dynamic and unpredictable, and mistakes can be made by considering ancient payment behavior that may not reflect the current situation. As a result, it can be difficult to predict your next payment, so we use 12 months of history. During this period, variables related to the individual’s credit history are generated, such as payment and borrowing habits, maximum and average delinquency, open transactions, telephone transactions, actual telephone contacts, card payments, consumption amounts, etc. At the observation point, socio-demographic variables such as age, marital status, province, region, etc. are generated.

After the observation point, we evaluate an individual’s payment behavior during a period called the "performance window". This window provides crucial information that helps us define good and bad individuals (dependent variable Y). Since payments are made monthly, we use a one-month window to determine whether a payment has been made or not. After that, we evaluate the individual’s payment behavior over a period of 6 months.

2.2. Dependent Variable Setting

The dependent variable Y is binary, with a value of 1 assigned to individuals marked as "Good Customer" and 0 assigned to those identified as "Bad Customer". We will be using two different definitions to evaluate performance. The first is based on payment events within a one-month performance window, while the second is based on monthly payment behavior within a 6-month performance window.

Definition 2.1. 

Payment Event

$$Y_1=\left\{\begin{array}{c}\begin{array}{cc}0:\hfill & \mathrm{if}\mathrm{paid}\mathrm{a}\mathrm{complete}\mathrm{installment}\hfill \\ 1:\hfill & \mathrm{otherwise}\hfill \end{array}\hfill \end{array}\right.$$

(4)

Definition 2.2. 

Monthly Payment Behavior

$$Y_2=\left\{\begin{array}{c}\begin{array}{cc}0:\hfill & \mathrm{if}\mathrm{paid}\mathrm{before}30\mathrm{days}\mathrm{of}\mathrm{arrears}\hfill \\ 1:\hfill & \mathrm{otherwise}\hfill \end{array}\hfill \end{array}\right.$$

(5)

With $Y_1$, the aim is to have as few clients as possible switch to higher arrears ranks. This definition will be used in each arrears range to discriminate between good and bad clients. On the other hand, with $Y_2$, the aim is to control the deterioration of the portfolio in the medium term, to avoid excessive losses.

3. Train and Test Models

A scoring model is created by identifying patterns within the predictor variables, which can be used to classify individuals into good and bad categories based on the event to be predicted. In machine learning, this model is developed through supervised learning, where the model is trained with data that is different from the data used in the training phase.

It is crucial to have a diverse dataset during the training phase to ensure that the model is trained on a wide range of information. In order to achieve this, the data needs to be randomly split into three datasets. The first dataset, comprising 60% of the data, is used for training. The second dataset, containing 25% of the data, is used for testing. Finally, the remaining 15% of the data is used for validation.

Table 1 the distribution of the training, testing, and validation samples. Meanwhile, Table 2 indicates the distribution of customers classified as good and bad, for $Y_1$ and $Y_2$.

3.1. Logistic Regression Training

Logistic regression is a widely used technique for predicting a categorical variable using a set of explanatory variables. It is a parametric method that is formulated as follows.

Consider N quantitative variables $X_1,...,X_N$. For each combination of these variables, the response variable Y follows a Bernoulli distribution [21].

$Y|(X_1=x_1,...,X_N=x_N)\rightarrowtail B(1,p(x_1,...,x_N))$

We are interested in modelling the conditional expectation.

$E\left[Y\right|(X_1=x_1,...,X_N=x_N)]=P[Y=1|X_1=x_1,...,X_N=x_N]=p(x_1,...,x_N)$

The multiple logistic regression model for Y in terms of the values of the variables X, can be modelled as:

$$p(x_1,...,x_N)={\displaystyle \frac{exp(\alpha +{\sum }_{n=0}^N{\beta }_n{x}_n)}{1+exp(\alpha +{\sum }_{n=0}^N{\beta }_n{x}_n)}}$$

(6)

with $\alpha ={\beta }_0$ and $x_0=1$

In matrix terms it would be

$$p\left(x\right)={\displaystyle \frac{exp\left({\beta }^{t}x\right)}{1+exp\left({\beta }^{t}x\right)}}$$

(7)

with $x=(1,x_1,...,x_N)$ and $\beta =({\beta }_0,...,{\beta }_N)$

Finally, a linear model for the logit transformation is obtained.

$$ln\left[{\displaystyle \frac{p\left(x\right)}{1-p\left(x\right)}}\right]=\sum _{n=0}^N{\beta }_n{x}_n$$

(8)

3.1.1. Unbalanced Problem

In some cases where we use logit, probit or linear probability models, the number of observations in one group is much smaller than in the other. For instance, in lending, the number of bad clients is expected to be much smaller than the number of good clients because if both were equal, the financial institution would face bankruptcy. Therefore, to reach accurate predictions, we need either a large dataset or a balanced sample containing equal proportions of both groups. In this case, we would consider all bad customers and sample the good customers to achieve a 50/50 ratio.

The question arises as to how we can analyze data in such cases. We suggest using a weighted logit (or probit or linear probability) model, similar to weighted least squares. If the logit model is used for estimating the coefficients of the explanatory variables, the different sample sizes for the two groups do not affect the coefficients [17].

Let $m_1$ and $m_2$ be the sample proportions of the two groups, with $m_2>m_1$. Since $m_1$ is the probability that an observation belonging to the first group is selected, and $m_2$ is the probability that an observation belonging to the second group is selected, when the samples are disproportionate the logistic function is shifted as follows:

$$ln\left[{\displaystyle \frac{p\left(x\right)}{1-p\left(x\right)}}*{\displaystyle \frac{m_2}{m_1}}\right]=\sum _{n=0}^N{\beta }_n{x}_n$$

(9)

When $m_1=m_2$ the logistic function cuts on the x-axis, at the value 0.5, as seen in Figure 5. Now, if $m_1=0.2$ and $m_2=0.8$ the curve shifts and cuts on the x-axis at 0.8, as seen in Figure 6.

Figure 5. Logit Function.

Figure 5. Logit Function.

Therefore, the disproportionality of the samples only affects the constant term of the model and one has that

$$p\left(x\right)={\displaystyle \frac{exp(\gamma +{\beta }^{t}x)}{1+exp(\gamma +{\beta }^{t}x)}}$$

(10)

where $\gamma =-ln\left(m^{\prime }\right)$, $m^{\prime }={\displaystyle \frac{m_2}{m_1}}$.

[13]

Figure 6. Shifted Logit Function.

Figure 6. Shifted Logit Function.

3.2. Extreme Gradient Boosting Training

XGBoost (Extreme Gradient Boosting) is a machine-learning algorithm introduced by Chen and Guestrin in 2016. It uses the concept of tree gradient boosting to improve its performance and speed. XGBoost was designed to reduce overfitting by introducing regularization parameters. Gradient boosting trees use regression trees in a sequential learning process as weak learners. These regression trees are similar to decision trees, but they assign a continuous score to each leaf that is then summarized to provide the final prediction [5].

Several hyperparameters are relevant when it comes to training a model. Some of them include the learning rate, column subsampling, and regularisation rate. Additionally, subsampling (which involves bootstrapping the training sample), the maximum depth of the trees, the minimum weights on the children’s scores to split, and the number of estimators (trees) are also commonly used to address the bias-variance-compensation. While higher values for the number of estimators, regularisation, and weights on secondary grades are associated with reduced overfitting, learning rate, maximum depth, subsampling, and column subsampling should have lower values to achieve reduced overfitting. However, setting extreme values for any of these hyperparameters can lead to model misfits.

3.2.1. Hyperparameter Selected and Tuning

Hyperparameters are settings or configurations of the methods (models), which are freely selectable within a certain range and influence model performance (quality).

Grid search in XGBoost is an optimization technique that seeks to find the set of hyperparameters that yields the most accurate predictive model. It operates by defining a grid of hyperparameter values and evaluating the model’s performance for each combination of these values. This process is facilitated by the use of cross-validation, typically k-fold cross-validation, to assess the performance of the model on different subsets of the training data, thereby ensuring that the model’s performance is robust and not overly dependent on the particularities of one set of training data. [2]

The hyperparameters commonly tuned in XGBoost through grid search include $max\_depth$, $min\_child\_weight$, $gamma$, $subsample$, $colsample\_bytree$, and $learning\_rate$ ($eta$). The grid search process evaluates the model for each combination of hyperparameters in the grid, which can be computationally intensive but is necessary for identifying the optimal parameters that minimize overfitting and maximize predictive performance.

3.2.2. XGBoost Hyperparameter Nrounds

The parameter $nrounds$ specifies the number of boosting steps and takes values between $[1,\infty [$ where only integer values are valid. Since a tree is created in each individual boosting step, $nrounds$ also controls the number of trees that are integrated into the ensemble as a whole. Its practical meaning can be described as follows: larger values of $nrounds$ mean a more complex and possibly more precise model, but also cause a longer running time.

3.2.3. XGBoost Hyperparameter Eta

The parameter $eta$ is a learning rate and is also called "shrinkage" parameter and takes values between $[0,1]$. It controls the lowering of the weights in each boosting step. It has the following practical meaning: lowering the weights helps to reduce the influence of individual.

3.2.4. XGBoost Hyperparameter max_depth

The $max\_depth$ hyperparameter in XGBoost refers to the maximum depth of a tree and takes values between $[0,n]$ where only integer values are valid. It is used to control how deep the decision trees within the model can grow during any boosting round. A deeper tree can model more complex patterns in the data, but it also increases the risk of overfitting. The default value is typically set to 6, but it can be adjusted depending on the complexity of the task and the amount of data available.

3.2.5. XGBoost Hyperparameter min_child_weight

Like gamma and maxdepth, $min\_child\_weight$ restricts the number of splits of each tree and takes values between $[0,\infty [$.

In the case of $min\_child\_weight$, this restriction is determined using the Hessian matrix of the loss function.

3.2.6. XGBoost Hyperparameter Subsample

In each boosting step, the new tree to be created is usually only trained on a subset of the entire data set, similar to random forest. The $subsample$ parameter specifies the portion of the data approach that is randomly selected in each iteration and takes values between $]0,1]$. Its practical significance can be described as follows: an obvious effect of small $subsample$ values is a shorter running time for the training of individual trees, which is proportional to the $subsample$.

3.2.7. XGBoost Hyperparameter colsample_bytree

The parameter $colsample\_bytree$ is the number of features is chosen for the splits of a tree and takes values between $]0,1]$. In XGBoost this choice is made only once for each tree that is created, instead for each split. Here $colsample\_bytree$ is a relative factor. The number of selected features is, therefore, $colsample\_bytree\times n$. $colsample\_bytree$ enables the trees of the ensemble to have a greater diversity. The runtime is also reduced, since a smaller number of splits have to be checked each time (if $colsample\_bytree<1$).

3.2.8. XGBoost Hyperparameter Lambda

The parameter $lambda$ is used for the regularization of the model. This parameter influences the complexity of the model and takes values between $[0,\infty [$. Its practical significance can be described as follows: as a regularization parameter, $lambda$ helps to prevent overfitting. With larger values, smoother or simpler models are to be expected.

3.3. Artificial Neural Networks Training

Training a neural network revolves around the following objects [7]:

• Layers, which are combined into a network (or model)
• The input data and corresponding targets
• The loss function, which defines the feedback signal used for learning
• The optimizer, which determines how learning proceeds

3.3.1. Building the Neural Networks

When feeding data into a neural network, it’s important to first apply one-hot encoding to the categorical variables. This means that for a variable with n categories, you would create $n-1$ dummy variables of 0s and 1s. After that, it’s essential to standardize the data so that all variables have the same scale. This standardized data is then used as the input for the first layer of the neural network. As for the Y variable, it is kept numerical with 1s and 0s.

A type of network that performs well on binary classification problem is a simple stack of fully connected ("dense") layers [7].

There are two key architecture decisions to be made about such stack of dense layers:

• How many layers to use?
• How many hidden units to choose for each layer?

The intermediate layers will use relu as activation function, and the final layer will use a sigmoid activation to output a probability (a score between 0 and 1, indicating how likely the sample is to have the target “1": that is, how likely the review is to be positive). A relu (rectified linear unit) is a function meant to zero out negative values, whereas a sigmoid "squashes" arbitrary values into the [0, 1] interval, outputting something that can be interpreted as a probability.

When setting up a neural network, it’s important to select a loss function and an optimizer. For a binary classification problem with network output as probabilities, it’s best to use the binary cross-entropy loss. Cross-entropy is a reliable choice for models that deal with probabilities, as it measures the distance between probability distributions or, in this case, the actual distribution and its predictions [7].

The optimizer of choice is Adam, (Adaptive Moment Estimation), Adam adjusts the neural network weights more efficiently by calculating adaptive learning rates for each parameter. It uses first and second-moment estimates of the gradients (i.e., the mean and non-centred variance) to perform parameter updates.

3.3.2. Adding Dropout

Dropout is a widely used regularization technique for neural networks, developed by Geoffrey Hinton at the University of Toronto. When dropout is applied to a layer during training, a certain number of output features are randomly set to zero [7]. For example, if a layer would normally return the vector $[0.2,0.5,1.3,0.8,1.1]$ for a given input sample during training, applying dropout might result in a vector like $[0,0.5,1.3,0,1.1]$.

The dropout rate is the fraction of features that are zeroed out, typically set between 0.2 and 0.5. During testing, no units are dropped out; instead, the layer’s output values are scaled down by a factor equal to the dropout rate to balance the fact that more units are active than at training time. The technique may seem strange and arbitrary, but why would it help reduce over-adjustment? Hinton says he was inspired, among other things, by a fraud prevention mechanism used by banks.

In his own words:“I went to my bank. The tellers kept changing and I asked one of them why. He said he didn’t know, but they changed them a lot. I assumed it must be because it would take cooperation among the employees to get the bank to cheat. This made me realize that randomly removing a different subset of neurons in each example would prevent conspiracies and thus reduce over-fitting" [7]. The central idea is that by introducing noise into the output values of a neural network layer, you can break random patterns that are not meaningful (what Hinton calls conspiracies), which the network will start to memorize if there is no noise.

4. Interpretability

4.1. Interpretation of Logistic Regression Coefficients

The estimated coefficients ${\beta }_n$ in a regression can be better understood by considering the concept of relative risk. Relative risk is the ratio of the probability of an event occurring (p) to the probability of it not occurring ($1-p$), also known as odds ratios. Odds ratios indicate how much the odds change per unit change in the explanatory variables[13].

The exponential of ${\beta }_n$, $exp\left({\beta }_n\right)$, represents the relative risk, which measures the influence of the variables $x_n$ on the risk of the event occurring, assuming all other variables in the model remain constant. Once the values of ${\beta }_n$ have been estimated, we can determine the probability of the event for different values of $x_n$.

The coefficients of logistic regression are not as easy to interpret as those of linear regression. While the ${\beta }_n$ coefficients are useful for model validation tests, $exp\left({\beta }_n\right)$ is easier to interpret. $exp\left({\beta }_n\right)$, represents the change in the odds ratio for each one-unit change in the variable $x_n$.

For example, take the variable V4 ($cp\_pl\le 0.77$) in the No Arrears Segment Model. It means “individuals whose value of instalments paid over time is up to 0.77". Its estimated coefficient ${\beta }_4$ is 1.612, so $exp\left({\beta }_4\right)$ = 5.014 indicates that the odds ratio of “individuals whose value of instalments paid over time is up to 0.77" (See Table A4from ), is 5.014 times higher than other customers if all other variables are held constant. In Other words, the probability that individuals with $cp\_pl\le 0.77$ will make a payment next month is 5.014 times higher than others.

4.2. Interpretation of XGBoost Models Results

XGBoost is often considered a “black box" algorithm because, while it is highly effective at making accurate predictions, it can be difficult to understand how it arrives at these predictions. This is due to the complexity of the decision tree models it creates and how these trees are combined to form the final model.

Machine learning models like XGBoost, which utilize ensemble and boosting techniques, generate multiple decision trees during the training process. Each tree is constructed to fix the errors of the previous one, leading to a final model that is a weighted sum of many trees [5].

Because of this combination of models and their interactions, it can be challenging to precisely determine which features are influencing the predictions and how they are doing so. Nevertheless, ongoing efforts are being made to enhance the interpretability of these models, including the use of feature importance techniques, SHAP values, and tree visualizations, which can provide insight into model decisions [16].

The significance of variables in the XGBoost algorithm pertains to the impact of each feature in the dataset on the accuracy of the model. From an interpretability standpoint, this helps in understanding which variables carry the most weight in the decisions made by the model and how each influences the final result.

In XGBoost, the importance of variables can be assessed in various ways, including information gain, coverage, or frequency of occurrence of a feature in the decision trees. These metrics offer a clear understanding of the relevance of each variable and enable data scientists and analysts to make well-informed decisions regarding feature selection and model optimization.

Gain represents the average contribution of a feature to model improvement each time it is utilized in a tree. A higher value signifies the feature’s greater importance in making splits that enhance model performance.

Weight refers to the frequency of a feature’s appearance in all trees of the model. A feature with a higher weight is deemed more significant.

Cover measures the frequency of a feature’s utilization in the trees, weighted by the amount of data passing through those splits. A high coverage suggests that the feature substantially impacts the model’s predictions.

Figure 7 shows the 10 variables that, in terms of gain, contribute to the splits that improve the performance of the segment model without arrears.

It has been noted that for the 0 - No Arrears Segment (Figure 7), the variables with the most significant influence on the calculation of the probability of payment in the next month are cp_pl, saldo_cuota_credito, and ctr_pl.

About Neural Networks Interpretation, like XGBoost, are often considered "black boxes" due to their complexity and the way they process information. Despite this, researchers have developed various techniques to help interpret how neural networks make decisions [19].

In this paper we do not address interpretation in neural networks and leave it as an open topic for future work.

5. Results and Discussion

Based on the data in Table 3, it appears that both XGBoost and Artificial Neural Networks (ANN) tend to outperform Logistic Regression (LR) in some segments. However, the superiority of one model over the other may depend on the specific segment.

• In Segment 0 Models, XGBoost (63.36%) and ANN (61.84%) outperform LR (56.42%).
• In Segment Model 1 - 30, XGBoost (51.38%) and ANN (50.35%) also outperform LR (47.32%). However, in the 31 - 90 Segment Model, ANN (38.77%) outperforms LR (36.62%), but XGBoost (34.47%) does not.
• Finally, in the All Segments Model, both XGBoost (74.05%) and ANN (73.59%) outperform LR (71.01%).

These findings suggest that XGBoost and ANN outperform LR in predicting events. It’s essential to consider that these results can vary based on the data characteristics, the quality of feature engineering, and the hyperparameters of the models, among other factors. Additionally, while XGBoost and ANN offer higher accuracy, they may also be more intricate and computationally demanding compared to LR. Hence, the choice of model might depend on balancing accuracy and computational efficiency, as well as the specific requirements of the prediction task.

Lastly, it is crucial to note that these results are specific to this dataset and cannot be generalized to other datasets or prediction tasks. Therefore, it is good practice to cross-validate and fine-tune the hyperparameters for each model and dataset.

6. Conclusions

• This paper compares three supervised learning models: logistic regression, XGBoost, and artificial neural networks, using the Kolmogorov-Smirnov (KS) statistic as a performance metric. The XGBoost algorithm consistently demonstrated superior performance across various segments, achieving accuracy rates of 63.36% for segments with no lags, 51.38% for segments with 1-30 lags, and 74.05% when all segments were analyzed together. These results indicate that XGBoost is more effective for binary classification compared to both logistic regression and neural networks.
• Although logistic regression requires more time for preprocessing and training due to feature engineering and sample balancing, its performance in terms of KS did not outperform XGBoost and neural networks in any of the arrears segments.
• In the 31-90 arrears segment, neural networks outperformed XGBoost with a 38.77% KS value, indicating that the complexity and adaptability of neural networks can be advantageous in certain scenarios, despite the longer training time required.
• The interpretation of XGBoost results relies on how much each variable contributes to the splits in the random trees. In contrast, neural networks are still being studied to achieve satisfactory interpretability. This indicates that when developing a scoring model, one must choose between interpretability and predictability. If the goal is to enhance predictive or discriminative power, XGBoost or neural networks are preferred options. These algorithms can be effectively utilized during the placement, servicing, and collection phases of the credit cycle, as there is a larger volume of data to analyze. This is particularly relevant in collections, where results can change rapidly.
• Future work involves exploring combinations of models to leverage the individual strengths of algorithms like XGBoost and neural networks, aiming to improve prediction accuracy across different segments or scenarios.