A Regularized Tree Forest for Classification in the Presence of Extreme Class Imbalance

1. Introduction

One of the most challenging issues when using a machine-learning method for binary classification is the uneven distribution of classes in the given training data. This problem is referred to as the class imbalance problem [1,2,3,4]. Several studies have been conducted on the issue of class imbalance [5,6,7,8]. Class imbalance problems can be addressed at four levels associated with various learning phases. To this end, ensemble learning, feature selection at the feature level, internal classifier modification at the classifier level, and resampling methods that change the class distributions can be used to tackle the issue. Among the resampling methods, random under-and oversampling is a non-heuristic method [9]. Under-and oversampling techniques are likely to improve learner generalization when datasets are significantly skewed [10,11,12]. Random under-sampling aims to balance the class distributions by randomly removing samples from the majority class. However, under-sampling can result in information loss and poor model performance.

By contrast, random oversampling attempts to equalize class distributions by randomly replicating minority class instances. Two problems arise from random oversampling. It first increases the likelihood of over-fitting because it perfectly replicates the minority class samples [13], and the second generation of a minority class can significantly increase the data size, resulting in a prolonged training time and high computational requirements. The synthetic minority oversampling technique (SMOTE) [14] provides more minority instances, which helps the machine-learning model learn more efficiently and assists in tackling the issue of class imbalance. However, certain disadvantages of SMOTE include its noise sensitivity and dependence on its nearest neighbor. An advanced sampling technique, boosting, was applied to mitigate this issue. Boosting focuses on the incorrectly classified instances. Boosting significantly affects the distribution of the training data. Other methods for finding training data close to the optimal value include heuristic, budget-sensitive, and progressive sampling methods [15,16]. OSPC, an oversampling technique based on preliminary classification, performs better in minority/majority class classification than SMOTE and under-sampling methods.

The goal of each of the aforementioned methods for changing class distributions is to address the issue of between-class imbalance. A cluster-based oversampling method was proposed [17] to simultaneously improve the accuracy of minority classes and address the issues of both within- and between-class imbalances. Both oversampling and under-sampling can be addressed using approaches, such as SMOTE, paired with the Tomek link and ENN [18] when there are only a few samples of the minority class or when the datasets are significantly skewed. Most machine-learning methods rarely yield satisfactory results in real-world applications with class imbalance problems. The key causes of the poor performance of the classifier on imbalanced datasets are accuracy, class distribution, and error costs. Numerous approaches have been proposed to solve these problems and to improve the efficiency of binary classification models [19].

Several other dedicated research endeavors have been conducted to address the class imbalance problem. The authors in [20] have provided a thorough comparison of several existing remedies for mitigating this problem. While most typical learning algorithms can achieve high global accuracy by correctly classifying majority class observations, improving the minority class accuracy remains a significant challenge [21]. To this end, the authors in [22] proposed the idea of a k-fold division of the majority class to generate a pool of classifiers that divides an imbalanced problem into numerous balanced ones, while guaranteeing that all available samples are used in the training process. In the context of ensemble learning, the model performance is significantly associated with the performance of the base classifier, weight computation method, and selection strategy when dealing with class-imbalanced datasets. This notion has led to the development of several ensemble learning methods. For example, the algorithm given in [23], which employs under-sampling of the majority class for classifying imbalanced datasets, has been proposed. Similar methods can be found in the literature that address the issue of class imbalances in binary classification problems. These techniques primarily exploit the idea of under-sampling the majority class observations or oversampling the majority class observations. Both these ideas have certain limitations, and the consequent machine-learning method may still fail to achieve the desired classification performance.

A binary classification model must perform well for both classes to achieve the desired classification performance. In certain situations, when there are insufficient instances in one or both classes, machine-learning algorithms [24] may find it challenging to identify patterns in the data and correctly classify minority class samples because of this imbalance. To address this issue, researchers developed learning algorithms to process and extract information from data with significantly skewed distributions. Specifically, instead of ignoring minority classes in the data, an imbalanced learning technique allows the prediction algorithm to predict the outcome variables with a more realistic level of accuracy. The model was trained to distinguish between instances belonging to two separate classes. Nevertheless, several factors can significantly affect the performance of a binary classification model, including class noise, class imbalance, and insufficient data.

Several tree-based methods have been modified to address class imbalance problems. However, little attention has been paid to simultaneously balancing data and selecting optimal base models for the final tree ensemble. In constructing an efficient ensemble, accuracy, and diversity are the main features that significantly regulate the overall performance of the final model. In the case of random forest, Breiman provided an upper bound for the overall prediction error as a function of the accuracy and diversity of the base tree models: i.e.

$$\mathsf{\epsilon }=\widehat{\mathsf{\rho }}{\mathsf{ϵ}}_{\mathbf{t}}$$

where $\mathsf{\epsilon }$ is the total prediction error of the forest, ${\mathit{ϵ}}_{\mathit{t}}$ is the estimate of any$\mathit{t}$tree in the forest, and $\widehat{\mathit{\rho }}$ is the weighted correlation between the residuals from two independent classification trees expressed as the mean correlation over the entire random forest. Therefore, developing a tree ensemble [25,26,27] that ensures the accuracy and diversity of the base model is believed to yield promising results. In the presence of class-imbalanced problems, allowing the ensemble to learn from balanced data leads to further improvements. Therefore, this study proposes an ensemble method that uses the above concepts to learn effectively from datasets with class-imbalanced problems.

To build an efficient tree ensemble, this study first balanced the given training data by generating new synthetic observations for the minority class. Consequently, datasets with extreme class imbalances were considered in this study. These data sets were obtained from several publicly available sources. Once the dataset was balanced, classification trees were grown on bootstrap/subsamples from the data. The training prediction accuracy of each tree was estimated, and the top-performing trees were selected for the final ensemble. Based on the above notion, this study attempted to increase the accuracy of individual tree models in forests, in addition to randomizing their construction. The accuracy was increased by balancing the given training data with the model selection based on individual prediction performance.

The remainder of this article is organized as follows. The proposed methods are given in Section 2, and the experiments and results are described in Section 3. Section 4 presents the simulations, which include both of the simulated scenarios for the proposed method. The final section concludes by summarizing the main findings and providing suggestions for future work.

2. Materials and Methods

2.1. Balancing the Training Data

Let Υ= (X, Y) be the given data; X is an $\mathbf{n}\times$p matrix, that is, $\mathbf{X}=[{\mathbf{x}}_{\mathbf{i}\times \mathbf{j}}{]}_{\mathbf{n}\times \mathbf{p}}$ where $\mathbf{i}=\mathrm{1,2},3,...,\mathbf{n},$ and $\mathbf{j}=\mathrm{1,2},3,...\mathbf{p}$, and Y are two-class vectors, that is, $\mathbf{Y}\mathsf{ϵ}$ (0,1) of length n. ${\mathit{y}}_{\mathit{i}}$ are the values of the binary response variable with a value 1for one class and a value 0 for the other class; that is,

$${\mathbf{y}}_{\mathbf{i}}=\left\{\begin{array}{c}1, if n={\mathbf{n}}^1\\ 0, if n={\mathbf{n}}^0, \end{array}\right.\mathbf{i}=\mathrm{1,2},3,...,\mathbf{n}.$$

Out of the total n sample points, there were ${\mathbf{n}}^1$majority and ${\mathbf{n}}^0$ minority class observations, with a severely skewed class distribution, i.e., i.e., ${\mathbf{n}}^1\gg {\mathbf{n}}^0$ . The following procedures were considered to balance the training data before building the tree ensembles. The given dataset Υ was balanced by selecting ${\mathcal{K}}_{\mathbf{b}}$=${\mathbf{n}}^1-{\mathbf{n}}^0$ bootstrap samples each of size, i.e. ${\mathbf{n}}^0$, equal to the number of observations in the minority class, from the minority class observations. Let the sample be ${\mathbf{X}}_{\mathbf{v}}$, where v = 1, 2, …, ${\mathcal{K}}_{\mathbf{b}}$ . Each bootstrap sample had ${\mathbf{n}}^0$ observations of p features. Each bootstrap sample was used to generate an additional row in the data by estimating the mean $\overline{\mathbf{u}}$of each column if it was numeric and mode $\tilde{\mathbf{u}}$ if it was categorical. Let there be ${\mathbf{p}}_1$ continuous features and ${\mathbf{p}}_2$ categorical features in the bootstrap samples; calculating the means and modes of the features gives a vector of $\mathbf{p}={\mathbf{p}}_1+{\mathbf{p}}_2$ observations, that is,

$${\mathbf{X}}_{\mathbf{n}\mathbf{e}\mathbf{w}}^{\mathbf{r}}=({\overline{\mathbf{u}}}_1, {\overline{\mathbf{u}}}_2,...,{\overline{\mathbf{u}}}_{{\mathbf{p}}_1},{\tilde{\mathbf{u}}}_1,{\tilde{\mathbf{u}}}_2,\text{…}, {\tilde{\mathbf{u}}}_{{\mathbf{p}}_2},$$

where r = 1, 2, …,p, and the elements in ${\mathbf{X}}_{\mathbf{n}\mathbf{e}\mathbf{w}}^{\mathbf{r}}$ are arranged according to the original order of the features in the bootstrap sample. The generated rows are arranged in the matrix, and the last column comprises the class labels of the minority class, that is,

$$\acute{\mathsf{{\rm Y}}}=(\widehat{\mathbf{X}},\mathbf{Y}=0{)}_{{\mathbf{n}}^{\mathbf{o}}\times \mathbf{p}},$$

where, $\widehat{\mathbf{X}}$ denotes a newly generated vector. The given training data was combined with the generated data $\acute{\mathsf{{\rm Y}}}$ to obtain balanced data, that is,

$$\widetilde{\mathsf{{\rm Y}}}= \mathsf{{\rm Y}} \cup \acute{\mathsf{{\rm Y}}}.$$

For each class, there were equal numbers of data points in $\widetilde{\mathsf{{\rm Y}}}$. To grow optimal trees, the data $\widetilde{\mathsf{{\rm Y}}}$ was used instead of the original data $\mathsf{{\rm Y}}$ . Two methods are used to grow and select optimal trees ensemble using balanced data $\widetilde{\mathsf{{\rm Y}}}$. The first method uses the corresponding out-of-bag (oob) observations to assess each tree individually. Trees were ranked based on how well they performed individually based on oob observations. The second method involved random subsets of training data for tree growth. In this study, balanced data, i.e., $\widetilde{\mathsf{{\rm Y}}}$ was used in conjunction with an ensemble method, i.e., optimal tree ensemble classifier using out-of-bag, i.e., OTEC (oob) and optimal tree ensemble classifier using sub-sample, i.e., OTEC (sub), on balanced data.

2.2. Out-of-Bag based Assessment with Balanced Data

The first proposed method, i.e., OTEC (oob) used out-of bag (oob) observations for trees grown on the balanced training data $\widetilde{\mathsf{{\rm Y}}}$ . Studies [28,29] have observed that samples during bootstrapping omit approximately 1/3 of the overall training data [30]. These observations play no role in model construction and can be used as independent validation data for model assessment. From the balanced data $\widetilde{\mathsf{{\rm Y}}}$ , we took T bootstrap samples, i.e., $\mathbf{B}\mathsf{\tau }$, $\widetilde{\mathsf{{\rm Y}}}$ = 1,2,3, …,T , and let ${\overline{\mathbf{B}}}_{\mathsf{\tau }}$ be the corresponding value of oob observations from each sample. G($\mathbf{B}\mathsf{\tau }$) represents the classification tree that was grown on $\mathbf{B}\mathsf{\tau }$. It was further assumed that ${\widetilde{\mathbf{e}\mathbf{r}\mathbf{r}\mathbf{o}\mathbf{r}}}_{\mathsf{\tau }}$ represented the oob error, which is the error of G($\mathbf{B}\mathsf{\tau }$) on ${\overline{\mathbf{B}}}_{\mathsf{\tau }}$, i.e.,

$${\widetilde{\mathit{e}\mathit{r}\mathit{r}\mathit{o}\mathit{r}}}_{\mathit{\tau }}={\displaystyle \frac{1}{\left|{\overline{\mathit{B}}}_{\mathit{\tau }}\right|}}{\sum }_{{\mathit{x}}_{\mathit{i}}\mathit{ϵ}{\overline{\mathit{B}}}_{\mathit{\tau }}}\mathit{\varphi }\left(\mathit{y}\ne \tilde{\mathit{y}}\right),$$

where, y is the true class label in the bootstrap sample, i.e.,${\overline{\mathbf{B}}}_{\mathsf{\tau }}$, $\tilde{\mathbf{y}}$ is the corresponding predicted value via classification tree G($\mathbf{B}\mathsf{\tau }$), and |${\overline{\mathbf{B}}}_{\mathsf{\tau }}$| is the size of the oob sample. $\mathsf{\varphi }$ is an indicator function, expressed as:

$$\mathsf{\varphi }(\mathbf{y}\ne \tilde{\mathbf{y}})=\left\{\begin{array}{c}1, if y\ne \tilde{\mathbf{y}} \\ 2, Otherwise.\end{array}\right.$$

G($\mathbf{B}\mathsf{\tau }$), After growing the desired number of classification trees, they were arranged in ascending order of their error rates on the oob samples. Let the top-ranked, second-top-ranked, etc. trees be

$${\mathit{G}}^{\mathit{r}1},{\mathit{G}}^{\mathit{r}2},\dots ,{\mathit{G}}^{\mathit{r}\mathcal{Q}}$$

A number of trees from the above-ranked trees were selected for the final ensemble. The ensemble was then used to predict the new/test data. The number of trees grown and the number of trees selected were two potential hyper-parameters for this method.

2.3. Sub-Sample based Assessment with Balanced Data

The second proposed method, i.e., OTEC(sub) used sub-sample based method where trees were grown on sub-samples from the balanced data ($\widetilde{\mathsf{{\rm Y}}}$). Unlike the oob observations, the remaining observations from each sample acted as test data for evaluating the predictive performance of each corresponding tree. Given that $\mathbf{B}\mathsf{\tau }$,$\mathsf{\tau }$ = 1,2,3, …, T be the random sample of size m < n. Additionally, let ${\overline{\mathbf{B}}}_{\mathsf{\tau }}$represent the corresponding remaining subset of observations of size n - m. G($\mathbf{B}\mathsf{\tau }$), where, $\mathsf{\tau }$=1,2, …,T, is the classification tree built on $\mathbf{B}\mathsf{\tau }$ . It is also assumed that the error of G($\mathbf{B}\mathsf{\tau }$) on ${\overline{\mathbf{B}}}_{\mathsf{\tau }}$ is represented by ${\widetilde{\mathbf{e}\mathbf{r}\mathbf{r}\mathbf{o}\mathbf{r}}}_{\mathbf{s}\mathbf{u}\mathbf{b}\mathsf{\tau }}$. On $\mathsf{\tau }$ ,$\mathsf{\tau }$ = 1,2, …,T, for the T classification trees. was used to estimate ${\widetilde{\mathbf{e}\mathbf{r}\mathbf{r}\mathbf{o}\mathbf{r}\mathbf{s}\mathbf{u}\mathbf{b}}}_{\mathsf{\tau }}$on each tree.

Let the top, second highest, and so on ranked trees be, that is,

$${\mathit{G}}^{\mathit{r}1},{\mathit{G}}^{\mathit{r}2},\dots ,{\mathit{G}}^{\mathit{r}\mathcal{Q}}$$

The remaining procedure was the same as that for OTEC(sub). This method might be useful in small-sample situations in which one wants to retain large amounts of training data to build trees. This method can also be tuned by selecting the optimal values for the initial number of trees grown and the number of trees selected for the final ensemble. The pseudo-code of the proposed ensemble is provided in Algorithm 1.

This study considered extremely imbalanced classification datasets to assess the performance of the proposed method for benchmark problems. The efficacy of the proposed method, i.e., OTEC(oob) and OTEC(sub) in comparison with the state-of-the-art methods, i.e., optimal tree ensemble (OTE), random forest in conjunction with using synthetic minority over-sampling technique (RF(smote)), random forest combined with over-sampling method (RF(over)), under-sampling method coupled with random forest (RF(under)), support vector machine, k-nearest neighbor (k-NN), artificial neural network (ANN), and classification tree (Tree) was assessed using 20 benchmark datasets. The evaluation metrics considered were the classification error rate, sensitivity, specificity, precision, recall, and F1 score, which is a measure that combines precision and recall into a single value, providing a balanced assessment of a model’s performance. R programming software was used to perform the experiments.

Table 1 provides a concise summary of the datasets considered. The data name is displayed in the second column and the numbers of instances/observations (n) and features (p) are shown in the third and fourth columns, respectively. The fifth column provides the classwise distribution, the sixth column provides the imbalance ratio ($n^1/n^0$), and the final column contains the data source. Additional information on the remaining datasets is provided in Table S1. This section describes the experimental design of this study.

The proposed methods, OTEC(oob) and OTEC(sub), were applied to extremely imbalanced datasets, with 95% of the observations belonging to the majority class and 5% to the minority class. This indicates that the observations were made at a 19:1 ratio. This was performed by randomly discarding minority class observations in which the original data did not have an uneven ratio of 19:1. A total of 1,000 realizations were made from the resulting data, which were split into 90% training and 10% testing parts. Model fitting was performed on the training part of the data, and evaluation was performed on the testing part. To generate T = 1,000 classification trees, bootstrap samples from 90% of the training data were obtained using the OTEC(oob) and OTEC(sub) methods described in Algorithm 1. The testing part was used for external validation.

The final ensemble contained all of the top $\mathcal{Q}$ trees that were chosen. The final result was the average of all 1,000 runs. The training and testing parts were the same for all methods under consideration. The findings presented in Table 2 and Table 3, along with the box plots displayed in Figure 1, Figure 2, Figure 3 and Figure 4, show the results of the proposed method, that is, OTEC(oob) and OTEC(sub), in terms of the classification error rate and precision compared with the other methods. The results for the rest of the methods, that is, OTEC(oob) and OTEC(sub), in terms of sensitivity, specificity, recall, and F1 score, are given in Tables S2–S5 of Supplementary Materials. The plots are shown in Figures S1–S14 in Supplementary Materials. Table 2 shows the results of the proposed method’s classification error rate for the datasets with those of the other state-of-the-art methods. The OTEC(oob) results are shown in bold numbers, and the OTEC(sub) results are italicized. It is evident from the tables that the proposed methods, that is, OTEC(oob) and OTEC(sub), outperform all other procedures in terms of the classification error rate, having the lowest error rate compared to the other methods. The proposed methods, that is, OTEC(oob) and OTEC(sub), showed minimal classification error rates ranging from 0 to 0.0005. OTEC(sub) did not perform well in breast cancer, drug classification, glass classification, or KDD. By contrast, RF (smote) yielded a low error rate on the glass classification dataset. The RF(under), k-NN, SVM, and ANN did not perform well on any of the datasets. In terms of the classification error rate, OTEC(oob) performed better on 19 out of 20 datasets than the other methods. For 17 of the 20 datasets, OTEC(sub) performed better in terms of classification error rate. However, the other methods failed to yield satisfactory results.

Table 3 presents a detailed comparison of the proposed method with other state-of-the-art methods on the datasets. OTEC(oob) yielded better results for the majority of datasets in terms of precision. The results given in Table 4 show that OTEC(oob), in terms of precision, yields promising results ranging from 93.36% to 100% for several datasets. This demonstrated the effectiveness of the proposed method. In contrast, OTEC(sub) provided better results in terms of precision for 14 datasets, except for the breast, liver disorder, and ionosphere. Moreover, RF (smote) and RF (over) demonstrate high precision in kc2 and glass classification. The box plots revealed the best findings of the proposed method, that is, OTEC(oob) and OTEC(sub), in terms of the classification error rate and precision with other methods given in Figure 1, Figure 2, Figure 3 and Figure 4. A box plot of the remaining datasets in terms of the classification error rate and precision is shown in Figures S1 and S2 of Supplementary Materials.

The parameter W, which represents the percentage of the best trees selected for the final ensemble, is crucial for the OTEC(oob) and OTEC(sub) methods. The diverse values of W indicate the different behaviors of the method. This study assessed the impact of W = (10%, 30%, 50%, 70%, 90%) of the total number of trees on the classification method. As shown in Figure 5, subsets of the total number of trees ranging from 10% to 90% were used to calculate the classification error rate. For each dataset, Figure 5 shows that an increase in the number of trees used by the ensemble decreased the OTEC(oob) classification error rate. Similar to OTEC(oob), Figure 6 shows that the classification error rate of OTEC(sub) decreased with an increase in the number of trees in the ensemble. In Figure 5 and Figure 6, the x-axis represents the percentage of the best trees selected, while the y-axis represents classification error rate results.

3. Simulation

In this section, we present two simulation scenarios for the proposed method. The first scenario is intended to show the circumstances in which the proposed method is useful, whereas the second scenario shows a data-generating setting where the proposed method might not be suitable. In total, 10,000 instances across 19 variables were synthetically generated. Each of these variables generates observations as a multivariate normal distribution with different means and variances, whereas the other generates observations with a multinomial distribution with four categories. To generate observations from a binary response, the first imbalance ratio specifies the imbalance in the observations. The binary response, $\mathbf{Y}=\mathbf{B}\left(\mathbf{p}\right)$, is generated using a logit-type function given the variables $\mathbf{P}\left(\mathbf{y}|\mathbf{X}\right)$:

$$\mathbf{p}\left({\mathbf{y}}_{\mathbf{i}\mathbf{m}\mathbf{b}}\right|\mathbf{X})={\displaystyle \frac{\mathbf{e}\mathbf{x}\mathbf{p}\left({\mathbf{y}}_{\mathbf{i}\mathbf{m}\mathbf{b}}\right)}{1+\mathbf{e}\mathbf{x}\mathbf{p}(1-{\mathbf{y}}_{\mathbf{i}\mathbf{m}\mathbf{b}})}}.$$

The imbalance ratio of the number of instances in the majority class compared to the minority class observations ${\mathbf{n}}^0$ was 5.67, of which 8500 belonged to the majority class and 1500 to the minority class. The values of $\mathbf{y}$ used in Equation 2 are as follows:

$$\mathbf{y}=\mathbf{X}\mathsf{\beta }+$$

This setup was used to generate 10,000 observations. All the methods presented in this study were applied using the same experimental setup as that used for the benchmark datasets. The second model was constructed similarly. The class-wise distribution was 50/50, where a class ratio of 1.02 indicated that the data were balanced. The difference between the two models was that the former contained an imbalanced class distribution, whereas the latter did not. A total of 100 realizations were performed. The proposed methods, OTEC(oob) and OTEC(sub), outperformed the other rivals in the first scenario. In the second scenario, OTEC(oob) and OTEC(sub) did not perform well because there was no class imbalance problem. Bar plots of the proposed method on the simulated datasets are shown in Figure 7 and Figure 8. The results demonstrate that the proposed methods are appropriate in the presence of a severe class imbalance problem in the data.

4. Conclusions

The uneven distribution of observations into two classes, where one of the classes significantly outperforms the other, adversely affects the performance prediction of machine-learning methods. Therefore, balancing the data by synthetically generating additional observations for minority classes, a class that is usually of high interest, along with model selection for a decision-tree ensemble, has resulted in improved prediction performance. The proposed ensemble algorithm, along with synthetic data generation for data balancing, effectively solved the key challenge of extremely imbalanced classification problems.

The two methods proposed in this study address the imbalanced data problem, as shown by the analysis of several benchmark and simulated datasets. The proposed methods outperformed conventional machine-learning tools, such as optimal tree ensemble, random forest with SMOTE, random forest with under-sampling, random forest with oversampling, k-NN, SVM, classification tree, and artificial neural networks, in terms of error rate, sensitivity, specificity, precision, recall, and F1-score. With the help of the proposed method, the performance of machine-learning models on extremely imbalanced binary classification problems can be significantly improved, producing more dependable and robust classification results in practical applications.

For future work in the direction of this paper, one may consider the use of SMOTE and ADASYN (Adaptive Synthetic Minority Oversampling Technique) for data augmentation methods customized to the dataset. Using these methods, the performance of the model can be improved by adding artificial data points for the minority class. Using the active learning strategy, the model may be trained to automatically search for new data points, particularly from the minority class. This will allow the underrepresented class most informative data to be the model’s key focus. Feature engineering methods that can generate new features that are more informative for the underrepresented class in conjunction with the proposed method might improve the performance further. Feature selection could also be used to determine which features are most relevant for classification in an unbalanced context. Other methods, such as cost-sensitive learning, could be combined with the regularized tree forest. Cost-sensitive learning enables the model to concentrate its performance on underrepresented data by providing greater weight to minority class misclassifications.