An Enhanced Tree Ensemble for Classification in the Presence of Extreme Class Imbalance

1. Introduction

The challenge of dealing with imbalanced problems in the field of machine learning has been a significant topic for a long time. Imbalanced problems, which have an uneven distribution of classes, can lead to biased and subpar results, affecting the accuracy of classification models. To tackle this issue, researchers have developed various methods, including sampling methods and specialized algorithms [1,2,3,4]. Oversampling techniques such as extreme anomalous oversampling technique (EXOT) [5], which synthesizes positive instances based on anomalous scores is an example of such methods. Another strategy involves increasing minority samples while maintaining low misclassification rates for the majority class [6]. Additionally, imbalanced problems can be tackled using re-sampling algorithms such as spread sub-sample, Class Balancer, SMOTE, and Resample [7]. However, the effectiveness of different sampling procedures may vary depending on the specific dataset and the nature of the problem [8,9]. In the context of cancer diagnosis, imbalanced problem is often encountered, and balancing techniques have shown significant improvements in classifier performance. It's crucial to utilize customized approaches because different cancer datasets respond differently to various balancing strategies and classifiers [10].

The performance of machine learning classifiers can be negatively impacted when working with imbalanced problems. The distribution of the training data greatly affects their performance [11]. In a study conducted by [12], several techniques for dealing with class imbalance, including the Synthetic Minority Oversampling Technique (SMOTE) in conjunction with Tomek links and Wilson’s Edited Nearest Neighbor Rule (ENN), were introduced for balancing datasets. However, a systematic comparison of these techniques with optimal tree forest classifiers is currently lacking in the available literature. Similarly, studies in [13,14] have assessed the performance of individual classifiers, such as One Class SVM, Cost-Sensitive SVM, Optimum Path Forest, and decision tree classifier (C4.5 Tree), in the context of imbalanced data.

The study by [15] investigates ensemble techniques, such as the random forest classifier and over-sampling methods, to address class imbalance. Moreover, the study given in [16] addresses the effects of class imbalance on forest point cloud data using a weighted random forest classifier. Most studies suggest that oversampling techniques outperform under-sampling techniques based on assessments made via area under the ROC curve (AUC). While some research [17,18] suggest that oversampling may not always impact performance, it is noted that more complex decision trees can be generated through oversampling. Specialized tree construction methods, such as minority condensation trees, have been proposed to enhance the performance of Random Forest in imbalanced environments [19]. Additionally, incorporating resampling techniques can improve the efficiency of ensemble methods like Random Forests, although these improvements may not always be statistically significant [20]. It has been shown that combining algorithmic strategies with resampling techniques can produce good results [21]. While the choice of splitting indices (GINI index or information gain) does not seem to impact accuracy for balanced or imbalanced data [22], other methods have shown enhanced results. For example, balanced bagging, an ensemble technique, improves the geometric mean, AUC scores, and accuracy of decision tree classification for imbalanced problems [23]. Additionally, using hybrid sampling, which combines techniques like random oversampling and under-sampling, can create balanced datasets that enhance the performance of C4.5 decision trees on imbalanced problems [24]. While there hasn't been a thorough investigation into the most effective combination of the implementation of decision tree and random forest classifiers with hyper-parameter tuning and resampling techniques, there hasn't been a thorough investigation into the most effective combination of these elements in the presence of extreme class imbalances [25].

The concept of optimal tree ensembles (OTE) has become popular in machine learning due to their ability to improve predictive performance and model robustness. Among the various techniques used, out-of-bag (OOB) samples and sub-sampling are crucial for optimizing tree ensembles. Many research papers [26,27] explore the effectiveness of using balanced data and optimal tree forest methods, specifically by utilizing OOB estimates and sub-sampling. In their paper, [28] introduces an improved random forest algorithm that assigns weights to decision trees based on OOB errors. OOB errors can be utilized to estimate the model's performance without needing a separate validation set [29]. The use of OOB observations in modified tree selection techniques for optimal trees ensemble (OTE) is examined by [30] who propose that using OOB observations can improve predictive accuracy for individual and group performance evaluations. Similarly, [31,32] discuss the Modified Balanced Random Forest (MBRF) algorithm, which employs under-sampling based on clustering techniques. This approach differs from the OOB and sub-sampling methods mentioned earlier. To tackle imbalanced data, [33,34,35] propose a random forest algorithm based on Generative Adversarial Networks (GANs), presenting a unique alternative to OOB and sub-sampling. Furthermore, [36] combines the skew-insensitivity of Hellinger distance decision trees with the diversity of Random Forest and Rotation Forest, providing another perspective for optimizing tree-based methods for imbalanced data. In conclusion, the literature suggests that OOB estimates and sub-sampling are valuable techniques in the context of balanced data and optimal tree forests, and can enhance the performance of tree-based models, especially when dealing with imbalanced datasets.

To this end, addressing extreme class imbalance in datasets can be effectively achieved through the use of ensemble tree-based classifiers like Random Forests, potentially enhanced by resampling methods or specialized tree construction techniques. The best approach may differ depending on the specific dataset and context, but a combination of algorithmic and data-level interventions seems to be a promising strategy for improving classifier performance in the presence of class imbalances.

Random forest [37,38], provided an upper bound for the overall prediction error as a function of the accuracy and diversity of the base tree models: i.e.,

$$\mathsf{\epsilon }\le \widehat{\mathsf{\rho }}{\mathsf{ϵ}}_{\mathbf{t}}$$

(1)

where $\mathsf{\epsilon }$ is the total prediction error of the forest, ${\mathit{ϵ}}_{\mathit{t}}$ is the estimate of any$\mathit{t}$tree in the forest, and $\widehat{\mathit{\rho }}$ is the weighted correlation between the residuals from two independent classification trees expressed as the mean correlation over the entire random forest. Therefore, developing a tree ensemble that ensures the accuracy and diversity of the base model is believed to yield promising results. Therefore, this study proposes an ensemble method that uses the above concepts to learn effectively from datasets with class-imbalanced problems. First, the given training data is balanced by generating new synthetic observations for the minority class. Consequently, datasets with extreme class imbalances are considered in this study. These data sets are obtained from several publicly available sources. Once the dataset is balanced, classification trees are grown on bootstrap/sub-samples from the data. The training prediction accuracy of each tree is estimated, and the top-performing trees are selected for the final ensemble. Based on the above notion, this study attempted to increase the accuracy of individual tree models in forests, in addition to randomizing their construction. The accuracy is increased by balancing the given training data with the model selection based on individual prediction performance.

The main contributions of this paper are:

• Mitigating the impact of extreme class imbalance on the random forest ensemble.
• Generating synthetic data from the minority class observations during the training of the tree forest.
• Exploring the concept of tree selection in combination with data balancing to achieve an overall improved ensemble.

The remainder of this article is organized as follows. The proposed methods are given in Section 2, and the experiments and results are described in Section 3. Section 4 presents the simulations, which include both of the simulated scenarios for the proposed method. The final section concludes by summarizing the main findings and providing suggestions for future work.

2. Materials and Methods

2.1. Balancing the Training Data

Let Υ = (X, Y) represent the given training dataset, where X is an n×p matrix, i.e., X $=[{\mathbf{x}}_{\mathbf{i}\times \mathbf{j}}{]}_{\mathbf{n}\times \mathbf{p}}$ where, $\mathbf{i}=1,2,3,...,\mathbf{n},$ and $\mathbf{j}=1,2,3,...\mathbf{p}$, where, p represents the number of features and n represents the number of samples. Y is a binary vector of length n, with elements in {0,1}. These numbers represent the binary response variable, with 1 indicating membership in one class and 0 indicating membership in the other class.

Assume that the given training data come with a severely skewed class distribution, i.e., ${\mathbf{n}}^1\gg {\mathbf{n}}^0$. To balance the training data before building the tree ensembles, the following procedure is considered. The dataset, i.e., Υ is balanced by adding ${\mathcal{K}}_{\mathbf{b}}$ observations, where, ${\mathcal{K}}_{\mathbf{b}}$=${\mathbf{n}}^1-{\mathbf{n}}^0$ by selecting bootstrap samples, each having the size of the minority class observations, i.e., ${\mathbf{n}}^0$. Let the sample be ${\mathbf{X}}_{\mathbf{v}}$ , where v = 1, 2, …, ${\mathcal{K}}_{\mathbf{b}}$. Each bootstrap sample has ${\mathbf{n}}^0$ observations and p features. The bootstrap samples included both continuous and categorical features; computing the means and modes of the features give a vector of observations, i.e., $\mathbf{p}={\mathbf{p}}_1+{\mathbf{p}}_2$. Each bootstrap sample is employed to add a row to the data by calculating the mean ${\left\{\overline{{\mathit{u}}_{\mathit{i}}}\right\}}_{\mathit{i}=1}^{\mathit{p}1}$ of each numeric column and the mode ${\left\{{\tilde{\mathit{u}}}_{\mathit{i}}\right\}}_{\mathit{i}=1}^{\mathit{p}2}$ of each categorical column. In the generated matrix, the rows represented the bootstrapped samples, and each row contained elements arranged in the original order of features. The last column of the matrix contained the class labels of the minority class, denoted as, ${\mathbf{X}}_{\mathbf{n}\mathbf{e}\mathbf{w}}^{\mathbf{r}}$, where r = 1, 2, …, p, that is,

$${\mathbf{X}}_{\mathbf{n}\mathbf{e}\mathbf{w}}^{\mathbf{r}}=({\left\{\overline{{\mathit{u}}_{\mathit{i}}}\right\}}_{\mathit{i}=1}^{\mathit{p}1},{\left\{{\tilde{\mathit{u}}}_{\mathit{i}}\right\}}_{\mathit{i}=1}^{\mathit{p}2}$$

(2)

In this way, a new vector is generated, denoted as $\widehat{\mathbf{X}}$.

$$\acute{\mathsf{{\rm Y}}}=(\widehat{\mathbf{X}},\mathbf{Y}=0{)}_{{\mathbf{n}}^{\mathbf{o}}\times \mathbf{p}},$$

(3)

It follows from Equation 3 that the given training data is combined with the generated data ($\acute{\mathsf{{\rm Y}}}$) to create a balanced dataset ($\widetilde{\mathsf{{\rm Y}}}$).

$$\widetilde{\mathsf{{\rm Y}}}= \mathsf{{\rm Y}} \cup \acute{\mathsf{{\rm Y}}}.$$

(4)

For each class, there is an equal number of data points in $\widetilde{\mathsf{{\rm Y}}}$ as shown in Equation 4. In order to grow optimal trees, balanced data ($\widetilde{\mathsf{{\rm Y}}}$) will be used instead of the original data ($\mathsf{{\rm Y}}$). Two methods are used to grow and select optimal tree using this balanced data ($\widetilde{\mathsf{{\rm Y}}}$). The first method utilized the corresponding OOB observations to assess each tree individually. Trees were ranked based on their individual performance using the OOB observations. The second method involved random subsets of training data for tree growth. In this study, balanced data is used in conjunction with an ensemble method, namely the enhanced tree ensemble using ${\mathit{E}\mathit{T}\mathit{E}}_{\mathit{S}\mathit{S}}$ and sub-sample ${\mathit{E}\mathit{T}\mathit{E}}_{\mathit{S}\mathit{S}}$ observations, on balanced data ($\widetilde{\mathsf{{\rm Y}}}$).

2.2. Enhanced Tree Ensembles via Out-of-Bag (${\mathit{E}\mathit{T}\mathit{E}}_{\mathit{O}\mathit{O}\mathit{B}}$) Observations:

The first approach, i.e., ${\mathit{E}\mathit{T}\mathit{E}}_{\mathit{O}\mathit{O}\mathit{B}}$ uses OOB observations for trees grown on the balanced training data $\widetilde{\mathsf{{\rm Y}}}$. From the balanced data $\widetilde{\mathsf{{\rm Y}}}$, we grow a total of T classification trees from the bootstrap samples $\mathbf{B}\mathsf{\tau }$, $\mathsf{\tau }$ = 1,2,3, …,T , where, T is the number of bootstrap samples used in building trees. Let ${\overline{\mathbf{B}}}_{\mathsf{\tau }}$ be the corresponding OOB observations resulted from bootstrap sample ($\mathbf{B}\mathsf{\tau }$). G($\mathbf{B}\mathsf{\tau }$) represents the classification tree that was grown on $\mathbf{B}\mathsf{\tau }$. Using Equation 6, compute ${\widetilde{\mathbf{e}\mathbf{r}\mathbf{r}\mathbf{o}\mathbf{r}}}_{\mathsf{\tau }}$ the OOB error for the tree grown from sample $\mathbf{B}\mathsf{\tau }$,

$${\widetilde{\mathit{e}\mathit{r}\mathit{r}\mathit{o}\mathit{r}}}_{\mathit{\tau }}={\displaystyle \frac{1}{\left|{\overline{\mathit{B}}}_{\mathit{\tau }}\right|}}{\sum }_{{\mathit{x}}_{\mathit{i}}\mathit{ϵ}{\overline{\mathit{B}}}_{\mathit{\tau }}}\mathit{\varphi }\left(\mathit{y}\ne \tilde{\mathit{y}}\right),$$

(5)

where, y is the true class label in the bootstrap sample, i.e.,${\overline{\mathbf{B}}}_{\mathsf{\tau }}$, $\tilde{\mathbf{y}}$ is the corresponding predicted value via classification tree G($\mathbf{B}\mathsf{\tau }$). The $\mathsf{\varphi }$ is an indicator function that takes value 1 if y does not equal to $\tilde{\mathbf{y}}$ otherwise 0 , expressed in Equation 6:

$$\mathsf{\varphi }(\mathbf{y}\ne \tilde{\mathbf{y}})=\left\{\begin{array}{c}1, \mathrm{if} \mathbf{y}\ne \tilde{\mathbf{y}} \\ 0, \mathrm{Otherwise}.\end{array}\right.$$

(6)

After growing the desired number of classification trees (G($\mathbf{B}\mathsf{\tau }$) arearranged in ascending order based on their prediction error (${\widetilde{\mathit{e}\mathit{r}\mathit{r}\mathit{o}\mathit{r}}}_{\mathit{\tau }}$) on OOB observations. The top H trees, that have the lowest error (${\widetilde{\mathit{e}\mathit{r}\mathit{r}\mathit{o}\mathit{r}}}_{\mathit{\tau }}$), are selected. Let the top-ranked (${\mathbf{G}}^{{\mathbf{W}}_1}$), second-top-ranked tree (${\mathbf{G}}^{{\mathbf{W}}_2}$),…and so one, be shown in Equation 7,

$${\mathit{E}\mathit{T}\mathit{E}}_{\mathit{O}\mathit{O}\mathit{B}}={\mathbf{G}}^{{\mathbf{W}}_1},{\mathbf{G}}^{{\mathbf{W}}_2},\dots ,{\mathbf{G}}^{{\mathbf{W}}_{\mathbf{H}}}$$

(7)

A certain number of trees from the above-ranked trees is selected for the final ensemble. The ensemble is then used to predict the new/test data.

2.3. Enhanced Tree Ensembles using Sub-Samples (${\mathit{E}\mathit{T}\mathit{E}}_{\mathit{S}\mathit{S}}$) Observations

The second proposed method, i.e., ${\mathit{E}\mathit{T}\mathit{E}}_{\mathit{S}\mathit{S}}$ used sub-sample based method where trees were grown on sub-samples from the balanced data ($\widetilde{\mathsf{{\rm Y}}}$). Unlike the OOB observations, the remaining observations from each sample acted as test data for evaluating the predictive performance of each corresponding tree. Given that $\mathbf{B}\mathsf{\tau }$,$\mathsf{\tau }$ = 1,2,3, …, T be the random sample of size m < n, let ${\overline{\mathbf{B}}}_{\mathsf{\tau }}$represent the corresponding remaining subset of observations of size n - m. G($\mathbf{B}\mathsf{\tau }$), where, $\mathsf{\tau }$=1,2, …,T, is the classification tree built on $\mathbf{B}\mathsf{\tau }$. It is also assumed that the error of G($\mathbf{B}\mathsf{\tau }$) on ${\overline{\mathbf{B}}}_{\mathsf{\tau }}$is represented by ${\widetilde{\mathbf{e}\mathbf{r}\mathbf{r}\mathbf{o}\mathbf{r}}}_{\mathbf{s}\mathbf{u}\mathbf{b}\mathsf{\tau }}$. On $\mathsf{\tau }$ ,$\mathsf{\tau }$ = 1,2, …,T, for the T classification trees. was used to estimate ${\widetilde{\mathbf{e}\mathbf{r}\mathbf{r}\mathbf{o}\mathbf{r}\mathbf{s}\mathbf{u}\mathbf{b}}}_{\mathsf{\tau }}$ on each tree.

Let the top, second highest, and so on ranked trees be, that is,

$${\mathit{E}\mathit{T}\mathit{E}}_{\mathit{S}\mathit{S}}={\mathbf{G}}^{{\mathbf{W}}_1},{\mathbf{G}}^{{\mathbf{W}}_2},\dots ,{\mathbf{G}}^{{\mathbf{W}}_{\mathbf{H}}}$$

(8)

The remaining procedure is the same as that for (${\mathit{E}\mathit{T}\mathit{E}}_{\mathit{S}\mathit{S}}$). This method might be useful in small-sample situations where one wants to retain large amounts of training data to build trees. This method can also be tuned by selecting the optimal values for the initial number of trees grown and the number of trees selected for the final ensemble. The pseudo-code and flow chart of the proposed ensembles is provided in Algorithm 1 and Figure 1.

Algorithm 1: Pseudo-code of the proposed method.

• Training data $\mathsf{{\rm Y}}$ consisting of ${\mathbf{n}}^1\gg {\mathbf{n}}^0$ observations and p variables;
• ${\mathbf{n}}^1$← Number of majority class observations;
• ${\mathbf{n}}^0$←Number of minority class observations;
• $\widetilde{\mathsf{{\rm Y}}}$←Balanced data
• ${\mathcal{K}}_{\mathbf{b}}$← Bootstrap sample
• If |${\mathit{n}}^0$| < |${\mathit{n}}^1$|; ${\mathcal{K}}_{\mathbf{b}}={\mathit{n}}^1-{\mathit{n}}^0$.
• for $\mathsf{\nu }\leftarrow$ 1 : ${\mathcal{K}}_{\mathbf{b}}$ : do
• Using the training data ($\mathsf{{\rm Y}}$), take a bootstrap sample from the minority class;
• If a feature is continuous, find its mean (${\overline{\mathit{u}}}_{\mathit{i}}$).
• If categorical, find its mode (${\tilde{\mathit{u}}}_{\mathit{i}}$)
• Concatenate the values in Steps 9 and 10 to get a new row arranged according to the original training data.
• Add the new row ($\acute{\mathsf{{\rm Y}}}$) to the training data $\left(\mathsf{{\rm Y}}\right)$.
• Combine the training data ($\mathsf{{\rm Y}}$) with generated data ($\acute{\mathsf{{\rm Y}}}$) to obtain the balanced data ($\widetilde{\mathsf{{\rm Y}}}$)
• end for
• for t $\leftarrow$ 1 : T do
• Take a bootstrap/sub-sample (${\overline{\mathbf{B}}}_{\mathsf{\tau }}$) from balanced training data ($\widetilde{\mathsf{{\rm Y}}}$).
• Store OOB/out of sample observations.
• Grow classification tree (G(B$\mathsf{\tau }$)) on the bootstrap/ sub-sample (${\overline{\mathbf{B}}}_{\mathsf{\tau }}$).
• Use OOB/out of sample observations and estimate prediction error (${\widetilde{\mathbf{e}\mathbf{r}\mathbf{r}\mathbf{o}\mathbf{r}}}_{\mathsf{\tau }}$).
• end for
• Arrange the trees ${\mathbf{G}}^{{\mathbf{W}}_1},{\mathbf{G}}^{{\mathbf{W}}_2},\dots ,{\mathbf{G}}^{{\mathbf{W}}_{\mathbf{H}}}$ in ascending order with respect to OOB/out-of-sample errors.
• Select the top ranked trees ($\mathbf{H}$) as the final ensemble

3. Experiments and Results

This study evaluated the performance of the proposed methods ${ETE}_{OOB}$ and ${ETE}_{SS}$ on extremely imbalanced classification datasets using a set of benchmark problemsThe effectiveness of these methods was compared to several state-of-the-art approaches, including optimal tree ensemble (OTE) [39,40], SMOTE random forest (${RF}_{SMOTE}$) [41,42], over-sampling random forest (${RF}_{OS}$), under-sampling random forest (${RF}_{US}$) [43], k-nearest-neighbor (k-NN) [44], support vector machine (SVM) [45], tree, and artificial neural network (ANN) [46].

The method, i.e., ${ETE}_{OOB}$ involves growing 1500 trees using bootstrap samples from the training data, and using OOB observations for tree assessment. Similarly, for MathType@Translator@5@5@MathML2 (no namespace).tdl@MathML 2.0 (no namespace)

1500 trees are grown using random samples without replacement. Each sample consists of 90% of the training data, with the remaining 10% used for assessing individual trees. The parameter H is set at 20% of T. Various hyperparameters of the individual tree model are tuned using the tune.rpart function from the R package e1071 [47]. The optimal tree depth and the number of splits are determined by testing values of 5, 10, 15, 20, and 25. To optimize the random forest model, we use the tune.randomForest function from the e1071 package to adjust key hyperparameters, including node size (nodesize), the number of trees (ntree), and the size of the feature subset (mtry). We evaluate different values for nodesize(10, 15, 20, 25, 30), ntree (1000, 1500, 2000), and mtry (sqrt(d)). The support vector machine (SVM) uses automatic sigma estimation from the kernlab package [48], with default settings for other parameters. For the k-nearest neighbors’ classifier (kNN), the tune.knn function from the e1071 package is used to select the optimal value for k, ranging from 3 to 15. To ensure consistent comparisons, the same training and testing data were used across all models, including ${ETE}_{OOB}$, ${ETE}_{SS}$, OTE), ${RF}_{SMOTE}$, ${RF}_{OS}$, ${RF}_{US}$, k-NN, SVM, tree, and ANN Model performance was evaluated using two metrics: classification error rate and precision. All experiments were conducted in R version 4.3.1 on a 1.30 GHz Intel Core i5-1235U with 8 GB of memory, running on a 64-bit operating system.

An overview of the datasets is provided in Table 1. The first column displays the acronyms used for the datasets (DS). The second column shows the names of the datasets. The third and fourth columns present the number of instances/observations (n) and features (p) respectively. The fifth column provides the class distribution, the sixth column shows the imbalance ratio ($n^1:n^0$) , and the last column gives the data source. The results from various training and testing scenarios are presented in Table 2, Table 3, Table 4 and Table 5.

The training and testing datasets were consistent across all methods under consideration. The results, summarized in Table 2, Table 3, Table 4 and Table 5, along with the box plots in Figure 1, Figure 2, Figure 3, Figure 4, Figure 5, Figure 6, Figure 7, Figure 8, Figure 9 and Figure 10, provide a comprehensive comparison of the proposed methods, ${ETE}_{OOB}$ and ${ETE}_{SS}$, against other techniques in terms of classification error rate and precision. Table 2 highlights the classification error rates of the proposed methods in comparison with other state-of-the-art approaches. The results for ${ETE}_{OOB}$ and ${ETE}_{SS}$ are shown in bold. The data clearly indicate that the proposed methods, ${ETE}_{OOB}$ and ${ETE}_{SS}$, consistently outperform the other methods in terms of classification error rate, achieving the lowest error rates across most datasets. The proposed methods exhibited minimal classification error rates, ranging from 0 to 0.0005. Generally, ${ETE}_{OOB}$ performs well on the ${DS}_1$, ${DS}_2$, ${DS}_3$, ${DS}_8$, ${DS}_9$, ${DS}_{10}$, ${DS}_{12}$, ${DS}_{14}$, ${DS}_{17}$, ${DS}_{18}$ and ${DS}_{19},$respectively, while ${ETE}_{SS}$ performs better on the ${DS}_4$, ${DS}_6$, ${DS}_7$, ${DS}_{11}$, ${DS}_{13}$, ${DS}_{15}$, ${DS}_{16}$, respectively. However, ${ETE}_{OOB}$ and ${ETE}_{SS}$ did not perform well on ${DS}_5$ and ${DS}_{20}$, respectively. In contrast, ${RF}_{SMOTE}$ achieved a low error rate on ${DS}_5$, while ${RF}_{OS}$, ${RF}_{US}$, k-NN, SVM, tree, and ANN did not perform well on any of the datasets.

Table 3 offers a detailed comparison of the proposed methods with other state-of-the-art techniques across various datasets, focusing on precision.${ETE}_{OOB}$ delivered better results for the majority of the datasets in terms of precision. Table 4 further reinforces these findings, showing that ${ETE}_{OOB}$ achieved precision rates ranging from 93.36% to 100% across several datasets, underscoring the effectiveness of the proposed methods. In contrast, ${ETE}_{SS}$ provided better precision results for ${DS}_4$, ${DS}_7$, ${DS}_8$, ${DS}_9$, ${DS}_{11}$, ${DS}_{14}$, ${DS}_{16}$, ${DS}_{17}$, ${DS}_{20}$, respectively. Additionally, ${RF}_{SMOTE}$ demonstrated high precision in the ${DS}_5$ datasets. The box plots provide visual confirmation of the superior performance of ${ETE}_{OOB}$ and ${ETE}_{SS}$ in terms of classification error rate and precision, as shown in Figure 1, Figure 2, Figure 3, Figure 4, Figure 5, Figure 6, Figure 7, Figure 8, Figure 9 and Figure 10. Furthermore, for Table 4 and Table 5 same conclusion could be drawn for the classification error rate and precision, with 90% training data and 10% testing.

A multi-line plot demonstrates the consistent improvement in performance and reduction in error rates achieved by the proposed methods, ${ETE}_{OOB}$ and ${ETE}_{SS}$, as the number of trees varies. These figures analyze the impact of different numbers of trees (H) within the ensemble on the error rate. The x-axis represents H as a percentage (indicating the proportion of trees selected for inclusion in the ensemble), while the y-axis shows the corresponding classification error rate. Figure 11 illustrates that increasing the number of trees in the ensemble consistently results in a reduction in the error rate on ${DS}_1$, ${DS}_2$, ${DS}_3$, ${DS}_4$, ${DS}_5$, ${DS}_6$ and ${DS}_7$, respectively. Similarly, Figure 12 shows that the error rate for ${ETE}_{SS}$ decreases as the number of trees in the ensemble increases. Furthermore, Figure 11 and Figure 12 support these findings by illustrating the relationship between the percentage of trees selected (10%, 30%, 50%, 70%, 90%) for the ensemble (x-axis) and the resulting error rates (y-axis). These figures collectively emphasize the importance of optimizing the number of trees in an ensemble to achieve better classification performance.

4. Simulation

In this section, we present two scenarios for creating scientific datasets for simulation. The first scenario involves generating a synthetic imbalanced dataset for simulation (${\mathit{S}\mathit{I}\mathit{D}\mathit{S}}_{\mathit{s}\mathit{i}\mathit{m}}$), while the second scenario involves creating a synthetic balanced dataset for simulation (${\mathit{S}\mathit{B}\mathit{D}\mathit{S}}_{\mathit{s}\mathit{i}\mathit{m}}$). We then evaluate the performance of the proposed method, i.e., ${\mathit{E}\mathit{T}\mathit{E}}_{\mathit{O}\mathit{O}\mathit{B}}$ and ${\mathit{E}\mathit{T}\mathit{E}}_{\mathit{S}\mathit{S}}$ using these datasets. The first scenario aims to demonstrate when the proposed method is beneficial, while the second scenario showcases a data-generating environment where the proposed method may not be appropriate. In total, 10,000 instances were synthetically generated, involving 19 variables. Each of these variables produces observations following a multivariate normal distribution with varying means and variances, except for 8 variable which generates observations following a multi-nomial distribution with four categories. To generate observations for a binary response, the first imbalance ratio specifies the imbalance in the observations. The binary response , denoted as $\mathbf{Y}=\mathbf{B}\left(\mathbf{p}\right)$, is generated using a logit-type function given the variables as shown in Equation 9 .

$$\mathbf{p}({\mathbf{y}}_{\mathbf{i}\mathbf{m}\mathbf{b}}|\mathbf{X})={\displaystyle \frac{\mathbf{e}\mathbf{x}\mathbf{p}\left({\mathbf{y}}_{\mathbf{i}\mathbf{m}\mathbf{b}}\right)}{1+\mathbf{e}\mathbf{x}\mathbf{p}(1-{\mathbf{y}}_{\mathbf{i}\mathbf{m}\mathbf{b}})}}.$$

(9)

The ratio of the number of instances in the majority class to the minority class observations was 5.67. There were 8500 instances in the majority class and 1500 in the minority class. The values used in Equation 2 are as follows:

$$\mathbf{y}=\mathbf{X}\mathsf{\beta }+ϵ.$$

(10)

This setup was used to generate 10,000 observations. All the methods presented in this study were applied using the same experimental setup as that used for the benchmark datasets. The second model was constructed similarly. The class-wise distribution was 50/50, where a class ratio of 1.02 indicated that the data were balanced. The difference between the two models was that the former contained an imbalanced class distribution, whereas the latter did not. A total of 100 realizations were performed. Bar plots of the proposed method on the simulated datasets are shown in Figure 13. The bar-plots results indicate that the proposed method, ${\mathit{E}\mathit{T}\mathit{E}}_{\mathit{O}\mathit{O}\mathit{B}}$and ${\mathit{E}\mathit{T}\mathit{E}}_{\mathit{S}\mathit{S}}$, has lower classification error rates compared to other state-of-the-art methods, especially on ${\mathit{S}\mathit{I}\mathit{D}\mathit{S}}_{\mathit{s}\mathit{i}\mathit{m}}$. These findings suggest that ${\mathit{E}\mathit{T}\mathit{E}}_{\mathit{O}\mathit{O}\mathit{B}}$and ${\mathit{E}\mathit{T}\mathit{E}}_{\mathit{S}\mathit{S}}$ is more effective for classifying imbalanced datasets than other state-of-the-art approaches in the presence of a severe class imbalance problem in the data. In the first scenario, the proposed methods, ${\mathit{E}\mathit{T}\mathit{E}}_{\mathit{O}\mathit{O}\mathit{B}}$and ${\mathit{E}\mathit{T}\mathit{E}}_{\mathit{S}\mathit{S}}$, outperformed the other rivals. However, in the second scenario, ${\mathit{E}\mathit{T}\mathit{E}}_{\mathit{O}\mathit{O}\mathit{B}}$ and ${\mathit{E}\mathit{T}\mathit{E}}_{\mathit{S}\mathit{S}},$ did not perform well due to the absence of a class imbalance problem. The model's performance was affected by the lack of a class imbalance problem, leading to sub-optimal outcomes in the second scenario.

5. Conclusions

The performance of machine learning models can be negatively impacted by a significant imbalance in the distribution of classes, with one class significantly outnumbering the other. To address this issue, balancing the data by creating additional observations for the minority class, typically the focus of interest, in combination with tree ensemble, has led to improved prediction accuracy. The proposed methods, along with synthetic data generation for balancing, effectively tackle the challenges presented by highly imbalanced classification problems. This study introduced two innovative methods, i.e., ${\mathbf{E}\mathbf{T}\mathbf{E}}_{\mathbf{O}\mathbf{O}\mathbf{B}}$ and ${\mathbf{E}\mathbf{T}\mathbf{E}}_{\mathbf{S}\mathbf{S}}$ that successfully address the issue of imbalanced data. These methods were shown to outperform traditional machine learning approaches, such as OTE, SMOTE random forest (${\mathbf{R}\mathbf{F}}_{\mathbf{S}\mathbf{M}\mathbf{O}\mathbf{T}\mathbf{E}}$), over-sampling random forest (${\mathbf{R}\mathbf{F}}_{\mathbf{O}\mathbf{S}}$), under-sampling random forest (${\mathbf{R}\mathbf{F}}_{\mathbf{U}\mathbf{S}}$), k-nearest-neighbor (k-NN), support vector machine (SVM), tree, and artificial neural network (ANN), across multiple metrics in analyses of various benchmark and simulated datasets.

For future work in the direction of this paper, one may consider the use of methods such as SMOTE and Adaptive Synthetic Minority Oversampling Technique (ADASYN) to create synthetic data tailored to the dataset. This can enhance model performance by better balancing the minority class. Additionally, consider developing new features that are particularly useful for the minority class to provide more relevant information for the model and improve its performance. Apply feature selection techniques to identify and use the most relevant features for classification in imbalanced scenarios. Incorporate cost-sensitive learning methods to adjust the model’s focus on the minority class by assigning higher costs to errors made on this class. For better results, combine this approach with the enhanced tree ensemble.

Our method is designed specifically for binary classification problems. However, it's important to note that the time complexity of the method increases exponentially as the data size grows. This challenge can be addressed by using parallel computing techniques, which allow computational tasks to be distributed across multiple processors, improving efficiency and reducing processing time. Implementing parallel computing could greatly improve the scalability of our method, making it more suitable for larger datasets.