Effective Hybrid Sampling Approach for Evaluating Classification Performance

1. Introduction

In the process of constructing a classification model, the data sampling method plays a crucial role. The model's learning capability is highly dependent on the quality of the sampled training set, while its accuracy is determined by the test set. If the sampling method results in a poor-quality training set for model construction and an overly simple test set, the model may achieve artificially high accuracy, leading to a misjudgment of its actual performance.

Therefore, during the data sampling phase, it is essential to carefully select samples to ensure their representativeness. This requires adopting appropriate sampling strategies and ensuring diversity within the sample to accurately reflect the overall data distribution. One of the most commonly used sampling methods is random sampling, which is both simple and efficient. In this approach, each data point has an equal probability of being selected, making the sampled dataset a reasonable representation of the entire population. Due to its ease of implementation, random sampling is frequently used in practical applications.

Random sampling has the drawback of generating different training and test sets in each iteration, leading to variations in accuracy results. A single instance of random sampling cannot fully reflect the model's performance, making it unreliable for evaluating classification models when only one training/test split is used. To achieve a more accurate assessment, multiple random samplings are typically required, and the average accuracy across all experiments is used as a measure of overall performance. However, this approach necessitates training multiple models, which is computationally expensive and impractical for real-world applications. Cross-validation offers a solution by mitigating the limitations of single random sampling. By systematically rotating different subsets of data into the training and test sets and averaging the results, cross-validation provides a more comprehensive evaluation of model performance. Given these limitations of random sampling, improving sampling strategies and efficiently selecting appropriate training and test sets have become crucial research topics in the field [7,8,9,10].

The random sampling method presents several challenges in constructing classification models, particularly in terms of efficiency, time consumption, and computing resource demands for model evaluation. These factors significantly impact its practicality in real-world applications. To address these issues, Shin and Oh [6] proposed an improved sampling method FWS based on the approach introduced by Kang and Ohs [4]. Their method involves performing multiple random samplings to generate multiple training/test datasets as candidates. For each set, feature-weighted distance between this set and original dataset is calculated to assess their suitability. A training/test set is then selected from these multiple candidates to optimize model performance. However, this method still faces difficulties in selecting the most appropriate training and test sets, requiring further refinement.

To improve the accuracy and reliability of model assessment, we employed different techniques for measuring data distribution similarity and calculating feature importance [25], which also can only handle datasets with numerical attributes. This article proposes a sampling approach which is capable of handling categorical attributes as well as mixed-type datasets. Additionally, we compare the training/test sets chosen by various sampling methods and evaluate their ability to approximate the average accuracy obtained from multiple random samplings.

2. Related Work

D. Kang and S. Oh introduced the R-value-based sampling (RBS) method [14] to enhance the evaluation of classification models. This method calculates the R-value for each data point p by identifying its k nearest neighbors and counting the number of neighbors that belong to a different class than p. As a result, the R-value of p ranges from 0 to k, indicating the degree of classification difficulty associated with that point. The RBS method groups data points based on their R-values and applies stratified sampling [11] within each group. The sampled training and test datasets from all groups are then combined to form the final training/test set. By utilizing the R-value as a quantitative indicator of classification difficulty, this method ensures that the training and test sets are constructed with comparable levels of classification complexity. Experimental results demonstrate that [14], compared to random sampling [5], D-optimal sampling [12], and MDC methods [13], the RBS method generates training/test sets that better align with the actual performance of classification models, leading to more reliable model evaluation.

Although the R-value-based sampling (RBS) method accounts for class overlap among data points, it does not consider the overall distribution distance or the importance of individual features. To address these limitations, Shin and Oh proposed the feature-weighted sampling (FWS) method [6], an improved version of RBS designed to generate multiple candidate training/test sets and select the most suitable one. One limitation of the original RBS method [14] is its reliance on stratified sampling, which often results in the repeated selection of the same training/test sets. To introduce more diversity, FWS replaces stratified sampling with random sampling, ensuring greater variability in the generated datasets. Additionally, FWS transforms the original dataset into histograms, applying the same transformation to each candidate training/test set. To evaluate which candidate set most accurately represents the overall population distribution, earth mover’s distance (EMD) [15] is used as a similarity measure. This approach allows for a more precise selection of training/test sets, addressing the shortcomings of RBS and improving the robustness of model evaluation.

Earth Mover’s Distance (EMD) [15] is a metric used to measure the similarity between distributions by calculating the minimum amount of effort required to transform one distribution into another. This transformation is conceptualized as moving units of "soil" from one distribution to match the target distribution. Given two distributions—P (representing the training/test dataset) and Q (representing the original dataset)—the EMD between P and Q is determined by the optimal transport distance and the corresponding amount of data that needs to be moved. During model training, the contribution of different features significantly impacts overall performance.

To account for this, Feature-Weighted Sampling (FWS) incorporates feature importance into the distance calculation. Specifically, it utilizes Shapley values [16] to quantify each feature’s contribution to the model. Shapley values, originating from cooperative game theory, are widely applied in machine learning to fairly assess the contribution of each feature to a model’s predictions [17]. In the context of game theory, the Shapley value provides a principled approach for distributing value among multiple players. When adapted to machine learning, it allocates the predictive influence among input features in an equitable manner. Despite its effectiveness, a notable limitation of the Shapley value is its reliance on numerical inputs, which restricts its direct applicability to categorical variables. This presents challenges when analyzing datasets that contain both numerical and categorical features.

After computing the Shapley value to determine the weight w(f_i) for each numerical feature f_i, and calculating the Earth Mover’s Distance (EMD) d(f_i) for each training/test set, the overall feature-weighted distance d is derived using Expression (1). In this expression, d_train(f_i) represents the EMD of feature f_i in the training set, while d_test(f_i) represents the EMD of feature f_i in the test set. Once the feature-weighted distances for all candidate training/test sets are computed, the FWS method selects the training/test set with the smallest feature-weighted distance, as it best preserves the overall data distribution and feature importance, ensuring optimal model evaluation.

$$d={\sum }_{i=1}^{n}w\left(f_i\right)\times \left(d_{train}\right(f_i)+d_{test}(f_i\left)\right)$$

(1)

3. Our Approach

In this section, we introduce our proposed sampling framework designed to more accurately evaluate the classification performance of algorithms on mixed-type datasets. We begin by enhancing the FWS method [6], which was limited to datasets composed solely of numerical attributes. To address this limitation, we further develop sampling techniques capable of effectively handling categorical attributes. Based on experimental findings, we ultimately propose an integrated sampling strategy that accommodates both numerical and categorical features. The subsequent subsections detail the sampling procedures for different attribute types.

3.1. The Sampling Methods on the Dataset with Numerical Attributes

In this subsection, we describe how our method conducts sampling on the dataset with only numerical attributes [25], which generates an optimal training and test set for accurately evaluating classification performance. Our approach enhances the FWS method [6] by refining data distribution distance measurements and feature importance calculations.

Our method consists of two phases. In the first phase, we apply a modified RBS [6] to generate 1,000 training/test candidate sets. In the second phase, we evaluate these candidate sets and select the one with the smallest feature-weighted distribution distance. To ensure the most suitable training and test set selection, our approach improves the FWS method by refining feature-weighted calculations and data distribution distance assessment. The overall architecture of our approach is illustrated in Figure 1 [25].

In calculating distribution similarity, FWS employs the Earth Mover’s Distance (EMD) [15] to measure the distance between the population data and the training/test datasets. However, the computation of EMD requires data discretization, which can lead to information loss and potentially impact the accuracy of the results. To address this limitation, our method utilizes three alternative approaches for measuring distributional distance: Energy Distance [18], distance correlation hypothesis testing (dcor) [19], and the Kolmogorov-Smirnov (K-S) test [20].

Energy distance, inspired by Newton's concept of gravitational energy, serves as a versatile statistical tool. It can be applied in various contexts such as testing statistical independence through distance covariance, assessing goodness-of-fit, performing non-parametric tests for distribution equality, extending analysis of variance (ANOVA), identifying change points, conducting feature selection, and more [18]. This metric is particularly useful for detecting similarities between two probability distributions. The formal definition of energy distance is provided in Expression (2), where X and Y represent two distributions, and N_X and N_Y ​denote the number of samples from each, respectively. One of the key strengths of energy distance lies in its sensitivity to both the shape and location of distributions, making it especially effective for testing distributional homogeneity. However, a notable drawback is that it requires computing pairwise distances between all sample values, which can become computationally expensive for large datasets. Additionally, the energy distance does not have a standardized scale.

$${ϵ}_{N_X,N_Y}\left(\mathrm{X},Y\right)={\displaystyle \frac{2}{N_X{N}_Y}}\sum _{i=1}^{N_X}\sum _{j=1}^{N_Y}‖x_i-y_j‖-{\displaystyle \frac{2}{{N_X}^2}}\sum _{i=1}^{N_X}\sum _{j=1}^{N_Y}‖x_i-y_j‖-{\displaystyle \frac{2}{{N_Y}^2}}\sum _{i=1}^{N_X}\sum _{j=1}^{N_Y}‖x_i-y_j‖$$

(2)

The value derived from the energy distance is not confined to a fixed range, which can result in it disproportionately influencing the overall weighted distance calculation during sample selection. To address this, Carreño and Torrecilla [19] introduced an implementation of energy-based metrics in the Python dcor package. This package supports computations such as distance covariance, distance correlation, partial distance correlation, and hypothesis testing. In our approach, we also utilize dcor to assess the similarity between different data distributions.

The dcor-Hypothesis test, as defined in Expression (3), accounts for differences in sample sizes to maintain comparability across datasets. It estimates the p-value through permutation testing: by repeatedly shuffling the samples and computing the energy statistic, it measures how often the permuted statistic exceeds the original, indicating the degree of similarity or dependence. This test is suitable for multi-dimensional data and is sensitive to variations in distribution shape, location, and scale. Unlike energy distance, the dcor-Hypothesis test outputs values within a fixed range of 0 to 1, providing a normalized measure of similarity. However, this method is computationally intensive, especially for large datasets, due to the repeated permutations involved in the testing process.

$$T={\displaystyle \frac{N_X{N}_Y}{N_X+N_Y}}{ϵ}_{N_X,N_Y}\left(X,Y\right)$$

(3)

The third distribution similarity calculation method employed to measure the distributional distance between the population dataset and the training/test sets is the Kolmogorov-Smirnov (K-S) test [20]. In multivariate statistics, the K-S test is a widely adopted method for assessing differences in data distributions [20]. It quantifies the similarity between two distributions by evaluating the maximum difference between their cumulative distribution functions, as illustrated in Expression (4). This maximum deviation is taken as the measure of distance between the distributions. Here, F(P) and F(Q) represent the cumulative distribution functions of distributions P and Q, respectively.

$$D=max\left|F\left(P\right)-F\left(Q\right)\right|$$

(4)

Compared to the Earth Mover’s Distance [15], the K-S test is more intuitive and easier to apply, especially for one-dimensional data where it is both simple and computationally efficient. Its output ranges consistently between 0 and 1. However, when applied to multi-dimensional data, it requires careful consideration and validation to ensure accuracy.

For feature importance estimation, the Shapley value used in the FWS method [6] considers all possible feature ranking combinations, leading to excessive computation time, making it impractical for real-world applications. Additionally, Shapley values can sometimes be negative, which may reduce the calculated distance and lead to inaccurate results. To address these issues, our approach adopts Analysis of Variance (ANOVA) to assess the relationship between each feature and the target variable. The resulting p-value is used as a measure of importance. In statistical analysis, the p-value indicates whether the null hypothesis can be rejected. Typically, a p-value below a predefined significance level (commonly 0.05) suggests that the feature has a statistically significant effect on the target variable. Therefore, we treat the $1-$p-value as an indicator of feature importance.

$$F={\displaystyle \frac{MSB}{MSW}}$$

(5)

$$MSB={\displaystyle \frac{{\sum }_{i=1}^k{n_i(\overline{X_i}-\overline{X})}^2}{k-1}}$$

(6)

$$MSW={\displaystyle \frac{{\sum }_{i=1}^k{\sum }_{j=1}^{n_i}{(X_{ij}-\overline{X_i})}^2}{\mathrm{N}-k}}$$

(7)

The F-statistic and p-value are commonly used as the basis for hypothesis testing in ANOVA, as shown in Expression (5). The F-statistic is calculated using the mean square between groups (MSB) which measures variation between groups (sample means), and the mean square within groups (MSW) which measures variation within each group. MSB and MSW are shown in Expression (6) and Expression (7), respectively, in which N and k denote the total number of observations and total number of groups, respectively, $n_i$ denotes the number of observations in the i-th group, $\overline{X_i}$ and $\overline{X}$ represent the mean of the i-th group (the average of all values in group i) and the average of all observations across all groups, respectively, and $X_{ij}$ is the j-th observation in the i-th group. Therefore, $(\overline{X_i}-\overline{X})$ is squared difference between group mean and grand mean, which represents how much group i deviates from the overall mean, and $(X_{ij}-\overline{X_i})$ is squared difference between an individual observation and its group mean, which presents how much individual values deviate from their own group mean.

3.2. The Sampling Methods on the Dataset With Categorical Attributes

In this subsection, we describe our proposed feature-weighted sampling methods on a categorical dataset. The framework of our sampling method for categorical dataset is illustrated in Figure 2. The process is divided into two phases: In the first phase, the modified RBS method [6] is used to generate 1000 training/test sets, which serve as candidate sets. In the second phase, each candidate set is evaluated by calculating the feature-weighted distribution distance between the population dataset and the training/test sets (candidate sets). The distribution distance between the population dataset and the training/test dataset is measured using Kullback-Leibler divergence (KLD) [21] and Jensen-Shannon divergence (JSD) [22]. Feature importance is assessed using the p-value from the Chi-square test (${\chi }^2$ Test).

We adopt Kullback-Leibler divergence (KLD) and Jensen-Shannon divergence (JSD) as methods for calculating distribution distance. The KL divergence is defined in Expression (8), where P and Q represent different distributions. Specifically, p(x) is the probability distribution of P, and q(x) is the probability distribution of Q. The term p(x) serves as a weighting factor, emphasizing events with higher occurrence probability and greater influence on the overall distribution. The Kullback-Leibler divergence (KLD) can be used to measure the degree of similarity between two probability distributions. When the two distributions are identical, the KLD is zero. It is also asymmetric, as indicated in Expression (9), and can be regarded as a form of information measure [23].

$$D_{KL}\left(P\parallel Q\right)={\sum }_{x}p\left(x\right)\mathit{ln}\left({\displaystyle \frac{p\left(x\right)}{q\left(x\right)}}\right)$$

(8)

$$D_{KL}\left(P\parallel Q\right)\ne D_{KL}\left(Q\parallel P\right)$$

(9)

Due to the asymmetry of KL divergence, it is necessary to compute the divergence in both directions: one using the population data (P) as the reference distribution, and the other using the training data (Q) as the reference distribution. If the difference between the two distributions is substantial, the KL divergence can grow without bound and approach infinity. JS divergence [22] can be used for fitness testing and to solve the asymmetry problem of KL divergence. Expression (10) is the definition of JS divergence, where P and Q represent different distributions, and their values ​​range from 0 to 1. Both JS divergence and KL divergence require the probability distribution of each data.

$$JSD\left(P\parallel Q\right)={\displaystyle \frac{1}{2}}{D}_{KL}\left(P\parallel {\displaystyle \frac{P+Q}{2}}\right)+{\displaystyle \frac{1}{2}}{D}_{KL}\left(Q\parallel {\displaystyle \frac{P+Q}{2}}\right)$$

(10)

For the feature importance, we adopt the Chi-square test [24] as the basis for calculating the feature importance of categorical dataset. The Chi-square test is a commonly used statistical method for significance testing, which assesses the relationship between feature variable and target variable.

4. Experimental Results

In this section, we first describe the datasets we used to perform the experiments and introduce the evaluation metric, and then we apply the four classification algorithms decision tree (DT), random forest (RF), K-nearest neighbor (KNN) and support vector machine (SVC) to build the classifiers. Finally, we assess the reliability of various sampling methods by analyzing the classification accuracy on test datasets, using the classifiers trained on the training sets derived from each sampling strategy.

4.1. Datasets and Classifiers

The datasets used in our experiments can be obtained from UCI repository (https://archive.ics.uci.edu/) and Kaggle (https://www.kaggle.com/), which are all used in classification tasks. We use five datasets with only numerical attributes, five datasets with only categorical attributes and five datasets with hybrid attribute types. Table 1 is the description of the datasets used in the experiments, in which N denotes the numerical attributes and C denotes the categorical attributes. Table 2 shows the parameters of the four algorithms used to build the classifiers.

4.1. Evaluation Metric MAI

To assess whether the training/test set generated by a sampling method meets the expected outcomes, a reliable evaluation metric is required. During the sampling process, it is challenging to determine in advance whether the selected training/test sets will produce the desired results. To address this, one common strategy is to use the accuracy obtained from random sampling as a baseline for comparison. However, relying on the accuracy of a single random sample can be misleading due to high variability across different samples. A more robust approach involves computing the average accuracy over multiple random samplings, providing a more stable and reliable reference metric. This average is referred to as the Absolute Evaluation Value (AEV) [4], as defined in Expression (11). In this context, test_acc_i denotes the accuracy of the i-th random sampling, m represents the total number of random samplings, and AEV is the mean accuracy computed across all m samplings.

$$AEV={\displaystyle \frac{{\sum }_{i=1}^m{test\_acc}_i}{m}}$$

(11)

Once the Absolute Evaluation Value (AEV) is determined, the Mean Accuracy Indicator (MAI) is used to evaluate whether a training/test set is appropriate [4]. As shown in Expression (12), MAI is calculated by subtracting the average accuracy of multiple random samplings (AEV) from the classification accuracy (ACC) of a given sampling method and then dividing the result by the standard deviation (SD) of the AEV. Intuitively, a lower MAI value indicates a smaller discrepancy between ACC and AEV, suggesting that the sampling method produces results closely aligned with the average performance of multiple random samplings, thereby demonstrating its effectiveness.

$$MAI={\displaystyle \frac{⌈ACC-AEV⌉}{\mathrm{S}D}}$$

(12)

4.3. Experimental Results On Numerical Datasets

We first conducted 1,000 random samples on five datasets with only numerical attributes in Table 1 to generate 1,000 training/test sets. Using four different classification algorithms, we built classification models and obtained accuracy on the test set for each sampling. We then calculated the average accuracy and standard deviation of the test dataset across the 1,000 classifiers built from the random samplings, as shown in Table 3.

We employ the Modified RBS sampling method [6] to generate 1,000 candidate training/test sets for each of the five numerical datasets. For each candidate set, we use different distribution distance calculation methods EMD, Energy Distance, dcor-Hypothesis testing and K-S test, and apply Shapely value and ANOVA ​​to calculate feature weights, respectively, in which the combination of EMD and Shapely value is the method FWS [6]. Subsequently, we calculate the feature-weighted distance between each candidate training/test set and the original dataset using Expression (1). The training/test set exhibiting the smallest feature-weighted distance is selected. The classification accuracy (ACC) of the test set for the four classifiers built by using the training set on the selected sets are presented in Table 4 and Table 5, in which Shapely value and ANOVA are used to calculate feature weight, respectively. According to Table 3 and from Table 4 and Table 5, the evaluation metric MAIs for the sampling methods based on different distribution distance calculation methods with Shapely value and ANOVA as feature-weighted computation are shown in Table 6 and Table 7, respectively.

Based on the results presented in Table 6 and Table 7, the average MAI of dcor-Hypothesis testing method is the smallest among all the methods, which demonstrates superior performance compared to other distribution distance calculation methods in most scenarios. By employing dcor-Hypothesis testing alongside ANOVA for feature-weighted calculation, the resulting training/test dataset selected based on the smallest feature-weighted distance yields the smallest average MAI values.

4.3. Experimental Results On Categorical Datasets

We also performed 1,000 random samplings on five datasets with only categorical attributes in Table 1, resulting in 1,000 distinct training/test set pairs. For each of these pairs, we applied four different classification algorithms to construct corresponding classification models and recorded the accuracy on the test dataset. Subsequently, we calculated the average test accuracy and its standard deviation across the 1,000 classification models generated from the random samplings. The results are presented in Table 8.

Table 9 shows the classification accuracy of test dataset obtained from four classifiers using five categorical datasets. For each dataset, candidate training/test sets were evaluated by computing their distribution distances from the original dataset using Kullback-Leibler Divergence (KLD), Jensen-Shannon Divergence (JSD), and Earth Mover’s Distance, incorporating feature importance calculated by the Chi-square test [24], computing feature-weighted distances. The training/test set with the smallest distribution distance to the original set was selected. For KLD, two variants were considered: one treating the original dataset as the reference distribution (P∥Q) and the other treating the candidate training/test set as the reference distribution (Q∥P). In addition, in order to enable the FWS method to be executed on these datasets, we flattened the categorical attribute values ​​and converted them into numerical attributes by one-hot encoding, and use Earth Mover’s Distance (EMD) and Shapley values to calculate feature-weighted distance.

According to Table 8 shows the evaluation metric MAIs for the sampling methods based on different feature-weighted distribution distances with feature importance calculated by the Chi-square test [24], and FWS method. From Table 10, we can see that both methods KL divergence and JS divergence outperform the FWS method [6], which transforms categorical attributes into numerical features through one-hot encoding. Among them, the average MAI of JS divergence with feature importance is the smallest among all the methods, which achieves the best overall performance.

4.3. Experimental Results On Mix-Type Datasets

We also performed 1,000 random samplings on five mix-type datasets containing numerical attributes and categorical attributes in Table 1, resulting in 1,000 distinct training/test set pairs. For each sampling, classification models were constructed using four different algorithms, and the corresponding test accuracies were recorded. Subsequently, we computed the mean accuracy and standard deviation of the test results across the 1,000 classifiers derived from these random samples, which is shown in Table 11.

Since the combination of the distribution distance method dcor-Hypothesis testing with ANOVA for calculating feature-weighted distribution distances yielded the best performance on numerical datasets, and the combination of JS divergence with the Chi-square test performed best on categorical datasets, we adopted a hybrid approach for mixed-type datasets. Specifically, for numerical attributes, we applied dcor-Hypothesis testing combined with ANOVA to calculate the feature-weighted distance between the original dataset and candidate sets. For categorical attributes, we used JS divergence combined with the Chi-square test for the same purpose.

In addition, we conducted another experiment in which all categorical attribute values were flattened and converted into numerical attributes using one-hot encoding. After transforming all categorical features in the mixed-type dataset into numerical form, we applied dcor-Hypothesis testing combined with ANOVA to calculate the feature-weighted distance between the original dataset and candidate sets, and the training/test set with the smallest feature-weighted distance was selected. The corresponding test accuracy and MAI values are presented in Table 12.

As shown in Table 12, our hybrid approach outperforms the best method (dcor-Hypothesis testing with ANOVA) for processing numerical attributes after converting categorical attributes into numerical attributes in the above experiments. This result demonstrates that our hybrid sampling method achieves strong performance on mixed-type datasets.

5. Conclusions

In the past, it was common to repeatedly split the dataset into training and test sets and build multiple classification models to evaluate classification accuracy, which requires substantial computational time for repeated modeling and evaluation, and becomes impractical when dealing with large-scale datasets. To address this, some researchers have proposed sampling methods to obtain a single training/test dataset that approximates the results of repeated modeling. Among the existing methods, the Feature-Weighted Sampling (FWS) method is currently regarded as the most effective. It calculates the distribution distance between the original dataset and the training/test dataset using Earth Mover's Distance (EMD) and employs Shapley values to compute feature importance. However, EMD requires data discretization prior to distribution similarity calculation, which may compromise data fidelity. Moreover, Shapley values are applicable only to numerical attributes.

To overcome these limitations, this study proposes improvements to the FWS method. Specifically, we introduce a sampling approach that does not require discretization when computing distribution similarity and also propose a sampling strategy capable of handling categorical datasets. Experimental results show that our proposed method achieves lower MAI (Mean Accuracy Indicator) and performs better than the original FWS method. Furthermore, in our sampling framework, the best-performing sampling strategies for numerical and categorical datasets are respectively applied to the corresponding attributes in mixed-type datasets. The results also demonstrate that our hybrid method