Investigating the Performance of a Novel Modified Binary Black Hole Optimization Algorithm for Enhancing Feature Selection

1. Introduction

The continuous development of online technology has created large volumes of high-dimensional data. These extreme volumes are generated from various sources such as sensor networks, data communications, web applications, manufacturing, network monitoring, and financial applications, as well as medical diagnosis reports. The current era can be called the era of big data, where massive amounts of high-dimensional datasets belonging to diverse domains are collected from social media, healthcare, and bioinformatics [1]. These datasets contain many features that are used for the aforementioned practical purposes. However, many features could be irrelevant or redundant, causing drawbacks such as slowing down the learning algorithm, consuming resources, and decreasing the performance of Machine Learning (ML) algorithms [2]. Therefore, high-dimensional datasets are considered time-consuming for model construction, making the data analysis process very complex and difficult to interpret. Therefore, the process of finding the optimal subset of features, which is called “minimal optimal”, via selecting the smallest possible subset of features that gives the best classification result is essential [3,4,5].

This process of feature selection (a.k.a dimensionality reduction) is used in preprocessing steps to enhance the performance of ML models. Dimensionality reduction can be divided into three methods: Feature Construction (FC), Feature Extraction (FE), also called feature projection, and Feature Selection (FS). FC is a process of increasing the expressive power of the original features by generating additional features, thereby revealing the relationships between the features and augmenting the space of features. In other words, FC aims to transform the original representation space of features to new ones that improve data mining objectives. It is important to note that in FC it is not necessary that all constructed features are useful [6,7]. On the other hand, FE is a process of generating new significant features from the original ones by performing transformation functions, while ensuring that the number of produced features is lower than the original ones. Meanwhile, FS is a process of selecting the best possible subset of features from the original space of features in order to improve the performance of the classifier [8]. In other words, FS is a technique used to select the best subset of features that aims to reduce the size of the dataset structure without significantly decreasing the performance of data mining or the classification algorithms [3].

Figure 1 shows the difference between feature selection and feature extraction. The optimal subset of features provides faster and more cost-effective predictors with better analysis of the selected features instead of high-dimensional complex datasets which contain a lot of variables [10].

FS is a very useful technique for various studies that tend to work with large data and complex problems. FS can be deployed in many fields such as image processing/computer vision [11] (e.g., image classification), text mining [12] (e.g., text clustering and classification), industrial applications [13,14,15] (e.g., fault diagnosis and life expectancy), bioinformatics [16] (e.g., biomarker discovery), biomedical engineering, electrical and electronic engineering, computer engineering [17], and detection systems like intrusion detection systems [18,19] and fraud detection systems [20].

Most FS approaches fall into three main categories: filter, wrapper, and embedded approaches [21]. Filter approaches assess features from the dataset without intervention from classification algorithms, using statistical functions to rank each feature and indicate its importance [22]. Wrapper approaches select the best feature subset from the dataset with learner intervention, generating subsets from the original one and using the learner to evaluate all of them and mark the best ones [23]. Embedded approaches are similar to wrapper approaches since both attempt to optimize the objective function of the learner based on an intrinsic model-building metric used during the learning process [16,24]. For example and by the way of demonstration, embedded methods integrate feature selection and learner into a single process, such as using a genetic method to search in feature space and using it as a classification algorithm [25]. Furthermore, embedded approaches calculate the importance of each feature, removing low-scoring ones and retaining high-scoring ones based on a calculation function [26]. For all of these three FS approaches, the main goal is selecting which inputs are highly affecting the classification algorithm.

FS algorithms are usually performed by domain experts to choose which variables of an instance (object) should be selected to describe and determine the output class of a given problem [27]. Due to the time and resource-intensive nature of exhaustive or precise search strategies aimed at assessing all potential feature subsets to identify the optimal one, employing such expansive search algorithms for FS is impractical [28]. For instance, when dealing with a sizable dataset comprising d-features, a thorough approach would generate $2^d$ potential subsets for evaluation, leading to performance issues. Given that FS is recognized as an NP-hard problem, researchers have consistently explored stochastic methods leveraging randomness for its resolution [3,28]. Randomness-based algorithms offer a practical and efficient alternative, as they can effectively conserve time and resources. Particularly, population-based algorithms have emerged as viable solutions to mitigate the resource drain associated with FS [3].

Although there are several prior works in feature selection (FS), they have several limitations that affect their usage in practical applications. First, traditional FS approaches often suffer from overfitting, especially in high-dimensional feature spaces [29]. While these methods may select variables that improve training performance, their generalization to unseen data can be poor. Second, embedded FS methods that rely on biased feature scoring techniques during training, such as binary decision trees, may not perform well during the testing phase [30]. Third, some FS approaches involve computationally expensive procedures, making them impractical for large-volume datasets. This is the case for forward [31] and backward [32] FS methods. Some FS methods assume independence among variables, which might not hold true in complex real-world problems. Furthermore, some FS methods are sensitive to noise and outlier features, leading to the selection of sub-optimal feature subsets [33].

In this paper, we propose a novel wrapper feature selection (FS) approach called MBHO (Modified Black Hole Optimization Algorithm). MBHO combines the black hole algorithm with inversion mutation. Empirically, numerous research studies based on the Black Hole Algorithm (BHO) have been proposed to solve diverse optimization problems across various domains [34,35,36]. These studies consistently demonstrate high-quality solutions and favorable outcomes. We will delve into specific works related to this algorithm in the upcoming related work section. The motivation behind adopting the BHO for FS can be illustrated through several key points:

• Astrophysics Inspiration: BHO draws inspiration from astrophysics, specifically the fascinating concept of black holes. This unique property sets it apart from traditional statistical methods. By mimicking the gravitational interactions of black holes, BHO explores feature spaces in a novel and non-deterministic manner.
• Randomness-Based Exploration: BHO leverages randomness and domesticity techniques to explore wider areas within the search space. Unlike deterministic approaches, it embraces stochasticity, allowing it to escape the local optima problem, found in many FS methods, and avoid premature convergence. This adaptability is particularly valuable in complex optimization landscapes.
• Simplicity and Computational Efficiency: The mathematical model underlying BHO is not overly complex. Consequently, it requires less computational expense compared to more intricate approaches. This efficiency makes it practical for real-world applications.

In summary, MBHO combines the power of black hole-inspired exploration with the practicality of simplicity, making it a promising candidate for feature selection tasks.

In our proposed FS approach, we develop a revised fitness function to assess the best feature subset and improve the exploration process in the problem’s search space. In particular, we consider the correlation among the features in the feature subset and the dependencies between each feature and the assigned label. We leverage the Spearman correlation function [37] to compute the dependencies we added to our proposed fitness function. We also utilize a transfer function, known as the V2 transfer function, to convert continuous values into discrete ones. This is because most evolutionary algorithms update their candidate solutions by artificially producing solutions with continuous values. The enhanced binary version of BHO, proposed in this work, adopts inversion mutation to enhance the population diversity to prevent premature convergence and to avoid the trap of local optima. Taken together, these additions allow our FS (MBHO) approach to explore broader areas in the search space and extract a better subset of features.

We rigorously evaluate MBHO using a diverse set of fourteen benchmark datasets spanning various domains, including healthcare, social life, and others. To assess MBHO’s performance, we adopt two key measurements: accuracy and the F1-score. Our evaluation encompassed datasets of varying sizes and dimensionality, ensuring a comprehensive assessment across different domain-specific problems. We also benchmark the performance of several popular FS methods, including a wrapper FS approach called Binary Cuckoo Search (BCS) [38], and three filter-based FS methods, namely Mutual Information Maximisation (MIM) [39], Joint Mutual Information (JMI) [40], and minimum Redundancy Maximum Relevance (mRMR) [41]. Notably, MBHO consistently achieved competitive results and demonstrated robustness over diverse datasets. This robustness is crucial for real-world applications, where data properties can significantly vary. An insightful observation from MBHO’s performance is the importance of considering feature dependence, which aligns with real-life scenarios. Although MBHO exhibits reasonable results across all validation datasets, it also showcases strong generalization capabilities, performing well on unseen data. These findings highlight its potential for practical deployment in handling real-life feature selection tasks. Moreover, MBHO achieves these outcomes efficiently, utilizing reasonable computational resources (as shown in our evaluation).

In summary, the main contributions of this work can be summarized as follows:

• We propose a novel wrapper feature selection (FS) approach, that combines the black hole algorithm with inversion mutation, to select the most descriptive subset of features from datasets that cover different application domains.
• We modify a well-established multi-objective function that focuses on the interleaved classifier and the number of selected features in the decision-making process. Additionally, we enhance the decision-making process by considering the correlation among the selected subset of features and the correlation of each feature within that subset with the corresponding label.
• We assess our approach using fourteen benchmark datasets. We benchmark the performance of a wrapper FS approach called Binary Cuckoo Search (BCS). We also benchmark the performance of three filter-based FS, namely Mutual Information Maximisation (MIM), Joint Mutual Information (JMI) and minimum Redundancy Maximum Relevance (mRMR).
• We release the source codes of our framework for the research community. The implementation link is: https://github.com/Mohammed-Ryiad-Eiadeh/A-Modified-BHO-and-BCS-With-Mutation-for-FS-based-on-Modified-Objective-Function.

2. Related Work

This section provides an overview of several proposed studies that have been conducted to develop FS systems to handle different application domains, detailed below.

2.1. Meta-Heuristic Algorithms for Feature Selection

Meta-heuristic algorithms are utilized to solve complex optimization problems when traditional solutions are not efficient to use. These algorithms are attractively search for better solutions by evolving the existed ones to explore better areas in the given problem search space [42]. Genetic Algorithm (GA) and immune clonal algorithm, have revolutionized FS in various domains. The work [43] proposed a hybrid approach combining GA and the immune clonal algorithm to overcome GA’s limitations, inspired by the immune system’s behavior. Similarly, [44] utilized an artificial immune system for FS, focusing on optimizing inter-class and intra-class distances. In biomedical classification, [45] introduced a hybrid FS technique merging the Binary State Transition Algorithm (BSTA) with the Relief filter approach, demonstrating its effectiveness in detecting bearing faults in brushless DC motors. Additionally, [46] combined envelope analysis and Hilbert-Huang transformation for FE from Hall sensor signals, highlighting the versatility of meta-heuristic approaches across a variety set of applications in different domains.

Particle Swarm Optimization (PSO) is inspired by the behavior of birds and has diverse applications. Initially proposed by [47], PSO was employed for classifying conducting particles in transformer oil [48], utilizing FE from denoised electrical signals and a Support Vector Machine (SVM) classifier. the work [49] proposed an improved multi-objective PSO by generating a Pareto front of non-dominated solutions, catering to various real-world task requirements. The article [50] utilized PSO for breast cancer detection, involving stages such as data acquisition and preprocessing. Additionally, the Black Hole Optimization (BHO), inspired by star behavior near black holes [34], was adapted for biological data classification [35], with further enhancements using chaotic map functions [36].

The Humpback Whale Optimization (HWO) algorithm, inspired by humpback whales’ hunting behavior, has been applied in various domains. The work [51] described the three steps of humpback whale hunting: searching for prey, encircling prey, and spiral bubble net attacking. The work [52] proposed HWO for disease classification, while the work [53] improved it for FS in text analysis. The article [54] introduced an Embedded Chaotic Whale Survival Algorithm for FS, incorporating wrapper, filter, and death mechanism components.

Additionally, GA, inspired by genetic variation and natural selection [55], remains one of the most popular optimization algorithms.

The work [56] utilized a binary GA for classifying features extracted from images in the Flavia dataset. Various GA approaches have been proposed for FS across different problem sizes [57,58], including hybrid GA methods integrating Wrapper and Embedded FS techniques for classification tasks. The work [59] explored the use of different L-regularization functions for FS.

The work [60] employed a binary Gravitational Search Algorithm (GSA) with a fuzzy-rule-based classifier for FS, while the work [61] proposed GSA with mutation and crossover mechanisms. The paper [62] introduced the Equilibrium Optimizer (EO) algorithm for FS, including a hybrid binary EO with SA [63,64]. Additionally, the work [65] addressed biological data classification using a General Learning EO, while the article [66] proposed an Improved EO Algorithm for FS in biomedical data, enhancing diversity in solution sets.

2.2. Approximate Algorithms for Search

There have been several works for using approximate algorithms in finding optimal (or near-optimal) solutions in search space. These algorithms are used to estimate the solutions for optimization problems (in particular NP-hard problems), where they aim to provide reasonable solutions with no high discrepancy from the optimal ones of the given problems [67]. The work [38] proposed Binary Cuckoo Search (BCS) for detecting thieves in power distribution systems. The work [68] modified BCS for FS, while the work [69] improved Chaotic BCS to overcome local optima. The work [70] used weighted BCS for speech emotion recognition and ensemble RF with BCS for feature importance analysis. The article [71] introduced Crow Optimizer to avoid local optima, and this work [72] applied it for optimization. The work [73] enhanced FS with V-shaped transfer functions and CSA, while the work [74] improved CSA with ten chaotic maps.

Bat Algorithm (BA) simulates microbats’ echolocation for prey searching [75], applied FS [76] and blind steganalysis [77]. Cat Swarm Optimization (CSO) mimics cat behavior [78], adapted for FS [79]. Grey Wolf Optimization (GWO) inspired by grey wolves’ behavior [80] is utilized for FS in various domains [81,82,83]. Jaya algorithm, based on solution improvement principles [84], outperforms application in FS [85,86]. Intelligent Water Drop (IWD) algorithm, inspired by natural water flow, is adapted for FS [87], including Neural-IWD models [85].

Pigeon Inspired Optimization (PIO) emulates pigeons’ homing behavior [88], applied in diverse areas including intrusion detection [89], image organization based on visual features [90], and others [91]. Recent works in FS include a PSO-based multi-objective memetic algorithm for massive datasets [92], and a micro-expression recognition system with evolutionary operators [93].

2.3. Filter-Based FS Approaches

Filter-based FS approaches have been used for several applications in the literature. Filter-based approaches enhance FS for heart disease classification [94]. Information Gain (IG) is commonly employed to measure feature importance in high-dimensional datasets [95]. Sympatric Uncertainty (SU) assesses the relationship between a feature and the target class by multiplying the Information Gain (IG) of a specific attribute class by two and dividing by the sum of class entropy and attribute entropy, mitigating IG’s bias issues [96,97]. Chi-square measures the dependency between an attribute and target label individually [98]. It assesses deviation from expected class value distribution while considering the independence of feature and class values. Chi-square is regarded as a robust FS method, particularly effective with multi-label datasets [99]. Fast Correlation Based Filter (FCBF) employs Sympatric Uncertainty (SU) to compute feature-class correlations, eliminating uncorrelated features through backward sequential search [100]. Inconsistency Rate (IR) measures the inconsistency rate across a dataset for a subset of features, providing insight into feature sets rather than individual features [101]. IR computation can achieve an approximate time complexity of $O\left(patterns\right)$ through hashing mechanisms [102]. Correlation-based Feature Selection (CFS) evaluates feature subsets using heuristic search functions instead of exhaustive search strategies [103]. Additionally, minimum Redundancy Maximum Relevance (mRMR) ranks the importance of feature subsets for classification problems [104,105].

For continuous data, F-statistic measurement computes the relevance between a given subset of features and a specified class [106]. The Markov Blanket (MB) of a target class represents the optimal subset of variables predicting that class, comprising features statistically dependent on the class while ignoring those independent ones [107,108]. Relief, a common weighting method for FS, estimates an accurate subset of features for classification tasks [109]. Relief’s stopping criteria are user-defined, indicating the total number of variables to be selected in the final subset, prioritizing lower-weight variables first. However, Relief is limited in handling large instances, noisy or incomplete data, and is restricted to two-class problems [110,111]. ReliefF was introduced to address these limitations, employing the K-Nearest Neighbor (KNN) classifier to update feature weights and selecting higher-weight variables initially until meeting the stopping criteria [112,113]. Fisher score identifies an optimal feature subset for class description by maximizing intra-class distances and minimizing inter-class distances. It computes scores for individual features independently and selects features with low individual scores but high relevance when grouped [114,115]. Fisher score is widely used and efficient for FS [116].

2.4. Contributions of Our Work

This current study pioneers the utilization of the BHO with an inversion mutation mechanism to select the most descriptive subset of features from datasets that cover different application domains, which helps in solving the diverse FS problems, as indicated by various research papers referencing FS methodologies. Additionally, we modify a well-established multi-objective function that focuses on the interleaved classifier and the number of selected features in the decision-making process. Furthermore, we enhance the decision-making process by considering the correlation among the selected subset of features and the correlation of each feature within that subset with the corresponding label. We elucidate our rationale for incorporating correlation functions into our approach, drawing from information theory principles. Through our evaluation (in which we compare our work with both filter-based and wrapper-based FS approaches), our work demonstrates its capability to be regarded as a prospective state-of-the-art in FS.

3. Background on Mutual Information

When discussing FS from the perspective of information theory, it is worthy to mention a very important terminology, Mutual Information (MI). MI is a metric that measures the amount of associated shared information between two variables (two features in our context) [117,118]. In another context, it quantifies the amount of information shared between two random variables. This quantity is always a non-negative value, and if it is zero, then X and Y are independent random variables. By way of illustration, MI measures the associated relationships among two variables. For example, for two given discrete variables X and Y, MI of X and Y is given by:

$$MI(X,Y)=\sum _x\sum _y{P}_{(X,Y)}(x,y)log\left(\frac{P_{(X,Y)}(x,y)}{P_X\left(x\right)P_Y\left(y\right)}\right),$$

(1)

where, the joint probability distribution of X and Y is given by:

$$P_{(X,Y)}(x,y)=P(X=x,Y=y)$$

(2)

This demonstrates the probability that X obtains x and Y obtains y respectively. This expression satisfies the following two conditions:

$$p(x,y)\ge 0,\forall x\in X,\forall y\in Y$$

(3)

$$\sum _{x\in X,y\in Y}p(X=x,Y=y)=1$$

(4)

This indicates that the probability for any combination of $x\in X,y\in Y$ occurring at same time is never less than zero, and this depicts the likelihood of an event occurring which cannot be negative. In the case where X and Y are variables of continuous values, MI would be given by:

$$MI(X,Y)={\int }_x{\int }_y{P}_{(X,Y)}(x,y)log\left(\frac{P_{(X,Y)}(x,y)}{P_X\left(x\right)P_Y\left(y\right)}\right)dxdy$$

(2)

In FS for classification problem, we care about the association between the features in a subset and between each feature and the class label. For a given feature subset $S=[v_1,v_2,...,v_d]$ and label l, $MI(v_1,v_2)$ measures the dependency between feature $v_1$ and feature $v_2$ while $MI(v_1,l)$ measures the dependency between feature $v_1$ and label l. For more details about how we take advantage of measuring the dependencies among variables in the subset and among each variable and the label, we refer to our detailed explanations in Section 4.5.

4. Methodology

In this section, we explain every step of our proposed FS approach, including the optimization problem, the population representation of features, the evaluation function, the proposed modification on the evaluation function, the correlation between two feature vectors, the binary improved black hole optimizer (BHO), and the mitigation of main issues of BHO optimizer. We also provide a time complexity analysis of the proposed approach.

4.1. Optimization Problem

The maximization problem is a type of optimization problem, which can be mathematically defined as follows:

$$Maximize\left(f\right(X\left)\right)$$

(3)

where $f\left(X\right)$ is the fitness function of X which is used to evaluate the candidate solution X. f is designed based on the given optimization problem. The candidate solution X for any optimization problem consists of $d-decision$ variables $(x_1,x_2,\dots ,x_d)$ in the search space, where $x_i$ is the $i_{th}$ decision variable that can be initialized as follows:

$$x_i=L_i+(U_i-L_i)\times r,$$

(4)

where $L_i,U_i$ are the lower and the upper bounds of $x_i$ respectively, and $r\in [0,1]$ is a uniform distribution variable. The optimization algorithms have demonstrated their ability to generate high-quality solutions efficiently. For instance, the work [119] developed a K-means transition algorithm to enhance CS and BHO algorithms, exhibiting competitive performance in solving the set covering problem. Additionally, the work [120] proposed a multi-stage GWO approach to enhance feature selection for fund performance evaluation, overcoming local optima issues and reducing feature dimensions and classification error rates. Furthermore, the work [121] introduced a PSO-based feature selection method with explicit representation and feature grouping, addressing memory and computational challenges in PSO.

4.2. Population Representation

Most population-based algorithms start by generating the population randomly, and then applying a sequence of rules to it during the optimization process, aiming to produce better generations. In our research, every solution is depicted as a binary vector consisting of either 0s or 1s, with a size of d, which corresponds to the quantity of variables (features) within the initial dataset. Each binary digit symbolizes an individual attribute. Note that the last feature ($F_d$) is the class label, which is always selected. Figure 2 shows an illustration of such representation of different solutions in a population for FS.

Precisely, the value of $i_{th}$ attribute (or feature) is $x_i$ which denotes whether this feature is selected in X or not. If it is equal to 1, this means that it will be selected, otherwise it will not. In the majority of optimization algorithms, the process begins by iteratively refining solutions towards the current best one. The initial population is not formed through prior knowledge but rather through randomness, essentially a random guess. Such a guess could potentially be distant from the optimal solution, because of the absence of prior information at the start by generating high-quality solutions at the beginning. To solve such a problem, Opposition-Based Learning (OBL) is used. To enhance the diversity and quality of the initial population, OBL technique generates solutions in a direction opposite to the initially generated ones. The main idea of OBL is introduced in [122]. In particular, the OBL of the value x is the opposite direction denoted by $\tilde{x}$. OBL mechanism can be defined as follows:

$${\tilde{x}}_i=U_i+L_i-x_i.$$

(5)

For binary problems like FS, 0/1 knapsack [123]. $U_i$ is set to 0 and $L_i$ is set to 1. The previous equation Eq. (5) can be reformulated as follows:

$${\tilde{x}}_i=1-x_i.$$

(6)

After calculating OBL, each solution will be compared with its opposite based on the given fitness function f, and the better solutions will be kept. The previous procedure can be formulated as follows:

$$x_i=\left\{\begin{array}{cc}{\tilde{x}}_i,\hfill & f\left({\tilde{x}}_i\right)>f\left(x_i\right)\hfill \\ x_i,\hfill & otherwise\hfill \end{array}\right.$$

(7)

For example, assume that we have a maximization problem, and two solutions $X_1,X_2$ where their fitness values are given by 1.49, 1.56 respectively, their opposites ${\tilde{X}}_1,{\tilde{X}}_2$ with the fitness values 1.39, 1.62, then based on Eq. (7), $X_1$ will remain, $X_2$ will be replaced by ${\tilde{X}}_2$ because $f\left(X_1\right)$ is better than $f\left({\tilde{X}}_1\right)$, and $f\left({\tilde{X}}_2\right)$ is better than $f\left(X_2\right)$. Note that here “better” means “higher” since the fitness function is used for the aforementioned maximization problem in Section 4.1.

4.3. Evaluation Function

In FS, the best subset of features is the one that maximizes the classification accuracy with a lower number of features. The fitness function $f(·)$ can be defined as follows:

$$f\left(X\right)=max(Acc\left(X\right)+w_f*(1-\frac{L_f}{L_t})),$$

(8)

where $Acc$ is the classification accuracy under the candidate solution X, $w_f\in (0,1]$ is a factor used to scale and adjust the contribution of a mathematical term of the given equation. In other words, $w_f$ is a factor used in math to reduce the weight of some mathematical term in the given equation. When $w_f$ is closer to 0, it reduces the credits of the corresponding term, while a $w_f$ closer to 1 maintains its full influence. $L_f$ is the number of features in the current solution, and $L_t$ is the total number of features. Note that in our work $L_t=d$ since we have d features in total. For example, assume a dataset with 40 attributes, $w_f=1$, two solutions achieved $97\%$ and $97\%$ accuracies respectively, where the first one has 20 features, and the second one has 30 features, then f of the first one is $97.5\%$ and f of the second one is $97.25\%$, which means that first solution is better than the second one. The first one is better since it yields the same accuracy as the second one but with 10 fewer features.

In this work, we used the K-Nearest Neighbor (KNN) classifier with 10-fold cross-validation. KNN is one of the easiest classifiers with effective performance (accuracy) [124]. KNN is applicable to handle different tasks like classification, regression, and searching problems. It is known as a lazy algorithm [125,126] which means that no calculations are conducted in the training phase till the point of prediction, which means that no training phase is associated with KNN. It starts by computing the distances when it reaches the first point in the testing phase to determine which class label the current data point refers to by calculating the similarity (distance) between each data point and all other data points that are booked for the training portion, then upon a predefined K value, KNN decides which class that the data point (tuple) refers to.

KNN uses a set of distance functions such that Euclidean distance (ED), Mahala Nobis distance, Manhattan Distance (MD) [127], Earth Movers distance, Chebyshev distance, and Canberra distance. We set K to 3 and we used MD measurement in this work. Recall that MD is defined as follows:

$$dis(X,Y)=\sum _{i=1}^d|x_i-y_i|,$$

(9)

where $dis(X,Y)$ is the distance between $X,Y$ data vectors (datapoints). Note that $|x_i-y_i|$ is the absolute subtraction between the two vectors of features. Here, two important factors that have to be taken in consideration when utilizing KNN classification algorithm are bias and variance. This is because the learning error rate increases (bias increased) by increasing K, but the testing error may be decreased (variance decreased) and vice versa.

4.4. Modified Evaluation Function

The aforementioned fitness function in the previous section is a multi-objective function that depends on the accuracy and the number of selected features. $f\left(X\right)$ tends to maximize the $Acc\left(X\right)$ and maximize the minimization process of the selected number of features $(1-\frac{L_f}{L_t})$. Note that the lower the number of selected features $L_f$, the higher the term $(1-\frac{L_f}{L_t})$. In other words, $f\left(X\right)$ is strictly increased by increasing $Acc\left(X\right)$, and strictly increased by decreasing $L_f$. One issue we should demonstrate in the current fitness function is that it does not care about an important measurement called the correlation [128]. According to $f(·)$, there is no such term(s) in $f\left(X\right)$ that considers the correlation among the features in the candidate solution or between each feature and the corresponding class label too. Therefore, this function has this limitation with respect to our earlier claim in Section 1. This is because it does not concern the relationship among the variables in the selected subset [129]. Furthermore, random-based search has no prior information about the correlation and it can be improved by improving the decision-making in random-based algorithms.

Motivational Example: Let us assume that we have a dataset with 100 features, and two subsets of features $S_1$ and $S_2$ which contain 40 and 36 features, respectively. Let the accuracy of $S_1$ be $0.94$ and the accuracy of $S_2$ be 0.92. According to Eq. (9), with $wf=1$, the fitness score of $S_1$ is 1.54 and the fitness score of $S_2$ is 1.56 which means that $S_2$ is better than $S_1$, however, the accuracy of $S_1$ is better than the accuracy of $S_2$. Thus, not capturing the correlation between features in the fitness function may lead to a higher number of features or decreased accuracy.

We tackled this complication by introducing new two terms to the fitness function in Eq. (9). These two terms represent the different correlation aspects, as explained below. Thus, $S_1$ will be selected instead of $S_2$ if and only if the correlation among its features is lower than the correlation among the features in $S_2$ and the correlation among each feature and the corresponding label in $S_1$ is higher than the correlation among each feature and the corresponding label in $S_2$. In this work, note that the label with different categories is tackled by hot encoding [130]. According to our claim, the fitness function $f(·)$ is modified and reformulated as follows:

$$f\left(X\right)=max(Acc\left(X\right)+w_f*\left((1-\frac{L_f}{L_t})-Correlation\left(X\right)+Correlation(X,l)\right),$$

(10)

where $Correlation\left(X\right)$ is the correlation among the features in X and $Correlation(X,l)$ is the correlation between each feature and the label l. According to the same example above with two candidate solutions $X_1$ and $X_2$, and under the same assumptions, consider that based on some correlation function, the term $correlation\left(X_1\right)$ is 0.01 and $Correlation(X_1,l)$ is 0.2, and the $correlation\left(X_2\right)$ is 0.04 and $Correlation(X_2,l)$ is 0.1, then the fitness score of $X_1$ is 1.55, and the fitness score of $X_2$ is 1.53 which means that $X_1$ is better than $X_2$.

4.4.1. The Dilemma Of the Weight Factor ($w_f$) in Wrapper Feature Selection (FS)

In the fitness function, we introduced in eq. (13), the hyper-parameter $w_f$ plays a crucial role since it controls the second part of the function which indicates the subset of features importance and its features correlations and further their correlations with the corresponding label. For more explanation, consider a scenario where our focus is minimizing the loss function $L(.)$ over a subset of features X. Here we tend to show that when $w_f$ approaches zero, the impact of the features on $L(·)$ diminishes. This can be mathematically captured as follows:

$$\underset{w_f\to 0}{lim}L(X,w_f)=L(X,0)$$

(11)

Hence, as $w_f$ approaches 0, $L(X,w_f)$ converges to the loss function without any weighting, therefore, the influence of $w_f$ diminishes. In other words, when $w_f$ approaches 0, it has no impact on the loss function, and then it is perfectly downplaying the importance of the features during the selection process.

On the other hand, when $w_f$ approaches one. It indicates that the importance of the features is perfectly influencing the process of decision-making. Hence, as $w_f$ approaches 1, means that the weight which is played by the features’ importance has reached the highest impact on the loss function or the decision-making process. In other words, this assumption considers that the features are highly matter. Yet this might lead to overfitting or a lack of generalization especially if some features are highly correlated or noisy or irrelevant. In conclusion, in the context of wrapper FS, the features should play a small but enough role in the process of decision-making. Yet, this is done in a controlled manner since the process is mainly guided by the classifier performance.

4.5. Correlation between Two Candidate Feature Vectors

4.5.1. Background about Correlation

In ML, the correlation depicts a statistical measurement that represents the strength or the degree of relationship between two or more variables [118]. In another context, correlation is used to measure the degree to which two or more variables depend on each other. Assume W and Z are two vectors of double values; they are said to be dependent if and only if the values associated with one of them affect the distribution of the other one. In contrast, W and Z are said to be independent if and only if the changes in one vector’s values do not affect the distribution of the other one. Positive correlation defines that both vectors move in the same direction while negative correlation indicates that each vector moves in the opposite direction from the other such that when W decreases, Z increases and vice versa. Furthermore, correlation values lay in [-1,+1], where -1 means that both vectors have the highest degree of negative relationship in contrast to +1. Also, zero correlation informs that W and Z are unrelated to each other (or independent) [131].

Z is said to be redundant if at least one other vector as W is highly correlated to it. So, a subset of correlated features that contain redundant and irrelevant features to the respect of the target class, should be dropped from the perspective of FS. Furthermore, the optimal subset of features should have the highest correlation values between each feature and the corresponding label.

4.5.2. Intuition of Adding Correlation Terms in Our Objective

Due to the importance of considering the correlation between features when doing feature selection, we included the terms $Correlation\left(X\right)$ and $Correlation(X,l)$ in our objective function in Eq. (10). Therefore, the optimal subset of features is the one that has features that are lightly correlated to each other and highly correlated to the label. From a mathematical optimization perspective, the modified fitness function is strictly increased when the term $Correlation\left(X\right)$ is decreased in the interval [-1,0), and strictly decreased when $Correlation\left(X\right)$ is increased in the interval (0,1]. On the other hand, $f\left(X\right)$ is strictly increased when $Correlation(X,l)$ is increased in the interval (0,1] and is decreased when $Correlation(X,l)$ is decreased in the interval [-1,0).

4.5.3. Used Correlation Functions in This Study

Numerous statistical functions are used to calculate the dependency between variables, however, in this study, we focused on one popular function. This correlation function is Spearman’s rank-order correlation coefficient. Spearman’s correlation coefficient is used to measure the strength of a monotonic association relationship among vectors [37].

Spearman’s rank-order correlation coefficient: This coefficient is a non-parametric (distribution-free) rank-based version of the Pearson coefficient which is utilized to measure the statistical dependency among the ranks of two vectors and measures the direction of the associated relationship among two vectors. It is most useful when the data distribution of the vectors has linear association and does not require the measure of these vectors in certain interval scale, because it is convenient for ordinal scale vectors which means that the data should be transformed to ranks before calculating this coefficient function [37]. It is defined as follows:

$$s=\frac{{\sum }_{i=1}^n\left((\mathrm{rank}\left(x_i\right)-\overline{\mathrm{rank}\left(X\right)})(\mathrm{rank}\left(y_i\right)-\overline{\mathrm{rank}\left(Y\right)})\right)}{\sqrt{{\sum }_{i=1}^n{(\mathrm{rank}\left(x_i\right)-\overline{\mathrm{rank}\left(X\right)})}^2{\sum }_{i=1}^n{(\mathrm{rank}\left(y_i\right)-\overline{\mathrm{rank}\left(Y\right)})}^2}}$$

(12)

where: $rank\left(x_i\right)$ and $rank\left(y_i\right)$ are the ranks of the observation in the sample, and $\overline{rank\left(X\right)}$ and $\overline{rank\left(Y\right)}$ are the mean rank of the variable X and the variable Y in the sample respectively. The value of Spearman correlation coefficient s lays in [-1,1] such that when s moves towards 0, that means that this is a weaker monotonic relationship between X and Y.

4.6. Binary Improved Black Hole Optimizer (BHO)

4.6.1. Background about Motivation for BHO

A Black hole is defined as what remains after a star has undergone complete gravitational collapse [132]. By the way of illustration and upon the strong gravity of the black hole, no object even light can get away from a black it. Furthermore, the chance that any object has to get away from the black hole after crossing its certain critical surface (event horizon) is 0. Moreover, when some mass reaches the field of the event horizon, it would be stretched toward the black hole and compressed perpendicularly as it falls, which thus is known as the spaghettification theory [133]. The spherical shape of the black hole is the event horizon and the radius of this sphere is named the Schwarzschild radius [134]. This radius is calculated as follows:

$$R=\frac{2GM}{c^2},$$

(13)

where G is the gravitational constant, M is the mass of the black hole, and c is the speed of light. By way of demonstration, any object that crosses R will be absorbed into the black hole and permanently disappear.

4.6.2. Mathematical Modeling of Black Hole Optimization

The mathematical modeling of BHO is depicted in this subsection. A number N-stars are placed in d-dimensional space (search space). All stars are assessed via an objective function and the best star is captured and marked as the black hole. According to the strong gravitational force of the black hole, stars start to move around the black hole which then would be absorbed. In other words, stars that move to the black hole and get closer to the event horizon of a black hole would be consumed. The black hole absorption procedure is formulated as follows:

$$X_{i,iter+1}=X_{i,iter}+rand*(X_{BH}-X_{i,iter})$$

(14)

where $X_{i,iter}$ and $X_{i,iter+1}$ are the locations of the old and new star at the current generation (iteration), respectively. $X_{BH}$ is the black hole in the current generation. $rand$ is a random number in the interval (0,1].

While stars are heading to the black hole, one of them may reach a better location according to the objective function, and in such cases, the black hole will move to that star and BHO will resume with the new black hole. Furthermore, when stars move toward the black hole, some of them may cross the event horizon according to some probability, and in such scenarios, these stars will be sucked by the black hole. To keep the number of candidate solutions constant, instead of each consumed star, a new one will be generated and distributed randomly in the search space.

The radius of the even horizon of BHO is calculated as follows:

$$R=\frac{f\left(X_{BH}\right)}{{\sum }_{i=1}^{N}f\left(X_i\right)}$$

(15)

where $f\left(X_{BH}\right)$ is the fitness of the black hole and $f\left(X_i\right)$ is the fitness of the star i in the current generation. Therefore, R is the ratio of the fitness score of the black hole over the summation of all fitness scores of all stars in the current generation. When the distance (like Manhattan distance in our work) between a candidate star and the black hole is less than R, then that star is absorbed and a new one is randomly generated.

4.6.3. Mitigating Main Issues in BHO Search Algorithm

Mitigating Local Optimal Issue: BHO, which is a population-based algorithm, suffers from the problem of falling into the trap of local optima because that may reach a point where all solutions are the same during the optimization process and cannot produce better ones. Therefore, we tend to overcome such a complication by utilizing a local search operator called inversion mutation [135]. This type of mutation is utilized for improving the diversity of the solutions in population-based approaches. It starts by selecting a subset of genes from the chromosome (solution) and inverts these genes such as flipping all genes to their opposite values like flipping all ones to zeros and vice versa, and the beginning and the ending of this subset are selected randomly.

BHO algorithm like other metaheuristic approaches suffers from the premature (early) convergence [136,137,138] and the trap of local optima [139]. Thus, searching in wider areas in the search space and exploring more locations to get better solutions than the current ones is essential. Here, we used a common evolution operator called mutation [140] in order to overcome the mentioned complications and improve the solutions’ diversity during the optimization process along the number of iterations. In binary problems like FS, each bit in the solution vector is flipped according to a ratio called Mutation Rate $MR$ which is set to $\frac{1}{|X_i|}$ where $|X_i|$ represents the length of the solution vector which is equal to the length of the given dataset. In this stage and for the given solution, the algorithm iterates over each bit of this solution and generates a random number in [0,1] and if this random is less than $MR$, it flips this bit from 1 to 0 and vice-versa otherwise, and moves to the next bit. Algorithm 1 shows the main steps of our modified BHO (MBHO) algorithm.

Mitigating Encoding Issue: In general, population-based algorithms evolve solutions according to a set of processes in order to generate the next generation which includes a set of vectors (chromosomes) of continuous genes (values). As our encoding criteria is based on a binary string of bits, continuous values should be converted to binary ones. A very reputable approach to handle this issue is Transfer Functions $TF$. These functions are common for such a task without changing the structure of the optimization algorithm. In this work, we utilized four V2-TF $(V_2=|tan\left(x_i\right)\left|\right)$[141] (shown in Figure 3). Each bit $x_j$ in the solution X is handled as follows:

$$x_i=\left\{\begin{array}{cc}0,\hfill & V_2\left(x_i\right)<0.5\hfill \\ 1,\hfill & V_2\left(x_i\right)\ge 0.5\hfill \end{array}\right.$$

(16)

Algorithm 1: Pseudocode of MBHO

Main Description of MBHO: The proposed approach is showcased in Algorithm 1. It outlines the Modified Binary Black Hole Optimization (MBHO) algorithm which is binarized through the $V_2$ transfer function where its behavior is shown in Figure 3, which is designed for tackling FS tasks. The algorithm begins by initializing parameters such as dataset path, correlation type, transfer function, maximum iterations, population size, and mutation rate. The algorithm evaluates the stars in the population, sorts them, and sets the best-performing star as the black hole. It then iterates through a loop for a specified number of iterations. Within each iteration, new positions for each star are generated and compared to the previous positions based on the modified fitness function (proposed in our work). If a new position outperforms the previous one, it is updated. The algorithm also calculates an event horizon and eliminates stars that cross it, replacing them with randomly generated ones. The mutation is applied to some stars with a certain probability, and if the mutation occurs, their positions are updated accordingly. The process continues until the maximum number of iterations is reached, and the best solution found, represented by the black hole (best feature set), is returned as the output of the algorithm.

4.7. Time Complexity Analysis

To analyze the complexity of our MBHO, we interpret each step in Algorithm 1 according to the big-O annotation and then provide the final or the total complexity with the knowledge that k is the number of keys, d is the number of variables (features) in the dataset, and n is the number of samples. The first statement is nothing but initializing the parameters of MBHO which takes constant time $O\left(C\right)$. The second statement evaluates all N stars (solutions) as a vital step for reporting the convergence curves and this takes $O\left(N\right)$, and since each solution, $X_i$ is evaluated as ML classification task by KNN, we consider the complexity for this as $O\left(kdn\right)$ and in total $O(N*kdn)$. The third statement indicates that the algorithm will stop if and only if the number of iteration counter $iter$ exceeds the bound max iteration M and this takes $O\left(M\right)$. Now, we have an inner loop iterates over the N solutions which take $O\left(MN\right)$, where a few statements inside this inner loop are interpreted as: first statement is generating new solution $X_{iter+1}$ from the current one $X_i$ by a function takes each bit from the solution vector as an input, and without the loss of generality, let the length of the solutions denoted by L, then this takes $O\left(L\right)$. The second and third statements are about comparing the new solution $X_{iter+1}$ with its old state $X_i$ and keeping the fitter one which takes $O(kdn+C)$. Fourth statement is assessing and comparing the solution $X_{iter+1}$ with the current best denoted as the black hole $X_{BH}$ and update the current best by $X_{iter+1}$ if and only if the score of $X_{iter+1}$ is better than $X_{BH}$ and this takes $O(kdn+C)$. After this loop, we calculate the event horizon R which indicates evaluating all N solutions and takes the summation of their scores and evaluate the current black hole and then its score is divided by that summation and this takes $O(N*kdn+C)$. Now, we have another loop iterates over all N solutions that the complexity becomes $O\left(M\right(N(L+2kdn+2C+N*kdn+C)+N)$, and the several statement in this loop are interpreted as: first statement is calculating the Euclidean distance between the solution and the black hole $X_{BH}$ and this takes $O\left(L\right)$. The second statement is eliminating the solution that crosses the event horizon R and replacing it with a new one and this takes $O(L+C)$. The third statement is replacing the black hole $X_{BH}$ by the new star $X_{iter+1}$ according to their fitness scores and this takes $O(kdn+L)$. The fourth, fifth, and sixth statements are about deciding whether to apply the mutation on the given solution $X_{iter}$ or not and this takes $O(3C+L+kdn)$. The last statement in this loop is updating the black hole $X_{BH}$ by the current solution $X_i$ if its score is better and this takes $O(kdn+C)$. And the final statement of MBHO is retrieving the black hole $X_{BH}$ as the optimal solution and this takes $O\left(C\right)$.

Therefore, the complexity becomes $O\left(M\right(N(L+2kdn+2C+N*kdn+C)+N(4L+5C+3kdn))+C)$, by simplifying that term, the final time complexity can be given by: $O\left(M\right(N(L+2kdn+N*kdn)+N(4L+3kdn)\left)\right)$, where k is the number of keys, d is the number of variables (features) in the dataset, and n is the number of samples.

We also emphasize that the main goal of this work is to extract a lower number of features while achieving good performance metrics (such as F-1 and Accuracy scores) for the classification problem, which is achieved through our proposed FS approach.

Having explained the main steps of our proposed MBHO FS approach and all related additions (including mitigating issues in BHO and involving correlation terms), we next present our extensive evaluation to test our proposed approach.

5. Experimental Results

In this section, we evaluate our proposed FS method. We first summarize our main key setups to ensure the proper execution and construction of all experiments. These include the specifications of the datasets used for model evaluation, the tools employed in the experiments, and the performance measurements applied across all experiments. We then show the main evaluation results. In particular, we discuss comparisons between our approach with different wrapper-based and filter-based FS approaches, statistical analysis based on the Friedman test, convergence curves, and box plots. All these elements collectively contribute to a comprehensive understanding of our experimental process for evaluating our FS method and the main findings.

5.1. Used Datasets

MBHO is validated using fourteen publicly available datasets. These datasets are sourced from the UCI ML repository [142]. The datasets are colon, Darwin, divorce, WDBC, leukemia, leukemia-3c, MLL, Parkinsons, sobar, sonar, SPECTFtest, SRBCT, urban, and WPBC. Colon dataset is widely used in binary classification, especially, predicting Colon cancer. Darwin dataset is widely used in ML tasks such as classification and regression analysis. Leukemia and Leukemia-3c datasets demonstrate various molecular or cellular features, widely used in machine learning tasks and differing in size. The MLL dataset includes various molecular or cellular features, widely utilized in data science tasks. The WDBC dataset represents various diagnostic measurements crucial for breast cancer research and analysis. The SRBCT dataset embodies diverse molecular or cellular attributes pertinent to machine learning research and analysis in the context of small round blue cell tumors. The Sobar dataset encapsulates diverse socio-behavioral features, facilitating research and analysis in relevant domains. The Parkinson’s dataset represents various clinical and demographic features relevant to Parkinson’s disease research and diagnosis. The Sonar dataset represents distinct acoustic signal features, valuable for research and analysis in underwater target detection or classification tasks. The Divorce dataset represents diverse socio-demographic and relationship features, crucial for machine learning research and analysis in understanding factors valuable in divorce prediction or prevention. The SpectTF dataset embodies various attributes related to spectral analysis or signal processing, pertinent for research and analysis in fields such as pattern recognition or medical diagnostics. The Urban dataset represents diverse urban features, facilitating research and analysis in urban planning, development, or sustainability studies. The WPBC dataset represents various clinical and demographic features pertinent to research and analysis in the context of breast cancer prognosis and treatment. As detailed above, this diverse selection of datasets with different applications helps us in better generalization of the performance of our FS approach for different datasets with different characteristics. A detailed description of all these datasets can be found in Table 1. These datasets were specifically selected due to certain characteristics such as high feature size, and binary classes, which are ideal for demonstrating the effectiveness of the feature selection algorithms (both our proposed one and those used as baselines in our evaluation). We underscore that the benefits of our suggested model apply to any given dataset with many features that need a careful and accurate FS approach.

5.2. Main Hyperparameters

We begin by detailing the primary hyperparameters utilized in various components of our framework. We use the KNN algorithm in the evaluation process with $k=3$ and the Manhattan distance function to measure the dissimilarity among a pair of instances. Also, we use k-fold cross-validation with 10-fold for better evaluation. The parameters for the MBHO were selected as follows: maximum iterations ($M=25$), population size which refers to a set of potential attack paths ($N=10$), mutation rate ($mr=0.5$), and weight factor ($w_f=0.001$). For the binary CS [38], we have set population size $N=30$, step-size $=1.5$, lambda $=2.5$, worst-nest-probability $=0.2$, delta $=1.5$, $mr=0.5$, and $M=25$. All tests were conducted using the Java language (JDK 17) on a machine with an Intel® Core™ i7-8750H CPU @ 2.20GHz (12 CPUs), and 32768MB RAM. The tools and libraries we used in this study are as follows: Java Development Kit 8 (JDK 8), Integrated Development Environment (IDE), IntelliJ ultimate version, Oracle Machine Learning Library, Tribuo 4.1.0, and XChart Library 3.8.0. Tribuo is a general-purpose open-source ML library written in Java that can be used for deep learning and Natural Language Processing applications as well [143]. Furthermore, we use the Apache Common Math 3 API for correlation calculation.

5.3. Baselines

We compare our proposed FS approach MBHO with four baseline approaches for selecting the most dominant subset of features over the fourteen datasets considered in our study. These baselines are both filter-based and wrapper-based FS approaches. The baselines are detailed as follows.

(i) Binary Cuckoo Search (BCS) [38]: Cuckoo Search (CS) is an optimization algorithm inspired by the reproductive behavior of cuckoo birds. These birds lay their eggs in the nests of other birds, increasing the chances of their own eggs hatching. To avoid detection, cuckoos mimic the appearance of the host bird’s eggs. Additionally, cuckoo eggs tend to hatch before the host’s eggs, allowing the chicks to assert dominance and secure a larger share of the food provided by the host bird.

(ii) Mutual Information Maximisation (MIM) [39]: MIM is a filter-based FS approach inspired by the idea of MI. It computes the MI among the class label and the selected features. In another term, MIM aims to select features that are highly informative about the target class while minimizing the redundancy, selecting the most relevant features that maximize the MI from the original dataset.

(iii) Joint Mutual Information (JMI) [40]: JMI is a filter-based FS approach that computes the amount of the shared information among at least two variables. It is an extension of MI. In other words, JMI tends to measure the relevance of features according to their joint information with the class label.

(iv) minimum Redundancy Maximum Relevance (mRMR) [41]: mRMR is a filter-based FS approach widely used in ML and applied mathematics aiming to select a subset of features that maximizes the correlation with the target class while minimizing the redundancy among the selected features.

5.4. Evaluation Measurement

To evaluate the performance of the MBHO, we utilize several metrics such as the duration of the FS process, $Accuracy$, $Sensitivity$, and the $F1$ score. These metrics, except for the duration of the FS process, are based on the confusion matrix, a crucial tool for gauging the effectiveness of machine learning algorithms. The confusion matrix consists of four main components: True Positive ($TP$), True Negative ($TN$), False Positive ($FP$), and False Negative ($FN$). In a scenario where we have a group of individuals, some of whom are infected with a specific disease, $TP$ refers to those who are indeed infected and correctly diagnosed, TN to those who are not infected and correctly identified as healthy, $FP$ to those who are healthy but mistakenly diagnosed as infected, and $FN$ to those who are infected but incorrectly identified as healthy. This classification task helps us understand the significance of these four components in the context of machine learning evaluation. Our main metrics are given as follows:

$$Acc=\frac{TP+TN}{TP+TN+FP+FN},$$

(17)

$$Racall=\frac{TP}{TP+FN},$$

(18)

$$Precision=\frac{TP}{TP+FP}$$

(19)

$$F1=2\times \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}.$$

(20)

Statistical Significance: We employ the Wilcoxon signed-rank test, a non-parametric statistical method. This test is used to compare two paired observations (FS algorithms in our evaluation here) across multiple cases (here, datasets). The goal is to determine if there is a significant difference in the median values of these two observations [144]. The test uses a p-value and a null hypothesis to ascertain this difference. The null hypothesis, which assumes no significant difference between the two observations, is only rejected if the p-value falls below a specified significance level $SL$. Here, we consider the $Acc$ as the main parameter for this statistical significance test. Friedman, as a non-parametric test, operates on non-uniform distribution values, thus bypassing the need for normality assumptions in decision-making. Its purpose is to determine if there are significant differences among at least three measurements within the same group of subjects concerning a skewed variable of interest, such as accuracy [145].

5.5. Evaluation Results

Having shown the main experimental setup and evaluation measurement, we now show our main evaluation results and their related discussions.

Evaluation of the Case with No Correlation:Table 2 presents a comparison among MBHO (with no correlation) and BCS (with no correlation), evaluated across various datasets. Each dataset is represented by a row, and the columns detail different performance metrics and characteristics of the methods. These include the name of the dataset, the classification accuracy, the F1 score, the time taken for each algorithm to execute, and the number of features used by each one. By comparing these metrics, one can evaluate the performance of the two FS algorithms across different datasets and conditions, such as which method has higher accuracy or F1 score, which runs faster, or which uses fewer features. This can assist in selecting the most suitable method for a specific task or dataset. It is important to note that the time is here formatted as hours: minutes: seconds: milliseconds, and the accuracy and F1 scores are likely proportions from 0 to 1. The features represent the number of variables used by the model to make its predictions, with a lower number indicating a less complex model, which can sometimes lead to better performance and interpretability. Moreover, according to the Wilcoxon test, these two algorithms are significantly different from each other due to the term of $Acc$ at $p<SL$.

In this comparison between MBHO (with no correlation) and BCS (with no correlation) algorithms over various datasets, MBHO generally outperforms BCS in terms of accuracy and F1 score, but it also tends to use more features and take a longer time to run. For instance, on the Colon and Darwin datasets, MBHO (with no correlation) achieves higher accuracy and F1 scores than BCS (with no correlation), despite using slightly more features and taking longer. On the Leukemia dataset, both methods perform equally in terms of accuracy and F1 score, but MBHO (with no correlation) takes longer and uses fewer features. Similar trends are observed on the Leukemia-3c and MLL datasets. On the WDBC, Sobar, and Divorce datasets, both methods perform equally in terms of accuracy and F1 score, but MBHO (with no correlation) takes longer and uses fewer features. For the SRBCT, Parkinsons, Sonar, and Urban datasets, MBHO achieves higher accuracy and F1 scores, but takes longer and uses more features. On the SpectTF and WPBC datasets, both methods have the same accuracy, but MBHO (with no correlation) has a slightly higher F1 score, takes longer, and uses more features. In total, MBHO (with no correlation) yields better accuracy in eight datasets of the fourteen datasets and a better F1 score in eight datasets of all datasets, compared to BCS (with no correlation). While MBHO (with no correlation) may provide a better performance, it may also be more computationally intensive and complex. Conversely, BCS (with no correlation) may be a more efficient choice in terms of runtime, but it may not perform as well as MBHO in terms of accuracy and F1 score.

Evaluation of Incorporating Proposed Correlation:Table 3 presents a comparison between MBHO (with correlation) and Binary CS (BCS) (with correlation)[38], based on the Spearman correlation function and a significance level $SL=0.05$. The metrics used for comparison are Accuracy (Acc), F1 score (F1), Time, and the number of Features. For the ‘Colon’ dataset, MBHO (with correlation) achieved an accuracy of 0.86 and an F1 score of 0.83 in 19:16:256 time with 1064 features, while BCS achieved an accuracy of 0.81 and an F1 score of 0.77 in 9:56:465 time with 1069 features. In the ‘Leukemia’ dataset, both MBHO and BCS achieved an accuracy of 0.99 and an F1 score of 0.94, but MBHO took more time (54:23:664) and used more features (1732) than BCS (31:49:948 time and 1730 features). For the ‘MLL’ dataset, both methods achieved an accuracy of 0.94 and an F1 score of 0.87, but MBHO took more time (14:32:17:233) and used more features (6500) than BCS (11:37:18:083 time and 6223 features). Lastly, for the ‘Divorce’ dataset, MBHO achieved an accuracy of 0.99 and an F1 score of 0.99 in 1:00:577 time with 14 features, while BCS achieved an accuracy of 0.98 and an F1 score of 0.98 in 35:622 time with 18 features. Moreover, according to the Wilcoxon test these two algorithms are significantly different from each other due to the term of $Acc$ at $p<SL$. In total, MBHO (with correlation) yields better accuracy in nine datasets of the fourteen datasets and better F1 scores in nine datasets. This shows the impact (enhancement) due to the correlation terms, proposed in this work, to enhance the fitness function used in the FS process.

Table 4 presents a comparison between two versions of the MBHO algorithm, categorized by their correlation status, detailing their performance across various datasets. The MBHO-correlation variant often outperforms the MBHO-no correlation counterpart in terms of $Acc$ and $F1$ scores across several datasets. Notably, MBHO-correlation achieves higher $Acc$ and $F1$ score in eight and five datasets, respectively, compared to MBHO-no correlation. However, for the SRBCT dataset, MBHO-no correlation achieves a better $F1$ score. In addition, both variants yield similar results in the remaining datasets for both $Acc$ and $F1$ scores, further emphasizing the effectiveness of incorporating correlation into the MBHO algorithm. For example, in the Leukemia dataset, MBHO-correlation achieves an $Acc$ of 99 % and an $F1$ score of 94 %, albeit with an execution time of 54 minutes and 23 seconds compared to MBHO-no correlation’s 9 minutes and 18 seconds. Similarly, in the MLL dataset, MBHO-correlation achieves higher $Acc$ and $F1$ scores but has a higher time for the execution compared to MBHO-no correlation. These results suggest that considering correlation leads to more robust FS, enhancing classification performance, yet requiring more computational time. Conversely, in datasets with a small number of features such as WPBC, both variants yield similar performance metrics and select the same number of features. Each variant has its benefit depending on the user’s need (MBHO-no correlation for better time and MBHO-correlation for better accuracy).

Evaluation of Comparing MBHO with Filter-based FS:Table 5 presents the performance comparison between MBHO with the Spearman correlation function and three baseline filter FS approaches: MIM [39], JMI [40], and mRMR [41]. Each method’s accuracy $Acc$ and $F1$ score are evaluated across fourteen datasets. The results show that MBHO with the Spearman correlation function generally achieves higher $Acc$ compared to the other methods. Specifically, in datasets such as Colon, Darwin, Leukemia, SRBCT, and Divorce, MBHO consistently outperforms the other methods in terms of both accuracy $Acc$ and $F1$ score. Additionally, the ranking analysis indicates that MBHO with the Spearman correlation function has the highest sum of ranks, implying its overall superiority across the evaluated datasets. However, it is worth noting that MBHO may require more time to generate the final subset of features compared to the other methods since it is an approximation approach. Overall, these findings suggest that MBHO with the Spearman correlation function is a promising approach for feature selection tasks for different applications (captured by different datasets), offering competitive performance across diverse datasets.

Evaluation of Convergence Speed of Proposed Approach: In this study, we analyzed the behavior of the convergence curves generated by the suggested algorithms. These algorithms are our method variants (MBHO-no correlation and MBHO-correlation), along with BCS variants. The experiment have been conducted based on the average fitness score of each algorithm over a series of iterations (ranging from 1 to 25). Figure 4 and Figure 5 illustrate the convergence curve using the datasets provided, where the X-Axis denotes the iteration count, and the Y-Axis signifies the $Acc$ value for each algorithm. According to Figure 4 and Figure 5, the proposed MBHO model has faster convergence than other algorithms due to its emphasis on maximizing the fitness score. Furthermore, the performance of the proposed algorithm has been enhanced (as shown in the figures by tending to find solutions with higher values) at each iteration. We emphasize that the mutation operation and correlation assessment have improved the performance (accuracy and F1) of the proposed algorithm by providing opportunities to explore the search space more extensively and replacing existing solutions with better ones, but led to slower overall time compared to BCS.

Evaluation of Distribution of Fitness Scores Across Iterations (Box Plots): In our research, we also examined the performance of the MBHO algorithm using the box plots produced. The box plots are used to visualize the distribution of fitness scores across different iterations. They help illustrate changes in the fitness scores over time, displaying the spread of values and central tendency. It provides important insights into the convergence behavior of the algorithm being used. The tests were carried out based on the mean fitness score of each algorithm across a range of iterations (from 1 to 25). The box plots are depicted in Figure 6 and Figure 7, which uses the provided datasets. The X-axis represents the number of iterations, while the Y-axis indicates the $Acc$ value for each algorithm. To provide a clear analysis, we first introduce the key factors of the box plot. The box depicts the interquartile range, the line inside the box is the median value of the average fitness score, and the lines extending from the box indicate the variability outside the upper and lower quartiles. According to Figure 6, MBHO with Spearman generally achieves the highest fitness scores across six datasets (of the eight datasets shown in this figure), although it exhibits considerable variability in certain datasets, suggesting fluctuating performance across different problem instances. Furthermore, according to Figure 7, MBHO with Spearman tends to have the highest fitness scores across four datasets (of the remaining six datasets shown in this second figure), with some variability observed. However, this variability indicates fluctuations in the performance across different datasets, highlighting the importance of considering the specific characteristics of each problem instance. Yet this fluctuation indicates that even when MBHO-correlation starts with low-quality solutions, through iterative evolution, it tends to reach better areas in the search space over time. This highlights the adaptive and evolutionary nature of the algorithm, which can progressively improve its performance through iterations, even when faced with initially suboptimal solutions. Regarding the fluctuated performance, the proposed approach, MBHO achieves the highest fitness score and accuracy across most of the used dataset.

6. Conclusions

FS is a binary optimization problem, acknowledged as one of the most intricate computational tasks. It is an NP-hard optimization problem that seeks to boost the effectiveness of learning algorithms by removing unnecessary and redundant variables from the data. This process benefits learners (e.g., ML classifiers) by enhancing their accuracy, decreasing the time needed for training and testing phases, and reducing time complexity, with a minor decrease in learner accuracy. In our research, we introduced a framework that employs a modified version of the black hole algorithm, named as MBHO. We also developed a revised fitness function to assess the best feature subset and improve the exploration process in the problem’s search space. We considered the interrelationships among the features in the feature subset and the dependencies between each feature and the assigned label. We utilized a transfer function, known as the V2 transfer function, to convert continuous values into discrete ones. This is because most evolutionary algorithms update their candidate solutions by artificially producing solutions with continuous values. This work presents an enhanced binary version of BHO by adopting inversion mutation to enhance the population diversity to prevent premature convergence and to avoid the trap of local optima. This allows these approaches to explore broader areas in the search space. We introduced the Spearman correlation function to compute the dependencies we added to our proposed fitness function.

Fourteen benchmark datasets from the UCI repository were used to evaluate the performance of our proposed FS approach, MBHO. The first comparison we made is with a popular wrapper-based FS method (BCS). This comparison is based on four evaluation measurements: $Acc$, $F1$ scores, execution time, and the number of features. The results demonstrated the capabilities of MBHO to improve accuracy, increase F1 score, and reduce the size of the dataset (via selecting effective features), which in turn reduces the time consumed during the training and testing phases. Based on the conducted experiments, the proposed model has shown better accuracy. We conducted additional experiments indicating the comparison between our approach and the other three filter-based FS approaches which are MIM, JMI, and mRMR. The results confirmed the capabilities of MBHO to improve accuracy, increase the F1 score, and show better performance than the other filters. Other experiments have been conducted including box plots and convergence curves. These comparisons showed that MBHO with the Spearman function has the fastest convergence speed among other algorithms in nine datasets and has the highest fitness score in nine datasets. Moreover, the results confirmed that the proposed model can be utilized as a promising mechanism for feature selection for real-world datasets with low, moderate, and high-dimensional variables. Additionally, the proposed model is expected to succeed in other fields such as engineering problems, data science, and many others. The proposed approach has the capability to be used with any other classifiers (e.g., KNN). This can help in enhancing the time complexity of that algorithm in selecting effective features.