Feature Selection with Weighted Ensemble Ranking for Improved Classification Performance on the CSE-CIC-IDS2018 Dataset

1. Introduction

Feature ranking plays a key role in the data preprocessing workflow of the development and training of a network intrusion detection system owing to the high dimensionality of the training data. Feature selection can contribute to faster training, better performance, reduced overfitting, and better interpretability [1]. In several cases, ensemble feature selection methods can outperform individual feature ranking methods when developing a classifier [2]. By the combination of multiple feature ranking algorithms, one can improve the robustness of the overall feature ranking. Thus the final set of selected features is less likely to be influenced by outliers or biases in any single ranking method. Additionally, ensemble feature selection methods can also contribute to better coverage of the feature space and a more comprehensive evaluation of the different attributes, which could improve the performance of the classifier. Another benefit of the combination of different feature selection methods is that they can improve the discriminative ability of the selected attributes by identifying features that would be neglected otherwise by some ranking methods. Furthermore, by reducing the sensitivity of the selected features to the given training data, ensemble feature selection methods can help improve the capability of the generated model to classify previously unseen data accurately. This is especially important in network intrusion detection systems, where the model needs to identify new and emerging threats accurately.

The above presented ideas led us to the development of a novel ensemble feature selection approach, where the final scoring of the features is calculated by a weighted average of the individual scores obtained by the application of six feature selection techniques, and the optimal values of the weights are determined using Taguchi’s design of experiments approach. The new method was used in the course of the development of network intrusion detection modules trained with different attack types available in the CSE-CIC-IDS2018 dataset. The primary contribution of this paper is the utilization of weighting techniques for calculating the feature score and the determination of the weights.

The rest of this paper s organized as follows. Section 2 gives a short overview of some ensemble feature selection methods. Section 3 presents the proposed ensemble feature selection method and the incorporated feature selection techniques. Section 4 describes the results obtained with the CSE-CIC-IDS2018 dataset. The conclusions are drawn in Section 5.

2. Related works

Ensemble Feature Selection (EFS) is an approach that combines the results of multiple individual feature ranking techniques to improve the selection of relevant attributes and enhance model performance. This technique offers several advantages, including better classification accuracy, increased efficiency on large data sets, and the ability to address data overfitting. Utilizing ensemble feature selection involves leveraging the strengths of various feature selection algorithms to identify significant features in a dataset. By doing so, it enhances classification accuracy, reduces overfitting, and ensures greater stability in the selected features. This approach proves particularly beneficial in machine learning-driven applications like intrusion detection systems, where feature diversity significantly impacts model accuracy and training duration. However, there are some drawbacks to using EFS as well. Running all models requires significant computational resources, and finding the right balance between model accuracy and computation time can be challenging [2].

In their research, M. Manonmani and S. Balakrishnan utilized the density-based feature selection (DFS) method as a filtering approach to rank the features of a dataset. The DFS results were then given to an envelopment-based optimization technique called Improved Teacher Learner Based Optimization (ITLBO) algorithm to find the optimal feature set containing the most important features for high-accuracy prediction. The results of the ensemble feature selection method were evaluated using (Support Vector Machine) SVM, Gradient Boosting (GB), and Convolutional Neural Network (CNN) classification algorithms. SVM achieved a high classification accuracy of 93%, Gradient Boosting achieved a high classification accuracy of 97%, and CNN achieved a high classification accuracy of 97.75% for the derived optimal feature set. The proposed work achieved a feature reduction of 62.5% for the 8 features selected using SVM and CNN classification algorithms and 66.6% for the 9 features selected using GB classification algorithm [3].

A. Hashemi, M. B. Dowlatshahi, and H. Nezamabadi modeled joint feature selection as a multi-criteria decision-making process (MCDM) for 10 real datasets with varying numbers of features. They utilized the VIKOR method to rank the features based on the evaluation of several feature selection methods as different decision criteria. The proposed method first obtains a decision matrix by ranking each feature according to different ranking criteria. Next, a score is assigned to each feature. Finally, the output is a ranking vector of features from which the user can select the desired number of features. The results of their approach demonstrate its superiority over other similar methods in terms of accuracy, F-score, and algorithm runtime. Their approach also performs quickly and efficiently.[4].

N. Hoque et al. present a method called Ensemble Feature Selection using Mutual Information (EFS-MI), which combines subsets of the results of different feature selection methods, including InfoGain, GainRatio, ReliefF, Chi-square, and SymmetricUncertainty, to obtain the optimal subset of features[5].

A.S. Sumant and D. Patil processed high-dimensional datasets (HDD) using multi-step methods, specifically, Chi-square integrated with RReliefF (ChS-R) and Symmetric Uncertainty integrated with RReliefF. The results were then validated using Random Forest (RF), k-nearest neighbors (KNN), and SVM classifiers.The proposed ChS-R system achieved an accuracy improvement of 13.28%, while SU-R achieved an accuracy improvement of 9.47% [6].

Chih-Fong Tsaiand Ya-Ting Sung in their research describe several feature selection methods, including Principal Component Analysis (PCA), Genetic Algorithm (GA), and C4.5 decision tree, specifically for high-dimensional, low sample size (HDLSS) data. We also explore their nine parallel and nine row combinatorial approaches to the results, including union, intersection, and intersection. Our test results indicate that the row-based ensemble feature selection approach is particularly suitable for processing very high-dimensional datasets [7].

In their study, J. Wang et al. used the UCI machine learning dataset to propose SA-EFS based on sort aggregation. The feature selection methods employed included Chi-square, maximum information coefficient, and XGBoost. The performance of the method was evaluated using KNN, Random Forest, and XGBoost classifiers [8].

3. Methodology

The following section of the article presents the feature selection methods utilized in the study. Subsequently, the results obtained are utilized to rank the characteristics by means of the ensemble feature selection method, whereby the weighted average of the method results is calculated.

3.1. Feature Selection Methods

The process of selecting features involves identifying the most relevant attributes that can be effectively used for classification or prediction [9,10,11]. This contributes to reducing the dimensionality of the problem, and therefore reduces resource requirements such as storage and computation. Additionally, it can improve the performance of machine learning algorithms [12] by enabling faster training, reducing overfitting, and sometimes leading to better prediction power. The following subsections provide a brief description of the feature selection methods used.

3.1.1. Information Gain

One of the most commonly used univariate methods for evaluating attributes is the Information Gain (IG) filter. It assesses features based on their information gain and examines each feature individually. The Information Gain filter employs a symmetrical measure. It sorts all features in a methodical manner and necessitates the establishment of a threshold for selecting a specific number of features based on the obtained order. A drawback of the Information Gain criterion is that it tends to favor features with more values, even when they do not provide more informative data [13].

3.1.2. Gain Ratio

Gain Ratio (GR) is a variant of IG that mitigates its partiality. Unlike the standard Information Gain, the GR accounts for the number and size of branches when selecting an attribute. It addresses the Information Gain’s bias by incorporating the intrinsic information of a split. The intrinsic information refers to the entropy of the distribution of instances across branches, which represents the amount of information required to determine which branch an instance belongs to. As the intrinsic information increases, the value of the attribute decreases [14].

3.1.3. Chi2

The chi-squared test for feature selection is a statistical technique used to identify the most relevant features for a given set of data for a target variable. It works by comparing the observed distribution of the values of a characteristic with the expected distribution under the assumption of independence between the characteristic and the target variable and selecting those characteristics for which the difference between the observed and expected distributions is the largest [15].

3.1.4. Symmetric Uncertainty

Symmetric uncertainty is a means of determining the fitness of features for feature selection. It involves computing the relationship between the feature and the target class. Features that exhibit a high value of SU are considered to be of greater importance [16].

3.1.5. Relief

Relief is a feature selection method that serves as an individual evaluation filter. It computes a proxy statistic for each feature, which can estimate its quality or relevance to the target concept (i.e., predicting endpoint value). These statistics are known as feature weights, or informally, feature scores, ranging from -1 (worst) to +1 (best) [17].

3.1.6. ANOVA

ANOVA is a widely recognized statistical method used for comparing multiple independent means. This technique evaluates features by computing the ratio of variances between and within groups and then ranks them accordingly [18].

3.2. Weighted Ensemble Ranking

Weighted ensemble ranking is a widely used approach for assessing samples, allowing for differential evaluation of each component based on its significance, importance, strength, or any other criterion referred to as its weight. By considering the contributions of multiple feature ranking methods, the weighted average of the feature scores is computed using Equation (1). This equation provides an overall evaluation score, reflecting the combined assessment of the ensemble.

$$R_{ens}=\frac{R_{IG}·w_{IG}+R_{GR}·w_{GR}+R_{SU}·w_{SU}+R_{{\chi }^2}·w_{{\chi }^2}+R_{Re}·w_{Re}+R_{AN}·w_{AN}}{w_{IG}+w_{GR}+w_{SU}+w_{{\chi }^2}+w_{Re}+w_{AN}},$$

(1)

where $R_{ens}$ is the score of the feature calculated by the ensemble method, $R_{IG}$, $R_{GR}$, $R_{SU}$, $R_{{\chi }^2}$, $R_{Re}$, $R_{AN}$ are the normalized feature scores obtained by the individual feature ranking methods included into the ensemble, while $w_{IG},w_{GR},w_{SU},w_{{\chi }^2},w_{Re},w_{AN}$ represent the weights associated to these methods.

By incorporating multiple feature ranking methods and assigning appropriate weights to each method, the ensemble approach effectively leverages the strengths of individual techniques while mitigating their weaknesses.

3.3. Weight Optimization using Taguchi’s DoE Approach

The arithmetic mean of individual feature scores is the simplest method for aggregating different scores, where each weight is identical. However, employing different weights can sometimes lead to feature scores that contribute more significantly to the selection of an improved feature subset. Such a subset enables better classification results to be achieved. Determining the optimal combination of weights is a challenging task due to the substantial time required for evaluating the various feature collections resulting from score calculations. Therefore, weight optimization with a minimal number of trials becomes necessary.

This recognition has led to the utilization of a well-known design of experiments (DoE) technique known as the Taguchi method. Developed by Genichi Taguchi in the 1950s, this approach originally aimed at quality management and design in the manufacturing industry [19]. The Taguchi method sought to identify and optimize the effects of different production parameters on product quality. By identifying optimal parameter settings during production, the Taguchi method reduces sensitivity to variations and enhances overall product quality.

To identify the optimal parameter setting, the Taguchi method employs the concept of "parameter design." This involves assigning process variables to predefined value ranges, conducting tests, and optimizing them. In the course of this research, six independent variables have to be tried, each of them at two levels. Therefore the $L_8{2}^7$ orthogonal design plan was adopted. In the case of each factor, two levels were used, coded by 1 and 2 (see Table 1).

3.4. Classification Methods

In the course of the current research, three classification methods were employed to evaluate the selected feature subsets. A brief description of each method is provided in the following subsections.

3.4.1. Decision Tree

The decision tree method is a well-known algorithm in machine learning that is used for both classification and regression tasks. It works by creating a tree-shaped model that represents decisions and their potential outcomes. Each node in the tree represents a decision based on a specific feature, while the branches represent the different possible values or outcomes of that feature. The end nodes, or leaves, correspond to the final predicted class or numerical value. The goal of the decision tree algorithm is to identify the best points to split the data, which either maximizes the information gained or minimizes the uncertainty, leading to effective decision-making. Decision trees are easy to interpret, comprehend, and can handle both categorical and numerical features. They have found wide applications in various fields such as finance, healthcare, and marketing, primarily due to their simplicity, versatility, and ability to capture complex relationships [20].

3.4.2. Random Forest

The random forest method is a powerful technique in machine learning that is commonly used for classifying and predicting outcomes. It works by creating many decision trees and combining their predictions to achieve accurate and reliable results. Each decision tree is built using a random selection of features and training data, which helps prevent overfitting and increases the diversity of the trees. When making predictions, the random forest combines the outputs of all the trees, either by taking the majority vote (for classification tasks) or by averaging (for regression tasks). This approach improves overall prediction performance, effectively handles noisy data, and is capable of handling high-dimensional feature spaces [21].

3.4.3. SVM

A support vector machine (SVM) is an effective supervised machine learning algorithm utilized for classification and regression tasks. Its main objective is to construct an optimal hyperplane that effectively separates different classes within a feature space of high dimensionality. By maximizing the margin between data points of distinct classes and minimizing classification errors, SVMs can handle both linearly separable and non-linearly separable data. This is achieved through the use of kernel functions, which map the data into higher-dimensional spaces [22].

4. Experimental Results

In the course of this research, we utilized the CSE-CIC-IDS2018 dataset [23]. The dataset was created by the Canadian Institute for Cybersecurity Laboratory and was chosen because it fulfilled all the research requirements, providing information on total traffic, labeled data, and multiple attack types. It encompasses various attack types, while our study specifically focused on FTP, SSH, SQL, XSS, and Web attacks. Each record in the dataset contains 80 attributes extracted from the recorded traffic using CICFlowMeter-V3 [24].

In our previous work [25], we successfully reduced the dimensionality of the problem by excluding certain features. These features either had single-valued columns or contained irrelevant information. Thus, we narrowed down to 69 the number of features to be considered and further investigated. Subsequently, various feature selection methods (see Section 3.1) were employed to evaluate the individual features. The resulting score values were later normalized and aggregated using the arithmetic mean, yielding a single-value evaluation. Based on these feature scores and ranking thresholds, we selected feature subsets and tested them with different classification methods. Finally, each classifier’s performance was evaluated using accuracy, precision, recall, and F1 measures on both the training and test datasets. In cases where the trained classifiers exhibited poor performance, it motivated us to explore further the weighted average approach.

To facilitate a better exploration of the weight search space with minimal experiments, we assigned the weight values of 0.0233 and 0.2336 to the two levels of the weight variables (called factors in DoE) included in the selected DoE design. The rationale behind this choice was to use values that are significantly distant from each other. Due to the considerable time required for experiments, conducting an exhaustive search was not feasible. For each feature, we determined eight sets of weights based on Table 1. The resulting scores after applying these weights can be found in the five tables provided in the Appendix A During the experiments the same datasets were used as in [25].

Primarily, we directed our attention toward cases where the previous investigation using the arithmetic mean did not yield satisfactory results. Our aim here was twofold: either identifying feature sets with fewer features while maintaining the original classification performance, or finding feature sets that could enhance classification performance using accuracy, precision, recall, and F1 scores as performance measures. The steps of the process are outlined in Figure 1.

The results of the investigation are summarized in Table 2, Table 3, Table 4, Table 5 and Table 6.

5. Discussion

Feature selection plays a critical role in training classifiers with large datasets, as it enables the identification of relevant and informative features, thereby improving performance and efficiency. This paper investigated the incorporation of a weighting mechanism to enhance the ensemble feature selection approach. Our hypothesis was that using a weighting mechanism can enhance our previous approach, which involved utilizing multiple individual feature scoring methods and calculating the arithmetic mean of their normalized scores.

During the investigation, the classification algorithms were trained and tested using a visual programming technique with Orange. The workflows were created by linking predefined widgets and parameterizing them.

For the FTP dataset, the number of features considered could be reduced from 8 to 5, while maintaining excellent performance across all three classification methods. Similarly, for the SSH dataset, we observed a comparable pattern. In this case, the number of features could be reduced from 7 to 6, while achieving the same or potentially enhanced performance.

Parallel to the improvement of SVM classification measures in the case of the SQL dataset, the number of necessary features could be reduced from 26 to 7. Similarly, for the XSS dataset, the number of necessary features could be reduced from 10 to 2. In the case of the Web dataset, the number of necessary features could be reduced from 44 to 13. Furthermore, while the simple average-based solution provided poor results with the SVM classifier for both the train and test datasets in the case of the SQL and Web datasets, the new approach resulted in a significant improvement in performance measures.

Evaluation of the resulting classification performance measures clearly demonstrates that weighting the scores provided by different feature scoring methods can lead to a better ensemble method. Future research will investigate the applicability of different fuzzy techniques (e.g., [26,27,28]) in the ensemble feature ranking method.