Detecting Anomalies in IoT Devices: A Machine Learning-Based Solution

1. Introduction

The Internet of Things (IoT) emerged in 1999 with the introduction of Radio Frequency Identification (RFID) tags [1]. Fast forward twenty-four years and the IoT landscape has grown exponentially, becoming increasingly in demand. According to Cambridge consultants, IoT connections across various sectors are projected to reach 155.7 million between 2016 and 2024 [2]. As IoT devices become more integrated into our daily lives, the attack surface for these devices expands. In fact, Kaspersky reported a staggering 140% increase in IoT breaches from January to June 2021 [3]. Researchers have been working tirelessly to address IoT vulnerabilities and enhance device security. However, due to IoT devices' limited processing power and memory, traditional security methods fall short in the IoT environment. Furthermore, as manufacturers prioritize rapid market entry over comprehensive security measures, security concerns in IoT applications are often overlooked. Unlike conventional computing methods, soft computing solutions require less computational power. Therefore, recent studies tend to adopt soft computing solutions that better suit resource-scarce IoT devices. As the number of interconnected devices rises, safeguarding them from potential attacks becomes increasingly challenging. This situation stresses the importance of scalable, efficient anomaly-based intrusion detection models driven by machine learning algorithms. Abomhara and Køien [4] identified various IoT security challenges, such as user privacy, identity management, data sharing and protection, authentication, and authorization. Intrusion detection systems (IDSs) are crucial to IoT security as they monitor networks for suspicious activities [5]. Intrusion detection models can be divided into two categories: signature-based and anomaly-based models. While signature-based IDSs offer high precision by matching network traffic to predefined known attacks, they struggle to detect less frequent or novel attacks, like zero-day threats. In contrast, anomaly-based IDSs improve recall by learning unusual behaviors and patterns but suffer from high false detection rates, which remains an ongoing research challenge.

• Motivations and contributions

Several researchers have proposed anomaly detection techniques, most of which are ML-based. There are also several studies on combining different deep-learning methods for anomaly detection [6,7,8]. But still, three important research problems persist: combining ML algorithms like Decision Trees and Random Forests to harness their collective power, classifying the sub-categories of anomalous traffic using optimum datasets, and making predictions more efficient. Thus, an ensemble ML-based intrusion detection model is a promising solution that is independent of potential negligence from manufacturers and end users. Moreover, several existing studies were based on datasets that are not sufficiently representative of real-world IoT security issues. In this study, instead, we performed our experiments on the IoTID20 dataset, which resembles real-world security issues arising in IoT environments, to improve the practicality of the proposed solution. The main contributions of this research are summarized as follows:

• We developed an ensemble ML-based detection method against IoT attacks using the IoTID20 dataset. The proposed model makes three-faceted predictions: binary classification for anomalous or normal traffic, multi-class prediction for broad categorization of the attack, and classifying attacks within the sub-category. This layered approach provides a more nuanced understanding of the threat landscape, which can significantly aid in effective threat mitigation and response. To the best of our knowledge, this is the first work that applies an ensemble machine learning approach specifically to the IoTID20 dataset. This dataset represents a specific and complex IoT environment with varied network attacks, which poses unique challenges that our model effectively addresses.
• We provided a proof of concept implementation of the proposed ensemble ML-based solution, which consists of a diverse mix of individual classifiers (Decision Trees, Extra Trees Classifier, K-Nearest Neighbors, and Random Forests), creating a more robust and reliable system for anomaly detection. This particular combination is a novel contribution that enhances the performance and stability of the detection system.

The paper is structured as follows: Section 2 reviews relevant work on detecting IoT attacks using the IoTID20 dataset. Section 3 explains the methodology behind constructing the IDS model. Section 4 presents the results achieved by the proposed IDS model, and Section 5 concludes the paper.

2. Related Work

In our earlier work, we conducted an extensive review of recently published studies in IoT intrusion detection [6]. The published survey provided a detailed analysis of widely used IoT datasets. The results of this analysis motivated our decision to use the IoTID20 dataset. In addition, the survey compared the performance of existing ML-based solutions built using the IoTID20 dataset. In this section, we extend the findings of our earlier study with more recent related work utilizing the IoTID20 dataset and different approaches followed by researchers to mitigate IoT abuse.

• Mitigate IoT Abuse

To overcome the shortcomings of anomaly detection methodologies that depended solely on cloud computing or single-source time series data, the authors [9] presented a distributed edge computing model for anomaly detection in IoT-based industrial sustainability. An Edge Computing based Anomaly Detection Algorithm (ECADA) was introduced to detect abnormalities in single-source and multi-source time series data. Experimental results showed that the algorithm was superior to its competitors when presented with novel anomalies. This research demonstrates the promise of edge computing in solving the transmission and processing difficulties of anomaly detection in real time.

CorrACC is a novel hybrid feature selection approach proposed by [10] to improve the accuracy of machine learning algorithms for identifying anomalous and malicious traffic in Smart Internet of Things (SIoT) networks. The authors emphasize the significance of feature selection for accurate anomaly and intrusion detection in IoT networks. Correlation Attribute Evaluation (CAE) and specific machine learning classifier accuracy (ACC) are used to select effective feature sets for a particular classifier which is used in their proposed method. The authors evaluate their approach on the Bot-IoT dataset and demonstrate that, relative to other feature selection techniques, their proposed approach obtains accuracy rates greater than 95%.

Another relevant study in distributed computing and mobile communication security presents DeepAutoD (D-AD), an innovative unpacking framework for secure feature extraction in distributed machine learning-based malicious code detection [11]. The authors address the challenge of malicious apps that employ advanced techniques to conceal high-risk code, which can hinder efficient mobile communication. By integrating deep deception call chains, D-AD effectively identifies mainstream Apps in the App-market and offers customizable algorithms for advanced Android systems. This work demonstrates that the proposed framework outperforms existing solutions regarding safety and effectiveness, making it a noteworthy contribution to the domain of Android security.

The authors [12] propose a novel method for identifying malicious traffic flows in IoT networks using machine learning algorithms. They demonstrate the efficacy of their approach by applying four distinct machine-learning algorithms to the Bot-IoT dataset. The experimental results indicate that the proposed method obtains an average accuracy of greater than 96%, which is superior to other feature selection techniques. The authors emphasize the importance of accurate feature selection for accurate malicious traffic detection in IoT networks and propose a new feature selection metric called CorrAUC, which eliminates inaccurate features and selects effective ones based on the area under the curve (AUC) metric. CorrAUC, a new feature selection algorithm based on the wrapper technique, accurately filters the features and selects the most effective ones for the chosen machine learning algorithm. Using an integrated TOPSIS and Shannon entropy based on a bijective soft set, the authors validate the desired features for identifying malicious traffic. This work significantly improves the security of IoT networks by providing an efficient and accurate method for detecting malicious traffic flows.

b.

Recent Related Work Utilizing IoTID20 Dataset

Authors in [13] proposed a MidSoit IDS, which consists of three stages; the first stage identifies connected IoT devices, stage two identifies whether the traffic is benign or malicious, and stage three identifies the attack type of the malicious traffic identified in stage 2. Finally, the results of the third stage are sent to an action manager component that triggers necessary actions. The proposed model was evaluated on IoTID20, CIC-IDS-2017, and BOT-IoT. The authors then tested the model on nine machine learning classifiers, including LSVM, QSVM, K-Nearest Neighbor (KNN), Linear discriminant analysis (LDA), QDA, multilayer perceptron, LSTM, autoencoder, and Decision Tree (DT) classifier. The proposed model achieved class category accuracy of 99.15% using the IoTID20 dataset. Furthermore, the analysis was performed without sampling, under-sampling, and Synthetic Minority Over-Sampling Technique (SMOTE). It was evident that the model performed better without sampling.

Authors in [7] proposed a DCNN model consisting of 8 layers. The first layer is a 1D convolution layer followed by a max pooling layer, a 1D convolution layer, a max pooling layer, a flattened layer, and finally, three DNN layers. Before evaluating the model, the authors performed data preprocessing and the ETC feature selection method resulting in 62 selected features that achieved information gain greater than 0.001. The authors tested five optimizers, where Nadam was the top-performing optimizer. The proposed model achieved an accuracy of 99.84% and an F1-score of 99.34% for binary classification. As for class category classification, the model achieved an accuracy of 98.12% and an F1-score of 97.46%. Finally, for the multiclass subcategory classification, the model achieved an accuracy of 77.5% and an F1-score of 76%.

Table 1 provides a summary of the related studies on the IoTID20 dataset. Most of these studies have modeled the IDS problem as a binary classification problem. However, while the binary classification model is essential to distinguish suspicious from normal traffic, further information is needed to provide a comprehensive defense before, during, and after attacks. Therefore, this work focused on identifying the category and subcategory instances in the IoTID20 dataset.

As the IoTID20 dataset is relatively new and closest to a real-world IoT environment, there is more research potential on the dataset, specifically implementing Ensemble machine learning to IoT environments. Three out of the outlined literature review have applied ensemble machine learning, out of which two were applied to binary classification, and the third was applied to multi-class classification achieving an accuracy of 87%. This paper will focus on building an Ensemble-based IDS model that can detect attacks targeting IoT devices using the dataset IoTID20.

3. Methodology

This section presents the proposed approach to detecting IoT-based attacks along with the selected features and implementation requirements of the intrusion detection model.

• IoTID20 Testbed Environment

To safeguard against security risks that face IoT devices, it is essential to understand the underlying architecture. The IoT architecture is divided preliminarily into three layers [5]. The perception layer is the physical layer which consists of edge devices such as sensors, CCTV cameras, and actuators. This layer is responsible for gathering information from its surroundings, making it a sensitive layer as it is relatively exposed. The network layer is responsible for connecting IoT devices to network devices and servers by acting as a median to transfer data from one point to another via various network protocols. The application layer is responsible for providing professional services to the end user. In this paper, a recent IoT dataset was chosen, which includes possible attacks on all three layers of the IoT environment. The studied IoT dataset includes four attack types as follows:

• Denial of Service Attack: an attack that tries to compromise the availability of its target by making it unresponsive.
• Mirai Attack: an evolved type of DoS attack named DDoS. DDoS attacks use the same method as DoS. However, this time a group of attackers are attacking the victim simultaneously.
• Man in the Middle Attack: an attack that tries to eavesdrop on communications taking place between two nodes, where the attacker starts acting as a proxy.
• Probing Attack: an attack that scans the network looking for vulnerabilities that could be exploited to help perform an attack on the device.
• Probing Attack: an attack that scans the network looking for vulnerabilities that could be exploited to help perform an attack on the device.

The IoTID20 dataset was created using Pcap files from the IoT Network Intrusion dataset [6,17] to detect malicious activities in IoT networks. The testbed resembles an environment similar to a smart home with a number of interconnected IoT devices. The IoTID20 testbed consists of an AI speaker (SKT NGU), security camera (EZVIZ Wi-Fi Camera ), smartphones and laptops connected to an access point. The dataset consists of 625,783 records and 86 features, of which 83 are network features and 3 are label features. Table 2 outlines the three label features and the number of records for each class.

Figure 1 outlines the general approach of the proposed IoT intrusion detection model. To build the model, we used a Windows 10 pro computer machine. The technical specifications of the CPU are (Ryzen 5 5600x, 6 cores 12 threads) and (16GB 3600 MHZ) RAM. The model was built on a Visual Studio Code environment. Python was used as a programming language utilizing Scikit-learn, Numpy, and Pandas libraries.

Data preprocessing was the first stage, followed by extracting the most relevant network features. The quality of data has a direct impact on the machine learning models' performance. The detailed description of this stage is as follows:

• Converted all data types to integer values to ensure data is understood by the machine learning model.
• Removed any infinity values found and replaced them with NaN values instead.
• Replaced NaN values utilizing the mean strategy ‘simple imputer’ imputation transformer from sklearn, which replaces NaN values by the mean column value.
• Performed label encoding on three columns (Binary, Multi-Class Category, and Multi-Class Subcategory) which transforms categorical columns into numerical values.
• Dropped the remaining categorical features to reduce margin of error (Flow_ID, Dst_IP, Src_IP, Timestamp)
• Dropped constant features as per the correlation matrix of the IoTID20 dataset; they contain only one value for all the outputs in the dataset and provide no information.

b.

Overall IDS Model Design Phase

Phase Two of the proposed approach includes testing the impact of two sampling techniques that could potentially enhance the machine learning model by solving the challenge of the imbalance found in the IoTID20 dataset. A random under-sampling technique [24] was utilized where the majority class is reduced to the minority class by randomly deleting instances to result in a balanced dataset. In addition, the SMOTE oversampling technique [25] was utilized to produce new synthetic data samples. The output of this phase will result in a balanced dataset.

Phase three of the suggested model studies the effect of dimensionality reduction techniques that reduce the number of input variables in a dataset where important features are chosen to help reduce computational time, cost of modeling, save storage space, improve the detection model, and avoid the overfitting of the ML model. Principle component analysis [26] was utilized as a dimensionality reduction method where principal components with a cumulative variance > 99% were chosen.

Phase four of the suggested model approach splits the dataset into training and testing datasets using stratified K-fold cross-validation with a K value of 10. Stratified cross-validation enhances K-fold cross-validation by preserving the class distribution in the training and testing datasets [27]. The training dataset is then fed into the machine learning model to train the model with possible attacks, while the test dataset is used to test and evaluate the machine learning model.

c.

Ensemble Machine Learning Model Selection Phase

Ensemble machine learning combines multiple base learners to train the dataset, creating a model with improved stability and performance. [28] outlined three reasons why ensemble learning outperforms single classifiers; (1) the training data may provide insufficient information to choose a single best classifier. The ensemble algorithm solves this by averaging their votes, which reduces the likelihood of selecting the incorrect classifier. (2) the learning algorithm's search process may be imperfect; ensemble learning solves this as the search would be originating from different points, which could lead to better accuracy. And finally, (3) the hypothesis space being searched may not express a true function. However, applying various weights of the hypothesis makes it possible to enlarge the space of the expressible function. As previously outlined, the IoTID20 dataset is an imbalanced dataset. Consequently, ensemble methods are more stable than single models and less affected by dataset class imbalance [29].

During the development of the ensemble machine learning model, multiple classifiers were tested to achieve optimum results. Of the eleven classifiers, Decision Tree, Random Forest, KNN, and Extra Tree achieved better results and are used to build an advanced ensemble ML classifier. Advanced ensemble methods are divided into three types: (1) Bagging, (2) Boosting, and (3) Stacking. Figure 2 displays the results of the best-performing ML ensemble classifier for the binary, multi-class category, and multi-class subcategory classification.

As depicted in Figure 2, the stacking ensemble model performed better than bagging and boosting. Stacking has the advantage of utilizing multiple ML classifiers as base learners, followed by a meta-classifier. The meta-classifier combines the different predictions made by the base learners and improves them, yielding better results. Figure 3 outlines the stacking ensemble model utilized throughout this paper.

4. Experiments and Results

The experiments and findings obtained for binary, category, and subcategory classifications are described in this section. In addition, the impact of sampling and dimensionality reduction on the machine learning model was tested in each experiment. Accuracy is a widely used metric to assess the performance of machine learning models and will be used to compare the results of this study with the previously outlined literature. However, accuracy is a misleading metric when dealing with imbalanced datasets as it would be predicting based on the majority class, yielding high accuracies [24]. Instead, in imbalanced datasets F1-Score is the metric to rely on.

• Binary Classification

In binary classification, the IDS model predicts whether the traffic is malicious or benign. Two experiments were performed in this stage. The first experiment was to test the effect of the dataset imbalance on the IDS model, followed by testing the effect of dimensionality reduction utilizing PCA. For dataset imbalance, a comparison was conducted between a balanced model utilizing random under-sampling and an imbalanced model. We found that the IDS model achieved an accuracy of 100% in both instances; however, a higher F1-score and recall in the imbalanced model of 100% compared with 97.1% and 98.1% in the balanced model. As the IDS model achieved better results while keeping the dataset imbalanced, further analysis has been performed by applying dimensionality reduction, phase three of the proposed model. After applying PCA dimensionality reduction, we noticed that the model performed better by utilizing all features of the IoTID20 dataset, as the F1-Score decreased from 100% to 93.70%. Likewise, the recall has also decreased from 100% to 92.40%. Hence, the IDS model for binary classification was built by utilizing all features of the IoTID20 dataset.

Table 3 outlines the detailed results of the stacking IDS model for binary classification per class type, and Figure 4 outlines the confusion matrix for binary classification.

In binary classification, the Ensemble IDS model achieved a recall of 100%, which means it can correctly identify benign and attack instances. Furthermore, the model achieved a precision of 99% of positive predictions in benign instances and 100% of positive predictions in attack instances. As the dataset is imbalanced F1-score is used as a measure to calculate the accuracy of the model, where it achieved 99%.

b.

Multi-Class Classification (Categories)

In multi-class category classification, the IDS model predicts whether the traffic is benign, DoS, Mirai, MITM or a Scan attack. Similar to binary classification two experiments were performed in this stage, the first experiment was to test the effect of the dataset imbalance on the IDS model. Followed by testing the effect of dimensionality reduction utilizing PCA. To test the impact of the imbalanced dataset on the IDS model, both under sampling and SMOTE oversampling techniques were utilized to achieve a balanced dataset and a comparison was made with a model built utilizing an imbalanced dataset. Table 4 outlines the six different methods tested for multi-class category classification. In each method the model was under sampled and/or oversampled to reach a value of one of the class types. The IDS model achieved good results on all methods where the difference between them was nearly non-existent. However, after PCA dimensionality reduction was applied it was noticed that the IDS model performed better utilizing all features of the IoTID20 dataset. The accuracy of the balanced datasets de-creased by 17-20%, while the F1-score and recall decreased by 32-37%.

Table 5 outlines the detailed results of the stacking IDS model per class type, and Figure 5 that outlines the confusion matrix for category classification.

In multi-class category classification, the Ensemble IDS model achieved a recall of 99% for the normal class, which means the model was able to identify 99% of benign instances and was able to detect attack instances where it achieved 100% for DoS, followed by 99% for MITM, Mirai, and Scan attack instances. Furthermore, the model achieved an overall precision of 97.2% of positive predictions. As the dataset is imbalanced F1-score is used as a measure to calculate the accuracy of the model, where it achieved an overall of 98%.

c.

Multi-Class Classification (Sub-Categories)

In multi-class subcategory classification, the IDS model predicts whether the traffic is benign, DoS, MITM, Brute Force, HTTP Flooding, UDP Flooding, ACK Flooding, Host Port or Port OS attacks. Similar to both binary and category classification two experiments were performed in this stage, the first experiment was to test the effect of the dataset imbalance on the IDS model. Followed by testing the effect of dimensionality reduction utilizing PCA. To test the impact of the imbalanced dataset on the IDS model, both under sampling and SMOTE oversampling techniques were utilized to achieve a balanced dataset and a comparison was made with a model built utilizing an imbalanced dataset. Table 6 outlines the ten different methods tested for multi-class subcategory classification. In each method the model was under sampled and/or oversampled to reach a value of one of the class types. The IDS model performed optimum results without sampling; however, the results were very close between the various methods tested. On the other hand, after applying PCA dimensionality reduction the IDS model performed better utilizing all features of the IoTID20 dataset and out of all class classifications, subcategory class was mostly affected by PCA where the accuracy decreased from 88% to 55%.

Analyzing the previous results, it is concluded that a stacking IDS model with no sampling and feature selection yields better results. To be able to secure IoT devices it is vital that the model achieves optimum results. Table 7 outlines the detailed results of the stacking IDS model built for subcategory classification. In addition, Figure 6 that outlines the confusion matrix for subcategory classification.

d.

Ensemble Techniques

The built stacking model achieved results better than other ensemble methods as previously outlined in Figure 2. Furthermore, the proposed stacking model was compared with individual ML classifiers where stacking achieved better results than all individual classifiers especially in multiclass subcategory classification. Figure 7, Figure 8 and Figure 9 outline the comparison between the proposed stacking model and individual ML classifiers where all models were tested on no sampling methods for a fair comparison.

e.

Comparison with Related Previous Work

This section outlines the comparison between the built stacking IDS model with the previously outlined literature as follows:

• Binary Classification: Out of the previously outlined literature, nine papers evaluated their models against binary classification. Figure 10 summarizes the results of the literature in comparison with the proposed IDS model. The proposed stacking IDS model performed better than some of the literature, and in some instances, the performance was similar with 100% accuracy.

2.

Multi-Class Category Classification: Out of the previously outlined literature, only three papers evaluated their models against multi-class category classification. Figure 11 outlines the results of the literature in comparison with the proposed IDS model. The proposed stacking IDS model performed better than the literature in the category classification task. The proposed model achieved an accuracy of 100% and recall, precision, and F1-score of 99%.

3.

Multi-Class Subcategory Classification: Out of the previously outlined literature, four papers evaluated their models against multi-class subcategory classification. Figure 12 presents the results of the literature in comparison with the proposed IDS model.

The main focus of the paper is to detect subcategory classification to be able to secure IoT environments. We should be able to detect what type of attack was initiated in order to better respond. A summarized comparison between the proposed models and earlier studies is given below.

• Researchers in [17] evaluated the IoTID20 dataset on six ML classifiers. Out of the six classifiers, the decision tree achieved the best result with 88%, followed by an ensemble method with 87%.
• Researchers in [22] evaluated the IoTID20 dataset on four classifiers. Of the four, DT performed the highest detection rate in detecting 6 of the nine classes. Figure 13 displays the ROC-AUC curve for the multi-class subcategory classification of this study. The proposed Ensemble model in this study achieved similar results to the DT classifier in [22] in terms of AUC except for the Scan host port class, where in [22], AUC is 96 compared to 81 in this study.
• The proposed IDS model achieved better results than both [21] and [7].

5. Conclusions

In this work, we developed a machine learning-based intrusion detection approach to detect malicious traffic in IoT networks and identify further details about the attack, which we referred to as the attack category and subcategory. We trained different ML algorithms on the IoTID20 dataset to perform binary and multi-class anomaly detection tasks. Our experiments led to the development of a staking ensemble anomaly detection model that achieved robust detection results. In particular, our binary classifier surpasses the results reported by earlier studies. Moreover, our multi-class classification models achieved F1-scores of 99% and 88% in the multi-class category and subcategory classification tasks, respectively. These results suggest the proposed ensemble model can reliably detect anomalies in IoT devices. While IoT devices produce massive data that enhance the potential of machine learning-powered IoT security solutions, the practical applicability of these solutions is highly challenged by the class imbalance with noisy data problems. Hence, our foreseen future work is focused on alleviating the adverse effect of class imbalance, using various preprocessing and feature selection techniques.