CICIoMT2024: Attack Vectors in Healthcare devices-A Multi-Protocol Dataset for Assessing IoMT Device Security

1. Introduction

Nowadays, the Internet of Things (IoT) has a growing presence in society’s daily lives. These lightweight devices can be easily deployed and maintained [1,2], enabling extensive adoption in different environments. The development of smart cities also relies on the advances of such technology [3]. Solutions that were once considered futuristic can be achieved by integrating ubiquitous systems with advanced analytics [4,5,6]. Besides, the current adoption of existing advanced technologies represents an ideal platform for new solutions [7,8]. For example, wearable devices (e.g., smart watches) can support health monitoring applications at scale once they have been widely adopted. This new paradigm provides multiple services that can be beneficial in a variety of areas, e.g., transportation [9], education [10], and energy [11]. IoT devices play very distinguished roles in each application while observing different levels of constraints (e.g., safety-critical systems). One of the most promising areas for the use of IoT devices is healthcare, referred to as the Internet of Medical Things (IoMT) [12,13,14].

There are several examples of healthcare services supported by IoMT devices. For example, an important area in patient care is health monitoring [15]. Observing different physiological metrics is vital in multiple treatments and continuous measurements can improve the process results [16]. In this case, IoMT devices can play the role of physiological monitors since they can be set up easily and provide critical information to medical personnel [17]. Also, they provide mobility and simplicity in the hospital environment. Another important area is smart medication delivery, in which the process of medicating patients is automated with the amount prescribed by the doctors [18]. In this case, infusion pumps are becoming more and more popular in healthcare institutions and support treatments with an automated and precise approach. Remote medicine is another concept that has been growing in popularity in the past few years [19]. Allowing patients to be treated and monitored remotely can improve the overall practice by providing more comfort and continuous checks. In this application, IoMT devices can play the role of sensors by collecting important data and actuators by providing some local service (e.g., automated medication delivery).

However, although IoT devices are beneficial in many applications, there is an increasing concern with the cybersecurity aspects of these devices [20]. In the past few years, several attacks against IoT infrastructures have been launched [21,22], and the consequences of such threats depend on the environment considered. IoT devices present a limited pool of computational resources compared to traditional systems (e.g., servers). This is a key factor to observe in terms of the device security, the simpler network profiles these components bring to the network (e.g., limited pool of protocols), and the possible open doors for attackers [23,24]. For instance, IoT devices can be exploited in Advanced Persistent Threats (APT) as key vectors. Furthermore, compared to traditional vulnerability assessment and patching, there is a current lack of standards for IoT devices. For example, the National Institute of Standards and Technology (NIST) maintains a repository for the vulnerability management of several devices called the National Vulnerability Database (NVD) [25]. Vulnerabilities discovered that can affect devices in use are shared in this database to enable the general public to be aware and patch their systems. Regarding IoT, devices are very diverse, with multiple brands, firmware, and purposes. Although there are some initiatives to create standard repositories for IoT [26], this diversity makes it difficult for vulnerabilities to be tracked and patched for all devices. The main issue refers to the increase of IoT as an important asset in many environments as well as a main target for attackers. Knowing the benefits of IoT to businesses motivates companies to invest in this technology, but the fragile security defences of such devices may lead attackers to exploit and disrupt IoT operations [27].

These benefits and cybersecurity concerns also apply to healthcare applications. Although adopting IoT devices can be beneficial in several medical procedures, the tradeoff between usefulness and cybersecurity needs to be observed [28]. Relying on insecure devices can bring risks that may compromise the quality of the treatment provided. Besides, some IoT devices are not widely adopted in specific environments. For example, infusion pumps are devices adopted in healthcare and unavailable in other environments (e.g., transportation). The limited use of this device is another factor in hardening the discovery and patching of potential vulnerabilities. For example, IoT devices have been targeted in many ransomware attacks against healthcare organizations in the past few years. In fact, security solutions need to consider this aspect in different areas of healthcare. Furthermore, there may be several consequences resulting from such attacks. In terms of confidentiality, sensitive information from patients may be disclosed [29]. In terms of integrity, vital readings can be manipulated to compromise the accuracy of the treatments provided. Regarding availability, attacks may disrupt critical services and lead the operations to unsafe states. Finally, considering healthcare as a safety-critical application for IoT, the consequences of attacks can be critical, and solutions to improve the protection of such operations are necessary.

Given the complexity and amount of data generated by IoT and IoMT network traffic, advanced methods have become especially useful in these environments. In this context, Machine Learning (ML) brings various techniques and solutions that can improve the detection, prevention, and mitigation of cyber-attacks [30,31]. By analyzing the networking patterns in a multidimensional space, ML can identify anomalies in the network, abnormal IoMT traffic, and characteristics of attacks that can be launched (e.g., Distributed Denial-of-Service - DDoS) [32,33]. ML can enable the automated detection of potential threats and help healthcare professionals deliver high-quality services with a continuous security monitoring approach. Moreover, the characteristics of IoT network traffic can vary greatly depending on the devices and environments. For example, a temperature sensor can be implemented to send temperature readings once a day. At the same time, an aircraft sensor can be implemented to send structural readings regularly [34]. Besides the temporal differences, IoT services can also rely on multiple protocols (e.g., MQTT), which can include specific patterns in the network traffic. Similarly, IoMT devices can operate differently depending on the application. For example, the traffic generated by an infusion pump is usually less frequent than a continuous physiological monitor. These differences can be learned or captured by ML techniques to identify if abnormal traffic is sent through the network.

Although there are some datasets available for the development of security applications in IoMT, there are important features not addressed in the current state-of-the-art contributions [35]. For example, operational diversity is a critical aspect to consider to mimic a realistic IoMT environment. Hence, there is a need for an extensive testbed composed of several IoMT devices comprising both network traffic for Intrusion Detection Systems (IDS) as well as for IoMT profiling. IoMT features and profiles can drive ML and other advanced analytics methods to improve the security of these systems. For this reason, establishing a realistic and extensive topology with IoMT devices of different subcategories is required. Another key feature to support new solutions is the adoption of multiple protocols. To mimic a healthcare IoMT infrastructure, a topology needs to rely on the use of multiple services that would naturally adopt different protocols (e.g., Wi-Fi, Bluetooth, and MQTT). Finally, special attention needs to be given to defining experiments to represent the data transmission in real topologies, both in terms of attacks and in everyday operations.

Thereupon, the main objective of this research is to propose a realistic benchmark dataset to enable the development and evaluation of IoMT security solutions. In this effort, 18 attacks were executed against an IoMT testbed comprising 40 IoMT devices (25 real devices and 15 simulated devices), considering the plurality of protocols used in healthcare (e.g., Wi-Fi, MQTT, and Bluetooth). These attacks are categorized into five classes: DDoS, DoS, Recon, MQTT, and spoofing. This effort aims to establish a baseline complementary to the state-of-the-art contributions and supports researchers in investigating and developing new solutions to make healthcare systems more secure using different mechanisms (e.g., machine learning - ML). The main contributions of this research are:

• Development of a Comprehensive IoMT Security Dataset: This paper introduces the CICIoMT2024 dataset, an advanced effort to construct a realistic and multi-protocol benchmark for the security of the Internet of Medical Things. By executing 18 different cyberattacks against a diverse set of 40 IoMT devices, we contribute to the Healthcare field by providing a comprehensive dataset containing most of the protocols in the devices in this field, namely Wi-Fi, MQTT, and Bluetooth protocols.
• Innovative Methodology in IoMT Attack Simulation and Data Collection: A unique aspect of this paper is the systematic approach to simulating and capturing IoMT network traffic under various cyberattack scenarios. Considering the complex network of healthcare organizations, it is essential to consider several attributes, such as understanding the effects of different types of attacks, including DDoS, DoS, Recon, MQTT, and spoofing, using a combination of real and simulated devices. Advanced network monitoring techniques and specialized hardware, such as network taps, ensure the high fidelity of data collection.
• Profiling IoMT Device Lifecycle for Enhanced Security Understanding: This research goes beyond merely conducting attacks on IoMT devices. We also attempt to capture the lifecycle of these devices in different vital phases of a device from the moment they join the network until they leave. Mobility in healthcare organizations is considered to be a regular aspect. Thus, profiling the lifecycle of these devices, encompassing various operational phases such as power interaction, idle, active, and interaction states, becomes very important. This study offers an in-depth understanding of the devices’ behavioral patterns by meticulously capturing and analyzing the behavior of IoMT devices from the moment they join the network. This profiling is critical in identifying and mitigating potential security vulnerabilities.
• Multi-dimensional Evaluation: This research extends beyond dataset creation to evaluate the efficacy of multiple machine learning algorithms in detecting and classifying IoMT cyberattacks. By assessing techniques like Logistic Regression, Adaboost, Random Forest, and Deep Neural Networks, the study not only benchmarks the current state of ML in IoMT security but also opens avenues for future exploration in algorithm optimization and feature engineering.

The paper is structured as follows: Firstly, Section 3 reviews related works and identifies the main aspects of this research. Secondly, Section 2 presents the primary aspects of IoMT applications in healthcare. After that, Section 4 introduces the CICIoMT2024 dataset and explains the phases involved in the data collection. Section 5 presents the method used in this research to evaluate multiple aspects of the data collected and Machine Learning (ML) algorithms. Then, Section 6 describes the feature extraction process and provides a description of the data collected. Finally, Section 7 and Section 8 present the ML evaluation in identifying different attacks and the conclusion of this research.

2. Internet of Medical Things (IoMT) in Healthcare

Nowadays, several medical solutions rely on IoMT devices to enhance or accelerate treatment. These devices bring many advantages compared to traditional medical equipment while ensuring the accuracy of the multiple procedures considered. Various areas in the healthcare practice can benefit from adopting IoMT infrastructure, and this Section presents a discussion on how such devices can be useful in this context as illustrated in Figure 1.

Remote medicine and alert systems use IoMT to simplify patient care in remote conditions while observing the continuity and readiness of services [36]. This area includes not only teleconsultations but also refers to remote diagnosis and therapy [37,38]. Remote care poses requirements other than the needs of in-person care. To address them and ensure effectiveness, alert systems are vital and vary depending on the patient’s condition. In this case, general emergency notification services are responsible for informing medical personnel that medical attention is required [13,39]. Also, critical monitoring informs the medical staff of physiological conditions based on multiple parameters [12,40], while fall detection services inform that patients have fallen and need support [41,42]. Besides, some solutions ensure the medications and treatments are used as medical doctors recommend and are also referred to as medication reminders [43,44]. In all these cases, IoMT offers a simpler approach to remote medicine and alert systems by integrating devices and software platforms [45,46].

Furthermore, another critical area in patient care is the evaluation of sleep patterns and oxygen levels in the blood. This area is called sleep and oxygen assessment and helps medical staff treat sleep and respiratory disorders and improve the overall medical practice by monitoring cardiovascular health [47,48,49]. These solutions comprise using IoMT devices for sleep analysis, including monitors and actuators [50]. Also, IoMT devices can act as oxygen saturation as part of the continuous monitoring framework alongside integrated systems (e.g., software platforms that can control IoMT devices) [51,51,52]. Another important category of IoMT devices in this context is wearable technology, which offers comfort and simplicity in patient care. Finally, alert systems can also be adopted in this context [53,54].

Monitoring technologies are present in many areas of medical practice, including heart rate and cardiovascular solutions. IoMT devices enable measuring and tracking parameters necessary to cardiovascular health, e.g., pulse rate, blood pressure, and oxygen saturation [55,56,56]. These parameters are critical to ensure patients’ well-being, and IoMT helps ensure good condition continuously. For example, heart rate tracking can be achieved using devices such as smartwatches, chest straps, and fitness trackers [57,58,59]. Similarly, Electrocardiogram (ECG) monitoring devices detect anomalies in the heart beating pace [60,61]. Besides, blood pressure and overall cardiac monitoring and telemetry are areas in which IoMT can enable effective and continuous real-time treatment. Finally, deploying IoMT solutions can also include using software platforms to simplify the data management of information sharing [51].

Patient care also includes wellness. In this case, IoMT can leverage the periodic and continuous measurement and evaluation of wellness indicators [62,63]. Solutions include using smart thermometers capable of sharing essential data with medical applications [64]. Tracking devices are especially important in this area through the use of fitness trackers and wearable devices [65,66]. Besides, chronic conditions can be continuously observed with such technologies, and treatment can be optimized [67]. The same applies to patients recovering from illnesses who can benefit from using IoMT devices. Finally, this area extends beyond illness detection, comprising wellness tracking in preventive efforts.

Chronic condition management comprises managing and treating chronic diseases in different cases, targeting diseases such as diabetes, hypertension, and asthma [68,69]. In this application area, IoMT devices are designed to attend to the particular needs of each patient. For example, continuous monitoring needs to be regarded for specific metrics such as blood pressure (hypertension) and blood glucose levels (diabetes) [56,70]. IoTM is essential in engaging medical staff and patients in chronic condition management because it enables remote care, personalized treatments, and preventive measures.

A particular area of care that IoMT devices can support is the respiratory function evaluation. These solutions monitor and evaluate the health of the respiratory system and are crucial in special conditions (e.g., asthma and sleep apnea) [71,72]. Devices can be used to measure the inhaled and exhaled airflow to ensure an appropriate lung function that can be vital in some treatments and can be used by smart inhalers [73]. The oxygen saturation level in the blood is another critical indicator for respiratory conditions in remote care practice.

Muscle activity and neurological function represent another medical care area that IoMT devices can improve. It uses advanced solutions to enhance muscular and neurological health conditions [74]. Such solutions act not only in treating diseases but also in the prevention and recovery. For example, Electromyography (EMG) devices can be used alongside neurological monitoring devices to evaluate the health of muscles continuously [75]. Physical rehabilitation is another area of IoMT applications since patients’ movement patterns can be closely assessed (e.g., balance) [76]. Finally, solutions for different types of treatment can be developed using IoMT devices to support remote care and patient comfort.

Furthermore, safety and posture detection helps medical personnel improve patients’ safety levels considering their postures and movement patterns [77]. These solutions enhance the treatment of different movement-related diseases and can include remote services such as fall detection and real-time posture monitoring [78,79]. Regarding remote care, sensors can be distributed in a smart home environment to ensure the patient’s continuous care and comfort [80]. IoMT solutions play a unique role in these applications, with devices deployed to support these functions and the software platforms that manage and share the data collected.

Moreover, medication management is crucial to ensure patients are treated efficiently. Infusion and medication management is an area of medical care that deals with methods to automate medication delivery and improve the patient’s treatment [81,82]. IoMT devices can be used in different ways in this context. For example, infusion pumps are widely used in hospitals and can provide several treatment benefits [83]. This extends to environments outside hospitals, where medications can be remotely infused, and data can be shared with the medical staff.

Finally, new applications are being developed using IoMT devices in healthcare [84]. These solutions aim to improve the medical practice by offering comfort, remote care, higher accuracy, and interconnection. Besides, the hardware of such devices is improving, and more robust solutions are expected in the next few years [85,86]. Although IoMT brings many advantages to healthcare, there are challenges to ensuring secure operations. In this sense, resources that enable the development of cybersecurity solutions in this context are vital, e.g., datasets that mimic natural environments with extensive testbed.

3. Healthcare dataset in IoT domain

In [87], the researchers proposed an intrusion detection system for IoT devices in the Healthcare environment. Their proposed method has merged network and biometric metrics to enhance IDS development. Their experimental setup involved a hybrid method, including an Electrocardiogram, PM4100 Six Pe Multi-Sensor Board, a Gateway simulated in the Windows operating system, Blood Oxygen Saturation (SpO2), and a device to measure Blood Oxygen Saturation (SpO2). They conducted attacks such as spoofing and data alteration and investigated scenarios where different vulnerabilities can be exposed.

In [88], the authors present the ECU-IoHT dataset to demonstrate the exploitation of IoT devices in a healthcare environment. An analysis of attack behaviour is conducted to support the development of new security solutions comprising many real devices (MySignals, temp sensor, BP sensor, HR sensor, Bluetooth and wireless adapter,Kali and Windows Laptob ). Concerning the attacks launched, the authors focus on scaling, ARP spoon, DoS, Smart, and injection.

The authors in [89] introduce a dataset focused on how attacks are executed against IoMT topologies with a special focus on Bluetooth. An important discussion on the technical aspects of this protocol is presented alongside its applicability to the healthcare context. Multiple devices are used, and several attacks are launched, followed by an in-depth ML analysis considering the evaluation of multiple algorithms (e.g., Support Vector Machine, K-Means, and Deep Neural Networks).

Hussain et al. [90] introduce a data generation method to support the design of IoT security solutions focused on healthcare. This method, named IoTFlock, enables the generation of both normal and malicious network traffic. The dataset proposed adopted many IoMT devices and resulted from the network traffic captured during. Several types of attacks can target the MQTT protocol. These attacks include Distributed Denial of Service (DDoS), brute force, SlowITE, and MQTT publish flood attacks.

The study described in [91] concentrates on a commonly used protocol in industrial healthcare systems, namely IEC 60 870-5-104. The research aims to address the issue of limited resources available for this protocol. Several attacks, such as Man-In-The-Middle and DoS, were executed as part of the research. Furthermore, the study evaluated the effectiveness of machine learning techniques in the analysis process.

Table 1 summarizes the contributions and compares them with the characteristics of this research. In fact, the CICIoMT2024 provides an extensive list of real IoMT devices while executing several attacks. Besides, extensive IoMT profiling is part of the contributions of this research.

Moreover, other IoT security datasets are not necessarily focused on healthcare applications. "IoT-SH [92] presents data related to twelve attacks, categorized into four classes, that were carried out against eight smart home devices.". Kitsune [93] is a dataset comprising four different classes of attacks conducted for 9 IoT devices. MedBIoT [94] is built using an IoT network topology with a combination of real and emulated devices. The authors in [95] introduce the IoT-23 dataset as a botnet dataset. IoTIDs (2020) [96] is another IoT security dataset comprising the execution of four attacks against two devices. On the other hand, the authors in [97] and [98]introduce the MQTT datasets. Finally, [99,100], and [28] propose IoT datasets comprising more extensive testbeds and experiments.

4. The Proposed CICIoMT2024

This Section introduces the details of how the CICIoMT2024 dataset was planned, generated, and captured. This discussion includes the facilities where all experiments were conducted, the topology adopted, malicious traffic generation, and aspects of IoMT profiling.

4.1. CIC IoT lab

Establishing an IoT lab with several devices is difficult for many reasons. Such devices require supporting network devices (e.g., routers, access points, and switches) to connect and a team capable of setting up all the configurations needed. Furthermore, purchasing these devices at scale requires planning and financial investments that are not readily available. The Canadian Institute for Cybersecurity (CIC) has invested in establishing a well-equipped IoT lab. This investment comprises the acquisition of tens of IoT devices for multiple purposes (e.g., healthcare devices, home automation devices, and next-generation devices), several supporting network systems (e.g., routers, switches, access points, servers, adapters, sniffers, and networks taps), IoT kits (e.g., Arduino and Raspberry Pi’s), and miscellaneous devices. Besides, a technical team is dedicated to maintaining and managing the current IoT devices, network, and inventory while analyzing new IoT devices that can be purchased and included in our topology.

Figure 2 illustrated the CIC IoT lab. Devices are connected across the IoT lab in multiple places as several power plugs are available. There are racks across the room to simplify the organization of devices. Finally, all devices are labelled to streamline management and identification.

4.2. IoMT Topology

The CICIoMT2024 comprises the use of several devices and multiple purposes. Figure 3 illustrates the devices used alongside the network segregation. The main goal of this topology is to simplify the process of capturing network traffic from different protocols while mimicking realistic operations. Besides, devices are separated based on the protocol used to enable protocol-specific attacks to be executed and captured.

An iPad acts as a remote controller for several devices across the network. This iPad and four Raspberry Pi’s are connected to an access point. In this case, the Raspberry Pi is a malicious device that launches all attacks. Then, this access point is connected to a Netgear switch, which gives access to the IoMT devices as well as to the Internet provider. This Internet connection is essential since many IoMT devices need to connect to remote servers.

Furthermore, all the traffic between the switch and the IoMT devices is captured using a network tap. This device is a dedicated hardware that enables real-time duplication of packets without affecting the network performance. The traffic is collected and stored by network monitors. Then, this network tap is connected to another access point that acts as a gateway for the IoMT devices.

This access point connects all IoMT devices. First, 15 simulated devices are connected to the network using the MQTT protocol. Table 2 lists all simulated devices and presents their IP addresses, time profiles, categories, and simulated value range. Second, 7 Wi-Fi devices are also connected to the CICIoMT2024 topology. Table 3 lists all devices alongside their MAC addresses. Note that the iPad remote controller and the attacking Raspberry Pi are also presented in the table. Finally, 14 Bluetooth Low-Energy (BLE) devices are connected through a smartphone. These devices are listed in Table 4.

4.3. Generation of Malicious Traffic

To enable the development of cybersecurity solutions, a benchmark dataset needs to mimic aspects of real IoMT deployments and operations. To accomplish this, we consider aspects of different protocols as well as the characteristics of individual devices.

4.3.1. Wi-Fi

The devices contained in the Wi-Fi topology involve 4 Raspberry Pi’s attackers, and a remote controller (iPad) connected to one Access point. The healthcare devices are connected to the other access point on the other side of the network tap. These devices comprise 3 healthcare devices and 4 cameras. Several attacks were executed against Wi-Fi devices. ARP spoofing was carried out to perform Man-in-the-middle attacks [101]. Regarding Dos and DDoS [102,103], we executed ICMP Flood, SYN Flood, TCP Flood, and UDP Flood. Finally, reconnaissance attacks were conducted [104,105], including Port Scan, OS Scan, Ping Sweep, and vulnerability scan.

4.3.2. MQTT

To simulate IoMT devices operating on the MQTT protocol, we used IoTFlock [90]. This simulation platform generates network traffic and is developed for IoMT applications. The simulator was run using a VMware image (Ubuntu OS running on version 18.04.6 LTS) containing an executable IoTFlock GUI C++, allowing the creation of executable environments using XML. Once an XML has been created, IoTFlock parses it, and runs the simulation. In fact, all simulated MQTT devices that publish numerical values do not include units of measurement. Also, the MQTT topology contains a local connection between a PC that hosts the IoTFlock virtual machine and a Ubuntu laptop running an MQTT broker.

A MQTT switch establishes this connection. The IoTFlock PC shares network interfaces with the Virtual Machine (VM). Furthermore, the iPad acts as a remote controller and also as an MQTT subscriber.

Moreover, three different attacks were executed against MQTT devices. First, the MQTT Connect Flood was executed. This attack floods the MQTT broker with several connect packets to establish a connection [106]. To accomplish this, a custom Python script was developed to conduct both Denial-of-Service (DoS) and Distributed Denial-of-Service (DDoS) attacks. Second, MQTT Publish Floods were also conducted. This threat uses publish packets under random topics and was executed in a DoS and DDoS format. Finally, the MQTT Malformed Data attack was executed. This attack uses the MQTTSA tool [107] and tries to sniff the broker by sending specific packets to understand its behavior. In this process, the attacker collects the names of topics that have been published to it and attempts to send malformed data. To accomplish this, we modified the tool to send each topic collected some malformed data.

4.3.3. Bluetooth Low Energy (BLE)

Given the criticality of the data BLE devices generate in the IoMT context, it is paramount that they are secured against potential attacks. However, BLE requires a particular approach to both execute attacks and capture the network traffic. To accomplish this, we used a smartphone connected to the BLE devices. Then, an attacker PC conducts the malicious activities.

In this case, the Bleak library is adopted as a tool for Bluetooth operations in Python and scanned nearby BLE devices in a discovery approach. The script attempts to establish a connection and fetch all the services and characteristics to determine the data format. This effort empowers attackers to potentially disrupt the device’s operations.

The script then iterates through each characteristic, attempting to write data packets of varying sizes, ranging from 20 bytes to 810 bytes in increments of 10 bytes. The data packets consisted of repeating numeric sequences, e.g., an arbitrary data packet would follow a trend as “0123456789012345...". All successful writing was logged, noting the characteristics’ Universally Unique Identifier (UUID) and the data format that succeeded.

After identifying the successful UUIDs and their corresponding data formats, the script entered a continuous loop, constantly writing data to these UUIDs. This continuous writing aimed to potentially overload the device or disrupt its regular operations, thereby executing a DoS attack.

To gather comprehensive data, we not only examined benign logs (Bluetooth snoop log on the Android device where the IoT device is supposed to communicate with using Android SDK) but also utilized the Ubertooth One sniffer to capture attack logs right from the PC where the Attack script originated. This dual-log approach ensured we had a holistic understanding of the device’s behavior under both normal and attack conditions.

Furthermore, this methodology was replicated for each healthcare device and the outcome provided valuable insights into the resilience and vulnerabilities of these devices in the face of BLE-based attacks:

• Lookee Sleep Ring: showcased resilience against the attack, operating without any noticeable disruptions.
• Powerlabs HR Monitor Arm Band: Similarly, this armband maintained its standard functionality during the attack.
• COOSPO HW807 Armband: In stark contrast, our attack had a significant impact on this device. It became overwhelmed and subsequently turned off.
• Livlov Heart Rate Sensor: This heart rate sensor managed to resist our attack, operating without any observed disruptions.
• Wellue O2 Ring: This device remained unaffected and continued to operate normally.
• Lookee O2 Ring: This device was vulnerable to our attack. Upon execution, the device became overwhelmed and turned off.
• Checkme BP2A: This device displayed a unique behavior. While it stored data during the attack, it only transmitted this data once a Bluetooth connection was re-established.
• SleepU Sleep Oxygen Monitor: This monitor resisted our attack, showcasing its resilience by maintaining standard functionality.
• Rhythm+ 2.0 (Scosche): significantly affected by our attack, this device became overwhelmed and subsequently shut down.
• Wellue Pulsebit EX: This device withstood the attack and continued to operate without disruptions.
• Checkme O2 Smart Wrist Pulse Oximeter: It resisted our attack, continuing its standard operations.
• Kinsa Thermometer: Our attack impacted this device uniquely. While under attack, it was not possible to reset the connection by turning off the thermometer. The only methods to terminate the connection were to either let the battery deplete or to halt the attack. The device behaved as it remained connected throughout the attack.

Finally, as illustrated in Figure 4, the Bluetooth experiments were executed inside of a Faraday Cage to block external signals that may interfere with the network traffic collected.

4.4. IoMT Profiling

Understanding the behavioral aspects of IoMT operations is vital to enhancing these systems’ security. To enable the development of such methods, we describe how the generation and collection of profiling data were conducted for different experiments.

4.4.1. Power Experiments

The power experiments for Wi-Fi enabled devices are carried out by disconnecting all the devices from the network. In these experiments, the MQTT network is not considered as a simulated network. Hence, we are only interested in the 7 Wi-Fi devices. Furthermore, all Raspberry Pi’s from the previous sections were disconnected. The iPad is the only device connected to the network to control and monitor devices.

The captures were done similarly to CICIoT2022 [100], wherein all devices are disconnected from the network. The device we are interested in is powered on, and its behavior is captured by filtering the MAC address for about 2 minutes. The device is later powered off, and capture is allowed to continue for another 3 minutes for leftover packets or until no packets are transmitted from the device. However, some devices did not have physical power buttons, and adjustments needed to be made to perform the experiments:

• Singcall Sensor: This device does not explicitly have a power button instead it has a reset button. The device is disconnected from the network when reset and connects back to it when we reconnect the device to the app.
• SOS Multifunctional Pager: This device includes the button and its base station. In these experiments, since the button depended on the base station, the base station was powered on and off.
• Sense U Baby: This device includes an MQTT base station (publisher) and a sensor that collects data and sends it to the base station. The data collected during this experiment is that of the base station. The sensor was excluded since it does not take part in communicating with the cloud service and only repeatedly collects data and transmits it to the base station for publishing. This behavior is later analyzed in idle/active experiments.

Similarly, the same procedure was conducted for 10 applicable Bluetooth Low Energy (BLE) enabled devices wherein their power behaviours were captured when connecting to the smartphone.

4.4.2. Idle Experiments

The idle experiments are carried out in a span of two 13-hour captures. The captures were performed on two nights between the hours of 6:00 pm, and 7:00 am to ensure no interaction with devices would happen. Among our Wi-Fi devices, only the Singcall SOS Button did not produce any traffic. Another similar device, the Multifunctional Pager (SOS button paired with base station) had some form of external communication with services.

In these experiments, the Sense-U Baby Monitor was disconnected from its base station since the sensor does not have any idle state. The base station, however, continuously measured the temperature and humidity of the environment.

The MQTT broker was connected to this capture, and the iPad was subscribed to it. The iPad (subscriber) was running for the full 26 hours. In fact, no simulated MQTT traffic was published during these experiments. It was expected that the broker would remain idle during the capture, and the only traffic produced was when the subscriber pinged the broker at different times to ensure it was awake and when the broker made DNS queries.

4.4.3. Active Experiments

The experiments were done in batches to accumulate a total of 26 hours. The devices were left to interact with one another as people actively (e.g., triggering SOS buttons, wearing baby sleep sensors, interacting with apps) or passively (e.g., motion detection from cameras) interacted with the devices. MQTT simulation was also included in these captures. Finally, It was observed during these captures that the M1T Laxihub camera sends unencrypted image frames to its cloud.

4.4.4. Interaction Experiments

The interaction experiments were carried out by capturing the interaction with devices either physically or through their companion apps. 3 types of interactions were considered where applicable:

• Physical: carried out where devices could be interacted with using physical buttons. These experiments were combined with LAN and WAN experiments where applicable, i.e., the apps were either connected to the same network as the IoMT devices or connected to another network.
• LAN: These experiments were carried out by making use of the device’s companion apps, and interacting with them while being on the same network as the IoMT devices.
• WAN: These experiments were carried out by making use of the device’s companion apps, and interacting with them while being on another network as the IoMT devices.

5. Methodology

Once the experiments have been conducted and the network traffic has been stored in PCAP format, the data needs to be organized to simplify access. Figure 5 illustrates the process of storing network traffic in PCAP format, converting this traffic into CSV files, conducting Machine Learning (ML) evaluation, and reporting integrated results.

First, the raw network traffic is stored in PCAP files. These are the original traffic data that can be used in future research directions to secure IoMT operations. These files are separated into MQTT, Bluetooth (BLE), and Wi-FI. Finally, profiling experiments are also stored in PCAP format.

Second, we divide the collected traffic into two groups: (i) train and (ii) test. The “train" group is intended to be used to develop new solutions (e.g., ML training), while the “test" group is designed to evaluate such solutions in unseen data. This division relies on defining a group of PCAP files comprising 80% (“train") and 20% (“test") of all PCAP files available.

After that, a process of converting PCAP files to CSV files is conducted for each attack. In this case, we target Wi-Fi and MQTT attacks due to the nature of their features. To accomplish this, a set of supporting tools needs to be used and are illustrated in Figure 6. First, TCPDUMP [108] is used to split a big PCAP file into a group of smaller PCAP files. This is done to enable the parallel execution of the conversion script. Then, for each PCAP chunk, DPKT [109] is used in parallel to extract features and save them into CSV format. As the traffic generated is extensive, we average packets with varying window sizes of 10 (i.e., ping sweep, vulnerability scan, Bluetooth benign, Bluetooth DoS, OS scan, MQTT malformed data, ARP spoofing, port scan, and benign) and 100 (e.g., MQTT DoS connect flood, MQTT DDoS connect flood, MQTT DoS publish flood, MQTT DDoS publish flood, DoS TCP, DoS ICMP, DoS SYN, DoS UDP, DDoS SYN, DDoS TCP, DDoS ICMP, DDoS UDP) packets. Finally, these resulting CSV files are combined into a single CSV file using Pandas [110].

Furthermore, different ML techniques are evaluated to assess their performance in detecting and classifying these attacks. In this research, we evaluate the performance of five widely used ML techniques, namely Logistic Regression [111], Adaboost [112,113,114], Random Forest [115], and Deep Neural Network [116]. The evaluation is focussed on three tasks: Binary classification (i.e., Begnign and Attack), Categorical Classification (i.e., benign, spoofing, DDoS, DoS, recon, and MQTT), and multiclass classification (i.e., including all classes available).

Finally, the integrated results are presented using multiple metrics. Considering TP True Positives, TN True Negatives, FP False Positive, and FN False Negatives, the metrics used in this research are [117]:

• Accuracy: evaluates the classification models by calculating the proportion of correct predictions in a dataset using the following expression:

  $$Acc=\frac{TP+TN}{TP+TN+FP+FN}$$

  (1)
• Recall: ratio classes identified to the total number of occurrences of this particular class:

  $$Rec=\frac{TP}{TP+FN}$$

  (2)
• Precision: ratio of correctly classified labels to the total number of positive classifications:

  $$Pre=\frac{TP}{TP+FP}$$

  (3)
• F1-Score: geometric average of precision and recall:

  $$F1=2\times \frac{Pre\times Rec}{Pre+Rec}$$

  (4)

6. Feature Extraction and Data Description

The process of extracting features from PCAP files to CSV aims to simplify threat detection. Several features can be extracted and engineered to represent multiple traffic characteristics (e.g., protocols and temporal aspects) in this context. Table 5 lists the features extracted and used in the ML evaluation process. Although this is a comprehensive list that represents several aspects of the network traffic, this selection can be easily increased by recurring to the original PCAP files. Table 6 describes the data for the CICIoMT2024 dataset by presenting the mean, standard deviation (std), minimum (min), 25% percentile (25%), 50% percentile (50%), 75% percentile (75%), and maximum (max) values for each feature.

Nevertheless, Figure 7 and Figure 8 illustrates the data distribution for each class present in the dataset. In fact, the amount of data generated by DDoS and DoS attacks is much larger than in any other experiment. These graphs show that the dataset presents a much larger proportion of attacks than benign traffic. It results from the number of attacks executed against the proposed topology. The categorial classification graph shows that most of the traffic collected stems from MQTT and benign traffic. The multiclass classification graph shows that benign and MQTT connect floods represent the majority of the traffic generated, while some attacks present reduced traffic due to the challenges of large-scale data generation (e.g., ping sweep). Nevertheless, Figure 8 depicts the traffic generated by DoS and DDoS. These two classes present a substantial number of instances, and the majority of the specific attacks executed refer to UDP and ICMP floods. Table 7 presents the number of instances for each attack, presenting the total count (i.e., attack and benign), the categorical count (i.e., spoofing, recon, MQTT, DoS, and DDoS), and the individual count (i.e., considering all individual attacks).

7. Machine Learning (ML) Evaluation

The experiments conducted in this research evaluated the performance of four ML techniques: Logistic Regression (LR), Random Forest (RF), Adaboost (AD), and Deep Neural Network (DNN). Three classification problems were considered: Binary classification (i.e., benign and attack), categorical classification (i.e., benign, spoofing, recon, MQTT, DoS, and DDoS), and multiclass classification (i.e., including all classes available). Figure 9 illustrates the experiments’ results regarding accuracy, recall, precision, and F1-score. Also, Table 8 lists the numerical results obtained in each scenario.

The overall results show that all ML techniques performed well in several scenarios. However, this graph also shows that some techniques presented a decrease in performance in some cases. Considering the binary classification, all methods could distinguish benign and attacks successfully. In this scenario, the rate of mislabeled instances is low as accuracy, and the F1-score reaches values above 0.98. In the categorical classification scenario, there is a clear impact on the performance of some algorithms. While RF can maintain high performance, LR significantly reduces the resulting values. The most challenging scenario refers to the multiclass classification. In this 19-class challenge, the overall performance is decreased for all metrics as detecting internal patterns for each attack becomes more difficult. Conversely, RF and DNN could still present accuracy results of 0.76 and 0.99, respectively. In terms of the F1-score, RF presents 0.907.

To understand the results presented, the confusion matrix highlights the challenges a given model faces for each class. Since RF outperformed the other techniques, Table 9, Table 10, and Table 11 present the confusion matrix of RF for binary, categorical, and multiclass classification, respectively. Most instances for all classes are correctly classified for binary and categorical classification. However, distinguishing recon and spoofing attacks from benign traffic is challenging. Also, separating MQTT and TCP-IP DoS attacks is a complex task. Furthermore, the results obtained for multiclass are similar in terms of the issues faced. For example, there is a misclassification between MQTT DoS and DDoS attacks. Besides, internal recon classes are also difficult to distinguish in some cases due to the traffic similarity (e.g., OS, Port, and Vulnerability scan). Finally, although some mistakes can be found among classes of the same category (e.g., UDP DDoS and SYN DDoS), RF is capable of correctly classy instances in most cases.

8. Conclusion

This research focuses on developing a multi-protocol dataset for assessing IoMT device security named CICIoMT2024. To accomplish this, 18 different attacks were against an IoMT topology comprising 40 IoMT devices. Besides, three protocols were targeted considering the characteristics of healthcare operations (i.e., Wi-Fi, MQTT, and Bluetooth). The main goal is to contribute to the existing state-of-the-art by defining a complementary baseline that supports researchers in investigating and developing new solutions for cybersecurity in healthcare and IoMT operations.

A comprehensive explanation of how the dataset was collected, processed, and stored was conducted alongside evaluating different ML algorithms. Besides, we conducted a discussion on how each attack was conducted, as well as feature extraction and data description. The dataset is available in PCAP and CSV formats and comprises the network traffic collected throughout the experiments. The www.unb.ca/cic/datasets/iomt-dataset dataset has been published on CIC’s dataset page, making it available for other researchers to use.

Finally, there are several possible future directions for future work. For example, the hyperparameter optimization of ML models, the analysis and engineering of new features that can be extracted from the PCAP files, and the integration of this dataset with other healthcare resources (e.g., datasets and simulation platforms).