Enhancing Cybersecurity Incident Response: AI-Driven Optimization for Strengthened Advance Persistence Threat Detection

1. Introduction

The cyber incident is analogous to a catastrophe as it is unexpected, sudden, and crucial. If the cyber incident is not quickly alleviated, it causes severe damage. These damages often worsen when a cyber incident is unmanageable for a longer time. For example, the adversary can destroy or withdraw confidential data. Moreover, in a recently published report [1], 20 percent of the companies report a loss in revenue, business opportunities, and a decrease in the number of their customers due to security incidents.

The companies implement incident response procedures to identify, hold, and abolish cybersecurity incidents. The incident response consists of a plan of action preparation, incident detection, limiting the adversary, removal of the adversary, healing from the incident, and post-accident tasks [2]. The incident Response Frameworks i.e., SysAdmin, Audit, Network, and Security (SANS) [3], and National Institute of Standards and Technology (NIST) [4] help companies in the formation of systematic incident response procedures.

The Cyber Security Incident Response Team (CSIRT) [5] is trained to hold and respond to cybersecurity incidents. The CSIRT discovers threats, prepares methods, and trains end users to mitigate cybersecurity incidents [6]. The CIRT uses Security Information and Event Management (SIEM) [7] tools like Splunk [8], QRadar [9], to analyze the log in real-time. The SIEM tools collect security logs from different devices like servers, firewalls, etc. These tools generate alerts for CSIRT if a cybersecurity incident is found.

Due to the huge streams of alerts received per unit of time in Security Operation Centers (SOC), CSIRT teams find it difficult or impossible to handle threats and attacks on time. Unfortunately, many incidents and attacks are missed. Another problem is the large amounts of false positives that overload the system and negatively impact its efficacy. Moreover, manual intervention of experts is required in many stages in the last post-alert phase, which slows the decision process [10].

We proposed a novel framework to enhance security event identification and reduce false positive alerts. The proposed framework includes many core elements to solve the above-mentioned problems. Firstly, we proposed six event detection modules to identify more malicious behaviors using a specific set of Indicators of Compromises (IOCs). These modules can detect 14 different types of malicious behavior using different Splunk queries. Secondly, the Advance Persistent Threat (APT) alert dataset [11] is extended by adding additional alert types, giving the system a large number of APT indicators. Thirdly, Machine Learning (ML) techniques i.e., Random Forest and Extreme Gradient Boosting (XGBoost) are trained and tested using the extended APT dataset. The proposed model improved the APT detection accuracy up to 99.6% and reduced false positive alerts generated by Splunk. The main contributions of the research study are as follows.

• To detect critical malicious behavior, we extended the APT alert dataset with more alert types using Splunk.
• To reduce false positive APT alarms, we re-engineered existing processes and developed an advanced learning framework with an accuracy of 99.6%.
• To automate the incident response process by reducing manual intervention in the post-alert decision using ML techniques.

The rest of the paper is structured as follows. In Section II, we summarize preliminaries and related literature. In section III, we detailed the proposed framework. In Section IV, we present the implementation and evaluation of the result in detail. Similarly, the conclusion of this work is drawn in section V.

2. Preliminaries and Related Literature

The following subsections discuss SIEM, ML, and literature related to CyberSecurity Incident Response optimization Using ML techniques.

2.1. Overview of Security Information and Event Management

SIEM was introduced by Mark Nicolett and Amrit Williams in the technical report of Gardner in 2005 [12]. SIEM combines Security Information Management (SIM) and Security Event Management (SEM) citesecurity. The SIM is used to store, analyze, and report log events. Similarly, SEM is used to monitor and correlate real-time events. Therefore, SIEM can be defined as it is a tool that gathers, analyzes, and validates log data [13]. Moreover, SIEM uses either rule-based or statistical events correlation and provides the log data in human-readable format i.e., reports and alerts. The core functionalities of SIEM are discussed in the following subsection.

• Log Collection: Network Devices like switches, hosts, firewalls, etc, generate log data. There are two methods defined for log data collection i.e., agent-based and agent-less [14]. In the agent-based method, an intermediary agent collects log data and then forwards it to the next module. Similarly, in the Agent-less method, there is no intermediate agent and the server gets data directly from the servers.
• Log Aggregation: It is a procedure for gathering, analyzing, and capturing structured data from the log data generated by network devices. In the literature [14], two methods are used for log data collection i.e., push and pull. i) Push: The network devices like switches, firewalls, etc, send logs to the servers. ii) Pull: The servers drag the log data from the network devices.
• Parsing: It is a procedure for transforming the log data into structured data. In a network environment, multiple parsers are used to transform log data received from multiple devices like firewalls, server switches, etc.
• Normalization: It receives log data from multiple sources and converts it into one standard reduced format. It removes the redundant logs from the log data.
• Threat analysis: SIEM identifies adversaries by comparing log data with the records in a known threat database. During threat analysis, SIEM uses statistical methods to identify different threat patterns.
• Response: SIEM is a very efficient tool because it can detect malicious activity quickly before it can cause serious damage. It has built-in pre-defined real-time alerts and a notification system.
• Reporting: SIEM tools contain different custom reporting templates for data visualization. Moreover, the data collected by SIEM is available to data analysts and experts for investigation.

2.1.1. SIEM’s Architecture and Components

The layered architecture of SIEM is depicted in Figure 1. It comprises a SIEM Engine, log collection, log sources, and presentation layer [14]. The SIEM architecture collects logs generated by different sources like servers, firewalls, applications, etc. Then, the SIEM engine filters the duplicate events and correlates the filtered events to find malicious activities. Moreover, the analysis result is displayed in the form of a report or alert at the presentation layer. The SIEM architecture also stores the log data for future analysis. The SIEM provides functionalities like threat detection, log storage, incident response, and threat reporting.

Gartner’s SIEM Magic Quadrant annual report [15] has categorized SIEM solutions into niche, challengers, leaders, or visionary. Splunk is a well-known SIEM tool recognized as a leader by the Gartner report.

2.1.2. Splunk: Implementations of SIEM

It is used to process log data and present it in a human-readable format. It consists of two modules i.e., the processing module and management module [16]. A high-level Splunk architecture is present in Figure 2. Furthermore, the processing module consists of forwarders, Indexers, and search heads [17]. The forwarders gathered log data from the network devices and forwarded it to other Splunk processing components. Similarly, an indexer is used to index and store log data. Furthermore, the search head analyzes the log data and generates visual graphs and reports. Similarly, the deployment server, license master, Indexer cluster master node, search head cluster deployer, and monitoring console are the sub-components of the management module.

Splunk can be deployed in two ways i.e., single server deployment and distributed deployment [18]. The single server deployment is used either to test Splunk components or for learning purposes. Similarly, in a multi-server deployment, the Splunk components are installed on several servers in a distributed manner.

2.2. Introduction to Machine Learning

ML is an Artificial Intelligence (AI) application that enables a system to learn from its experiences. ML identifies patterns in a given data and generates predictions. ML has gained much attention due to its applications in diversified domains like Image processing, digital forensics, incident response, etc [19]. ML is also known as learning from data. Conventional softwares are based on deduction methods whereas ML is based on induction methods. Therefore, an ML algorithm automatically discovers rules by looking at many examples, and these practices can then be applied to other similar data. The following are the well-known ML models in the existing literature.

2.3. Random Forest

Random Forest [20] is a powerful ML classifier that combines the predictions of multiple decision trees. The working of random forest can be divided into the following three steps. 1) Random Forest begins by extracting random samples from the original data by using either sample with replacement or random feature selection method. This randomness in the training phase protects the model from overfitting. 2) Then, a decision tree is generated for each sample. 3) Finally, the Random Forest combines the predictions of each decision tree using either classification or regression. The Random Forest is well known because of its simplicity, usability, and potential to deal with large-size datasets.

2.4. XGBoost

XGBoost [21] is an improved version of the gradient boosting technique to reduce the loss function. It is an ensemble ML model that combines numerous decision trees to generate a strong prediction model. Unlike random forests, which build trees separately, boosting builds trees consecutively, with each new tree attempting to remedy the flaws committed by the preceding ones. Moreover, it uses L1 and L2 regularization techniques to avoid overfitting. XGBoost classifier is famous for its speed due to parallel processing, accuracy, and ability to work with large-size datasets.

2.5. Related Works

Cybersecurity has strongly accepted ML in various fields i.e., botnet identification [22], malware, and intrusion detection [23,24,25]. This research focuses on the application of ML to enhance and automate decision-making in the post-alert stage of SIEM systems. The related works are discussed in the following paragraphs.

In [11], the authors have developed an efficient ML-based model to detect APT attacks quickly and accurately. The model consists of three core phases i.e., APT attacks detection, correlation, and prediction. The model detected the APT attacks with a true positive rate of 81.8% and a false positive rate of 4.5%. However, the accuracy is still low and needs to be improved. Similarly, the authors, in [26], have proposed a framework to protect organization workstations and servers from APTs. Agents are used to monitor these computers. The agents perform two functions i.e., security violation identification and alert generation. The proposed TerminAPTor module correlates the alerts generated by agents using Information Flow Tracking (IFT). This IFT is used to detect APTs. However, the authors simulated only two APT scenarios which resulted in high false positive alerts. Similarly, the proposed context-based framework introduces a conceptual attack model called the attack pyramid [27]. It has defined sensitive data at the top level and the lower levels consist of a platform for recording events related to the APTs. However, sufficient knowledge is required to set up the framework.

The authors, in [28], discussed the deployment of ML to categorize and predict SOC alerts. The proposed solution aims to reduce human analyst intervention by automating the process of threat detection using an ML model. The proposed ML model depends on the response from five senior SOC experts. Each expert shares his experience in handling the alerts. They used a Random Forest Classifier while the dataset was generated using SIEM. Although the aim was to automate the decision-making concerning false positive alerts, the final decision was still taken by the experts. Similarly, the authors, in [29], have proposed an efficient ML-based threat detection model with minimized false positive alerts. To select standardized threat use cases, several MITRE’s taxonomies (ATT&CK, CAPEC, and MAEC) were examined. KMeans is used to analyze uploaddownload data traffic, because of its simplicity and fast grouping ability. Linear regression and Random Forest were used to identify malicious activities while SVM was used to identify malicious child and parent activities. Moreover, the authors have claimed that existing intrusion detection mechanisms are inefficient and proposed context-aware IDS by integrating ML and DRL models [30]. The robustness of the proposed model is enhanced by integrating the denoising autoencoder. The proposed model is trained and tested using NSL-KDD, UNSW-NB15, and AWID datasets. The authors claimed that the proposed model showed better accuracy and less FPR as compared to their counterparts. However, the proposed model is not tested in real-world scenarios. Moreover, a quick ML-based incident response approach is proposed in [31]. The authors generated their APT dataset by analyzing 2.5 million log events. The dataset consists of a thousand vectors, where each vector contains nine features labeled as either positive or negative. ZeroR, OneR, SVM, NaiveBayes, J48, and Random Forest classifiers were trained and tested. The Random Forest attained a high accuracy of 95.54% using the extracted APT dataset.

Moreover, in [32], the authors have proposed a spam and junk mail detection mechanism based on Bayesian spam filtering. They identified APT’s behavioral patterns and proposed a novel self-destructive method to protect individual or organizational data from APTs. Furthermore, several classification models i.e., random forest, KNN, XGBoost, etc., are used on four different datasets i.e.,e CSE-CIC-IDS2018, CIC-IDS2017, NSL-KDD, and UNSW-NB15, to detect APT at the early stage [33]. However, Random Forest and XGBoost were selected as the base classifiers. Then, the simple classifier aggregator is used to merge the predictions of the two classifiers. The proposed hybrid model was then trained and tested on the four different datasets. The summary of the existing related works is presented in Table 1.

Finally, the state-of-the-art APT detection mechanisms face several challenges i.e., high false positive rate, detection of APT in early stages, new APT scenarios, and behavior. To address those challenges, we proposed a novel mechanism for APT detection.

3. Overview of the Proposed Framework

The proposed architecture uses an advanced ML Technique for Cybersecurity Incident Response (CIR) optimization. The proposed architecture consists of an Event Logging, Event Detection Module, Alert Correlation Module, and Reporting as shown in Figure 3.

3.1. Notations

The notations used in this section are given in Table 2.

3.2. Event Logging

The following logs were collected from the hosts. Splunk collects log files from different sources i.e., Syslog, MySQL Logs, Apache Access Logs, DNS logs, FTP server Logs, and Authentication Logs. The components of an event are given in Eq. 1.

$$\begin{array}{c}\hfill {\mathcal{EVENT}}_j=\{S\_IP,D\_IP,S\_port,D\_port,timestamp,\\ \hfill type,infected\_host\}\end{array}$$

(1)

Formally, we defined a log file as a set of events i.e., $\mathcal{LOG}$ set records all the events generated in a system as shown in Eq. 2.

$$\mathcal{LOG}\supseteq {\cup }_{j=1}^{n}Even{t}_j$$

(2)

3.3. Event Detection Module

The event detection module is proposed to detect malicious behavior in a log of events. We use Splunk to collect log files from different sources like firewalls, routers, hosts, and servers. Splunk examines these logs against malicious behavior indicators to detect malicious behavior. If a malicious behavior is detected then Splunk generates an alert. We proposed six sub-modules for malicious behavior detection i.e., unusual network traffic, unusual system behavior, user behavior, file behavior, malicious IP, and malicious domain as shown in Figure 3. Each module generates an alert when a specific behavior is detected.

3.3.1. Unusual Network Traffic (UNT) Behavior Detection:

The Splunk query is used to detect Unusual Network Traffic as shown in Table 3. Unusual Network Traffic detection can be an indicator of several stages in the APT life cycle i.e., Command and Control, Exfiltration, and Maintenance. 1) Command and Control (C2): An unusual network traffic pattern can be produced when the adversary uses non-standard communication methods to reach the compromised host. 2) Exfiltration: Extracting sensitive data from the compromised host can increase network traffic. 3) Maintenance: The attacker performs additional activities to maintain communication with the compromised host, which can produce unusual network traffic.

We defined a Unusual Network Traffic Set ($\mathcal{UNTS}$) that contains all those events that cause unusual network traffic as shown in Eq. 3.

$$\begin{array}{c}\hfill \mathcal{UNTS}=\{{\mathcal{UNT}}_0\cup {\mathcal{UNT}}_1\cup {\mathcal{UNT}}_2\cup ......{\mathcal{UNT}}_n\}\\ \hfill ={\cup }_{i=1}^n{\mathcal{UNT}}_i\end{array}$$

(3)

3.3.2. Unusual System Behavior Detection:

Unusual system behavior like unexpected system crashes, error messages, and modifications to system configuration are the indicators of many stages in the APT life cycle. We used the following indicators to detect unusual system behavior of APT attacks. 1) Indicators at Exploitation phase: Malicious activities can result in unusual system behavior like unexpected system crashes and error messages. 2) Indicators at Installation phase: Adversaries may install malicious software like backdoors, rootkits, etc, which can result in a modification to system configurations or new process creation. 3) Indicators at the Data Exfiltration phase: At this phase, we use increased network traffic as indicators of compromise. We defined $\mathcal{USB}$ as a superset of all the events of unusual system behavior as shown in Eq. 4.

$$\mathcal{USB}=\{{\mathcal{USB}}_0\cup {\mathcal{USB}}_1\cup {\mathcal{USB}}_2\cup ......{\mathcal{USB}}_n\}={\cup }_{j=1}^n{\mathcal{USB}}_j$$

(4)

3.3.3. Unusual User Behavior Detection:

It recognized unusual user behavior from the logs collected by Splunk using the following indicators. 1) Indicators at Reconnaissance Phase: Unusual network scans. 2) Indicators at Initial Compromise Phase: At this phase, the following indicators are used i.e., Unusual pattern of failed login attempts, Failed login attempts from the same IP address into many accounts, and successfully logging into an account followed by numerous failed login attempts. 3) Indicators at Lateral Movement Phase: Unusual activity from privileged accounts, change in user privileges, and access to sensitive data. A sample Splunk query is given in Table 4 for detecting failed attempts before successful login. We defined $\mathcal{UUB}$ as a superset of all the events of unusual user behavior as shown in Eq. 5.

$$\begin{array}{c}\hfill \mathcal{UUB}=\{({\mathcal{SL}}_0\cup {\mathcal{SL}}_1\cup {\mathcal{SL}}_2\cup ......{\mathcal{SL}}_n)\\ \hfill \cup ({\mathcal{SBA}}_0\cup {\mathcal{SBA}}_1\cup {\mathcal{SBA}}_2\cup ......{\mathcal{SBA}}_n)\}\\ \hfill \cup ({\mathcal{CUP}}_0\cup {\mathcal{CUP}}_1\cup {\mathcal{CUP}}_2\cup ......{\mathcal{CUP}}_n)\}\\ \hfill \cup ({\mathcal{NS}}_0\cup {\mathcal{NS}}_1\cup {\mathcal{NS}}_2\cup ......{\mathcal{NS}}_n)\}\\ \hfill =\{\left({\cup }_{i=1}^n{\mathcal{SL}}_i\right)\cup \left({\cup }_{j=1}^n{\mathcal{SBA}}_j\right)\cup \left({\cup }_{k=1}^n{\mathcal{CUP}}_k\right)\cup \left({\cup }_{l=1}^n{\mathcal{NS}}_l\right)\}\end{array}$$

(5)

3.3.4. File Behavior Detection:

We used the following indicators of file behavior to identify an APT. 1) File execution: APTs often use malicious files that execute upon being opened. 2) File creation and deletion: APTs may create or delete new unexpected files. 3) File encryption: APTs may use encryption to keep their task undetected. We monitored system logs for unexpected file encryption to detect APT. 4) File transfer: We monitored system logs for unexpected file transfers between network hosts. A sample Splunk query is given in Table 5 for the detection of malicious file creation and deletion by a particular host. We defined Unusual File Behavior Set $\mathcal{UFBS}$ as a superset of all the events generated in response to unusual File behavior i.e., malicious domain, unusual file transfer, malicious hash as given in Eq. 6.

$$\begin{array}{c}\hfill \mathcal{UFB}=\{({\mathcal{MD}}_0\cup {\mathcal{MD}}_1\cup {\mathcal{MD}}_2\cup ......{\mathcal{MD}}_n)\\ \hfill \cup ({\mathcal{UFT}}_0\cup {\mathcal{UFT}}_1\cup {\mathcal{UFT}}_2\cup ......{\mathcal{UFT}}_n)\}\\ \hfill \cup ({\mathcal{MH}}_0\cup {\mathcal{MH}}_1\cup {\mathcal{MH}}_2\cup ......{\mathcal{MH}}_n)\}\\ \hfill =\{\left({\cup }_{j=1}^n{\mathcal{MD}}_i\right)\cup \left({\cup }_{j=1}^n{\mathcal{UFT}}_j\right)\cup \left({\cup }_{k=1}^n{\mathcal{MH}}_k\right)\}\end{array}$$

(6)

3.3.5. Malicious IP Detection:

This module uses the following indicators for malicious IP addresses. 1) Known malicious IP address. 2) Unusual increase in network traffic from an IP. 3) Communication with C2 server. We defined Malicious IP Set $\mathcal{MIPS}$ that contains blacklisted IP addresses as shown in Eq. 7.

$$\begin{array}{c}\hfill \mathcal{MIP}=\{{\mathcal{MIP}}_0\cup {\mathcal{MIP}}_1\cup {\mathcal{MIP}}_2\cup ......{\mathcal{MIP}}_n\}\\ \hfill ={\cup }_{j=1}^n{\mathcal{MIP}}_j\end{array}$$

(7)

3.3.6. Malicious Domain Detection:

The proposed framework used the following indicators to detect malicious Domains. 1) Similar domains that were used in previous APT attacks. 2) Only TOR-accessible domains. 3) malicious DNS.4) Newly registered domains. 5) Domains having malicious SSL certificates. We defined Malicious Domain Set($\mathcal{MD}$) as a superset of all the events generated in response to malicious domain behavior i.e., malicious domain, malicious DNS, malicious SSL Certificate as shown in Eq. 8.

$$\begin{array}{c}\hfill \mathcal{MD}=\{({\mathcal{MD}}_0\cup {\mathcal{MD}}_1\cup {\mathcal{MD}}_2\cup ......{\mathcal{MD}}_n)\\ \hfill \cup ({\mathcal{MDNS}}_0\cup {\mathcal{MDNS}}_1\cup {\mathcal{MDNS}}_2\cup ......{\mathcal{MDNS}}_n)\}\\ \hfill \cup ({\mathcal{MSSL}}_0\cup {\mathcal{MSSL}}_1\cup {\mathcal{MSSL}}_2\cup ......{\mathcal{MSSL}}_n)\}\\ \hfill =\{\left({\cup }_{i=1}^n{\mathcal{MD}}_i\right)\cup \left({\cup }_{j=1}^n{\mathcal{MDNS}}_j\right)\cup \left({\cup }_{k=1}^n{\mathcal{MSSL}}_k\right)\}\end{array}$$

(8)

3.3.7. Alert ($\mathcal{ALERT}$)

In the proposed architecture $\mathcal{ALERT}$ consists of the following attributes as shown in Eq. 9. However, Eq. 10 shows the different types of alerts generated by the proposed system.

$$\begin{array}{c}\hfill \mathcal{ALERT}=\{S\_IP,D\_IP,S\_port,D\_port,timestamp,\\ \hfill activity,alter\_type,C\_hostandstage\}\end{array}$$

(9)

$$\begin{array}{c}\hfill alert.type=\{aler{t}_{MD}\vee aler{t}_{ME}\vee aler{t}_{MH}\vee aler{t}_{MDNS}\\ \hfill \vee aler{t}_{SL}\vee aler{t}_{IP}\vee aler{t}_{SSL}\vee aler{t}_{NS}\vee aler{t}_{SDA}\\ \hfill \vee aler{t}_{CUP}\vee aler{t}_{UNT}\vee aler{t}_{USB}\vee aler{t}_{tor}\vee aler{t}_{UFB}\}\end{array}$$

(10)

3.3.8. Alert Set ($\mathcal{AS}$)

$\mathcal{AS}$ contains all the alerts generated by the system in response to unusual behaviors.

$$\mathcal{AS}\supseteq {\cup }_{j=1}^{\infty }{\mathcal{ALERT}}_j$$

(11)

3.3.9. APT Life Cycle Graph ($\mathcal{APTLCG}$)

$\mathcal{APTLG}$ is a superset of all the APT attacks sequences. It is a directed graph where nodes represent alerts and the edges between nodes represent the sequence of alerts i,e., $({\mathcal{ALERT}}_i$, Sequence of ${\mathcal{ALERT}}_i)$.

$$\mathcal{APTLCG}=\{{\mathcal{ALERT}}_i,{\mathcal{ALERT}}_j,{\mathcal{ALERT}}_k,....\}$$

(12)

3.3.10. APT Attack Sequence ($\mathcal{APTAS}$)

$\mathcal{APTAS}$ It contains all sequences of APT attack alerts.

$$\begin{array}{c}\hfill \mathcal{APTLAS}=\{{\mathcal{APTLAS}}_1\cup {\mathcal{APTLAS}}_2\cup ......{\mathcal{APTLAS}}_n\}\\ \hfill ={\cup }_{i=1}^n{\mathcal{APTLAS}}_i\end{array}$$

(13)

3.3.11. APT Incomplete Attack Sequence (${\mathcal{APTIAS}}_i$)

It contains an incomplete sequence of APT attacks on host i.

$$\begin{array}{c}\hfill {\mathcal{APTIAS}}_i\subset \{(alert.stage==S1)\\ \hfill \cup (alert.stage==S2)\cup (alert.stage==S3)\\ \hfill \cup (alert.stage==S4)\}\end{array}$$

(14)

3.3.12. APT Incomplete Attack Sequence ${\mathcal{APTCAS}}_i$

It contains a complete sequence of APT attacks on host i.

$$\begin{array}{c}\hfill {\mathcal{APTCAS}}_i=\{(alert.stage==S1)\\ \hfill \cap (alert.stage==S2)\cap (alert.stage==S3)\\ \hfill \cap (alert.stage==S4)\}\end{array}$$

(15)

3.3.13. Event Detection Function

The Event_Detection () takes logs of events as input and detects malicious events based on the type of event stored in the log file as shown in Eq. 16. The different types of events are discussed in Table 6.

3.3.14. Implementation of Malicious Behavior Detection Algorithm

The proposed architecture is designed to detect malicious behavior in a system using input logs as shown in Algorithm 1. The Algorithm 1 takes a set of logs as input which consists of one or more events. As an output, the Algorithm 1 generates an alert or error message. We used an Event_Detection() function to extract the type of each event from the input logs. The Algorithm 1 examines the type of each event and divides it based on specific types of malicious behavior. The Algorithm 1 generates an alert if the event type matches the malicious behavior. The details of the malicious behaviors and alerts are given in Table 6. Then, it adds the generated alert to the $\mathcal{APTLCG}$ Graph. However, if the event type does not match any of the malicious behaviors then an error message is generated which shows that malicious behavior is not found.

Algorithm 1 Malicious Behavior Detection

1: Input: $\mathcal{LOG}\supseteq {\cup }_{j=1}^{\infty }{\mathcal{EVENT}}_i$  2: Output: Alert or Error  3: Event_Detection(): $\mathcal{LOG}$–> ${\mathcal{EVENT}}_i.type$  4: if {${\mathcal{EVENT}}_i.type$ = ($\mathcal{UNT}$ or $\mathcal{TOR}$)} then  5:     $\mathcal{UNTS}$ = $\mathcal{UNTS}$∪${\mathcal{EVENT}}_i$  6:     Alert():${\mathcal{EVENT}}_i$–> {$aler{t}_{UNT}$ or $aler{t}_{TOR}$}  7:     $\mathcal{AS}$ = {$\mathcal{AS}$∪ {$aler{t}_{UNT}$ or $aler{t}_{TOR}$}  8:     $\mathcal{APTLCG}$ = $\mathcal{APTLCG}$∪ {$aler{t}_{UNT}$ or $aler{t}_{TOR}$}  9: else if {${\mathcal{EVENT}}_i.type$ = ($\mathcal{MD}$ or $\mathcal{MDNS}$ or $\mathcal{MSSL}$)} then 10:     $\mathcal{MDS}$ = $\mathcal{MDS}$∪${\mathcal{EVENT}}_i$ 11:     Alert():${\mathcal{EVENT}}_i$–> {$aler{t}_{MD}$ or $aler{t}_{MDNS}$ or $aler{t}_{MSSL}$} 12:     $\mathcal{AS}$ = $\mathcal{AS}$∪ {$aler{t}_{MD}$ or $aler{t}_{MDNS}$ or $aler{t}_{MSSL}$ } 13:     $\mathcal{APTLCG}$ = $\mathcal{APTLCG}$∪ {$aler{t}_{MD}$ or $aler{t}_{MDNS}$ or $aler{t}_{MSSL}$} 14: else if {${\mathcal{EVENT}}_i.type$ = ($\mathcal{ME}$ or $\mathcal{UFT}$ or $\mathcal{MH}$)} then 15:     $\mathcal{UFBS}$ = $\mathcal{UFBS}$∪${\mathcal{EVENT}}_i$ 16:     Alert():${\mathcal{EVENT}}_i$–> {$aler{t}_{ME}$ or $aler{t}_{UFT}$ or $aler{t}_{MH}$} 17:     $\mathcal{AS}$ = $\mathcal{AS}$∪ {$aler{t}_{ME}$ or $aler{t}_{UFT}$ or $aler{t}_{MH}$ } 18:     $\mathcal{APTLCG}$ = $\mathcal{APTLCG}$∪ {$aler{t}_{ME}$ or $aler{t}_{UFT}$ or $aler{t}_{MH}$ } 19: else if {${\mathcal{EVENT}}_i.type$ = $\mathcal{MIP}$} then 20:     $\mathcal{MIPS}$ = $\mathcal{MIPS}$∪${\mathcal{EVENT}}_i$ 21:     Alert():${\mathcal{EVENT}}_i$–> $aler{t}_{MIP}$ 22:     $\mathcal{AS}$ = $\mathcal{AS}$∪$aler{t}_{MIP}$ 23:     $\mathcal{APTLCG}$ = $\mathcal{APTLCG}$∪ {$aler{t}_{MIP}$ } 24: else if {${\mathcal{EVENT}}_i.type$ = ($\mathcal{SL}$ or $\mathcal{SDA}$ or $\mathcal{CUP}$ or $\mathcal{NS}$)} then 25:     $\mathcal{USBS}$ = $\mathcal{UFBS}$∪${\mathcal{EVENT}}_i$ 26:     Alert():${\mathcal{EVENT}}_i$–> {$aler{t}_{SL}$ or $aler{t}_{SDA}$ or $aler{t}_{CUP}$ or $aler{t}_{NS}$} 27:     $\mathcal{AS}$ = $\mathcal{AS}$∪ {$aler{t}_{SL}$ or $aler{t}_{SDA}$ or $aler{t}_{CUP}$ or $aler{t}_{NS}$} 28:     $\mathcal{APTLCG}$ = $\mathcal{APTLCG}$∪ {$aler{t}_{SL}$ or $aler{t}_{SDA}$ or $aler{t}_{CUP}$ or $aler{t}_{NS}$} 29: else if {${\mathcal{EVENT}}_i.type$ = $\mathcal{USB}$} then 30:     $\mathcal{USBS}$ = $\mathcal{USBS}$∪${\mathcal{EVENT}}_i$ 31:     Alert():${\mathcal{EVENT}}_i$–> $aler{t}_{USB}$ 32:     $\mathcal{AS}$ = $\mathcal{AS}$∪$aler{t}_{USB}$ 33:     $\mathcal{APTLCG}$ = $\mathcal{APTLCG}$∪ {$aler{t}_{USB}$ } 34: else 35:     Error (No known malicious behavior found) 36: end if

3.4. Alert Correlation Module

The alert correlation module is proposed to correlate the alerts generated by different modules and predict the exact attack type. This module accepts the APT Life Cycle Graph as input. Then this module uses supervised ML techniques like Random Forest and XGBoost, to correlate and predict the exact attack type alert. It reduces the false positive alerts and filters the repeated alerts generated by the proposed six modules. The main functionalities of the alert correlation module are depicted in Figure 4.

The IOCs for different kinds of attacks can often overlap because adversaries frequently use identical tactics and techniques. For example, several attacks may use IOCs like malicious URLs, IP addresses, file signatures, and malicious behavior patterns. To effectively identify and respond to attacks, the proposed framework using AI techniques detects the exact attack IOCs associated with each attack. The APT attack detectable stages and alerts are given in Table 6.

3.4.1. Alert Function

The Alert () takes events as input and generates the corresponding alert based on the event type as shown in Eq. 17. The Alert () compares the event.type with pre-defined malicious behavior. If a match is found then it generates the corresponding alert. The different types of alerts are discussed in Table 6.

3.4.2. Alert Correlation Function

The Alert_correlation() takes alerts store $\mathcal{APTAS}$ as input and add its to ${\mathcal{APTIAS}}_i$ or ${\mathcal{APTCAS}}_i$ set. In other words, it correlates the alerts and classifies them into complete APT Attacks and incomplete APT Attacks. The alert correlation function is shown in Eq. 18.

$$\begin{array}{c}\hfill Correlation\left(\mathcal{APTAS}\right)=\left\{\begin{array}{c}\left|\right|{\mathcal{APTIAS}}_i={\mathcal{APTIAS}}_i\hfill \\ \cup aler{t}_i,\hfill \\ if\left\{\right({\mathcal{APTIAS}}_i\subset (\varnothing \cup S1\hfill \\ \cup S2\cup S3\cup S4\left)\right)\}\hfill \\ \left|\right|{\mathcal{APTCAS}}_i={\mathcal{APTCAS}}_i\hfill \\ \cup aler{t}_i,\hfill \\ if\left\{\right({\mathcal{APTCAS}}_i==(S1\hfill \\ \cap S2\cap S3\cap S4\left)\right)\}\hfill \end{array}\right\}\end{array}$$

(17)

(16)

3.4.3. Implementation of Alerts Correlation Algorithm

The proposed architecture correlates alerts and classifies them into complete APT Attacks and incomplete APT Attacks as shown in Algorithm 2. The Algorithm 2 takes $\mathcal{APTLC}$ as input and adds it to either the complete APT alert set or the incomplete APT alert set i.e., ${\mathcal{APTCAS}}_i$, ${\mathcal{APTIAS}}_i$. In step 3, the Algorithm 2 checks if the APT attack sequence i.e., $\mathcal{APTAS}$, for host i is empty. In step 4, if empty, it adds the alert from $\mathcal{APTLC}$ to $\mathcal{APTAS}$ of host i. In step 6, if $\mathcal{APTAS}$ is not empty, the Algorithm 2 calls a loop that iterates through the remaining alerts in $\mathcal{APTLC}$. In step 7, the Algorithm 2 executes a correlation check between the current alert (${\mathcal{ALERT}}_i$) and all alerts already in $\mathcal{APTAS}$. The correlation check examines three parameters i.e., stage, source IP address, and timestamp. Firstly, it inspects whether the current alert’s stage is greater than or equal to the stage of any existing alert in $\mathcal{APTAS}$. It shows a potential development in the attack. Secondly, it inspects whether the current alert’s source IP address matches any existing alert in $\mathcal{APTAS}$. It indicates a potential connection between the events. Lastly, it inspects whether the current alert’s timestamp is greater than or equal to any existing alert in $\mathcal{APTAS}$. It aids in maintaining chronological order within the APT sequence. In step 8, If the correlation check passes, the current alert is added to $\mathcal{APTAS}$. Moreover, it invokes the $\mathit{A}ler{t}_{C}orrelation\left(\right)$, which returns either an incomplete APT attack sequence i.e., ${\mathcal{APTIAS}}_i$ or a complete APT attack sequence i.e., ${\mathcal{APTCAS}}_i$. In step 9, if the correlation check fails, the Algorithm 2 returns "similar alert already exists" and starts the execution of a new alert in the loop.

Algorithm 2 Alerts correlation Algorithm

1: Input: $\mathcal{APTLC}$ 2: Output: $complete\_APT\_alert$ or $uncomplete\_APT\_alert$ 3: if {$\mathcal{APTAS}$ == ∅} then 4:     $\mathcal{APTAS}$ = $\mathcal{APTAS}$∪${\mathcal{ALERT}}_i$ 5: else 6:     while {Length($\mathcal{APTLC}$)} do 7:   if {{${\mathcal{ALERT}}_i$.stage ≥${\mathcal{ALERT}}_j$.stage} ∧{${\mathcal{ALERT}}_i$.S_IP == ${\mathcal{ALERT}}_j$.S_IP} ∧ {${\mathcal{ALERT}}_i$.timestamp ≥${\mathcal{ALERT}}_j$.timestamp}} then 8:          $\mathcal{APTAS}$ = $\mathcal{APTAS}$∪${\mathcal{ALERT}}_i$ 9:            $Correlation\left(\right):\mathcal{APTAS}$ –> {${\mathcal{APTIAS}}_i$ or ${\mathcal{APTCAS}}_i$} 10:       else 11:          Similar alert already added 12:       end if 13:     end while 14: end if

3.4.4. APT Reporting

APT offers substantial issues due to its stealth and long-lasting nature. Early diagnosis and rapid intervention are critical for minimizing damage. This module reports two types of APT alerts i.e., complete APT attack alert and incomplete APT attack alert. This reporting allows the incident and response team to make swift and appropriate decisions based on the information provided by the proposed system.

4. Experimental Setup and Results Evaluation

The experimental setup and obtained results are discussed in the following sections.

4.1. System Setup

We used the VMWare ESXi physical server to install the virtual machines with different operating systems and services. We installed vulnerable web applications i.e., Damn Vulnerable Web Application (DVWA), X-operating system, Apache, Mysql, Php, Perl (XAMPP) Server on the first VM. Similarly, the Domain Name System (DNS) Server and File server are installed on the other two VMs. The adversary attacks the servers from the Public Network. Each server has a log-capturing capacity. The log server is used for filtering and storing logs. Moreover, the networks and server logs are periodically transferred to the Splunk server as shown in Figure 5.

4.2. Dataset Collection

We extended the dataset used in [11] by adding more alert types as shown in Table 6. Our dataset contains the source IP address, destination IP address, source port number, destination port number, timestamps, alert types, infected hosts, and attack steps. This dataset is used as the basis for ML models to detect and investigate complete and incomplete APT attacks.

4.3. Data Pre-Processing and Encoding

Initially, our dataset had 3886 records with 9 attributes as discussed in section B. After careful analysis of the attributes, we discarded the alter_id attribute because it carries no information for the ML algorithm. Furthermore, we deleted the source IP attribute as it duplicates the IPs of the infected host attribute.

In the next step, we annotated the dataset into two classes i.e., ’complete APT attack’ and ’incomplete APT attack’. We identified 1031 incomplete APT attacks and 337 complete APT attacks as shown in Figure 6.

In the next step, we performed data encoding. Initially, we converted infected host IPs and destination IPs into their respective IP classes i.e., A, B, and C. In our dataset, alert type, step, destination IP and infected host IP are categorical attributes. We used a one-hot encoding technique to transform these attributes. There are 13 distinct values for alert type, therefore, after one hot encoding there are 13 columns each representing a different alert type. Similarly, there are 3 columns for each class of IP address. The aggregation criteria for the alert type, destination IP address, and infected host IP address attributes are set to sum. Moreover, the time stamp in our data is a continuous attribute. Therefore, we used ’mean’ as the aggregation criteria for the time stamp. The destination port and source port are nominal attributes, therefore we set ’median’ as the aggregation criteria. After data pre-processing and encoding we get 1368 records (rows) and 24 columns (attributes).

We selected the Random Forest and XGBoost classifiers because the majority of the attributes in our dataset are categorical and also, we used one-hot-encoding. Random Forest is a powerful machine-learning homogeneous ensemble classifier that is composed of multiple decision trees. Each tree is built using random sub-samples of the training data. The final decision is made using a majority vote criteria. Random Forest is useful because it reduces overfitting, and increases the robustness of the model. An overfitted model poorly performs on unseen data or cannot generalize. unlike random forest, XGBoost implements gradient boosting where each new tree is trained to work on the errors produced by preceding trees. Moreover, XGBoost uses a regularization mechanism in tree formation, that prevents the model from overfitting.

4.4. Results and Discussion

The training dataset is split into training and testing datasets with a famous 20:80 ratio. We performed out-of-sample experiments also called hold-out testing or external validation. The testing data is unseen for the model to provide a realistic assessment of the model. As mentioned earlier we have used Random Forest and XGBoost for the prediction task. The Random Forest and XGBoost classifier’s prediction accuracy, True Positive Rate (TPR), True Negative Rate (TNR), False Negative Rate (FNR), and False Positive Rate (FPR) are given in Table 7.

The confusion matrix and Receiver Operating Characteristic curve (ROC curve) for the Random Forest classifier are shown in Figure 7 and Figure 8 respectively.

Similarly, the confusion matrix and Receiver Operating Characteristic curve (ROC curve) for the XGBoost classifier are shown in Figure 9 and Figure 10 respectively.

We computed the importance of each feature/attribute in the prediction task as shown in Figure 11. We used all the features in our prediction step, however, less number of features can give nearly the same results. Thus, making the predictor suitable for embedded systems or the Internet of Things.

We can conclude that if the data is properly pre-processed and encoded, ML techniques can greatly enhance the performance of any SEIM tool.

4.5. Comparisons among Proposed and State-of-the-Art Framework

Our model has outperformed the state-of-the-art techniques in terms of multiple performance metrics as shown in Table 8.

5. Conclusion

This research paper covers the issues that the CSIRT faces while dealing with enormous amounts of alerts and time-consuming manual operations in the SOC. This study proposes a novel architecture to improve malicious behavior detection and reduce false positive alarms. We implemented six different malicious behavior detection modules. These modules can detect 14 malicious behaviors. These malicious behaviors are reflected in the APT dataset by adding the new alert types. Then, ML approaches, such as Random Forest and XGBoost, are trained and tested on the extended APT dataset. The proposed Random Forest-based model detects malicious behaviors with an accuracy of 98.54% and FPR of 1.46%. However, the XGBoost-based model generation detection accuracy of 99.6% and FPR of 0.0%.

The dataset was generated in a controlled environment, therefore only four stages of the APT life cycle are considered. In the future, we will work to detect APT at a more fine-grained level by incorporating more APT steps in real-world scenarios. Moreover, we will work to identify more malicious behaviors by adding more detection modules.