Detecting Malware C&C Communication Traffic Using Artificial Intelligence Techniques

1. Introduction

Cybercrime poses a serious danger to cybersecurity [1], and according to [2], the cost of cybercrime reached 8 trillion dollars in 2023, with malware such as banking malware accounting for a large proportion of this total cost. In recent years, banking malware has emerged as a major concern because malicious actors can make huge profits from these types of malware variants [3], and the cost to businesses is high. For instance, Emotet banking malware infections can cost up to 1 million dollars per incident to remediate [4]. Banking malware attacks also continue to rise [5], and the discovery of new banking malware variants also continues to increase for example, over a thousand variants of the banking malware Godfather were discovered in 2023 alone [6].

The Zeus banking malware has emerged as one of the most notorious banking malware variants ever developed [7] and since the release of the Zeus source code, many additional variants of Zeus have been developed and emerged and these include Ramnit, ZeusPanda and Ramnit [8].

1.1. Need for Detecting Banking Malware

New malware variants are always being discovered and are becoming more sophisticated in the way they attack systems [9], and they will continue to increase [10]. Banking malware follows the same trajectory, and, in this category, Zeus and its variants are still the most prevalent. For example, Figure 1 shows that Zeus and Ramnit were amongst the top ten banking malware variants discovered in Q3 of 2022 [11]. Figure 2 depicts the number of banking malware attacks that were detected during the same time and an upward trend is clearly visible [11].

Banking malware have also diversified their tactics and expanded their capabilities and have, overtime, evolved into more sophisticated software tools that leverage several attack vectors to cause financial loss. The fact that threat actors can use Malware-as-a-service (MaaS) providers to target victims has also led to their increased prevalence and some of these MaaS providers can charge up to $4000 per month to discharge their services [12].

Many strategies exist to detect banking malware, and these include signature-based, anomaly based, behavior based and heuristic approaches [13], but these do have limitations [14]. These approaches and limitations are explored further over the next few sections however, some of these limitations include:

• The need to update signature-based malware systems.
• The inability of these systems to detect newer malware variants.
• Unable to detect malware that uses sophisticated obfuscation techniques.
• The inability to detect zero-day malware.

This research paper is broken down into the following sections. Firstly, a review is conducted of the Zeus, Zeus Panda and Ramnit banking malware variants and then a banking malware family tree is proposed. The purpose of this review is to understand the similarities and differences between the various banking malware variants. Next, a literature review of related research is provided followed by a problem statement. Finally, this paper proposes a machine learning approach for detecting banking malware and also, proposes a feature section approach that supports the proposed machine learning approach.

The main contributions of this paper are to develop a methodology and approach to detect several banking malware variants’ Command and Control communication network traffic (C&C) and distinguish them from benign network traffic using binary classification Machine Learning (ML) algorithms. Three ML algorithms and an ensemble approach are all examined, analyzed and compared in this paper and these include the Decision Tree (DT), Random Forest (RF) and K-Nearest Neighbors KNN ML algorithms. An ensemble approach is also examined and this uses and combines all three algorithms.

This paper will compare the results of the binary ML classification algorithms to determine which one provides the best detection results when used to detect the Zeus banking malware C&C network traffic and other variants of the Zeus banking malware. The minimal number of features that could be utilized to identify the same banking malware variants is also determined. This paper aims to:

• From all the ML algorithms being analyzed, identify which one performs the best.
• Establish whether the features used to detect the Zeus banking malware can also be used to detect the other banking malware variants.
• Determine a minimum set of features that could be used for detecting Zeus.
• Determine a minimum set of features that could be used for detecting the other variants of the Zeus malware.
• Compare the performance results of all the ML algorithms.

1.2. Overview of the Zeus Banking Malware

There are several traits that the Zeus banking malware has been seen to portray and these include [15]:

Zeus malware is known to propagate like a virus and is normally spread using spam emails that contain a trojan horse with the malware embedded within the trojan.

Zeus normally attacks Windows systems and targets data such as usernames and passwords and other important information such as banking information.

Once the Zeus malware gets onto the device, it needs to perform several actions to infect the device. The Zeus bot inserts malicious code into the winlogin.exe process after copying itself to the system 32 directory and this is achieved by escalating its privileges and manipulating the Winlogin.exe and svchost processes. Two files are created, local.ds, which is used to download the configuration file and user.ds, which is used to transmit stolen data back to the threat actors’ C&C servers. The additional code injected into the svchost process is used by the Zeus bot for communication purposes and Zeus communicates using a Command and Control (C&C) channel which can either use a centralised or P2P architecture. In the centralised architecture, the IP address of the C&C server is hardcoded into the Bot’s binary file which leaves the bot vulnerable because if the C&C channel is discovered and blocked, the Zeus bot becomes inactive and is unable to recover [16]. Modern day variants of the Zeus malware use a P2P architecture as this is more resilient to disconnections and are much harder to detect and block [17]. One reason is simply because the IP address is not hardcoded into the bot binary and because in the P2P network, multiple bots can act as C&C servers. This architecture also allows stolen data to be routed through the bot network via these intermediary C&C bots and crucially, allows bots to recover from failures [18]. This recovery is possible because each peer C&C bot can essentially provide support to a failed bot for example, by sending the failed bot an updated IP address to help it resume malicious communications.

1.3. Overview of the Zeus Panda Banking Malware

The Zeus Panda banking malware portrays similar characteristics to the Zeus banking malware and research shows that it infects devices using spam emails and exploit kits and it has been known to spread like a virus [15]. The Zeus Panda’s communication architecture is similar to the Zeus banking malware architecture however, its communication is generally encrypted using RC4 or AES [15]. The Zeus Panda authors have also enhanced the code to allow it to detect and evade security protection tools such as anti-virus software and firewalls [19]. Zeus Panda is intelligent enough to detect that it is running in a virtual environment and upon sensing such an environment, it can disable itself to ensure researchers are unable to detect communication patterns [19]. The Zeus Panda malware is difficult to detect and can persist on a device for a long time and researchers have concluded that the Zeus Panda is a sophisticated variant of the Zeus malware [20].

1.4. Overview of the Ramnit Banking Malware

Ramnit is an enhanced version of the Zeus malware [21] and spreads between devices just like a virus and it incorporates code from the Zeus banking malware. It is delivered to a device via spam emails or via exploit kits and once the C&C communication is established, the C&C communication traffic is encrypted using custom encryption techniques [22]. Ramnit can also use HTTPS to obfuscate the communication channel and hide any data that is transmitted between systems. Ramnit is sophisticated enough to detect and evade security tools and once it infects a device, it can persist on the device for a long time [22]. Ramnit can also use evasion techniques to avoid detection and some of these include the use of anti-debugging techniques, code injection and process handling, polymorphic capabilities, encryption and environmental awareness [23].

1.5. Proposed Banking Malware Tree

This section examines and discusses the relationship between the three banking malware variants that were discussed above and establishes a timescale of when they emerged. Figure 4 shows that Zeus was discovered in 2006 [24], Ramnit was discovered in 2010, and Zeus Panda was discovered in 2016 [25] and research indicates that they all share similar code and perform similar actions. Based on this research [24,25,26,27,28,29,30], this paper proposes that all banking malware variants belong to a specific family of banking malware, and this proposed family tree can be seen in Figure 3. A historical timeline of a selection of banking malware variants can be seen in Figure 4 which suggests that all banking malware variants can be traced back to one of the parent banking malware variants, Zeus, Snifula and Gozi, as shown in Figure 3.

The key conclusions drawn from this research are that:

• Most of the banking malware variants belong to one of the three families identified in Figure 3.
• All the banking malware variants have borrowed code from each other.
• Most new banking malware variants still share code and perform similar actions to those shown in Figure 3.
• Banking malware variants continue to evolve and are becoming more effective at targeting victims.

The rest of the paper is organized as follows: Section 2 discusses some of the key research that has been conducted in this field. In section 3, a problem statement is presented. Section 4 proposes a framework to detect the banking malware variants discussed in sections 1.2 to 1.4, section 5 analyses and compares the research findings and section 6 concludes with a summary and conclusion.

2. Related Studies

Malware detection approaches can be categorized using several methods. However, the method discussed by [31] is used in this paper which is that malware detection can either be signature based, heuristic based or behavioral based. Also, malware detection tools can either be host based, or network based [32], and this research examines a network-based approach as it is focusing on the C&C network communication traffic.

The authors in [33] used the SVM machine learning algorithm to develop an intrusion detection system which uses the NSL-KDD dataset to classify network traffic. To select appropriate features, [33] used a hybrid feature selection approach which ranks the features using a feature selection approach called Information Gain Ratio (IGR) and then, refines this further by using the k-means classifier. They achieved an accuracy of 99.32 and 99.37 when used with 23 and 30 features.

A simple yet effective method was developed by [34] which involves extracting statical features called ‘function call frequency’ and ‘opcode frequency’ from Windows PE files. These features are extracted from both the executable files’ header and from the executable’s payload and the features were extracted from a total of 1,230 executable files. The dataset contained 800 malware and 430 non-malware executable files, and the experimental work was conducted using a tool called WEKA. Several classifiers were experimented with, and the results of these experiments can be seen in table 1.

Research was carried out by [35] who used an unsupervised machine learning algorithm to detect bot communication traffic. They used datasets obtained from the university of Georgia which contained bot traffic from both the Zeus and Waledac malware variants. Features were extracted from the dataset by using a tool called Netmate which extracts the traffic as flows and then analyses each flow to calculate their statistical features. The datasets were clustered using the WEKA tool and, all the experimental analysis was also conducted using this tool. The experimental results can be seen in table 2.

BotOnus [36] is a tool that extracts a set of flow specific feature vectors and then uses an online fixed-width clustering algorithm to arrange these features into unique clusters. Suspicious botnet clusters are defined as flow clusters that have at least two members that have an intra-cluster similarity above a certain threshold. All hosts inside these clusters are considered to be infected with bots. BotOnus is an online detection technique that uses unsupervised machine learning algorithms for detecting malicious bots. Because BotOnus employs an unsupervised method that is motivated by the inherent features of botnets, like group behaviors, it can identify unknown botnets without any prior knowledge of them. Table 3 shows the experimental results that were obtained by using BotOnus.

RCC Detector (RCC3) is a tool developed by [37] that uses a multi-layer perception (MLP) and a temporal persistence (TP) classifier to analyze network traffic flows to identify malware communication traffic. The botnet detection system was trained and tested using the DETER testbed and two datasets were used, the DARPA and LBNL datasets. The authors aimed to predict Zeus, Blackenergy and normal traffic and the key to this paper is that RCC examines traffic generated from a host. The tool achieved a detection rate of 97.7%.

3. Problem Statement

The goal of this study is to create a methodology and framework for predicting banking malware using machine learning approaches. Many malware detection approaches already exist and have been researched and used by researchers and some of these include signature-based approaches [38,39] and anomaly-based detection approaches [40,41] however, these do have limitations [42]. For instance, signature-based systems are not able to detect zero-day malware or unknown malware variants, and they must be updated frequently to accommodate newly emerging malware variants.

Many of these problems can be solved with the aid of machine learning [43], and this research has developed a framework and method that can identify various banking malware variants using machine learning. Although some experimental work has been done in this field [33,34,35], little to no research has been done that seeks to detect a variety of banking malware variants by only training one dataset which belongs to only one banking malware variant. The goal of this research study is to develop a machine learning model using a single training dataset. Subsequently, this approach is employed to identify other banking malware variants, and the detection results are compared to identify which ML algorithms perform the best. This research also examines the minimum number of features that could be used to obtain satisfactory prediction result.

4. Research Methodology

This research paper aims to classify C&C network traffic flows as belonging to Zeus which indicates that the C&C network traffic is malicious. Bot samples are collected as pcap files and these pcap files are made up of network flows. A flow is defined as a sequence of packets flowing between a source and a destination host. Each flow is referred to as an ML sample and the features are extracted from these samples. For this research, supervised ML algorithms were chosen as these algorithms are well suited for solving predication and classification problems such as the one being researched in this paper [44]. This paper analyses three supervised ML algorithms and these are the Decision Tree (DT), Random Forest (RF) and the K-Nearest Neighbor (KNN) ML algorithms and also examines an ensemble approach.

4.1. Machine Learning Algorithms

Artificial Intelligence (AI) is made up of several fields which include deep learning, neural networks and machine learning. Figure 5 depicts the various fields of AI [45].

The most widely used approaches in machine learning are supervised, unsupervised and reinforced learning and Figure 6 illustrates the various types of machine learning approaches [46] that could be used. For this paper, supervised ML approaches are used.

There are several types of supervised ML approaches that could be considered for the problem being researched in this paper and these are [46]:

• Binary Classification – Two possible classifications can be predicted for example, an email can either be spam or not spam. The two possible classes are usually either normal or abnormal.
• Multi-Class classification - Multiple classes are involved and each data point is classified into one of the available class options.
• Multi-Label classification - Multiple classes can be predicted for each data point. For example, a house could be present in multiple photos.

For this research, the binary classification approach was selected as this has been used by many researchers to solve similar problems. For the supervised ML algorithms used in this research, a brief description of these is provided below.

One of the most effective and noteworthy machine learning methods for predictive modeling is the Decision Tree (DT) algorithm, which performs exceptionally well when dealing with binary classification problems [47]. Since this research aims to ascertain whether the network flow is malicious (banking malware traffic) or benign, the decision tree technique is a good fit for this prediction problem. Additionally, the decision tree algorithm learns and makes predictions extremely quickly [47].

In comparison to the decision tree algorithm, the Random Forest (RF) algorithm can be more effective, can produce better prediction results, and can lessen the likelihood of overfitting [48]. It can do this because the RF algorithm can construct multiple decision trees which can help to improve prediction results. When utilizing the RF method, it is crucial to adjust the parameters of the algorithm, to improve the prediction accuracy. It can be challenging to foresee the ideal values in advance, and the parameters are chosen by experimentation. One of these parameters is the quantity of trees constructed during the training and testing phases and research shows that constructing more than 128 trees can raise the cost of training and testing while offering no appreciable improvement in accuracy [49]. Constructing between 64 and 128 trees has shown to be the ideal number of trees that should be used, so, the experimental analysis for this research also used between 64 and 128 trees [50].

The K-Nearest Neighbors (KNN) ML algorithm is a supervised ML algorithm which is simple to implement and can produce good results. KNN is considered to be a non-parametric method [51], meaning that it makes no assumptions about the underlying data. Following the computation of the distance between each new data point and every other training data point, the algorithm can classify the new data point in relation to the trained data points [52].

An ensemble approach [53] is also used in this research and for this, the random forest, decision tree and the k-nearest neighbor ML algorithms are all used in the ensemble approach. A voting classifier was used to combine the results of all the models, and for this research, a soft vote [54] was used for predicting the malware. The soft voting approach is useful because it can select the average probability of each class [55].

4.2. System Architecture and Methodology

The system architecture shown in Figure 7 depicts the steps that are completed for the experimental work completed during this research. These include:

• The datasets are identified and collected.
• Features are extracted from these datasets.
• The extracted features are transferred to a csv file and prepared.
• The features are selected for training and testing.
• The algorithm is trained and tested, and a model is created. Only one dataset is used for the training.
• The model is tuned and trained and tested again if required.
• The model is used to test and evaluate the remaining datasets.
• Deploy the final model and test all the data samples and create a report highlighting the evaluation metrics.

4.3. Data Samples

In this study, a variety of datasets were collected, and these datasets represent real-world activity that were captured by various reputable organizations. Six datasets were used for this research, and these were collected from Zeustracker [56], Stratosphere [57], Abuse.ch [58] and Dalhouse University [59]. Abuse.ch correlates samples from commercial and open-source platforms such as VirusTotal, ClamAV, Karspersky and Avast [58]. Dalhousie universities’ botnet samples are part of the NIMS botnet research project and have been widely utilized by many researchers [59]. Table 4 defines all the data sets that were used during this research and provides some details around the banking malware variant, the year that the samples were collected and the number of flows extracted from these samples.

4.4. Feature Selection

The procedure for gathering and preparing the data can be seen in Figure 8 and this is discussed further in this section. The features were extracted and exported into a CSV file and a total of 44 features were extracted. It is important to select the appropriate and best features as this helps reduce overfitting and computational cost and helps the ML algorithm learn faster [60,61]. Several approaches can be used to identify the appropriate features, and the three predominant approaches are [62]:

• Filter method - Feature selection is independent of the ML algorithm.
• Wrapper method - Features are selectively used to train the ML algorithm and through continual experimental analysis, the best features are selected for the final model. This method can be very time-consuming.
• Hybrid – Which is a fusion of the filter and wrapper approaches.

For this research, the features were analyzed using the filter-based approach and three automated feature selection algorithms were used for this analysis, and these included the ANOVA [63], RFE [64] and SelectKBest [65] feature selection algorithms. This empirical analysis determined that the following ten features would be the most appropriate and minimum number of features required for predicting the different malware variants.

• mean_fpktl.
• min_fpktl.
• min_bpktl.
• min_fiat.
• mean_fiat.
• mean_biat.
• min_biat.
• sflow_fpackets.
• sflow_fbytes.
• Duration.

The above ten features were used in the training and testing of the machine learning algorithms and the results of the experimental analysis can be viewed in section 5.

4.5. Evaluation Approach of the Experimental Analysis

The evaluation metrics, precision, recall, and f1-Score are used for the experimental analysis conducted for this research. Precision is the percentage of correctly identified positive cases from the whole data sample [66] and recall is the percentage of correctly identified positive cases from the positive samples only [67]. The formulas used are:

$$Precision ={\displaystyle \frac{TP}{TP+FP}}$$

(1)

$$Recall ={\displaystyle \frac{TP}{TP+FN}}$$

(2)

The f1-Score considers both the positive and negative cases combined, and the formula used to calculate the f1-score is set out below [68]:

$$F1-Score ={\displaystyle \frac{2\ast \left(Precision\ast Recall\right)}{Precision+Recall}}$$

(3)

A confusion matrix [69], shown in table 5, will also be generated for each experiment that is conducted. The confusion matrix calculates the True Positives (TP), True Negatives (TN), False Positives (FP) and False Negatives (FN) scores for each dataset.

5. Results

This section presents the training and testing results for all the ML algorithms and compares the prediction results for each of the datasets. For each ML algorithm, two tables are presented. The first table shows the precision, recall and f1-score results and the second table depict the following information: the number of samples tested; the number of samples correctly classified (true positives); the number of samples misclassified (false negatives). The table also depicts the prediction results of the benign C&C network samples.

5.1. Training and Testing the Decision Tree Machine Learning Algorithms

The DT ML algorithm was trained using the features defined in Section 4.4 and for the training, 3 folds were used. A training accuracy of 0.974 was achieved and table 6 shows the testing results and table 7 depicts a confusion matrix for each of the datasets tested. By examining the key metric, which is the recall score for the malware, most of the recall scores are above 95. The lowest recall rate was 66 for dataset6 and the highest recall score was 99 achieved by both datasets 3 and 4.

5.2. Training and Testing the Random Forest (RF) Machine Learning Algorithm

The results of testing the RF ML algorithm can be seen in table 8 and table 9. A training accuracy of 0.997 was achieved and by examining the key metric, which is the recall score for the malware, most of the recall results are above 95. The lowest recall score was for dataset6, which was 66, and the highest recall score was obtained by datasets 3 and 4, which 99.

5.3. Training and Testing the K-Neaest Neigbor (KNN) Machine Learning Algorithm

The KNN testing results can be seen in table 10 and table 11. A training accuracy of 0.950 was achieved and by examining the key metric, which is the recall score for the malware traffic, most of the malware recall results are above 90. The lowest malware recall rate was 50, which was achieved by dataset6, and the highest malware recall score was 100 achieved by dataset3.

5.4. Training and Testing Using the Ensemble Machine Learning Approach

An ensemble approach was used to train and test all the datasets and the results of these can be seen in table 12 and table 13. Again, focusing on the malware recall score for each dataset, the highest malware recall score was achieved with both datasets 3 and 4 with a score of 99. The lowest malware recall score achieved was for dataset6 which was 66.

5.5. Comparing the Predication Results of All the Algorithms Tested

The results obtained from testing all the algorithms are compared in this section. Figure 9 shows the malware recall results of all the algorithms when tested against all the datasets and Figure 10 shows the benign traffic recall scores. An expanded view of the results can be seen in Figure 11 and Figure 12 which show both the recall and precision scores for both the malware and benign traffic samples.

The results obtained during the testing phase indicate that the decision tree algorithm performed consistently well across all the datasets with high accuracy scores for datasets 1 - 4. Dataset6 does show lower performance with an accuracy score of 0.74, and this seems to indicate that the decision tree algorithm faced some challenges when used for testing on a large dataset. The random forest algorithm also performed well and achieved similar results to the decision tree algorithm. The KNN algorithm performed extremely well across datasets 1 - 4 as the accuracy scores were close to 100 for all these datasets.

The random forest and decision tree algorithms generally perform better than the k-nearest neighbor and ensemble approaches on most datasets, especially for smaller and less complex datasets. K-nearest neighbor performs exceptionally well for smaller datasets but suffers with larger or more complex data. The ensemble approach does provide a balanced approach but performs poorly when compared to the decision tree and random forest ML algorithms, especially for larger datasets.

The experimental results and the patterns observed suggest that the random forest and decision tree models are more robust and consistent across all the datasets, while the k-nearest neighbor and ensemble models may face some difficulties with larger or more complex data. Dataset 6 seems particularly challenging, reducing performance across all the datasets.

This paper has demonstrated an approach that can be used to detect banking malware and some its variants and has demonstrated that the methodology does work across multiple datasets and other variants of the Zeus malware.

6. Conclusions

The framework's ability to identify banking malware and its variants have been demonstrated by the empirical analysis conducted during this research. The research shows that the methodology and framework used for this study can identify both older and newer versions of the Zeus banking malware. It is possible that this approach can be used to detect a large number of banking malware variants without having to examine each one in order to understand its characteristics. This is because the framework and technique developed during this research identified key features that could be used and may also predict other banking malware variants. Also, this research has shown that a reduced set of features can be used for detecting banking malware and this should help increase the performance and time required for training and testing machine learning or deep learning algorithms, especially for large datasets. This research will also benefit other researchers as they should be able to adopt this approach in their own research and will have a good base to begin conducting experiments of a similar nature.

It may be possible to advance this research in the future by improving the methodology to include more banking malware variants, especially variants belonging to a different banking malware family. Additionally, more study may be done to identify other malware types and increase the prediction accuracy of these predictions. The results of this study may also be utilized by researchers to develop an intrusion detection system (IDS) that can identify a variety of malware, and by anti-virus manufacturers to support their development of malware detection tools. Once the infection has been identified, action can also be taken against the malicious communications. Researchers can improve their work by using the results of this study to create their own malware prediction systems.