Threat Actor Attribution Applying a Tactics-Techniques-Procedures Approach: An Empirical Investigation

1. Introduction

Due to the growing global cyberattack surface comprising organizations and individual entities, the risk and complexity of cyberattacks have increased significantly [1]. The most prevalent cyberattacks include Denial-of-Service (DoS) and malware attacks, phishing, zero-day exploitation, and the identification and exploitation of unpatched vulnerabilities. Investigations of current trends indicate that the frequency and potential impact of these attacks have been increasing over recent years [2,3] while the techniques used by malicious actors to launch in cyber attacks are constantly evolving to improve their success rates[4].

A current trend in proactive defense approaches is the use of threat intelligence as a means of collecting and sharing information relating to cyberattacks, including the specific attack indicators and techniques observed; such actionable intelligence is commonly shared through threat reports by security vendors[5]. This information may often utilize the tactics, techniques and procedures (TTPs) in the MITRE ATT&CK Framework [6,7] ; this provides a standardized format that allows security operation center (SOC) teams to detect and counteract threats targeting their environments. Accurate threat sector attributing has been found particularly important as a means of increasing the efficiency of the cybersecurity response [8].

Threat actor attribution is a cybersecurity defense method that identifies the origin of cyber attacks and classifies or associates attacks with specific malicious actors (threat actors) such as advanced persistent threat (APT) groups. This enables a deeper analysis of the tools and techniques used by threat actors to understand the different attack approaches and how these are tailored to the targeted victims’ location and industry affiliation [9].

1.1. Research Background

1.1.1. Attacks and Threat Intelligence

As a result of the evolving cyber threat landscape, significant cyber incidents with victim governments, multinational corporations, and small businesses continue to occur. Different categories of attackers are involved, including state-sponsored threat actors, financially motivated hacker groups, and hacktivists. Attacker motivation varies from espionage to financial gain, or even cause damage and harm tu human life [10].Security Operation Centre (SOC) teams monitor networks and devices within their organizations’ digital infrastructures and investigate alerts and suspicious activities that may be a part of a cyberattack. This is often performed with the use of Security Information Event Management (SIEM) systems as a centralized platform to perform SOC operations [11].

The MITRE ATT&CK Framework (developed in 2013) classifies the methods and approaches s used in cyberattacks by identifying attack TTPs – activities used at the different stages of an attack [12]. Currently, the database comprises a total of 203 techniques and 453 sub-techniques [13].

Another source of threat intelligence are the ndicators of compromise (IOCs) are also shared by commercial security vendors and industry- an government-based agencies. IOCs provide IP addresses, suspicious file hashes, and domain names which are often associated with specific threat actors [14]. However, IOCs are less effective than TTPs as an approach to improving the effectiveness of the organization’s security controls attacks. While attackers can change their IOCs by using a different IP address or a URL or editing a file to change its hash value, whereas TTPs are often more difficult to alter [15,16]. Gopalakrishna et al. [17] places TTPs at the top of the tactical layer of their ‘Extended Pyramid of Pain’, highlighting their cybersecurity defense potential.

1.1.2. Threat Actors

Identifying and understanding the tactics and techniques used by threat actors in cyberattacks has demonstrated value in improving the design of cybersecurity defense systems [18]. Understanding threat actors’ techniques can enhance their organization’s ability to detect threats and enable a pro-active approach to providing protection against emerging threats [19]. However, the threat actor attribution process of identifying the perpetrator behind a cyberattack is a complex task that may be hard to accomplish. It involves the analysis of pertinent technical data such as IOC and TTP digital footprints, as well as a variety of supporting information, for example, targeted victim prefile including the relevant industry sector and their geographical location [20].

But even if only probable an not completely reliable, threat actor attribution results give equip organizations valuable knowledge about their most relevant adversaries and threats [21]. By attributing a cyberattack to a threat actor, the organizations gain the ability to deter threats by counteracting the relevant TTPs and reduce cyberattack success rates. Additionally, threat actor attribution allows SOC teams to prioritize their response to incidents [22]. .

1.2. Related Research Work

1.2.1. Cybersecurity Data Generation Approaches

The MITRE ATT&CK Framework provides security analysts with the capability to decipher attacks in relation to a comprehensive and standardized catalogue of TTPs. Bhole et al. [23] explored the datasets related to threat actor activities through threat intelligence and real-world incidents that were used to analyze cyber threat groups. The public databases ThaiCERT, Malpedia, ICS-CERT, and the European Repository of Cyber Incidents (EuRepoC) which are compatible with the MITRE ATT&CK’s TTP format and have been used in a number of empirical investigations. However, prior research has also demonstrated that the use of heterogeneous cyber intelligence data sources and hybrid feature sets enhances threat actor attribution. For example, Chen et al. [24] successfully extracted TTPs from threat intelligence reports and used them for APT group clustering based on their attributes and techniques.

Machine Learning Models

To mitigate the challenges posed by manual TTP extraction, Domschot et al. [25] compared various machine learning (ML) techniques to associated specific words with different TTPs and extract relevant TTPs directly from intelligence reports; they achieved an accuracy of 69.8% In a related study, Permana et al. [26] attempted to extract TTP form unstructured data (cyber threat news reports) by parsing and searching for selected matching words. They formatted the in formation gathered using structured threat information expression (STIX). The STIX format is machine -readable and is supported by a number of industry and community platforms [27]. Huang et al. [28] also aimed to address the deficiencies of manual TPP extraction. They developed a deep learnings (DL) model for the efficient extraction and classification of MITRE techniques from unstructured cyber threat intelligence ( CTI ) reports.

DL-based generative artificial intelligence (GenAI) models can extract attack patterns and indicators of attack from threat intelligence data to help cybersecurity teams [29]. For example, Saddi et al. [30] proposed a conceptual GenAI model for automated threat intelligence gathering, updating and analysis. GenAI applications such as a large language model (LLM) can be an effective tool for identifying and extracting TTPs from threat intelligence reports, as shown by Fieblinger et al. [5]. For example, MITRE have enhanced the the accuracy their open-source threat report ATT&CK mapper (TRAM) by deploying an LLM [31]. The use of the commercially available ChatGPT (an LLM) as a tool to analyze security-related information from a variety of sources and identify behaviors and attack patterns has been suggested in more recent research [32,33].

Prompt Engineering for LLMs

The performance of LLM models is sensitive to the quality of the ‘prompting’( structured input provided by the user). By applying prompt engineering (PE) best practices, the use can the model performance, increase the accuracy of the response, and overall, gain a greater control over the process [34]. For example, the LLM can be instructed to double-check its output and to re-attempt the task if there are any inconsistencies [24]. Such PE techniques aim to increase the adequacy an the credibility of the LLM’s responses.

Schmidt et al. [35] recently found that prompt model performance improves when several prompt components are included; this ensures that the LLM have sufficient information to respond with optimized and accurate results. Providing comprehensive prompts means to build a strong instructional foundation for the LLM to follow. In the flipped interaction pattern, the user the user instructs the LLM to asks questions (which the user replies to) and continue till it has enough information to perform the required; in the reflection pattern, the user asks the LLM to provide supporting rationale on how and why it responded with the answer to the prompt. Furthermore, it is suggested by Tripaldelli et al. [36] that specifying he role the LLM (e.g., by stating in the prompt that the LLM is an expert on the subject), or by inclusion of step-by-step instructions on how the LLM should perform its task and the specific output required also results in improved responses to prompts

1.2.2. Theat Actor Attribution Approaches

Threat actor attribution process is a time-consuming and challenging task if conducted manually [37]. Noor et al. [38] suggest that ML modeling for their analysis and attribution to criminal identities based on dataset of TPP features extracted from threat intelligence can produce useful and reliable results to support cyberattack investigation However, research om actor and threat attribution is relatively scarce Abou et al.[39] and Desa et al. [40] have examined work threat actor typologies and classifications, motivation and behavior and have noted that there is still a need to develop reliable methods that can enhance threat actor profiling and cybersecurity management and best practicers.

Data extracted from cyberthreat technical descriptions such as the TPPs and the low- and high-level IOCs in David Bianco’s ‘Pyramid of Pain’ framework [41] have been extensively used in current empirical research [24,42,43,44,45,46,47]. This allow to utilize the collective cybersecurity intelligence knowledge and also, to align threat actor profiling with cybersecurity practicians’ expectations[48]. Several studies have reported high accuracy levels.

For example, Duan et al. [42] gathered indicators ranging from low-level IOCs, such as IP addresses and domain names, to higher-level indicators, including malware and TTPs. For attribution, their study used the information about the country and the industry sector targeted by the attack. The authors identified the belongs-to relationship between their dataset of threat actors and a corresponding incident dataset, in which they utilized a heterogeneous graph technique for mapping nodes to threat actors. The dataset comprised 1027 threat actor nodes; an 80/20 split was used to train, and validate and test the model. The authors compared the performance of eight different learning models against their dataset; the highest precision model achieve 54.42% accuracy in positioning the threat actor within the top five most probable threat actors behind an incident.

Kida et al. [43] explored threat actor attribution with regards to malware that was used in attacks and applied a range of classification algorithms to match an attack to a threat actor: random forests (RF), support vector machines ( SVM,), naïve Bayes (NB), and K-Means Clustering. The authors used d a sample of 3594 malware file hashes mapped to 12 actor threat groups (with an 80/20 split for training and testing). The best performing model achieved 89% accuracy in associate malware hashes with threat actors, the authors were able to perform threat actor attribution with an accuracy performance of 89%.

Xiao et al. [44] used IOCs, malware, and TTPs to derive the datapoints needed for threat actor attribution modeling . The authors developed a baseline for matching threat actors to threat intelligence reports through the use of multilevel heterogeneous graphs. The authors utilized data from 1300 threat intelligence reports associated with 21 threat actors. They extracted features from IOCs, TTPs, and malware and optimized the model to achieve threat actor attribution accuracy of 83.2% (associating a threat actor with a threat intelligence report).

Irshad et al. [45] used IOCs, TTPs, malware, and data about targeted countries, industry sectors and targeted applications for feature extraction and training ML models (RM and SVM). Both algorithms achieved 84% accuracy in identifying the correct threat actor behind an attack. Similarly, a study conducted by Chen et al. [24]on threat actor clustering highlighted the success of attribute analysis of attack indicators such as TTPs and IOCs using decision trees (DT).

To improve their results, Chen et al. [24] used natural language processing (NLP) to extract beavertail characteristic’s form unstructured cyberattack and threat reports. Irshad et al. [46] also explored a hybrid feature set comprising both technical features and behavioral features representing the attack context. The best performing Ml model (long short-term memory- LSTM) achieved prediction accuracy of 97% .

Saha et al. [47] have developed a two-level ML model for automatic APT attribution that first identifies malicious samples in the context of a threat campaign, and then identifies samples that operated by the same threat actor. The model uses input from a range ty of heterogeneous file (both executables and documents) and searches for connections between features. It was s trained on a MITRE reference dataset and a dataset of APT samples belonging to a number of APT threat groups. Somewhat similar, Xiang et a. [37] approach threat attribution by margin a large real-world network behaviors dataset with threat intelligence data and applying a K-means clustering algorithm. . The proposed system (IPAttributor) assesses similarities and links between IP data and applies dynamic weighted threat segmentation to identify attacker entities. The highest reported output accuracy was 88.89%

In addition to the methodologies and algorithms used in the studied above, other pattern or characteristic matching techniques considered in current research include the of neural networks and DL algorithms. The effectiveness of the chosen technique varies depending on the number of attributes of each entry, the structure of the data, and the size of the datasets [49]. For example, Böge et al. [50] developed a e DL bidirectional encoder representations from transformers (BERT) models for entity recognition based on commands that are executed within an attack. The research results supported the applicability of BERT models for threat actor attribution. Zhang et al. [51] use an LLM to construct a schematic representation of an attack’s s technique and model the attack implementation details with a higher layer of granularity compared to features extracted from technical cyberthreat descriptors. The proposed automatic system for threat attribution (APTChaser) exhibited a high mean reciprocal ranking.

1.3. Research Aim, Questions and Contribution

As shown by the review of the extant research, a current problem with threat actor attribution approaches is that the uncertainty about correctly identifying the threat actor group behind a cyberattack is still relatively high. This is due in part to the lack of f reliable technical input to assist the threat actor identification; additionally, the process is complex and time-consuming [52,53].

1.3.1. Research Aim and Question

Building on prior research in threat actor attribution approaches, tools and techniques and considering the importance of TTPs as threat actor indicators, this study explores machine learning (ML)-based methods for effective threat actor attribution. In addition to using TTP data for the extraction of data input features, the study extended the feature space by incorporating input features from other cyber incident characteristics, such as the geographical location and the industry sector of the victim targeted by the cyber attack.

The main purpose of the study was to identify feasible options for creating an effective ML model for threat actor attribution. The main research question of the study was formulated as: ‘What ML models and feature selection methods can be used for effective threat actor attribution?’

1.3.3. Research Contributions

Currently, a total of 203 techniques and 453 sub-techniques are classified in the MITRE ATT&CK Framework [13], with threat actor groups utilizing specific subsets of these techniques and sub-techniques. This study proposes and tests a novel framework for threat actor attribution based caber attack TPPs s and the contextual characteristics of cyber attack target. The results suggest that the feature set used in this study can be used for successful threat actor attribution, , excluding the need to collect data about the indicators at the lower levels of the Pyramid of Pain such as IOCs, domain information,, IP addresses, and hashes .

The rest of the paper is organized as follows. The literature review section explores cyber attacks, threat intelligence, and the current threat actor attribution landscape. The Materials and Methods describes the data sources used in this study including the use of ChatGPT to create a dataset of 783 cyber incident reports , and describes experiment set up. The results and discussion sections present and analyze the s findings of the experiments. . .

2. Materials and Methods

2.1. Cyber Incident Data Source

To build the ML models, it was necessary to create first a suitable dataset of cyber incidents including the threat actors conducting cyberattacks, the MITRE ATT&CK TTPs as observed during the attack, and the victim’s geographical location and industry sctor. Section 1.2 identifies several databases that could be utilized for this research;, based on desired fields available in the database, the amount of records included, the accuracy of its data, and the frequency of updates to its data, the EuRepoC database [54]was selected to build the cyber incident dataset used in this research

The EuRepC was set up by the EuRepoC consortium ( established in 2022)l the aim was to support cybersecurity-related discussions and policy-making by providing a reliable cyber incident database and analytical frameworks. Trustworthy contributors such as European Union (EU) government agencies, including the German Federal Foreign Office and the Danish Foreign Ministry, helped build the database which is available for public use and research [55]. This research used the 4 March 2205 downloadable version of this database (eurepoc_global_dataset_1_3.xlsx) [54].

This data consists of 3414 rows with 85 data points each. Each row contains data related to a specific cyber incident, including the description of the cyberattack, the date of when the attack started and the date the attack ended, the date the attack was added to the database , a description of the attacker (e.g., the name the country of the threat actor and the threat group), a description of the victim (e.g., industry sector and country name, and the cyber incident report or article source URLs. The database record includes as well information about the response to the attack and its impact, the source of the attribution, and the number of threat actpr group attributions for each cyberincident.

The following fields were used to create this study’s dataset of cyber incidents:

1. 

incident_id

The incident_id field indicates the specific cyberattack event that occurred.

2. 

receiver_country

The receiver_country field specifies the victim’s geographic location.

3. 

receiver_category_subcode

The receiver_category_subcode field specifies the victim’s industry sector.

4. 

initiator_name

The initiator_name field indicates the threat actor group name. This attribute is an essential datapoint that is used to test and validate the threat actor attribution model proposed in this study. This study focuses on the ‘advanced persistent threat’ (APT) actor groups (group name prefix APT) The groups were selected as APT attacks are among the most dangerous ones in cyberspace. Often, they are hard to detect as the highly skilled malicious actors apply a range of tactics and use different tactics to carry out covert and sophisticated attack campaigns [56] .

5. 

number_attributions

The number_attributions field shows how many threat actor groups were attributed to the cyber incident. As the attributions were based on information sourced from sdiffrent channels ( e.g., self-attributions, technical reports conducted by IT companies, political, direct, or anonymous statements in the media, legal action proceedings), some incidents were attributed to multiple threat actors. In this study, we limited the scope to cyberincidents that had been attributed to one threat actor group only (number_attrivutions=1) to avoid ambiguity.

6. 

sources_url

The “sources_url” field includes one or more cyber incident links to articles or report links which contain additional information about the specific cyberattack. This field was selected as the study used the articles and the to extract the TTPs associate with the specific cyber incidents.

2.2. LLM-based TTP Extraction Prompt Engineering

This study used an AI-supported TTP data extraction method. In their study of the effectiveness of AI in sentence parsing Sewunetie et al. [57] found that ChatGPT version 3.5 was ca[able of n understanding, parsing, and extracting requested information from input prompts. In this research we used ChatGPT version 4.0 to extract specific TTPs from a reference cyber incident dataset ’s dataset which contained cyber incident report URLs in the field sources_url.

The ChatGPT prompt was engineered specifically to achieve high response accuracy for accuracy of the responses. As shown below, the prompt was very specific and detailed. It instruct ChatGPT to extract TTPs from the sources associated with the specific cybercide, using the TTP definitions in the MITRE ATT&CK Framework.

Table 1. Example of a ChatGPT prompt engineered to rxtract MITRE ATT&CK Framework TTP information from the report: https://securityaffairs.com/132452/hacking/apt28-hacked-nato-think-tank.html.

Prompt component	Description
Steps to follows	Step 1. Visit the report link and read the information presented. Step 2. Identify specific MITRE ATT&CK Framework techniques and sub-techniques found in the body of the report that are explicitly stated or very strongly inferred by the text. Vague inference is not acceptable. Step 3. Extracted techniques and sub-techniques must be completely based on the information found in the report and not based on any external knowledge of the threat actor or the attack. Step 4. Each extracted technique and sub-technique should be supported with evidence in the format of a direct quote from the report for user validation. If the article is not in English, then translate the direct quote to English and do not output the quote in the original language. Only include the direct quote in the Supporting Evidence field in the output format template. Step 5. All techniques and sub-techniques must be based on the information found on https://attack.mitre.org/techniques/enterprise/ and should be able to be justified based on the direct quote and relating to the technique or sub-technique description found on https://attack.mitre.org/techniques/enterprise/. Step 6. Avoid unnecessary enumeration of sub-techniques if they are inferred and not explicitly mentioned. Step 7. Follow steps 1-6 again and verify that the results are the same. If they are not, then repeat 1-6 again until the results are consistent.
Output Format	The output must be in the following JSON format: { "incident-techniques": [ "TXXXX", // Technique or Sub-technique Name – Supporting Evidence "TYYYY" // Another Technique or Sub-technique Name – Supporting Evidence ], 'targeted-countries': [ 'Country1', 'Country2' ], 'targeted-industries': [ 'Industry1', 'Industry2' ], 'URL': 'Report URL' }
Additional clauses	Do not include any MITRE ATT&CK Framework techniques or sub-techniques unless they are directly supported by a quote from the report. Do not infer techniques based on the known behavior of a threat actor or group, unless these techniques or sub-techniques are strongly inferred within the report. Each technique or sub-technique must be directly justified by a quote from the report, which must be included word-for-word in the JSON output. If a technique cannot be explicitly tied to a quote, it must be excluded. Do not list techniques based on general attack stages or assumptions; only include those with specific textual evidence. If any technique is included without a matching direct quote from the article, the output is invalid. If the report explicitly lists any MITRE ATT&CK techniques or sub-techniques using MITRE technique IDs (e.g., T1059 or T1566.001), all of these must be extracted into the JSON output. Do not omit any technique listed with a T-ID, even if it is not described in full descriptive text. If a MITRE ATT&CK matrix or table is included in the report, all techniques in that matrix must be extracted and included in the output. Any omission of an explicitly listed technique or sub-technique invalidates the output. Strongly inferred techniques and sub-techniques must be included if they are supported by the report text. Inference is allowed only if it is strongly supported by specific quotes or sentences in the report and can be directly mapped to a MITRE ATT&CK technique. Vague, weak, or assumed inferences must be excluded. Provide the supporting evidence after the JSON output, in addition to it, for manual verification of the techniques. Before responding to this request, ensure that all these instructions, clauses, and format are strictly followed.

Table 1. Example of a ChatGPT prompt engineered to rxtract MITRE ATT&CK Framework TTP information from the report: https://securityaffairs.com/132452/hacking/apt28-hacked-nato-think-tank.html.

Prompt component	Description
Steps to follows	Step 1. Visit the report link and read the information presented. Step 2. Identify specific MITRE ATT&CK Framework techniques and sub-techniques found in the body of the report that are explicitly stated or very strongly inferred by the text. Vague inference is not acceptable. Step 3. Extracted techniques and sub-techniques must be completely based on the information found in the report and not based on any external knowledge of the threat actor or the attack. Step 4. Each extracted technique and sub-technique should be supported with evidence in the format of a direct quote from the report for user validation. If the article is not in English, then translate the direct quote to English and do not output the quote in the original language. Only include the direct quote in the Supporting Evidence field in the output format template. Step 5. All techniques and sub-techniques must be based on the information found on https://attack.mitre.org/techniques/enterprise/ and should be able to be justified based on the direct quote and relating to the technique or sub-technique description found on https://attack.mitre.org/techniques/enterprise/. Step 6. Avoid unnecessary enumeration of sub-techniques if they are inferred and not explicitly mentioned. Step 7. Follow steps 1-6 again and verify that the results are the same. If they are not, then repeat 1-6 again until the results are consistent.
Output Format	The output must be in the following JSON format: { "incident-techniques": [ "TXXXX", // Technique or Sub-technique Name – Supporting Evidence "TYYYY" // Another Technique or Sub-technique Name – Supporting Evidence ], 'targeted-countries': [ 'Country1', 'Country2' ], 'targeted-industries': [ 'Industry1', 'Industry2' ], 'URL': 'Report URL' }
Additional clauses	Do not include any MITRE ATT&CK Framework techniques or sub-techniques unless they are directly supported by a quote from the report. Do not infer techniques based on the known behavior of a threat actor or group, unless these techniques or sub-techniques are strongly inferred within the report. Each technique or sub-technique must be directly justified by a quote from the report, which must be included word-for-word in the JSON output. If a technique cannot be explicitly tied to a quote, it must be excluded. Do not list techniques based on general attack stages or assumptions; only include those with specific textual evidence. If any technique is included without a matching direct quote from the article, the output is invalid. If the report explicitly lists any MITRE ATT&CK techniques or sub-techniques using MITRE technique IDs (e.g., T1059 or T1566.001), all of these must be extracted into the JSON output. Do not omit any technique listed with a T-ID, even if it is not described in full descriptive text. If a MITRE ATT&CK matrix or table is included in the report, all techniques in that matrix must be extracted and included in the output. Any omission of an explicitly listed technique or sub-technique invalidates the output. Strongly inferred techniques and sub-techniques must be included if they are supported by the report text. Inference is allowed only if it is strongly supported by specific quotes or sentences in the report and can be directly mapped to a MITRE ATT&CK technique. Vague, weak, or assumed inferences must be excluded. Provide the supporting evidence after the JSON output, in addition to it, for manual verification of the techniques. Before responding to this request, ensure that all these instructions, clauses, and format are strictly followed.

Due to errors and inconsistencies caused by ChatGPT’s learning and adaptation to its own responses, results appear to lose their accuracy over time of using the service, as also noted in [58]. To ensure that ChatGPT strictly followed o the instructions presented to it in the crafted prompt, a ‘restart’ prompt was sent between requests, to ensure that these instructions were followed explicitly: ‘Forget everything you have learnt, clear memory, and start a new instance’.

Overall, ChatGPT extracted TTPs from 783 reports, each representing a cybersecurity incident. The authors analyzed manually the supporting evidence (direct quote) for every technique or sub-technique identified by ChatGPT to ensure that the source report that was accurately associated with the technique or sub-technique and that the direct quote provide clear evidence for it . A ChatGPT response that was found invalid, was removed from the results.

If a ChatGPT response was judged ambiguous by the researcher, the following ‘repeat’ prompt was given to ChatGPT, requesting the model to provide more information: ‘Reevaluate each technique and sub-technique and compare it to the MITRE ATT&CK Framework descriptions and remove any that do not adhere to the provided instructions and criteria’. The aim of this prompt was to make ChatGPT analyze its findings and provide additional justifications or evidence to support techniques it identified and remove techniques that were weakly inferred.

A total of 352 MITRE ATT&CK unique techniques and sub-techniques were extracted. As shown in Figure 1, there was a significant variance is a wide variance in the number of TTPs that ChatGPT was able to extract from each usable report, ranging from one TTP to 35 TTPs extracted from a report.

2.3. Cyber Incident Dataset

To create a cyber incident dataset for training and testing the ML models, we selected APT actors who met the following criterion: to be associated with at least 15 cyber incidents. The selection process resulted in a set A of ten MITRE threat actors; The resulting cyber incident dataset comprised data about 783 incidents A (Table 2).

Each record of the dataset contained the threat actor ID, the targeted countries/ regions, targeted industry sector and subsector (manually extracted and summarized from the EuRepoC cyber incident dataset), and the corresponding TTPs, extracted from the 738 reports associated with each of the cybersecurity incidents (extracted using ChatGPT, as described above). Table 3 describes the variables included in th4 dataset.

To create the study dataset T, then variables described in Table 3 we applied a one-hot encoding , mapping evet variable to a binary value of 0 or 1 [59]. The resulting study dataset T contained 783 data rows representing individual cyber incidents. Each row was labeled with a label form the set A and comprised 471 features columns of binary values. The resulting structured cyber incident dataset T was slightly imbalanced as 52% of the datapoints (the cyber incidents) belonged to three classes only (APT_29 ,APT_144, and APT_28).

2.4. Experiment Design

Five ML algorithms were selected were selected and used to build an Ml multi0class classification model Four of the algorithms (DT, RF, SVM, NB) were used in the related work reviewed in section 2 were selected based on their use in previous published research (refer section 1.2). The fifth ML algorithm (KNN), is a supervised learning algorithm that classifies a new datapoint based on the classification of similar datapoints in the training dataset [60]; it was considered suitable for this research as threat actors are identified based on the TTPs observed during an attack and the similarities between targeted countries/regions and industry sectors.

To deploy and experiment with the ML algorithms, a series Python scripts were created by utilizing the Python scikit-learn open-source ML library in a Python IDE (PyCharm Community Edition 2023.3 [61]). The scripts were written and tested on an Asus ROG Strix G16 Laptop with 64GB of RAM, 13th Gen Intel i7 CPU, Nvidia GeForce RTX 4060 GPU, running on Windows 11 Home Edition.

In all experiments, the input dataset was split into a training dataset (70%) and testing dataset (30%). To ensure consistency across all algorithms regarding splitting the training and testing data, the ‘random state’ value was set to 7 across all models. To reduce overfitting and bias due to the majority of TTPs extracted from a relatively small number cybercide reports, a stratified 10-fold cross-validation method [62] was adopted in all experiments where the input dataset included all TTP features (refer section 3.1 and 3.2). Stratified 3-fold cross-validation was used in the experiments which compared model performance across of specified TTP feature ranges (refer section 3.3).

The performance of the models was evaluated using the performance metrics of precision (PR), recall (RC), F1-score (the harmonic mean of PR and RC), and mean model accuracy (mean AC) and standard deviation (STD).

While the mean AC is an effective indicator of the overall model performance, the F1-score is also a significant performance metric in the context of the experience: both false positives (the threat actor attributed to the incident is not the one behind the incident), and false negatives (the threat actor behind the incident was not recognized) have a significant impact on the operational and managerial decisions made on the basis of the classification output. To account for the unbalanced class size, the weighted values of PR, RC and F1-score were considered.

3. Experiments and Results

In the three subsections following, we provide the results of six of experiments, each series involving a different feature selection method. To illustrate the step-wise approach in training and testing the different ML models, we present performance data perc class for KNN models, and average performance data for all models. The KNN models were tested with three different values of K (1, 2 and 3); in the results below, K was set to 1, as the best performing parameter.

3.1. Threat Actorr Attrubution Using Unweighted Features

In the first two series of experiments, the ML models were trained and tested using all encoded TTP features in the cyber incident dataset T. In the second series, all encoded TPP features and all encoded cyberattack target features (i.e., the data about the targeted country/region, and industry sector and subsector of the attack receiver) were included as input features in the training and testing data sets. Table 4 shows the per-class metrics of the KNN model while Table 5 shows the weighted PR, RC and F-score values for the multi-class classification, and the mean AC and STD achieved by the five models (applying 10-fold cross validation). As seen, all models achieved both higher accuracy and higher F1-score when using all features.

3.2. Threat Actorr Attrubution Uisng Unweighted Features

In the subsequent experiments, several different approach were applied to improver the models’ performance by apply weights to the selected features

3.2.1. Rarity-Based Feature Weights

In this approach, the encoded TTP and attack target features were assigned weights based on their usage frequence in the cyber incident dataset T. The assumption underlying the approach was that the more uncommon a TTP or a characteristic of the targeted cyberattack victim get characteristic was, the higher the weight it should be given when used for threat actor attribution. Table 6 shows the algorithm for calculation rarity-based feature weights.

Similarly to above, two series of experiments were conducted , training and testing all ML models on an input dataset. In the first serie4s, the input file included the TTP-features in the cyber incident dataset T (weighted based on rarity). In the second series, the input file included all features in the dataset T, rarity-based weighted.

Table 7 shows the per-class metrics of the KNN model while Table 8 shows the weighted PR, RC and F-score values for the multi-class classification, and the mean AC and STD achieved by the five models (applying 10-fold cross validation).

As in the previous tests, the mean accuracy of the models was significantly higher when all features were included. While the models’ mean accuracy using weighted and unweighted features was compatible for both types of features selection (TTP features only, and all feature, respectively), the respective weighted F1-scores of the models using all features weighted were higher than the weighted F1-scores of th4 models using all features , unweighted.

3.2.2. Pre-Set Feature Weights

This manually weighting approach explored the impact of a number of combinations of pre-determined weights in the range of 1 to 3 to the different feature categories. It was found through experimentation and testing that the highest accuracy levels were achieved when TTP features s were given a weight of 1, and the attack target features were given a weight of 3. Table 9 shows the algorithm.

Table 10 shows the per-class metrics of the KNN model while Table 11 shows the weighted PR, RC and F-score values for the multi-class classification, and the mean AC and STD achieved by the five models (applying 10-fold cross validation). Overall, these results shoe an improvement in comparison to the previous sets of results. The average F1-score were improved. .

3.2.2. Hybrid Feature Weights

In this last series of experiments with, the TTP features were assigned rarity-based weights (refer section 3.2.1), while the attack target features were assigned fixed when the weights of the attack target features were set to 3 (as this value gave the best accuracy). Table 12 shows the per-class metrics of the KNN model while Table 13 shows the weighted PR, RC and F-score values for the multi-class classification, and the mean AC and STD achieved by the five models (applying 10-fold cross validation).Both the mean accuracy across the models and the averaged F1-score were improved for the KNN, SVN and NB models.

To check the accuracy of the results, the final version of the KNN model was modified to include an interactive data input function to the Python script. The function prompts the user to input TTPs, target country and target industry information using the dataset T format. Once the user enters this information, the model performs the classification threat actor attribution and outputs the predicted threat actor attribution

Using the input function, the model was tested on new samples of cyber incidents incident data (five per each threat actor class). As shown in Table 14, the average classification accuracy achieved was 86.,00% .

An additional experiment was conducted to explore the impact of small-sized classes on the model performance. When the smallest class APT_32 (with the lowest weighted F1-score of at 0.44) was removed to create more balanced input dataset, the model’s mean AC increased slightly to 93.19%.

3.3. Threat Actor Attribution by TTP Ranges

To explore the impact of the high number of TTP feature in the feature set of the feature set of cyber incident dataset T, we split the dataset T into subsets, based on t the number of TTP features with a value of 1 (i.e., the number of TTPs extracted from the respective cyber incident report), and removing the TTP feature columns that had 0 values only. We aggregated the subsets to create input datasets with TTP counts in the ranges shown in the first column of Table 15. Three tests were performed, using the KNN algorithm to build the ML classifier, using: : (i) Unweighted TTP features only (as described in section 3.1); (ii) Rarity-based weighted TTP features only (as described in section 3.2.1; and (iii) Rarity-based TTP features and pre-determined flat weighting of the attack target features (as described in section 3.2.2). Table 15 shows the mean AC for each if the tested ranges, using stratified 3-fold cross-validation.

4. Discussion

This research identified ML algorithms that have shown effectiveness in threat actor attribution. The effectiveness of the respective models was explored and compared using different methods of feature selection and weighting.

4.1. Results Overview

Overall, the NB models performed consistently wore compared to the other four ML algorithms. The highest mean AC was obtained by the KNN algorithm utilizing rarity-based weighted TTP features and predetermined weights for the attack target features. The second-best performing algorithm was SVN with a mean AC=92.20% and weighted F1-score = 94% (the highest across all models). The RF algorithm achieved the highest mean AC of 65.75% and the best F1-score of 69% when using TTP features alone. Using different types of feature weighting did not have an impact on the performance of the RF and DT models.

The KNN model responded most positively to using predetermined weighting, achieving the highest mean AC of 91.95% (with the attack target features weighted three times more than the TTP features). However, the weighted F1-score of the KNN model was relatively low (88%). Lastly, when TTP features were weighted based on rarity and the attack target features were given a flat weighting, the best performing model (KNN) achieved a mean AC of 92.59 and a weighted F1-score of 90% (second highest across all models). The manual verification of the results achieved by the KNN model using hybrid feature weighting indicated that the model was performing well for all classes. Additionally, the user input function provides a means for user testing of the black box ML model thus increasing user confidence in the predicted threat actor attribution.

4.2. ML Models and Feature Selection Methods for Thret Actor Attreubutiin

Returning to the research question guiding the study, it may be concluded that four of the explored ML models (DT, RF. SVN and KNN) can be used to build effective ML models. Regarding the use of TTP features only in the input datasets, the experiments indicated that using the extended feature space including the TTP features which represent the cyber attack methods. and the features characterizing the cyber attack context (geographical location and industry) improved significantly all models’ performance.

In particular, when using TTP features alone, the classification accuracy of the four models varied between 55.16% and 65.75%; the best-performing model (RF) achieved mean AC of 65.75% for both weighted and unweighted TTP features. When the attack target features were added, the classification accuracy varied between 79.30% and 92.59%

The choice of feature weighing method had a significant impact on the performance of the KNN and SVN models only (with the classification accuracy increased from 70.94% and 84.78% to 92.59% and 92.20% for the KNN and SVM models, respectively); the classification accuracy of the DT and RF models was not affected by applying feature weighting (highest AC achieved was 84.92% and 91.82%, respectively).

The highest mean AC values were achieved by KNN and SVM, respectively (applying rarity-based weighting to the TTP features and flat weighting g to the attack target features; the highest F1-scores were also achieved by the same model/ feature selection combination but in reverse order (the F-score of SVN was 94.00% compared to the F-score of KNN which was 90.00% ). These results also indicated the setting the K value to 1 for the KNN algorithm, did not cause significant overfitting. This is important as a real-life cyber data includes information about well known threat actors and about new entrants to the cyber crime scene.

4.3. Cyber Incident Class and Feature Set Size

4.3.1. Class Sample Size

In this study, the study dataset was constructed to include threat actors with more then 15 reported cyber incidents in the MITRE database this resulted in having five ‘large’ classes (sample size between 18 and 58), and five ‘small’ classes (sample size between 82 and 169). This affected the performance of the ML models. For example, the performance metrics of the class that had a number of samples very close to the cutoff point (APT_32, with 18 samples) were consistently the lowest, compared to the other classes.

When the KNN model was tested on new data, the classification accuracy of four of the smaller classes was below the leverage while the classification accuracy od four of the large classes was above the average, reflecting imbalance. However, ,runing the KNN model over a resampled dataset that excluded APT_32 showed only small (0.65%) increase of the classification accuracy.

4.3.2. Feature Set Size

As pointed out earlier, the cyber incident dataset T had a very large feature set as it included all TTPs extracted from the 783 cyber incident reports. Comparing the results from the tests described in section 3.3 it can be seen that the actor attribution accuracy was considerably higher for datasets with a smaller number of TTPs Figure 2). However, the results are inconclusive as the datasets with higher number of TTP feature counts contained a small number of samples.

Further analysis of the permeance of the KNN model across datasets with different number of TTPs indicates that the highest classification accuracy when using TTP features only was archived when the number of TTP features was between 1 and 6 or above 18, wth model performance trend similar for unweighted and weighted TTP features. . This may indicate that the TTP features in these dataset included a higher number of unique TTPs compared to the other datasets, which may have included a higher number of common TTPs. Furthermore, when all features were used and weighted (rarity based for TTP features, flat weighing fit attack target features) the model permeance tended to drop with the increase of the number of TTP features in the set. Ultimately, three model performed with same accuracy for datasets with 18 and more TTP features for all three methods of feature selection and weighting.

The number of TTP features per sample was determined by t the ChatGPT ouput. With effective prompt engineering, providing adequate instructions detailing precisely how to extract the TTPs from each report and including strict criteria to be followed, accurate TTP extraction can be performed. However, mistakes can occur in this process, with ChatGPT often producing vague inferences. It was required to manually verify ChatGPT’s responses to ensure the correctness of the threat extraction. This process may have affected the quality of the threat extraction.

4.4. Comparison with Relateds Work

Table 16 presents a comparison between this study and four related threat actor attribution studies that were reviewed in section 2. Th reported accuracy of the studies varies from 54.42% to 96%. This study compares favorably to the first three e studies in terms of achieved accuracy. While the fourth study has a higher accuracy, it uses more feature categories, making it harder to implement in real-life contexts.

4.5. Study Contributions

This research makes several contributions. First it demonstrates how specific ML algorithms can be used to model threat actor attribution with high accuracy, using TTP data as well as data about the attack context. Second, the study has demonstrated the effectiveness of using an LLM to analyze and extract TTPs from cyber incident reports, following provided TTP specifications. Third, the study has contributed a multi dimensional dataset of 783 samples attributed to 10 threat actors that can be used in related research.

The outcomes of this study may help cybersecurity professionals estimate the number and types of datapoints that are required to adequately attribute a threat actor to a cyberattack and how to improve the accuracy of the classification by ifgting the input dataset features.

4.6. Study Limitations

The study has several limitations. First, only five ML algorithms were used n the experiment. Second, there were a number of limitations related to the study dataset: the numbers of the threat actors in the dataset was relatively small; the number of TTP features with a binary value of 1 (i.e., the TTP features specific to the sample) varied widely, with 69% of the samples in the dataset represented by r six or fewer TTPs; the class size also varied, with the smallest class represented d by 18samples only.

Lastly, even TTPs are at the top of the Pyramid of Pain and are considered as a reliable and stable cyber attack attributes, threat actors may alter their techniques, and new threat actors may appear . To ensure trustworthy threat actor attribution, , the most up-to-date information should be used to train. This study did not consider altered, newly seen existing techniques, or emerging techniques that were yet to be defined by the MITRE ATT&CK Foundation.

4.7. Further Research Directions

There are several aspects of this research that can be fer explored in future research to expand on the capabilities of threat actor attribution. Increasing the number r number of threat actors and cyber incidents and generating a mor balanced dataset may increase the accuracy of the classification and achieve a better understanding of the effect of increasing the number of 1-valued TTP features per sample.

Second, the manual testing conducted within this research can be expanded upon to explore variance in threat actor attribution using this dataset for training and the final KNN model against cyber incidents that were not included within the training dataset and additional real-world incidents, or against further subsets of TTPs from reports already utilized in the training dataset.

Third, future research can explore the impact of inclusion of additional cyber incident attributes from the Pyramid of Pain as a means of reefing the clarification model output , such as IP addresses, hashes, domains, malware, or tools. Another avenue for further research is to investigate the effectiveness of other ML algorithms and improve the classification accuracy by using an ensembles MK model.