A Machine Learning-Enhanced Cybersecurity and Optimization Framework for Intelligent Threat Detection and System Efficiency

1. Introduction

As the digital system and network has become more and more complex, the number and the sort of the threats have increased enormously. Common intrusion detection system (IDS) systems use exact static rules that are not adaptive to the attack pattern and they produce high rates of false positives [1,2]. To address these limitations, the area of machine learning (ML) and cybersecurity cross-boundaries has emerged as a front-line research direction to enable dynamic anomaly detection and assurances on future threats [3,4].

Hybrid machine learning models such as ensemble, deep learning and reinforcement learning have been introduced to improve the detection accuracy of the IDS, particularly in high-throughput and heterogeneous networks [5,6]. For example, convolutional neural network (CNN) is able to capture more complicated spatiotemporal features within network traffic data [7], and ensemble learning methods like XGBoost and Random Forest are interpretable while performing robustly on imbalanced classificaiton tasks [8,9].

But a correct detection is not by itself sufficient to make a good cyber security detection. Besides, other constraints for real-world systems such as limited computational resources, energy efficiency and the need to respond in real-time also needs to be considered [10]. Therefore, it is valuable to search for multi- objective solutions, inspection and system optimisation are combined [11,12]. Production deployments require load balancing, adaptive thresholding and intelligent alert routing to decrease latency and resource costs [13].

Towards this direction, in this paper, we develop the Machine Learning enhanced Cybersecurity and Optimization Framework (MLECOF) that includes ML-based threat detection and dynamic optimization module for resource scheduling and response coordination. MLECOF is fully configurable and consists of an anomaly pre-processing filter, an IDS core, for which the implementation relies on DNNs, and a multi-objective optimization engine based on Genetic Algorithms (GA) to optimize over security, efficiency and energy consumption aspects.

This integration of multi-level interventions reflects recent reports. Some works applied ML to cyber-physical systems [14], edge computing platforms [15], cloud data centers [16], and IoT networks [17]; however, none of these systems integrate detection and operation optimization at scale across a modular, distributed framework. To the best of our knowledge, our work bridges this gap through the following comparison, (i) comparison of diverse classifiers (Random Forest, XGBoost, CNN), and (ii) evaluation of effects of optimizations in terms of CPU usage, latency, and energy savings employing a set of experiments in real-time.

We also validate mode with different well-known benchmarks such as NSLKDD, CICIDS2017 1, and UNSW-NB15 2 [18,19,20]. Such data sets make it possible to carry out exhaustive experiments on various classes of intrusion, on various representation spaces and for a variety of attack vectors, thus evaluating their generality and transferibility.

2. Related Works

Machine Learning (ML) based security models have been the focus of many researchers as they provide flexibility and smart threat detection. Previous works were proposed to adopt statistical model and rule-based detection which encountered great difficulty to detect zero-day attacks and high rate of false positive [21]. Due to the spectral complexity of network traffic, ML-based IDS systems have emerged as more dynamic and resilient solutions [22,23,24]. They can adapt to evolving threat landscapes and reduce dependence on human-engineered rules.

Supervised methods, such as Decision Trees, Support Vector Machines (SVM), k-Nearest Neighbors (k-NN), are also widely used in anomaly detection [25,26,27,28]. But, in most cases, they rely on a huge labeled dataset, which is unrealistic for a lot of real-world problems. One of the reasons for this has inspired an interesting line of semi-supervised and unsupervised approaches, which might avoid the need for so much labeled input [29,30,31]. So while such procedures can detect deviations in behavior, they might not be suitable for new types of attacks.

Various deep learning techniques such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Autoencoders have revolutionized the intrusion detection scenarios [32,33,34,35]. Because of their ability to perform hierarchical feature learning and to exhibit sequential behavior, RNNs are well suited for the modeling of temporal and spatial dependencies in network flows. The hybrid model structures such as CNN combined with LSTM or GRU have been proven to enhance the performance of the system for advanced attack detection as per the literature [36,37,38].

Ensemble models, in particular Random Forest and gradient boosting like XGBoost, have been generally successful on multi class-classification such as network intrusion detection [39,40,41]. These techniques apply bagging and boosting to avoid overfitting and generalize well against different kinds of attacks. Ensemble classifiers have been successfully used on the NSL-KDD [42], CICIDS2017 [43], and UNSW-NB15 [44].

Trending toward edge computing has increased demands for lightweight and power-efficient IDS frameworks [45,46,47]. There is some research towards the development of detection architectures that can be deployed under a constrained computational budget without significantly compromising the detection accuracies. Transfer learning and federated learning have also been proved to be the effective paradigms in distributed IDS systems [48,49,50].

Optimisation techniques are crucial to improve the performance and the resource utilisation of ID systems. Some existing literatures (GA [51], PSO [52], ACO[53]) also reported that the evolutionary algorithms, namely GA, PSO and ACO, are employed in feature selection task, hyperparameter tuning and rule optimization [54,55]. Multi-objective GA (MOGA) [55,56,57] also generalise this method considering the accuracy versus the computational time and power.

The integration of ML in SDN and network function virtualization (NF G) has made scalable and flexible deployment of security functions possible [58,59,60]. These paradigms transform IDS modules into traffic pattern, policy changes & threat-aware. RL for the first-hop decision in real-time has been developed with the use of SDN-based architectures [61,62,63].

Adversarial machine learning and security of" ML models themselves have recently attracted attention as well. Recent work has showed that the ML-based IDS is vulnerable to adversarial samples, poisoning attacks as well as model inversion attacks [64,65,66]. This has driven the investigation of different secure training methods based on adversarial training [67], differentially private learning [68], and secure federated learning protocols [67,68,69,70].

Recent research focuses on the interpretability and explainability of AI models in cyber security [71,72,73]. As AI models become more complex, interpretability and trust in the outputs from them is critical for security analysts. Techniques like SHAP, LIME, or fuzzy logic based inference have been used for interpretable models.

DSLs also similarly remains to influence the efficacy of IDS. The move from old datasets KDD’99 to newer realistic datasets such as CIC-IDS2018, BoT-IoT, and TON_IoT, which are more extensive and real, represents efforts to obtain more realistic scenarios for training and prediction [74,75,76,77]. These repositories include real network traffic, IoT devices and cloud infrastructures data, therefore they are more generalizable.

There are also researches that propose techniques to incorporate the ML-based IDS into cloud-based security orchestration system [78,79,80]. These include detection and auto-response system, to offer a complete remedy of any threat. Microservices and containers As the security service world has adopted a modular architecture built on microservices and containers, this means that security code has been able to be integrated into many platforms.

In Cyber-Physical systems (CPS) and Internet of Things (IoT) domain, the dedicated intrusion detection mechanisms have also been proposed to secure the resource-constrained devices and data flows from the time domain [81,82,83]. Efficient deployment on edge nodes is actively being investigated from a number of perspectives such as lightweight models, compression technology, and energy-aware scheduling.

Recent progress and adoption of attention mechanisms and transformers in cyber-security (a) will lead to forward move in temporal feature extraction as well as (b) in global context modeling [84,85,86]. These state-ofthe-art approaches that have reoriented NLP are now being explored for log analysis, threat intelligence, and malware classification.

Last but not least, the requirement of dynamic configuration and self-optimization for IDS has driven the emergence of closed-loop systems. These systems autonomously adjust model thresholds, train modules, and redistribute resources in accordance with environmental feedback [87]–[90]. These self-adaptive structures are required to provide resilience in highly distributed and dynamic cyber spaces.

3. Methodology

3.1. Framework Overview

We will design a solution MLECOF which as an end to end solution that scales horizontally, is adaptive and intelligent in defense against sophisticated-by-design cyberattacks (Figure 1). Andreas Dieckow It is a collection of three major software modules, each of them looking at specific steps in the cybersecurity workflow. Specifically, the DIPM, the TDCE, and the RORU.

The DIPM is the entry point of the framework. It is designed to be used as a passive data extractor for network traffic & system log data, as well as the appropriate analysis tools such as (Ldap and CSV format output). DIPM ensures that MLECOF can be applied in different operational contexts (e.g., data centers, cloud infrastructures, or edge networks) by having the ability to ingest data from various sources like PCAPs, Netflow records, or system event logs.

Then the preprocessed data is submitted to the TDCE by which the key threat analysis is performed here. In this system, unsupervised learning and supervised learning models are combined together for comprehensive threat verification. The first reason is that it uses anomaly detection algorithms to separate the suspicious traffic from the benign one. Then, the filtered pruned data is designed to put into the classifiers with high accuracy to classify the attacks such as DOS, probe, R2L, U2R and botnet to monitor.

Finally, the RORU dynamically adjusts resource settings and security values to ensure that an exemplary system performance and security are maintained. Evolutionary optimization makes a trade-offs among computer efficiency and detection precision and power consumption. By reusing these building blocks of MLECOF, emphasises a single end-to-end security pipeline which handles both threat intelligence and autonomous adaptation.

3.2. Data Preprocessing

Data pre-processing is the footing the MLECOF pipeline is established upon, quality of the input data set determines how well (both performance-wise and efficiency-wise) downstream ML modules will work. The framework is heavily based on two popular cybersecurity datasets, CICIDS2017, and UNSW-NB15, which are available in the public domain. We choose these datasets because of the diversity of attack vectors, the complexity of features, and the reality of traffic.

The first step of pre-processing is imputation of missing values: we identify where the dataset has nulls, or out of range values, and we address them. Such irregularities are common in raw logs, and can induce extremely large bias if uncorrected. For the missing value, the it fills (by mean/mode for numerical/categorical feature respectively) or drops (if the number of them is big enough) based on which feature type it is.

And then we have feature normalization to ensure the different data are with the same scale, which is very important for distance-based learning algorithms and gradient-based optimizers. There are two types of normalization methods: Z-score normalization which normalizes features zero mean and unit variance and Min-Max scaling which normalizes the columns to a range of [0,1]. These add a little bit in order to prevent the modeling bias to dominate the over population numerically.

One hot encoding is applied to categorical variables such as protocol types and service names. This processed the categorical attributes into binary vectors without destroying the nominal relationship in the input data, which made it suitable for a machine learning model and it takes numerical input. For instance, protocol such as TCP, UDP, ICMP, etc. are mapped to mutually exclusive binary columns.

Since the feature space is extremly high dimensional, particularly the statistical ones over CICIDS2017 dataset, the final step of the pre-processing is the dimensionality reduction via principal component analysis (PCA). PCA map data onto the axes of the greatest variance and orthogonality to remove noize and to avoid age effects and correlation among features. The number of kept components is calculated by eigenvalue thresholding and cross validation for removing overfitting of the model.

3.3. Threat Detection Module

The central processing unit in MLECOF is its TDCE. For this purpose, this model combines unsupervised and supervised learning techniques for an effective and efficient detection of potential threats. The motive behind this hybrid solution is to reduce the benign requests served in order to save on computational resources, and enforce more accurate classification models in determining what type of malicious behaviour it actually represents.

The firststage of the TDCE is an unsupervised anomaly detection filter using Autoencoders. Autoencoders are the name given to neural networks that are trained to reconstruct their input. During training only normal traffic is used. Then at the time of inference, inputs that have high reconstruction error are considered as anomalies. This allows hot drop of high risk traffic without the reliance on labelled data – A massive boon for truly dynamic landscapes that are continuing to evolve over time, and where new (or adapted) threats/traffic can end up unclassified for a time.

Samples flagged to be potentially malicious by the anomaly filter are sent to the supervised classifiers. Three main threat classification algorithms are integrated: Random Forest, XGBoost and Convolutional Neural Networks (CNN). We choose the robust and interpretable model of Random Forest. As tree models, tree interpreters can assist cyber analysts in understanding the importance of features and the decision rules learned.

The XGBoost classifier: as a type of gradient boosting decision tree performs better than other like models as scikit-learn to make prediction and abstract features, especially in the circumstance of imbalanced data as UNSW-NB15. Continuing and categorical inputs are handled in a very natural manner, with regularisation incorporated to prevent over-fitting. Additionally, complex properties of training data, such as packet byte sequences or logs presented in form of time series, are fed into CNNs. CNNs are particularly good at detecting stealthy or polymorphic attacks, since they learn spatial structure of data within the input data.

Model performance is evaluated in terms of the following metrics: accuracy, precision, recall, F1-score, and ROC-AUC. # # Accuracy is the gross measure of rightness but precision and recall inform us as we trade-off false positives/false negatives. Thus, F1-score unites these two measures, and ROC-AUC focuses on classifier performance across thresholds in general. These evaluation measures are averaged over k-fold cross-validation and test test sets for the robustness and reproducibility of the results.

3.4. Optimization Engine

Asides detection an add-on of MLECOF to provide with its RORU model also autonomous system resilience and performance improvement. In general, security adds more computation overhead and may impair the availability and/or throughput of a system. RORU addresses these problems by applying MOGA to optimize the operation configuration and resource allocation at runtime.

MOGA is a biologically inspired optimization technique which uses a biologic analogy to arrange compromises among conflicting objectives. These are to save resources ( such as CPU, memory), maximize detection probability, and minimize response time in MLECOF. It includes thresholds for IDS configuration, cut-points for model confidence, and tactics for VM assignment, as well as as network segment rules.

One major application is the load balancing among virtual machines. Resource optimization and reorganization: RORU first examines the current resource usage, as well as the distribution of attack traffic, and reassigns the work of the IDS from the underutilized nodes to prevent overload or bottleneck. Another use is in the thresholding of the IDS parts. For example, confidence levels are dynamically configured for anomaly detectors and classifiers depending on the monitored traffic patterns and system workload.

The third use case is energy aware deployment for edge networks with limited power requirements. The middleware saves energy by shutting down some critical parts of it and by applying the lightweight components in the low critical environments. RORU also periodically retrains R with the detection output and system performance, and an adaptive loop is formed for self-improvement.

Accordingly, with the combination of real-time optimization and the machine learning-based detection, MLECOF can keep a safe-effective system posture able to face the variation of the threat landscapes as well as of the system loads.

4. Experimental Setup

4.1. Dataset Details

To experimentally validate the effectiveness and generalization ability of the proposed MLECOF framework, two publicly available and common benchmark datasets in network security, i.e., CICIDS2017 were employed. These databases offer balanced blending of mechanism of latest and old network attacks which can approximate to scenarios and behaviors of real time working network environment.

The CICIDS2017 dataset, curated by the CIC is a dataset generated from simulated network traffic and has about 2.8 million samples and 78 features - It comprises a diverse range of attack types like DoS, DDoS, Botnet, Brute Force, Intrusion, Web attacks. We have tried to balance the generated dataset by the axis of balanced records, which means benign and malicious records are treated equally. This trade-off promotes good training and low bias in the classifier performance, particularly for precision and recall.Together, they offer a diverse and challenging benchmark dataset to assess MLECOF's capability of detection and optimization in multiple sorts of cyber-threat scenario.

4.2. Training Environment

We conducted all experiments on high-performance clusters for training and testing of ML/Opt models. The system was an Intel Core i9, 64 GB RAM, NVIDIA RTX 3090 GPU 24 GB VRAM. This framework is important for training deep learning models (like the CNNs), approximate execution of iterative optimization, and skeletons used in the Resource Optimization and Response Unit (RORU).

Software And Tools Python was selected as the primary programming language because of the existence of a great deal of libraries to machine learning, data preprocessed and analasis of networks. Traditional models like Random Forest and XGBoost were implemented using scikit-learn, whereas deep learning approaches like Autoencoders and CNNs were implemented using TensorFlow. In this study, the NSGA-II multi-objective optimization algorithm on RORU is implemented via the DEAP (Distributed Evolutionary Algorithms in Python). DEAP provided the ability to define evolutionary operators, fitness functions and genetic representations for our purpose.

This combination of robust hardware and working software-stack dependencies facilitated the reusable, scalable and customizable MLECOF framework in simulation and potential deployment contexts.

5. Results and Discussion

5.1. Detection Performance

The performance (computational efficiency and detection accuracy) of MLECOF was tested on three major machine learning classifiers such as Random Forest, XGBoost, and CNN. Five well-known performance metrics (accuracy, precision, recall, F1-score and ROC-AUC) were used to evaluate the performance.

As reported in Table 1, the performance of the CNN model was better than the other models for all metrics, indicating the ability to capture complex patterns in the data naturally. Meanwhile, XGBoost also showed excellent performance at its accuracy-interpretabilty trade-off.

This Figure 2 illustrate Accuracy, Precision, Recall, F1-score, ROC-AUC of models. Each set of bars, one for each metric, show that CNN is superior and XGBoost superior over Random Forest.

Our visual results verify that, although all three models show effectiveness, CNN is more proper in the case of high-dimensional data with spatial or sequential relationships, and XGBoost yields a good tradeoff between speeding and accuracy.

5.2. Optimization Performance

In addition to the detection itself, MLECOF RORU contributes to the optimal system operation by minimizing overhead and the response time and energy of the system. This module was evaluated through simulation under different loads and edge-computing constraints.

Table 2. Performance Gains by RORU.

Metric	Baseline (No RORU)	With RORU	Improvement
Average CPU Usage	68%	53%	22%↓
IDS Response Latency (ms)	210	151	28%↓
Energy Usage in Edge (kWh/day)	4.2	3.49	17%↓

Table 2. Performance Gains by RORU.

Metric	Baseline (No RORU)	With RORU	Improvement
Average CPU Usage	68%	53%	22%↓
IDS Response Latency (ms)	210	151	28%↓
Energy Usage in Edge (kWh/day)	4.2	3.49	17%↓

It has been shown that the multi-objective genetic algorithms (MOGA) have significantly enhanced the performance of this system. The intelligent distribution of the workload causes a reduction of CPU by 22%. IDS latency was cut by 28% and threat mitigation was achieved much more quickly.

These plots (Figure 3) show the real-time advantages of adaptive resource reallocation, in particular in resource-constrained setups such as cloud edge systems or low-power devices.

5.3. Ablation Study

To verify the effectiveness of each of the MLECOF modules, we conducted an ablation study. This included manually disabling the anomaly detection pre-filter and resource optimization unit, and monitoring differences in resource utilization and response times.

The removal of the anomaly detection layer caused on average an 11% higher total resources used by the system, due to the fact that benign traffic had to be run through complex classifiers without any possibility to avoid this process (Table 3). Similarly, the removal of the optimization module produced a 9.5% raise in response latency, and the response time passed over the critical value, which then slowed down the response of the system in waves of attacks.

This Figure 4 visualization serves to emphasize the modular significance of each component in the framework re-affirming the architectural choice to keep separate but dependent modules for detection and optimization.

6. Conclusions

In this work, we introduce MLECOF as an efficient and flexible cognitive cybersecurity framework to meet the two goals of accurate intrusion detection and system tuning. By performing intense data preprocessing, the hybrid model and the dynamic optimization scheme, MLECOF obtains the best defense accuracy (CNN: 98.7%, XGBoost: 98.1%) and can also adapt the resource allocation and the system behavior using its RORU module. Experiment with CICIDS2017/UNSW-NB15 The experiment with CICIDS2017 shows that the unified AD pre-filter reduces computational complexity under a different number of anomalies generating, and the optimization engine has comprehensive performance improvement in CPU usage, response time, and energy efficiency in general. Ablation study also ensures the cooperation and necessity of each part of the system. Because of its scalable style and insensitivity to high-dimensional imbalanced and diverse cyber threats, MLECOF has potential applications in real-time mission-critical systems such as data centers, IoT edge devices or cloud networks. One future research direction is to explore the integration of PSDS with both real time threat intelligence feed and continuous learning to better adapt in the evolving threat landscapes.