Security Evaluation of Classical Machine Learning Models Under Poisoning, Evasion, and Model Extraction Attacks

1. Introduction

Traditional machine learning classifiers remain widely used in practical systems where computational efficiency, interpretability, and deployment constraints are important considerations. Despite the increasing dominance of deep learning approaches, models such as Logistic Regression, Support Vector Machines, and Random Forests continue to support applications in domains where transparency and deployment constraints are critical.

However, these models are exposed to multiple adversarial threats that affect different stages of the learning pipeline. Poisoning manipulates training data to influence the learned model, evasion modifies inputs during inference to cause incorrect predictions, and model extraction exploits prediction interfaces to reconstruct model behavior. Although these threats have been extensively studied individually, their combined impact across classical ML architectures remains insufficiently characterized.

Existing research commonly focuses on a specific attack category or a particular model family, making it difficult to determine how architectural characteristics influence adversarial vulnerability. In particular, linear classifiers and ensemble models rely on fundamentally different decision mechanisms: LR and Linear SVM learn global decision boundaries, whereas RF relies on aggregated nonlinear decision rules. These structural differences may produce different vulnerability profiles depending on the attack scenario.

Therefore, this study investigates the adversarial vulnerability profiles of classical ML models under a unified evaluation framework. The objective is to analyze how model architecture influences susceptibility to poisoning, evasion, and extraction attacks across multiple data domains.

The main contributions of this study are:

• Unified adversarial evaluation framework. We develop an experimental framework that evaluates classical machine learning models under training-time poisoning, inference-time evasion, and black-box model extraction attacks using consistent datasets, metrics, and evaluation procedures.
• Comparative analysis of architectural vulnerabilities. We analyze how differences between linear classifiers and ensemble-based models influence vulnerability across different adversarial settings.
• Cross-domain security evaluation. We evaluate adversarial behavior across image-based and text-based classification tasks to examine how dataset characteristics influence attack effectiveness.
• Practical security insights. We investigate the relationship between attack strategy, model architecture, and vulnerability patterns in order to provide insights into the suitability of classical machine learning models under different security conditions.

The remainder of this paper is organized as follows. Section 2 reviews related work on adversarial machine learning, including poisoning, evasion, and model extraction attacks. Section 3 presents the conceptual foundation linking model architecture and adversarial vulnerability. Section 4 describes the experimental methodology, including datasets, models, attack configurations, and evaluation metrics. Section 5 presents the experimental results. Section 6 discusses the security implications, limitations, and future research directions. Finally, Section 7 concludes the paper.

2. Related Work

2.1. Adversarial Machine Learning and Security Threats

The increasing deployment of machine learning systems in real-world applications has motivated extensive research into adversarial machine learning, where an attacker attempts to compromise model reliability, integrity, or confidentiality. Existing studies commonly categorize adversarial threats into three major classes: poisoning attacks, evasion attacks, and model extraction attacks.

Previous studies have demonstrated that malicious data manipulation can significantly degrade model performance, particularly when attackers identify influential samples or exploit structural vulnerabilities in the optimization pipeline. In a foundational work, Biggio et al. [1] demonstrated that an adversary could exploit mathematical optimization properties to construct optimal training-time poisoning instances specifically targeting the hyperplanes of Support Vector Machines. Yerlikaya and Bahtiyar [2] expanded on this by providing a comprehensive analysis of data poisoning attacks against machine learning algorithms, showing that different learning architectures exhibit different sensitivity profiles depending on the attack strategy. Recent studies have also investigated poisoning threats against ensemble models, demonstrating that Random Forests may provide increased resilience against certain untargeted attacks while remaining vulnerable to targeted manipulation strategies [3].

Evasion attacks represent another major category of adversarial threats, where the attacker modifies input samples during inference to cause incorrect predictions without altering the model itself. Biggio et al. [4] initially formalized gradient-based test-time evasion against classical learning systems such as linear classifiers and SVMs. Following this, the seminal work of Goodfellow et al. [5] introduced gradient-based adversarial examples and demonstrated the vulnerability of neural models to carefully crafted perturbations. Although most adversarial robustness research has focused on deep learning, classical machine learning models can also be affected by adversarial perturbations. Recent research has investigated the detection and mitigation of evasion attacks across different model families, highlighting the importance of evaluating robustness beyond neural architectures [6].

Model extraction attacks represent a different security challenge. Tramèr et al. [7] demonstrated that prediction APIs can leak sufficient information for reconstructing machine learning models, establishing model stealing as an important privacy and security concern. To formalize the downstream threat of model extraction, Papernot et al. [8] demonstrated that an adversary could systematically extract black-box models using surrogate classifiers (such as Logistic Regression) to launch high-transferability evasion attacks offline. Subsequent research showed that high-fidelity extraction can be achieved even when direct access to model parameters is unavailable, demonstrating that prediction interfaces may reveal substantial information about model behavior [9]. More recent studies have explored query-efficient extraction strategies, including active learning-based approaches and data-free extraction methods [10,11,12]. These studies show that the difficulty of extraction depends not only on the number of queries but also on the structure of the target model and the information available to the attacker.

2.2. Security Properties of Classical Machine Learning Models

Logistic Regression, Support Vector Machines, and Random Forests are frequently applied in domains where transparency and computational constraints are important.

Linear models have well-defined decision structures, making them attractive for analysis but potentially easier to approximate through query-based approaches. Previous studies have demonstrated that model structure and decision behavior influence vulnerability to adversarial manipulation, particularly in the contexts of adversarial examples, robustness evaluation, and model extraction [5,7,13]. In contrast, ensemble-based models such as Random Forests construct predictions through collections of decision rules, which may introduce different robustness characteristics compared with linear classifiers [14,15].

Research on Random Forest robustness has produced mixed findings. Some studies suggest that ensemble aggregation can improve resilience against certain forms of data corruption [3], while other works demonstrate that targeted attacks can still compromise ensemble behavior. This indicates that robustness is highly dependent on the attack formulation and threat model rather than being an inherent property of model complexity.

2.3. Adaptive Querying and Model Extraction

A central challenge in model extraction is obtaining high fidelity while minimizing the number of queries issued to the target model. Random querying provides a simple baseline but may require many interactions to approximate complex decision functions. To improve efficiency, several works have investigated adaptive and active-learning-based extraction strategies. Active learning provides a theoretical framework for selecting informative samples based on uncertainty and expected information gain, reducing labeling or querying requirements compared with random sampling [16].

Recent studies have explored query-efficient extraction methods using public data, uncertainty-based sampling, and active learning frameworks [12,17]. These approaches aim to prioritize informative samples near uncertain regions of the decision space, reducing the number of required queries. However, the effectiveness of adaptive querying may depend on the target architecture, dataset representation, and query constraints.

Although active extraction methods have been studied extensively for neural networks, less attention has been given to their behavior when applied to classical machine learning models, particularly non-linear ensemble models operating on sparse or high-dimensional representations. Understanding whether adaptive querying consistently improves extraction efficiency across different model structures remains an open empirical question.

2.4. Research Gap

Although previous studies have significantly contributed to the understanding of adversarial vulnerabilities in machine learning systems, most existing works investigate individual attack categories or focus on specific model families. In particular, poisoning, evasion, and model extraction attacks are commonly studied independently, making it difficult to determine how architectural characteristics influence vulnerability across multiple threat scenarios. A comparative evaluation framework that simultaneously analyzes classical machine learning models under different adversarial settings remains limited. Table 1 summarizes representative research directions.

To address these limitations, our study presents an empirical comparative evaluation framework for classical machine learning architectures under multiple adversarial threat models. Unlike previous studies that examine isolated attack mechanisms, this study analyzes the relationship between model structure, decision boundary characteristics, and vulnerability patterns across poisoning, evasion, and extraction scenarios.

3. Conceptual Foundation

The vulnerability of machine learning models is expected to depend on the interaction between model representation, decision structure, and attack mechanism. This section defines the theoretical assumptions guiding the experimental evaluation.

3.1. Decision Representation and Adversarial Sensitivity

Linear classifiers such as LR and Linear SVM rely on global decision boundaries determined by learned feature weights. Consequently, adversarial modifications affecting influential features may directly alter classification outcomes, particularly for samples located close to decision margins [5,18].

Random Forest models represent predictions through multiple decision paths generated by independent trees. This distributed decision mechanism may reduce sensitivity to some perturbations [14,15], but it can also introduce vulnerabilities when attackers identify influential regions of the ensemble structure.

3.2. Model Complexity and Extraction Difficulty

Model extraction attacks exploit access to model predictions to approximate the behavior of a target classifier [7]. The effectiveness of extraction depends on the amount of information revealed through queries and the complexity of the target decision function.

Models with simpler prediction structures may be easier to approximate because their input-output relationships can be learned using fewer observations. Linear models provide a relatively constrained representation, which may facilitate surrogate learning when sufficient prediction queries are available. In comparison, ensemble models such as Random Forests represent predictions through multiple local decision rules, potentially increasing the difficulty of reconstruction [9].

However, structural complexity alone does not guarantee extraction resistance because query strategy and data distribution strongly influence attack effectiveness [10].

3.3. Attack-Dependent Vulnerability of Classical Models

In this study, adversarial robustness is treated as an attack-dependent property rather than a fixed characteristic of a classifier. Different attack types target distinct components of the learning pipeline, and therefore may expose different structural weaknesses in a model.

Poisoning attacks affect the training process by modifying the data used to learn model parameters. Their impact depends on how strongly individual samples influence the learned decision function.

Evasion attacks operate at inference time by introducing perturbations to input samples without modifying the trained model. Their effectiveness is influenced by the geometry of the decision boundary and the local sensitivity of the classifier.

Extraction attacks target model confidentiality by leveraging prediction outputs to approximate the underlying decision function. Their success depends on the amount of information revealed through queries and the structure of the target model.

Based on these distinctions, the experiments in this study evaluate classical machine learning models under each attack category independently to characterize architecture-dependent vulnerability patterns.

4. Experimental Setup

This study evaluates the security behavior of three classical machine learning architectures under three adversarial scenarios: poisoning attacks during training, evasion attacks during inference, and model extraction attacks after deployment. The selected models represent different learning mechanisms: Logistic Regression as a linear probabilistic classifier, Linear SVM as a margin-based classifier, and Random Forest as a non-linear ensemble model.

All experiments follow the same evaluation protocol to enable direct comparison between architectures. Each model is trained using the original training partition, exposed to the corresponding attack scenario, and evaluated using the same test partition. Performance degradation or attack effectiveness is measured relative to the clean model performance.

The experimental workflow consists of four stages:

• training baseline models using unmodified datasets;
• applying adversarial manipulation according to the selected attack strategy;
• evaluating model performance after the attack;
• comparing vulnerability patterns across architectures;
• statistical significance analysis.

4.1. Datasets and Preprocessing

Experiments were conducted on three benchmark datasets representing different data modalities: MNIST, CIFAR-10, and AG News. MNIST and CIFAR-10 were used to evaluate image-based classification behavior, while AG News was selected to represent high-dimensional text classification. To maintain computational feasibility while preserving dataset diversity, the maximum number of training samples was limited to 15,000 using class-stratified sampling. The data pipeline implements the following preprocessing configurations:

• MNIST consists of 28 x 28 pixel grayscale handwritten digit vectors (d = 784 features). Pixel values are normalized to a uniform range of [0, 1];
• CIFAR-10 consists of 32 x 32 x 3 color image matrices flattened into continuous feature coordinates (d = 3,072 features). Features are cast to 32-bit floating-point arrays and scaled via minimum-maximum transformation to the range [0, 1];
• AG News represents a high-dimensional topic classification task structured across four distinct categorical classes. Text sequences are constructed by concatenating headline string records with body descriptions. The text data is vectorized into a continuous sparse representation using a term frequency-inverse document frequency (TF-IDF) transformation bounded at M = 5,000 maximum features alongside English stop-word filtering.

Following pipeline normalization, a global scaling pass is executed across all training instances using an empirical minimum-maximum estimator:

$$\overline{x}=\left(\mathrm{x}\text{-}\min \right({\mathrm{x}}_{\mathrm{train}}\left)\right)/\left(\max \right({\mathrm{x}}_{\mathrm{train}}\left)\text{-}\min \right({\mathrm{x}}_{\mathrm{train}}\left)\right)$$

To prevent data leakage, validation and test partitions are transformed using the parameters fitted exclusively on the clean training partition, followed by an element-wise clipping function to enforce a strict boundary condition of x_test ∈ [0.0, 1.0].

4.2. Target Estimator Architecture and Configuration

Three classical learning architectures chosen for their structural properties and prominence in production environments were evaluated: Logistic Regression, Linear Support Vector Machine and Random Forest. Estimators are configured with explicit parameter boundaries:

• Logistic Regression, structured as a multi-class multinomial estimator utilizing the Limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) optimization solver. Optimization tolerances are set to τ=10^(-3), the inverse regularization strength is bounded at C = 5.0, and the maximum iteration limit is fixed at 2,000;
• Linear Support Vector Machine instantiated via a linear support vector classifier minimizing a squared hinge-loss objective function under automated primal-dual selection. Optimization thresholds are maintained at to τ=10^(-3), the regularization parameter is fixed at C = 5.0, and the maximum iterations are bounded at 10,000. To handle downstream probability collection, the estimator is wrapped in a CalibratedClassifierCV meta-estimator utilizing a 3-fold cross-validation calibration split;
• Random Forest, configured as a non-linear structural ensemble comprising B = 100 independent decision estimators. Tree complexity is constrained with a maximum depth threshold of d_max = 10 to prevent unconstrained variance fitting.

4.3. Attack Implementation

4.3.1. Poisoning Attacks

Training-time robustness was evaluated using label poisoning attacks. Two poisoning strategies were considered:

• Random label flipping, where selected training labels are randomly modified;
• Confidence-based targeted poisoning, where high-confidence samples are selected using a surrogate model and assigned incorrect target labels.

The poisoning ratio was evaluated at multiple levels, including 1%, 5%, and 10% of the training data. The primary evaluation metric was the change in classification accuracy compared with the clean baseline.

4.3.2. Evasion Attacks

Inference-time robustness was evaluated using gradient-based perturbation attacks:

• Fast Gradient Sign Method (FGSM);
• Projected Gradient Descent (PGD).

Perturbations were generated by modifying input features within predefined perturbation budgets (ε). For Random Forest, which does not provide differentiable gradients, a surrogate Logistic Regression model was used to generate transferable perturbations.

Robust accuracy after perturbation was used as the main evaluation metric.

4.3.3. Model Extraction Attacks

Model confidentiality was evaluated through black-box extraction experiments. The attacker was assumed to have access only to model predictions and an unlabeled query pool.

Two query strategies were evaluated:

• random sampling;
• entropy-based adaptive sampling.

Extraction effectiveness was measured using fidelity between the extracted surrogate model and the target model across increasing query budgets: Q ∈ {1,000; 5,000; 10,000}.

Higher fidelity indicates a more successful reconstruction.

4.4. Statistical Validation and Metrics

Experimental results were averaged across multiple initialization seeds S ∈ {42, 43, 44}. Confidence intervals were calculated to quantify variability between runs.

Pairwise model comparisons were performed using two-tailed paired Student’s t-tests. Due to the limited number of experimental repetitions, statistical results are interpreted as indicators of observed differences rather than definitive population-level conclusions.

4.5. Reproducibility Settings

The main experimental parameters are summarized below:

• TF-IDF maximum features: 5,000
• Surrogate training fraction: 30%
• Calibration cross-validation folds: 3
• Query budgets: 1,000, 5,000, 10,000
• Poisoning rates: 1%, 5%, 10%

This experimental design provides a consistent basis for comparing the vulnerability profiles of classical machine learning models under different adversarial conditions.

4.6. Implementation Details and Computational Environment

All experiments were executed on a localized workstation to evaluate the computational efficiency of the attacks. The environment consisted of a MacBook Pro equipped with an Apple M3 Chip and 16 GB of Unified Memory. The software environment was managed under macOS Tahoe (Version 26.2) using Python 3.12, scikit-learn 1.5, and the Adversarial Robustness Toolbox (ART) for attack implementations.

5. Empirical Results

This section presents the empirical results of the security evaluation across the poisoning, evasion, and model extraction scenarios. The collected data suggest distinct, architecture-dependent trends when the target models are evaluated under adversarial conditions.

5.1. Baseline Performance

Prior to executing adversarial evaluations, point-of-reference classification benchmarks were established by training and testing all target models on clean, non-perturbed data splits. The aggregated accuracy and Area Under the Receiver Operating Characteristic curve (ROC-AUC) metrics, alongside their corresponding two-tailed Student's t-distribution 95% confidence intervals (CI95, ν=2), are provided in Supplementary Table S2, while a high-level summary of these baseline metrics is provided in Table 2.

The baseline metrics suggest strong discrimination capabilities across the MNIST and AG News domains, indicating that the target estimators are well-fitted to these underlying data distributions prior to attack deployment. For instance, on the MNIST dataset, the non-linear Random Forest ensemble achieves a peak baseline accuracy of 94.29% ± 0.10% and a near-ideal ROC-AUC of 0.997. Similarly, on the text-based AG News dataset, the log-linear decision surfaces of Logistic Regression yield a robust baseline accuracy of 88.40% ± 0.13% and an ROC-AUC of 0.973.

Conversely, a lower performance profile is observed across all three architectures when processing the CIFAR-10 data modality, with baseline accuracies bounded between 32.46% and 41.06%. This systemic performance limitation is consistent with the known mathematical boundaries of traditional algorithms, which require raw three-channel color pixel matrices to be flattened into single-dimensional continuous feature vectors (d = 3,072). Because these estimators lack the specialized spatial hierarchy awareness and translation-invariant inductive biases inherent to convolutional or localized deep learning architectures, they may struggle to map complex visual relationships within uncooperative feature spaces. This indicates that while these classical models are effective for structured or low-complexity modalities, their performance profiles are constrained by high-dimensional structural transitions.

5.2. Poisoning Attack Results

To evaluate training-time robustness, two poisoning strategies were investigated: random label flipping and confidence-based targeted poisoning. These attacks represent different levels of adversarial knowledge, ranging from unspecific corruption of the training distribution to targeted manipulation of influential samples. Observed performance trends across models, datasets, and poisoning strategies at the maximum configuration are presented in Table 3 and graphically represented in Figure 1 and Figure 2.

The results suggest that the impact of poisoning is influenced by both the attack strategy and the underlying model architecture. Under random label flipping, Random Forest generally exhibits the lowest performance degradation across the evaluated datasets. For example, at a poisoning rate of 10%, Random Forest shows a limited accuracy decrease of 0.21% on MNIST and CIFAR-10, while Logistic Regression and Linear SVM exhibit larger drops. This indicates that ensemble aggregation can mitigate the effect of randomly introduced label noise by distributing the influence of corrupted samples across multiple decision paths.

However, this apparent resilience appears to lessen when evaluating targeted poisoning scenarios. By selecting high-confidence samples and altering their labels, the attack strategy tends to target samples that contribute meaningfully to the learned decision function. Under this scenario, the difference between model architectures becomes less predictable. The most notable degradation within this setting is observed for Random Forest on the AG News dataset, where accuracy decreases by 13.68% at the 10% poisoning rate. This suggests that although ensemble models may tolerate random corruption, targeted manipulation of informative samples can influence their learned decision structures.

Linear models show a more consistent degradation pattern across poisoning configuration. Logistic Regression and Linear SVM show higher sensitivity within the text classification domain, where variations in high-dimensional TF-IDF feature distributions can alter the learned separating hyperplanes. On the AG News dataset, random poisoning is associated with accuracy reductions of 8.01% for Logistic Regression and 7.63% for Linear SVM, highlighting the susceptibility of linear decision functions to training distribution changes.

Overall, the results indicate that poisoning robustness is attack-dependent rather than representing an immutable property of a model architecture. Random Forest shows higher resistance against unstructured label noise within these parameters, while targeted attacks appear capable of exploiting structural characteristics in the learned representation. Therefore, evaluating only random poisoning scenarios may underestimate the vulnerability of classical classifiers.

Figure 2. Accuracy degradation under increasing confidence-based poisoning attacks.

Figure 2. Accuracy degradation under increasing confidence-based poisoning attacks.

5.3. Evasion Attack Results

Inference-time robustness was evaluated using both single-step (FGSM) and iterative multi-step (PGD) continuous perturbations to identify performance drop-off thresholds across image-based domains. Because Logistic Regression minimizes a smooth multinomial cross-entropy loss function (L_CE) and Linear SVM optimizes a piece-wise squared hinge loss (L_Hinge), their reactions to local gradient-based steps differ structurally. Furthermore, the non-differentiable step boundaries of Random Forest were probed via a surrogate-gradient transfer approach using a separately trained Logistic Regression model.

The aggregated classification accuracies collected across the MNIST and CIFAR-10 data partitions are summarized in Table 4 and Table 5 and graphically represented in Figure 3 and Figure 4.

The empirical trends point toward a sharp, non-linear decline in robust accuracy across all traditional models as the perturbation budget increases. On Logistic Regression configurations, iterative multi-step optimization consistently reduces performance slightly more than single-step methods. For example, at ϵ = 0.05 on MNIST, robust accuracy drops to 31.92% ± 2.77% under PGD compared to 34.82% ± 2.84% under FGSM. This behavior is consistent with standard optimization trends: the smooth, continuously differentiable cross-entropy loss function appears to allow iterative gradient steps to traverse the classification space effectively.

Interestingly, our evaluations reveal an optimization paradox where the multi-step PGD attack yields marginally higher robust accuracy scores than the single-step FGSM on Linear SVM models at intermediate budgets (e.g., at ϵ = 0.02, MNIST accuracy tracks at 55.60% ± 1.25% under PGD versus 54.28% ± 1.15% under FGSM). Rather than a data anomaly, this behavior suggests a gradient-stalling phenomenon often associated with the piece-wise linear configurations of squared hinge-loss functions. Smaller iterative steps (α = ϵ/4) cause perturbed samples to cross the decision margin early in the optimization loop. Once inside the alternate class region, local margin gradients can zero out or oscillate, preventing the attack from optimizing further. Single-step FGSM avoids this stalling effect by using its full perturbation budget at once, vaulting samples deep past the decision hyperplanes in a single mathematical step.

For the Random Forest model, the robust accuracy scores remain nearly identical under both single-step and iterative attacks across all evaluation steps (e.g., hovering around 40.31% under FGSM and 40.32% under PGD at ϵ = 0.01 on CIFAR-10). This stability highlights the structural transferability gap between linear models and tree-based ensembles. Because the forest relies on non-differentiable, axis-aligned step functions, calculating continuous surrogate gradients across multiple iterations does not appear to yield a cumulative advantage over a single large step. Within this setting, the attack evaluates the raw transfer boundary directly.

5.4. Model Extraction Results

The final vector of our security assessment analyzes the model extraction boundary conditions, tracking how vulnerable different classifier geometries are too hard-label replication queries. By using the unlabeled query pool x_train, we evaluated the replication process under two distinct sampling heuristics - Random and Entropy-Based Adaptive sampling - across query capacities of Q ∈ {1,000; 5,000; 10,000}.

The empirical target extraction fidelities (F), along with their corresponding two-tailed Student's t-distribution 95% confidence intervals (CI95, ν=2), are separated into Table 6 for the passive baseline distribution and Table 7 for the active selection framework. For an easier overview, the visual trends are displayed in Figure 5 and Figure 6.

The empirical results suggest a clear contrast in extraction behaviors between linear hyperplanes and tree ensembles. For log-linear models (LR and SVM), increasing the query volume tracks with a steady increase in extraction fidelity across both strategies. However, when transitioning from random sampling to entropy-based adaptive querying at Q = 10,000 on the AG News dataset, the extraction fidelity for Logistic Regression increases from 93.72% up to a peak of 95.07%. These results suggest that active sampling may effectively guide the surrogate model toward the target's decision boundary, indicating that it can duplicate continuous separating hyperplanes using a fraction of the available data space.

Conversely, the Random Forest architecture exhibits a distinct complexity buffer under active extraction configurations. As shown in Table 6, scaling the passive query volume from 1,000 to 10,000 on the AG News dataset allows the surrogate forest to improve its fidelity from 68.44% to 73.39%. However, when active adaptive querying is applied at Q = 10,000 (Table 7), the Random Forest's extraction fidelity is limited to 52.34%, trailing the random sampling strategy by 21.05%.

This variance in performance highlights a potential limitation of active sampling frameworks: entropy-driven heuristics prioritize queries located near low-confidence regions. While this strategy appears suited for mapping smooth, continuous linear hyperplanes, it may be systematically misled by the irregular, step-like decision boundaries characteristic of tree ensembles. The active query selector samples heavily around unrepresentative local leaf-node corners, which is consistent with hypothesis that the ensemble's extraction resistance behaves as a structurally driven trait under active adversarial exploration.

5.5. Comparative Analysis

To synthesize the empirical findings across training-time and inference-time environments, a comprehensive comparative analysis is presented to map out the trade-offs of each architecture. Figure 7 and Table 8 cross-examine the security posture of the models across all three threat vectors, suggesting an architectural trade-off where structural traits that secure an estimator against one class of attack may introduce vulnerabilities to another.

5.5.1. Poisoning Vulnerability

Linear models, particularly Logistic Regression, exhibit sensitivity to training set manipulation within these parameters, which is consistent with their reliance on a global separating hyperplane. Even a 10% poisoning rate can skew these boundaries, leading to notable accuracy losses, such as the 8.01% drop observed for LR on AG News dataset. Random Forest initially presents as more robust to random noise, showing a limited accuracy decrease of 0.21% at a 10% rate on MNIST. However, this resistance appears to be conditional; targeted confidence-based poisoning resulted in a 13.68% performance decrease on the AG News dataset, indicating that ensemble model vulnerabilities can be highly attack-dependent.

5.5.2. Evasion Vulnerability

Evasion remains a critical threat across all evaluated architectures. Linear models show a susceptibility to gradient-based shifts, with the Linear SVM dropping to 10.65% accuracy on MNIST under the ε = 0.05 perturbation threshold. On the CIFAR-10 dataset, this performance drop-off is more pronounced, with the estimators approaching a breakdown point at minimal perturbation levels. Although Random Forest demonstrates a statistically higher robust accuracy under transfer conditions (retaining 28.75% accuracy at ε = 0.01 on MNIST), this pattern is largely associated with the transferability gap, which reflects the structural misalignment between the linear surrogate model and the non-differentiable decision paths of the target ensemble.

5.5.3. Extraction Vulnerability

The threat of model extraction appears to be strongly influenced by the geometric complexity of the decision boundary. LR and SVM show high susceptibility to duplication within this framework, achieving over 90% replication fidelity under limited query budgets. Random Forest provides a larger complexity buffer, requiring higher query volumes to achieve comparable fidelity in high-dimensional feature fields such as CIFAR-10. Our analysis of adaptive querying further suggests that while active heuristics can accelerate extraction at higher query budgets, they may be less effective at lower budgets by focusing on unrepresentative boundary samples.

In summary, while traditional models are often preferred for their computational efficiency and transparency, these same traits facilitate adversarial exploitation. Our results emphasize that within the scope of this evaluation, increasing model complexity does not necessarily eliminate risk, but can shift the vulnerability from direct exploitation to more sophisticated, transferability-based attacks.

Table 8. Comprehensive security synthesis and empirical trade-off matrix.

Architectural feature	LR	Linear SVM	RF
Decision Surface Geometry	Continuous, smooth linear hyperplanes	Continuous, piece-wise linear hyperplanes	Discontinuous, orthogonal step-boundaries
Optimization Objective	Unbounded Multinomial Cross-Entropy	Bounded Squared Hinge-Loss	Discrete Gini Impurity Heuristics
Poisoning Profile ($\mathbf{10}\mathbf{\%}$)	High vulnerability to feature distribution shifts; steady log-linear decay.	High vulnerability to edge corruption; susceptible to support vector skewing.	Outstanding resistance to unstructured noise; extreme fragility under targeted attacks.
Evasion Profile ($\mathit{ϵ}=\mathbf{0.02}$)	Moderate robust accuracy; continuous loss gradient provides stable attack directions.	Lower robust accuracy; susceptible to single-step margin-collapse (gradient stalling).	High transferability buffer; non-differentiable thresholds break localized analytical attacks.
Extraction Profile ($\mathit{Q}=\mathbf{10}\mathrm{k}$)	Critical vulnerability; smooth decision planes are completely exposed to active sampling.	Critical vulnerability; active query heuristics quickly map explicit linear separating margins.	Robust resistance buffer; active adaptive query selections are misled by jagged surfaces.

Table 8. Comprehensive security synthesis and empirical trade-off matrix.

Architectural feature	LR	Linear SVM	RF
Decision Surface Geometry	Continuous, smooth linear hyperplanes	Continuous, piece-wise linear hyperplanes	Discontinuous, orthogonal step-boundaries
Optimization Objective	Unbounded Multinomial Cross-Entropy	Bounded Squared Hinge-Loss	Discrete Gini Impurity Heuristics
Poisoning Profile ($\mathbf{10}\mathbf{\%}$)	High vulnerability to feature distribution shifts; steady log-linear decay.	High vulnerability to edge corruption; susceptible to support vector skewing.	Outstanding resistance to unstructured noise; extreme fragility under targeted attacks.
Evasion Profile ($\mathit{ϵ}=\mathbf{0.02}$)	Moderate robust accuracy; continuous loss gradient provides stable attack directions.	Lower robust accuracy; susceptible to single-step margin-collapse (gradient stalling).	High transferability buffer; non-differentiable thresholds break localized analytical attacks.
Extraction Profile ($\mathit{Q}=\mathbf{10}\mathrm{k}$)	Critical vulnerability; smooth decision planes are completely exposed to active sampling.	Critical vulnerability; active query heuristics quickly map explicit linear separating margins.	Robust resistance buffer; active adaptive query selections are misled by jagged surfaces.

5.5.4. Deconstructing the Linear vs. Ensemble Structural Trade-Offs

The experimental tracks highlight a security contradiction when choosing between continuous linear models (LR and SVM) and discontinuous ensemble estimators (RF). This interaction is best analyzed by looking at how their internal geometries respond to adversarial pressure:

• The smooth hyperplane vulnerability: Logistic Regression and Linear SVM suffer from a shared susceptibility to black-box model extraction. Because their decision boundaries consist of continuous, flat hyperplanes, an adversary can efficiently map their global orientation using minimal boundary query samples. This structural exposure is visible under active adaptive sampling, where extraction fidelity values climb to 96.22% for LR and 93.84% for SVM on the MNIST domain. However, this same spatial continuity provides an optimization benefit during inference-time evasion. Under multi-step PGD pressure at $ϵ=0.02$, Logistic Regression retains a robust accuracy of 77.32% ± 1.17% on MNIST, outperforming the Random Forest's transfer defense. The continuous, smooth landscape allows the model's loss landscape to decay gracefully rather than collapsing abruptly.
• The jagged ensemble complexity buffer: The Random Forest resists active model extraction attacks in these settings, which is consistent with its complex decision surface. Its boundary is composed of numerous discontinuous, axis-aligned orthogonal step functions, creating a jagged surface. When an entropy-based active query selector attempts to map this space, it samples heavily around unrepresentative local leaf-node corners, which can misdirect the attack loop and limit extraction fidelity to 52.34% ± 1.33% on the AG News dataset. However, this irregular geometry introduces vulnerabilities during training-time poisoning. When targeted confidence-based label flipping is deployed, the Random Forest's accuracy on AG News collapses by 13.68%, dropping from a clean baseline 76.81% to 63.13% ± 2.05%. Because tree splits rely on categorical partitions, corrupting high-confidence training targets may alter early parent-node splits, causing downstream subtrees to misclassify instances.

5.5.5. Unified Security Recommendations for Deployment

These empirical findings suggest that within the scope of the evaluated implementations and parameter configurations, no single model architecture provides a secure, all-around defense against modern adversarial threat vectors. Security engineers must select and configure estimators based on the specific threat constraints of their deployment environments:

• Environments dominated by extraction and reverse-engineering risks: If a model interface is exposed via public web APIs where intellectual property theft or surrogate extraction is the primary threat vector, non-linear tree-based ensembles (RF) should be prioritized over linear hyperplanes. The irregular decision surfaces of ensembles provide a natural buffer that resists adaptive sampling attacks.
• Environments vulnerable to training corruption and data ingestion risks: For applications that continuously retrain on user-generated inputs or open-source streams where label-tampering and data poisoning are prominent risks, linear models or robust hyperplanes are structurally preferable. Their global regularized objectives minimize the impact of targeted shifts, preventing the substantial degradation that compromises tree-based ensembles.
• Environments exposed to real-time inference threats: When models face automated, real-time input manipulation at inference time, continuous formulations utilizing cross-entropy objectives should be selected. Smooth loss surfaces handle multi-step local feature shifts more predictably, avoiding the sudden margin collapses and gradient-stalling vulnerabilities that compromise piece-wise linear hinge margins.

5.6. Statistical Significance Analysis

To evaluate whether the observed differences between model architectures represent consistent performance variations rather than random experimental fluctuations, pairwise statistical comparisons were conducted across the three adversarial scenarios. Two-tailed paired Student’s t-tests were applied to the evaluated model pairs using the results obtained from the experimental repetitions. The complete statistical results, including t-statistics, p-values, and significance indicators, are provided in Supplementary Table S5.

For poisoning attacks, statistical analysis was performed at the maximum poisoning intensity (10%), where the largest differences in classification degradation were observed. The results suggest that model sensitivity depends on both the attack strategy and the dataset characteristics. In particular, differences between linear classifiers and Random Forest become more evident under targeted poisoning conditions, indicating that adversarial sample selection influences models differently depending on their learning mechanism.

For evasion attacks, statistical comparisons were conducted at the selected perturbation threshold used for robustness evaluation. The analysis indicates performance differences between several architectures, particularly between Logistic Regression and the other evaluated models. However, comparisons between Linear SVM and Random Forest showed weaker statistical separation, indicating that their robustness differences may depend on the specific perturbation setting and dataset.

For model extraction attacks, comparisons were performed at the highest query budget. The results show statistically significant differences between target architectures under both random and adaptive querying strategies. These findings support the observation that model structure influences extraction susceptibility, with simpler decision functions generally allowing more accurate surrogate reconstruction.

Although the statistical tests provide evidence of differences between evaluated configurations, the results should be interpreted considering the limited number of experimental repetitions. Therefore, the conclusions are restricted to the evaluated datasets, attack implementations, and parameter configurations, and the statistical analysis is considered exploratory to support the observed trends rather than to establish universal conclusions about model robustness.

6. Discussion and Security Implications

The empirical evaluation conducted across poisoning, evasion, and model extraction scenarios suggests that traditional machine learning models exhibit distinct vulnerability profiles that depend on both the underlying model architecture and the characteristics of the adversarial strategy. Rather than indicating a universal robustness hierarchy, the results support the view that different model structures provide different forms of resistance and exposure. Linear models, such as Logistic Regression and Linear Support Vector Machines, utilize relatively simple and continuous decision boundaries, while ensemble models such as Random Forest rely on an increased structural complexity that influences their behavior under different attack conditions.

The following discussion interprets these findings by relating the observed performance changes to the architectural characteristics of the evaluated models and by considering the practical implications for deploying traditional classifiers in adversarial environments.

6.1. Conditional Robustness of Ensemble Architectures Under Poisoning Attacks

The poisoning experiments suggest that the impact of training-time manipulation is strongly influenced by both the attack strategy and the model structure. Under random label-flipping poisoning, Random Forest exhibited relatively stable performance compared with the linear models. For example, at a 10% poisoning rate on the MNIST dataset, Random Forest experienced an accuracy variation of approximately 0.21%, whereas Logistic Regression showed a larger degradation of approximately 3.74%. This behavior is consistent with the distributed decision process of ensemble models, where multiple decision paths can mitigate the influence of isolated corrupted samples.

However, the results also indicate that this resistance behaves as a conditional rather than a universal trait. When the poisoning strategy was changed from unstructured random corruption to confidence-based targeted poisoning, Random Forest experienced a more substantial performance reduction, reaching a 13.68% accuracy decrease on the AG News dataset. Unlike random noise, targeted poisoning modifies the training distribution in a more systematic manner by introducing samples selected according to their influence on the learned decision process.

For tree-based ensembles, such attacks appear to alter the learned partition structure because decision tree construction relies on feature-based split optimization. Manipulating high-confidence samples can therefore affect local decision regions and influence subsequent classification paths. These findings indicate that the resistance of Random Forest against random corruption does not necessarily translate into general protection against strategically designed poisoning attacks. Therefore, ensemble complexity should be interpreted as a conditional robustness factor rather than an inherent defense mechanism. The effectiveness of such models remains dependent on the relationship between the attack objective and the structural properties of the learned decision boundaries.

6.2. Architectural Trade-Offs Between Evasion Resistance and Extraction Vulnerability

The trends suggest a clear trade-off between susceptibility to model extraction and behavior under adversarial evasion attacks. This variance appears to be strongly related to how the evaluated models represent their decision boundaries.

Linear model approximations, LR and Linear SVM construct global linear decision surfaces, allowing their behavior to be approximated efficiently from query responses. In the extraction experiments, entropy-based adaptive querying achieved high replication fidelity at larger query budgets, reaching 96.22% fidelity for LR and 93.84% for Linear SVM on MNIST. These results suggest that models with simpler and more regular decision structures can be efficiently approximated when an attacker can strategically select informative queries.

At the same time, the continuous structure of linear models may contribute to more predictable degradation patterns under the evaluated evasion settings. For example, Logistic Regression maintained an adversarial accuracy of 77.32% ± 1.17% under PGD attacks at an intermediate perturbation budget ($ϵ$ = 0.02) on MNIST. This behavior suggests that gradual perturbation strategies may interact differently with linear decision boundaries compared with more irregular decision surfaces.

Ensemble model approximations. In contrast, Random Forest demonstrated greater resistance to extraction under the evaluated black-box query strategies. The adaptive extraction procedure achieved a lower fidelity of 52.34% ± 1.33%, which was also lower than the fidelity achieved through random querying in some settings. This result may be associated with the difficulty of efficiently mapping the irregular decision regions generated by multiple decision trees using limited query information. However, the same structural characteristics that complicate extraction can introduce challenges under evasion attacks. Because Random Forest predictions are determined by discrete decision paths, small perturbations may move samples across different regions of the learned partition space. Therefore, the observed resistance of Random Forest to extraction should not be interpreted as general adversarial robustness.

Overall, the results indicate that architectural complexity can provide resistance against specific attack strategies while simultaneously introducing other vulnerabilities. Security evaluation should therefore consider multiple threat models rather than relying on a single robustness indicator.

6.3. Optimization Effects in Gradient-Based Adversarial Evaluation

The adversarial evaluation of Linear SVM highlights an important consideration regarding the interpretation of iterative gradient-based attacks. At an intermediate perturbation level on MNIST ($ϵ$ = 0.02), PGD resulted in higher adversarial accuracy for the Linear SVM (78.03% ± 0.36%) compared with FGSM (54.28% ± 1.15%). This behavior does not automatically indicate stronger robustness against iterative attacks.

A potential explanation is related to the optimization characteristics of the loss function used by the model. Linear SVM commonly relies on hinge-based objectives that contain regions with limited gradient information once samples move beyond the margin boundary. Under certain attack configurations, iterative optimization may therefore encounter reduced gradient effectiveness, limiting the ability of PGD to generate stronger perturbations.

This observation highlights a methodological consideration in adversarial benchmarking: improved performance against an iterative attack should not automatically be interpreted as evidence of stronger security. Attack effectiveness depends not only on the model architecture but also on the interaction between the optimization procedure, perturbation parameters, and loss formulation.

Therefore, adversarial evaluation of classical models should carefully validate attack effectiveness and avoid conclusions based solely on the apparent success or failure of a specific optimization method.

6.4. Security Implications and Practical Considerations

The results provide several practical implications for deploying traditional architectures in security-sensitive applications.

First, model selection should consider the expected threat environment, as models that demonstrate resistance against one attack category may remain vulnerable to another. For example, Random Forest showed strong tolerance to random poisoning noise but was more affected by targeted poisoning, while linear models were easier to extract due to their simpler decision structures.

Second, under the evaluated parameters, reducing model transparency through increased structural complexity does not provide universal protection. Although complex ensemble models may be more difficult to replicate through black-box queries, their decision behavior may still be influenced by carefully designed adversarial manipulation.

Third, evaluation protocols should include multiple adversarial perspectives, since assessing only one threat category may provide an incomplete view of model security. An estimator that performs well under evasion testing may still expose intellectual property risks through extraction attacks, while a model resistant to extraction may remain sensitive to training-time manipulation.

6.5. Limitations and Future Research Directions

The findings of this study should be interpreted within the scope of the evaluated experimental conditions. The conclusions are restricted to the evaluated datasets, attack implementations, and parameter configurations. Therefore, these limitations provide distinct opportunities for future investigation.

First, the statistical analysis was performed using three initialization seeds (n = 3). Therefore, the reported significance values should be considered exploratory indicators of performance variation rather than population-level conclusions. Larger experimental blocks would provide stronger confidence estimates.

Second, multiple comparisons were conducted across datasets, attack strategies, and perturbation levels. The reported statistical tests were not corrected using procedures such as Bonferroni or Holm adjustments; therefore, significance results should be interpreted with caution.

Third, the evasion evaluation of Random Forest relied on surrogate-based gradient transfer attacks because tree ensembles do not provide direct differentiable gradients. Therefore, the results represent resistance against the evaluated transfer-based attack setting rather than universal robustness against all possible adversarial strategies.

Future work should investigate broader robustness evaluation frameworks, including adaptive attacks specifically designed for non-differentiable models, larger-scale statistical validation, and defensive strategies such as robust training, adversarial data filtering, and security-aware model construction methods.

Overall, the results suggest that traditional machine learning security is highly dependent on the interaction between model architecture and adversarial strategy. Rather than identifying a universally secure model, this study highlights the importance of evaluating the operational strengths and weaknesses of different architectures under diverse attack conditions.

7. Conclusions

This study presented a unified evaluation of traditional machine learning architectures - Logistic Regression, Linear SVM, and Random Forest - under three adversarial threat categories: training-time poisoning, inference-time evasion, and black-box model extraction. By examining these models across image and text datasets using an exploratory statistical evaluation framework, this work provides empirical insights into how architectural characteristics are associated with vulnerability patterns under different adversarial scenarios.

The findings suggest that for the settings examined in this framework, traditional machine learning security is characterized by architecture-dependent trade-offs, where resistance to one attack category does not imply broader adversarial robustness:

• Poisoning and extraction trade-offs: The Random Forest ensemble exhibited reduced extraction fidelity under the evaluated active query strategies, suggesting that its complex decision structure can increase the difficulty of black-box approximation. However, this advantage did not translate into universal robustness, as targeted poisoning attacks produced notable performance degradation, indicating that structured manipulation of training data can influence ensemble decision behavior.
• Linear model behavior under adversarial evaluation: Logistic Regression and Linear SVM demonstrated higher susceptibility to model extraction within these parameters, which is consistent with the premise that simpler decision structures can be approximated efficiently through query-based methods. At the same time, these models showed more predictable performance variations under the evaluated evasion attacks, reflecting the interaction between linear decision boundaries and specific perturbation strategies.
• Influence of optimization characteristics: The evasion experiments suggest that adversarial evaluation results can depend strongly on attack optimization procedures. The observed differences between FGSM and PGD performance for Linear SVM suggest that loss formulation and optimization dynamics may influence the apparent effectiveness of iterative attacks and should be carefully considered when interpreting adversarial robustness measurements.

Overall, the observed trends suggest that traditional machine learning models possess distinct and conditional security profiles rather than absolute robustness properties. Therefore, model selection for security-sensitive applications should consider the specific threat environment, including risks associated with training-data manipulation, adversarial inputs, and model replication.

Future research should investigate larger-scale statistical evaluations, adaptive attacks specifically designed for non-differentiable models, and security-aware training strategies for classical machine learning systems. Extending these comparative evaluations to additional architectures and application domains may further improve our understanding of structural factors influencing machine learning security.