MAD-OOD: A Deep Learning Cluster-Driven Framework for an Out-of-Distribution Malware Detection and Classification

1. Introduction

The potency of malware to successfully infiltrate any system no matter how sophisticated made it an indispensable tool available to cybercriminals today, as malware had proven to be highly successful in the extraction of sensitive data which could be used by cybercriminals against their victim. Several machine and deep learning based approaches had been widely proposed to combat the rampant threat of malware attack but the close-world assumption of identical samples are quickly violated whenever state-of-the-art models are exposed to previously unseen out-of-distribution malware attack. Hence, the persistent proliferation of malicious software (malware) poses an ever-evolving challenge to cybersecurity systems worldwide. With the advent of increasingly sophisticated malware variants and the ubiquity of polymorphic and metamorphic transformations, traditional malware detection mechanisms face significant limitations in achieving high detection accuracy, particularly in the context of out-of-distribution (OOD) detection [1,2,3]. While deep learning and machine learning have emerged as dominant paradigms in malware classification tasks, most state-of-the-art models demonstrate considerable performance degradation when evaluated on previously unseen malware families or variants not represented in their training data [4,5,6]. This research is motivated by a critical observation: the existing approaches do not effectively exploit the variation in latent embedding spaces between individual variants within the same malware family—a characteristic that is both prevalent and underutilized [7,8,9,10,11,12].

Malware classification typically relies on the assumption that samples from a particular class provide a comprehensive representation of that class. However, in the domain of malware, this assumption falls short. Each malware family comprises multiple variants, often generated using obfuscation techniques such as encryption, packing, or polymorphism, which drastically alter the feature representations of these samples while preserving their malicious intent [4,13,14,15,16]. Consequently, a single malware variant does not sufficiently represent the diversity of the family in feature space. This insight is a central tenet of our approach, and we argue that it accounts for the poor performance of existing OOD detection methods when applied to malware datasets.

Contribution

The primary contributions of this research are threefold:

• Novel Framework for OOD Detection: We propose a two-stage framework for OOD detection and malware classification that exploits the variation in embedding space between variants of the same malware family. The first stage uses spherical decision boundaries defined through Gaussian discriminant analysis to evaluate the likelihood of a test sample being in-distribution, while the second stage integrates this information into a deep neural network for final prediction.
• Distance-Based Confidence Modeling: Our method utilizes statistical measures, including Z-score and the coefficient of variation, to determine how far a test sample lies from the centroid of a class distribution. A key insight is the use of multiple z-score comparisons across class boundaries, where a sample must lie within a standard range relative to at least one centroid to be considered in-distribution. This multi-boundary approach accounts for the multidimensional nature of malware distributions.
• Empirical Validation: We provide a comprehensive evaluation of our model using malware datasets containing 25 known malware families and multiple out-of-distribution variants. Our model outperforms existing baseline methods in OOD detection, achieving superior AUC scores, particularly for novel malware types.

2. Background and Related Work

Out-of-distribution detection refers to the task of identifying inputs that do not belong to the distribution of the training data. This is critical for deploying machine learning systems in real-world environments, where they frequently encounter data that differs from what they were trained on. Traditional OOD detection methods in computer vision, such as those based on softmax confidence scores, Mahalanobis distances, and reconstruction error from autoencoders, have demonstrated promising results in standard benchmark datasets like CIFAR-10 and ImageNet [17,18,19,20,21]. However, the application of these techniques in the malware domain has been limited and underwhelming due to the distinct characteristics of malware data, including high intra-class variance and distributional overlap among different malware families.

Prior works in malware detection using deep learning have typically employed convolutional neural networks (CNNs) trained on image representations of binaries [10], recurrent neural networks on opcode sequences [22], or graph-based models leveraging control-flow information [23]. Although these approaches achieve high classification accuracy under closed-world settings, their performance declines significantly when tasked with recognizing novel, unseen malware families. One-class classification methods, such as One-Class SVM and Support Vector Data Description (SVDD) [24], have been explored to address this issue [25,26]. SVDD, for example, constructs a hypersphere around training data to capture the normal class distribution and flag samples falling outside the sphere as anomalous. However, these methods often lack scalability or robustness when confronted with the complex embedding distributions exhibited by modern malware datasets.

Recent advances have sought to integrate distance-based anomaly detection techniques into deep learning pipelines. For instance, works like Deep SVDD [27] and Geometric Transformations for OOD detection [25] attempt to reshape the representation space to better separate in-distribution and out-of-distribution samples. Nonetheless, these methods rarely consider the granularity of variation within individual classes, especially in datasets where class boundaries are inherently fuzzy due to polymorphic behaviors. Our work builds upon this direction by explicitly modeling the intra-class variance within malware families through spherical decision boundaries derived from Gaussian discriminant analysis.

3. Research Methodology

3.1. Spherical Boundary Modeling for Malware Classification

Since each family of malware has different variants leading to variation in the embedding spaces. Finding the spherical decision boundary for each class can be instrumental to determines whether a test sample had been previously seen during training or not by integrating the concept of Gaussian discriminant analysis Figure 1 into deep neural networks. This enables it to learn class-conditional distributions that are explicitly modeled as separable Gaussian distributions using the distance of a test sample from each class-conditional distribution to define the confidence score which is eventually used to identifying OOD samples.

$$\widehat{L}(u,v)=\{\begin{array}{cc}1\hfill & \mathrm{if}u=v\mathrm{and}{d}_v\ne 0\\ -{\displaystyle \frac{1}{\sqrt{d_u{d}_v}}}\hfill & \mathrm{if}(u,v)\in E\\ 0\hfill & \mathrm{otherwise}\end{array}$$

(1)

To effectively model the various spheres within the dataset, each sphere is treated as distinct class-conditional distributions, specifically assuming isotropic Gaussian distributions to ensure that OOD samples are positioned farther from all predefined class distributions without relying on OOD samples for validation.

$$\begin{array}{c}\hfill \mathcal{N}(x\mid {\mu }_k,\Sigma )=\\ \hfill & {\displaystyle \frac{1}{{\left(2\pi \right)}^{d/2}{|\Sigma |}^{1/2}}}exp\left(-{\displaystyle \frac{1}{2}}{(x-{\mu }_k)}^T{\Sigma }^{-1}(x-{\mu }_k)\right)\hfill \end{array}$$

(2)

In the proposed approach, an innovative two-stage framework is introduced to address the limitations of current OOD detection methods in malware datasets. The first stage involves modeling class-conditional distributions using spherical decision boundaries derived from Gaussian Discriminant Analysis (GDA) [28]. This approach assumes isotropic Gaussian distributions for each class, enabling the estimation of distances between test samples and class centroids to determine likelihood and confidence scores for OOD detection.

This approach provides a statistical basis for OOD classification using metrics such as the z-score, which measures the deviation of a sample from the mean in standard deviation units. By using a standard range (e.g., -1 to +1) to classify samples as in-distribution or out-of-distribution, the model effectively introduces a robust, interpretable decision framework. While similar to prior attempts at distance-based classification, this method innovatively combines these statistical insights with deep learning embeddings tailored to the unique characteristics of malware datasets [29,30]. This approach is closely related to recent advancements in Deep SVDD [5], which integrates the SVDD loss function into neural network training to learn a compact representation of in-distribution data. Similarly, the proposed method leverages the structure of latent embedding spaces to form well-defined spherical clusters, offering a more granular understanding of family-wise variations.

By measuring the distance between a test sample and the derived class-conditional distributions, we can determine the probability and confidence level of the sample belonging to each class. The theoretical foundation for integrating the distance-based classifier into deep neural networks (DNNs) is provided by Gaussian discriminant analysis (GDA).The qualitative analyses on the latent space obtained by this method provide the strong evidence that it places OOD samples more clearly distinguishable from ID samples in the space compared to the others. The task of data description, also referred to as one-class classification, involves defining a representation of the training data and utilizing it to determine whether a test sample belongs to the same distribution. Earlier research in this area primarily explored kernel-based methods, which sought to establish decision boundaries through identified support vectors. One notable approach, supporting vector data description (SVDD), constructs a closed spherical boundary (hypersphere) that encompasses most of the data.

3.2. Cluster-Aided Malware Detection and Final Prediction Network

Following the initial cluster-based classification, the framework employs a second deep learning model that integrates the initial prediction from the clustering algorithm, the output of a supervised DL model, and the input image itself to refine the final prediction. This ensemble-like design enhances the robustness of classification by combining complementary information sources. Notably, unlike conventional vision-based DL models used in malware detection [6], which typically rely solely on image-to-label mappings, this architecture leverages structural knowledge of the latent space through embedding-based features and prior statistical analysis. This strategy offers a unique advantage in handling polymorphic malware samples, which often defy simple visual patterns.

Figure 2. Confusion Matrix on sample test set containing previously unseen out-of-Distribution variants.

Figure 2. Confusion Matrix on sample test set containing previously unseen out-of-Distribution variants.

Table 1. AUC Score on an Out-of-Distribution Malware Classification.

Malware Family	AUC Score	Malware Family	AUC Score
Adposhel	0.0059	InstallCore	0.539
Agent	0.105	MultiPlug	0.576
Allaple	0.113	Neoreklami	0.629
Amonetize	0.166	Neshta	0.585
Androm	0.266	VBA	0.619
BrowseFox	0.257	Sality	0.573
Elex	0.305	Snarasite	0.830
Expiro	0.360	Stantinko	0.846
Fasong	0.383	Out-of-Dist. (Novel)	0.911
HackKMS	0.424	VBKrypt	0.916
Hlux	0.463	Vilsel	0.993
Injector	0.499

Table 1. AUC Score on an Out-of-Distribution Malware Classification.

Malware Family	AUC Score	Malware Family	AUC Score
Adposhel	0.0059	InstallCore	0.539
Agent	0.105	MultiPlug	0.576
Allaple	0.113	Neoreklami	0.629
Amonetize	0.166	Neshta	0.585
Androm	0.266	VBA	0.619
BrowseFox	0.257	Sality	0.573
Elex	0.305	Snarasite	0.830
Expiro	0.360	Stantinko	0.846
Fasong	0.383	Out-of-Dist. (Novel)	0.911
HackKMS	0.424	VBKrypt	0.916
Hlux	0.463	Vilsel	0.993
Injector	0.499

Table 2. Comparison of MAD-OOD with other state-of-the-art Out-of-Distribution models on Benchmark Malware Dataset.

Method	Model Comparison and Evaluation
	AUC	AP-Id	AP-OOD	FPR	AR-OOD	ACC
MSP	0.611	0.464	0.322	0.613	0.526	53.82
OE	0.247	0.634	0.709	0.751	0.592	0.407
EnergyOE	0.651	0.660	0.792	0.736	0.808	0.682
OCL	0.637	0.529	0.558	0.771	0.690	0.625
PASCL	0.209	0.405	0.392	0.229	0.393	0.592
OS	0.692	0.442	0.842	0.793	0.712	0.827
Class Prio	0.596	0.376	0.816	0.728	0.693	0.606
BERL	0.846	0.572	0.561	0.807	0.737	0.812
MAD-OOD	0.906	0.951	0.861	0.940	0.817	0.907

^a

Table 2. Comparison of MAD-OOD with other state-of-the-art Out-of-Distribution models on Benchmark Malware Dataset.

Method	Model Comparison and Evaluation
	AUC	AP-Id	AP-OOD	FPR	AR-OOD	ACC
MSP	0.611	0.464	0.322	0.613	0.526	53.82
OE	0.247	0.634	0.709	0.751	0.592	0.407
EnergyOE	0.651	0.660	0.792	0.736	0.808	0.682
OCL	0.637	0.529	0.558	0.771	0.690	0.625
PASCL	0.209	0.405	0.392	0.229	0.393	0.592
OS	0.692	0.442	0.842	0.793	0.712	0.827
Class Prio	0.596	0.376	0.816	0.728	0.693	0.606
BERL	0.846	0.572	0.561	0.807	0.737	0.812
MAD-OOD	0.906	0.951	0.861	0.940	0.817	0.907

^a

Table 3. Model Evaluation on Benchmark Malware Dataset.

OOD Malware	Proposed Model Generalization
Dataset	AUC	AP-Id	AP-OOD	TPR	AR-OOD	ACC
MaleVis	0.911	0.864	0.822	0.813	0.926	93.82
BODMAS	0.847	0.834	0.809	0.851	0.792	0.907
Virus-MNIST	0.851	0.860	0.792	0.936	0.908	0.882
Stamina	0.837	0.929	0.858	0.871	0.890	0.925
MalImg	0.906	0.951	0.861	0.940	0.817	0.907

^a Summary of OOD-MAD model against benchmark malware dataset

Table 3. Model Evaluation on Benchmark Malware Dataset.

OOD Malware	Proposed Model Generalization
Dataset	AUC	AP-Id	AP-OOD	TPR	AR-OOD	ACC
MaleVis	0.911	0.864	0.822	0.813	0.926	93.82
BODMAS	0.847	0.834	0.809	0.851	0.792	0.907
Virus-MNIST	0.851	0.860	0.792	0.936	0.908	0.882
Stamina	0.837	0.929	0.858	0.871	0.890	0.925
MalImg	0.906	0.951	0.861	0.940	0.817	0.907

^a Summary of OOD-MAD model against benchmark malware dataset

4. Metric Evaluation

Our metric evaluation is based on the centroid of each decision boundary as I can now measure the distance between a test sample and the derived class-conditional distributions and utilize it to determine the probability and confidence level of any particular sample belonging to a class to determine whether it is an in-distribution or out-of-distribution sample. In this, utilizing the class conditional distributions to determine the centroid of each decision boundaries. After this comes the challenge of how far new data points could be to any of the centroid before classifying it as in-distribution or an out-of-distribution sample.

Figure 3. ROC Performance Evaluation of the Proposed Framework on the prediction of Previously Unseen Out-of-Distribution Classification.

Figure 3. ROC Performance Evaluation of the Proposed Framework on the prediction of Previously Unseen Out-of-Distribution Classification.

We computed the standard deviation of data point from any new samples to see how far apart it is to each of the centroid of the spherical boundary, the results was that of a low standard deviation between data points of news sample to the centroid and high standard deviation for others, then arises the problem of no single rule to determine a low standard deviation across several centroids. This lead to computation of the CV (coefficient of variation) which is the ratio of the distance between the mean and standard deviation considering it has specific range to determine and outliner. Z-score as a statistical method can detect outliers by measuring how far a data point deviates from the mean in terms of standard deviations using the formula:

$$z={\displaystyle \frac{x-\mu }{\sigma }}$$

(3)

Where:

• z z-score (the number of standard deviations a data point is from the mean)
• x is the observed value
• $\mu$ is the mean of the data point within the cluster
• $\sigma$ is the standard deviation of the new data point to the centroid cluster standard deviation

• Computation of the Mean and Standard Deviation: Calculate the mean ($\mu$) and standard deviation ($\sigma$) of the selected feature across a dataset of benign and potentially malicious samples.
• Z-score computation: Determine the Z-score for each family represented by the centroid.
• Pre-Determine in-distribution Threshold: Common thresholds for outlier detection
  Z > 1 (or Z < -1) → Possible outlier
  Z > 2 → Highly suspicious
• Initial Input Classification: If a sample has a Z-score beyond the chosen threshold, it is flagged as an outlier and so is initially classify a possible out-of-distribution sample

So, to compare new data point from the test set, I use z-score because it has a standard range to determine an outlier data point. z-score (z) = (new data point - mean)/standard deviation The standard to determine outlier in z-score is that anything outside the range of -1 and +1 is an outliner. Hence, for any sample to be classify as an in-distribution sample, at least one of the z-score to each of the centroid of the spherical boundary must be within the range of -1 and +1. If all of the z-scores falls outside the -1 and +1 range, then it is consider as an out of distribution sample. If the z-score is less than -1 or greater than 1, the data point is an outlier as it is further away from the mean than majority of the other data points.

4.1. Training a Computer Vision Based-Deep Learning Classifier for Initial Family Prediction

Splitting Data Train, validation, and test sets are carefully divided into: Train & Validation Sets: Contain only in-distribution (known malware and benign) data along with sample representation from out-of-distribution set. Test Set: Contains both in-distribution and OOD samples for evaluation.

Training Procedure Step 1: Train on In-Distribution Data Loss Function: Use cross-entropy for classification (malware vs. benign). Optimizer: Adam and SGD with learning rate scheduling. Regularization: we applied dropout, and batch normalization

Evaluation Metrics In-Distribution Performance: Accuracy, and Confusion matrix for malware classification. OOD Detection Performance: AUROC (Area Under ROC Curve): Measures separation between in-distribution and OOD

4.2. Training a Computer Vision Based-Deep Learning Classifier for Final Prediction

Second Model is a deep neural network-based model which takes three arguments Argument 1-prediction from the cluster analysis Argument 2: prediction output from the first model Argument 3: Input image Unlike the first model, we make use of the new embedding spaces from the spherical boundary to train the second model since the final prediction comes from the second model. There are two stages to it here i. Initial prediction from the second model, since the second model is train on the new embedding spaces, We can expect a better predictive output from it ii. z-score Calculation

While previous work has explored embedding-based OOD detection [3,4], few have addressed the unique challenges posed by malware datasets. This study fills a critical gap by acknowledging and explicitly modeling the intra-family variation in malware, which is often overlooked in favor of simplifying assumptions. The integration of GDA-inspired decision boundaries, z-score analysis, and a multi-input deep learning classifier offers a holistic approach that outperforms existing methods in separating in-distribution malware from novel, unseen threats. Moreover, the empirical results support the efficacy of the proposed method. For instance, the Area Under the Curve (AUC) scores for known malware families such as Regrun (0.761), Snarasite (0.830), and Stantinko (0.846) indicate improved separability in the latent space, while the detection score for novel OOD samples reaches up to 0.911—highlighting its practical potential in real-world malware detection.

5. Contributions over Prior Work

This research introduces a novel two-stage framework for out-of-distribution (OOD) detection in malware classification by exploiting the inherent variability among malware variants within the same family. Unlike traditional models that assume class uniformity, this approach integrates unsupervised clustering with deep neural networks and Gaussian discriminant analysis (GDA) to establish spherical decision boundaries around malware family embeddings, effectively identifying unseen samples without requiring OOD data during training. A z-score–based statistical analysis enhances discrimination between in-distribution and outlier data. The second stage refines classification using enriched embeddings and multi-source inputs, resulting in superior predictive and OOD detection performance. Achieving an AUC score of 0.911, the model outperforms conventional methods, offering a robust, scalable, and statistically grounded solution for accurate malware detection, with potential extensions to real-time cybersecurity applications.