Trust Your Generator: Enhancing out of Model Scope Detection

1. Introduction And Motivation

One of the key factors contributing to the success of deep learning in computer vision is its ability to learn intricate patterns directly from the training data, eliminating the need for manual feature engineering. However, deep learning models are susceptible to overfitting [1] and their performance may degrade when faced with data that lie outside of the training set’s distribution [2].

A neural network trained on images of handwritten digits illustrates this issue: while some implementations [3] achieve near-perfect recognition rates on test data, in real-world scenarios [4] where the inputs are unbounded or uncontrolled [5], there is no guarantee that the images will belong to the same distribution as the training data. In this example, how would a network react to an image of a human? It would classify it as one of the 10 digits. A more tangible example would be autonomous driving, specifically when the car encounters an unknown traffic sign or has to operate in a previously unseen environment. In these cases, there is a need to develop and deploy a Monitoring function which recognizes Out-of-Model Scope inputs and reacts accordingly.

We present a novel perspective on the concept of Out-of-Model Scope [6] inspired by the validation processes described in the Safety of the Intended Functionality (SOTIF) norm [7] and in ISO 26262 [8]. We shed light on critical considerations for enhancing the robustness and reliability of Machine Learning (ML) systems deployed in the computer vision domain. To the best of our knowledge, there is no norm prescribing a specific method for detecting an OMS sample. The implementation of State-Of-The-Art (SOTA) monitors therefore varies and consists of hard-coded boundary binary classifiers that use features extracted from different layers [9], input images, or a combination of both [10]. Our approach aims to incorporate diverse information processed during the prediction phase, as can be seen in Figure 1. We refer to the monitored model as the Encoder and the approximated inverse version of its IMS distribution as the Generator. The detection based on the input images, generated images, and latent feature representations is solved by a learnable Monitor. Throughout this article, we will use the terms "OMS detection" and "OMS monitoring" interchangeably.

Our contributions in the domain of the OMS detection are as follows:

• We describe the theoretical background of OMS detection incorporating a generator of synthetic images.
• We devise a novel multi-input OMS detection pipeline that facilitates monitoring of any pre-trained classifier.
• We demonstrate the robustness of our method against distributional shifts and adversarial attacks.
• Through an ablation study, we evaluate the sensitivity of the OMS monitor to the various inputs.
• We show that our pipeline achieves SOTA results in the computer vision domain.

2. Related Work

2.1. Functional Safety Versus Computer Vision

For practical reasons, the objective of our contribution is aligned with industry norms. As mentioned in the introduction, the uncertainty of machine learning models brings a potential risk in real-world applications. We therefore consider the definitions and processes described in EN IEC 61508 [11] which define a general clustering of safety components in the Safety-Integrity-Level (SIL). On the other hand, the functional safety norm ISO 26262 [8] defines guidelines that minimize the occurrence and severity of hazardous situations related to components’ malfunction or failure. Furthermore, the ISO PAS 21448 [7], also referred to as SOTIF, which builds upon ISO 26262, extends the development process by addressing potential risks in situations where a system could operate incorrectly or unsafely.

The detection of OMS falls within the scope of SOTIF [7]. Nevertheless, the OMS detection pipeline presented in this paper is derived from the concept of hardware (HW) malfunction detection described in ISO 26262 [8]: in the event of a HW failure, the objective is to transition the system to a safe state and enable the operator to maintain control. To achieve this, the safety-critical function and its inverse implementation are monitored by the plausibility function, as highlighted in Figure 2. The implementation of such software (SW) measures reduces the risk of potential undetected malfunctions to a predefined and acceptable level [8].

In Computer Vision (CV), and particularly in deep learning, lossy information compression (Pooling Layers and activation functions) prevents the direct implementation of an inverse function. Additionally, in real-world conditions, the monitoring function must contend with the uncertainty of the monitored model, which is caused either by the model’s limited generalization capabilities [12], or by missing features in the training data [2]. These factors led to the design of a monitoring function [13] based on hard boundaries. The Monitor’s role is to detect the scope in which the model operates safely. Although the Monitor can predict whether the input belongs to the known data distribution, the uncertainty of the OMS detection persists [14].

2.2. OOD and OMS

The terms "Out of Distribution" and "Out of Model Scope" are often used interchangeably, but their definitions vary [15]. In this work, we adopt the definitions and guidelines provided by [6].

2.2.1. Out of Distribution

Let us consider a machine learning task on a domain $\mathcal{X}$, such as classification or detection, defined by an oracle function $\mathsf{\Omega }$. The oracle function $\mathsf{\Omega }$ maps points $x\in \mathcal{X}$ from the task domain to their corresponding ground truths $\mathsf{\Omega }\left(x\right)=y\in \mathcal{Y}$. Using the oracle function $\mathsf{\Omega }$, we can define the OOD scope and OMS of any domain. We denote the operational domain as data points $(x_1,y_1),\dots ,(x_n,y_n)\in \mathcal{X}\times \mathcal{Y}$. Considering the predefined OOD settings, we assume that the data follows a probability distribution $p_{\mathrm{ID}},$ where

$$\forall i=1,\dots ,n;x\sim p_{\mathrm{ID}}.$$

(1)

Given a new input instance $x\in \mathcal{X}$, our goal is to determine whether it originates from the same distribution $p_{\mathrm{ID}}$ or if it resides outside of this distribution. This leads to the formulation of an out-of-distribution monitor $M:\mathcal{X}\to \{0,1\}$ given by

$$\forall x\in \mathcal{X},M\left(x\right)=\left\{\begin{array}{cc}0\hfill & \mathrm{if}x\sim p_{\mathrm{ID}},\hfill \\ 1\hfill & \mathrm{else}.\hfill \end{array}\right.$$

(2)

2.2.2. Out of Model Scope

Referring to the nomenclature established in Section 2.2.1, we define the model scope $S_f$ as follows:

$$S_f=\{x\in \mathcal{X}:f\left(x\right)=\mathsf{\Omega }\left(x\right)\},$$

(3)

where f denotes the deployed model.

We aim to develop an OMS monitor $M_f:\mathcal{X}\to \{0,1\}$ capable of identifying all data points that fall outside the model scope $S_f$. This monitor function is defined as follows:

$$\forall x\in \mathcal{X},M_f\left(x\right)=\left\{\begin{array}{cc}0\hfill & \mathrm{if}x\in S_f,\hfill \\ 1\hfill & \mathrm{else}.\hfill \end{array}\right.$$

(4)

As can be seen in Equation 2, OOD detection focuses on distinguishing among an infinite number of probability distributions within the domain $\mathcal{X}$ and a specific In Distribution $p_{\mathrm{ID}}$. However, capturing all boundaries within a CV domain is only possible if the condition $\left|\mathcal{X}\right|<+\infty$ is fulfilled [14]. Proposed solutions for OOD detection in a stochastic environment are consequently vulnerable to adversarial attacks and shifts within the ID, as demonstrated in [16]. Furthermore, the definition of OOD can be significantly influenced by the domain and definition of the machine learning task. These ambiguities led to the implementation of various monitoring functions and the attainment of diverse results, as shown in [17].

In contrast to OOD, the definition of OMS provides a more precise and consistent framework for detecting samples that our model should not classify or make predictions on, as they are not drawn from $p_{\mathrm{ID}}$. The conditions for learnable OMS remain the same as for OOD [14]; nonetheless, its clearer definition eliminates the need to differentiate between outlier detection [18,19], anomaly detection [20,21], novelty detection [22], and the open-set recognition problem [23,24], as was the case with OOD. Due to these findings, we will focus only on OMS detection instead of OOD detection. Based on the OMS Monitor definition in Equation 4, our OMS encompasses both covariate and semantic shifts, as well as adversarial attacks. These distribution shifts can be introduced into the training data through various means, such as:

• Covariate shift: altering certain image aspects, such as brightness, to a degree that causes the classifier to fail.
• Semantic shift: introducing semantic content that has not been reviously encountered in our domain $\mathcal{X}$.
• Adversarial attack: incorporating malicious perturbations in the input data, imperceptible to human eyes.

2.3. Oms Metrics And Monitors

Although the monitors and metrics discussed in this chapter were originally developed for detecting OOD samples, they can also be utilized for OMS detection [6]. Moreover, it is common to cluster OOD detection methods based on the used inputs. As highlighted in Figure 1, the following possibilities are available:

• feature-based
• probability-based
• logit-based
• feature and logit-based

Our method is a novel generative approach utilizing features and logits.

2.3.1. Metrics

Among the possible feature-based methods, the distance between any point $\widehat{x}=(x_1,x_2,x_3,\dots ,x_N)$ within the operational domain $\mathcal{X}$ and the ID distribution $p_{\mathrm{ID}}$, called the Mahalanobis distance (MAHA), is the most frequently used. Introduced by P.C. Mahalanobis [25], the Mahalanobis distance was further employed by Lee et al. [26] through a Gaussian discriminant analysis applied to the softmax layer. Lee et al. discovered that abnormal samples are better represented in the feature space of Deep Neural Networks (DNNs) rather than the "label-overfitted" output space of softmax-based posterior distribution, as demonstrated in earlier studies [27,28].

The Max Softmax Probability (MSP) method, pioneered by Hendrycks and Gimpel [13], established OMS detection based on the hypothesis that correctly classified examples should exhibit higher probability compared to other classes. This involved determining a statistical threshold for a binary classifier using a validation set. On top of this work, Liang et al. built their ODIN (Out-of-Distribution detector for Neural Networks) system [29]. They used temperature scaling to maximize the discriminability of the softmax outputs for In- and Out-Of-Distribution images. Hendrycks et al. further focused on large-scale datasets anomaly detection and proposed using the negative of the maximum of the un-normalized logits for an anomaly score, which they called MaxLogit [30].

As pointed out by Liu et al. [24], relying solely on softmax probabilities led to high confidence for misclassified samples. Liu et al. therefore incorporated an energy-based score that is theoretically aligned with the probability density of the inputs and less susceptible to overconfidence.

2.3.2. Monitors

The Outside-the-Box method, introduced by Henziger et al. [23], addresses novelty detection by employing a monitor on top of the arbitrary hidden layers of a pre-trained classifier. During training, the method samples class-related responses and clusters them, thus improving monitor accuracy. Novelties are recognized using a set of box abstractions that determine whether a new data point is inside or outside of the projected box.

Motivated by activation patterns analysis, Sun et al. [31] noted that activations of OMS samples differ significantly from those of In Model Scope samples. Their Rectified Activation (ReAct) method further showed that gathering activations from other than the penultimate layer does not enhance accuracy, as early layers capture less distinctive features.

The latter statement, asserted by [31], was negated by Lin et al. [9], who evaluated the possibility of detecting an OOD sample from different intermediate layers. They proposed a new energy-based score to dynamically terminate an early exit during inference. They picked their early exit based on the number of bits needed to encode the compressed image, consequently proving on many datasets that less complex OMS inputs can also be reliably discovered from the early layers.

2.3.3. Other Evaluation Metrics

Class imbalances and other non-uniformities in the number of False Positives (FP) and False Negatives (FN) must be considered during the evaluation of any classifier. To address this, the Area Under the Receiver Operating Characteristic (AUROC) [32] is frequently used in conjunction with F-score. AUROC evaluates the True-Positive Rate (TPR) against the False-Positive Rate (FPR), where the TPR is calculated as $TP/(TP+TN)$ and the FPR analogously as $FP/(FP+TN)$, where TN stands for True Negative samples. Nonetheless, as mentioned in [13], AUROC may not be suitable when the positive and negative classes have notably different base rates. In such cases, the Area Under the Precision-Recall curve (AUPR) is preferred, as it adjusts for the varying positive and negative base rates. Another metric we use to combat class imbalance in the data is the balanced accuracy $AC{C}_B=(TPR+TNR)/2$ where $TPR$ is the true positive rate as defined before and the True Negative Rate (TNR) is given by $TN/(TN+FP).$ From now on, we refer to accuracy in OMS domain as balanced accuracy.

3. Oms Monitoring Pipeline

All components of the OMS monitoring pipeline are depicted in Figure 3. Our primary goal is to enhance the robustness of OMS Detection by providing multiple inputs to our OMS monitor. The weights ${\theta }_E$ of the Encoder E represent the feature space extracted from the operational domain $\mathcal{X}$, which, in return, represents the IMS distribution $p_{S_f}$. The Generator G produces reconstructed synthesized images $\widehat{x}$ based on a combination of latent features z. Ideally, if the input sample x belongs to the model scope $S_f$, G generates an image from the same distribution. On the other hand, when an OMS input is presented, the generator combines different features based on the Encoder’s latent feature representation z in the generated image. This representation z is extracted by a Latent Space Wrapper L and can contain information from different layers of the Encoder and its softmax prediction $\widehat{y}$. In the last step, the OMS binary classification is addressed by a multi-input OMS monitor $M_{\theta }$. This monitor receives three inputs: the original input image x, the reconstructed image $\widehat{x}$, and the latent feature representation z. Conceptually, our OMS monitor can be seen as a mapping $OMS({\theta }_M,{\theta }_G,{\theta }_E,{\theta }_L):\mathcal{X}\to \{0,1\}$. The description of the inference process is presented in Algorithm 1.

Based on the theoretical and practical expertise we gathered during experimentation, we have identified that the following properties of the OMS pipeline need to be fulfilled.

• The input space $\mathcal{X}$ cannot be infinite, as described in [14].
• The influence of regularization of the Encoder’s feature space on the Generator’s performance should be investigated: we use Dropout [1] as well as latent feature space normalization [33], which calculates a class-related cosine distance and maximizes it during the training of the Encoder.
• The Generator should have high reconstruction capability for samples within the Encoder’s scope $S_f$ and lose this ability for samples outside the scope $S_f$: to achieve this, during the generator’s training, we minimize the classification loss between the Encoder’s prediction on the original and generated images from the model scope.
• To be able to compare the contribution of our multi-input OMS pipeline, we investigate the performance fluctuation of the Monitor with the ablation study.
• To increase the robustness we use Dropout directly on the inputs provided to the OMS monitor, namely x,$\widehat{x}$, and z.

Algorithm 1 OMS Monitor during inference.

Require: $\left|\mathcal{X}\right|<+\infty$     for $x\in \mathcal{X}$ do        $\widehat{y},z=E_{\theta }\left(x\right)$             ▹ $\widehat{\mathbf{y}}\in {\mathbb{N}}^C$, $\mathbf{z}\in {\mathbb{R}}^D$, $\mathbf{x}\in {\mathbb{R}}^{3\times M\times N}$       $\widehat{x}=G_{\theta }(z,E_{\theta }\left(x\right)|\widehat{y})$                   ▹ $\widehat{\mathbf{x}}\in {\mathbb{R}}^{3\times M\times N}$       $\widehat{o}=M_{\theta }(x,z,\widehat{x})$                         ▹ $\widehat{\mathbf{o}}\in \{0,1\}$     end for

In general, the Encoder can be any deep neural network that has been trained on the operational domain $\mathcal{X}$; its implementation must allow access to intermediate layers. The Generator has to follow an inverse information flow with respect to the Encoder. In our case, the generator produces synthetic images $\widehat{x}$ with the same resolution as the input image x. This allows us to directly evaluate the image fidelity with the Fréchet Inception Distance (FID) [34] and the Inception Score (IS) [35]. There are no further constraints on the implementation of the Generator, and its architecture can be similar to Generative Adversarial Networks (GANs) [36], or Latent Diffusion models [37]. What distinguishes our approach from our predecessors’ is that the OMS monitor contains learnable parameters and receives synthetic images from the approximated density function of the Encoder, namely: the Generator.

4. Experiments

The evaluation setup of our experiment was inspired by [6,9]. However, instead of using pre-trained networks, we explore the influence on the OMS performance by training custom ResNet models with a different number of parameters and layers.

4.1. Ims And OMS Datasets

We chose the CIFAR-10 [38] as our initial dataset, following common practice in the OOD and OMS community. We split all datasets uniformly: 70% were used for the training set, 20% for validation, and the remaining 10% for testing purposes.

The OMS dataset consists of semantic shifts, covariate shifts, adversarial attacks, and FP samples. We built the set of semantic shifts using publicly available datasets, namely MNIST [39], fashion-MNIST [40], k-MNIST [41], SVHN [42] and DTD [43]. To test our pipeline’s robustness against covariate shifts in the data, we introduced various modifications to the original CIFAR-10 dataset. These modifications were applied until the Encoder failed to recognize otherwise correctly classified IMS samples. The applied modifications included blurring, brightness adjustments, image rotations, and noise addition. Lastly, we introduced imperceptible malicious perturbations to the IMS data, also known as adversarial attacks [44]. These perturbations are designed to be undetectable by human observers. Our approach involved leveraging the Fast Gradient Sign Method (FGSM) [45], Projected Gradient Method (PGD) [46], DeepFool [47] and AutoAttack [48]. The implementation in the Torchattacks [49] library was used in our case. The FPs samples, which are identified after the training of the Encoder is completed, form a default OMS dataset. Consequently, the identified TPs form the IMS ($S_f$) dataset. A selection of random samples and transformations from all datasets can be seen in Figure 4. Furthermore, a summary of all methods that were used to construct the OMS datasets is highlighted in Table 1.

4.2. Training

The OMS monitoring pipeline consists of four main components: the Encoder E, the Generator G, the OMS monitor O, and the Latent Space Wrapper L. Each component is trained independently, which ensures a plug-and-train monitoring system that can be applied to any pre-trained Encoder (e.g. a ResNet).

4.2.1. Encoder

To establish the foundation for the IMS dataset, we commenced with the training of the Encoder, which defines the IMS dataset as per Equation 3. We employed ResNet models [50] with 18, 34, 50, and 101 layers, leveraging the CIFAR-10 dataset divided by the guidelines outlined in Section 4.1. Each model underwent training with the Adam optimizer [51], for a total of 500 epochs, employing a learning rate of $0.001$ and following a cyclic learning rate scheduler, all while utilizing a batch size of 1024. The Categorical Cross Entropy was used as loss function. Additionally, we applied various data augmentation techniques available in PyTorch [52], including random horizontal flipping, random cropping, and random affine transformations.

Many OMS detection methods assign a score to an input sample based on its representation in the feature space. Given the infinite variations in OMS data, we have to rely on the IMS feature space generated by our Encoder. Should this feature space representation of our IMS data lack relevance, it may falsely contribute to the detection of OMS examples. As a result, we applied regularization techniques related to the latent feature space. This involved implementing a dropout mechanism with a probability of 0.2 for zeroing out and incorporating a Spatial Feature (SF) regularizer. The SF regularizer calculates both inner and outer cosine class similarities. During training, it maximizes the inner similarity while minimizing the outer similarity. For a visual comparison of the feature spaces before and after regularization, we refer to Figure 5, which showcases the Uniform Manifold Approximation and Projection (UMAP) [53] results.

4.2.2. Generator

Let us start with the theoretical description of the role of our Generator in the example of the vanilla GAN architecture. The GAN architecture is comprised of two main components: the Generator (G) and the Discriminator (D). The Discriminator is trained to differentiate between an original image x and a synthetic image created by the Generator $G\left(z\right)$. This is reflected in its loss function: ${\mathbb{E}}_{x\sim p_{\mathrm{data}}\left(x\right)}[log\left(D\left(x\right)\right)]+{\mathbb{E}}_{z\sim p_z\left(z\right)}[log(1-D\left(G\left(z\right)\right))]$. Concurrently, the Generator aims to produce images that  l the Discriminator, optimizing its loss function: ${\mathbb{E}}_{z\sim p_z\left(z\right)}[log(1-D\left(G\left(z\right)\right))]$. The Generator uses a latent vector z as input, typically sampled from a Normal distribution with mean=0.0 and standard deviation=1.0. During the training, the vanilla GAN model attempts to minimize the KL divergence [54] between the IMS data distribution $p_{S_f}\left(x\right)$ and the implicit generator data distribution $p_G\left(z\right)$. This often leads to learning instability and difficulty in capturing structural and geometric features [55]. Consequently, this has spurred the development of enhanced GAN architectures such as convolutional GAN [56], Self-Attention GAN [57], BigGAN [58], and Style GAN [59], along with improved training setups like Wasserstein GAN with gradient penalty [55] and spectral normalization [60].

The Generator in our pipeline aims to densely approximate the IMS distribution $p_{S_f}\left(x\right)$. Its task is to reconstruct the input x given its latent feature space representation z and the predicted class $\widehat{y}=E\left(x\right)$ of the Encoder E. More specifically, the generator learns $p_{S_f}\left(x\right|\widehat{y},z)$. It should be noted that during the training with the IMS dataset, we can use y in place of $\widehat{y}$, and thus use $p_{S_f}\left(x\right|y,z)$. Our goal is to minimize the distance between $p_{S_f}\left(x\right)$ and $p_{S_f}\left(x\right|\widehat{y},z)$; but instead of using a random noise, we utilize the latent feature space from the Encoder E. Consequently, we minimize the classification loss ${\mathcal{L}}_{\mathrm{cls}}(G,E)=CCE(y,E\left(G\left(z\right)\right))=-{\sum }_{c=1}^C{y}_{c}log\left({\widehat{y}}_c\right)$, where $CCE(y,E(G\left(z\right)\left)\right)$ is the Categorical Cross Entropy (CCE) between ground truth label y and the classified generated image $E\left(\widehat{x}\right)=E\left(G\left(z\right)\right)$. In our case, the representation of z can correspond to the latent feature space of one or more layers. The complete training loss function for the Generator G is detailed in Equation 5.

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{Generator}}(G,D,E)& ={\mathbb{E}}_{x\sim p_{\mathrm{data}}\left(x\right)}[log\left(D\left(x\right)\right)]\hfill \\ \hfill & +{\mathbb{E}}_{z\sim p_z\left(z\right)}[log(1-D\left(G\left(z\right)\right))]\hfill \\ \hfill & -\alpha {\mathbb{E}}_{y\sim P\left(y\right)}\left[{\displaystyle \sum _{c=1}^C}{y}_{c}log\left({\widehat{y}}_c\right)\right],\hfill \end{array}$$

(5)

where ${\mathbb{E}}_{y\sim P\left(y\right)}$ denotes the expected value over the probability distribution of the true labels y, C is the number of classes, $y_c$ is a ground-truth indicator, ${\widehat{y}}_c$ is the predicted probability and $\alpha$ is a scaling factor for training stability, empirically set to $0.1$.

We applied the aforementioned training approach to all three models, WGAN [55], BigGAN [58] and Class Conditional Latent Diffusion Model [37] (CCLDM). To summarize, the following hyper-parameters were explored in terms of image fidelity and contribution to OMS detection:

• generator’s architecture and size,
• dimension of the latent feature space z,
• class/feature-embedding method,
• combination of intermediate layers building the latent feature space z,
• implementation of the latent space wrapper L,
• and integration of a CCE between ground truth label y and the classified generated image.

For the first two GAN-based models(WGAN and BigGAN), we utilized the training loss described in Equation 5 and followed the hyperparameters’ settings from the original papers. In the case of the CCLDM model, the class embedding information was directly concatenated with the denoising parameters and the original image. During the training, the original image, with added noise, was reconstructed and the Mean Squared Error between the original and denoised image was minimized. The main component of the diffusion architecture is the denoising U-Net [61] model.

By incorporating the latent features z extracted by the encoder as class conditioning information, we encourage our generator to synthesize images of the same class with similar features learned during the Encoder’s training. Our hypothesis is guided by the expectation that even in the event of an adversarial attack, where the sample falls outside of the model’s scope, the Generator will still reconstruct an image containing features from the IMS distribution. To achieve this, we explore the hyper-parameter space and evaluate which combinations are the most effective for OMS detection in general.

4.2.3. Latent Space Wrapper

As previously mentioned, we use information from the latent feature space of the Encoder. Since this feature space can stem from any arbitrary deep neural network, we have to devise a mechanism that will aggregate the information from different feature maps of the model and produce a single latent representation z. We deploy a non-trainable approach, by applying Average and Max-Pooling layers directly to the intermediate layers, as well as a trainable approach with convolutional layers with a stride of the size of the kernel. Our Latent Space Wrapper, which adapts to the size of each given layer and whose parameters are tuned during the training of the Generator, can be seen in Appendix in Figure A1.

4.2.4. OMS Monitor

In the final phase, we train the OMS Monitor. The primary task of the OMS Monitor is to perform binary classification based on the similarities between its three inputs. Specifically, the Monitor predicts whether the input from the Encoder is OMS or not.

During the training of the OMS Monitor, we construct the OMS datasets using the methods and combinations described in Section 4.2. It is noteworthy that if we solely consider FPs as OMS examples, the size of the OMS dataset is contingent upon the performance of the Encoder. In scenarios where the Encoder achieves 100% accuracy, the OMS dataset would consist of an empty set of images. This underscores the dependency between the size of the OMS dataset and the performance of the Encoder, thereby justifying the necessity of integrating supplementary OMS data. To address potential issues of dataset imbalance during training, we employ Focal Loss [62] and weight random sampling techniques.

Notably, we do not apply any data augmentation on the IMS dataset. This precaution is to prevent a sample from inadvertently being categorized as OMS. Building on the training setup of the Encoder, we apply Dropout before each Linear Layer, and we use Binary Cross Entropy as the loss function.

4.3. Evaluation of Encoder and Generator

As Encoder, we trained different ResNet models, namely with 18, 34, 50, and 101 layers. All models consist of 4 blocks, which serve as binding points to our Latent Space Wrapper. The accuracy of the Encoders for each dataset is presented in Table 2.

As elaborated in detail in Section 4.2, the generator is trained on the IMS dataset. The highest classification ACCuracy on the generated images $AC{C}_{E\left(G\right(z\left)\right)}$, was achieved by the bigGAN model with a latent feature space dimension of 64, combining the features extracted from ResNet34 blocks 1 and output from the softmax layer. This model reached $91.80\%$ for the training set and $91.96\%$ for the validation set. Image fidelity and image variance were evaluated by the Frechet Inception Distance (FID) [34] and Inception Score (IS) [35], which correlates well with human judgment [63]. The same model achieved FID of $28.04\%$ and IS of $7.92\%$. To increase the evaluation set and as recommended in the original FID paper, all samples from the IMS dataset were included in the assessment.

The best model of each generator’s architecture can be found in Table 3. As we conditioned the training loss from Equation (5) by $AC{C}_{E\left(G\right(z\left)\right)}$, our model finds the optimum between image fidelity and the accuracy of the Encoder. Consequently, our generator achieves lower IS and higher FID scores than reported in the original papers.

As mentioned at the beginning of Section 4.2.2, several hyper-parameters influence the generator’s and consequently the monitor’s performances. The influence of the dimension of the latent feature z on the $AC{C}_{E\left(G\right(z\left)\right)}$ is highlighted in Figure 6. An ablation study of different combinations of layers further shows that the most relevant features are contained in the deeper layers. As can be seen in Figure 6, a generator trained on combinations of features extracted from multiple layers outperforms a generator that was trained only on features from one specific layer.

Although the highest classification accuracy on CIFAR-10 was achieved by ResNet-34, as highlighted in Table 4, the combination of Generator with ResNet-18 showed similar results in the OMS detection task as the combination with ResNet-34. The deployment of more complex Encoders with a higher number of layers didn’t improve the performance of the Generator. On the contrary, as visible in Table 5, applying dropout during the training of the Encoder improved the Generator’s performance. The final overview presented in the Appendix in Table A2 summarizes all hyper-parameters and techniques related to the Generator’s training.

4.4. Evaluation OMS Detection

In the case of OMS monitor, we deal with a binary classification problem involving the identification of two classes. However, it is a common scenario for the volume of OMS data to be larger than the IMS data, both in real-world applications and experimental setups [64]. As written in Section 2.3, to address this issue, we employ metrics that consider a trade-off between TPs and FPs and calculate the balanced accuracy instead of vanilla accuracy.

As mentioned in Section 4.2, we train the OMS monitor on each IMS and OMS training set presented in Table 1 and validate it on all the IMS and OMS validation and test sets. In our experiment, we explore various combinations of OMS datasets to discern the individual impact of each dataset on the OMS detection instances. Achieved results are presented in Table 7.

4.5. Ablation Study Of The OMS Monitor

Ablation studies in machine learning involve systematic evaluation of a model by removing or modifying its components to understand their impact on its performance. In the context of our OMS Monitor, the key elements under consideration are the original input x, the latent feature representation z, and the generated image $\widehat{x}$, all of which serve as inputs to the OMS monitor.

The results, as presented in Table 8, show the contributions of each input to the overall OMS detection. The accuracy drops by $25.20\%$ after removing the original input image x, proving that the primary source of relevant information for the OMS monitor still lies within the original input data. On the other hand, incorporating the feature space and generator increases the overall classification accuracy by approximately $4\%$. This subtle improvement can be explained by insufficient approximation of the IMS distribution $p_{S_f}\left(x\right)$. Namely, the best bigGAN model achieved an accuracy of $91.96\%$ on the IMS training set. Images identified as FP by $E\left(G\right(z\left)\right)$ could be considered for removal from the original IMS training set. However, this action would only superficially boost the $AC{C}_{E\left(G\right(z\left)\right)}$ to $100\%$, but shrink the valid IMS samples training set, thereby impairing the Monitor’s ability to generalize.

5. Conclusions and Future Work

Our novel OMS Monitoring pipeline represents an advancement in the development of functional safety monitors. It effectively detects samples that were not part of the training setup, preventing actions that could lead to severe consequences. As demonstrated in Section 4.4, our OMS Monitoring pipeline, inspired by the HW-Malfunction detection, achieves a balanced accuracy of $99.52\%$ on OMS samples from previously unseen test sets. Furthermore, it exhibits the highest AUROC, $99.46\%$, among all evaluated monitors and metrics. As can be seen in Table 7, our pipeline excels in detecting adversarial attacks, which were designed to deceive the potentially safety-critical Encoder. The overall detection accuracy was achieved by incorporating OMS distribution shift methods during OMS Monitor training. However, it is important to note that detecting false positives, which were excluded from the original dataset, remains the most challenging aspect of OMS detection. This behavior is not unique to our monitoring pipeline but is consistent across state-of-the-art methods, as shown in Table 10. Our method achieved the best results quantified by Area Under the Receiver Operating Characteristic, and the probability that a negative (OMS) example is misclassified as positive (IMS), namely FPR95TPR. Our weaker performance in AUPR compared to the current SOTA methods can be explained by the optimization process of the OMS monitor, via binary cross entropy, which rewards both TP and TN. This dichotomy can result in a slight decrease in precision, as can be seen in Table 9.

In Section 4.3 we thoroughly discussed the contribution of the Generator to the OMS Monitoring pipeline. The approximation of the IMS distribution of the Encoder improved mainly due to the incorporation of the classification loss on the reconstructed images and by utilizing the Latent Space Wrapper. Consequently, the OMS training improved in stability as well as in accuracy. A notable drawback of using the Diffusion architecture as a Generator throughout our experiment was a high inference time caused by an iterative diffusion process. Even though the FID and IS scores of the diffusion model were better than those of the bigGAN, the bigGAN model achieved the highest $AC{C}_{E\left(G\right(z\left)\right)}$ which, as mentioned in Section 4.2.2, is more relevant in the context of the OMS detection task.

To summarize the key takeaways, our OMS Monitoring pipeline allows the integration of any pre-trained Encoder, regardless of the working domain. We demonstrated the potential of a robust OMS sample detection, where the main focus was laid on the utilization of the Generator as a universal approximator of the Encoder’s latent space. Due to all our improvements the OMS detection pipeline achieved high accuracy in detecting various Adversarial Attacks, $98.58\%$. We hope that this potential will open doors to further topics and future work, namely:

• Adjustment of our OMS detection pipeline to other tasks in computer vision and other fields of machine learning, such as Natural language processing.
• Research of a possible approximation of the intermediate encoders’ layers and investigation of end-to-end training which we found based on the definition of IMS, currently not applicable.

Appendix A.1. Latent Space Wrapper

Figure A1 depicts a non-trainable computational graph. Within the computational process, we scale the inputs of each layer to the dimension of the tensor z with a given feature map of dimensions $(C,M,M)$. The left part of the graph depicts the latent feature space reduction through Average and Max-Pooling layers. The right branch gives an example of applied layers when the number of channels C is smaller than the required feature space dimension of z. In this chart, operations such as up-scaling and reshaping, which are necessary for achieving the desired output tensor shape, are not visualized. The trainable Wrapper replaces the Maximum Pooling layers with Convolutional layers and uses the calculated K as kernel size and stride.

Figure A1. This diagram represents a computational graph of a non-trainable latent space wrapper.

Figure A1. This diagram represents a computational graph of a non-trainable latent space wrapper.

Appendix A.2. Performance of Generator

During training, the Generator G minimizes the reconstruction loss between the original $x_i$ and the generated image $\widehat{x_i}$. This implies that the weights ${\theta }_G$ capture an approximated inverse function to our model E. Consequently, the reconstructed image $\widehat{x_i}$ is the outcome of $G_{\theta }(z_i,\widehat{y_i})$, where $z_i$ generally represents a combination of aggregated feature spaces from various layers. As we want to enhance the robustness of our OMS detection pipeline by the reconstructed image $\widehat{x_i}$ from the Generator, we have to ensure its best performance. We therefore conducted several additional experiments with learnable and non-learnable latent space wrappers as can be seen in Table A1. The most robust performance was achieved by extracting the features via maxPool layer. All techniques and their influence are further summarized and listed in Table A2.

Table A1. Variations of learnable and non-learnable latent space wrapper only for the bigGAN generator, showing the differences in $IS$, $FID$ and $AC{C}_{E\left(G\right(z\left)\right)}$.

$\left|z\right|=64$	↑IS	↓FID	$\uparrow AC{C}_{E\left(G\right(z\left)\right)}$
avgPool	7.87	29.22	90.78
maxPool	7.92	28.04	91.96
learnable	6.89	32.07	78.45

Table A1. Variations of learnable and non-learnable latent space wrapper only for the bigGAN generator, showing the differences in $IS$, $FID$ and $AC{C}_{E\left(G\right(z\left)\right)}$.

$\left|z\right|=64$	↑IS	↓FID	$\uparrow AC{C}_{E\left(G\right(z\left)\right)}$
avgPool	7.87	29.22	90.78
maxPool	7.92	28.04	91.96
learnable	6.89	32.07	78.45

Table A2. Summary of all hyper-parameters and techniques and their influence on the generator’s training. FS stands for Feature Space.

Method / Hyper-parameter	$\uparrow AC{C}_{E\left(G\right(z\left)\right)}$	Reference	Reasoning
Encoders depth	Small	Table[4]	Deploying more complex ResNet models on the CIFAR-10 dataset didn’t improve the accuracy of the encoder, nor did it enhance the performance of the decoder.
Encoders FS regularization	Middle	Table[5]	Training with Dropout and Feature regularization influences FS’s compactness and consequently improves the Generators’ training performance.
Generators Architecture	High	Table[3]	Almost similar results were achieved by the bigGAN model and Stable Diffusion model with latent class embedding. Compared to bigGAN, the WGAN achieved lower performance by 21%.
FS dimension	Middle	Figure[6a]	Grid search method settled around the dimension of size 64. Smaller and higher feature space dimension results in lower $AC{C}_{E\left(G\right(z\left)\right)}$.
FS layers combination	Middle	Figure[6b]	Combination of features from more than one and deeper layers results in higher $AC{C}_{E\left(G\right(z\left)\right)}$.
Trainable Latent Wrapper	Small	Table[A1]	The best results were achieved by finetuning the latentwrapper with Max Pooling Layers. The combination with trainable Conv2d layer didn’t improve the $AC{C}_{E\left(G\right(z\left)\right)}$.

Table A2. Summary of all hyper-parameters and techniques and their influence on the generator’s training. FS stands for Feature Space.

Method / Hyper-parameter	$\uparrow AC{C}_{E\left(G\right(z\left)\right)}$	Reference	Reasoning
Encoders depth	Small	Table[4]	Deploying more complex ResNet models on the CIFAR-10 dataset didn’t improve the accuracy of the encoder, nor did it enhance the performance of the decoder.
Encoders FS regularization	Middle	Table[5]	Training with Dropout and Feature regularization influences FS’s compactness and consequently improves the Generators’ training performance.
Generators Architecture	High	Table[3]	Almost similar results were achieved by the bigGAN model and Stable Diffusion model with latent class embedding. Compared to bigGAN, the WGAN achieved lower performance by 21%.
FS dimension	Middle	Figure[6a]	Grid search method settled around the dimension of size 64. Smaller and higher feature space dimension results in lower $AC{C}_{E\left(G\right(z\left)\right)}$.
FS layers combination	Middle	Figure[6b]	Combination of features from more than one and deeper layers results in higher $AC{C}_{E\left(G\right(z\left)\right)}$.
Trainable Latent Wrapper	Small	Table[A1]	The best results were achieved by finetuning the latentwrapper with Max Pooling Layers. The combination with trainable Conv2d layer didn’t improve the $AC{C}_{E\left(G\right(z\left)\right)}$.