Increasing the Robustness of Image Quality Assessment Models Through Adversarial Training

1. Introduction

Image quality assessment (IQA) is essential to developing image- and video-processing algorithms. These algorithms assess the perceptual quality of processed images and reduce reliance on costly subjective tests. IQA models fall into two main categories, depending on the availability of the original image: full reference (FR), which involves estimating the difference between the original and processed images; reduced reference (RR), which involves calculating the difference between features extracted from the original and processed images; and no reference (NR), which only involves analyzing the distorted image using scene statistics gathered from a large dataset.

In recent years, IQA models have shifted toward deep learning, surpassing traditional approaches in correlation with subjective quality. However, this shift has also introduced new vulnerabilities, as all trainable methods, including IQA models, are susceptible to adversarial attacks. These attacks can occur in many real-life image-quality-measurement scenarios [1,2], such as cheating on benchmarks, manipulating web-search results, increasing the energy consumption and latency of streaming video, and storing unclear video-surveillance footage. Researchers often use IQA models to estimate the perceptual quality of image enhancement algorithms, e.g., underwater image enhancement [3,4], low-light image enhancement [5]. When IQA models are used as the optimization objective to improve the visual quality, unintended attacks may occur, leading to visible image artifacts. Current non-robust learning-based IQA models exhibit low resistance to adversarial attacks, and their use as optimization objectives may lead to visible artifacts.

Recent papers have proposed adversarial attack methods for IQA models [6,7,8,9,10], but only a few of them offer defense methods. Adversarial training is essential to improve neural network resistance to these attacks. Surprisingly, however, little research has considered adversarial training for IQA models. Only in one paper suggesting a new adversarial attack, did the authors try adversarial training for three models [6]. They concluded that adversarial training is a promising defense for deep IQA models and said further research is necessary to investigate more-sophisticated techniques.

The main drawback of applying vanilla adversarial training to increase IQA models’ robustness is a falloff in their correlation with subjective scores, as a previous study has shown [6]. his effect is because these models are more sensitive to image noise than are computer-vision models such as object classification and detection. When attacking the IQA model, the adversarial noise affects the subjective-quality labels. Even an unsuccessful attack that leaves the IQA score unchanged will likely alter an adversarial image’s perceptual quality. Unlike this case, an unsuccessful attack on an image classifier leaves the adversarial image’s target class unchanged. Also, training an IQA model from scratch only on the attacked data generated at each iteration, as in the standard adversarial training approach, may decrease the correlation.

This paper studies adversarial training’s efficiency in improving IQA model robustness to adversarial attacks. We propose a new strategy to adjust perceptual-quality scores for adversarial images during training. Our main contributions are the following:

• A new adversarial-training approach with a perceptual-quality-labels penalty. It outperforms standard adversarial training by correlating with subjective quality.
• A new method for comparing the robustness of trained IQA models, as well as a new robustness-evaluation metric.
• Comprehensive experiments that tested our method’s efficiency on four IQA models: LinearityIQA, KonCept512, HyperIQA, and WSP-IQA.

The paper is organized as follows. Section 2 discusses related work: the task and trends in image quality assessment, the existing adversarial attacks on IQA models, adversarial training as a defense against adversarial attacks, and the existing IQA models designed to be robust to adversarial attacks. In section 3, we describe the proposed method: first, we formulate the problem, and then we describe the main ideas of our adjustments to the vanilla adversarial training procedure. Section 4 describes our experiments’ methodology, including a list of datasets and IQA models, evaluation metrics, and main details of experiment implementation. In section 5, we show the results of the experiments. Section 6 discusses the main findings from the results, and section 7 summarizes the conclusions of this work.

2. Related Work

2.1. Image Quality Assessment

IQA methods aim to make quantitative evaluations that closely approximate human perceptual judgments. NR-IQA models function without an original image or any information about that image; they may serve a broader range of applications than other models, such as controlling video-streaming quality and assessing the quality of text-to-image generation. Initially, the IQA model’s design depended on the task and how its developers understood visual-quality perception by the human eye. Early approaches seldom involved machine learning [11]: examples include error visibility (e.g., PSNR), structural similarity (e.g., SSIM [12]), and information theory (e.g., VIF [13]).

Because subjective quality image and video datasets have proliferated, new IQA methods employ deep learning to model empirical statistics of natural images in the NR case or feature learning in the FR case. The latest comparisons reveal that deep-learning-based IQA outperforms traditional alternatives [14,15]. Image-quality measurement is widening thanks to the appearance of new narrow task-specific models. For example, recognition-aware IQA evaluates image quality as computer vision’s ability to detect or recognize objects, as well as image generation or image super-resolution models. Many IQA models that handle perceptual quality employ CNN-based architectures [16,17,18,19,20], but many also employ ViT [21,22].

2.2. Adversarial Attacks on IQA Models

Adversarial examples are specially crafted inputs that add small imperceptible perturbations to images such that the perturbed input can, with high confidence, mislead an IQA model to yield a false prediction [23]. Depending on the attacker’s knowledge of the victim model, adversarial attacks are either white-box or black-box. In the former case, an attacker has some knowledge about the model, such as its architecture, weights, loss function, or training data. These attacks often use gradients and generate high-confidence adversarial examples that are invisible to the human eye. In the black-box case, an attacker has no direct access to the target model and sometimes only evaluates it on the basis of chosen examples without gradient feedback. Therefore, adversarial examples in this situation are weaker than in the white-box situation. White-box attacks normally have few applications. In the case of IQA models, however, they are practical because IQA model architectures are often available, as with public benchmarks. White-box attacks can obtain gradients to optimize the objective function by taking a single small step (one-step attacks) or multiple small steps (iterative attacks) along the gradient direction in image space.

Several adversarial attacks have targeted IQA models. UAP [8], for example, creates a universal adversarial noise that increases the IQA model score. The generated perturbation is independent of the input image. FACPA [9] is an improved version of this attack; it produces adversarial noise for each input image, increasing success and perceptual quality. The main challenge in attacking IQA models is hiding the perturbations. Korhonen et al. [6] suggested a Sobel filter to hide adversarial distortions in textured regions. An invisible one-iteration (IOI) attack [10] also uses high-frequency filtering to mask perturbations. A subjective comparison proves this attack is invisible, even when applied to videos frame by frame. Zhang et al. [7] used Chebyshev distance, SSIM, LPIPS, DISTS, and other FR metrics to manage visual distortions.

2.3. Adversarial Training

Adversarial training is a popular way to improve the resistance of deep learning models to adversarial attacks. Each training step requires generating a perturbation, thus increasing the computational time. Researchers have used an early version of adversarial training, the Fast Gradient Sign Method (FGSM) [24], to generate perturbed examples. But adversarial training based on noniterative attacks such as FGSM only yields resistance to weak attacks; the model remains vulnerable to stronger iterative attacks. A way to overcome this limitation is adversarial training through multistep Projected Gradient Descent (PGD) [25], which achieves resistance to iterative attacks but can be computationally expensive. Wong et al. designed Fast adversarial training [26], which combines FGSM adversarial training and the random perturbation initialization of PGD attacks to accelerate training. This method is almost as effective as PGD-based training but has a much lower computational cost.

In their recent work, Singh et al. [27] generated adversarial examples using two-step Auto PGD (APGD-2) [28]. This adaptive attack realizes a higher loss than PGD in the same iteration budget, reducing both the number of iterations for adversarial training and the computational cost. For NR-IQA models, Korhonen et al. [6] applied adversarial training to make KonCept512, PHIQNet, and TRIQ more robust. These models exhibited lower correlation with subjective quality than the original ones did.

2.4. Robust IQA Models

Few papers thus far have focused on making particular IQA models more robust. Netflix introduced VMAF NEG (No Enhancement Gain) [29], which is more resistant to contrast enhancement than the original VMAF. VMAF utilizes support-vector regression based on fundamental low- and mid-level frame attributes. The NEG version entails regression over clipped values of base features. In particular, the authors introduced VIF and DLM enhancement-gain limits. These parameters allow limits on the maximum VIF and DLM scores, which are features of the VMAF support-vector regressor. Although this modification increases VMAF’s resistance to attacks [30], limiting base features in a fixed range only protects the model from extreme attacks—for example, increasing scores by 100–200%. An attacker can still increase the VMAF NEG score in its clipped range, however.

Kettunen et al. [31] proposed E-LPIPS, a modification of LPIPS [32]. It applies a series of random transformations to the image before passing the altered image through the neural network. It then calculates the average output over all inputs. This ensemble approach can make manipulating the results more difficult, but it also requires many model runs to generate a single prediction, limiting its real-world applicability. R-LPIPS [33] is another LPIPS variation. The authors left the network architecture unchanged and used adversarial training, during which time they applied adversarial perturbations to just one image in a triplet. They employed cross-entropy loss to generate the adversarial example.

Currently, there are very few studies focusing on improving the stability of NR-IQA models. Recently, Liu et al. [34] proposed Norm regularization Training (NT) method using $l_1$-regularization of the gradient norm to enhance the robustness of NR-IQA models against adversarial attacks.

3. Proposed Method

3.1. Problem Setting

Adversarial attacks on IQA models. Our investigation assumes an attacker is trying to increase the IQA model scores by adding small perturbations to images. Recent papers have examined these attacks, which may occur more frequently in the real world than other attacks. Attempts to decrease an image’s estimated quality are all the same up to the attack direction sign. Therefore, this approach avoids violating our study’s generality.

For a given NR-IQA model $f_{\theta }:X\to \mathbb{R}$ parameterized by the vector of weights $\theta$, where $X\in {[0,255]}^{3\times H\times W}$ is a distribution of images, we can mathematically describe an adversarial attack on an input image x with a perturbation $\delta$ bounded by $\epsilon$ with a norm of p as follows:

$$\underset{{\parallel \delta \parallel }_p\le \epsilon }{max}{f}_{\theta }(x+\delta ).$$

(1)

An attack on FR-IQA models can be reformulated as an attack on NR-IQA by manipulating just one of the reference and distorted images. For a given FR-IQA model $f_{\theta }$ parameterized by $\theta$ which takes a reference image $\widehat{x}\subseteq {[0,255]}^{3\times H\times W}$ and distorted image $x\subseteq {[0,255]}^{3\times H\times W}$ as an input, an adversarial attack with a perturbation $\delta$ bounded by $\epsilon$ of norm p is formulated as follows:

$$\underset{{\parallel \delta \parallel }_p\le \epsilon }{max}{f}_{\theta }(\widehat{x},x+\delta ).$$

(2)

Adversarial training for IQA models. Given a NR-IQA or FR-IQA model $f_{\theta }$ parameterized by $\theta$; a distribution of the training data D, where each training sample contains an image $x\subseteq {[0,255]}^{3\times H\times W}$ and associated quality label $y\in \mathbb{R}$; and a loss function $\mathcal{L}$ of the model $f_{\theta }$, vanilla adversarial training is a min-max problem where the inner maximization problem aims to discover strong adversarial examples $x+\delta$ bounded by the allowable perturbation magnitude $\epsilon$ of a norm p. The outer minimization problem fine-tunes the model parameters as follows:

$$\underset{\theta }{min}{\mathbb{E}}_{(x,y)\sim D}\left[\underset{{\parallel \delta \parallel }_p\le \epsilon }{max}\mathcal{L}(f_{\theta }(x+\delta ),y)\right]$$

(3)

As we mentioned in our related work discussion (Section 2), the perceptual quality of perturbed images may change, leading to different quality labels. We therefore add another term to the objective function:

$$\underset{\theta }{min}{\mathbb{E}}_{(x,y)\sim D}\left[\underset{{\parallel \delta \parallel }_p\le \epsilon }{max}\mathcal{L}(f_{\theta }\left(x\right),y)+\mathcal{L}(f_{\theta }(x+\delta ),y^{\prime })\right],$$

(4)

where y is the original subjective quality score and $y^{\prime }$ is the adjusted subjective quality score.

3.2. Method

We propose the following improvements to the vanilla adversarial training procedure to adapt it for IQA models.

Pre-training and fine-tuning. Many IQA models extract deep features from convolutional neural networks pre-trained on the ImageNet dataset to perform image classification [17,18,19,20,32]. Such transfer learning enables the receipt of more stable features than training on smaller datasets labeled for IQA. Because adversarial training is a nontrivial optimization problem, we propose initializing it from pre-trained standard IQA model weights and then fine-tuning them on clean and perturbed images. Doing so allows us to make the model resistant to adversarial perturbations while keeping a high correlation with subjective quality—a feature we demonstrated in our experiments.

Subjective scores correction. obtaining a subjective quality score for perturbed images during each training epoch would be extremely expensive. Therefore, we propose three strategies to correct the subjective quality scores of clean images after adding adversarial perturbations.

Minimal MOS. The simplest approach to adjusting scores for adversarial images is to set the minimum value for the entire dataset D:

$$y^{\prime }=\underset{y\sim D}{min}y$$

(5)

Percentage-based penalty. Another strategy involves penalizing the original subjective evaluation with a constant. During training, adversarial images receive an assigned target value as follows:

$$y^{\prime }=y-p\times \mathrm{diam}\left(D\right),$$

(6)

where p is 5/100 or 10/100 in our experiments, and $\mathrm{diam}\left(D\right)={\displaystyle \underset{y,z\in D}{sup}}\left\{\right|y-z\left|\right\}$. This approach’s complexity lies in the emergence of a new hyperparameter, which may be inconvenient in practice.

FR-based penalty. We extended our approach by penalizing subjective quality labels with FR metrics such as SSIM [12], MS-SSIM [35], and PSNR. Also, we incorporated penalties based on the trainable metric LPIPS [32], which emphasizes perceptual similarity. The formula for the target-value assignment then becomes the following:

$$y^{\prime }=y-\mathit{norm}(M(x,x+\delta ),M)\ast y,$$

(7)

where $M\in \{$SSIM, MS-SSIM, PSNR, LPIPS}. Here, norm denotes a metric-dependent mapping of the original values that transforms the range of M to the closed interval $[0,1]$:

$$\mathit{norm}(x,M)=\left\{\begin{array}{c}1-x,\mathrm{if}\mathrm{M}\mathrm{is}\mathrm{SSIM}\mathrm{or}\mathrm{MS}-\mathrm{SSIM},\hfill \\ {\displaystyle \frac{75-x}{45}},\mathrm{if}\mathrm{M}\mathrm{is}\mathrm{PSNR},\hfill \\ x,\mathrm{if}\mathrm{M}\mathrm{is}\mathrm{LPIPS}.\hfill \end{array}\right.$$

(8)

Attack methods and magnitudes. The choice of attack for generating adversarial examples during training is crucial to adversarial learning’s effectiveness. Previous research shows that models trained using PGD are resistant to weaker attacks. However, choosing a stronger attack for training may cause the model’s performance to decrease more. Therefore, we applied three attacks previously used for adversarial training of image classifiers [24,26,27]: a one-step FGSM attack [24]; a one-step PGD (PDG-1) attack, which is an FGSM with an initial random perturbation [26]; and an adaptive Auto PGD (APGD-2) attack [28], for which we used two iterations. For FR-IQA models, we propose attacking both distorted images simultaneously for greater strength compared with only one image, as in [33] suggests. Additionally, we examined the impact on model robustness of the adversarial-perturbation magnitude during training. Previous works on adversarial training commonly consider a single value for the perturbation magnitude $\epsilon$. Our study explores different $\epsilon$ values from the set {2, 4, 8, 10} / 255.

The complete proposed training procedure is summarized in Algorithm 1 and Figure 1. Our ablation study (Section 5.1) shows the impact of each component of our adversarial training.

Algorithm 1 Proposed adversarial training for T epochs, given attack magnitude $\epsilon$, step size $\alpha$, and dataset D for pre-trained NR-IQA model $f_{\theta }$, loss function $\mathcal{L}$, and FR metric M.

for $t=1\dots T$ do    for $(x_i,y_i)\in D$ do       // PGD-1 adversarial attack:       $\delta =Uniform(-\epsilon ,\epsilon )$       $\delta =\delta +\alpha ·sign\left({\nabla }_{\delta }\mathcal{L}(f_{\theta }(x_i+\delta ),y_i)\right)$       $\delta =max(min(\delta ,\epsilon ),-\epsilon )$       // Subjective score correction:       $y_i^{\prime }=y_i-\mathit{norm}(M(x_i,x_i+\delta ),M)\ast y_i$ // norm is calculated according to Equation (7)       $x_i=concatenate(x_i,x_i^{\prime })$       $y_i=concatenate(y_i,y_i^{\prime })$       $\theta =\theta -\beta ·{\nabla }_{\theta }l(f_{\theta }(x_i+\delta ),y_i)$// Update model with some optimizer and step size β    end for end for

4. Materials and Methods

4.1. Experimental Setup

NR-IQA datasets and models. We used KonIQ-10k [17], a popular IQA dataset that commonly serves for training and evaluating IQA models, to perform adversarial training of the NR-IQA model. This dataset comprises 10,073 images paired with subjective quality values based on Mean Opinion Scores (MOSs). To add an extra evaluation layer, we also used the NIPS2017 [36] dataset, which serves in attack development and testing and contains 1,000 images without MOSs. The decision to incorporate the NIPS2017 dataset in our experiments was to validate model robustness across varied datasets, extending beyond the scope of their original training domain.

We evaluated our method using four NR-IQA models originally developed using KonIQ-10k and showed a high correlation with subjective quality in public benchmarks. KonCept512 [17] was the first IQA model trained on the KonIQ-10k dataset. WSP-IQA [19] employs weighted spatial pooling (WSP) instead of global average pooling (GAP) to aggregate spatial information from different-size weight maps. HyperIQA [18] uses a hyper-convolutional network to predict the weights of fully connected layers. LinearityIQA [20] introduces the norm-in-norm loss function, which enables faster convergence than MSE or MAE when serving with the ResNet architecture.

FR-IQA datasets and models. For full-reference assessment we enhanced LPIPS [32]. . Among learning-based FR-IQA models that have proven vulnerable to adversarial attacks, it’s the most widely used option with an open training dataset [31,37]. Training of the original model version used the BAPPS [32] dataset, so our approach involves modifying the training process using this dataset. BAPPS consists of 36,344 image triplets: one image is the reference and the other two are distorted. Each triplet has two alternative forced-choice (2AFC) scores, indicating which of the two distorted images is more similar to the reference image. Additionally, we used the KADID-10k dataset [38], which contains 10,125 distorted images derived from 81 reference images. Each distorted image is associated with a subjective quality score based on MOS.

4.2. Evaluation Metrics

Performance evaluation. We used the Spearman rank order correlation coefficient (SROCC) to measure the correlation of NR-IQA model scores with subjective ratings. For FR-IQA models, we measured two-alternative forced choice (2AFC), which Zhang et al. [32] used.

Robustness evaluation. We applied FGSM and PGD attacks with 10 iterations to test adversarially trained models. Our evaluation of IQA model robustness employed two metrics.

Robustness score (R) [7] is the average ratio of the maximum allowable quality-prediction change to the actual change over all adversarial images in a logarithmic scale:

$$R={\displaystyle \frac{1}{N}}\sum _{i=1}^{N}log\left({\displaystyle \frac{max\{{\beta }_1-f\left(x_i\right),f\left(x_i\right)-{\beta }_2\}}{|f\left(x_i\right)-f\left(x_i^{\prime }\right)|}}\right),$$

(9)

where f is an IQA model, $x_i$ and $x_i^{\prime }$ are, respectively, the clean and perturbed versions of $i^{th}$ image from the dataset; ${\beta }_1$ is the maximum MOS value; and ${\beta }_2$ is the minimum MOS value.

Integral Robustness Score (IR-score). We introduce a new method to measure the resistance of adversarially trained models to perturbations of various magnitudes. Unlike R, our integral robustness score (IR-score) accounts for attack success versus perturbation-budget dependency, as Carlini et al. recommended [39].

We generated adversarial examples with different perturbation sizes $\epsilon$ from {2, 4, 8, 10} / 255 for each image in the test set and each IQA model. We then obtained the IQA-model scores for images before and after the attack and performed min-max normalization. Our next step was to calculate the minimum and maximum scores for each IQA model using the training dataset. Afterward, we transformed the IQA-model scores into the unified domain using neural optimal transport [40]. This step is necessary because the scores of different IQA models are nonuniformly distributed. We therefore trained a small neural network with four linear layers and one-dimensional input, as in [1]. suggested. Our experiments used LinearityIQA as a unified domain. Next was computing the absolute gain: the score difference for each image pair before and after the attack. We performed the abovementioned procedure for the original and adversarially trained models. The final step was to calculate the IR-score as the difference between two integrals (for the original and adversarially trained models) of absolute gain-$\epsilon$ curves averaged over all test-dataset images. Formally, for a given IQA model f parameterized by $\theta$ and adversarially trained model $f_{AT}$ parameterized by ${\theta }_{AT}$, the IR-score for a set of images ${\left\{x_i\right\}}_{i=1}^N$ is the following:

$$\begin{array}{c}\hfill \mathrm{IR}-\mathrm{score}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\int }_{\epsilon }\left(\widehat{f}\left(x_i\right)-\widehat{f}(x_i+\delta (\theta ,\epsilon ))-{\widehat{f}}_{AT}\left(x_i\right)+{\widehat{f}}_{AT}(x_i+\delta ({\theta }_{AT},\epsilon ))\right)d\epsilon ,\end{array}$$

(10)

where $\delta (\theta ,\epsilon )$ is the attack perturbation with a magnitude of $\epsilon$ applied to the model parameterized by $\theta$, and $\widehat{f}\left(x_i\right)$ is a metric score mapped by neural optimal transport (NOT) [40]:

$$\begin{array}{c}\hfill \widehat{f}\left(x_i\right)\leftarrow \mathrm{NOT}\left(\mathrm{diam}\left(f_{\theta }\left(x_i\right)\right)\right).\end{array}$$

(11)

4.3. Implementation Details

Pretraining of each IQA model employed the settings described in the source articles. Subsequently, we conducted 30 fine-tuning epochs for NR-IQA models, applying our proposed method with the hyperparameters listed in the supplementary materials. LPIPS involved five epochs using the Adam optimizer with a learning rate of ${10}^{-4}$ as well as five epochs with a linearly decreasing learning rate. To train their LPIPS model, the authors of [41] chose the features of various classification models trained on ImageNet. Following the R-LPIPS researchers [33], we used the AlexNet features.

Selection of the best model first required analysis of the SROCC for WSP-IQA, HyperIQA, and LinearityIQA and of the Pearson linear correlation coefficient (PLCC) for KonCept512, in accordance with the original paper’s recommendation [17]. We measured correlation coefficients on the validation subset after each training epoch and saved the best-performing model. During adversarial retraining, we saved the best model after three epochs, as we expected to obtain a more robust model exhibiting a natural decrease in correlation with subjective scores over more epochs. All our experiments with NR-IQA models were performed using eight Nvidia Tesla A100 80-Gb GPUs, while experiments with FR-IQA models used four NVIDIA RTX A6000 GPUs.

5. Results

5.1. Ablation Study

Role of pre-training and training data in adversarial robustness. Table 1 shows pre-training’s impact on the performance and robustness of the adversarially trained LinearityIQA as measured by the SROCC and IR-score. Similarly, Table 2 shows the performance and robustness of the adversarially trained FR-IQA model LPIPS. AT means vanilla adversarial training with an FGSM attack, +clean means concatenated batches of clean and adversarial data, +pretr. means pre-training. For training, we set the allowable perturbation magnitude $\epsilon$ to 4/255.

Although vanilla adversarial training for classification typically involves training the model solely on perturbed data, our experiments showed this approach is ineffective for IQA models. Adversarial training of these models without pre-training causes the model’s correlation with subjective quality to decline sharply.

Subjective score correction. Table 3 shows that assigning minimal MOS labels and PSNR-based score correction produces a greater decrease in correlation with subjective quality on clean images, a phenomenon we want to avoid. The best correlations are for fixed penalties of 5% and 10%, followed by FR-based penalties on LPIPS and SSIM. A fixed rate, however, will require manual selection of a penalty percentage, that may depend on image content, its corruption level and other properties. In total, LPIPS-based penalty shows better IR-score than fixed rate, keeping high correlation with original quality labels.

Impact of attack type during training. Table 3 shows the correlation between model predictions and subjective-quality scores for the original NR-IQA model LinearityIQA and its adversarially trained versions. The results showed that NR-IQA models trained with a PGD-1 attack correlated better with subjective quality. PGD-1 is a stronger FGSM version requiring one iteration of forward and backward passes. Interestingly, use of a stronger attack during image-classifier training decreases the model’s accuracy on clean images [25,26]. Table 4 presents the results for different versions of the FR-IQA model, LPIPS. More-sophisticated attacks increased robustness and decreased SROCC.

Impact of perturbation magnitude during training. Figure 2 illustrates the impact of the allowable perturbation values on the NR-IQA model’s correlation and robustness. The upper row of plots illustrates the resistance of each model to a simple FGSM attack. The lower row illustrates the same models’ resistance to a stronger PGD attack with 10 iterations. Models trained using a lower $\epsilon$ value correlate less well with MOS values, but during evaluation, they become more resistant to a stronger attack. LPIPS exhibits a similar effect, as Table 4: smaller magnitudes are used while adversarial training yields better resistance to PGD.

5.2. Overall Results

Our final adversarial training method employed the best strategies according to our experiment results noted in the ablation study (Section 5.1): we fine-tuned pre-trained IQA models on clean and attacked images using the PGD-1 attack with $\epsilon =2/255$; for NR-IQA models, we computed a penalty for subjective scores using the SSIM metric. Table 5 compares the original NR-IQA models with the robust versions from vanilla adversarial training, Norm regularization Training, and our proposed method. Our adversarially trained models demonstrate better resistance than the base and NT versions. Moreover, the SROCC decrease of models trained with our process is no more than 1.8% — a substantial improvement over vanilla adversarial training.

Table 6 shows the adversarial robustness of the FR-IQA LPIPS model trained with different approaches. Our approach to attacking two images simultaneously demonstrates improved robustness while insignificantly reducing the SROCC compared to vanilla adversarial training used in R-LPIPS.

For each final model, we provide information on training time. Although our method for NR-IQA requires more GPU hours than previous methods, this does not affect inference time.

6. Discussion

Applicability of proposed method to other adversarial attacks. We tested the robustness of the adversarially trained NR-IQA Linearity model to 16 other attacks. Appendix A provides the details about attacks and their hyperparameters. Figure 3 shows the R values for each pair of attack and version of IQA model. The proposed approach improves NR-IQA model resistance to almost every attack, except AdvCF [42]. This is because the training process used the noise attack, and models were therefore intended to be resistant to perturbations rather than color distortion which is how AdvCF works. Also, the SSIM-based label penalty chosen in the ablation study as the best strategy showed the best results in this experiment since it provided the best robustness for 13 of 16 unforeseen attacks.

Performance on clean images. Since vanilla adversarial training decreases model performance on clean data, we tested the performance of our adversarial training method when images did not contain adversarial noise. This property is more pronounced for IQA models because we added adversarial noise to images during training. The proposed approach helped reduce this effect of NR-IQA models: it decreased the correlation by less than 1.8% for four NR-IQA models. For LPIPS, our approach yielded 4.6% lower SROCC compared with the original LPIPS and less than 1% compared with R-LPIPS.

Empirical probabilistic estimation. When comparing the robustness of adversarially trained models, we launched each attach once and varied only the perturbation norm. We conducted an empirical study involving 100 attack launches for each parameter set to show that our adversarially trained models provide statistically significant improvement. Then, we used the Wilcoxon signed-rank test with the alternative hypothesis that the attack gains for the original and adversarially trained models differ. The test yielded a p-value of 0, confirming that adversarial training enhances model resistance to adversarial attacks. Details of the experiment appear in the Appendix B.

7. Conclusions

This paper presents a novel adversarial training method for IQA models. Unlike the vanilla approaches created for computer-vision models, our method is for IQA specifically, a feature that allowed us to create an IQA model that is robust to adversarial attacks without significantly reducing correlation with human image-quality perception. We conducted a comprehensive ablation study to determine the best adversarial training components for IQA. We also introduced a new evaluation strategy for our method and compared the robustness of IQA models. Our results showed that the best approach is to fine-tune IQA models on clean and adversarial images with a low perturbation level ($\epsilon =2/255$ in our experiments). In the case of NR-IQA models, the adjusted perceptual-quality scores for perturbed images generated during adversarial training maintain a high correlation with subjective quality. For FR-IQA LPIPS model, our approach of adding perturbations to two images during training improves robustness relative to other methods.

Future work. This study primarily focused on applying attacks during training with a restriction on the $l_{\infty }$ norm. An interesting variation would be to explore perceptually constrained attacks for quality metrics.