FFA: Foreground Feature Approximation Digitally Against Remote Sensing Object Detection

1. Introduction

The development of remote sensing (RS) imaging techniques has led to the advancement of many important applications in RS, including image classification [1,2,3], image segmentation [4,5], object detection [6,7,8], and object tracking [9,10]. Recent studies have found that image-processing techniques based on deep learning exhibit greater potential than traditional image-processing techniques, which makes them a hot research direction and leads to the current implementation of these tasks using deep neural networks (DNNs).

Recent research has demonstrated that DNNs are susceptible to certain perturbations, and the results of detectors can be altered by adding a perturbation to an image that is unrecognizable to the human eye, despite the fact that deep learning has achieved increased success [11,12]. This phenomenon reveals the great security risks of DNNs and also inspires people to research their security. Adversarial examples (AEs), which can be created by subtly altering the source image to increase the success rate of attacks, were initially identified by Szegedy et al. [13]. Since then, more and more people have been devoted to studying the vulnerabilities of DNNs and improving their security and robustness.

Object detection [14], as an important research area in image processing, has likewise given birth to some important applications, such as autonomous driving [15,16] and intelligent recognition systems [17,18], and naturally has yet to escape the attack of AEs. Compared with attacking image classification, attacking object detection is more difficult [19]. In image classification, each image contains only one category, which is easier to realize. In object detection, in which each image contains multiple objects, it is necessary to find out the object as well as determine its location and lock its size, which leads to object detection being more practical but more difficult to attack. Therefore, it is more meaningful to attack the object detection system and study its vulnerabilities.

Adversarial attacks can be classified into digital and physical attacks, depending on the domains in which they are applied [19]. Digital attacks mainly make the detector detect errors by changing the image pixels, which is applicable to all images, such as the DFool attack proposed by Lu et al. [20] and the DAG attack proposed by Xie et al. [21]. Physical attacks mainly make the detector misclassify by adding patches to the objects, mainly for images with a large percentage of objects. Kurakin et al. [22] first experimentally verified that AEs are also present in the actual world, and Thys et al. [23] successfully spoofed a pedestrian monitor by generating adversarial patches.

Remote sensing images (RSI), with their high resolution and wide spatial distribution, add some difficulties for object detection and likewise lead to their difficulty in realizing physical attacks  [24,25]. Therefore, the majority of the current attacks on RSI are digital attacks, and a few physical attacks are limited to RSI with a large percentage of targets. Czaja et al. [24] first investigated the AEs in RSI and successfully attacked the image classification models. Lu et al. [25] successfully designed an adaptive patch to attack the object detectors (ODs) for RSI. Du et al. [26] and Lian et al. [27] successfully applied adversarial patches against RSI to the physical world. More and more people have been devoted to the work of RS adversarial attacks.

The work of Xu et al. [3] and Li et al. [19] inspires us. Xu et al. [3] investigated the generic AEs for RSI classification models, and Li et al. [19] devised a method for adaptively controlling the perturbation size and location based on the behaviour of the ODs. It is well known that object detection has three main tasks: determining the object location, determining the object size, and determining the object category [14]. These are indispensable, and the attack can be realized by successfully attacking any one of the three tasks. Determining the object category is the image classification task, so we choose to attack the classifier in the ODs. We found that for object detection, using global adversarial perturbation will result in a significant resource waste since the objects in the RSI only represent a minor fraction of the entire image, and the background accounts for a much larger portion than the foreground, so we use local adversarial perturbation, which only changes the pixels in the foreground portion.

We suggest a digital attack method in this paper, which we call Foreground Feature Approximation (FFA), which mainly utilizes the backbone network of the ODs (feature extraction network) to attack the classifier. As an important part of the detection task and is most effective in utilizing it for the attack. The attack effect of FFA is shown in Figure 1.

The contributions of this paper are as follows:

• We propose the FFA attack, which uses the high-quality prediction of the ODs as the foreground, extracts shallow features using the ODs’ backbone network, and approximates the features of the input foreground image towards the features of the hybrid foreground image only by changing the pixels in the foreground part of the image, which achieves the targetless attack;
• We replaced the hybrid image with an image of the specific class, constructed the target foreground based on the high-quality prediction of the ODs, extracted the shallow features of the target foreground using the backbone network as well, and later approximated the input foreground features towards the target foreground features to achieve the targeted attack;
• The results of attacks on seven rotating ODs on the RSOD datasets DOTA and UCAS-AOD demonstrate that our method can produce AEs with higher attack capability, higher transferability, and higher imperceptibility.

2. Related Work

We touch on the theory of adversarial attacks in this section, including those used in image classification, object detection and RS.

2.1. Adversarial Attacks in Image Classification

Image classification is a critical task in machine learning, the goal of which is to enable the computer to assign input images to the real categories [28]. Given an input image $\mathit{x}$, whose true label is y, and the model’s prediction function is $f(·)$. The task of correctly classifying the image is even if the model’s prediction is equal to the true label of the target, that is $f\left(\mathit{x}\right)=y$. Generating AEs ${\mathit{x}}^{\prime }$ for an image classification model requires generating an adversarial perturbation $\mathit{\delta }$, then ${\mathit{x}}^{\prime }=\mathit{x}+\mathit{\delta }$. By limiting the size of the perturbation, it is possible to generate AEs that appear unchanged to the human eyes but that the computer cannot correctly classify, even if $f\left({\mathit{x}}^{\prime }\right)\ne y$. The above method is an targetless attack, which only requires that the computer be rendered incapable of correct classification. Adversarial examples also enable targeted attacks, also known as directed attacks, where the goal is to make the computer assign our carefully designed AEs to a particular class, assuming that the label of the particular class is p, and $f\left({\mathit{x}}^{\prime }\right)=p\ne y$.

In the beginning, people studied white-box attacks [29,30,31]. This method assumes that the structure and parameters of the models we want to attack are known, and when we attack according to this known information, the attack success rate of these methods is higher. For example, the optimization-based method L-BFGS [13] and gradient-based method FGSM [32] are white-box attack methods. However, in practice, it is difficult to get the framework and settings for the target models, so black-box attacks arise [33]. A black-box attack assumes that the framework and settings for the target models are unknown, and we need to discover the vulnerability of the target models through the known models and other information to realize the attack. The attack success rate of this method is generally low but has a high value of use. For example, transfer-based attacks Curls&Whey [33], FA [2], etc., query-based attacks ZOO [34], MCG [35], etc. and decision boundary-based attacks BoundaryAttack [36], CGBA [37], etc. are black box attack methods.

2.2. Adversarial Attacks in Object Detection

Object detection, an advanced task of image classification and segmentation, occupies an increasing share in computer vision and is also applied in task scenarios such as face recognition [38], unmanned vehicles [39], aircraft recognition [40], terrain detection [41], etc. The goal is for the computer to find the part of interest in an image and give it the correct classification, location, and size. The emergence of AEs also puts higher demands on the security of object detection.

Xie et al. [21] first discovered the existence of AEs in object detection and semantic segmentation tasks. They proposed the DAG attack with good transferability between multiple datasets and different network structures. The target of the attack is placed in the region of interest (RoI), and the attack expression is:

$${\mathit{\delta }}_m=\sum _{t_n\in T_m}[{\nabla }_{X_m}{f}_{l_n^{\prime }}(X_m,t_n)-{\nabla }_{X_m}{f}_{l_n}({\mathrm{X}}_m,t_n)],$$

(1)

$${\mathit{x}}_{m+1}={\mathit{x}}_m+{\displaystyle \frac{\alpha }{{∥{\mathit{\delta }}_m∥}_{\infty }}}{\mathit{\delta }}_m,$$

(2)

where ${\mathit{\delta }}_m$ denotes the perturbation generated by the mth iteration, ${\mathit{x}}_m$ denotes the AE generated by the mth iteration. $t_n$ denotes a certain object among all objects $T_m$ in the image, $f(·)$ denotes the result function of object detection, $l_n$ refers to the true category label of the attacked object, and $l_n^{\prime }$ refers to the category label specified by the attack. The perturbation is also normalized in order to limit the range of the perturbation. $\alpha$ denotes the iterative learning rate, which controls the range of the perturbation and is a fixed hyperparameter value.

Lu et al. [20] proposed the DFool attack to construct AEs against intersection stop signs in videos to successfully attack the then more advanced Faster RCNN [42] model and implement the attack in the physical world as well. It generates AEs by minimizing the average score produced by Faster RCNN for all stop signs in the training set. The average score of stop signs is:

$$\mathsf{\Phi }\left(\mathcal{T}\right)={\displaystyle \frac{1}{N}}\sum _{i=1}^{N}b\in B_s\left({\mathcal{I}}^{mean}({\mathcal{M}}_i,\mathcal{T})\right){\mathsf{\Phi }}_s\left(b\right),$$

(3)

where $\mathcal{T}$ denotes the texture of the stop sign in the root coordinate system, $\mathcal{I}({\mathcal{M}}_i,\mathcal{T})$ denotes the image $\mathcal{I}$ formed using the mapping ${\mathcal{M}}_i$ superimposed on the $\mathcal{T}$. ${\mathcal{M}}_i$ is the mapping formed by the correspondence between the eight vertices of the stop sign and the vertices of the $\mathcal{T}$. $B_s\left(\mathcal{I}\right)$ is the set of stop signs detected by the image $\mathcal{I}$ through the Faster RCNN, b is a certain detection box, and ${\mathsf{\Phi }}_s\left(b\right)$ denotes the score predicted by the Faster RCNN for the detection box.

Afterwards a small gradient descent direction is used for optimization:

$${\mathit{\delta }}^{\left(n\right)}=\mathrm{sign}\left({\nabla }_T\mathsf{\Phi }\left({\mathcal{T}}^{\left(n\right)}\right)\right),$$

(4)

$${\mathcal{T}}^{(n+1)}={\mathcal{T}}^{\left(n\right)}+\epsilon {\mathit{\delta }}^{\left(n\right)},$$

(5)

where sign(·) denotes the sign function, which is used to control the size of the perturbation, ${\mathcal{T}}^{(n+1)}$ denotes the AE generated by the n+1st iteration. $\epsilon$ denotes the iteration step size and ${\mathit{\delta }}^{\left(n\right)}$ denotes the adversarial perturbation generated by the nth iteration.

Both of the above methods are global perturbation attacks [3,43], i.e., each pixel in the image is changed to generate the AE. The defects of these methods are very obvious. They change the foreground and background at the same time, resulting in a waste of attack resources and prolonging the attack time. Based on this, researchers have discovered local perturbation attacks [19,44] and gradually developed them as patch attacks [23,27] in the physical world.

The Dpatch attack was put forward by Liu et al. [45], which centres on generating a patch with the ground truth detection box property and optimizing it into a real object through backpropagation. In addition, the OD is made to ignore other possible objects. For two-stage ODs, such as Faster RCNN [42], the idea of the attack is to make the RPN network unable to generate the correct candidate network so that the generated patch becomes the only object. For single-stage ODs, such as YOLO [46], the core of which is the bounding box prediction and confidence scores, the attack is to make the grid where the patch is located be regarded as an object and ignore other grids. The objective function is optimized by training the patches, and the resulting patches are:

$$\widehat{p}=arg\underset{p}{max}{E}_{x,t,l}[log\mathcal{P}\left(\widehat{y}\right|\mathcal{A}(\mathit{p},\mathit{x},l,t))],$$

(6)

where $\mathcal{A}(\mathit{p},\mathit{x},l,t)$ represents that the input image $\mathit{x}$ places the patch $\mathit{p}$ on l through a transformation t, the transformation t includes basic transformations such as rotation scaling. $\mathcal{P}\left(\widehat{y}\right|\mathcal{A})$ represents the likelihood that the input $\mathcal{A}$ is associated with the label $\widehat{y}$. The principle of Dpatch is to maximize the loss of the ODs to the true label and the bounding box label.

Thys et al. [23] designed the first printable adversarial patch for pedestrians, which achieved better attack results against the YOLO V2 OD in the physical world. Their optimization objective consists of three parts.

One is the non-printable score $L_{nps}$, which donates whether the patch can be printed using a printer:

$$L_{nps}=\sum _{p_{\mathit{patch}}\in P}\underset{c_{\mathit{print}}\in C}{min}\left|p_{\mathit{patch}}-c_{\mathit{print}}\right|L_{nps},$$

(7)

let $p_{\mathit{patch}}$ represent a pixel in P, and $c_{\mathit{print}}$ represent a colour in a set of printable colours in C. This loss benefits colors in the image that closely resemble the colors in the set of printable colors.

The second is the total cost $L_{tv}$ in the image, which is used to ensure a smoother transition between the patch and the image:

$$L_{tv}=\sum _{m,n}\sqrt{{(p_{m,n}-p_{m+1,n})}^2+{(p_{m,n}-p_{m,n+1})}^2},$$

(8)

The third is the maximum objective score $L_{obj}$ of the image, which represents the score assigned to the object detected by the detector. The three are combined according to reasonable weights, and adversarial patches are generated iteratively.

2.3. Adversarial Attacks in Remote Sensing

Adversarial attacks have been investigated in ground imagery but they also have numerous uses in RS. Li et al. [19] designed an adaptive local perturbation attack method aiming at generalizing the attack to all ODs, which establishes an object constraint to adaptively utilize foreground-background separation to induce perturbations as a way of controlling the magnitude and distribution of perturbations so that perturbations are attached to the object region of the image. It employs three loss functions: classification loss, position loss, and shape loss, to generate the AE, and the three parts of the loss are rationally combined using a weighted approach.

$$L=\sum _{n=1}^N\alpha {\mathcal{L}}_{shape}(b_n,b_n^{\prime })/N+\sum _{n=1}^N\beta {\mathcal{L}}_{loc}(b_n,b_n^{gt})/N+\sum _{n=1}^N\tau {\mathcal{L}}_{cls}(p_n,{\ell }_b),$$

(9)

where N denotes objects’ number in the image, ${\mathcal{L}}_{shape}$ denotes the shape constraint, ${\mathcal{L}}_{loc}$ denotes the location constraint, ${\mathcal{L}}_{cls}$ denotes the classification constraint, and $\alpha ,\beta ,\tau$ refers to the weights of the three constraints. Each object is assigned a random bounding box $b_n^{\prime }$ for shape attack, a real border $b_n^{gt}$ for localization attack and a background category ${\ell }_b$ for classification attack. In addition, the distribution of perturbations is adaptively controlled using the foreground separation method so that the perturbations are attached to the foreground as much as possible.

Lian et al. [27] investigated physical attacks in RS. They proposed the patch-based Contextual Background Attack (CBA), which aims to achieve steganography of the object in the detector without contaminating the object in the image. It places the patch outside the object instead of on the object, which is more conducive to the realization of physical attacks. An AE with an adversarial patch can be precisely described as:

$${\mathbf{x}}_{adv}=(1-{\mathcal{M}}_{{\mathbf{p}}^{*}})\odot \mathit{x}+{\mathcal{M}}_{{\mathbf{p}}^{*}}\odot {\mathbf{P}}^{*},$$

(10)

where ${\mathbf{P}}^{*}$ denotes the antipatch, ${\mathcal{M}}_{{\mathbf{p}}^{*}}$ denotes the mask of the antipatch, foreground pixels are 1 and background is 0, and ⊙ denotes the Hadamard product.

Objectivity loss and smoothness shrinkage are also used to update the adversarial patches. Objectivity loss refers to updating the AEs by detecting differences between objects in the inner and outer parts of the patch, and smoothing contraction reduces the distance between the patch and the target to make the transition smoother.

3. Methodology

3.1. Problem Analysis

To design a compliant AE, two aspects need to be considered (1) where the perturbation is placed; (2) how the perturbation should be designed.

3.1.1. Location of the Perturbations

To enhance the effectiveness of the perturbation, we need to choose a suitable location to add the perturbation. Compared with the global adversarial perturbation for the image classification task, it is not very suitable to design a global adversarial perturbation for the object detection task. Especially for RSI object detection, the number of objects is large, and the area occupied is small. In this case, we need to design a more useful perturbation generation strategy to limit the perturbation to a suitable range.

3.1.2. Magnitude of the Perturbations

After determining the location of the perturbation, the next step is to determine its design scheme. The principle of perturbation generation is to successfully deceive the ODs and ensure that human eyes cannot perceive them. Therefore, the design of the perturbation can be seen as a joint optimization problem by considering multiple factors and balancing multiple losses so that the final generated AEs are globally optimal rather than locally optimal.

The input picture is expected to be $\mathit{x}$ and there are n objects in the image. ${B^i}_n=(w_n,h_n,x_n,y_n,l_n,s_n)$ is the parameter of the ith object, which represents the width, height, centroid horizontal coordinate, centroid vertical coordinate, object label, and object score of the object box, respectively. When detecting rotated objects, the object box’s rotation angle parameter must be added. The goal of AE design is to identify an appropriate perturbation $\mathit{\delta }$ that satisfies:

$$arg\underset{\mathit{\delta }}{min}{P}^i(\mathit{x}+\mathit{\delta })\ne {B^i}_n,$$

(11)

where P denotes the detector, $P^i(\mathit{x}+\mathit{\delta })$ represents the image’s ith detection result, and the generated AE is $\mathit{x}+\mathit{\delta }$. It can be seen from this that there are three manifestations of a successful attack: first, the size of the detection box is incorrect; second, the location of the detection box is incorrect (including no detection box, i.e., the object is "invisible" in the detector), and third, the category of the detected object is incorrect. These three manifestations can be superimposed.

3.2. Overview of the Methodology

The topic of attacks against ODs has been extensively studied. Since ODs contain several important components, attacks against these specific components are effective. Still, they also pose a huge problem at the same time: specific ODs contain specific frameworks and components, which prevent the generalization of the attacks [19]. In order to improve the transferability of attacks, we need to discover the similarities between different ODs. All ODs are inseparable from a common backbone network. The most commonly used backbone network is the ResNet network [8]. So we can change the features of the image for the attack and can form an AE with high transferability. The process of extracting features is visually represented in Figure 2.

We first use an OD to detect the image, separate the high-quality foreground and the background, make a record of the foreground data of the image, including the object category as well as the position and the object box size, and cut the image of all the object boxes as a backup. After that, we need to design a hybrid image with the principle that it can’t contain objects of either category, and its dimensions are identical to those of the original image. Utilize the backbone network for obtaining the features of hybrid foreground and high-quality foreground, design the body of the loss function by minimizing the KL-divergence between the latter and the former, and use the prediction result of the OD to perform the assisted attack. To accomplish the attack, the original image’s pixels should finally be altered so that the hybrid image’s features roughly match those of the original image. The flowchart of FFA attack is shown in Figure 3.

3.3. Foreground Feature Approximation (FFA) Attack

3.3.1. Targetless Attack

1. Location of the perturbation

For the object detection task, only the foreground section of the image, which is a small part, is relevant, while the background portion is irrelevant. Therefore, to attack the object detection, we only need to add perturbations to the portion of the ODs that is of interest. We use the attacked ODs to predict the image, get the prediction boxes and perform further filtering based on the prediction boxes scores:

$${\mathit{T}}_{bboxes}=\left\{\begin{array}{cc}{P}_{bboxes}\left(\mathit{x}\right),\hfill & score{s}_{max}>\alpha \hfill \\ 0,\hfill & score{s}_{max}<\alpha \hfill \end{array}\right.(0<\alpha <1),$$

(12)

where ${\mathit{T}}_{bboxes}$ is the object to be attacked, ${\mathit{P}}_{bboxes}$ is the prediction boxes of the ODs, $score{s}_{max}$ is the maximum value of the prediction scores for each category, $\alpha$ is the threshold of the prediction scores, the prediction boxes with prediction scores greater than $\alpha$ are selected as the object to be attacked, and the prediction boxes with prediction scores less than $\alpha$ are considered to have no value to be attacked, and we do not attack them.

2.The size of the perturbation

Most of the previous methods design perturbations based on the predictive information of ODs and construct a variety of loss functions based on the object boxes detected by the ODs as well as the object categories, including position loss, shape loss, category loss, and so on. These methods depend on the accuracy of the ODs and the fact that the generated AEs do not have high transferability. Based on this, we propose a feature approximation strategy to change the features of the image in the view of the backbone network to mislead the detectors and achieve the attack effect. The loss function is designed as follows.

Feature Loss: After selecting the foreground portion to attack, we need to design a suitable loss function to update the perturbation iteratively. Before starting, we need to generate a hybrid image $\tilde{\mathit{x}}$ to determine the direction of each iterative update. We choose ten images from the training set of the dataset to be superimposed, and the final image formed by the superimposition is a hybrid image that does not contain any object, which we call the full background image.

Using the prediction boxes obtained from the ODs, the input image ${\mathit{x}}_{adv}^i$ and the hybrid image $\tilde{\mathit{x}}$ of the ith iteration are cropped separately to obtain the input foreground ${\mathit{x}}_{adv\_f}^i$ and the corresponding hybrid image ${\tilde{\mathit{x}}}_f$. We define the feature loss function as the KL-divergence between the two images’ pixel distributions:

$${\mathcal{L}}_{feat}(\mathit{\theta },{\mathit{x}}_{adv\_f}^i,{\tilde{\mathit{x}}}_f)=-\sum _{R=1}^{N_R}\sum _{C=1}^{N_C}\sum _{K=1}^{N_K}{f}_{\mathit{\theta }}{\left({\mathit{x}}_{adv\_f}^i\right)}^{(R,C,K)}log{\displaystyle \frac{f_{\mathit{\theta }}{\left({\mathit{x}}_{adv\_f}^i\right)}^{(R,C,K)}}{f_{\mathit{\theta }}{\left({\tilde{\mathit{x}}}_f\right)}^{(R,C,K)}}},$$

(13)

where ${\mathcal{L}}_{feat}$ denotes the feature loss, $\mathit{\theta }$ denotes the backbone network parameters, and $N_R,N_C,N_K$ denotes the number of rows, columns, and channels of the feature map, respectively. $f_{\mathit{\theta }}(·)$ denotes the mapping function for extracting shallow features using the backbone network. Note that the formula is preceded by a negative sign in order to minimize the KL-divergence of the clean and mixed foregrounds.

To improve the effectiveness of the attack, we also have to design the prediction loss to assist the attack based on the ODs’ prediction results. The classification loss and the object box loss are the two components of this loss.

Classification Loss: Let the AEs carry incorrect labelling information to mislead the classifier. Since the probability that the predicted object is background is calculated during the detector prediction process, it has a background label, which we assume to be $l_{bg}$. Assume that the prediction results in m objects ${\mathit{x}}_0,{\mathit{x}}_1,\cdots {\mathit{x}}_{m-1}$. Their labels are finding the true label of a prediction box by calculating the cross-entropy loss of the true box to the prediction box. By reducing the cross-entropy loss between the background category and the prediction box category, the attack is accomplished. The classification loss for each object is:

$${{\mathcal{L}}^j}_{cls}({\mathit{x}}_j,l_{bg})=\sum _{C=1}^{N_L}{l_{bg}}^{\left(C\right)}logP{\left({\mathit{x}}_j\right)}^{\left(C\right)},$$

(14)

where $N_L$ indicates the number of categories in all images, $P(·)$ is the mapping function of the ODs’ detection results, ${\mathit{x}}_j$ is the jth object, and the total classification loss for this image is:

$${\mathcal{L}}_{cls}(\mathit{x},l_{bg})=\sum _{j=1}^m{\mathcal{L}}_{cls}^j({\mathit{x}}_j,l_{bg}),$$

(15)

Object box loss: this loss is used to control the offset between the real object box and the prediction box. By increasing this offset, the gap between the detection result and the real result gets bigger and bigger to carry out the attack. Assuming that the centroid of the jth prediction box has the coordinates $O_j=(h_j,v_j)$, and the centroid of the real box corresponding to this prediction box has the coordinates $O=(h,v)$, the Euclidean distance $D_j$ between the two is calculated, and the object box loss is defined in terms of the Smooth L1 loss at this distance:

$${{\mathcal{L}}^j}_{box}=SL1\left(D_j\right)=\left\{\begin{array}{cc}0.5D_j^2,\hfill & D_j<1.0\hfill \\ D_j-0.5,\hfill & otherwise\hfill \end{array}\right.,$$

(16)

The object box loss for the entire image is then the sum of the object box losses for each prediction box:

$${\mathcal{L}}_{box}=\sum _{j=1}^m{{\mathcal{L}}^j}_{box}\left(D_j\right),$$

(17)

The prediction loss is the sum of the categorization loss and the object box loss, both of which are equally weighted:

$${\mathcal{L}}_{pre}={\mathcal{L}}_{cls}+{\mathcal{L}}_{box},$$

(18)

Perceptual Loss: In order to make the perturbation look more natural, we also need to control the range of the perturbation. The perceptibility of the AEs is reduced by minimizing the L2 distance between the AEs and the original images. Assuming that the ith iteration generates an adversarial example ${\mathit{x}}_{adv}^i$, we define the perceptual loss as:

$${{\mathcal{L}}^i}_{per}={∥{\mathit{x}}_{adv}^i-\mathit{x}∥}_2,$$

(19)

The ultimate loss function is a weighted combination of the three losses:

$$\mathcal{L}={\mathcal{L}}_{feat}+\beta {\mathcal{L}}_{pre}+\gamma {\mathcal{L}}_{per},$$

(20)

The AE generated in the ith iteration is:

$${\mathit{x}}_{adv}^i={\mathit{x}}_{adv}^{i-1}+{\nabla }_{\mathit{x}}{\mathcal{L}}^i,$$

(21)

where ${\nabla }_{\mathit{x}}{\mathcal{L}}^i$ denotes the gradient of the computational loss ${\mathcal{L}}^i$ with respect to the original image $\mathit{x}$.

Algorithm 1 gives the implementation details of the FFA to achieve the targetless attack.

Algorithm 1: Foreground Feature Approximation (FFA).

Input:        -Input image $\mathit{x}$ and its ground truth $B_n$;       -A object detector D with the prediction function $P(·)$ and the feature extraction       function $f(·)$;       -Hybrid image $\tilde{\mathit{x}}$.    Initialization:       1. ${\mathit{x}}_{adv}^0\leftarrow \mathit{x},I\leftarrow 50,\alpha \leftarrow 0.75,\beta \leftarrow 0.1\gamma \leftarrow 0.1.$       2. $T_{bboxes}\leftarrow P^D{\left(\mathit{x}\right)|}_{score{s}_{max}>a}$   Iteration:        for t in $range(0,I)$ do          1. ${\mathit{x}}_{adv\_f}^i\leftarrow T_{bboxes}\left({\mathit{x}}_{adv}^i\right),{\tilde{\mathit{x}}}_f\leftarrow T_{bboxes}\left(\tilde{\mathit{x}}\right)$          2. ${\mathcal{L}}^i\leftarrow \mathrm{Combine}\{{\mathcal{L}}_{feat}^i,{\mathcal{L}}_{pre}^i,{\mathcal{L}}_{per}^i\}$          3. ${\mathit{x}}_{adv}^i\leftarrow Update\{{\mathit{x}}_{adv}^{i-1},{\mathcal{L}}^i\}$   end for   return${\mathit{x}}_{adv}^I$

3.3.2. Targeted Attack

1. Location of the perturbation

As with the targetless attack, we use Equation (12) to obtain the objects $T_{bboxes}$ of the attack.

2.The size of the perturbation

The targeted attack is executed by deceiving the classifier in the detector, causing it to classify all of the objects in the image as the specific target. The loss function is designed as follows.

Feature Loss: To realize the targeted attack, we use an image of the target category instead of the original hybrid image and extract a target in the target image as a benchmark. Due to the large number of prediction boxes and their different sizes, we have to redesign the target foreground before performing feature approximation. First, create a blank image that is the same shape as the input image, identify the location and size of each prediction box, and resize the target image so that the two are of the same size, following which the target image is transferred into the corresponding position in the blank image to create a new target foreground ${\tilde{\mathit{x}}}_{f\_t}$. After that, the feature loss is computed:

$${\mathcal{L}}_{feat\_t}={\mathcal{L}}_{feat}(\mathit{\theta },{\mathit{x}}_{adv\_f}^i,{\tilde{\mathit{x}}}_{f\_t}),$$

(22)

Classification Loss: The design principle of targeted attack is to move the classification result of the classifier continuously closer to the target label, assuming that the target label is $l_t$. The classification loss is calculated as follows:

$${\mathcal{L}}_{cls\_t}={\mathcal{L}}_{cls}(\mathit{x},l_t),$$

(23)

Object box loss: in contrast to targetless attacks, the object box loss in targeted attacks is designed to allow the detection box to coincide with the real box gradually, and thus, the loss is designed to be:

$${\mathcal{L}}_{box\_t}=-\sum _{j=1}^m{{\mathcal{L}}^j}_{box}\left(D_j\right),$$

(24)

As with the targetless attack, the total of the object box loss and the classification loss is the prediction loss.

Perceptual Loss: As with the targetless attack, we construct the perceptual loss by minimizing the L2 distance between the clean image and the AE.

The ultimate loss function is a weighted amalgamation of the three loss functions. The adversarial perturbation is determined by calculating the gradient of the loss function in relation to the input image. Adding the perturbation to the input image creates the AE.

4. Experiments

4.1. Experimental Preparation

4.1.1. Datasets

We examine our method on two frequently used RSOD datasets, DOTA [47] and UCAS-AOD [48].

DOTA [47]: Since the original images of this dataset are too large, we cropped the original dataset to facilitate the processing. The whole dataset has more than 5000 PNG images with 1024×1024 pixels after cropping. After our filtering, we get 1500 images covering 15 categories. This research utilizes the DOTA dataset to validate the efficacy of FFA in targetless attacks.

UCAS-AOD [48]: This dataset contains a total of 1,510 images and 14,596 image instances of only two categories, aircraft and vehicles. The counterexample images in the dataset are of less value for the study of adversarial attacks, and thus, we do not consider them. As with the DOTA dataset, we cut the images and processed them to contain a total of 3000 plane images and 1,500 vrhicle images, each with pixels of 680×680. We filter the dataset finely and remove similar images to obtain 1,000 aeroplane images and 1000 car images, which constitute our experimental dataset. This dataset is used to validate the performance of FFA with targeted attack.

4.1.2. Detectors

In total, we selected seven rotated ODs as both training and attacked detectors, four for two-stage detectors, namely Oriented R-CNN (OR) [49], Gliding Vertex (GV) [50], RoI Transformer (RT) [51], and ReDet (RD) [52], and three in total for single-stage detectors, namely Rotated Retinanet (RR) [53], Rotated FCOS (RF) [54] and S2A-Net (S2A) [55]. For the feature extraction network, we use ResNet50 [8], hereafter referred to as R50. All the above models are implemented using the open-source MMrotate [56] toolbox.

4.1.3. Evaluation Metrics

For targetless attacks, we use the mean accuracy (mAP) to evaluate the attack effect of the FFA attack. As long as the detected object class is not a real class, the attack is judged to be successful. The smaller the mAP, the better the attack effect is. For targetless attacks, we use mAP and the number of detection boxes $N_t$ of the target class detected by the detector to validate the effect of the attack. The smaller the mAP, the bigger the $N_t$ indicates that the effect of the attack is better. The IoU threshold is set to 0.5. For image perceptibility, we use Information content weighted Structural SIMilarity (IW-SSIM) [57] to evaluate the AEs’ perceptibility. The smaller the value, the lower the perceptibility of the image perturbation and the better the effect. For observational comparison, we multiply both mAP50 and IW-SSIM data by 100. In addition, we assess the attack speed of various algorithms by calculating the average attack time of an image computed using an NVIDIA GeForce RTX 3060 (12GB) graphics card.

4.1.4. Parameter Setting

The iterations number I is set to 50, the learning rate is assigned a value of 0.1, and the threshold for high-quality prediction box filtering $\alpha$ is set to 0.75. For the targetless attack, the weight of feature loss is set to 1, and the weights $\beta$ and $\gamma$ of prediction loss and perceptual loss are set to 0.1. For the targetless attack, the weight of prediction loss $\beta$ is set to 1, and that of perceptual loss $\gamma$ is set to 0.1.

This study was implemented on the Pytorch platform.

4.2. Targetless Attack

The test of targetless attack is performed on the DOTA [47] dataset using seven rotating ODs, whose backbone networks are all ResNet50 [8]. We test the attack capability of the attacks using the mAP with an IoU threshold of 0.5 (hereafter referred to as ${\mathrm{mAP}}_{50}$), with smaller values indicating stronger attack capability. We test the imperceptibility of the perturbation using IW-SSIM [57], with smaller values indicating that the perturbation is imperceptible. We use the average attack time of each image to test the attack speed of all attacks; the smaller the value, the faster the attack speed. Table 1 illustrates the experimental outcomes. In order to facilitate the observation, we multiply the value of ${\mathrm{mAP}}_{50}$ with the value of IW-SSIM by 100 to get the data in the table.

From the data in the table, we can see that the attack capability of FFA is significantly stronger than the current state-of-the-art methods, especially since the attack transferability obtains a large improvement. The reason is that we designed a generic attack for the current commonly used backbone network by changing the characteristics of the image in the view of the backbone network, attacking the model’s attentional mechanism. Thus generating the adversarial perturbation with a more powerful attack. The imperceptibility of the perturbation generated by FFA is slightly weaker than the LGP attack due to the fact that LGP designs an adaptive perturbation region updating strategy to select more cost-effective regions to attack. On the other hand, FFA selects the attacked regions based on the prediction of the ODs and the real labels of the clean images, which implies that the ODs’ detection accuracy and the attacking effect of FFA are somewhat correlated, and it will inevitably select some cost-ineffective regions for attacking during the attacking process, which attenuates the imperceptibility of the perturbation. In terms of attack speed, FFA is slightly slower than LGP because FFA uses the feature approximation strategy, and the feature distance has to be calculated during the attack process, which increases the amount of computation. FFA sacrifices part of the imperceptibility and attack speed for the improvement of attack effectiveness.

The visualization results of the targetless attack are displayed in Figure 4. Based on the figure, it is evident that the FFA produces better attack results not only for large and sparse objects but also for small and dense objects.

4.3. Targeted Attack

4.3.1. White Box Attack Performance

Targeted attacks were tested on the UCAS-AOD [48] dataset using seven rotating ODs whose backbone networks are all ResNet50. We tested the method’s attack capability using ${\mathrm{mAP}}_{50}$ and the number of targets $N_t$ with the IoU threshold of 0.75 (hereafter referred to as $N_{t0.75}$), where a smaller value of ${\mathrm{mAP}}_{50}$ and a larger value of ${\mathrm{mAP}}_{50}$ indicate a stronger attack capability. The test results are shown in Table 2. From the table, it can be seen that FFA’s attack capability is significantly stronger than the existing advanced attack methods. This shows that FFA can realize a high success rate by changing the object features in the image.

4.3.2. Transferability Experiments

To verify the targeted attack transferability of FFA, we designed the following experiment. A two-stage OD OR [49] and a single-stage OR S2A [55] are selected as trained detectors to attack the remaining five detectors, with the original categories of planes and vehicles and the corresponding target categories of basketball courts and ground track fields, respectively. The evaluation metrics are ${\mathrm{mAP}}_{50}$ and $N_{t0.75}$. The empirical findings are displayed in the Table 3.

The data presented in the table demonstrates that FFA has better transferability compared to other methods. We also find that FFA has a stronger attack effect on simple models than on complex models. ${\mathrm{mAP}}_{50}$ of FFA for single-stage detectors is, on average, 2 percentage points lower than that of dual-stage detectors, while $N_{t0.75}$ is, on average, more than 200. In addition, the accuracy of the trained OD also affects the attack effect. Since the accuracy of the two-stage OD is slightly higher than that of the single-stage OD, the attack effect of the attack using the two-stage OD is stronger than that using the single-stage OD. At the same time, the success rate of targeted attacks on planes is significantly higher than that of vehicles because the ODs’ detection accuracy for planes is higher than that of vehicles.

4.3.3. Imperceptibility and Attack Speed Test

Similar to the targetless attack, we also tested the perturbation’s imperceptibility and the algorithm’s attack speed. The empirical findings are displayed in the Table 4.

The data presented in the table demonstrates that the perturbation imperceptibility type of FFA is slightly weaker, and the attack speed is slower than LGP. The reason is that we will sacrifice some of the perturbation imperceptibility with the attack speed to get a high-transferability attack. In addition, we also find that the success rate of attack against single-stage ODs is higher than that of two-stage ODs due to the simpler model structure and longer attack time of single-stage ODs, leading to better attack performance.

The visualization results of targeted attacks are displayed in Figure 5 and Figure 6. We can see that FFA has better attack results on different targets in different detectors.

4.4. Effect of Iteration Number

The number of iterations is a direct parameter that impacts the efficiency of the attack, so we assessed the influence of the iteration number using multiple evaluation metrics. The results are displayed in Table 5 and Figure 7.

From the results, we can see that only the attack time is positively correlated with the iteration number, and the more iteration numbers, the longer the attack time. The relationship between the attack effect and imperceptibility and the iteration number is more complicated: the attack effect is first enhanced and then weakened with the increase of iteration number, while the imperceptibility of the perturbation is exactly the opposite, first weakened and then enhanced. It shows that there is a contradiction between the attack effect and imperceptibility, and the stronger the attack effect is, the weaker the imperceptibility of the perturbation is. Since the results in the table show that the imperceptibility of the perturbation is still at a low level, taking into account, we set the iteration number of FFA to 50.

4.5. Ablation Experiment

In this section, we verify the effect of different losses used in the method on the attack efficacy. The quantitative results are displayed in Table 6, and the visualization results are displayed in Figure 8.

5. Conclusion and Discussion

This work explores a strategy for generating AEs for RSOD. Most existing methods directly attack the predictive information of the image to achieve the effect of fooling the ODs. Unlike these methods, we propose an attack method that utilizes the information of the image itself by changing the feature of the image in the view of the backbone. Specifically, we first use the OD to filter out high-quality predictions as the object to implement the attack, create a hybrid foreground without any target, and use KL-divergence to minimize the shallow features of the attack object and the shallow features of the hybrid foreground to achieve a targetless attack. Replace the hybrid foreground with the target foreground and repeat the above process to realize the targeted attack. Although this method is relatively simple, the results of attacking seven rotating ODs on two datasets show that this method can generate AEs with high transferability.

There are still some directions that can be improved in this study: (1) When selecting the object to implement the attack, we use the prediction scores and IoU for threshold screening, which can result in poor-quality attack objects, leading to the weakening of the effect of the attack, and it can be improved by adding a better screening strategy. (2) The imperceptibility of the perturbation is weak. More effective perturbation control methods can be set to increase the imperceptibility of the perturbation. (3) The attack speed of the algorithm is slow, and there are still some redundant calculations in the algorithm, which increases the amount of calculation, and a more efficient iteration method can be designed to reduce the amount of calculation.