C-RISE: A Post-hoc Interpretation Method of Black-box Models for SAR ATR

1. Introduction

Synthetic Aperture Radar (SAR) is a kind of active earth-observation system which can produce high-resolution image all day, has been widely used in ground observation and military reconnaissance. One of its primary applications is the detection and identification of various military targets [1,2]. With the enhancement of SAR data acquisition capability, Synthetic Aperture Radar Automatic Target Recognition (SAR ATR) [3] has become a key technology and research hotspot of radar signal processing. Traditional SAR target recognition methods [4] merely rely on artificial experience for feature extraction and selection, which lead to a certain degree of subjectivity and bias. Additionally, it is challenging to guarantee the effectiveness of recognition results [5]. In recent years, deep learning methods [6], especially Convolutional Neural Networks (CNN), have been extensively used in computer vision [7,8] and demonstrating remarkable achievements. Meanwhile, based on deep learning, the image processing method has also been successfully extended to the field of remote sensing images [9,10], presenting a new direction and breakthrough for SAR target recognition [11,12,13].

At present, CNN has become one of the most effective network architecture for image recognition tasks. As the earliest CNN network, LeNet-5, proposed by LeCun et al. [14] in 1998 for handwritten digit recognition, was regarded as the first CNN structure. Over time, researchers have continuously refined and optimized the classic CNN architecture and its features, leading to the design of more complex and high-performing CNNs, such as Alexnet [15], GoogLeNet [16], VGGNet [17], Resnet [18], etc. Despite the outstanding performance achieved by classic CNN structures, the neural network has a low level of transparency and is also known as the black boxes [19] due to the lack of a clear visual explanation for the representation of internal features and parameter organization. These limitations significantly constrain people’s ability to understand and interpret the internal workings of neural networks, consequently restricting their potential applications in specialized fields, such as medicine, finance, transportation, military, and other domains [20,21]. There are currently two primary research directions for interpretability, which are Intrinsic Explanation and Post-hoc Explanation [22]. Intrinsic Explanation aims to enhance the interpretability of the model itself, enabling users to understand the calculating process and rationale without requiring additional information or algorithms. In contrast, Post-hoc Explanation mainly focuses on explaining the behavior and decision-making process of black-box models [23]. Retraining the model can be too costly in terms of time and resources since the model has already been trained and deployed. As such, the Post-hoc Explanation approach is often more appropriate in such cases. Representation visualization, as an intuitive method in post-hoc interpretation, mainly involves combining the input, middle layer parameters, and output information of the pre-trained model to achieve an interpretation of the decision results. Gradient-based methods, Perturbation, and Class Activation Map (CAM) are three widely adopted methods for achieving representation visualization [22,24].

The gradient-based method [25,26,27,28,29,30,31] backpropagates the gradients of a specific class into the input image to highlight image regions that contribute positively or negatively to the result. The methods are fast computation and high resolution of the generated images but usually suffer from excessive noise. CAM is one class of the most important methods specifically designed for CNNs [24,32,33,34,35,36,37]. The method utilizes the form of a heatmap to visually highlight the regions most relevant to the particular category. The CAM-based method was first proposed by Zhou et al. [33] in 2016. They believed that with the deepening of CNN layers, the feature map of the intermediate layer contains less and less irrelevant information, and the last convolutional layer of the CNN achieves the highest-level semantic information. After that, numerous CAM methods have been proposed, including Grad-CAM [34], Grad-CAM++ [35], Grad-CAM [36], Group-CAM [32], Score-CAM [24], Ablation-CAM [37], etc. Although these methods have demonstrated good performance in image interpretation, they may suffer from low resolution and spatial precision in some cases. Interpretability methods based on perturbation [38,39,40,41] typically utilize the element-wise product of generated masks and the original image to obtain the perturbed input images, which are then fed into the model to observe the changes in the prediction result. The information generated is used to optimize the weighted mask to obtain the final interpretation result image. Among them, RISE [41] randomly generates a large number of masks through Monte Carlo sampling method to occlude different parts of the input image. And the final saliency map is generated by the weighted sum of the masks and the scores predicted by the base model on the masked images.

In this paper, we propose a post-hoc interpretation method of black-box models for SAR ATR called Randomized Input Sampling for explanation based on Clustering (C-RISE). We demonstrate the effectiveness of C-RISE through extensive experimental validation and comparative analysis. Specifically, our method exhibits superior performance when dealing with SAR images that suffer from severe noise interference, as well as cases where adjacent pixels exhibit mutual influence and dependence. C-RISE offers several advantages over other neural network interpretable algorithms, including white-box methods:

1.

The method is a black-box interpretation method, and the calculation process does not need to use the weight, gradient, feature map and other information of the model so that it has better robustness and transferability. Furthermore, the approach avoids errors caused by unreasonable weight selection and information loss during feature map upsampling in Class Activation Mapping (CAM) methods;

2.

Compared with RISE, our algorithm can group mask images that capture similar fusion features into different groups by clustering strategy. This allows for the concentration of more energy in the heatmap on the target area, thereby increasing the interpretability of the model.

3.

C-RISE employs Gaussian blur to process masked regions, as opposed to simply setting occluded pixels to 0. This technique ensures the consistency and integrity of the original image structure while covering certain areas. As a result, it reduces the deviation of network confidence caused by the destruction of spatial structure, leading to more credible results when compared to other perturbation-based interpretation methods.

The contents of this article are organized as follows: In Section 2, we introduce the principle of the RISE algorithm and CAM methods. Section 3 elaborates on the details of the C-RISE algorithm. Section 4, we verify the effectiveness and robustness of the proposed method through both qualitative judgment and quantitative description. Finally, in Section 5, we discuss the experimental results, clarify any confusion, and explore potential future work.

2. Related Work

In this section, we first review the existing classical methods of CAM [24,32,33,34,35,36,37] and the RISE [41] algorithm. Since both CAM methods and RISE interpretation methods display in the form of heatmaps, we focus our subsequent experiments [41] on comparing the effects of different CAM methods, RISE, and C-RISE. This chapter provides theoretical support for the design and experimentation of C-RISE.

2.1. CAM Methods

Zhou et al. [33] proposed the Class Activation Map (CAM) method which utilizes the final convolutional layer of CNN to extract the most abstract target-level semantic information. Its corresponding feature map contained the most abstract target-level semantic information and each channel detected different activated parts of the target. Thus, the class activation map relevant to the recognition result of class c can be generated by the channel-wise weighted summation of the final feature maps. The formal representation of this process can be expressed as follows:

$$L_{CAM}^c=ReLU\left(\sum _{k=1}^n{w}_k^c{\mathit{A}}_k^L\right)$$

(1)

where $w_k^c$ represents the connection weight of the kth neuron pair classified as class c in the Softmax layer, and ${\mathit{A}}_k^L$ represents the feature map of the kth channel in the lth convolutional layer. The disadvantage of this method is that it can only be applied to the last layer feature map and the full connection is GAP operation. Otherwise, it requires the user to modify the network and retrain, and such costs are sometimes substantial. To overcome the disadvantages, Selvaraju et al. [34] proposed a method named Grad-CAM and updated the weight generation method in Equation (1) as follows:

$$w_k^c=\frac{1}{Z}\sum _i\sum _j\frac{\partial y^c\left(\mathit{x}\right)}{\partial {\mathit{A}}_{k,i,j}^L}$$

(2)

where the sum element is the gradient of the calculated class score($y^c\left(x\right)$) with respect to the pixel values at each position of ${\mathit{A}}_k^L$, and Z represents the normalization factor. Compared to the CAM method, Grad-CAM is more generalized and can be used for different model structures. Both Grad-CAM++[35] and XGrad-CAM [36] are improved algorithms based on Grad-CAM method. The basic form of Grad-CAM++ is the same as Grad-CAM, but the difference is that the combination of higher-order gradients is used as the channel weight in Grad-CAM, which improves the visualization effect of multi-object images and the positioning is more accurate. XGrad-CAM achieves better visualization of CNN decisions through a clear mathematical interpretation.

Different from the improvement idea based on gradient, Score-CAM [24] is a gradient-free algorithm for visualizing CNN decisions. It defines the concept of Increase of Confidence (CIC), which measures the increment of confidence relative to a baseline image. The CIC score for a particular feature map ${\mathit{A}}_k^L$ is computed as:

$$C\left({\mathit{A}}_k^L\right)=f\left(\mathit{X}\circ {\mathit{A}}_k^L\right)-f\left({\mathit{X}}_b\right)$$

(3)

where $\mathit{X}$ is the input image, ∘ represents the Hadamard product, and ${\mathit{X}}_b$ is the baseline image, which can be set to an all-0 matrix with the same size as the original image. $f(·)$ denotes the neural network’s output score for the target class. The algorithm then computes CIC scores for all feature maps in a particular layer and updates the scores using the Softmax operation. These updated scores are used as the weights for the corresponding feature maps. Finally, the different feature maps are weighted and summed to generate a visual image.

The CAM approach has been demonstrated to be effective in visualizing the important regions of objects in various optical image datasets. However, when applied to Synthetic Aperture Radar (SAR) images, several challenges arise such as gradient dispersion, energy unconcentration, and inaccurate positioning. These challenges are primarily due to the unique characteristics of SAR images which include:

1.

SAR images are often characterized by low resolution and low Signal-to-Noise Ratio (SNR), which makes it challenging to visualize important features and information accurately. Additionally, the imaging principle of SAR images is based on active imaging, which introduces a significant amount of interference spots in the image, thereby making SAR images significantly different from optical images. These interference spots can significantly impact the visualization process, leading to inaccurate feature localization and reduced effectiveness of CAM-based visualization methods;

2.

The relatively small difference between different categories in SAR image datasets poses a challenge to visualization techniques such as CAM, which heavily rely on distinguishing features between different categories. Furthermore, the target area of SAR images is often highly localized, which makes accurate positioning critical for the interpretation of visualizations. However, different CAM methods typically use feature maps to upsample to the size of the original image, which can introduce positioning deviations. Despite ongoing efforts to generate high-resolution feature maps, the visualization effect of SAR images using CAM methods remains suboptimal.

2.2. RISE

Randomized Input Sampling for Explanation (RISE) [41] is a perturbation-based visualization method in local interpretation, which is, for the prediction result of a single image, a heatmap with prominent areas is obtained as the interpretation result by combining randomly sampled masks. The detailed architecture of RISE is presented in Figure 1. Firstly, based on Monte Carlo sampling method, a large number of masks with the same size as the original image are generated. After that, the element-wise product of masks and the original image are made to obtain the corresponding perturbed images. Then, the masked images were input into the black-box model to obtain the prediction probability of the inferred category. Finally, the prediction probability is used as the weight to sum the masks, so as to superimpose the areas in the original image that play an important role in the specified category. Randomized Input Sampling for Explanation (RISE) [41] is a perturbation-based method that generates a heatmap to highlight the important regions of an input image with respect to the prediction of a black-box model. The detailed architecture of RISE is presented in Figure 1. RISE generates a large number of randomized binary masks and applies them to the input image to obtain a set of masked images. The CNN is then applied to each masked image to obtain a set of output scores. The final explanation map is generated by aggregating the scores obtained from all the masked images. RISE has been shown to be effective in providing local interpretability for various image classification models. Moreover, Score-CAM is a gradient-free method that is inspired by RISE [24].

RISE method is a black-box interpretation method, which does not need to use the weight, gradient, feature map and other information in the calculation process. Since the Monte Carlo sampling method is a stochastic approximate inference method, the idea of this method is to find the expected value of the function $f(·)$ under the complex probability distribution $p\left(z\right)$, as shown in Equation (4).

$$\begin{array}{cc}\hfill & E_{z\mid x}\left[f\left(z\right)\right]=\int p(z\mid x)f\left(z\right)dz\cong \frac{1}{N}\sum _{i=1}^{N}f\left(z_i\right)\hfill \end{array}$$

(4)

In the RISE algorithm, the predicted probability of the black-box model for the category to which the perturbed image belongs can be viewed as the importance of the region retained by the mask. Then the importance of the prominent region of the final generated image can be viewed as the expectation obtained from all masks, as shown in Equation (5).

$$\begin{array}{cc}\hfill & S_{I,f}\left(\lambda \right)=E_M[f(I\circ M)\mid M\left(\lambda \right)=1]\hfill \end{array}$$

(5)

where $\lambda$ denotes the pixel with a value of 1 in the mask, and $S_{I,f}\left(\lambda \right)$ represents the expected score obtained by inputting the pictures under different masks M into the model $f(·)$. $S_{I,f}\left(\lambda \right)$ can be intuitively interpreted as the greater the prediction probability after the pixel-wise multiplication of the mask and the image, the more important the region retained by this mask.

Then, we can expand the expression according to the definition of expectation as follows:

$$\begin{array}{cc}\hfill S_{I,f}\left(\lambda \right)& =\sum _{m}f(I\circ m)P[M=m\mid M\left(\lambda \right)=1]\hfill \\ \hfill & =\frac{1}{P\left[M\right(\lambda )=1]}\sum _{m}f(I\circ m)P[M=m,M\left(\lambda \right)=1]\hfill \end{array}$$

(6)

where

$$\begin{array}{cc}\hfill P[M=m,M(\lambda )=1]& =\left\{\begin{array}{c}0,\mathrm{if}m\left(\lambda \right)=0\hfill \\ P[M=m],\mathrm{if}m\left(\lambda \right)=1\hfill \end{array}\right.\hfill \\ \hfill & =m\left(\lambda \right)P[M=m]\hfill \end{array}$$

(7)

By substituting Equation (7) into Equation (6), we can get:

$$\begin{array}{cc}\hfill & S_{I,f}\left(\lambda \right)=\frac{1}{P\left[M\right(\lambda )=1]}\sum _{m}f(I\circ m)·m\left(\lambda \right)·P[M=m]\hfill \end{array}$$

(8)

Since the mask m has a 0-1 distribution, we can obtain Equation (9):

$$\begin{array}{c}\hfill P\left[M\right(\lambda )=1]=E\left[M\right(\lambda \left)\right]\end{array}$$

(9)

$$\begin{array}{cc}\hfill & \therefore S_{I,f}\left(\lambda \right)=\frac{1}{E\left[M\right(\lambda \left)\right]}\sum _{m}f(I\circ m)·m\left(\lambda \right)·P[M=m]\hfill \end{array}$$

(10)

It can be seen from Equation (11) that the heatmap can be obtained by the sum of masks obtained from random sampling by weighting, while the weight is the predicted probability of the perturbed image. When masks are sampled by uniform sampling, $P[M=m]$ can be expressed as:

$$\begin{array}{cc}\hfill & P[M=m]=\frac{1}{N}\hfill \end{array}$$

(11)

So Equation (10) can be updated to:

$$\begin{array}{c}\hfill S_{I,f}\left(\lambda \right)\approx \frac{1}{E\left[M\right]·N}\sum _{i=1}^{N}f\left(I\circ M_i\right)·M_i\left(\lambda \right)\end{array}$$

(12)

Considering that pixle-wise masks can cause huge changes in the prediction of the model, and the computational cost of sampling a pixle-level mask is exponential, during mask generation, small masks are generated first and then upsampled back to the image size in order to ensure smoothness.

3. Our Method

As a post-hoc interpretation algorithm based on perturbation, RISE algorithm has a more intuitive and understandable presentation than the visual interpretation method based on back propagation. At the same time, RISE also overcomes the limitations of general CAM methods by avoiding the generation of unreasonable weights and the problem of small feature maps during the up-sampling process. However, the effectiveness of RISE and other optical image-based interpretive methods in SAR ATR scenarios is limited. This is because the active imaging mechanism of SAR images results in multiplicative noise, which causes problems such as noise, energy dispersion, and inaccurate positioning when applying optical image-based interpretive methods to SAR image recognition [3,4]. To address this issue, we propose an algorithm based on RISE, called Randomized Input Sampling for Explanation based on Clustering (C-RISE), which is a post-hoc interpretation method for black-box models in SAR ATR. Our algorithm considers the structural consistency and integrity of SAR images and highlights the regions that contribute to category discrimination in SAR images. Figure 2 illustrates the workflow of our proposed approach.

3.1. Mask Generation

As shown in Section 2.2, pixle-level occlusion may have a huge impact on the model, and the computational complexity of sampling is high. Therefore, in order to ensure the smoothness and the consistency of the target space structure when generating masks, small masks are generated first and then upsampled back to the image size. The basic process is shown in Figure 3. Formally, the process of generating masks can be described as follows:

1.

N binary masks $\{gri{d}_1$, $gri{d}_2,\dots$, $grid{}_N\}$ are randomly generated based on Monte Carlo sampling, where $gri{d}_i\in {\mathbb{R}}^{s\times s},i=1,2,\dots ,N$. s is smaller than image size H and W. In  $gri{d}_i$, each element independently to 1 with probability p and to 0 with the remaining probability;

2.

Upsample $gri{d}_i$ to $gri{d}_i^{\prime }\in {\mathbb{R}}^{(s+1)H\times (s+1)W}$;

3.

A rectangular area was randomly selected from ${grid}_i^{\prime }$ as $M_i$, where $M_i\in {\mathbb{R}}^{H\times W},i=1,2,\dots ,N$.

After obtaining N masks, we introduce Gaussian blur to the occluded part of the original image, which is in order to make the image after the mask processing can retain the maximum consistency of the original image, and smoothly occlusion of the region. Gaussian blur is an image blurring filter that computes the transformation of each pixel in an image with a normal distribution. The normal distribution equation in 2-dimensional space can be written as:

$$\begin{array}{c}\hfill G\left(X\right)=\frac{1}{2\pi {\sigma }^2}{e}^{-\left(u^2+v^2\right)/\left(2{\sigma }^2\right)}\end{array}$$

(13)

where $(u,v)$ denotes the pixel position and $\sigma$ means the standard deviation of the normal distribution. It is worth noting that in 2-dimensional space, the contours of the surface generated by Equation (13) are normally distributed concentric circles from the center. The value of each pixel is a weighted average of the neighboring pixel values. The value of the original pixel has the largest Gaussian distribution value, so it has the largest weight, and the neighboring pixels get smaller and smaller as they get farther from the original pixel. The Gaussian blur preserves the edge effect more than other equalization blur filters, which is equivalent to a low-pass filter.

Based on Gaussian blur, We can use Equation (14) to obtain the image after mask processing:

$$\begin{array}{c}\hfill X_i^{\prime }=X\circ M_i+G\left(X\right)\circ ({\mathit{1}}^{H\times \mathit{W}}-M_i),i=1,2,..,N\end{array}$$

(14)

where $X\in {\mathbb{R}}^{H\times W}$ denotes the original image, ${\mathit{1}}^{H\times \mathit{W}}\in {\mathbb{R}}^{H\times W}$ means an all-1 matrix and its shape is $H\times W$.

3.2. Clustering

The masked image $\{X_1^{\prime }$, $X_2^{\prime },\dots$, $X_N^{\prime }\}$ are input to the black-box model $f(·)$ to obtain the output vector $\{{\mathbf{a}}_1$, ${\mathbf{a}}_2,\dots$, ${\mathbf{a}}_N\}$. Moreover, we use ${\mathbf{a}}_i\in {\mathbb{R}}^{1\times m},i=1,2,\dots ,N$ as the feature vectors to cluster $M_i$ by k-$means$. m is the number of categories. The process is shown in Equations (15)–(17).

$$\begin{array}{c}\hfill {\mathit{a}}_i=f\left(X_i^{\prime }\right),i=1,2,\dots ,\mathrm{N}\end{array}$$

(15)

$$\begin{array}{c}\hfill \left({\mathit{c}}_1;{\mathit{c}}_2;\dots ;{\mathit{c}}_k\right)=k-means\left(\left[\left({\mathrm{M}}_1,{\mathit{a}}_1\right),\left({\mathrm{M}}_2,{\mathit{a}}_2\right),\dots ,\left({\mathrm{M}}_N,{\mathit{a}}_N\right)\right]\right)\end{array}$$

(16)

$$\begin{array}{c}\hfill {\mathit{c}}_i=\left\{M_j^i\right\},i=1,2,..,k;j=1,2,...,N_i\end{array}$$

(17)

where $c_i$ denotes the ith cluster, $M_j^i$ denotes the jth mask in ith cluster, k and $N_i$ represent the number of clusters and the number of elements in the ith cluster, respectively.

If the original image is identified as class l after the black-box model, we can obtain:

$$\begin{array}{c}\hfill {\alpha }_j^i={\mathit{a}}_j^i\left[l\right],i=1,2,..,k;j=1,2,...,N_i;l\le m\end{array}$$

(18)

where ${\mathit{a}}_j^i$ denotes the feature vector from $M_j^i$ and ${\alpha }_j^i$ can be seen as the contribution of the jth mask in the ith cluster to the model. After that, we use ${\alpha }_j^i$ to estimate the weight of a specific mask and calculate the weighted sum in each cluster $C{M}_i$ as follows:

$$\begin{array}{c}\hfill C{M}_i=\sum _{j=1}^{N_i}{\alpha }_j^i·{\mathbf{M}}_j^i,i=1,2,..,k\end{array}$$

(19)

After that, we calculated the $CIC$ value of $C{M}_i$ through Equation (3) and used it as the classificatory information that $C{M}_i$ was concerned about. Finally, the final result $H^{C-RISE}$ is generated by weighted summation of the feature maps of different clusters. The process is formulated as Equations (20) and (21). The pseudo-code is presented in Algorithm 1.

$$\begin{array}{c}\hfill {\alpha }_i^{\prime }={\left[f\left(X\circ {CM}_i\right)-f\left(X_b\right)\right]}_l,i=1,2,...,k\end{array}$$

(20)

$$\begin{array}{c}\hfill H^{C-RISE}=\sum _{i=1}^k{\alpha }_i^{\prime }·{CM}_i\end{array}$$

(21)

Algorithm 1: C-RISE

4. Experiment

4.1. Experimental Settings

This study employs SAR images of ten vehicle target types under standard operating conditions (SOC) from the Moving and Stationary Target Acquisition and Recognition (MSTAR) dataset [42] as the experimental data. The dataset comprises 5172 SAR images with dimensions of $1\times 100\times 100$, with 2536 images used for training and 2636 for testing. The ten target categories include $2S1$, $BRDM2$, $BTR60$, $D7$, $SN\_132$, $SN\_9563$, $SN\_C71$, $T62$, $ZIL131$, and $ZSU\_23\_4$. Figure 4 displays ten representative SAR images for each category.

During the experiment, the Alexnet model [5] was utilized as a classifier, and its structure is depicted in Figure 5. It is worth mentioning that, as the C-RISE algorithm is primarily tailored for black-box models, alternative efficient models may be employed in place of Alexnet. After conducting multiple iterations of training, the neural network achieved a recognition rate of 97.6%, which indicates the effectiveness of using various methods to generate saliency maps. However, since this paper primarily focuses on interpreting and analyzing the network structure using different visualization methods, the training techniques and processes are not extensively discussed. During the implementation of the C-RISE algorithm, several parameters were set, including $k=4$, $N=2000$, $s=8$, $p=0.5$. It should be emphasized that the experimental results were sensitive to the number of clusters, and selecting $k=4$ or 8 yielded relatively optimal results. Hence, for the purpose of this paper, k was specified as 4.

4.2. Class Discriminative Visualization

Since the class activation map generated by CAM method and the saliency map generated by C-RISE algorithm are presented in the form of heatmap, we focus on comparing the experimental effects of different CAM methods, RISE algorithm and C-RISE algorithm in the following experimental part, referring to the comparison method in [41]. In this section, we randomly selected ten graphs that were correctly classified in different networks from the testset, and used Grad-CAM [34], Grad-CAM++ [35], XGrad-CAM [36], Score-CAM [24], RISE [41] and C-RISE to visually analyze the model recognition process, and the comparison is shown in Figure 6.

We can verify the fairness and localization ability of the C-RISE algorithm from a qualitative and quantitative perspective. It can be intuitively seen from Figure 1 that compared with CAM methods and RISE, the highlighted areas of the heatmap generated by our method are more closely concentrated near the target and the degree of energy dispersion is smaller. The heatmap is an image composed of different color intensities, and the intensity of a pixel’s color corresponds to its importance. Analyzing from a quantitative point of view, we measure the quality of the saliency map by the localization ability. From an energy-based perspective, we are concerned with how much energy of the salient map falls in the bounding box of the target object. Therefore, we adopted a similar measure to [24], the specific process is shown in Figure 7. Firstly, we annotated the bounding boxes of the objects of all images in testset, and then binarized the images according to the rule that the inner region of the bounding box is set to 1, and the outer region is 0. The processed image is then multiplied by the heatmap and summed to obtain the energy within the target bounding box. We use the ratio of the internal energy of the bounding box to the total energy of the heatmap $proportion$ to measure the localization and recognition capabilities of different methods. The mathematical expression is shown in Equation (22).

$$\begin{array}{c}\hfill Proportion=\frac{\sum E_{(i,j)\in bbox}}{\sum E_{(i,j)\in bbox}+\sum E_{(i,j)\notin bbox}}\end{array}$$

(22)

where $E_{(i,j)}$ denotes the energy value of the pixel at position $(i,j)$ in the heatmap.

It is worth mentioning that the information contained in each image in the MSTAR dataset is a single target. And in different pictures, the position occupied by the target is usually a large area of the image, which facilitates us to label each subset. Figure 8 shows the binarization results of ten groups of data randomly selected. We calculate $proportion$ of images in each category of the testset separately, and the results are shown in Table 1.

4.3. Conservation and Occlusion Test

In this section, we use the occlusion and conservation test [36,42] to analyze the localization capability of different methods quantitatively. The Conservation and Occlusion tests represent experiments in which only part of the area is preserved or abandoned, respectively. The experiments measures the effectiveness of the energy-concentrated regions in heatmaps by inputting the mask/reverse mask processed images into the black-box model and observing the change in scores, and the masks/reverse masks the resulting map obtained by binarization of the heatmap at different thresholds. The way masks generated is shown as Equations (23) and (24).

$$\begin{array}{c}\hfill M_{threshold}(i,j)=\left\{\begin{array}{c}1,\mathrm{if}{H}^{C-RISE}(i,j)\ge threshold\\ 0,otherwise\end{array}\right.\end{array}$$

(23)

$$\begin{array}{c}\hfill {\overline{M}}_{threshold}={\mathit{1}}^{H\times \mathit{W}}-M_{threshold}\end{array}$$

(24)

where $threshold\in [0,1]$, $H^{C-RISE}$ denotes the pixel value of the heatmap from C-RISE. $M_{threshold}$ and ${\overline{M}}_{threshold}$ mean the masks/reverse masks, respectively.

Based on Equation (23) and (24), we could use the element-wise product to get the processed images I/$\overline{I}$ after masked/reverse masked and the results after masked/reverse masked are shown in Figure 9.

$$\begin{array}{c}\hfill I=M_{threshold}\circ X\end{array}$$

(25)

$$\begin{array}{c}\hfill \overline{I}={\overline{M}}_{threshold}\circ X\end{array}$$

(26)

However, directly replacing some pixels with black may produce high-frequency sharp edges [43], and these artificial traces may also lead to changes in the prediction probability, which cannot guarantee the fairness and objectivity of the model recognition process. In order to solve the above problems, we improved the original experiment and proposed two new measures, namely, introducing multiplicative noise and Gaussian blur to the occluded region. The follow two experiments show the effectiveness and rationality of our algorithm.

4.3.1. Based on Multiplicative Noise

In the experiments, we firstly add multiplicative noise to the occluded region and updated Equations (23) and (24) to Equations (27) and (28). The reason for adding multiplicative noise is based on the physical scattering mechanism of SAR coherent imaging. We believe that the intensity of each resolved element of SAR image is modulated by the Radar Cross Section (RCS) [3] of the ground object in the element and a multiplicative noise whose intensity follows the exponential distribution of unit mean (mean = 1). So we can consider the SAR image as the product of the RCS of the ground object in the scene and the noise of the unit mean exponential intensity distribution. Therefore, in the process of signal processing, we generally consider the noise of SAR image as multiplicative noise [3,6]. Figure 10 shows the above processing of the same image.

$$\begin{array}{c}\hfill I=M_{threshold}\circ X+{\overline{M}}_{threshold}\circ Noise\left(X\right)\end{array}$$

(27)

$$\begin{array}{c}\hfill \overline{I}={\overline{M}}_{threshold}\circ X+M_{threshold}\circ Noise\left(X\right)\end{array}$$

(28)

where $Noise\left(X\right)$ denotes add high-variance Gaussian multiplicative noise to the input image X.

Then we define $confidence\_drop(a,b)$ to represent the divergence in the confidence that the processed image b and the original image a are classified into the same category. The mathematical expression of $confidence\_drop(a,b)$ is shown in Equation (29).

$$\begin{array}{c}\hfill confidence\_drop(a,b)=\frac{S^c\left(a\right)-S^c\left(b\right)}{S^c\left(a\right)}\end{array}$$

(29)

where $S^c\left(x\right)$ is used to represent the score of the input image x being classified as class c. Based on this, we use $confidence\_dro{p}^{con}(X,I)$ and $confidence\_dro{p}^{occ}(X,\overline{I})$ to represent the scores in the conservation and occlusion test, respectively. The process is shown as Equations (30) and (31).

$$\begin{array}{c}\hfill confidence\_dro{p}^{con}(X,I)=\frac{S^c\left(X\right)-S^c\left(I\right)}{S^c\left(X\right)}\end{array}$$

(30)

$$\begin{array}{c}\hfill confidence\_dro{p}^{occ}(X,\overline{I})=\frac{S^c\left(X\right)-S^c\left(\overline{I}\right)}{S^c\left(X\right)}\end{array}$$

(31)

It is worth noting that the smaller $confidence\_dro{p}^{con}(X,I)$, the greater the difference between the values of $S^c\left(X\right)$ and $S^c\left(I\right)$, and the generated heatmap can be considered to be located in the salient feature part of the target. Similarly, the larger the $confidence\_dro{p}^{occ}$, the lager the difference between the values of $S^c\left(X\right)$ and $S^c\left(\overline{I}\right)$, and the main features after image processing can be considered to be preserved.

The $confidence\_dro{p}^{con}(X,I)$ and $confidence\_dro{p}^{occ}$ of various methods under different thresholds including Grad-CAM, Grad-CAM++, XGrad-CAM, Score-CAM, RISE and C-RISE, are shown in Table 2 and Table 3.

4.3.2. Based on Gaussian Blur

From Table 2 and Table 3, we can see that compared with other methods, C-RISE achieved relatively optimal performance under different thresholds. Similarly, we can also use high-variance Gaussian blur to process the masked area, and the processed results are shown in Figure 11. Experimental indicators are shown in Table 4 and Table 5 respectively. The mathematical expressions are updated from Equations (23) and (24) to Equations (32) and (33).

$$\begin{array}{c}\hfill I=M_{threshold}\circ X+{\overline{M}}_{threshold}\circ G\left(X\right)\end{array}$$

(32)

$$\begin{array}{c}\hfill \overline{I}={\overline{M}}_{threshold}\circ X+M_{threshold}\circ G\left(X\right)\end{array}$$

(33)

where $G\left(X\right)$ denotes introduce high-variance Gaussian blur to the input image X.

4.4. Insertion and Deletion Test

In this experiment, we compared different methods by insertion-deletion test [41]. The experiment is a metric used to evaluate visual interpretation methods and measures the ability of visual interpretation to capture important pixels. During the deletion experiment, the k most important pixels in the heatmap are successively removed, and then we calculate the degree of change in the prediction probability. The insertion curve is the opposite. The curves are shown in Figure 12, with smaller $AUC$ of deletion curves and higher $AUC$ of insertion curves indicative of a better explanation. We randomly select an image from the test set for demonstration and plot its deletion and insertion curves of different algorithms. The results are shown in Figure 13. We calculate $AUC$ of both curves and the $over\_all$ score [32] ($AUC\left(insertion\right)-AUC\left(deletion\right)$) of all images from the test set as a quantitative indicator. The average results over 2636 images is reported in Table 6. We found that C-RISE achieves splendid results, indicating that the pixel importance revealed by the visualization method is in high agreement with the model and has great robustness.

5. Conclusions

This paper introduces C-RISE, a novel post-hoc interpretation method for black-box models in SAR ATR, which builds on the RISE algorithm. We compare the interpretation effects of different methods and C-RISE algorithm using both qualitative analysis and quantitative calculation. C-RISE offers several advantages, including its ability to group mask images that capture similar fusion features using a clustering strategy, which allows for concentration of more energy in the heatmap on the target area. Additionally, Gaussian blur is used to process the masked area, ensuring the consistency and integrity of the original image structure and taking into account both global and local characteristics. Compared with other neural network interpretable algorithms and even white box methods, C-RISE’s black-box model-oriented characteristics make it more robust and transferable. Furthermore, C-RISE avoids the error that can be caused by the unreasonable weight generation method in general CAM methods and the small feature map in the CNN model during the up-sampling process to the original image size. In our future work, we aim to explore the potential of C-RISE in identifying improper behaviors exhibited by black-box models and leveraging it to guide parameter adjustments. This will involve a systematic investigation of the capabilities of our proposed approach in identifying and diagnosing the sources of model inaccuracies and devising strategies to improve the performance of the black-box models. Such research endeavors will contribute to enhancing the interpretability and robustness of black-box models in various practical applications.