1. Introduction

2. A Target Detection Model Incorporating Multiple Attention Mechanisms

2.2. Transform Attention Component (TAC)

The self-attentive component transformer is a stacked model architecture of multiple encoders and decoders that computes global correspondences between outputs and inputs using a unique multi-head attention mechanism. The extracted feature map and position encoding are used as inputs. The feature map is expanded into a one-dimensional sequence of H × W (height × width) features to be passed to the encoder, and the target-level information is extracted by the mechanism of mutual attention. This not only improves attention to the target region but also reduces background interference by focusing on the overall input information [14].

The working principle of TAC is to reconstruct image features and key matrices by multiplying the input feature sequence with different weighting matrices. $\mathit{Q}$ is the query matrix, that is, the feature matrix of the image; $\mathit{K}$ is the key matrix; and $V$ is the value matrix.

$$\begin{array}{l}\mathit{Q}=\mathit{F}\mathit{e}\mathit{a}\mathit{t}\mathit{u}\mathit{r}\mathit{e}\cdot \mathit{W}\mathit{q},\\ \mathit{K}=\mathit{F}\mathit{e}\mathit{a}\mathit{t}\mathit{u}\mathit{r}\mathit{e}\cdot \mathit{W}\mathit{k},\\ \mathit{V}=\mathit{F}\mathit{e}\mathit{a}\mathit{t}\mathit{u}\mathit{r}\mathit{e}\cdot \mathit{W}\mathit{v},\end{array}$$

(1)

where $\mathit{W}\mathit{q}$,$\mathit{W}\mathit{k}$, and $\mathit{W}\mathit{v}$ are the learnable weights of different matrices. The feature matrix $\mathit{Q}$ and key matrix $\mathit{K}$ are multiplied using the softmax method to obtain the attention matrix. To further realize numerical aggregation weighted by the attention weights, the attention matrix was multiplied by V to obtain the correlation between the targets in the SAR image. Finally, it can be represented in the TAC by multiple heads of attention.

$$\mathrm{TAC}(\mathit{Q},\mathit{K},\mathit{V})={\displaystyle \sum _{m=1}^M\left[\left(Ssoft\max \left(\frac{\mathit{Q}\cdot {\mathit{K}}^T}{\sqrt{dk}}\right)\right)\cdot \mathit{V}\right]}.$$

(2)

where M represents the number of attention heads and d_k is the dimension of K. In the feature extraction phase of the backbone network, the input image is segmented into multiple subfeature maps. Some subfeature maps are selected for processing by the TAC module, which utilizes a self-attention mechanism to model the associations and extract features from different regions in the feature maps. The TAC module can capture global contextual information and learn the dependencies between different locations in the feature maps, which can convert the input feature maps into more expressive sub-feature maps.

Figure 2. Transform Attention Component.

Figure 2. Transform Attention Component.

2.3. Channel and Spatial Attention Component (CSAC)

Transformer-based attentional mechanisms with deep semantic features have a larger sensory field; however, a larger downsampling factor results in a loss of positional information. In addition to the transformer-based self-attention mechanism used to form a feature map that focuses on interrelationships, attentional mechanisms include channel attention, pixel attention, multilevel attention, and other methods of focusing on key features [8,15].

Channel Attention is used to bring the attention of CNN on the channel dimensions. Hence, the input feature layer F(H × W × C) provides the average pooling (Avg-Pool) and maximum pooling (Max-Pool) operations, after which it is compressed into a vector (H × W × 1). Subsequently, using two fully connected (TFC) layers, the vector was mapped to the weight and bias vectors. Finally, the corresponding weight of each channel is calculated using the activation function, and a new feature map F_c is generated, which accounts for the importance of different feature channels.

$$\begin{array}{l}Favg=\frac{1}{H\times W}{\displaystyle \sum _{i=1}^H{\displaystyle \sum _{j=1}^{W}Avg-Pool\left(F\right)}},\\ Fmax=\frac{1}{H\times W}{\displaystyle \sum _{i=1}^H{\displaystyle \sum _{j=1}^{W}Max-Pool\left(F\right)}}，\\ F\mathrm{c}=sigmoid(TFC(Favg)+TFC(Fmax\left)\right)\end{array}$$

(3)

Based on the calculation results from the channel attention, the spatial attention performs average and maximum pooling operations on F_c. The compressed feature layer is then focused on the most useful data of the spatial region, and this vector is converted into a weight matrix using TFC. A new feature map, F_s, indicates the importance of different spatial positions.

$$\begin{array}{l}Fc\_avg=\frac{1}{H\times W}{\displaystyle \sum _{i=1}^H{\displaystyle \sum _{j=1}^{W}Avg-Pool\left(Fc\right)}},\\ Fc\_\max =\frac{1}{H\times W}{\displaystyle \sum _{i=1}^H{\displaystyle \sum _{j=1}^{W}Max-Pool\left(Fc\right)}}，\\ Fs=sigmoid(TFC(Fc\_avg)+TFC(Fc\_\max \left)\right)\end{array}$$

(4)

Figure 3. Channel attention and spatial attention.

Figure 3. Channel attention and spatial attention.

The CSAC module integrates feature space and feature channel information in two dimensions by introducing spatial and channel attention. In the target detection model, the feature-mapping layers at various scales employ the channel attention mechanism to determine the feature dependencies between different channel maps and calculate the weighted values for all channel maps. The spatial attention mechanism was used to weigh each spatial location in the feature mapping layer to strengthen the model’s ability to perceive and utilize spatial location features.

Figure 4. Channel and Spatial Attention Component.

Figure 4. Channel and Spatial Attention Component.

2.4. TAC_CSAC_Net

The backbone network of feature fusion is established in the TAC_CSAC_Net model, which adopts a multiattention mechanism to maximize the mining of target features and their associations with each other in a large scene image, whereas the GIoU loss function is used to optimize the traditional target detection loss function. Its multi-attention mechanism is primarily reflected in the use of TAC to process the global information and obtain the correlation between different pixel points and in the use of CSAC to integrate the feature space and feature channel information in two dimensions. The structure of TAC_CSAC_Net is shown in Figure 5.

The overall loss of the model included and bounding-box losses. The cross-entropy loss feeds the classification loss and the bounding box loss includes the L₁ loss between the true value, b_i, predicted , and losses.

$$Lclass=-\frac{1}{N}{\displaystyle {\sum }_{i=1}^N\left(pi\log \widehat{p}\widehat{\sigma }i+\left(1-pi\right)\log \left(1-\widehat{p}\widehat{\sigma }i\right)\right)}$$

(5)

$$Lbox\left(yi,{\widehat{y}}_{\sigma (i)}\right)={\displaystyle {\sum }_1^N\left[\lambda 1‖bi-\widehat{b}\sigma (i)‖+\lambda GIoULGIoU\left(bi,\widehat{b}\sigma (i)\right)\right]}$$

(6)

The IoU reflects the intersection and concatenation ratio between the prediction frame and the real frame; the larger the IoU, the greater the coincidence of the predicted and real boxes. Hence, IoU can be used as an optimization function [16]. The GIoU describes the minimum bounding box required for optimization when the gradient is zero (i.e., the predicted and real boxes do not overlap). Assuming that the coordinates of the true box are gt and the coordinates of the predicted box are pb, the IoU is obtained by

3. Results and Discussion

3.1. Experimental Procedure

3.1.1. Datasets

In this study, three public datasets—LS-SSDD[17], SAR-Ship-Dataset[18], and HRSID[19]—were used as experimental test datasets.

(1) LS-SSDD adopts Sentinel-1 satellite data and contains a total of 30 large-scene SAR images. The large-scene images were taken from 30 original large-scene satellite-based SAR images. The polarization modes included two modes—VV and VH, and the IW— which has the distinctive features of large-scene ocean observation, small-scale ship detection, a variety of pure backgrounds, a fully automated detection process, and a variety of standardized benchmarks. Figure 6 shows a sample large-scene image from the LS-SSDD dataset. The large-scene image of the LS-SSDD dataset has a size of 24,000 × 16,000 pixels, three-channel grayscale image format, 24-bit depth JPG, and XML annotation format, which record the target position information. During the experiment, the first 20 original large-scene SAR images were selected as the training set, and the remaining images were selected as the test set. Each 24,000 x 16,000 pixel SAR image was directly cropped into 600 800 x 800 pixel sub-images without processing.

(2) The SAR-Ship Dataset, created by Wang et al. and labeled by SAR experts, is the most extensive publicly available dataset for multi-scale ship detection. It comprises 102 Chinese Gaofen-3 images and 108 Sentinel-1 images, totaling 43,819 ship chips. The chips have a resolution of 256 pixels and contain ships of various scales and backgrounds. The Gaofen-3 images were captured using ultrafine strip chart (UFS), Fine Strip Chart 1 (FSI), Fully Polarized 1 (QPSI), Fully Polarized 2 (QPSII), and Fine Strip Chart 2 (FSII) imaging modes, with resolutions ranging from 3 to 10 meters. Sentinel-1 images were acquired in S3 strip map (SM), S6 SM, and IW modes. The dataset also includes ships in complex scenes such as offshore, island, and harbor environments. Furthermore, the dataset covers scenarios with high ship densities and small target sizes (less than 15 × 15 pixels). Table 1 provides an overview of the dataset, with the training, validation, and test sets accounting for 70%, 20%, and 10% of the dataset, respectively.

(3) The HRSID dataset is designed specifically for ship detection, semantic segmentation, and instance segmentation tasks in high-resolution SAR images. It consists of a total of 5,604 images, including 99 Sentinel-1B, 36 TerraSAR-X, and 1 TanDEM-X images. Within these images, there are 16,951 ship instances, with small target scenes accounting for approximately 54.8% of all ships present. Similar to the construction process of the Microsoft COCO (Common Objects in Context) dataset, the HRSID dataset incorporates SAR images with varying resolutions, polarizations, sea states, sea areas, and coastal ports. This diversity enables researchers to benchmark and evaluate their methods effectively. The SAR images in the HRSID dataset have resolutions of 0.5 m, 1 m, and 3 m. To facilitate the development of algorithms, the dataset is split into three subsets: a 70% training set, a 20% validation set, and a 10% test set. This partitioning allows researchers to train their models, tune hyperparameters, and evaluate performance in a controlled manner.

Figure 7. Image samples. (a) SAR-Ship-Dataset (b) HRSID

Figure 7. Image samples. (a) SAR-Ship-Dataset (b) HRSID

3.1.2. Evaluation Metrics

The main evaluation metrics used in the detection model were precision, recall, and F1 score, which are defined as follows:

$$\begin{array}{l}P=N_{TD}/N,\\ R=N_{TD}/N_{GT},\\ F1-score=\frac{2\times Precision\times Recall}{Precision+Recall}.\end{array}$$

(7)

where N_TD denotes the number of correctly detected ship targets, N_GT is the actual number of ship targets, and N is the total number of detected ship targets. Different IoU thresholds were used to calculate different numbers of ship targets P(R). This value represents the precision–recall curve, and the purpose of the AP is to find the area under the precision–recall curve because it is the core index used to measure detection accuracy. The mean average precision (mAP) is the average value of all detection types. Because only one type of ship target is detected, mAP and AP have the same value.

(8)

(9)

4. Conclusions

References

1. Xiao, X.; Zhou, Z.; Wang, B.; Li, L.; Miao, L. Ship Detection under Complex Backgrounds Based on Accurate Rotated Anchor Boxes from Paired Semantic Segmentation. Remote Sensing 2019, 11, 2506. [Google Scholar] [CrossRef]
2. Kang, M.; Leng, X.; Lin, Z.; Ji, K. A Modified Faster R-CNN Based on CFAR Algorithm for SAR Ship Detection. In Proceedings of the 2017 International Workshop on Remote Sensing with Intelligent Processing (RSIP); IEEE; 2017; pp. 1–4. [Google Scholar]
3. An, Q.; Pan, Z.; You, H. Ship Detection in Gaofen-3 SAR Images Based on Sea Clutter Distribution Analysis and Deep Convolutional Neural Network. Sensors 2018, 18, 334. [Google Scholar] [CrossRef] [PubMed]
4. Yue, B.; Zhao, W.; Han, S. SAR Ship Detection Method Based on Convolutional Neural Network and Multi-Layer Feature Fusion. In Proceedings of the Advances in Natural Computation, Fuzzy Systems and Knowledge Discovery: Volume 1; Springer, 2020; pp. 41–53. [Google Scholar]
5. Shi, W.; Jiang, J.; Bao, S. Ship Detection Method in Remote Sensing Image Based on Feature Fusion. Acta Photonica Sin 2020, 49, 57–67. [Google Scholar]
6. Yonggang, L.; Weigang, Z.; Qiongnan, H.; Yuntao, L.; Yonghua, H. Near-Shore Ship Target Detection with SAR Images in Complex Background. Systems Engineering & Electronics 2022, 44. [Google Scholar]
7. Ma, W.; Li, N.; Zhu, H.; Jiao, L.; Tang, X.; Guo, Y.; Hou, B. Feature Split–Merge–Enhancement Network for Remote Sensing Object Detection. IEEE Transactions on Geoscience and Remote Sensing 2022, 60, 1–17. [Google Scholar] [CrossRef]
8. Ting, L.; Baijun, Z.; Yongsheng, Z.; Shun, Y. Ship Detection Algorithm Based on Improved YOLO V5. In Proceedings of the 2021 6th International Conference on Automation, Control and Robotics Engineering (CACRE), IEEE, 2021; pp. 483–487. [Google Scholar]
9. Tian, Y.; Luo, P.; Wang, X.; Tang, X. Deep Learning Strong Parts for Pedestrian Detection. In Proceedings of the Proceedings of the IEEE international conference on computer vision; 2015; pp. 1904–1912.
10. Ouyang, W.; Zhou, H.; Li, H.; Li, Q.; Yan, J.; Wang, X. Jointly Learning Deep Features, Deformable Parts, Occlusion and Classification for Pedestrian Detection. IEEE Trans. Pattern Anal. Mach. Intell. 2018, 40, 1874–1887. [Google Scholar] [CrossRef] [PubMed]
11. Jiao, J.; Zhang, Y.; Sun, H.; Yang, X.; Gao, X.; Hong, W.; Fu, K.; Sun, X. A Densely Connected End-to-End Neural Network for Multi-scale and Multiscene SAR Ship Detection. IEEE Access 2018, 6, 20881–20892. [Google Scholar] [CrossRef]
12. Sun, Z.; Meng, C.; Cheng, J.; Zhang, Z.; Chang, S. A Multi-Scale Feature Pyramid Network for Detection and Instance Segmentation of Marine Ships in SAR Images. Remote Sensing 2022, 14, 6312. [Google Scholar] [CrossRef]
13. Yang, R.; Pan, Z.; Jia, X.; Zhang, L.; Deng, Y. A Novel CNN-Based Detector for Ship Detection Based on Rotatable Bounding Box in SAR Images. IEEE J. Sel. Top. Appl. Earth Observations Remote Sensing 2021, 14, 1938–1958. [Google Scholar] [CrossRef]
14. Amjoud, A.B.; Amrouch, M. Object Detection Using Deep Learning, CNNs and Vision Transformers: A Review. IEEE Access 2023, 11, 35479–35516. [Google Scholar] [CrossRef]
15. Khan, A.; Rauf, Z.; Sohail, A.; Rehman, A.; Asif, H.; Asif, A.; Farooq, U. A Survey of the Vision Transformers and Its CNN-Transformer Based Variants. arXiv 2023, arXiv:2305.09880. [Google Scholar]
16. Rezatofighi, H.; Tsoi, N.; Gwak, J.; Sadeghian, A.; Reid, I.; Savarese, S. Generalized Intersection over Union: A Metric and a Loss for Bounding Box Regression. In Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2019; pp. 658–666.
17. Zhang, T.; Zhang, X.; Ke, X.; Zhan, X.; Shi, J.; Wei, S.; Pan, D.; Li, J.; Su, H.; Zhou, Y.; et al. LS-SSDD-v1.0: A Deep Learning Dataset Dedicated to Small Ship Detection from Large-Scale Sentinel-1 SAR Images. Remote Sensing 2020, 12, 2997. [Google Scholar] [CrossRef]
18. Wang, Y.; Wang, C.; Zhang, H.; Dong, Y.; Wei, S. A SAR Dataset of Ship Detection for Deep Learning under Complex Backgrounds. Remote Sensing 2019, 11. [Google Scholar] [CrossRef]
19. Wei, S.; Zeng, X.; Qu, Q.; Wang, M.; Su, H.; Shi, J. HRSID: A High-Resolution SAR Images Dataset for Ship Detection and Instance Segmentation. IEEE Access 2020, 8, 120234–120254. [Google Scholar] [CrossRef]
20. Ren, S.; He, K.; Girshick, R.; Sun, J. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. arXiv 2015, arXiv:1506.01497. [Google Scholar] [CrossRef] [PubMed]
21. Lin, Z.; Ji, K.; Leng, X.; Kuang, G. Squeeze and Excitation Rank Faster R-CNN for Ship Detection in SAR Images. IEEE Geoscience and Remote Sensing Letters 2018, 16, 751–755. [Google Scholar] [CrossRef]
22. Liu, S.; Qi, L.; Qin, H.; Shi, J.; Jia, J. Path Aggregation Network for Instance Segmentation. In Proceedings of the Proceedings of the IEEE conference on computer vision and pattern recognition; 2018; pp. 8759–8768.
23. Cai, Z.; Vasconcelos, N. Cascade R-Cnn: Delving into High Quality Object Detection. In Proceedings of the Proceedings of the IEEE conference on computer vision and pattern recognition; 2018; pp. 6154–6162.
24. Cui, Z.; Li, Q.; Cao, Z.; Liu, N. Dense Attention Pyramid Networks for Multi-Scale Ship Detection in SAR Images. IEEE Transactions on Geoscience and Remote Sensing 2019, 57, 8983–8997. [Google Scholar] [CrossRef]
25. Zhao, Y.; Zhao, L.; Xiong, B.; Kuang, G. Attention Receptive Pyramid Network for Ship Detection in SAR Images. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 2020, 13, 2738–2756. [Google Scholar] [CrossRef]
26. Wu, Y.; Chen, Y.; Yuan, L.; Liu, Z.; Wang, L.; Li, H.; Fu, Y. Rethinking Classification and Localization for Object Detection. In Proceedings of the Proceedings of the IEEE/CVF conference on computer vision and pattern recognition; 2020; pp. 10186–10195.
27. Woo, S.; Park, J.; Lee, J.-Y.; Kweon, I.S. Cbam: Convolutional Block Attention Module. In Proceedings of the Proceedings of the European conference on computer vision (ECCV); 2018; pp. 3–19.
28. Zhang, T.; Zhang, X.; Ke, X. Quad-FPN: A Novel Quad Feature Pyramid Network for SAR Ship Detection. Remote Sensing 2021, 13, 2771. [Google Scholar] [CrossRef]
29. Jocher, G.; Stoken, A.; Borovec, J.; Chaurasia, A.; Changyu, L.; Hogan, A.; Hajek, J.; Diaconu, L.; Kwon, Y.; Defretin, Y.; et al. Ultralytics/Yolov5: V5. 0-YOLOv5-P6 1280 Models, AWS, Supervise. Ly and YouTube Integrations. Zenodo 2021.
30. Wang, C.-Y.; Bochkovskiy, A.; Liao, H.-Y.M. YOLOv7: Trainable Bag-of-Freebies Sets New State-of-the-Art for Real-Time Object Detectors. In Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2023; pp. 7464–7475.