SAINet: Spatially Adaptive Interaction Network for Semantic Segmentation of High-Resolution Remote Sensing Images

1. Introduction

With the increasing prevalence of high-resolution remote sensing satellites in global Earth observation missions, high-resolution remote sensing data has become abundant and the primary source for Earth observation. Semantic segmentation of high-resolution remote sensing images [1,2,3,4,5] plays a vital role in understanding the distribution of ground object features, enabling refined urban management, environmental monitoring, natural resource assessment, crop analysis, precise surveying, and mapping. High-resolution remote sensing images possess distinct characteristics, including complex background information, dense targets, and rich ground object features. Consequently, effectively leveraging global and local features of targets amidst complex backgrounds and bridging the differences between various features becomes crucial for enhancing the accuracy of semantic segmentation.

Semantic segmentation of high-resolution remote sensing images [6,7,8] can be broadly categorized into two types: single-branch segmentation networks and dual-branch networks. Figure 1a illustrates the single-branch segmentation networks [9,10,11], which employ pure convolutional neural networks (CNNs) to enhance feature extraction capabilities. These single-branch architectures typically consist of well-known backbone networks such as FCN [12], UNet [13], DeepLabV3 [14], and their combinations. However, these methods often overlook the effective interaction between global and local features of target objects within the complex backgrounds of high-resolution remote sensing images. Moreover, these approaches commonly involve downsampling the high-resolution remote sensing images before feature extraction, leading to the loss of either high-resolution details or spatial contextual information.

In recent years, the dual-branch structure has gained significant attention and achieved remarkable advancements in semantic segmentation. Figure 1b illustrates these methods [15,16,17,18], which consist of two branches dedicated to extracting global and local features, respectively. The final segmentation is generated by aggregating high-level feature maps from both branches. This network architecture greatly enhances segmentation performance and effectively demonstrates the efficacy of extracting different features through separate branches. However, previous methods fused all the information from different branches without a mechanism for selectively incorporating spatial and positional features, treating each feature equally. Additionally, the fusion of global and local information was achieved through explicit interactions, such as simple concatenation or addition, lacking an adaptive mechanism.

In this paper, we propose a novel spatially adaptive interaction network (SAINet) for the dynamic interaction of information across different features. As shown in Figure 1c, SAINet concatenates the entire network through the spatial refined module (SRM) and spatial interaction module (SIM). To be specific, the whole image is first input to the backbone network to obtain coarse segmentation results. To ensure the coarse regions and enhance the segmentation accuracy, SRM is used to select local patches that are difficult to be segmented by the backbone network. Then crop the patches and they are input to the backbone network to obtain fine local features. To further enhance the segmentation accuracy, SIM is used to facilitate interaction between local salient areas and global information. As a result of integrating local and global information, it is possible to capture contextual information adaptively from multiple perspectives. To evaluate the effectiveness of the model, we conducted extensive experiments demonstrating that our proposed model outperforms state-of-the-art methods on the publicly available high-spatial-resolution image datasets DeepGlobe, Vaihingen, and Potsdam.

Our main contributions are summarized as follows:

• We propose a novel spatially adaptive interaction network (SAINet) for filtering and dynamic interaction among different spatial features in the remote sensing semantic segmentation task.
• We present a new spatial refined module that uses local context information to filter spatial positions and extract salient regions.
• We devise a spatial interaction model to adaptively modulate mechanisms to dynamically select and allocate spatial position weights to focus on target areas.
• We demonstrate the effectiveness of our approach by achieving state-of-the-art semantic segmentation performance on three publicly available high-resolution remote sensing image datasets.

2. Related Works

2.1. Semantic Segmentation

The Fully Convolutional Network (FCN) [12] pioneered the use of end-to-end convolutional networks for semantic segmentation, laying the foundation for this field. FCN eliminated connected layers, making it adaptable to inputs of any size. It utilized simple bilinear interpolation for initialization and employed deconvolution layers to upsample the feature maps from the final convolutional layer, preserving spatial positional information for pixel-wise predictions. Optional application of Conditional Random Fields (CRF) [19] improved classification mapping and segmentation results. The encoder-decoder framework was subsequently introduced with influential models like UNet [13] and SegNet [20]. UNet, structured as a U-shape, employed deconvolutional upsampling and feature map concatenation from corresponding scales in preceding layers. It achieved high speed and became a baseline network for medical image segmentation tasks. SegNet, although not the first encoder-decoder structure, successfully generalized the architecture, balancing memory (parameters) and accuracy. It utilized unspooling for feature map upsampling, reducing parameters and preserving high-frequency information. The concept of dilated convolutions [21] introduced background modules that used dilated convolutions for multi-scale aggregation, achieving dense prediction results without increasing parameters. Deeplabv2 [22] proposed the pyramid-like atrous spatial pyramid pooling (ASPP) technique, employing multiple parallel atrous convolution layers with different sampling rates for effective multi-scale processing. EMRN [23] proposes a multi-resolution features dimension uniform module to handle dimensional features from images of varying resolutions. PSPNet [24] enhances the ResNet [25] structure using atrous convolutions and incorporates a pyramid pooling module to capture contextual information at different scales. Deeplabv3 [14] improves spatial ASPP by cascading multiple atrous convolution structures and introduces batch normalization after each parallel convolutional layer. In 2018 [9], DeepLabv3 is adopted as an encoder architecture with an effective decoder module. It also explores the use of improved Xception and depth-wise separable convolutions to enhance the model’s performance in semantic segmentation tasks.

Recently, attention mechanism methods [26,27,28,29,30] have been introduced to semantic segmentation. Transformer mechanisms and self-attention mechanisms [31] were initially introduced in the field of natural language processing and later sparked widespread research interest in the field of computer vision [32,33,34,35]. The pioneering work, Vision Transformer (ViT) [36], utilizes multiple Transformer blocks to process non-overlapping image patches, establishing a convolution-free image classification model. PVT (Pyramid Vision Transformer) [37] draws inspiration from the pyramid structure of convolutional neural networks. It gradually reduces feature resolution by segmenting the input image into blocks of different scales. Transformer encoding blocks are then applied to each scale for feature extraction. This approach empowers PVT to handle features at various levels, resulting in remarkable performance across diverse image classification tasks. CFIL [38] proposes a frequency-domain feature extraction module and feature interaction in the frequency domain to enhance salient features. MFC [39] proposes a frequency-domain filtering module to achieve dense target feature enhancement. To avoid excessive attention computation, Swin Transformer [40] employs window-based local attention to confine attention within local windows. To fully leverage the advantages of Convolutional Neural Networks (CNN) in local feature extraction and the capabilities of Transformers in global relationship modeling, [41,42,43,44,45,46] combining these two methods allows for the simultaneous capture of image details and overall context. This integration aims to enhance image analysis and understanding and has the potential to yield improved results across various computer vision tasks. In certain situations, different regions may possess varying degrees of importance. Deformable attention [47] enables models to dynamically adjust their focus on different regions of an image based on object shapes and positions. This adaptive tuning allows the model to more accurately emphasize important object-related areas, thus enhancing its performance in image processing tasks. Recently, to reduce computational costs, researchers have introduced a coarse-fine-grained Visual Transformer (CF-ViT) [48] to alleviate the computational burden while maintaining performance.

2.2. Dual-Branch Architecture

In visual tasks involving natural images [49,50,51,52], remote sensing images [15,17,53,54], and visible light images [55,56,57,58,59], a substantial body of work has sought to address the subjectivity issue in balancing this trade-off by learning to integrate multi-scale information. Specifically, these approaches learn representations from multiple parallel networks and then aggregate information across different scales before making the final predictions. Taking the remote sensing domain as an example, GLNet [15] consists of a global branch and a local branch, tailored to address both global downscaled images and localized cropped image blocks. GLNet achieves high-quality segmentation outcomes, effectively balancing precision and memory consumption. UHRSNet [16] enhances and streamlines the local and global feature fusion approach of GLNet, enabling small blocks to gather information from surrounding ones. This effectively addresses the challenges posed by cropping and downsampling. HPGN [60] proposes a novel pyramid graph network targeting features, which is closely connected behind the backbone network to explore multi-scale spatial structural features. To fully harness the potential of the multi-branch architecture, MBNet [17] introduces a scaling module. The Zoom module relinquishes the prior L2 normalization before feature concatenation, instead exploiting the acquired attention to fuse distinct features. This approach maximizes the advantages of multi-resolution capabilities. The existing dual-branch architectures have shown promising results by seamlessly integrating patches and downsampled images during training. However, these approaches have primarily focused on merging information from different branches without considering the importance of spatial location features. They treat all extracted features uniformly, resulting in direct interactions (such as concatenation or addition) without an adaptive mechanism.

3. Method

We propose a spatially adaptive interaction network to utilize both global and local information better and achieve effective interaction between local saliency regions and global information: spatially adaptive interaction network, SAINet, a segmentation network based on high-resolution images. In Section 3.1, we first give an overview of the network. From Section 3.2 to Section 3.3, we further introduce the composition of the network, including the backbone network, spatially refined module (SRM), and spatial interaction module (SIM).

3.1. Overview

SAINet network, as shown in Figure 2, consists of the following parts: (1) Extracts global information from downsampled images; (2) Introduces a simple yet effective spatial refinement module, which utilizes local contextual information for spatial position filtering, extracting salient regions and their indices; (3) Extracts local detailed information; (4) Utilizes a spatial adaptive modulation mechanism for dynamically selecting and allocating spatial position weights, achieving the interaction of local salient regions and global information. The global feature extraction and local feature extraction are performed using the ResNet50-based FPN (Feature Pyramid Network), with the parameters of the two backbone networks shared.

Specifically, the proposed approach first uses the backbone network to extract features from the input image X, generating the global feature map $F_g$. It then uses the spatial refine module to obtain the saliency regions and their indices $SRI$. Subsequently, SAINet uses $SRI$ to crop the original image, obtain the cropped image block $X_c$, and apply the same backbone network to extract features from $X_c$, resulting in the local feature map $F_l$. To facilitate more effective interaction between the global feature $F_g$ and the local feature $F_l$, we introduce the spatial interaction module (SIM), a novel attention mechanism that combines linear attention and softmax attention. Our network can be formulated as:

$$F_g=B_{RA}\left(R\left(X\right)\right)$$

(1)

where the matrix X represents the input image, $R(·)$ denotes the resize operation, and $B_{RA}$ is the backbone network, using the Feature Pyramid Network (FPN) with ResNet50. $F_g$ is the global feature map with coarse extraction results.

$$SRI=SRM\left(F_g\right)$$

(2)

where $SRI$ represents the saliency region index, and $SRM$ denotes the spatial refine module, with details shown in subSection 3.2.

$$F_l=B_{RA}\left(X_c\right)=B_{RA}\left(SRI\left(X\right)\right)$$

(3)

where $SRI(·)$ denotes the cropping operation with SRI. The matrix $X_c$ represents the cropped image. $B_R(·)$ is the backbone network the same as formula 1. $F_l$ denotes the local feature map with more detailed information.

$$F_i=SIM\left(F_g,F_l\right)$$

(4)

where $F_i$ signifies the newly obtained features through interaction, and $SIM(·)$ stands for the spatial interaction module, with operational details in subSection 3.3.

3.2. Spatial Refine Module

Not all the extracted global features from down-sampled images are inaccurate, meaning not every local area needs enhancement. Especially in high-resolution remote sensing images, some objects are often continuous and occupy a large proportion of the entire image. Therefore, we have introduced a spatial refinement module that utilizes local context information to screen spatial positions and extract salient regions.

As shown in Figure 2, the global features $F_{\mathrm{Global}}\in {\mathbb{R}}^{H\times W\times \mathrm{Class}}$ are first passed through the spatial refinement module, followed by a softmax operation, and then a 1x1 convolution is employed to obtain the saliency level on the channel dimension, denoted as $P_{ij}\in (0,1)$. Hence, the confidence level of the global image is noted as $S_{\mathrm{Global}}\in {\mathbb{R}}^{H\times W\times 1}$, and Equation (2) presents the calculation formulas.

$$S_{\mathrm{global}}={Conv}_{1\times 1}\left(softmax\left(F_g\right)\right)=\left[\begin{array}{cccc}{p}_{11}& p_{12}& \cdots & p_{1j}\\ p_{21}& p_{22}& \cdots & p_{2j}\\ ⋮& ⋮& \ddots & ⋮\\ p_{i1}& p_{i2}& \cdots & p_{ij}\end{array}\right],1\le i\le H,1\le j\le W$$

(5)

To further enhance the expressiveness of salient regions, a sliding window is used to calculate the local region scores, resulting in the ScoreMap. In this paper, the appropriate sliding window size of 3x3 was determined through subsequent ablation experiments. The final window size s is set to 3x3. The threshold for determining salient regions denoted as T, is determined using the formula:

$$\mathrm{T}=t\times {log}_2^{\mu }$$

(6)

In this study, the values of t and $\mu$ are set to 0.08 and 7, respectively. Areas of the ScoreMap with scores below the threshold T are considered salient regions (SR) along with their corresponding index (SRI), where

$$\left\{\begin{array}{c}SR=\frac{1}{s\times s}\sum _{\begin{array}{c}1<i\le s\\ 1<j\le s\end{array}}{P}_{ij}\hfill \\ SRI,SR\le T\hfill \end{array}\right.$$

(7)

Subsequently, the original image is cropped using the SRI to determine local crop patches. The weight matrix of the salient region $W_S$ is learned by the spatial interactive module, for details see Section 3.3.

3.3. Spatial Interaction Module

Traditional feature fusion involves merging global and local features entirely, and the fusion method is merely a simple addition or concatenation. This is not only inefficient but also results in insufficient fusion. We all know that the large and sometimes even global, receptive field of the Transformer model gives it superior representational abilities compared to its CNN counterparts, especially in terms of understanding global context. Therefore, this paper presents a fusion method based on Transformer. However, simply enlarging the receptive field can also lead to some issues. On the one hand, using dense attention in ViT can result in excessive memory and computational costs, and the features may be affected by irrelevant parts beyond the region of interest. On the other hand, the sparse attention adopted in PVT or Swin Transformer is data-agnostic and may limit the ability to model distant relationships. Therefore, this paper proposes a novel interactive attention mechanism, where the interactive positions are selected in a data-dependent manner. This flexible attention mechanism can focus on relevant areas and integrate more informative features.

As shown in Figure 2, our spatial interaction module (SIM) is an innovative mode of attention that integrates the powerful Softmax attention with the efficient linear attention, represented as a quadruple (Q, S, K, V). With two inputs of N tokens represented as $F_g,F_l\in {\mathbb{R}}^{N\times C}$, $F_g$ and $F_l$ are obtained from Eq.(1) and Eq.(3) respectively. So self-attention can be formulated as follows in each head:

$$\begin{array}{c}Q=F_g{W}_Q,K=F_l{W}_K,V=F_l{W}_V,\\ O_i={\displaystyle \sum _{j=1}^N}\frac{Sim\left(Q_i,K_j\right)}{{\sum }_{j=1}^{N}Sim\left(Q_i,K_j\right)}{V}_j,\end{array}$$

(8)

where $W_{Q/K/V}\in {\mathbb{R}}^{C\times d}$ denote projection matrices, C and d are the channel dimension of the module and each head, and $Sim(·)$ represents the similarity function. When using $Sim(Q,K)=exp\left(Q{K}^T/\sqrt{d}\right)$ in Eq.(8), it becomes Softmax attention [31]. So Softmax attention can be abbreviated as:

$$O^{\sigma }=\sigma \left(Q{K}^T\right)V\triangleq {Attn}^{\sigma }(Q,K,V)$$

(9)

Where $W_{Q/K/V}\in {\mathbb{R}}^{C\times d}$ denote query, key, and value matrices and $\sigma (·)$ represents Softmax function. When similarity is measured as $Sim(Q,K)=\varphi \left(Q\right)\varphi {\left(K\right)}^T$ in Eq.(8), it becomes Linear attention [61]. So linear attention can be abbreviated as:

$$O^{\varphi }=\varphi \left(Q\right)\varphi {\left(K\right)}^{T}V\triangleq {Attn}^{\varphi }(Q,K,V)$$

(10)

Softmax and linear attention suffer from either excessive computation complexity or insufficient model expressiveness. So this paper introduces a set of additional saliency region tokens $S,\in {\mathbb{R}}^{d\times C}$into the traditional attention module. As shown in Figure 2, our interaction module consists of two Softmax attention operations. We initially treat tokens S as queries and perform attention calculations between S, K and V to aggregate local salient features $O^{V_S}$:

$$O^{V_S}=\sigma \left(S{K}^T\right)V\triangleq {Attn}^{\sigma }(S,K,V)$$

(11)

The saliency region tokens S first act as intermediaries for the query tokens Q, aggregating information from K and V, and then broadcasting the information back to Q. The number of saliency region tokens is determined by the RM module and is far less than the number of query tokens. The saliency region attention is notably more efficient than the widely used Softmax attention, while still preserving the global context modeling capability.

Subsequently, we utilize S as keys and $O^{V_S}$ as values in the second attention calculation with the query matrix Q, interactions the global information and local salient information to obtain the final output $O_S$:

$$\begin{array}{cc}\hfill O^{\mathrm{S}}& =\sigma \left(Q{S}^T\right)O^{V_S}\hfill \\ \hfill & ={\varphi }_q\left(Q\right){\varphi }_k{\left(K\right)}^{T}V\hfill \\ \hfill & =\underset{\mathrm{Generalized}\mathrm{Linear}\mathrm{Attn}}{\underbrace{{Attn}^{{\varphi }_{\mathrm{q}/\mathrm{k}}}(Q,K,V)}},\hfill \end{array}$$

(12)

Then our attention used in the interaction module can be written as:

$$O^{\mathrm{S}}=\underset{\mathrm{Information}\mathrm{Interaction}}{\underbrace{{Attn}^{\sigma }(Q,A,{\underbrace{{Attn}^{\sigma }(A,K,V)}}_{\mathrm{Local}\mathrm{Salient}\mathrm{Information}})}}$$

(13)

3.4. Loss

In the first training phase, we employ cross-entropy loss as a loss function to measure the difference between the prediction results of the backbone network and the ground truth data.

$$Loss=-ylog\left(p\right)-(1-y)log(1-p)$$

(14)

where p is the prediction result, and y is the ground truth data.

In the second training phase, We use the Focal Loss [62] with $\tau =6$ as the optimization target.

$$FocalLoss\left(p_t\right)=-{\alpha }_t{(1-p_t)}^{\tau }log\left(p_t\right)$$

(15)

where $p_t$ represents the probability that the second backbone network predicts for a certain category; ${\alpha }_t$ is used to balance the number of positive and negative samples, with a smaller ${\alpha }_t$ value assigned to categories with more samples and a larger ${\alpha }_t$ value to those with fewer samples; $\tau$ is used to adjust the imbalance between hard-to-separate and easy-to-separate samples. In this paper, we take $\tau =6$ to reduce the loss of easy-to-separate samples with a power function.

4. Experiment

We conduct experiments on three datasets to verify the effectiveness of our proposed method. DeepGlobe dataset, which primarily focuses on rural areas, and the Vaihingen and Potsdam datasets, which emphasize urban regions.

4.1. Datasets

Table 1. Three public semantic segmentation datasets of ultra-high-resolution (UHR) remote sensing images are used in this paper.

Dataset	Max Size	Number of Classes	%of 4K UHR Image	#Image
DeepGlobe [63]	6M pixels (2448×2448)	7	100%	803
Vaihingen *	28M pixels (2494×2064)	6	100%	33
Potsdam *	103M pixels (6000×6000)	6	100%	38

* http://www2.isprs.org/commissions/comm3/wg4/semantic-labeling.html.

Table 1. Three public semantic segmentation datasets of ultra-high-resolution (UHR) remote sensing images are used in this paper.

Dataset	Max Size	Number of Classes	%of 4K UHR Image	#Image
DeepGlobe [63]	6M pixels (2448×2448)	7	100%	803
Vaihingen *	28M pixels (2494×2064)	6	100%	33
Potsdam *	103M pixels (6000×6000)	6	100%	38

* http://www2.isprs.org/commissions/comm3/wg4/semantic-labeling.html.

Figure 3. Image sample and corresponding ground truth of(a) DeepGlobe and (b) Vaihingen and (c) Potsdam. The mask of DeepGlobe is an RGB image with seven classes following the Anderson Classification. Label colors are given at the bottom left of Figure 3. The labels of Vaihingen and Potsdam challenges include six categories; they are given at the bottom right of Figure 3.

Figure 3. Image sample and corresponding ground truth of(a) DeepGlobe and (b) Vaihingen and (c) Potsdam. The mask of DeepGlobe is an RGB image with seven classes following the Anderson Classification. Label colors are given at the bottom left of Figure 3. The labels of Vaihingen and Potsdam challenges include six categories; they are given at the bottom right of Figure 3.

The DeepGlobe Land Cover Classification dataset is the first publicly available benchmark that offers high-resolution sub-meter satellite imagery with a focus on rural regions. DeepGlobe provides real pixel masks for seven classes: urban in blue, agriculture in yellow, rangeland in purple, forest in green, water in blue, barren in white, and unknown. It consists of 1146 annotated satellite images, all sized at 2448×2448 pixels. Compared to previous land cover classification datasets, DeepGlobe has a much higher resolution and is more challenging.

The Vaihingen dataset comprises thirty-three images with a spatial resolution of 9 cm and an average image size of 2494 × 2064 pixels. Each image consists of three channels: red, green, near-infrared (NIR), and their corresponding digital surface model. The Vaihingen dataset contains six classes: impervious surfaces, buildings, low vegetation, trees, cars, and background. Sixteen images with ground truth are used for network training and validation, while the remaining seventeen images are reserved for testing. It’s worth noting that the digital surface model was not utilized in our experiments.

The Potsdam dataset consists of twenty-eight images, all of which have the same dimensions of 6000×6000 pixels, with a spatial resolution of 5cm for both the top-level imagery and DSM. Similar to the Vaihingen region, this dataset includes remote sensing TIFF files with three bands and a single-band DSM. Each remote sensing image covers the same geographic area, ensuring that both the images and DSM are defined within the same reference system (UTM WGS84). Additionally, each image is accompanied by an affine transformation file, enabling the decomposition of the image into smaller sections as needed. All images are densely labeled with the same six classes as the Vaihingen dataset. Twenty-four images, each containing ground truth data, are utilized for network training and validation, while the remaining fourteen images are reserved for testing purposes. To streamline computation, we chose not to utilize the DSM in our experiments. It is noteworthy that only the red, green, and blue channels were employed in our experiments.

4.2. Experimental Settings

To validate the performance of the proposed method, we compared SAINet with widely used single-branch networks and GLNet-based segmentation networks on the DeepGlobe, Vaihingen, and Potsdam datasets. In all the experiments, none of the networks utilized pre-trained weights. To demonstrate that our fusion method is more effective than GLNet’s fusion approach, both the downsampled global image and the cropped local patches share the same size as GLNet, 500×500 pixels. Neighboring patches have a 50-pixel overlap to avoid boundary vanishing for all the convolutional layers.

The experiments were implemented under the Pytorch framework, with Python version 3.8, and using an NVIDIA Geforce RTX2080Ti 11-GB GPU. To accelerate convergence during model training, we use the Adam optimizer $({\beta }_1=0.9,{\beta }_2=0.999)$ and set the momentum to 0.9 and weight decay to $1\times {10}^{-4}$ for training the global branch, and $2\times {10}^{-5}$ for the local branch. The number of training epochs was set to 150, and a minibatch size of 6 for all training. We used an early stopping mechanism in the training process to terminate the training when the performance of the test dataset did not increase in 10 epochs.

4.3. Evaluation Matrix

To evaluate the performance of the algorithm in this paper, four evaluation common metrics are used: the overall accuracy (OA), the F₁ score, the mean of classwise intersection over union (mIoU), and the accuracy of foreground categories (Acc). The OA is used to assess the proportion of correctly classified pixels to the total pixels. (OA) is defined as:

$$\mathrm{OA}=\frac{TP+TN}{N}$$

(16)

where TP(True Positive): the prediction result is a positive example, and the actual value is also a positive example, the prediction is correct, it is true; TN(True Negative): the prediction result is negative, the actual value is positive, and the prediction error is false.

For each category, the F₁ score can be computed by:

$$F_1=\left(1+{\beta }^2\right)·\frac{\mathrm{Precision}·\mathrm{Recall}}{{\beta }^2·\mathrm{Precision}+\mathrm{Recall}}$$

(17)

where $\beta$ is the equivalent factor of the precision rate and recall rate and is set to one.

We use the pixel-wise Intersection over Union (IoU) score as our evaluation metric. It was defined slightly differently for each class, as there are multiple categories. Assuming there are n images, the formulation is defined as:

$$Io{U}_j=\frac{{\sum }_{i=1}^{n}T{P}_{ij}}{{\sum }_{i=1}^{n}T{P}_{ij}+{\sum }_{i=1}^{n}F{P}_{ij}+{\sum }_{i=1}^{n}F{N}_{ij}}$$

(18)

where TP_ij is the number of pixels in image i that are correctly predicted as class j, FP_ij is the number of pixels in image i that are wrongly predicted as class j, and FN_ij is the number of pixels in image FP_ij that are wrongly predicted as any class other than class j. Note that we have an unknown class that is not active in our evaluation (i.e., the predictions on such pixels will not be added to the calculation and thus do not affect the final score). Assuming there are k land cover classes, the final score is defined as the average IoU among all classes as in Eqn. (18).

$$mIoU=\frac{1}{k}\sum _{j=1}^k{I}_j{U}_j$$

(19)

4.4. Results

To demonstrate the superiority of our proposed algorithm, we compared it with traditional semantic segmentation algorithms and the dual-branch GLNet network. Our network, as well as the single-branch networks, can only produce prediction results for patches. Therefore, we combine these patches to obtain the overall prediction result for the entire image. GLNet offers two methods to obtain predictions for the entire image, and we have chosen the combination method, which yields better results.

4.4.1. Results on DeepGlobe

To verify the performance of SAINet, we compare the model with other CNN-based and GLNet-based semantic segmentation networks on the DeepGlobe dataset. GLNet only computed the mIoU, so we are consistent with it here. The specific results are shown in Table 2. The results show that double-branch GLNet-based methods outperform single-branch CNN-based methods on the DeepGlobe dataset. SAINet has a significant performance improvement compared to other networks. Compared to GLNet [15] and MBNet [17], the mIoU of SAINet is improved by 6.2% and 5.2%, respectively. Compared to DeepLabv3+ [9], the top-performing single-branch network, the mIoU of SAINet is improved by 20.7%. Hence, our network achieves optimal results in all classes and the mIoU matrix, indicating that our model outperforms GLNet, UHRSNet [16], and MBNet. As UHRSNet and MBNet are based on improvements to GLNet, our experiments on the DeepGlobe dataset include an extended range of experiments incorporating UHRSNet and MBNet.

We also compared the inference speed of the networks. For SAINet and GLNet, the prediction time is for the entire original image. However, for single-branch networks like DeepLabv3+, ICNet [64], and PSPNet [24], the prediction time is the sum of the prediction time for all the smaller patches obtained by cropping the original image. As shown in Figure 5, our method and GLNet demonstrate superior inference speed compared to single-branch networks. Thanks to the design of the spatial refine module and spatial interaction module, our network achieves faster inference for the entire original image while maintaining accuracy, outperforming GLNet by 105.24 seconds. While DeepLabv3+ achieves high accuracy, its inference efficiency is compromised. Although ICNet improves inference efficiency, it struggles to maintain high accuracy. PSPNet, on the other hand, increases accuracy but at the cost of significantly longer inference time.

4.4.2. The Result on the Vaihingen Dataset

Table 2. Comparison on the DeepGlobe Dataset.

Method	Barren	Rangeland	Forest	Water	Urban	Agriculture	mIoU (%)
ICNet[64]	35.2	31.3	24.2	42.1	24.3	69.2	37.7
FCN[12]	36.1	36.4	35.5	38.9	26.2	72.3	40.9
U-Net[13]	34.1	33.5	28.2	45.1	26.1	71.7	39.8
PSPNet[24]	39.3	36.9	53.6	64.2	48.2	74.7	52.8
SegNet[65]	43.1	38.2	54.8	67.8	53.1	75.2	55.4
DeepLabv3+[9]	44.1	42.2	56.3	68.4	54.1	77.3	57.1
UHRSNet[16]	81.2	61.3	72.3	70.1	78.9	68.4	72.0
MBNet[17]	64.1	87.3	41.5	80.6	83.1	78.9	72.6
Baseline[15]	63.6	38.6	79.8	82.6	78.1	86.8	71.6
SAINet	86.5	46.1	83.1	84.4	79.2	87.3	77.8

Table 2. Comparison on the DeepGlobe Dataset.

Method	Barren	Rangeland	Forest	Water	Urban	Agriculture	mIoU (%)
ICNet[64]	35.2	31.3	24.2	42.1	24.3	69.2	37.7
FCN[12]	36.1	36.4	35.5	38.9	26.2	72.3	40.9
U-Net[13]	34.1	33.5	28.2	45.1	26.1	71.7	39.8
PSPNet[24]	39.3	36.9	53.6	64.2	48.2	74.7	52.8
SegNet[65]	43.1	38.2	54.8	67.8	53.1	75.2	55.4
DeepLabv3+[9]	44.1	42.2	56.3	68.4	54.1	77.3	57.1
UHRSNet[16]	81.2	61.3	72.3	70.1	78.9	68.4	72.0
MBNet[17]	64.1	87.3	41.5	80.6	83.1	78.9	72.6
Baseline[15]	63.6	38.6	79.8	82.6	78.1	86.8	71.6
SAINet	86.5	46.1	83.1	84.4	79.2	87.3	77.8

Figure 4. Inference time on the DeepGlobe dataset.

Figure 4. Inference time on the DeepGlobe dataset.

To verify the robustness of SAINet, we conducted comparative experiments on the Vaihingen dataset. The results are shown in Table 3 where SAINet yields mean F1 of 90.7%, OA 91.9%, mIoU 82.5% respectively and is quantitively superior to other methods. Specifically, compared with the second method, the proposed SAINet improvement in mean F1 is 0.9%, OA 1.3%, and mIoU% respectively. The "Building" class gains the highest classification accuracy of 96.5%. Especially in high-resolution remote sensing images, the Car, as a relatively small object, is difficult to identify in the Vaihingen dataset. Even so, the proposed SAINet reached 89.4% of the F1 score, significantly exceeding the secondary method. The experimental results show that SAINet maintains stable performance on the Vaihingen dataset, which indicates that SAINet has a better practical generalization ability.

4.4.3. Results on Potsdam

From the Table 4, it can be seen that the SAINet proposed in this paper also outperforms previous segmentation methods on the Postdam dataset. In detail, compared with BGFNet, SAINet obtains increments of 0.4% and 0.3% in mean F1 and OA, respectively. Notably, our method scored 96.2% on F1 in the “Car” category, outperforming other networks by more than 0.65%. Compared to the DeepGlobe dataset, our algorithm exhibits a slightly smaller improvement in the Vaihingen and Potsdam datasets. This is primarily because the Vaihingen and Potsdam datasets are already mature and well-established. As shown in Figure 3, relative to the DeepGlobe dataset, the Vaihingen and Potsdam datasets primarily focus on urban areas, characterized by well-defined boundaries and intricate details.

4.5. Ablation Studies and Analysis

In the previous subsection, we have shown the superiority of SAINet by comparing it to state-of-the-art methods. In what follows, we comprehensively analyze intrinsic factors that lead to SAINet’s superiority in the dataset, including: (1) The role of SRM and SIM. (2) The effectiveness of variant SRM. (3) The impact of variant SIM. (4) Compared with different attention modules.

The role of SRM and SIM. It can be observed from Table 5 that the two modules significantly influence the performance of the algorithm, and their effects on different datasets are consistent: the effect of using SIM alone is better than using SRM alone, while the best performance is achieved when both are used simultaneously. When SRM is used alone, the algorithm adopts the simplest addition operation to interact with the global and local features, which leads to less improvement compared to GLNet, as GLNet performs more complex interactions between all global and local features. However, there is still an improvement in mIoU compared to using a single-branch network. When SIM is used alone, it interacts with all features, resulting in some improvement, although not significant, as not all features require interaction, and excessive interaction may have a detrimental effect. Only when both SAM and SIM are used together, the network in this paper can achieve the best performance.

Figure 5. The effectiveness of variant window size of SRM.

Figure 5. The effectiveness of variant window size of SRM.

The effectiveness of variant window size of SRM. To explore which window size of SRM gives the best performance boost, we set the window size to $3\times 3,5\times 5,7\times 7$ respectively, and select mIoU as the evaluation metric. As shown in Figure 5, when the window size is $3\times 3$, the segmentation results are the best. With a window size of $5\times 5$, the segmentation results are slightly worse, and with a window size of $7\times 7$, the segmentation results are inferior. The size of the window of SRM is an important parameter when filtering spatial locations and extracting salient regions. According to the results, the segmentation results are best when the window size is $3\times 3$. This is because global features have already been extracted before the first backbone network. Therefore, a $3\times 3$ window is already quite large, and increasing the window size further would only introduce unnecessary interference or noise without improving the results. Hence, choosing a $3\times 3$ window size is wise in this scenario, as it provides the best segmentation results.

Compared with different attention modules. To further demonstrate the capability of our proposed spatial interaction module in understanding spatial relationships in data, we replaced the spatial interaction module with another attention mechanism and calculated the mIoU on three datasets. From Table 6, it can be seen that replacing our spatial interaction module with the convolutional block attention module results in a better mIoU on all three datasets compared to replacing it with the channel attention module and spatial attention module. The best results obtained after replacing with the channel attention module and spatial attention module are close, with differences of 0.4%, 0.3%, and 0.1% on the DeepGlobe dataset, Vaihingen dataset, and Potsdam dataset, respectively, indicating that the channel attention module and spatial attention module focus on important features to a similar extent. The convolutional block attention module, which concatenates the channel attention module and spatial attention module, further enhances the capability to focus on local information. The results of the proposed spatial interaction module on the three datasets are respectively 5.5%, 3.4%, and 8.6% better than the results of the convolutional block attention module on the three datasets. This is because our designed spatial interaction module is more compatible with the proposed spatial refine module, and it better achieves the interaction between global and local information.

The correlation between segmentation accuracy and patch size. As can be seen from Table 7, our network’s segmentation accuracy remains relatively stable despite variations in patch size. The highest and lowest segmentation accuracies of GLNe across different patch sizes differ by 7%, whereas the difference between the highest and lowest segmentation accuracies of our network across different patch sizes is only 3.5%. This is attributed to our proposed Spatial Refine Module and Spatial Interaction Module, which adapt better to features of different sizes. This stability allows the network to be trained with smaller patches, consequently reducing the hardware requirements.

4.6. Visual Results and Analysis

To further evaluate the performance of our algorithm on different datasets and highlight its advantages, we selected five representative image samples from both the Vaihingen dataset and the Postdam dataset. We analyzed and compared the prediction results using visualizations. For the single-branch network, we chose Deeplabv3+ as the representative, and for the multi-branch network, we selected GLNet.

From the visualization results in Figure 6, it can be seen that our algorithm shows a remarkable improvement on the Postdam dataset. Compared to the prediction results of other algorithms, our algorithm outperforms in predicting buildings, clutter, and low vegetation. In Figure 6-(1), the area of low vegetation within the black dashed box is small. Deeplabv3+ fails to segment it accurately, and although GLNet identifies low vegetation, it is not very precise and falsely classifies a small portion as clutter. In contrast, our algorithm yields the correct segmentation result, resembling the true label. In Figure 6-(2) and Figure 6-(3), it is evident that Deeplabv3+ and GLNet easily confuse clutter and low vegetation, whereas our algorithm in this paper can better discriminate between these two categories. For simple scenes like Figure 6-(4) and (5), while other algorithms can correctly segment it, our algorithm can extract more detailed segmentation. In Figure 6-(5), the small red clutter within the black dashed box is incorrectly classified as a car by Deeplabv3+. However, GLNet and our algorithm can accurately segment it, highlighting the advantage of the dual-branch network. The global branch in the dual-branch network takes the entire original image as input, enabling correct segmentation of clutter.

Figure 7 compares segmentation results using different algorithms on the Vaihingen dataset. It can be seen that our algorithm overall performs better, especially in building segmentation, where it produces more complete results with more accurate edges. This indicates that our algorithm can accurately separate buildings from other areas and produce clearer outlines. Additionally, our algorithm exhibits greater accuracy in segmenting smaller objects. Our algorithm can effectively identify and segment smaller targets, leading to more precise segmentation results.

5. Conclusions

In this paper, we propose a novel spatially adaptive interaction network (SAINet). By addressing the limitations of the previous methods from focus salient regions and spatial adaptive interaction, our module achieves an impressive result. Incorporating this adaptable method empowers the dual-branch network to prioritize pertinent regions, enhancing the network’s capacity to encompass richer informative features. Consequently, this refinement contributes to heightened segmentation accuracy. Extensive experiments demonstrate the effectiveness of our model over competitive baselines. Although a complete analysis model has been established, there are still several limitations: first, in the comparative experiments on the Vaihingen dataset, our method achieves slightly lower segmentation F1 scores for the categories "Impervious Surface" and "Tree" compared to the best-performing algorithm. Second, We only conducted comparative experiments on high-resolution remote sensing images and lacked comparative experiments on high-resolution images in other scenarios. In future work, we aim to further improve segmentation accuracy while also conducting comparative experiments on high-resolution images in other scenarios.