Few-Shot Remote Sensing Scene Classification Based on Diffusion Augmentation and Multimodal Feature Fusion

1. Introduction

Remote sensing scene classification has been widely used in the fields of land use [1] urban planning [2], environmental monitoring [3], disaster detection [4], and so on. In recent years, the methods of remote sensing scene classification based on deep learning have made great progress. However, these methods typically require a large number of labeled samples [5] for model training, and the scarcity of labeled data limits their generalization ability heavily, especially in real-world scenarios where labeled data are lacking or unavailable.

In order to deal with the limitation of labeled data scarcity, few-shot remote sensing scene classification (FSRSSC) has gradually become a research hotspot. Different from the general classification tasks, FSRSSC aims to learn transferable prior knowledge from a large number of labeled data base class for classification of the remote sensing scene with limited samples. In accordance with the training approach, FSRSSC methods can be categorized as follows：1) Transfer learning-based methods [6,7,8,9,10]; 2) Attention mechanisms-based methods [7,11,12,13,14,15,16]; 3) Novel network architectures-based methods [11,12,17,18,19].

The current FSRSSC methods mainly extract remote sensing image features through convolutional neural networks and convert them into feature vectors through the pooling layer, thereby achieving classification. However, since remote sensing images are captured from a longer distance and a wider range of viewpoints through sensing devices such as satellites, aircraft or other unmanned aerial vehicles, a single image often contains complex and diverse ground scenes as well as spatially scattered objects. It results in the large intraclass and small interclass variances in remote sensing images, as is shown in Figure 1. If only the global features are paid attention to, it may lead to the loss of the useful local details in the images, thereby affecting the classification accuracy. Therefore, while extracting the global features of remote sensing images, the local features with more details are combined to enhance the representational ability in this paper. It should be noted that it is crucial that the extracted local features must be related to the classified image closely, otherwise, the irrelevant local features will reduce the classification accuracy. Meanwhile, semantic textual information is leveraged to refine the prototypes of the support set in few-shot scenarios, bringing them closer to the true class prototypes, thereby enhancing accuracy [20].

In summary, to solve the scarcity problem of labeled data and the separability problem of few-shot classifier for the FSRSSC task, an effective approach is proposed in this paper which is based on data augmentation and multimodal feature fusion. The primary contributions of this work can be summarized as follows:

1) We use the diffusion mode to implement data enhancement, which is applied to solve the challenges of data scarcity in FSRSSC, and in the case of meeting the actual data distribution, it reduces the similarity and performance gap between synthetic and real data.

2) Proposal frames are divided for marking local landscape according to the type and spatial distribution of ground objects, and the local feature is extracted from the above frames to improve the feature representation.

3) A multimodal feature fusion module (MFFM) is constructed in this paper. This module obtains the image characteristics with high quality by combining the global channel features, the local space and the semantic ones, to develop classification accuracy in the FSRSSC task.

2. Related Work

2.1. Few-Shot Remote Sensing Scene Classification

FSRSSC is an important branch of remote sensing image classification, which aims to learn a classifier to recognize remote sensing image classes during training with limited labeled examples. Xing et al. [6] proposed to exploit two pre-trained models to classify the remote-sensing scene, respectively. The models comprehensively consider the contribution of various decisions and further improve the discrimination of features. Gong et al. [7] introduced a novel two-path aggregation attention network with quad-patch data augmentation. The network makes it easier to focus on the key clues in a targeted manner. Ji et al. [8] introduced a baseline model using a standard cross-entropy loss leveraging two auxiliary objectives to capture intrinsical characteristics across the semantic classes. Li et al. [9] proposed a multiform ensemble self-supervised learning framework for FSRSSC. The framework can achieve an effective compromise between expensive computational cost and classification accuracy. Ji et al. [10] proposed a two-stage framework that first learns a general-purpose representation and then propagates knowledge in a transductive paradigm. This framework finds an expected prototype having the minimal distance to all samples within the same class. Although transfer learning has shown good results, it requires additional training time and a large amount of labeled data to adapt to the features of the target domain, which may lead to a decline in its performance with few shots. This paper addresses the issue of data scarcity by using the diffusion mode.

Li et al. [11] introduced an attention metric network to improve the performance of one-shot scene classification. The network is composed of a self-attention embedding network and a cross-attention metric network. Li et al. [12] proposed a discriminative learning of adaptive match network. The network can incorporate the channel attention and spatial attention modules, achieving “discriminative learning.” Li et al. [13] introduced an end-to-end framework called self-supervised contrastive learning-based metric learning network. The framework learns representative image features from few annotated samples through multi-task learning, and fuses multi-scale spatial features through a novel attention module. Zeng et al. [14] proposed a task-specific contrastive learning model. This model learns feature correlations and reduces the background interference through a self-attention and mutual-attention module. Xu et al. [15] introduced an end-to-end metric learning framework named attention-based contrastive learning network. The framework employs the attention-based feature optimization module to align and enhance target image features. Zhang et al. [16] proposed a few-shot multi-class ship detection algorithm with attention feature map and multi-relation detector. The algorithm enhances the features of the target and optimizes the detection head of YOLO. Although attention mechanisms can improve the classification performance, but they ignore the semantic information of the images. This paper enables the multimodal representations to more closely approximate the real prototype with semantic embeddings.

Cui et al. [17] proposed a method called meta-kernel networks via integrating a parametric linear classifier into the meta-learning framework. The PLC learns prior knowledge and the meta-kernel strategy remaps low-dimensional indistinguishable features to a high-dimensional space. Li et al. [18] proposed a RS-MetaNet. It raises the learning level by organizing training in a meta way, and learns a metric space that can well classify remote sensing scenes from a series of tasks. Qin et al. [19] proposed the deep updated subspace network. The network uses class subspace as a metric benchmark to represent the commonality of a category and can effectively mitigate the negative impact of irrelevant objects on the classifier. Although novel network architectures have acquired good performance, but they have a large number of parameters to be tuned with experience. This paper uses the general four convolution blocks (Conv4) as base network, and the proposal frame and self-attention module are applied to obtain the local features. This network architecture has reduced the total number of parameters while improving the classification performance.

2.2. Data Augmentation

In both general and few-shot image classification, data augmentation expands the number of available images per class and generates novel classes and tasks [21]. To overcome the data scarcity problem for FS-RSSC, several methods have been proposed to augment the training data in different ways. For example, the vast amount of visual data was harvested simply through 2d rotations [22,23,24] and crops [25]. And some methods used mixup [26] and cutmix [27]. In recent years, some research has employed more refined strategies. Li et al. [28] used GAN networks to emulate the target data distribution. Gong et al. [7] proposed the quad patch method to generate synthetic samples by cutting and reassembling patches from existing images. Ni et al. [29] found that it was more effective by increasing the number of query samples and tasks during meta-testing than increasing the number of support samples during meta-training. Yang et al. [30] proposed to enrich the feature space by simulating the distribution of neighboring classes with spatial vector enhancement. Chen et al. [31] performed data augmentation on images using the SimSiam method. Hou et al. [32] proposed a random augmentation sampling strategy for augmented samples. However, the samples produced through the methods above are always either low quality or may encounter pattern collapse or distortion. In this paper, we embed iterative minor changes into the initial data to generate new samples. It can reduce the performance gap between synthetic and real data, as well as capturing the initial data distribution and ensuring data coverage [33].

3. Materials and Methods

In this section, taking into account the constraints of FSRSSC data scarcity, coupled with the challenges posed by the high intraclass and low interclass variance, we introduce a multimodal feature fusion network (MMFF-Net), as shown in Figure 2. It consists of diffusion augmentation (DA), multiscale feature fusion (MSFF), dual attention fusion module (DAFM), and information interaction mutual attention (IIMA) module. The DA uses the diffusion model for data enhancement through increasing and reducing gaussian noise to the original support data. And the MSFF focuses on obtaining the local spatial details. In addition, the DAFM is utilized to extract critical remote sensing image features with parallel spatial and channels attention. What is more, the IIMA is employed to facilitate interactions between the target features of the support set and the query set, generating the cross-attention map.

3.1. Problem Definition

In this study, each RS scene dataset D is divided into three sets: training set D_train, validation set D_val, and test set D_test. The scene categories in D_train, D_val and D_test are distinct and mutually exclusive, that is, D_train∩D_val = ∅, D_train∩D_test= ∅, D_val∩D_test = ∅, D_train∪D_val∪D_test = D. There are three corresponding tasks which are training tasks, validation tasks, and test tasks, respectively. In each meta-task, we utilize the classical N-way K-shot setup, that is, N distinct categories are randomly selected with K + L scene images in every category. The K labeled samples are designated support set D_S = (x_i^s , y_i^s) _i=1^N×K and the remaining L scene images are for query set D_Q = (x_j^q , y_j^q)_j=1^N×L，where x_i^s denotes the ith support image and y_i^s represents the label of x_i^s. It is similar for x_j^q and y_j^q. The network is trained on the D_S and its performance is evaluated on the D_Q. It is important to note that for each task, the remote sensing scene images in the D_S will not appear in the D_Q (D_S ∩ D_Q = ∅). In this paper, we set N to 5 and K to 1 or 5 by the general experience, corresponding to the five-way one-shot and five-way five-shot settings.

3.2. Method Overview

The architecture of the MMFF-Net proposed in this paper is depicted in Figure 2. We utilize DA for data enhancement in the D_S. Then the enhancement data is put into Conv4 followed by MSFF for extracting local features F_L based on spatial attention. Meanwhile the representative global features F_G based on channel attention are extracted from the proposal frames of the enhancement data through Conv4. Then the F_L and F_G are fused to F_s by DAFM. In order to interact between the D_S and D_Q information, the IIMA module is utilized for the fusion features F_fusion, which is more distinct than F_s and F_q. And semantic features are extracted through the Word2Vec [34] model trained on the Wikipedia corpus during training. Finally, the WM Loss function based on the semantic correlation matrices is employed to maintain clear boundaries between different scene categories, ensuring distinctiveness among them.

3.3. Diffusion Augmentation

The DA utilizes the diffusion model to enhance the data, where adds and reduces Gaussian noise to the D_S image features to generate labeled samples for augmentation. As is shown in Figure 3, the DA module consists of two processes: forward noise-adding and reverse denoising. In the process of the forward process, the diffusion module gradually adds Gaussian distribution noise to the D_S. For a sample x in the D_S, it is transformed from x₀^s to x₁^s, x₂^s,…, x_T^s based on the time step t. The transitioning probability from x_t^s to x_t-1^s at step t is derived as following:

$$P\left(x_t^s|x_{t-1}^s\right)=G\left(\sqrt{1-{\alpha }_t}{x}_{t-1}^s,{\alpha }_{t}N\right)$$

(1)

where $x_t^s$ and $x_{t-1}^s$ denote the remote sensing images with noise-adding at time steps t and t-1. G(.) represents the Gaussian distribution, ${\left\{{\alpha }_t\right\}}_{t=0}^T\in (0,1)$ is the diffusion rate at time step t, and N denotes the standard normal distribution. Thus $\sqrt{1-{\alpha }_t}{x}_{t-1}^s$ and ${\alpha }_{t}N$ represent the mean and variance of $P\left(x_t^s|x_{t-1}^s\right)$ respectively.

During the reverse denoising process, the diffusion module gradually denoises the remote sensing images with noise-added generated in the forward process. At the tth step, the remote sensing image instance $x_{\mathrm{T}-t}^s$ is obtained from the instance $x_{T+1-t}^s$ with Gaussian noise through the denoising process, and the conditional probability is computed according to the following formula:

$$P^{\text{'}}\left(x_{T-t}^s|x_{T-t+1}^s\right)=G\left({\mu }_{\theta }(x_{T-t+1}^s,t),{\sigma }_{T-t+1}^{2}N\right)$$

(2)

where ${\mu }_{\theta }(x_{T-t+1}^s,t)$ and ${\sigma }_{T-t+1}^2$ represent respectively the mean and variance of $P^{\text{'}}\left(x_{T-t}^s|x_{T-t+1}^s\right)$. Thus the augmented image dataset is named D_aug, which comprises the D_S and the augmentation images generated through the diffusion module. It will be utilized for training the proposed MMFF-Net and optimizing it parameters.

3.4. Dual Attention Fusion Modul (DAFM)

The proposed DAFM is shown as Figure 4. The conv4 is utilized to extract image features as the basic network. It consists of four convolution blocks, each of which includes 3 × 3 convolution, a normalized, a ReLU, and a 2 × 2 max pooling layer. About 32, 64, 128, and 256 are the output channels of the four convolution blocks, respectively [12]. The input RSS image is denoted as $x_i\in {\Re }^{C\times H\times W}$, where its channels number, height and weight are C, H and W respectively. Then a feature map F can be obtained through the Conv4, which is represented as the following equation:

$$F=g_{\phi }(x_i;\phi )$$

(3)

where $F\in {\Re }^{C^{\text{'}}\times H^{\text{'}}\times W^{\text{'}}}$, $g_{\phi }$ with the parameters $\phi$ represents the Conv4. Based on the feature map F, we can acquire the precise representations of the RSS image through DAFM, which effectively captures and fuses the abundant feature information from both spatial attention and channel attention. The spatial attention is employed to focus on spatial details, such as object positions and local structures, through a local branch in the image, whereas the channel attention is used to integrate global semantic information through a global branch.

1) Local Feature Based on Spatial Attention: We utilize the multiscale feature fusion (MSFF) module to obtain the local spatial details. As is shown in Figure 5, we employ four different scale feature extraction branches based on the output of the Con4. Each branch comprises a convolutional, a normalized, and a ReLU layer, and the convolution kernels of the four branches are set to the typical sizes (1×1, 3×3, 5×5 and 7×7). Then the four branch outputs are separately input to the self-attention module (SAM) for catching the relationships among various regions in the RSS images, and the feature map F_c is respectively converted into three different feature spaces U, V and W through a 1×1 convolution.

In order to reduce the channels number, the input channels C_in is set C′/4 and output channels is set C_in- /8 in U and V, while the output channels C_out is set C′/4 in W [35]. Then U, V and W are reshaped to $\tilde{U}$, $\tilde{V}$ and $\tilde{W}$along the spatial dimensions. Then the relationships between positions in the spaces $\tilde{U}$ and $\tilde{V}$ can be obtained

$$R={\tilde{U}}^T\tilde{V}$$

(3)

where $R\in {\Re }^{H\text{'}W\text{'}\times H\text{'}W\text{'}}$, ${\tilde{U}}^T\in {\Re }^{H\text{'}W\text{'}\times (C_{in}/8)}$, ${\tilde{V}}^T\in {\Re }^{(C_{in}/8)\times H\text{'}W\text{'}}$, $R_{ij}$ denotes correlation between the ith pixel position and the jth pixel position in the feature space. The self-attention feature map between the spaces $\tilde{U}$ and $\tilde{V}$can be computed according to the following equations:

$$A_{ij}=Soft\max (R_{ij})=\exp (R_{ij})/{\displaystyle {\sum }_{j=1}^{H\text{'}W\text{'}}\exp (R_{ij})}$$

(4)

where Softmax(·) represents rowwise normalization, A_ij denotets the attention weight of the jth column on the ith row pixel position, and ${\displaystyle {\sum }_{j=1}^{H\text{'}W\text{'}}{A}_{ij}=1}$. And the self-attention feature map F_a of the three spaces can be calculated as follows:

$$reshape(R^{C_{in}\times H\text{'}W\text{'}})=R^{C_{in}\times H\text{'}\times W\text{'}}$$

(5)

$$F_a=reshape(\tilde{W}{A}^T)$$

(6)

where the ith column of $\tilde{W}\in {\Re }^{C_{in}\times H\text{'}W\text{'}}$ represents values of all channels at the ith pixel position in the space $\tilde{W}$. Then, the final feature map $F_c^{\text{'}}$ through the SAM is denoted as follows:

$$F_c^{\text{'}}=\gamma F_a+F_c$$

(7)

where $F_c^{\text{'}}\in {\Re }^{C_{in}\times H\text{'}\times W\text{'}}$, and$\gamma$denotes the learnable scale parameter.

Then we can obtain four different scale feature maps with self-attention in the feature learning branches. The feature maps are presented as $F_{c1}^{\text{'}}$, $F_{c2}^{\text{'}}$, $F_{c3}^{\text{'}}$, and $F_{c4}^{\text{'}}$ with the same size ${\Re }^{C_{in}\times H\text{'}\times W\text{'}}$, and they are concatenated to the multiscale feature map F_m with the size ${\Re }^{C\text{'}\times H\text{'}\times W\text{'}}$. Then the local feature F_L is obtained based on spatial attention.

2) Global Feature Based on Channel Attention: In order to obtain the representative local landscapes in the RSS images, we employ a proposal frame extraction strategy [36] which is designed to emphasize on the spatial distribution and types of land cover within the scene. It bases on the case that the more diverse and denser the distribution of land features, the more likely a proposal frame represents a local landscape. As is shown in Figure 6, we choose (including, but not limited to) three typical representative land feature types from the diverse types and manifestations of local landscapes: buildings, vegetation and impervious surfaces, and extract the corresponding semantic pixel set by using MBI index [37], NDI index [38] and BCI index [39]. Then the set of proposal frames can be extracted with a clustering index DS_loc through combining the richness of feature types and their distribution density.

The DS_loc is computed according to the following formula:

$$D{S}_{loc}={\displaystyle \sum B_m(1-\frac{d_i}{H})}/{\displaystyle \sum (1-\frac{d_i}{H})}$$

(8)

where H denotes the diagonal length of the current proposal frame F_rame, d_i signifies the distance from the particular pixel i to the center pixel of F_rame, and B_m (m = 1, 2, 3) denotes respectively the values of MBI, ND and BCI for the given pixel i. The specific steps in detail are as follows:

Step 1: For a particular pixel i in the image, it is used as the center and performed proposal frame growth with 8-CONNECTIVITY [40]. Proposal frame growing stops and the current proposal frame is recorded as Frame_i when the DS_loc mentioned above increases continuously for three iterations and then decreases continuously for three iterations. Otherwise, if it reaches the image boundary during proposal frame growth, there is no corresponding proposal frame for pixel i.

Step 2: Repeat Step 1 by traversing all pixels in the image to obtain all the proposal frame collection, denoted as Frame_all.

Step 3: To avoid falling into local optima, the total number of image pixels is represented as pixel_total and divided into ten equal intervals. On this basis, the number of pixels is calculated in each proposal frame of Frame_all and they are clustered based on the respective intervals. Finally, the optimal proposal frame collection Frame_opt is obtained by choosing the proposal frames with the highest DS_loc value (or tied highest values) from each interval for the image x_i.

Then the Frame_opt is input into the Conv4 followed by channel attention for extracting the representative global feature F_G.

3) Feature Fusion Based on Dual Attention: After generating the $F_G$ and $F_L$, the DAFM is utilized for their fusion. As is shown in Figure 4, To minimize the number of parameters, the parameter matrices for Conv₁1×1 and Conv₂1×1 are set respectively as $M_1\in {ℝ}^{{\displaystyle \raisebox{1ex}{$c\times c$}\!\left/ \!\raisebox{-1ex}{$r$}\right.}}$ and $M_2\in {ℝ}^{{\displaystyle \raisebox{1ex}{$c$}\!\left/ \!\raisebox{-1ex}{$r\times c$}\right.}}$, and r is a constant. As is mentioned above in the paper, $F_G$ and $F_L$ concentrate on global semantics and local detail of the input RSS image respectively. They are fused according to the following formula:

$$F_s=F_L\oplus F_G$$

(9)

where $\oplus$ denotes element-wise addition, thus the $F_s$ is more detailed and richer in the image information.

3.5. Information Interaction Mutual Attention (IIMA)

To interact the information between the query and support sets, the IIMA module is utilized to generate a cross-attention map, represented as A_s (A_q), which serves to emphasize the regions corresponding to the target objects. It can assist the network in learning more distinct image features. As is shown in Figure 7, the IIMA module comprises two primary modules:

1) Feature map fusion module: The support set feature map $F_s\in {\Re }^{C\text{'}\times H\text{'}\times W\text{'}}$ and the query set feature map $F_q\in {\Re }^{C\text{'}\times H\text{'}\times W\text{'}}$ are standardized by a 1 × 1 convolution operation across the channel dimension. Then the fusion feature $F_{fusion}\in {\Re }^{H\text{'}\times W\text{'}\times H\text{'}\times W\text{'}}$ is calculated as follows:

$$F_{fusion}(x_s,x_q)=sim(F_s,F_q)$$

(9)

where x_s and x_q represent the corresponding spatial positions within the feature maps F_s and F_q, and sim denotes the cosine similarity function.

2) Cross-attention computation module: To pay more attention to the information of the target regions, this cross-attention module is utilized for identifying these regions through the similarities shared by the support and query sets. In the support set branch, the H’ and W’ dimensions of the feature map$F_{fusion}$ are eliminated by a GAP operation, then the resulting tensor is downsampled and upsampled respectively through the convolution module. The weights ω of the target regions are computed as follows:

$$\omega =conv(Relu(GAP(F_{fusion})))$$

(10)

where conv represents the convolution operation, while Relu denotes a nonlinear activation function, and GAP signifies the global average pooling. Then the final attention map$A_s\in {\Re }^{H\text{'}\times W\text{'}\times H\text{'}\times W\text{'}}$is calculated as follows:

$$A_s(x_s)={\displaystyle \sum _{x_q}\frac{\exp (C({\omega }^T{x}_s,{\omega }^T{x}_q)/\tau )}{{\displaystyle {\sum }_{x_s}^{H\text{'}\times W\text{'}}\exp (C({\omega }^T{x}_s,{\omega }^T{x}_q)/\tau )}}}$$

(11)

where x_s and x_q represent the spatial positions within the final feature map, and $C({\omega }^T{x}_s,{\omega }^T{x}_q)$ signifies the matching values of x_s and x_q according to their similarity, while$\tau$denotes the temperature factor.

In the support set D_S, the final embedding feature map is signified by $s\in {\Re }^{C_s\times H\times W}$, which is calculated as the following equation:

$$s={\displaystyle \sum _{x_s}{A}_s(x_s)}\otimes F_s(x_s)$$

(12)

where$\otimes$denotes element-wise multiplication operation. Similarly, the embedding feature map $q\in {\Re }^{C_q\times H\times W}$ in the query set D_Q is computed as follows:

$$q={\displaystyle \sum _{x_q}{A}_q(x_q)}\otimes F_q(x_q)$$

(13)

In the IIMA module, the cross-attention map is the critical portion, which focuses on the information of the target regions within both the query set and the support set. Thus the module can emphasize the relevance of features and the final feature map is more distinct.

3.6. Loss Function

In order to compare the final embedding feature map F (s or q) with the semantic information of the scene, we utilize the Word2Vec model trained on the Wikipedia corpus to transform each scene class name into a semantic vector, as is shown in Figure 2, and obtain the semantic vector sets $u=\left\{u_1,u_2,\mathrm{...},u_v\right\}$, where v is the total number of scene categories. For a scene class $u_i$, the loss function based on the popular LSE Loss (least square embedding loss) [41] is as the following equation:

$$L(u)={{\displaystyle \sum ‖F-u_i‖}}^2+\lambda ‖Z‖$$

(14)

where F represents s or q, $\lambda$ is a regularization parameter, and Z denotes a randomly initialized encoding matrix for the semantic information branch to align the visual feature and semantic feature.

4. Results

4.1. Dataset Introduction

To validate the efficacy of our proposed approach, experiments were conducted on three public benchmark RS datasets: UC Merced [42], AID [43], and NWPU-RESISC45 [44]. In order to make a fair comparison, the datasets were divided into training, validation, and test sets with the scheme used in [12,45,46]. Table I shows the details of three RS scene datasets used for the FSRSSC task.

The UC Merced (UCM) dataset was created by the UC Merced Computer Vision Laboratory. The dataset consists of 21 scene categories with 100 images in each class, accumulating to a total of 2100 images. Every image is 256 × 256 pixels with a spatial resolution of 0.3 m per pixel. As is shown in Table I, the training set includes 10 classes, the validation set includes 5 ones, and the testing set includes the remaining 6 ones [12,45].

The AID dataset, created by Wuhan University and Huazhong University of Science and Technology, consists of 30 scene categories with a total of 1000 RGB images. Each class comprises 220∼420 images with 600 × 600 pixels, and the spatial resolution varies from 0.5 to 8 m. As shown in Table 1, we adopt 16, 7 and 7 classes for training, validation, and testing, respectively [46].

The NWPU-RESISC45 dataset, provided by Northwestern Polytechnical University (NWPU), contains 45 scene categories with a total of 31500 images. Each class has 700 images with 256 × 256 pixels. The spatial resolution of the most images varies from 0.2 to 30 m per pixel. As shown in Table I, the number of classes for the training, validation, and testing are 25, 10, and 10, respectively [12,45].

4.2. Implementation Details

All models were implemented within the PyTorch framework, and run on hardware with an NVIDIA A40 GPU. Prior to model input, images underwent preprocessing to a fixed spatial resolution of 256×256 pixels. And each task was set as 5-way 1-shot and 5-way 5-shot as usual, with each meta-task comprising 15 query samples chosen for each class. During the meta-training phase, we trained for 100 epochs, with each epoch consisting of 1000 training meta-tasks and 600 validation ones. The classification accuracy of our method was the average of 5 runs randomly sampling 2000 meta-tasks in each run, with a 95% confidence interval. The final chosen hyperparameter settings were outlined as follows: the learnable scale parameter $\gamma$ in the SAM was set to 0.6, the constant r used to minimize the number of parameters in the DAFM was set to 8, the temperature coefficient$\tau$for the IIMA module was set to 5, and the regularization parameter$\lambda$ was set to 0.05. Furthermore, we utilized SGD to refine the model parameters with an initial learning rate of 0.0001 and a weight decay factor of 0.0005. At 1-shot and 5-shot, the learning rate decayed every 100 epochs and 50 ones with a decay factor of 0.5 and 0.1 respectively.

4.3. Results Comparison of Multiple Models

To clearly validate the advantages of our MMFF-Net, we conduct a direct comparison with several FSL methods, including MatchingNet [47], MAML [48], ProtoNet [49], RelationNet [50], LLSR[51], DeepEMD [43], DLA-MatchNet [12], FRN [52], MCL-Katz [53], GLIML [54], SPNet [45], SCL-MLNet [13], SPFSR [55], MPCLNet [56], HProtoNet [57], MSoP-Net2024 [58], and FS-RSSCvS [35]. The experimental comparison results on the three RS datasets are shown in Table 2. The optimal results are emphasized in bold, while the sub-optimal results are marked with an underline, respectively.

The comparison results of the classification accuracy on the UCM,AID and NWPU-RESISC45 datasets are shown in Table II. Our method attains better classification results than the other approaches listed above, except that its performance is slightly worse than that of DLA-MatchNet on AID within the 5-way 1-shot task. It maybe relative to the image sizes of the datasets, which are 600×600 pixels in the AID dataset, while we uniformly resized all the images to 256×256 pixels as inputs in the experiments of our method. It caused more feature loss to the images in the AID dataset while resizing. Thus the feature extraction ability of the proposed multiscale feature extraction network is negatively affected. At the same time, there is only one labeled sample per class within the 5-way 1-shot task. The inaccuracy of the extracted features affects the representation for the scene images and leads to the lower accuracy.

In addition, Table 3 records the algorithmic efficiency results of our method and the ones [35]. As is shown, parameter count (Params), floating-point operations (FLOPs), and inference time (Time) are compared. Although the efficiency of our method is not the best, the difference between it and the FRN or the MCL-Katz is not very obvious. It is superior to other methods for the FSRSSC according to the classification accuracy and efficiency.