Multi-Granularity Text and Scale-Aware Visual Encoding for Remote Sensing Image-Text Retrieval and Prompt-Based Scene Recognition

1. Introduction

Text-based access to Earth observation archives depends on connecting image content with captions, geographic annotations, and domain knowledge[1]. This connection supports semantic search and scene-level interpretation across heterogeneous remote sensing products[2], but it is difficult to learn when sensors, spatial resolutions, and description styles vary.

Vision-language models provide a shared representation for image retrieval, scene interpretation, and geospatial knowledge discovery. CLIP[3] is an attractive starting point because its contrastive objective supports transfer through natural-language supervision. Its pre-training data and architecture, however, were developed for natural images and generic captions rather than nadir-view Earth observation scenes[4].

Three transfer failures are especially relevant here. First, remote sensing captions place object names, attributes, spatial relations, and scene context in the same sentence, so an end-of-text embedding can obscure local terminology. Second, a scene may contain both a large land-cover structure and small but diagnostic objects, which are easily diluted by global visual pooling. Third, unmatched pairs are not equally unrelated: airports and industrial areas, for example, can share paved surfaces and geometric patterns. The study addresses these semantic-granularity, visual-scale, and negative-pair ambiguities.

Work on scale-adaptive patching and boundary-preserving segmentation shows why local spatial evidence matters in high-resolution remote sensing[5,6]. We use this observation without turning retrieval into a segmentation task. SAM supplies candidate crops for local-view augmentation, whereas the downstream model remains a dual encoder. The complete framework combines multi-granularity text encoding, hierarchical scale-aware visual features, SAM-guided training views, and a membership-aware contrastive objective. Experiments are limited to daytime optical imagery from RSICD and a restricted scene adaptation dataset; no large-scale remote sensing re-pretraining is used.

The study makes four technical contributions:

(1) We introduce a multi-granularity text encoder retains word-level, phrase-level, and sentence-level information and combines the three branches by cross-granularity attention.

(2) We introduce a scale-aware visual encoder links four receptive-field configurations to hierarchical feature maps, refines each level independently, and combines level descriptors through self-attention and gated concatenation.

(3) SAM-guided local-view augmentation adds object-focused evidence during training and validation without detector labels; inference uses only the dual encoder.

(4) A semantic membership-aware objective modifies InfoNCE[7] with detached weights and a bounded discount, reducing the penalty for semantically close negatives while keeping every negative contribution positive.

2. Related Work

2.1. Vision-Language Pre-Training and CLIP-Based Multimodal Learning

Vision-language pre-training differs mainly in where cross-modal interaction occurs. ViLBERT[8] maintains separate visual and linguistic streams linked by co-attention, whereas UNITER[9] uses joint multimodal encoding with region-level visual inputs. Both established explicit region-token interaction, but their detector-dependent pipelines are less direct than a retrieval-oriented dual encoder.

Later models reduced the cost of visual processing or changed the order of alignment and fusion. ViLT[10] projects image patches directly into a Transformer; ALBEF[11] aligns unimodal features before multimodal fusion; BLIP[12] combines understanding and generation objectives; and FLIP[13] masks image patches during pre-training. These designs improve general vision-language learning, yet they do not explicitly model the semantic hierarchy and scale distribution of remote sensing scenes.

CLIP learns separate image and text embeddings and compares them in a shared space. This structure is efficient for retrieval, but its text output is a single sentence vector and its visual encoder is tuned to natural-image statistics. Direct transfer can therefore miss local domain terms and small scene elements that determine remote sensing categories.

Remote sensing adaptations address this domain gap through parameter-efficient transfer[14], visual priors[15], or domain-specific pre-training[16]. RemoteCLIP[17] and GeoRSCLIP[18] show the value of large remote sensing image-text corpora. The present study asks a narrower question: whether explicit semantic granularity, visual scale, and graded negative-pair weighting can improve a CLIP-style model when large-scale domain re-pretraining is unavailable.

2.2. Fine-Grained Semantic Modeling for Multimodal Alignment

A remote sensing caption may specify a target, its attributes, its relation to nearby objects, and the wider geographic setting. Compressing these elements directly into one global embedding can weaken the distinctions between scenes that share most of their content. Fine-grained alignment therefore requires local lexical and relational features to remain available when the sentence representation is formed.

Hierarchical text encoding and multi-level alignment provide relevant precedents. Text-focused alignment has been studied in vision-language models[19], while multi-level methods align local and global representations at different stages[20,21]. These studies motivate structured text features, but they do not determine how remote sensing terminology should be combined with phrase and sentence context.

Our design uses WordPiece tokenization to retain subword cues in abbreviations and compound identifiers, then processes the resulting sequence through word-level, phrase-level, and sentence-level branches. It does not add a character encoder. Cross-granularity attention combines the branches before projection to the shared embedding space.

2.3. Multi-Scale Visual Representation and Semantic-Aware Contrastive Learning

Spatial scale changes the evidence available to a retrieval model. Vehicles and individual buildings occupy small regions, whereas airports, ports, and urban blocks require broader context. Feature pyramids address related variation in object detection, but image-text retrieval must convert the resulting local and global evidence into one embedding rather than predict boxes or masks.

Feature-fusion architectures[22,23] and domain-adaptation methods[24,25] provide mechanisms for combining levels or transferring visual representations. Remote sensing vision-language models also use prior knowledge, calibration, and specialized pre-training. What remains unresolved for the present setting is how to preserve level-specific information while producing a compact dual-encoder representation.

The contrastive objective introduces a second problem. InfoNCE[7] treats all unmatched pairs as negatives, although their gradients already vary with similarity. In remote sensing, semantically related categories can share spectral responses, textures, or layouts. Uniformly enforcing category separation can therefore conflict with partial scene overlap.

Soft objectives and adaptive weighting have been investigated as alternatives[26,27]. The method proposed here uses a more restricted intervention: a similarity-derived membership weight can only reduce a negative term within a fixed bound, and the weight is detached so it cannot directly attract an unmatched pair.

Table 1 summarizes the resulting distinction. The framework changes both encoders rather than adding an adapter, uses SAM only to generate unlabeled local training views, preserves four visual levels through self-attention and gated concatenation, and applies bounded membership weights to negative terms. Because evaluation uses two modest optical datasets and no large-scale domain re-pretraining, the model is an application-specific adaptation rather than a remote sensing foundation model.

3. Methods

3.1. Task Formulation and Framework Overview

Global text aggregation, limited explicit scale modeling, and the absence of se-mantic weighting among negatives can constrain CLIP-style alignment on remote sensing imagery. We address these three gaps with three interacting modules to map a remote sensing image and its caption to a 512-dimensional shared space for bidirectional retrieval and prompt-based scene recognition. Figure 1 shows the three interacting module: a multi-granularity text encoder, a scale-aware visual encoder, and a semantic membership-aware contrastive objective. SAM-guided crops enter the visual branch during training and validation but are not required at inference.

The text encoder keeps lexical, phrase, and sentence information separate until attention fusion. The visual encoder associates four spatial resolutions with hierarchical ResNet features and combines the resulting level descriptors. The loss then uses semantic membership to distinguish semantically close negatives from unrelated negatives while retaining the bidirectional contrastive formulation.

Given paired samples $D={\left\{\left(I_i,T_i\right)\right\}}_{\left(i=1\right)}^N$, the image and text encoders produce representations $F_{\mathrm{i}\mathrm{m}\mathrm{g}}$ and $F_{\mathrm{m}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}}$. Both are projected to 512 dimensions, matching the CLIP embedding width. Training increases the similarity of matched pairs and preserves separation from unmatched pairs, with the negative penalty adjusted as described in Section 3.5.

The complete feature extraction and alignment process can be formulated as:

$$F_{\mathrm{i}\mathrm{m}\mathrm{g}}=E_{\mathrm{i}\mathrm{m}\mathrm{g}}\left(I_i\right)$$

(1)

$$F_{\mathrm{m}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}}=E_{\mathrm{t}\mathrm{e}\mathrm{x}\mathrm{t}}\left(T_i\right)$$

(2)

Here, $E_{\mathrm{i}\mathrm{m}\mathrm{g}}\left(\cdot \right)$ and $E_{\mathrm{t}\mathrm{e}\mathrm{x}\mathrm{t}}\left(\cdot \right)$ denote the scale-aware visual encoder and multi-granularity text encoder; $F_{\mathrm{i}\mathrm{m}\mathrm{g}}$ and $F_{\mathrm{m}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}}$ are their outputs.

3.2. Multi-granularity Semantic Modeling

CLIP forms its text embedding from the end-of-text token after Transformer encoding. Although token interactions are contextualized, the output does not expose separate lexical, phrasal, and sentence representations. We retain these three granularities explicitly. WordPiece tokenization supplies subword cues for abbreviations and compound identifiers, so no character-level branch is introduced.

Figure 2 groups the text pathway into semantic hierarchy decomposition, parallel branch encoding, and cross-granularity attention fusion.

3.2.1. Semantic Hierarchy Decomposition

Remote sensing captions often encode several semantic roles at once. In “a long bridge crossing a river near dense residential buildings,” the nouns identify objects, the modifiers describe geometry and density, and the prepositions specify spatial relations. The encoder therefore retains local units before combining them with sentence context.

From the perspective of semantic hierarchy decomposition, remote sensing text can be represented as a hierarchical structure:

$$T=\left\{T_w,T_p,T_s\right\}$$

(3)

where $T_w$, $T_p$, and $T_s$ denote word-level, phrase-level, and sentence-level semantic components, respectively.

WordPiece subwords enter all three branches. The word branch emphasizes object and attribute tokens; the phrase branch captures short-range combinations and spatial relations; and the sentence branch models the full caption. These roles define the semantic hierarchy used below.

Before projection to the shared space, the model maintains local detail and global context in a structured representation:

$$P\left(T_w,T_p\mid T_s\right)$$

(4)

The conditional form indicates that local components are interpreted in the context of the complete sentence rather than used as independent descriptors.

3.2.2. Multi-Branch Encoder Architecture

The three branches process the same tokenized caption but use different receptive fields. This parallel design exposes information that would otherwise be available only implicitly inside a single sentence-level Transformer output.

The sentence branch uses a Transformer with positional and token-type embeddings. Sequences are limited to 77 tokens, and the hidden state is linearly projected to 512 dimensions to obtain $F_s$.

The word branch applies a convolution with kernel size $2\times 512$ and the stated channel configuration, followed by a Transformer layer with hidden dimension 256. Its output is $F_w$.

The phrase branch uses a convolution with kernel size $3\times 512$ and 128 channels, followed by Transformer refinement, to produce $F_p$.

[WORD], [PHRASE], and [SENTENCE] token-type embeddings identify the branch roles. Each output is normalized and projected to 512 dimensions before fusion.

3.2.3. Cross-Granularity Attention Fusion

The branches should not contribute equally to every caption. Concatenation would retain all components but would not condition their importance on sentence content. We instead use the sentence representation as a global query over the word and phrase features.

The sentence-level representation $F_s$ provides the query, and the word-level and phrase-level representations provide local semantic evidence. Their first attention operation is:

$$F_{\mathrm{m}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}}=\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\left(F_s,\left[F_w,F_p\right],\left[F_w,F_p\right]\right)$$

(5)

Here, $Attention\left(\cdot \right)$ denotes multi-head attention. The attention weights select local components in the context of the complete description.

A residual connection and feature normalization yield the final fused representation:

$$F_{\mathrm{m}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}}=\mathrm{L}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\left(\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\left(F_s,\mathrm{L}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r}\left(\left[F_w,F_p\right]\right),\mathrm{L}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r}\left(\left[F_w,F_p\right]\right)\right)+F_s\right)$$

(6)

Here, $\mathrm{L}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r}\left(\cdot \right)$ is the feature projection and $\mathrm{L}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\left(\cdot \right)$ is layer normalization. The fused text vector $F_{\mathrm{m}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}}$ remains 512-dimensional.

This fusion retains lexical and relational cues while keeping them conditioned on sentence meaning.

3.3. Scale-Aware Visual Encoding

Remote sensing scenes distribute evidence across markedly different spatial extents. A port requires broad layout information, whereas ships, vehicles, and individual buildings may occupy only a few local regions. The visual branch therefore uses hierarchical features rather than a single pooled map.

Global aggregation can suppress small but diagnostic targets, while fine features alone cannot represent the organization of large geographic structures. The proposed branch assigns fine and coarse receptive-field configurations to corresponding ResNet feature levels.

The branch produces one image embedding for alignment; it does not predict classes, boxes, or masks. It combines SAM-guided local-view augmentation with four-level feature extraction and hierarchical association.

Figure 3 shows the two inputs and the subsequent scale-aware feature pathway.

3.3.1. SAM-Guided Local-View Augmentation

A whole-scene CLIP representation may overlook a target that occupies a small image fraction. Local-view augmentation supplies an additional crop when SAM identifies a usable region. SAM is run offline before image encoding. It generates candidate regions without target labels or detector supervision. The crop is an auxiliary visual input, not a segmentation output. Region selection uses the implementation's visual-similarity and spatial-consistency criteria.

For image $I$, a selected SAM proposal is cropped to form $I_{local}$. If no proposal passes the reliability criterion, the original image is reused, which keeps the input pathway defined for every sample.

The global image and local view are encoded independently:

$$z_{global}=f_v\left(I\right)$$

(7)

$$z_{local}=f_v\left(I_{local}\right)$$

(8)

Here, ${\mathrm{f}}_v\left(\cdot \right)$ is the CLIP visual encoder, and $z_{global}$ and $z_{local}$ are the global and local representations.

$$z_v=Norm\left(\alpha z_{global}+\left(1-\alpha \right)z_{local}\right)$$

(9)

The coefficient $\alpha$ controls their relative contributions. This operation adds local evidence without introducing a detector at inference. The deployment-time network remains the dual encoder.

3.3.2. Scale-Aware Feature Representation

The scale-aware prior configuration allocates receptive fields to visual patterns of different extent. Although expressed with scale and aspect-ratio priors, it is used to organize representation learning rather than to produce detections.

The original three scales $128\times 128$, $256\times 256$, and $512\times 512$ are extended to four levels: $64\times 64$, $128\times 128$, $256\times 256$, and $512\times 512$. The added fine scale targets small structures, and the coarse scales retain wider scene context.

The scale set is:

$$S=\left\{64\times \mathrm{64,128}\times \mathrm{128,256}\times \mathrm{256,512}\times 512\right\}$$

(10)

Here, $S$ is the receptive-field scale set. The fine scales $64\times 64$ and $128\times 128$ correspond to the 56 × 56 and 28 × 28 feature maps (P2 and P3), which retain local boundaries and small objects. The coarse scales $256\times 256$ and $512\times 512$ correspond to the 14 × 14 and 7 × 7 maps (P4 and P5), which encode larger structures and scene context.

The aspect-ratio set is expanded to cover compact and elongated structures:

$$R=\left\{1:\mathrm{1,1}:\mathrm{2,2}:\mathrm{1,2}:2\right\}$$

(11)

Here, $R$ lists the candidate shapes used by the encoder.

Together, the four scales and expanded aspect ratios provide complementary local and contextual features.

3.3.3. Hierarchical Feature Association

Feature depth is matched to spatial scale. Shallow, high-resolution maps retain edges and textures, whereas deeper maps contain stronger semantic abstraction. Fine-scale configurations are associated with shallow feature levels, and coarse configurations with deeper levels. This pairing avoids forcing all spatial evidence through a single resolution.

For a ResNet-based extractor[28,29,30], the intermediate maps are:

$$F=\left\{F_2,F_3,F_4,F_5\right\}$$

(12)

where $F_2$, $F_3$, $F_4$, and $F_5$ denote the four network stages.

Their aggregation is formulated as:

$$F_{\mathrm{i}\mathrm{m}\mathrm{g}}=\mathsf{\Phi }\left(F_2,F_3,F_4,F_5\right)$$

(13)

where $\mathsf{\Phi }\left(\cdot \right)$ denotes hierarchical fusion of the level-specific features.

Channel attention then recalibrates the aggregated responses:

$$F\text{'}=\sigma \left(W_2\delta \left(W_{1}F\right)\right)\odot F$$

(14)

where $W_1$ and $W_2$ are learnable transformations, $\delta \left(\cdot \right)$ is the nonlinear activation, $\sigma \left(\cdot \right)$ is the sigmoid function, and $\odot$ denotes element-wise multiplication.

The visual vector is finally projected to the 512-dimensional text space:

$$F_{\mathrm{i}\mathrm{m}\mathrm{g}}=\mathrm{L}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r}\left(F\text{'}\right)$$

(15)

where $\mathrm{L}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r}\left(\cdot \right)$ is the projection layer.

The aggregation differs from a standard FPN in three implementation choices. Each pyramid level has an independent residual head with two 3 × 3 convolutions, normalization, and nonlinear activation. The four globally pooled descriptors are then treated as a sequence and processed by multi-head self-attention. Finally, softmax-normalized gates modulate the concatenated descriptors rather than summing them, preserving level-specific information in the final image embedding.

3.4. Cross-modal Feature Alignment

3.4.1. Feature Normalization in the Shared Space

Image and text features are linearly projected and normalized before comparison. This removes feature-magnitude differences between modalities and makes cosine similarity depend on embedding direction.

The projected visual feature $F_{{\mathrm{i}\mathrm{m}\mathrm{g}}_i}$ and text feature $F_{{\mathrm{m}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}}_j}$ are normalized as follows:

$${\tilde{I}}_i={\displaystyle \frac{F_{{\mathrm{i}\mathrm{m}\mathrm{g}}_i}}{‖F_{{\mathrm{i}\mathrm{m}\mathrm{g}}_i}‖}}$$

(16)

$${\tilde{T}}_j={\displaystyle \frac{F_{{\mathrm{m}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}}_j}}{‖F_{{\mathrm{m}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}}_j}‖}}$$

(17)

where ${\tilde{I}}_i$ and ${\tilde{T}}_j$ are the normalized image and text embeddings.

Both embeddings have 512 dimensions, so they can be compared directly.

3.4.2. Cross-Modal Similarity

Cosine similarity measures the correspondence between an image and a caption:

$$\mathrm{S}\mathrm{i}\mathrm{m}\left(I_i,T_j\right)={\displaystyle \frac{F_{{\mathrm{i}\mathrm{m}\mathrm{g}}_i}\cdot F_{{\mathrm{m}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}}_j}}{‖F_{{\mathrm{i}\mathrm{m}\mathrm{g}}_i}‖\hspace{0.33em}‖F_{{\mathrm{m}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}}_j}‖}}$$

(18)

where $\mathrm{S}\mathrm{i}\mathrm{m}\left(I_i,T_j\right)$ is the similarity between image $I_i$ and text $T_j$.

Pairwise similarities for a batch form the matrix used by the contrastive objective.

A temperature coefficient $\tau$ scales the logits:

$$P\left(I_i,T_j\right)={\displaystyle \frac{\exp \left(\mathrm{S}\mathrm{i}\mathrm{m}\left(I_i,T_j\right)/\tau )\right)}{{\displaystyle \sum _{k=1}^N}\exp \left(\mathrm{S}\mathrm{i}\mathrm{m}\left(I_i,T_k\right)/\tau )\right)}}$$

(19)

Smaller $\tau$ produces a sharper distribution; larger values smooth it.

3.4.3. Cross-Modal Feature Interaction

Before contrastive optimization, the model combines the multi-granularity text vector and scale-aware image vector to emphasize dimensions supported by both modalities.

This interaction is learned jointly with the encoders and does not require an additional hand-crafted correction at inference.

The fused representation is:

$$F_{\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}}=\alpha F_{\mathrm{m}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}}+\left(1-\alpha \right)F_{\mathrm{i}\mathrm{m}\mathrm{g}}+\beta \hspace{0.17em}\mathrm{S}\mathrm{i}\mathrm{m}\left(I,T\right)\hspace{0.17em}\left(F_{\mathrm{m}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}}\odot F_{\mathrm{i}\mathrm{m}\mathrm{g}}\right)$$

(20)

where $\alpha$ and $\beta$ are balancing coefficients and $\odot$ denotes element-wise multiplication. The interaction term increases the contribution of dimensions with aligned image and text responses.

3.5. Semantic Membership-Aware Contrastive Learning

Unmatched remote sensing pairs can still share scene content. Airports and industrial areas contain similar paved regions, and bridges and ships can have elongated shapes. The objective therefore discounts, but does not remove, the penalty assigned to high-membership negatives.

3.5.1. Semantic Membership Estimation

For image $I_i$ and text $T_j$, the membership weight $w_{ij}$ lies in [0, 1]; larger values indicate greater semantic proximity.

A row-adaptive sigmoid converts similarity to membership:

$$w_{ij}= \sigma \left(k · \left(s_{ij}- {\mu }_i\right)\right), {\mu }_i={\displaystyle \frac{{\Sigma }_j{s}_{ij}}{B}}$$

(21)

Here, $\sigma (·)$ is the logistic function, k is a learnable slope initialized to 5.0, and ${\mu }_i$ is the row mean. Because $\sigma (·)$ maps to (0, 1), the weights remain bounded. Diagonal values are set to $w_{ii}=0$, so matched pairs are not discounted.

A second threshold δ gates low-confidence associations before the weights enter the loss:

$${\tilde{w}}_{ij}= w_{ij}· 1\left[ s_{ij}> \delta \right], i \ne j; {\tilde{w}}_{ii}= 0$$

(22)

Here, $1\left[ \bullet \right]$ is the indicator function. The row mean μ_i centers similarity within each row but remains dependent on batch composition. The scalar δ instead gates the raw cosine similarity: pairs below δ keep the full negative penalty, even if they exceed a weak row mean. Diagonal entries are excluded from discounting.

3.5.2. Membership-aware Contrastive Objective

The membership weight changes only the negative terms in the partition function. Standard InfoNCE already produces similarity-dependent gradients; the additional factor reduces the contribution of negatives estimated to be semantically close.

The image-to-text partition is:

$${\mathcal{L}}_{i2t}= -\left({\displaystyle \frac{1}{B}}\right){\Sigma }_i\log \left[{\displaystyle \frac{\exp \left(\frac{s_{ii}}{\tau }\right)}{\exp \left(\frac{s_{ii}}{\tau }\right)+ {\Sigma }_{\left\{j\ne i\right\}\left(1 - \lambda · {\tilde{w}}_{ij}\right)}·\exp \left(\frac{s_{ij}}{\tau }\right)}}\right]$$

(23)

The text-to-image loss is defined analogously:

$${\mathcal{L}}_{t2i}= -\left({\displaystyle \frac{1}{B}}\right){\Sigma }_j\log \left[{\displaystyle \frac{\exp \left(\frac{s_{jj}}{\tau }\right)}{\exp \left(\frac{s_{jj}}{\tau }\right)+ {\Sigma }_{\left\{i\ne j\right\}\left(1 - \lambda · {\tilde{w}}_{ij}\right)}·\exp \left(\frac{s_{ij}}{\tau }\right)}}\right]$$

(24)

With λ ∈ [0, 0.95], the factor (1 − λ·${\tilde{w}}_{ij}$) remains in [0.05, 1]. Thus, the discount cannot invert or eliminate a negative penalty. Log-sum-exp is used for numerical stability.

The final loss averages the two retrieval directions:

$$L_{\mathrm{M}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}-w}={\displaystyle \frac{1}{2B}}\left({\displaystyle \sum _{i=1}^N}{L}_{I_i}+{\displaystyle \sum _{i=1}^N}{L}_{T_i}\right)$$

(25)

Matched pairs retain the full alignment term. High-membership negatives receive a smaller penalty, whereas low-membership negatives retain the standard contribution.

Because membership is computed from the similarity matrix being optimized, gradients through the weights could create a feedback path. We therefore calculate $w_{ij}$ with stop-gradient. The weight changes penalty magnitude but cannot directly encourage an unmatched pair to become more similar. Together with the positive lower bound and δ gate, an erroneous membership estimate can only attenuate a penalty.

This formulation performs bounded gradient reweighting rather than redefining unmatched pairs as positives.

4. Experiments and Results

4.1. Datasets and Evaluation Metrics

RSICD[31] is the primary benchmark and originally contains 10,921 images with 54,605 captions. We retain the official image-level split. Before training, captions duplicated across different images are removed; within-image near duplicates are removed at a Jaccard threshold of 0.9; captions without a geographic or spatial entity are filtered; and lightweight keyword annotations are appended online only to training captions.

After cleaning, RSICD contains 6,922 images and 12,964 captions. The train/validation/test splits contain 5,177/740/1,005 images and 8,692/1,539/2,733 captions, with 1.68/2.08/2.72 captions per image. Merging the three residential subclasses leaves 28 categories. This preprocessing changes the task relative to raw RSICD: it removes one source of ambiguous positives, but the score effect has not been isolated by a direct comparison. RSICD[31] is used for both prompt-based scene recognition and scene-constrained bidirectional retrieval.

The second dataset contains 4,622 optical images of man-made target scenes and one bilingual caption per image. Its four functional groups are port and shipyard facilities, surface and industrial vessels, ground vehicles, and ground installation sites.

Captions follow a common template covering viewpoint, target category and count, surface or berthing context, and surrounding land cover. Splits are assigned by physical target or site so that near-duplicate views do not cross partitions. Usage restrictions prevent public release, which limits independent reproduction despite the documented annotation and grouping rules.

Figure 4. Examples from RSICD and the remote sensing scene adaptation dataset.

Figure 4. Examples from RSICD and the remote sensing scene adaptation dataset.

4.1.1. Image-Text Retrieval Evaluation

Retrieval uses a scene-constrained RSICD protocol: queries and galleries belong to the 28 known scene categories. This closed taxonomy represents search within a predefined scene ontology, not open-world retrieval. Every baseline in Table 5 uses the same cleaned split, gallery, and ranking procedure.

Recall at K (R@K) records whether the matched item occurs among the top K results.

We report R@1, R@5, and R@10 for image-to-text (I2T) and text-to-image (T2I) retrieval. Mean recall (mR) is the average of all six values:

$$mR = \left({\displaystyle \frac{1}{6}}\right)· {\sum }_{\left\{K \in \left\{\mathrm{1,5},10\right\}\right\}}\left( R@K^{\left\{I2T\right\}}+ R@K^{\left\{T2I\right\}}\right)$$

(26)

Throughout the paper, mR refers only to this bidirectional retrieval average. The mAP reported for scene recognition is a separate metric.

At test time, dual-softmax reranking recalibrates the cosine-similarity matrix using its row and column distributions:

$$P_{I2T}=Softma{x}_{row}\left(S/\tau \right)$$

(27)

$$P_{T2I}=Softma{x}_{col}\left(S/\tau \right)$$

(28)

where $S$ is the original similarity matrix and τ is the temperature.

The two directions are combined as:

$$S_{fused}=\beta P_{I2T}+\left(1-\beta \right)P_{T2I}$$

(29)

where $\beta$ controls their relative contribution.

Because dual-softmax uses the complete test-gallery score distribution, this protocol is transductive and differs from direct cosine ranking[32,33]. Absolute values should therefore be compared only under the same fixed-gallery protocol.

4.1.2. Prompt-Based Scene Recognition Evaluation

For prompt-based scene recognition, mean Average Precision (mAP) is adopted to evaluate whether the learned image-text representation can correctly identify semantic scene categories.

The average precision of each category is calculated according to the precision ranking of predicted semantic scores[34,35], and mAP averages over categories:

$$mAP=\left({\sum }_{i=1}^{C}A{P}_i\right)/C$$

(30)

where $C$ is the number of scene categories.

This is an in-domain prompt evaluation because category labels affect prompt construction and training-time sampling. It is not cross-dataset zero-shot recognition. Table 2 shows that the five-template ensemble with normalized class names gives the highest validation setting, 87.5% mAP, and is used for the remaining recognition experiments.

4.2. Implementation and Comparison Protocol

Models are implemented in PyTorch (The version of PyTorch used was 1.8.0). The visual branch uses CLIP ResNet-50 because the scale-aware module requires the C2-C5 hierarchy, which a flat ViT-B/32 feature map does not provide. Training uses a batch size of 64 for 100 epochs. SAM crops are generated offline and excluded from training time; approximately 59.7 million trainable parameters belong to the deployable dual encoder. Dual-softmax reranking is applied only at test time and is not part of model training.

Recognition baselines are CLIP with ViT-B/32[3], ViLT[10], ALBEF[11], and BLIP[12]. Retrieval additionally compares SkyCLIP[36], CLIP-Adapter[37], RemoteCLIP[17], and fully fine-tuned CLIP[3]. Public pre-trained weights initialize all baselines, which are fine-tuned on the same data partitions and evaluated with the same task-specific protocols.

Optimization settings are kept fixed across the two datasets. This controls one source of variation but does not separate architecture, initialization, and data-composition effects.

4.3. Comparative Evaluation

The comparisons test prompt-based scene recognition on both datasets and scene-constrained bidirectional retrieval on RSICD. Results are interpreted within the shared training and evaluation settings defined above.

4.3.1. Comparison with State-of-the-Art Models

Table 3 reports prompt-based scene recognition on RSICD. The proposed model reaches 87.5% mAP and 78.3% Top-1 accuracy, compared with 74.8% and 65.2% for CLIP. It also exceeds ViLT, ALBEF, and BLIP under the same in-domain prompt protocol.

Relative to CLIP, the gains are 12.7 percentage points in mAP and 13.1 points in Top-1, with approximately 59.7 million rather than 151 million trainable parameters. Training takes 1.88 h versus 1.50 h, so parameter count does not translate directly into training time. The ablations in Section 4.4 separate the contributions of the three proposed components.

4.3.2. Results on the Scene Adaptation Dataset

To further evaluate the remote sensing semantic adaptation capability of the proposed framework, experiments are conducted on the remote sensing scene adaptation dataset. Table 4 reports in-domain prompt recognition on the restricted scene adaptation dataset. The proposed model obtains 88.2% mAP and 78.5% Top-1 accuracy, compared with 73.8% and 66.4% for CLIP. The mAP gain is larger than on RSICD, whereas the Top-1 gain is smaller; differences in category composition and supervision prevent attributing this contrast to one module.

The result is consistent with the intended use of domain-specific captions, which encode target relations, spatial arrangement, and functional attributes. Improvement on two separate in-domain evaluations shows that the architecture can use the available domain supervision. It does not establish cross-dataset generalization.

Category counts in the scene adaptation dataset range from 287 to 2,143 across four functional groups. Because aggregate retrieval recall would be dominated by this imbalance without category-wise analysis, retrieval is reported only on the more balanced RSICD scene-constrained protocol.

4.3.3. Scene-Constrained Image-Text Retrieval on RSICD

Table 5 reports I2T and T2I recall on the cleaned RSICD test set; mR averages the six recall values.

Table 5. Scene-constrained image–text retrieval performance on RSICD.

	RSICD Dataset
Method	I2T			T2I			mR
	R@1	R@5	R@10	R@1	R@5	R@10
CLIP[3,38]	6.77	15.37	23.15	5.01	15.75	24.21	15.04
ALBEF[11,39]	7.82	19.34	30.41	6.18	20.35	33.45	19.59
BLIP[12,39]	9.15	22.96	32.02	10.67	27.75	40.00	23.76
SkyCLIP[36,40]	6.59	16.10	26.53	7.14	22.34	34.29	18.83
CLIP-Adapter[37,40]	7.11	19.48	31.01	7.67	24.87	39.73	21.65
RemoteCLIP[17,39]	13.36	32.94	44.83	10.76	32.83	48.75	30.58
Full-FT CLIP[3,40]	13.54	30.83	43.46	11.55	33.14	49.83	30.39
Ours	17.36	35.84	50.85	15.64	34.51	51.99	34.37

Table 5. Scene-constrained image–text retrieval performance on RSICD.

	RSICD Dataset
Method	I2T			T2I			mR
	R@1	R@5	R@10	R@1	R@5	R@10
CLIP[3,38]	6.77	15.37	23.15	5.01	15.75	24.21	15.04
ALBEF[11,39]	7.82	19.34	30.41	6.18	20.35	33.45	19.59
BLIP[12,39]	9.15	22.96	32.02	10.67	27.75	40.00	23.76
SkyCLIP[36,40]	6.59	16.10	26.53	7.14	22.34	34.29	18.83
CLIP-Adapter[37,40]	7.11	19.48	31.01	7.67	24.87	39.73	21.65
RemoteCLIP[17,39]	13.36	32.94	44.83	10.76	32.83	48.75	30.58
Full-FT CLIP[3,40]	13.54	30.83	43.46	11.55	33.14	49.83	30.39
Ours	17.36	35.84	50.85	15.64	34.51	51.99	34.37

The proposed model reaches 34.37% mR, 3.79 percentage points above RemoteCLIP and 3.98 points above fully fine-tuned CLIP. Its trainable parameter count is approximately 59.7 million, but efficiency relative to RemoteCLIP cannot be resolved without the exact evaluated backbone and trainable parameter count for that baseline.

The largest differences from CLIP occur at R@1: 17.36% versus 6.77% for I2T and 15.64% versus 5.01% for T2I. The comparison shows better first-rank retrieval in this setting, but it does not separate encoder effects from the proposed loss or establish superiority over adapter tuning beyond this protocol.

4.4. Ablation Studies

To investigate the contribution of each proposed component and understand the underlying mechanism of performance improvement, ablation experiments are conducted on the RSICD test set. Three key modules are individually removed or replaced, including the multi-granularity semantic modeling module, the scale-aware visual representation module, and the semantic membership-aware contrastive learning objective. Table 6 removes one component at a time on the RSICD prompt-recognition task.

Scheme A replaces the multi-granularity text encoder with the original single-level CLIP text representation. Scheme B removes the scale-aware priors and hierarchical feature association. Scheme C replaces the membership-aware objective with standard InfoNCE. Scheme D is the complete model.

Every single-component removal reduces both mAP and Top-1 accuracy. Without multi-granularity text encoding (Scheme A), mAP falls from 87.5% to 80.7% and Top-1 from 78.3% to 73.1%. The 6.8-point mAP loss is consistent with a contribution from explicit lexical and phrase features. The original CLIP text encoder mainly relies on global sentence-level embeddings, which may weaken fine-grained semantic concepts when descriptions contain multiple types of information. After removing this module, the model no longer has dedicated branches for lexical and phrase-level features; the lower recognition scores are consistent with a benefit from this explicit decomposition.

Removing the scale-aware visual branch (Scheme B) lowers mAP to 81.2% and Top-1 to 74.5%, decreases of 6.3 and 3.8 percentage points. This result supports using hierarchical visual evidence when local targets and global scene structure coexist. This result verifies that scale adaptation is an important factor for remote sensing visual representation learning. Remote sensing scenes usually contain objects with large variations in spatial coverage, and dominant background structures may suppress small but semantically meaningful objects during feature aggregation.

Replacing the proposed loss with InfoNCE (Scheme C) lowers mAP to 79.7% and Top-1 to 73.7%, decreases of 7.8 and 4.6 points. Among the three removals, this produces the largest mAP decrease in the tested configuration. This demonstrates that semantic-aware negative sample optimization is particularly beneficial for fine-grained discrimination in prompt-based scene recognition. In remote sensing scenarios, many categories exhibit similar spectral responses and spatial configurations, generating large numbers of hard negative samples. Conventional InfoNCE does not explicitly discount these samples according to semantic membership, which may introduce excessive optimization constraints. The proposed membership-aware objective modifies the contribution of negative samples according to semantic proximity, reducing unnecessary separation between semantically related categories while maintaining discrimination against irrelevant samples.

Table 7 tests how membership weights are computed. The detached default reaches 87.5% mAP and 78.3% Top-1; allowing gradients through the weights gives 72.9% and 69.8%, while fixed scene-label weights give 85.7% and 77.6%. This comparison supports detaching the weights in the tested configuration. The lower scores without detachment are consistent with the feedback concern discussed in Section 3.5.

5. Discussion

SAM generated a valid local proposal for 98.5% of training images and 97.4% of validation images, but coverage fell to 86.6% for meadow, 87.5% for forest, and 90.6% for desert. Texture-dominated scenes provide fewer discrete boundaries, so these samples more often use the global-view fallback. Removing SAM while retaining the other scale-aware operations lowers mAP from 87.5% to 83.3% and Top-1 from 78.3% to 77.4% (Table 8). Removing the entire scale-aware branch produces 81.2% mAP and 74.5% Top-1. SAM therefore contributes to the branch, especially to ranked category performance, but the current results do not identify which confusions it resolves. Because SAM is used during training and validation but not inference, the model also contains a train-test view mismatch; distilling local-view information into the global encoder is a direct way to test whether that gap can be reduced.

The retrieval and recognition protocols expose a second limitation. Dual-softmax reranking uses the full test-gallery score distribution, so Table 5 is not directly comparable with open-set retrieval or direct cosine ranking. Internal comparisons remain fair because every baseline uses the same gallery and reranking. Prompt formulation has an equally large effect: mAP ranges from 56.9% for attribute-enhanced prompts with raw class names to 87.5% for the five-template ensemble with normalized names. Bare class names also outperform the generic template “a photo of [class].” A plausible explanation is that the randomly initialized text encoder is trained on domain-specific captions, but the experiment does not isolate this mechanism. Prompt selection should therefore be performed on validation data for the intended domain.

The evidence is further bounded by model and data choices. The text encoder is trained from random initialization, and the visual backbone is a CLIP-initialized ResNet-50. Comparisons with language-pretrained text encoders, larger backbones, or hierarchical transformers remain open. All images are daytime RGB observations; photometric augmentation does not establish transfer to SAR, hyperspectral, or real night-time imagery.

Our future work should target the unresolved boundaries rather than add modules without diagnosis. Cross-dataset and cross-sensor tests with held-out categories would measure out-of-distribution behavior. A lightweight trainable proposal head could test whether local views can be learned end to end with lower preprocessing cost, and a high-resolution pathway could quantify the accuracy-cost trade-off at the finest feature level.

6. Conclusions

This study adapted a CLIP-style dual encoder to remote sensing by combining multi-granularity text encoding, hierarchical scale-aware visual features with SAM-guided training views, and a detached semantic membership-aware contrastive loss. On cleaned RSICD, the model achieves 87.5% mAP for in-domain prompt-based scene recognition and 34.37% mR for scene-constrained retrieval with transductive dual-softmax reranking, using approximately 59.7 million trainable parameters. These values apply to the reported daytime optical datasets, prompt construction, cleaned category taxonomy, and fixed-gallery protocol. The present experiments do not establish open-set, cross-dataset, or cross-sensor generalization, and SAM is unavailable at inference. Future evaluation should address these boundaries and test whether local-view knowledge can be transferred to the deployment-time encoder.