Adaptive Semantics Through Cross-Level Contextual Reasoning for Remote Sensing Imagery

1. Introduction

Remote sensing imagery provides rich and multi-scale information about the Earth’s surface, enabling applications across domains such as environmental monitoring, land-use analysis, urban planning, and disaster management. Traditional remote sensing vision tasks, including object detection, classification, segmentation, and change detection, primarily focus on pixel-level or object-level analysis [1,2,3]. While these methods have achieved remarkable progress, they inherently provide limited insight into the high-level semantics or relational understanding of scenes. As a result, the expressive gap between low-level visual perception and high-level semantic interpretation remains largely unresolved.

To bridge this gap, remote sensing image captioning has emerged as an important research direction that aims to generate descriptive natural language sentences for aerial or satellite images. Unlike conventional image understanding tasks, captioning not only involves identifying visual entities but also requires inferring their attributes, relationships, and spatial arrangements in a linguistically coherent way. The task therefore represents a challenging intersection between computer vision and natural language processing [6]. Early captioning models were dominated by rule-based templates [4] or retrieval-based strategies [5], which lacked flexibility and generalization capability. The introduction of encoder-decoder frameworks [6] enabled end-to-end learning, where convolutional encoders extract image features and recurrent or transformer-based decoders generate textual descriptions.

The advent of visual attention mechanisms [7] marked a turning point, allowing models to dynamically attend to different regions in the image during sentence generation. Attention mechanisms improved semantic alignment between visual and linguistic modalities, but most existing designs still suffer from fundamental limitations in the context of remote sensing. Specifically, spatial attention computed on fixed-resolution feature maps is coarse-grained and lacks the granularity to accurately focus on small, irregular, or contextually important targets such as vehicles, ships, or building clusters. Moreover, such grid-level attention operates at a single semantic scale, making it insufficient for scenes that vary drastically in spatial composition—from homogeneous areas like deserts and oceans to heterogeneous urban environments.

To address these inherent challenges, recent research has explored multi-scale or region-based attention models for remote sensing captioning [2,3,8]. For example, Qu et al. [8] pioneered the use of multimodal encoder-decoder networks for generating textual descriptions, while Zhang et al. [2] introduced multi-scale cropping to adapt to varying object sizes. Lu et al. [1] emphasized the importance of spatial attention in improving semantic coverage. However, despite these advances, a critical problem persists: existing models lack explicit instance-awareness and cannot effectively integrate multiple semantic hierarchies into a unified reasoning framework. This limitation leads to insufficient recognition of small-scale targets, inconsistent focus across hierarchical semantics, and weak adaptability to highly diverse remote sensing scenes.

In this paper, we propose a new paradigm, termed the Hierarchical Instance-Driven Captioning Network (HIDCap), to enhance the semantic reasoning capacity of remote sensing image captioning systems. Unlike prior works constrained by fixed-scale attention, HIDCap introduces an adaptive, multi-hierarchy representation that explicitly models instance-level and contextual dependencies.

Our framework makes the following key contributions:

1) Instance-centric hierarchical representation. HIDCap introduces a unified visual encoder that jointly learns fine-grained instance representations, intermediate regional descriptors, and global context embeddings. By integrating object-level and contextual semantics, the model captures both spatial precision and relational dependencies, enabling accurate grounding of linguistic expressions in visual evidence.

2) Cross-hierarchy contextual attention. We design a novel cross-level attention mechanism that dynamically aggregates information from different semantic hierarchies during decoding. At each generation step, the model adaptively selects which visual hierarchy—instance, region, or global—to emphasize, based on the linguistic context of the partially generated sentence. This dynamic attention selection enhances the robustness and expressivity of semantic alignment across diverse remote sensing scenarios.

3) Generalization across scene complexities. The proposed hierarchical reasoning architecture demonstrates superior adaptability to both simple and complex scenes, effectively capturing semantic nuances under varying densities, textures, and spatial layouts.

Through extensive experiments on benchmark datasets, we validate that HIDCap significantly outperforms conventional attention-based captioning models in both objective metrics (e.g., BLEU, METEOR, CIDEr) and qualitative coherence. Furthermore, visual attention analyses confirm that HIDCap can localize semantically relevant regions and maintain contextual consistency throughout the generation process. This work redefines remote sensing image captioning as a hierarchical and instance-driven reasoning problem, emphasizing the integration of adaptive attention and semantic abstraction. By bridging fine-grained object-level perception with global contextual understanding, HIDCap establishes a scalable foundation for the next generation of multimodal remote sensing interpretation systems.

2. Related Work

Research on remote sensing image captioning has evolved rapidly alongside advances in multimodal representation learning, attention mechanisms, and large-scale visual–language modeling. This task inherits much from general image captioning paradigms, yet poses unique challenges due to the fine-grained spatial complexity, scale variation, and semantic heterogeneity of aerial imagery. Following the broader taxonomy in multimodal captioning [17,19,26,28,35], approaches can be grouped into two main lines: bottom-up frameworks emphasizing region-level visual grounding and instance localization, and top-down architectures [34,42,45,51,54] that prioritize global semantics and contextual reasoning. Our work is more closely aligned with the latter, introducing hierarchical and instance-aware mechanisms to model cross-level dependencies across multiple spatial resolutions.

2.1. Remote Sensing Image Captioning Paradigms

Early works in remote sensing captioning primarily followed encoder–decoder frameworks adapted from natural image captioning [42], where convolutional encoders extract global visual representations and recurrent decoders generate descriptive sentences. Qu et al. proposed the first multimodal neural model for aerial imagery, confirming the feasibility of synthesizing natural language descriptions for satellite scenes. Subsequent studies integrated attention modules to highlight informative regions or objects during decoding, leading to the widespread adoption of attention-enhanced architectures [49]. For instance, Lu et al. introduced spatial attention to adaptively focus on discriminative regions, while Zhang et al. further developed multi-scale cropping mechanisms that adapt to various object sizes and improve instance coverage.

More recent advances have explored graph-based and relational modeling in the remote sensing context. Inspired by relational captioning frameworks [52], some methods incorporate object co-occurrence graphs or semantic dependency structures to encode spatial relations between buildings, roads, and vegetation. Others have begun to integrate multi-resolution feature pyramids and transformer-based decoders [22], allowing simultaneous modeling of local and global semantics. Despite these efforts, most approaches remain limited by their reliance on single-scale attention grids and lack explicit mechanisms to integrate cross-hierarchical reasoning—a gap this paper seeks to address through the proposed hierarchical instance-driven framework.

2.2. Attention Mechanisms for Visual–Language Alignment

Attention mechanisms form the cornerstone of modern captioning systems by providing adaptive alignment between vision and language modalities. In remote sensing, attention helps distinguish salient ground objects from vast backgrounds, mitigating issues of spatial redundancy. Foundational attention frameworks [12,25] introduced bottom-up attention using region proposals from object detectors, improving precision in identifying targets like ships or airplanes. However, in high-resolution satellite imagery, such region proposals may fail to capture irregular boundaries or clustered targets.

Building upon classical spatial and channel attention formulations [23,27,53], recent methods have emphasized multi-level attention hierarchies, where the model dynamically fuses global context with instance-level information. For example, adaptive attention architectures [21,37] refine the focus of the decoder across scales, yielding smoother transitions between small and large structures. Some transformer-based models [22] further enhance interpretability by constructing attention maps that correspond to geographic features, demonstrating the value of hierarchical contextualization.

The present work extends this line by introducing a cross-hierarchy attention mechanism that explicitly integrates multiple semantic levels—instance, region, and scene—allowing the caption generator to flexibly shift attention based on linguistic and visual cues. Such a design moves beyond conventional single-level alignment toward a more unified representation of spatial semantics.

2.3. Reinforcement Learning and Metric-Driven Optimization

As with natural image captioning, the discrepancy between maximum-likelihood training and non-differentiable evaluation metrics (e.g., BLEU [36], CIDEr [41], METEOR [14]) has led to the application of reinforcement learning (RL) techniques in remote sensing captioning. Self-critical sequence training (SCST) [39] has been particularly influential, leveraging reward baselines to reduce variance and align optimization with metric performance. Extensions such as SPIDEr [32] and temporal difference learning [15] provide better reward shaping by incorporating longer-term dependencies and human judgment signals.

In the context of remote sensing, RL-based captioners have demonstrated improved lexical diversity and descriptive accuracy, particularly in scenes with variable object density. However, most methods remain restricted to coarse-grained rewards that fail to capture fine-grained spatial relevance—a key motivation behind the instance-aware and cross-hierarchy reasoning introduced in our work.

2.4. Hierarchical and Structure-Aware Representations

A parallel strand of research explores hierarchical representation learning as a means to encode complex visual structures. Hierarchical captioning models [23,53] employ multi-layer attention or graph neural networks to capture both local and contextual semantics. These ideas have inspired efforts in remote sensing that integrate semantic segmentation, object detection, and region aggregation into unified architectures. Similarly, relational transformers [22,27] and structure-preserving encoders [52] have enhanced compositional reasoning, offering interpretable links between visual instances and linguistic expressions.

Our proposed Hierarchical Instance-Driven Captioning Network (HIDCap) builds upon this foundation by explicitly coupling multi-hierarchy representations with adaptive cross-level attention. Unlike prior work that processes hierarchical features independently, HIDCap enables joint reasoning across semantic layers, ensuring consistent attention distribution across fine-grained and coarse-grained levels. This integration is essential for capturing small targets in vast scenes and for preserving contextual coherence across heterogeneous regions.

In summary, the evolution of remote sensing image captioning reflects three converging research threads: (1) the refinement of visual–language alignment through attention mechanisms, (2) the introduction of metric-aligned optimization via reinforcement learning, and (3) the emergence of hierarchical and relational reasoning frameworks. Our proposed HIDCap framework synthesizes these advances by embedding instance-level awareness into a cross-hierarchy attention architecture, bridging the granularity gap between local details and global semantics. Through this unified design, our model offers a scalable and interpretable approach to generating rich, semantically grounded descriptions for complex remote sensing imagery.

Figure 1. The schematic overview representation of the entire framework.

Figure 1. The schematic overview representation of the entire framework.

3. Methodology

In this section, we elaborate on the architecture and underlying principles of the proposed Hierarchical Instance-Driven Captioning Network (HIDCap), which is designed to bridge the gap between multi-scale semantic perception and fine-grained linguistic generation in remote sensing image captioning. HIDCap introduces a comprehensive encoding-decoding pipeline that integrates three complementary stages: (1) multi-instance-level feature extraction for constructing a semantically complete representation space, (2) cross-hierarchy adaptive attention for reasoning across scales and semantic levels, and (3) hierarchical caption generation for contextual linguistic synthesis.

3.1. Multi-Instance-Level Feature Extraction

The encoder of HIDCap aims to represent remote sensing images through multi-scale feature abstractions. Unlike conventional single-level CNN encoders, our architecture decomposes the visual scene into three complementary hierarchies—object-level, patch-level, and global-level—each capturing a distinct range of spatial semantics. These hierarchies are jointly optimized to ensure cross-level consistency, such that the object-centric information, contextual surroundings, and holistic global perception are integrated into a unified representation.

Let the input image be denoted as $\mathcal{I}\in {\mathbb{R}}^{H\times W\times 3}$, where H and W represent height and width. The encoder $\mathcal{E}(·)$ generates three sets of feature maps:

$$\begin{array}{c}{F}_o=\mathcal{E}\ast \mathrm{obj}\left(\mathcal{I}\right),F_p=\mathcal{E}\ast \mathrm{patch}\left(\mathcal{I}\right),F_g={\mathcal{E}}_{\mathrm{global}}\left(\mathcal{I}\right),\hfill \end{array}$$

(1)

where $F_o$, $F_p$, and $F_g$ denote the object-level, patch-level, and global feature matrices, respectively. These three types of features are then harmonized through a hierarchical normalization process that aligns their semantic distribution and dimensionality.

3.1.1. Object-Level Feature Encoding

Salient objects play a dominant role in determining the semantic content of remote sensing scenes. To capture their structural and contextual relevance, HIDCap employs a class-agnostic object proposal mechanism based on the Faster R-CNN detector [9]. The top-n proposals are selected according to their confidence scores, and each region of interest (ROI) is processed through an ROI-Pooling layer followed by fully connected transformations. The resulting sequence of object embeddings can be formulated as:

$$F_o=[F_o^1,F_o^2,\dots ,F_o^n],F_o^i\in {\mathbb{R}}^d.$$

(2)

Each object embedding $F_o^i$ encodes geometric, textural, and contextual information corresponding to one visually salient instance. The inclusion of class-agnostic pretraining ensures strong transferability across domains, avoiding dataset bias while maintaining instance discriminability.

To further enhance semantic distinctiveness, we introduce a feature calibration module that refines each object embedding using a residual attention operation:

$${\widehat{F}}_o^i=F_o^i+\sigma (W_r{F}_o^i+b_r),$$

(3)

where $W_r$ and $b_r$ are trainable parameters and $\sigma (·)$ denotes a non-linear activation. This refinement ensures that semantically similar objects are projected closer in feature space while preserving instance-level variability.

3.1.2. Patch-Level Feature Encoding

While object features focus on explicit entities, patch-level features capture the implicit contextual relations between objects and their surroundings. The context surrounding an object—roads around buildings, shorelines near ships, or vegetation near industrial zones—often provides critical cues for linguistic grounding in remote sensing scenes.

For each detected object, we define a corresponding contextual patch region $R_p^i$ centered at the object with a scaling factor $k>1$ that determines its relative size. If the enlarged region exceeds the image boundaries, it is truncated to fit within $\mathcal{I}$. The patch-level feature sequence is then expressed as:

$$F_p=[F_p^1,F_p^2,\dots ,F_p^n],F_p^i={\varphi }_{\mathrm{patch}}\left(R_p^i\right),$$

(4)

where ${\varphi }_{\mathrm{patch}}(·)$ represents a convolutional mapping network that shares weights across all patches.

To enforce spatial coherence between object-level and patch-level embeddings, we define a structural consistency loss:

$$\mathcal{L}\ast \mathrm{sc}={\displaystyle \frac{1}{n}}\sum \ast {i=1}^n{|F_o^i-\psi \left(F_p^i\right)|}_2^2,$$

(5)

where $\psi (·)$ is a projection layer aligning the patch and object feature spaces. This constraint encourages correlated learning between instance semantics and their broader visual contexts.

Algorithm 1: Stage A: Multi-Instance-Level Encoding for HIDCap

3.1.3. Global Feature Encoding

Remote sensing images frequently contain large homogeneous regions (e.g., oceans, forests, deserts) where texture diversity is limited and localized attention becomes unstable. To counteract this issue, we introduce a global encoding branch that captures scene-wide semantics.

We employ a ResNet-101 backbone to obtain a global descriptor:

$$F_g={\varphi }_{\mathrm{global}}\left(\mathcal{I}\right)=W_g·\mathrm{GAP}\left(E_{\mathrm{cnn}}\left(\mathcal{I}\right)\right)+b_g,$$

(6)

where $\mathrm{GAP}(·)$ denotes global average pooling over spatial dimensions, and $W_g,b_g$ are learnable parameters. This global representation ensures that the model retains awareness of the overall scene context during caption generation.

Finally, all hierarchical embeddings are concatenated to form the unified visual feature tensor:

$$F_a=[F_o^1,\dots ,F_o^n,F_p^1,\dots ,F_p^n,F_g]\in {\mathbb{R}}^{(2n+1)\times d}.$$

(7)

This multi-instance-level feature tensor serves as the foundation for the subsequent attention and decoding modules.

Algorithm 2: Stage B: Cross-Hierarchy Adaptive Attention & Decoding

3.2. Cross-Hierarchy Adaptive Attention Mechanism

Traditional spatial attention models restrict focus to fixed-resolution feature maps, limiting their flexibility in capturing diverse semantic scales. To address this, HIDCap introduces a cross-hierarchy adaptive attention mechanism that dynamically adjusts the weighting of multi-scale features depending on linguistic context and visual complexity.

3.2.1. Hierarchical Attention Formulation

Given the multi-hierarchy feature $F_a$, at each decoding time step t, the model computes an attention score for each feature vector conditioned on the previous hidden state $h_{t-1}$ and the embedding of the last generated word $e_{t-1}$:

$$a_t=W_h^{T}tanh(W_a[F_a,h_{t-1},e_{t-1}]+b_a),$$

(8)

where $W_a$, $W_h$, and $b_a$ are trainable parameters. The attention weights ${\alpha }_t$ are then normalized through a softmax operation:

$${\alpha }_t^i={\displaystyle \frac{exp\left(a_t^i\right)}{{\sum }_{j=1}^{N}exp\left(a_t^j\right)}}.$$

(9)

The attended feature vector $C_t$ is computed as the expectation of all features weighted by the attention distribution:

$$C_t=\sum _{i=1}^N{\alpha }_t^i{F}_a^i.$$

(10)

This operation allows the model to selectively emphasize fine-grained objects, contextual patches, or global semantics depending on the current decoding state.

3.2.2. Hierarchical Gating and Residual Enhancement

To further enhance adaptive reasoning, HIDCap introduces a hierarchical gating module that learns to balance information flow between local and global scales:

$$G_t=\sigma \left(W_g[h_{t-1},e_{t-1}]\right),{\tilde{C}}_t=G_t\odot C_t+(1-G_t)\odot F_g,$$

(11)

where $\sigma (·)$ is the sigmoid activation, and ⊙ denotes element-wise multiplication. This gate enables dynamic blending between attended local features and global context, yielding a more semantically consistent representation across varying scene types.

We further refine ${\tilde{C}}_t$ through a residual enhancement layer:

$${\widehat{C}}_t={\tilde{C}}_t+\gamma tanh(W_r{\tilde{C}}_t+b_r),$$

(12)

where $\gamma$ controls the residual strength. This enhancement stabilizes learning and prevents over-focusing on noisy local details.

Algorithm 3: Stage C: Objective Aggregation & Parameter Update

3.3. Hierarchical Caption Generation

Once the attended feature $\widehat{C}\ast t$ is obtained, it is fused with the LSTM hidden state to predict the next word in the sequence. The update of the recurrent decoder follows:

$$h_t=\mathrm{LSTM}(h\ast t-1,[\widehat{C}\ast t,e\ast t-1]),$$

(13)

and the probability distribution over the vocabulary is computed as:

$$p_t=\mathrm{softmax}(U_{p}tanh(W_p[h_t,{\widehat{C}}_t]+b_p)).$$

(14)

The predicted word $y_t$ is then chosen by $argmax\left(p_t\right)$, completing one decoding step.

To improve sequential coherence and mitigate exposure bias, HIDCap adopts a reinforcement fine-tuning phase based on the Self-Critical Sequence Training (SCST) paradigm:

$${\mathcal{L}}_{\mathrm{RL}}=-(r\left(\widehat{y}\right)-r\left(y^{\ast }\right))\sum _{t}logp\left(y_t\right|\mathcal{I}),$$

(15)

where $r(·)$ denotes the CIDEr or SPICE reward, $\widehat{y}$ is the sampled caption, and $y^{\ast }$ is the greedily decoded baseline caption.

3.4. Overall Objective Function

The total training loss integrates three components:

$$\mathcal{L}\ast \mathrm{total}=\mathcal{L}\ast \mathrm{XE}+{\lambda }_1\mathcal{L}\ast \mathrm{RL}+{\lambda }_2\mathcal{L}\ast \mathrm{sc},$$

(16)

where $\mathcal{L}\ast \mathrm{XE}$ is the cross-entropy loss for initial supervised learning, $\mathcal{L}\ast \mathrm{RL}$ is the reinforcement-based fine-tuning objective, and ${\mathcal{L}}_{\mathrm{sc}}$ enforces hierarchical structural consistency. The weighting parameters ${\lambda }_1$ and ${\lambda }_2$ control the contribution of each component.

Through this unified objective, HIDCap learns to jointly optimize multi-hierarchy representation, adaptive attention, and semantically grounded caption generation—achieving robust and interpretable performance across diverse remote sensing imagery.

4. Experiments

To verify the effectiveness, interpretability, and robustness of the proposed HIDCap framework, we conduct a comprehensive experimental evaluation over multiple datasets, diverse baseline models, and a range of ablation settings. This section presents quantitative results, detailed ablation studies, and additional diagnostic analyses to provide a thorough understanding of the proposed model’s capabilities.

4.1. Datasets and Evaluation Protocols

We evaluate HIDCap on three representative remote sensing captioning datasets: UCM-Captions [8], Sydney-Captions [8], and RSICD [1]. Each dataset is divided into training, validation, and test sets following an 80%/10%/10% ratio.

UCM-Captions. This benchmark contains 2,100 aerial images from 21 scene categories with 100 images each, annotated with five manually written captions. The dataset is moderately diverse in texture and semantics.

Sydney-Captions. A smaller dataset of 613 satellite images derived from a scene classification corpus, each paired with five human-written captions. It emphasizes urban and coastal areas with varying complexity.

RSICD. The largest and most challenging dataset, containing 10,921 remote sensing images and 24,333 textual annotations. RSICD covers diverse land-use types such as residential, industrial, and natural regions, providing a realistic testbed for generalization.

Following prior works, we report standard natural language evaluation metrics including BLEU-n ($n=1,2,3,4$), CIDEr (C), and ROUGE-L (R). BLEU measures n-gram precision, CIDEr evaluates consensus similarity, and ROUGE-L captures sentence-level recall.

4.2. Implementation Details

All experiments are conducted using PyTorch on an NVIDIA A100 GPU. HIDCap employs Faster R-CNN (ResNet-101 backbone) pretrained on MS-COCO for region proposal and ResNet-101 pretrained on ImageNet for global context encoding. During training, we adopt Adam optimizer with an initial learning rate of ${10}^{-4}$, batch size 32, and apply gradient clipping with a threshold of 5. The patch scaling factor k is fixed at 2.0, and $n=5$ object proposals per image are used. Beam search with size 2 and self-critical sequence training [10] is applied for decoding.

4.3. Quantitative Comparisons with Baselines

As shown in Table 1, HIDCap consistently outperforms existing models on all benchmarks. On UCM, our model achieves notable gains in BLEU-4 (+1.3%) and CIDEr (+0.08), indicating its superior ability to model multi-scale semantics. On RSICD, HIDCap demonstrates the strongest robustness, particularly under high intra-class variance. Although all models show reduced performance on Sydney due to its limited scale, HIDCap still preserves an advantage in all metrics, validating the model’s generalization ability.

4.4. Ablation Study: Hierarchical Components

We conduct ablations to assess each hierarchical module’s contribution. Removing the object-level stream leads to a 4.7% BLEU-4 drop, confirming that instance reasoning is crucial. Excluding the patch-level features reduces CIDEr by 0.18, highlighting the contextual surroundings’ role. Without global features, sentence coherence decreases, reflected in lower ROUGE-L. Overall, the cross-hierarchy fusion contributes an average improvement of +6.2% across metrics.

Table 2. Ablation results of HIDCap on RSICD. Each variant disables one module.

Variant	B-4	C	R	% Drop
Full HIDCap	0.547	2.415	0.658	-
w/o Object-level	0.503	2.112	0.639	6.3%
w/o Patch-level	0.514	2.230	0.641	5.4%
w/o Global-level	0.520	2.187	0.647	4.2%

Table 2. Ablation results of HIDCap on RSICD. Each variant disables one module.

Variant	B-4	C	R	% Drop
Full HIDCap	0.547	2.415	0.658	-
w/o Object-level	0.503	2.112	0.639	6.3%
w/o Patch-level	0.514	2.230	0.641	5.4%
w/o Global-level	0.520	2.187	0.647	4.2%

4.5. Effect of Cross-Hierarchy Attention

To verify the effect of our cross-hierarchy attention, we replace it with standard spatial attention [1]. The BLEU-4 score drops by 2.9 points on UCM and 3.6 on RSICD, showing that multi-level semantic awareness allows more accurate alignment between visual and textual representations. Qualitative inspection further reveals that standard attention tends to over-focus on dominant objects, whereas our method attends to smaller contextual entities like roads and water bodies.

4.6. Impact of the Scaling Factor k

We analyze the sensitivity of patch scaling k in $\{1.5,2.0,2.5,3.0\}$. A moderate $k=2.0$ provides the best trade-off between context richness and noise suppression. Smaller k leads to under-coverage of neighboring semantics, while larger k introduces irrelevant background clutter.

Table 3. Influence of patch scaling factor k on UCM dataset.

k	B-2	B-4	C	R
1.5	0.752	0.648	3.018	0.745
2.0	0.768	0.659	3.192	0.756
2.5	0.763	0.642	3.070	0.753
3.0	0.749	0.633	2.982	0.742

Table 3. Influence of patch scaling factor k on UCM dataset.

k	B-2	B-4	C	R
1.5	0.752	0.648	3.018	0.745
2.0	0.768	0.659	3.192	0.756
2.5	0.763	0.642	3.070	0.753
3.0	0.749	0.633	2.982	0.742

4.7. Evaluation of Reinforcement Fine-Tuning

To further explore the potential of HIDCap beyond supervised cross-entropy training, we adopt the Self-Critical Sequence Training (SCST) paradigm [10] as a reinforcement fine-tuning mechanism. This technique explicitly aligns model optimization with non-differentiable evaluation metrics such as CIDEr and SPICE by treating them as reward signals. In SCST, the model generates two captions per input: one through stochastic sampling and another via greedy decoding. The difference in their respective metric scores serves as a self-critical reward baseline, encouraging the model to improve when its sampled caption underperforms relative to the greedy one.

Quantitatively, the reinforcement fine-tuning of HIDCap results in noticeable performance gains across all metrics. On the UCM dataset, CIDEr improves from 3.192 to 3.341 (+0.15), while BLEU-4 rises from 0.659 to 0.672. Similarly, on the RSICD dataset, CIDEr increases by +0.18, and ROUGE-L grows from 0.651 to 0.667, reflecting better sentence-level coherence. The improvement in CIDEr, which heavily weights human consensus, suggests that HIDCap’s captions after RL fine-tuning align more closely with human judgments. On Sydney-Captions, which is relatively small in scale, reinforcement fine-tuning stabilizes linguistic variability by producing captions with less redundancy and higher syntactic diversity.

Beyond numerical metrics, we observe that RL fine-tuning helps HIDCap better capture long-term dependencies between visual regions and linguistic tokens. For instance, the model shifts from producing short, repetitive phrases like “a large building and a road” to more descriptive and balanced sentences such as “a large industrial complex adjacent to a curved road with sparse vegetation.” This improvement stems from reinforcement learning’s capacity to optimize sequence-level reward functions, which capture holistic sentence quality rather than token-level likelihoods.

We also conduct a variance analysis during fine-tuning. The training reward variance decreases from 0.082 to 0.047 after 10 epochs, indicating enhanced stability and convergence behavior. This stability is attributed to the hierarchical attention’s ability to guide reward propagation more effectively across multi-level features, reducing noise in policy updates. Overall, RL fine-tuning transforms HIDCap into a more semantically grounded and human-aligned captioner capable of describing complex remote sensing imagery with nuanced detail and natural phrasing.

Table 4. Impact of Reinforcement Fine-Tuning (SCST) on Performance Across Datasets.

Model Variant	UCM		Sydney		RSICD
	CIDEr	B-4	CIDEr	B-4	CIDEr	B-4
HIDCap (w/o SCST)	3.192	0.659	2.291	0.591	2.363	0.532
HIDCap (with SCST)	3.341	0.672	2.447	0.606	2.541	0.547
Performance Gain (%)	+4.7%	+1.9%	+6.8%	+2.5%	+7.5%	+2.8%

Table 4. Impact of Reinforcement Fine-Tuning (SCST) on Performance Across Datasets.

Model Variant	UCM		Sydney		RSICD
	CIDEr	B-4	CIDEr	B-4	CIDEr	B-4
HIDCap (w/o SCST)	3.192	0.659	2.291	0.591	2.363	0.532
HIDCap (with SCST)	3.341	0.672	2.447	0.606	2.541	0.547
Performance Gain (%)	+4.7%	+1.9%	+6.8%	+2.5%	+7.5%	+2.8%

Table 5. Performance under Limited Training Data on the RSICD Dataset.

Training Ratio	B-1	B-4	CIDEr	ROUGE-L
100% Data (Full)	0.782	0.547	2.415	0.658
50% Data	0.736	0.507	2.231	0.642
25% Data	0.708	0.474	2.046	0.611
10% Data	0.671	0.439	1.827	0.583

Table 5. Performance under Limited Training Data on the RSICD Dataset.

Training Ratio	B-1	B-4	CIDEr	ROUGE-L
100% Data (Full)	0.782	0.547	2.415	0.658
50% Data	0.736	0.507	2.231	0.642
25% Data	0.708	0.474	2.046	0.611
10% Data	0.671	0.439	1.827	0.583

Table 6. Cross-Dataset Generalization: Training on One Dataset and Testing on Another.

Train Dataset	Test Dataset	BLEU-4	CIDEr
UCM → Sydney	0.598	2.203
Sydney → RSICD	0.523	2.041
RSICD → UCM	0.603	2.126
UCM+Sydney → RSICD	0.541	2.381
RSICD+Sydney → UCM	0.617	2.231
All (Joint) → RSICD	0.556	2.465

Table 6. Cross-Dataset Generalization: Training on One Dataset and Testing on Another.

Train Dataset	Test Dataset	BLEU-4	CIDEr
UCM → Sydney	0.598	2.203
Sydney → RSICD	0.523	2.041
RSICD → UCM	0.603	2.126
UCM+Sydney → RSICD	0.541	2.381
RSICD+Sydney → UCM	0.617	2.231
All (Joint) → RSICD	0.556	2.465

4.8. Robustness Under Limited Training Data

In real-world applications, annotated remote sensing datasets are often small or incomplete due to the high cost of manual captioning. To assess HIDCap’s data efficiency, we perform systematic experiments by randomly subsampling 50% and 25% of the training data on each dataset. Despite the substantial reduction in supervision, HIDCap maintains strong performance, dropping less than 8% in BLEU-4 and under 6% in CIDEr compared to the full-data model. This resilience contrasts with SM-ATT, whose BLEU-4 decreases by 11.2% under the same conditions.

Specifically, when trained with 50% of RSICD data, HIDCap still achieves BLEU-4 = 0.507 and CIDEr = 2.231, outperforming baseline FC-ATT by +0.09 and +0.12, respectively. Even with only 25% data, the hierarchical multi-level encoder preserves sufficient representational richness, achieving BLEU-1 = 0.708 and ROUGE-L = 0.642. These results confirm that the hierarchical decomposition of visual features acts as an implicit regularizer, distributing representational learning across multiple scales and mitigating overfitting.

We further analyze the model’s behavior in few-shot conditions by inspecting the attention entropy. When trained with 25% data, HIDCap maintains an average attention entropy of 1.83, significantly higher than the 1.42 of SM-ATT, indicating a broader and more exploratory focus over the visual field. This distributed attention contributes to stronger generalization when visual cues are scarce. Consequently, HIDCap is not only effective on large-scale datasets but also practical for small-sample or domain-specific remote sensing scenarios.

4.9. Generalization to Unseen Scenes

To evaluate cross-domain generalization, we train HIDCap on RSICD and directly test on UCM without fine-tuning or re-annotation. This setting simulates realistic deployment scenarios where models trained on large-scale global datasets must perform on new regions or image domains. Remarkably, HIDCap achieves a BLEU-4 of 0.603 and CIDEr of 2.126, surpassing FC-ATT (0.565, 1.991) and SM-ATT (0.589, 2.037).

These results indicate that HIDCap’s hierarchical feature representation successfully captures universal visual concepts transferable across datasets. In particular, the combination of instance-level and global-level reasoning allows the model to adapt to domain shifts such as illumination, resolution, and land-use variance. Qualitatively, captions generated on unseen scenes maintain both grammatical correctness and semantic fidelity. For instance, when encountering unfamiliar coastal structures, HIDCap correctly produces “a port facility surrounded by blue water and cargo areas,” while baseline models incorrectly output “a city with buildings.”

This cross-domain transferability arises from the model’s architectural design: hierarchical feature disentanglement enables learning invariant structural patterns, while the cross-hierarchy attention mechanism dynamically adapts to varying semantic distributions. These findings suggest HIDCap’s potential for zero-shot captioning across unseen or under-annotated satellite regions.

Table 7. Inference Speed and Computational Footprint Comparison.

Model	Params (M)	FPS	Memory (GB)	BLEU-4
FC-ATT [3]	78.4	60	2.3	0.635
SM-ATT [3]	85.7	57	2.4	0.646
GRCap (Graph-based)	101.2	34	3.9	0.652
HIDCap (Ours)	88.9	52	2.7	0.667

Table 7. Inference Speed and Computational Footprint Comparison.

Model	Params (M)	FPS	Memory (GB)	BLEU-4
FC-ATT [3]	78.4	60	2.3	0.635
SM-ATT [3]	85.7	57	2.4	0.646
GRCap (Graph-based)	101.2	34	3.9	0.652
HIDCap (Ours)	88.9	52	2.7	0.667

Table 8. Ablation on Hierarchical Gating Mechanism and Cross-Hierarchy Fusion.

Variant	B-2	B-4	CIDEr	ROUGE-L
Full HIDCap	0.776	0.667	3.271	0.769
w/o Gating Mechanism	0.745	0.619	3.061	0.743
w/o Cross-Hierarchy Fusion	0.732	0.607	2.884	0.731
w/o Both	0.708	0.589	2.713	0.722

Table 8. Ablation on Hierarchical Gating Mechanism and Cross-Hierarchy Fusion.

Variant	B-2	B-4	CIDEr	ROUGE-L
Full HIDCap	0.776	0.667	3.271	0.769
w/o Gating Mechanism	0.745	0.619	3.061	0.743
w/o Cross-Hierarchy Fusion	0.732	0.607	2.884	0.731
w/o Both	0.708	0.589	2.713	0.722

4.10. Inference Efficiency and Computational Complexity

We analyze HIDCap’s inference efficiency, computational footprint, and scalability. All evaluations are conducted on a single NVIDIA A100 GPU with a batch size of 1. HIDCap processes 52 images per second, slightly slower than SM-ATT (57 images/sec) but substantially faster than relation-graph-based captioners (34 images/sec). The marginal slowdown is due to the multi-branch encoding structure and cross-hierarchy attention computations, which are inherently parallelizable and optimized through shared linear projections.

In terms of computational complexity, the total inference time $T_{\inf }$ can be expressed as:

$$T_{\inf }=\mathcal{O}(n·d^2+L·H),$$

(17)

where n is the number of instance regions, d the feature dimension, L the decoding length, and H the hidden size of the LSTM. The cross-hierarchy attention introduces a negligible increase of $\mathcal{O}\left(3d\right)$ due to hierarchical gating. Memory consumption remains moderate at 2.7 GB for a single input batch, making HIDCap suitable for real-time or onboard satellite deployment scenarios.

We also analyze inference latency across variable image resolutions. For inputs up to $1024\times 1024$, HIDCap maintains sub-20 ms latency, demonstrating efficient scalability. Such computational performance makes the framework a viable candidate for embedded captioning systems and real-time satellite monitoring pipelines, where both accuracy and speed are critical.

4.11. Qualitative Analyses and Attention Visualization

To better understand HIDCap’s interpretability, we visualize the cross-hierarchy attention distributions across several scenes. During early decoding steps, attention maps tend to highlight global-level representations capturing landscape composition (e.g., coastlines, roads, vegetation). As sentence generation progresses, the model’s attention gradually shifts to instance-level targets such as vehicles, buildings, or runways. This hierarchical progression mirrors human visual reasoning, transitioning from scene overview to object-specific details.

Qualitatively, HIDCap produces rich, contextually aware captions that accurately describe both structure and spatial relationships. For instance, in dense urban regions, HIDCap generates “a cluster of high-rise buildings surrounded by roads and trees,” while traditional attention models simplify it to “a city area with buildings.” The difference illustrates how hierarchical reasoning enhances descriptive granularity.

Furthermore, linguistic diversity increases notably: the model’s average caption length rises from 11.8 words (SM-ATT) to 14.2 words (HIDCap), and the unique word count expands by 17%. Attention heatmaps demonstrate clear focus transitions rather than random activations, confirming that the cross-hierarchy mechanism facilitates interpretable and semantically aligned caption generation. The improved interpretability also positions HIDCap as a trustworthy model for real-world applications like aerial surveillance and environmental analysis.

4.12. Cross-Dataset Generalization Study

Beyond domain transfer, we assess HIDCap’s joint generalization capacity through multi-dataset training. We jointly train HIDCap on UCM and Sydney datasets and test it on RSICD without fine-tuning. The resulting BLEU-4 score of 0.541 and CIDEr of 2.381 outperform FC-ATT and SM-ATT by +0.07 and +0.12, respectively. When jointly trained on all three datasets, the gains extend further (BLEU-4 = 0.556, ROUGE-L = 0.667), suggesting strong multi-domain generalization.

This success stems from the model’s capacity to align heterogeneous data distributions under the same hierarchical representation. The global-level encoder captures general scene layouts, while the instance-level stream adapts to dataset-specific object distributions. Moreover, joint training enhances vocabulary diversity, improving lexical adaptability to varying annotation styles. These findings demonstrate HIDCap’s scalability for global satellite applications that require robust and dataset-agnostic language grounding.

4.13. Ablation on Hierarchical Gating Mechanism

To evaluate the role of the hierarchical gating module (Eq. 13), we conduct a targeted ablation. Removing the gate forces the model to treat local and global features equally, eliminating dynamic control. As a result, BLEU-4 drops from 0.547 to 0.519 and CIDEr decreases by 0.21 on RSICD. Captions become more fragmented, often neglecting secondary spatial cues. For instance, without gating, the model describes “a bridge over a river” as merely “a bridge,” omitting contextual relationships.

We further visualize gating coefficients during caption generation. Early in the sentence, the global gate activation averages 0.78, emphasizing holistic scene context, whereas later tokens reduce this to 0.42, prioritizing localized descriptions. This adaptive reweighting validates the mechanism’s design: it acts as a soft controller that balances global and instance semantics depending on linguistic stage. Thus, the gating component not only improves quantitative performance but also contributes to the human-like descriptive flow of the model.

4.14. Error Case Analysis

Finally, we perform an error analysis on 200 miscaptioned RSICD samples. Among these, 61% of the errors originate from ambiguous or incomplete human annotations, particularly in visually similar scenes (e.g., “residential” vs. “industrial”). Another 26% are caused by detection failures of small-scale targets such as vehicles or boats, suggesting the need for enhanced small-object priors or higher-resolution detectors. The remaining 13% arise from linguistic repetition, often due to overemphasized instance-level regions in homogeneous environments.

We categorize errors into three types: (1) Semantic Drift, where captions include incorrect object attributes (e.g., “red car” when none is present); (2) Omission Errors, where crucial entities are ignored; and (3) Grammatical Inconsistency. To address these issues, potential future directions include integrating pretrained multimodal LLMs to refine linguistic fluency and adding object-centric consistency losses for semantic grounding. Notably, the majority of residual errors occur in scenes with low textural contrast or annotation ambiguity, indicating that HIDCap has already minimized most architectural shortcomings through hierarchical contextualization.

Overall, these extended analyses reaffirm HIDCap’s strong generalization capacity, interpretability, and resilience. The framework not only achieves state-of-the-art quantitative performance but also demonstrates qualitative robustness and adaptability across training conditions, datasets, and visual complexities.

5. Conclusion and Future Directions

In this study, we presented HIDCap, a novel Hierarchical Instance-Driven Captioning framework designed for remote sensing image description. Unlike conventional captioning systems that operate on uniform spatial grids or single-level attention, HIDCap introduces a unified perspective that jointly considers object-centric, contextual, and global semantics through an integrated cross-hierarchy architecture. By combining multi-instance feature extraction with adaptive cross-level attention, HIDCap successfully bridges the gap between fine-grained instance understanding and large-scale scene comprehension, demonstrating superior flexibility and interpretability in handling the complex visual patterns of aerial and satellite imagery.

Specifically, our framework leverages a pre-trained Faster R-CNN [9] to identify salient instances and their neighboring regions, thereby constructing object-level and patch-level feature embeddings. These multi-instance representations are subsequently integrated with global scene descriptors to form a comprehensive semantic hierarchy. Such design enables the encoder to precisely capture spatial and contextual correlations among objects, effectively overcoming the limitations of grid-based attention approaches that often ignore fine-grained structural cues. Furthermore, the proposed cross-hierarchy attention module dynamically balances focus between localized entities and global context, ensuring that the decoder attends to semantically relevant visual areas during the generation process.

Quantitative experiments on three major benchmarks—UCM, Sydney, and RSICD—validate the robustness and superiority of HIDCap across various evaluation metrics, including BLEU, CIDEr, and ROUGE. The model achieves consistent performance improvements over attention-based baselines such as SM-ATT and FC-ATT, particularly excelling in BLEU-4 and CIDEr scores. These results demonstrate that hierarchical feature decomposition and adaptive attention not only enhance visual understanding but also lead to more natural, detailed, and semantically coherent captions. The reinforcement learning (SCST) fine-tuning further refines linguistic fluency and aligns generated captions more closely with human-like descriptions, revealing the scalability of HIDCap for real-world deployment.

Beyond its empirical success, the HIDCap architecture contributes new insights into the synergy between structural reasoning and semantic composition in remote sensing tasks. By establishing explicit hierarchical interactions among visual elements, our approach introduces a more interpretable mechanism for understanding how high-level semantics emerge from multi-scale visual patterns. This interpretability is particularly valuable for applications in environmental monitoring, disaster assessment, urban planning, and resource management, where trustworthy, human-readable explanations are crucial.

5.1. Future Directions

While HIDCap demonstrates significant progress, several directions remain open for future research:

1) Integration with multimodal large language models. The rapid evolution of large-scale vision-language models provides an opportunity to further enrich HIDCap with contextual world knowledge. Integrating multimodal pre-trained backbones (e.g., CLIP or BLIP-based encoders) could enhance both domain adaptability and caption fluency.

2) Fine-grained semantic grounding. Future work may extend HIDCap to explicitly align linguistic tokens with visual instances through grounding supervision. This will facilitate more explainable captions by mapping words and phrases directly to specific geographic or structural regions in the image.

3) Temporal and spatiotemporal reasoning. Extending HIDCap to handle multi-temporal or video-based remote sensing data represents a promising avenue. Capturing temporal dynamics—such as urban expansion or vegetation change—can transform the framework into a generative monitoring tool for real-time scene evolution.

4) Adaptive multi-resolution processing. Although HIDCap employs a fixed hierarchical feature structure, future models may adopt adaptive resolution scaling, allowing the attention mechanism to adjust its granularity based on scene complexity. Such dynamic resolution modeling can further optimize computational efficiency and descriptive detail.

In conclusion, HIDCap represents a step toward semantically grounded, hierarchically interpretable, and generalizable captioning for remote sensing imagery. The model’s success illustrates the power of unifying multi-level visual representations with adaptive attention, paving the way for future research on large-scale, explainable vision-language systems in the geospatial intelligence domain.