Adaptive Transformer with Sequence-Guided Decoders for Enhanced Vision Captioning

1. Introduction

Image captioning, a critical task in computer vision [1], involves generating coherent and descriptive textual summaries for images. This process requires not only recognizing the visual elements within an image but also understanding their spatial arrangements, interactions, and contextual relationships. By bridging the gap between visual content and natural language, image captioning has wide-ranging applications, including assisting visually impaired individuals, enhancing image retrieval systems, and improving multimedia content accessibility [2].

Historically, early image captioning approaches were predominantly categorized into retrieval-based and template-based methods [7]. Retrieval-based methods relied on matching input images to existing datasets of captions through similarity metrics [8,9]. These approaches, though efficient, often struggled to generate novel captions and heavily depended on the quality of the dataset. Template-based approaches, on the other hand, utilized predefined syntactic templates combined with visual concept detections to construct captions [10,11,12]. While these methods provided structured outputs, their reliance on handcrafted rules and feature engineering limited flexibility and scalability.

With advancements in artificial intelligence, neural network-based methods have emerged as a transformative force in image captioning [13,14,15]. The introduction of encoder-decoder architectures [18] marked a significant leap, enabling the generation of captions by mapping image features to textual sequences. Subsequent innovations, such as multimodal learning [16,17], attention mechanisms [19,20], and compositional architectures [21], have further enhanced the capability of neural models to generate contextually accurate and semantically rich descriptions. Attention mechanisms, in particular, allow models to focus on specific image regions while generating captions, mimicking human attention patterns [26].

Transformers, characterized by their self-attention mechanisms and parallel processing capabilities, have become a dominant framework in image captioning [27,28]. A standard Transformer-based model processes visual features extracted from Convolutional Neural Networks (CNNs) through an encoder, which maps these features into intermediate representations. The decoder, often leveraging Recurrent Neural Networks (RNNs) or alternative architectures, then generates captions sequentially [4,16,18]. This architecture has demonstrated remarkable success due to its ability to model long-range dependencies and efficiently process multimodal data.

Despite these advancements, one critical aspect often overlooked in existing models is the explicit representation and utilization of geometric and positional relationships between objects in an image. For instance, captions such as "a boy standing on a skateboard" and "a boy holding a skateboard" convey different meanings, yet without spatial awareness, models may fail to distinguish between them. While some methods incorporate geometric information at the encoder level, such as the Geometry Self-Attention (GSA) proposed in [24], they often neglect detailed positional reasoning during decoding.

To address these limitations, this paper introduces the Adaptive Geometry-Integrated Transformer (AGIT), which integrates geometric and positional reasoning at both encoding and decoding stages. The proposed framework leverages a Geometry Attention Refiner (GAR) to enhance the encoder’s ability to capture spatial relationships explicitly. Additionally, it incorporates Position-Sensitive LSTMs (PS-LSTMs) in the decoder, enabling precise word-order modeling and semantic consistency in generated captions. These innovations ensure that AGIT not only captures the "where" and "how" of visual elements but also translates these insights into grammatically and contextually accurate textual descriptions.

2. Related Work

Recent advancements in encoder-decoder Transformer architectures, particularly those enhanced with attention mechanisms, have sparked significant interest in the field of image captioning. Numerous methodologies have been proposed to address the challenges of generating coherent and descriptive captions for images, including approaches that utilize soft and hard attention mechanisms [29], residual connections [30], and the meshed-memory Transformer [31].

2.1. Advancements in Transformer Architectures

Transformer architectures [24,27,32,33] have established themselves as a dominant framework in image captioning due to their flexibility and effectiveness. Typically, these architectures comprise an encoder and a decoder. The encoder extracts features from image regions, while the decoder generates textual descriptions based on these features. Both components employ multi-layer residual networks, which improve gradient flow and enable deeper models. A key feature of these architectures is the self-attention mechanism, which captures long-range dependencies and contextual relationships among input data [27,28].

A noteworthy enhancement in this framework involves the integration of geometric cues. For example, additional features such as object centers, sizes, and spatial relationships [24] have been incorporated into encoder inputs to provide richer contextual information. Positional encoding techniques, such as sinusoidal functions, have further improved the sequential nature of generated text by maintaining word order consistency [26]. These innovations have significantly improved the quality of generated captions.

2.2. Enhancements in Attention Mechanisms

The attention mechanism plays a pivotal role in image captioning, enabling models to focus on relevant parts of an image while generating captions. Beyond the basic attention mechanism [29], various extensions have been developed to address specific limitations. Multi-head attention [26] allows the model to attend to different parts of an image simultaneously, providing diverse perspectives. Gate-controlled attention [21] selectively filters information, enhancing the alignment between image regions and generated words.

Other notable attention mechanisms include fully attentive paradigms [28,34], which leverage comprehensive attention across all input tokens, and meshed-connection attention [31], which establishes dense connections between layers to improve information flow. Dual attention mechanisms [35] combine spatial and semantic attention, offering a more holistic approach to aligning image features with textual outputs. These advancements have contributed to the robustness and versatility of image captioning systems.

2.3. Future Directions in Image Captioning

Despite the significant progress achieved with Transformer-based architectures, there remains ample scope for further innovation. Future research could focus on designing more sophisticated network architectures that better capture the hierarchical relationships within images. For example, leveraging graph neural networks (GNNs) to model the structural relationships between objects could enhance spatial reasoning capabilities.

In addition, integrating richer feature representations, such as high-resolution spatial details and temporal dynamics, could further improve caption quality. Advanced geometric encoding strategies that incorporate not only relative positions but also three-dimensional spatial information could be explored. Similarly, more dynamic positional encoding methods that adapt to varying text lengths and content could enhance decoder performance.

Lastly, the incorporation of multi-modal learning techniques, which combine visual, textual, and auditory data, represents a promising avenue for expanding the capabilities of image captioning systems [36]. By embracing these directions, the field of image captioning can continue to evolve, addressing increasingly complex tasks and real-world applications.

3. Methodology

Our proposed method, Adaptive Geometry-Integrated Transformer (AGIT), generates grounded captions by dynamically attending to specific image regions at each step. It retains the foundational encoder-decoder architecture while introducing novel enhancements to improve performance. In the encoder, a Geometry Self-Attention Refiner (GSR) optimizes image representations by integrating spatial relationships. The decoder employs a Position-Aware Self-Attention mechanism to generate word sequences that accurately reflect the contextual and spatial intricacies of the image.

3.0.0.1. Task Definition.

Given an input image I, we represent its appearance features as $({\mathit{X}}_A\in {\mathbb{R}}^{N\times d})$, where N is the number of image regions, and d is the feature dimension. The corresponding caption sequence is denoted as $(\mathit{y}=\{{\mathit{y}}_1,\cdots ,{\mathit{y}}_T\})$, with T indicating the number of words.

3.1. Geometry Self-Attention Refiner

To enhance the encoding of spatial relationships, the Geometry Self-Attention Refiner incorporates object geometry into the traditional self-attention mechanism. This integration ensures that both visual and spatial information contribute to feature representation.

Incorporating Geometry Features. The geometry features of each object are represented as ${\mathit{X}}_g\in {\mathbb{R}}^{N\times 5}$, with each row containing:

$$(x_{min},y_{min},x_{max},y_{max},S)$$

(1)

where $(x_{min},y_{min})$ and $(x_{max},y_{max})$ are the coordinates of the bounding box, and S represents the normalized area of the bounding box. These features are embedded into a higher-dimensional space through a non-linear transformation:

$${\mathit{X}}_G=\mathrm{ReLU}({\mathit{X}}_g{\mathit{W}}_G+{\mathit{b}}_G)$$

(2)

where ${\mathit{W}}_G$ and ${\mathit{b}}_G$ are learnable parameters.

Combining Appearance and Geometry. The appearance features ${\mathbf{X}}_A$ and geometry features ${\mathit{X}}_G$ are combined to compute enhanced queries and keys:

$$\begin{array}{cc}\hfill {\mathit{Q}}^{\prime }& =[{\mathit{X}}_A{\mathit{W}}_{Q_A};{\mathit{X}}_G{\mathit{W}}_{Q_G}]\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill {\mathit{K}}^{\prime }& =[{\mathit{X}}_A{\mathit{W}}_{K_A};{\mathit{X}}_G{\mathit{W}}_{K_G}]\hfill \end{array}$$

(4)

where ${\mathit{W}}_{Q_A}$, ${\mathit{W}}_{Q_G}$, ${\mathit{W}}_{K_A}$, and ${\mathit{W}}_{K_G}$ are learnable weights. The attention scores are calculated as:

$${\mathbf{\Omega }}^{\prime }={\displaystyle \frac{{\mathit{Q}}^{\prime }{{\mathit{K}}^{\prime }}^T}{\sqrt{d}}}$$

(5)

The refined attention output is given by:

$${\mathrm{Attention}}_G\left(\mathit{X}\right)=\mathrm{softmax}\left({\mathbf{\Omega }}^{\prime }\right){\mathit{V}}_A$$

(6)

Gate-Controlled Refinement. To further refine the attention output, a Gate-Controlled Linear Unit (GLU) modulates the attention features:

$$\begin{array}{cc}\hfill \mathit{G}& =\sigma ({\mathit{W}}_G{\mathit{X}}_A+{\mathit{b}}_G)\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill \mathit{Output}& =\mathit{G}\odot {\mathit{Attention}}_G\left(\mathit{X}\right)\hfill \end{array}$$

(8)

where $\sigma$ denotes the sigmoid function, and ⊙ represents the element-wise product.

3.2. Position-Aware Decoder

The decoder integrates positional encoding into the self-attention mechanism, ensuring the sequential coherence of generated captions.

LSTM-Based Positional Encoding. At each time step t, the input to the LSTM includes the word embedding ${\mathbf{w}}_t$ and the pooled visual features $\mathit{v}$:

$${\mathit{x}}_t=[{\mathit{w}}_t;\mathit{v}],\mathit{v}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\mathit{X}}_A^{\left(i\right)}$$

(9)

The LSTM updates its hidden state as:

$${\mathit{h}}_t,{\mathit{c}}_t=\mathrm{LSTM}({\mathit{x}}_t,{\mathit{h}}_{t-1},{\mathit{c}}_{t-1})$$

(10)

Decoder Self-Attention. Using the refined features ${\mathit{X}}^r$ and the LSTM encoding ${\mathit{h}}_t$, the decoder computes word distributions as:

$$P\left({\mathit{y}}_t\right|{\mathit{y}}_{1:t-1})=\mathrm{softmax}({\mathit{W}}_o{\mathit{h}}_t+{\mathit{b}}_o)$$

(11)

where ${\mathit{W}}_o$ and ${\mathit{b}}_o$ are learnable parameters.

By integrating geometry-aware attention and position-sensitive decoding, AGIT achieves superior captioning performance, as demonstrated in our experiments.

4. Experiments

4.1. Datasets and Metrics

We evaluate our proposed AGIT model using two widely-adopted datasets: MS COCO [37] and Flickr30k [38]. MS COCO is among the largest and most comprehensive datasets for image captioning, comprising 123,287 images with five human-annotated captions per image. To ensure consistency in evaluation, we use the well-known "Karpathy" split [16], which includes 113,287 images for training, 5000 for validation, and 5000 for testing. Similarly, Flickr30k consists of 31,783 images, each annotated with five captions. For both datasets, captions exceeding 16 words are truncated, and all sentences are converted to lowercase to establish experimental vocabularies of 9,487 words for MS COCO and 7,000 words for Flickr30k.

To comprehensively evaluate the performance of AGIT, we utilize five standard metrics: BLEU [39], METEOR [40], ROUGE [41], CIDEr [42], and SPICE [43]. These metrics collectively measure various aspects of the generated captions, including n-gram precision, semantic alignment, and overall coherence.

4.2. Implementation Details

Our implementation employs a pre-trained Faster R-CNN model with a ResNet-101 backbone [4] to extract 36 features for each image, with each feature vector being 2048-dimensional. To optimize memory usage, these features are projected into a 512-dimensional space. The LSTM hidden state size is set to 1024, while the input dimensions for the Geometry Self-Attention Refiner (GSR) and self-attention modules are both configured to 512. Furthermore, the multi-head self-attention mechanism utilizes eight heads, and both the encoder and decoder are composed of three layers.

Dropout regularization is applied during training, with a dropout rate of 0.5 for the LSTM and 0.1 for the self-attention layers. During the cross-entropy training phase, the learning rate is initialized to $(5\times {10}^{-4})$ and decays by a factor of 0.8 every three epochs. For CIDEr optimization, the learning rate is set to $(2\times {10}^{-5})$ and follows the same decay schedule. All models are trained using the Adam optimizer with a batch size of 50. For inference, we employ beam search with a beam size of five across all experiments.

4.3. Ablation Experiments

To evaluate the contributions of individual components within AGIT, we conduct a series of ablation studies on the MS COCO dataset. Our baseline is a vanilla Transformer model [24], which lacks geometric features and employs sinusoidal position encodings.

Effect of Geometry Self-Attention Refiner

To assess the impact of the GSR module, we integrate it into the baseline model. The GSR incorporates explicit geometric relationships into the encoder, refining raw image representations. As shown in Table 1, the CIDEr score improves significantly from 109.0 to 115.1. This enhancement demonstrates that without geometric relations, the baseline model is prone to misinterpreting irrelevant regions. The GSR effectively addresses this issue, enabling the model to identify precise spatial contexts and generate geometry-aware captions.

Effect of Position-Aware LSTM Decoder

Next, we replace the sinusoidal position encoding of the baseline model with the position-aware LSTM decoder. This module enhances the decoder’s capacity to incorporate sequential information. As indicated in Table 1, the CIDEr score increases by 5.9 points, underscoring the effectiveness of position-aware encoding in guiding the decoder to focus on semantically relevant regions.

Geometry Queries and Keys

To evaluate the efficiency of incorporating geometric features into queries and keys, we compare two strategies: addition and concatenation. Table 2 shows that concatenating appearance and geometric features yields superior performance. For instance, the BLEU-1 score increases from 75.0 (baseline) to 77.5 with concatenation, compared to 76.0 with addition.

Role of Gated Linear Units (GLUs)

GLUs refine the outputs of self-attention layers by selectively emphasizing important features. Table 2 indicates that integrating GLUs into the encoder achieves the best results, while applying them to both encoder and decoder leads to diminishing returns. This observation aligns with findings in [20], highlighting the importance of balanced module integration.

4.4. Comparison with State-of-the-Art Models

We compare AGIT against state-of-the-art models on MS COCO and Flickr30k datasets. These comparisons include SCST [3], Up-Down [4], ORT [27], AoANet [20], and others. Table illustrates that AGIT consistently outperforms these methods, achieving substantial improvements in BLEU-4, METEOR, and CIDEr scores.

4.5. Caption Text Comparisons

To qualitatively demonstrate the advantages of our proposed AGIT model, we present qualitative comparisons of captions generated by AGIT, the baseline ‘Vanilla Transformer’ model, and ground truth captions (GT1-GT3) for six selected images. It is evident that AGIT produces captions with a more comprehensive semantic structure and accurate positional relations. For instance, in the top-left image, AGIT correctly identifies that “people” are “under umbrellas” and “in front of the store,” whereas the baseline fails to capture such nuanced spatial relationships. Similarly, for the middle-right image, AGIT appropriately describes the “dog laying under the chair” instead of incorrectly stating “laying on the chair”.

The ability of AGIT to produce captions with precise geometric and positional relationships is primarily attributed to its Geometry Self-attention Refiner (GSR) module. This module explicitly integrates spatial correlations among image regions into the object feature representations. Furthermore, the position-LSTM plays a pivotal role in guiding the decoder to attend to relevant visual objects during each decoding step. Collectively, these enhancements enable AGIT to describe almost all key objects within an image, such as “people,” “umbrellas,” and “restaurant” in the aforementioned example, while also capturing their spatial interrelations. As a result, AGIT consistently generates reliable and spatially-aware captions that align closely with human interpretation.

5. Conclusions

In this paper, we introduce the Attention-based Geometry-Integrated Transformer (AGIT), a significant extension of the Transformer architecture specifically designed for image captioning tasks. AGIT effectively refines image representations by embedding geometric features of visual objects into their respective region encodings. Additionally, the position-LSTM module enhances the decoder by accurately encoding the sequential order of caption words.

Our ablation experiments reveal the effectiveness of the GSR module for geometric feature integration and the position-LSTM for positional encoding. Each module independently contributes to substantial performance improvements when integrated with a baseline model. Furthermore, comparative experiments conducted both offline and online validate the superiority of the AGIT framework over state-of-the-art methods. AGIT consistently outperforms competitors on benchmark datasets such as MS COCO and Flickr30k, highlighting its robustness and reliability in generating spatially-aware and semantically rich image captions.

6. Future Work

Future work will focus on extending AGIT to handle multimodal inputs, such as integrating audio or temporal video data for broader applications like video summarization and scene understanding. Enhancing the geometric representation by incorporating 3D spatial information, such as depth or point clouds, is another promising direction, especially for domains requiring precise spatial reasoning like robotics and medical imaging. Efficiency improvements through techniques like model pruning or quantization will enable deployment on resource-constrained devices. Further, exploring self-supervised or semi-supervised learning paradigms could expand AGIT’s applicability to scenarios with limited labeled data. Finally, improving explainability by visualizing attention mechanisms and providing user-friendly controls will increase trust and adaptability in critical applications.