Adaptive Semantic Fusion for Contextual Image Captioning

1. Introduction

Image captioning, often referred to as image-to-text translation, is a fundamental task at the intersection of computer vision and natural language processing [2,4,7,12]. It aims to generate meaningful and descriptive textual captions for given images by integrating scene understanding and language generation.

Despite significant progress, many existing approaches tend to over-rely on frequently occurring n-grams in the training data, leading to captions that may not accurately reflect the content of the image [7]. To address this issue, visual concept prediction methods have been proposed to bridge the semantic gap by extracting meaningful tags from images and incorporating them into the caption generation process [7,8,26,27,28]. These methods typically predict the likelihood of semantic concepts derived from a predefined image-grounded vocabulary, enabling better alignment between visual content and generated captions.

However, most prior approaches utilize Long Short-Term Memory (LSTM) networks [11] as language decoders, which, while effective, suffer from sequential processing limitations that hinder parallelization. In contrast, the Transformer architecture [22], originally developed for neural machine translation, has demonstrated exceptional parallelism and adaptability across various applications [6,9,18,20,24]. By leveraging self-attention mechanisms, Transformers offer a powerful alternative for image captioning tasks.

Humans typically adopt a dual-cognitive process when describing images, alternating between “thinking” of words to construct coherent sentences and “looking” at image regions to capture specific content. This insight has inspired adaptive attention models [17], which observe that not all words in a caption are directly grounded in visual content. Functional words, such as determiners or conjunctions, can often be inferred through language modeling, while content words, such as nouns and verbs, require explicit visual grounding.

Building on this understanding, we propose a novel framework called Dynamic Iterative Refinement Model (DIRM). This framework integrates a language-decoder-guided gating mechanism to dynamically modulate visual semantic vectors, ensuring that captions are both contextually coherent and semantically aligned with the image content. The key contributions of our work are summarized as follows:

-

We introduce a novel dynamic gating mechanism that refines the language decoder’s output using visual semantic vectors, enabling the generation of more descriptive and accurate captions.

-

A scatter connection layer is proposed to effectively align visual-semantic features with the decoder’s vocabulary, ensuring robust semantic representation.

-

Extensive experiments on the MS-COCO dataset demonstrate the superiority of our approach, achieving state-of-the-art performance compared to existing visual-semantic-based captioning models.

2. Related Work

The task of image captioning has attracted substantial attention due to its potential applications in assistive technologies, content retrieval, and multimedia generation. This section reviews key advancements in image captioning methodologies, focusing on visual-semantic alignment, language modeling, and the integration of attention mechanisms.

2.1. Image Captioning with Visual-Semantic Alignment

Visual-semantic alignment is a foundational aspect of image captioning. Early approaches in this domain focused on encoding image features using convolutional neural networks (CNNs) and decoding captions with recurrent neural networks (RNNs). These models leveraged global image features, but their coarse-grained nature often resulted in generic captions. To address this limitation, researchers introduced object-level visual features, which involved detecting and encoding objects in an image to provide more fine-grained semantic representations [2]. The use of object detection models such as Faster R-CNN [21] enabled these systems to extract localized features, thereby improving caption quality. Concurrently, visual concept prediction methods [8,28] sought to predefine semantic tags associated with image content, which were then used as additional inputs to enhance caption generation. Despite these advancements, many models struggled to dynamically integrate semantic representations with language generation. This motivated the development of adaptive attention mechanisms [17], which allowed models to selectively focus on relevant image regions during different stages of caption generation. These approaches demonstrated that incorporating visual-semantic alignment at both global and local levels significantly enhances the quality of generated captions.

2.2. Transformer-Based Architectures for Captioning

The advent of Transformer architectures [22] has revolutionized many areas of natural language processing and computer vision, including image captioning. Unlike RNNs, Transformers employ self-attention mechanisms to process entire sequences in parallel, making them more computationally efficient and scalable. Transformers have been successfully applied to image captioning in models such as Image Transformer [20] and Transformer-based encoder-decoder frameworks. These models leverage pre-trained vision encoders, such as Vision Transformers (ViT), to extract image embeddings, which are then processed by the Transformer decoder to generate captions. Recent works have focused on enhancing Transformer architectures for captioning by incorporating additional semantic cues. For instance, the OSCAR model uses object tags as auxiliary inputs to improve visual grounding. Similarly, M2 Transformer introduces multi-head cross-attention layers to better integrate visual and textual features. These methods have demonstrated state-of-the-art performance on benchmark datasets such as MS-COCO.

2.3. Reinforcement Learning and Evaluation Metrics in Captioning

Traditional supervised learning approaches for image captioning optimize cross-entropy loss, which does not directly align with evaluation metrics like BLEU, METEOR, ROUGE, and CIDEr. As a result, reinforcement learning (RL) techniques, particularly policy gradient methods, have been adopted to directly optimize these metrics. Self-critical sequence training (SCST) is a widely used RL-based approach in image captioning. It utilizes the model’s own predictions as baselines to compute rewards, enabling it to generate captions that better align with human judgments. Additionally, RL has been used to optimize diverse objectives, such as diversity and coherence in caption generation. Despite the success of RL, challenges remain in balancing metric optimization with linguistic quality. Recent efforts have explored hybrid loss functions that combine RL with supervised learning to achieve a balance between human-like fluency and metric-based optimization.

2.4. Limitations of Existing Approaches

While significant progress has been made, existing image captioning models face notable limitations. Many approaches rely on static vocabularies, which restrict their ability to generalize to unseen visual concepts. Moreover, the focus on global evaluation metrics often overlooks the importance of fine-grained linguistic coherence and semantic richness. Addressing these challenges requires models that can dynamically refine semantic representations and adaptively align them with language generation. Our proposed framework builds upon these advancements by introducing a novel method for iterative refinement of visual semantics, enabling more descriptive and contextually accurate captions.

3. Methodology

This section details the proposed Dynamic Iterative Refinement Model (DIRM), which integrates a Transformer-based relational encoder, a semantic-aware decoder, and a visual-concept refinement mechanism to generate high-quality captions. The model is designed to iteratively refine semantic representations, enabling better alignment between visual content and textual descriptions.

3.1. Relational Encoding with Transformers

Inspired by [2], we adopt Faster R-CNN [21] with ResNet-101 [10] as the base object detector. The detector generates M object proposals using a Region Proposal Network (RPN) and computes mean-pooled convolutional features for each proposal, resulting in a 2048-dimensional feature vector per object.

Let the extracted features for all proposals be denoted as:

$$\mathbf{X}={[{\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_M]}^{\top }\in {\mathbb{R}}^{M\times 2048}.$$

To reduce the feature dimensionality, we apply a fully connected layer:

$${\mathbf{X}}_r=\mathbf{X}{\mathbf{W}}_{\mathbf{r}},$$

where ${\mathbf{W}}_{\mathbf{r}}\in {\mathbb{R}}^{2048\times 512}$ is a learnable weight matrix, producing ${\mathbf{X}}_r\in {\mathbb{R}}^{M\times 512}$. Next, a Transformer encoder with $N_{\mathrm{enc}}$ layers processes the reduced features to capture object-object relationships [29]:

$$\mathbf{F}=\mathrm{TransformerEncoder}\left({\mathbf{X}}_r\right).$$

The output, $\mathbf{F}\in {\mathbb{R}}^{M\times 512}$, represents object-wise relational features that serve as input to the caption generation module.

3.2. Semantic-Aware Caption Decoder

The caption generator employs a Transformer decoder with $N_{\mathrm{dec}}$ layers. For a given caption sequence of length T, the decoder operates as follows: 1. Input words are first embedded via a word embedding layer and enriched with positional encodings:

$${\mathbf{E}}_t={\mathbf{W}}_e{\mathbf{w}}_t+{\mathbf{P}}_t,t=1,2,\dots ,T,$$

where ${\mathbf{W}}_e\in {\mathbb{R}}^{\left|\mathcal{V}\right|\times 512}$ is the embedding matrix, and ${\mathbf{P}}_t$ encodes positional information. 2. A masked self-attention sublayer computes contextual relationships within the sequence, attending only to prior tokens. 3. A cross-attention sublayer integrates multi-modal information by aligning ${\mathbf{E}}_t$ with visual features $\mathbf{F}$:

$${\mathbf{C}}_t=\mathrm{softmax}\left({\displaystyle \frac{{\mathbf{E}}_t{\mathbf{F}}^{\top }}{\sqrt{d_k}}}\right)\mathbf{F}.$$

4. Finally, a feed-forward network (FFN) processes the attention outputs:

$${\mathbf{H}}_t=\mathrm{FFN}\left({\mathbf{C}}_t\right).$$

Each sublayer is wrapped with residual connections [10] and layer normalization [3], ensuring stable training and effective gradient flow.

3.3. Dynamic Visual-Concept Refinement

Visual Concept Layer

To capture semantic concepts, we construct an image-grounded vocabulary ${\mathcal{V}}_{\mathrm{tag}}$ comprising the K most frequent nouns, verbs, and adjectives in the dataset [8], with $K=1000$. For each image, visual features $\mathbf{F}$ are transformed into a K-dimensional concept vector:

$$\mathbf{v}=\sigma \left(\mathrm{concat}({\mathbf{f}}_1{\mathbf{W}}_0,{\mathbf{f}}_2{\mathbf{W}}_0,\dots ,{\mathbf{f}}_M{\mathbf{W}}_0)\right),$$

where ${\mathbf{W}}_0\in {\mathbb{R}}^{512\times (K/M)}$ and $\sigma \left(x\right)=1/(1+exp(-x\left)\right)$. The resulting $\mathbf{v}\in {\mathbb{R}}^K$ represents the likelihood of each concept in ${\mathcal{V}}_{\mathrm{tag}}$.

Decoder-Guided Refinement

To dynamically modulate the visual-concept vector, we compute a decoder-guided gating mechanism:

$${\mathbf{g}}_t=\sigma \left({\mathbf{W}}_1{\mathbf{h}}_t+{\mathbf{W}}_2\mathbf{F}\right),$$

where ${\mathbf{h}}_t\in {\mathbb{R}}^{512}$ is the t-th decoder hidden state, and ${\mathbf{g}}_t\in {\mathbb{R}}^K$ is a gating vector. The refined concept representation is:

$${\mathbf{o}}_t={\mathbf{g}}_t\odot \mathbf{v}.$$

Scatter-Connected Mapping

The refined concepts are integrated with the decoder vocabulary using scatter mapping. For each word $j\in {\mathcal{V}}_{\mathrm{cap}}$:

$${\mathbf{h}}_t\left[j\right]=\left\{\begin{array}{cc}{\mathbf{h}}_t\left[j\right]+{\mathbf{o}}_t\left[k\right],\hfill & \mathrm{if}{\mathcal{V}}_{\mathrm{cap}}\left(j\right)={\mathcal{V}}_{\mathrm{tag}}\left(k\right),\hfill \\ {\mathbf{h}}_t\left[j\right],\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

The final vocabulary distribution is computed using a softmax function.

3.4. Training with Reinforcement Learning

The entire model is optimized using a hybrid loss function comprising cross-entropy and reinforcement learning (RL). For RL, we employ a policy gradient approach, where the reward is based on CIDEr [23] scores:

$$L_{\mathrm{RL}}=-{\mathbb{E}}_{{\pi }_{\theta }}\left[R\left(\widehat{y}\right)\right],$$

where $R\left(\widehat{y}\right)$ is the CIDEr score for the generated caption $\widehat{y}$. The total loss is:

$$L=L_{\mathrm{XE}}+\lambda L_{\mathrm{RL}},$$

where $\lambda$ balances supervised and RL losses.

4. Experiments

This section presents a comprehensive evaluation of the proposed Dynamic Iterative Refinement Model (DIRM). We detail the experimental setup, quantitative results, qualitative analysis, and additional insights through ablation studies and discussions.

4.1. Experimental Setup

Dataset and Evaluation

The MS-COCO dataset [16] is employed for evaluating DIRM. Following the Karpathy split, the dataset includes 113,287 images for training, 5,000 for validation, and 5,000 for testing, with each image paired with 5 human-generated captions. This diverse dataset provides a robust foundation for evaluating caption generation quality across varied scenes and contexts. We adopt the commonly used evaluation metrics BLEU [19], ROUGE-L [15], METEOR [5], CIDEr-D [23], and SPICE [1] to assess the generated captions. These metrics collectively evaluate fluency, semantic relevance, and adequacy.

Implementation Details

DIRM is implemented using PyTorch. The embedding dimension (D) is set to 512, with the encoder and decoder each comprising 3 Transformer layers for efficient yet effective learning. The batch size is set to 50, and the number of attention heads is 8. The feed-forward network (FFN) has a hidden size of 2048. To limit computational overhead, the maximum number of extracted object features per image is set to 50. Word embeddings are initialized randomly.

Optimization is performed using the Adam optimizer [13] with ${\beta }_1=0.9$ and ${\beta }_2=0.98$. The dropout rate is set to 0.1 to prevent overfitting, and early stopping is applied with a patience of 5 epochs. Beam search decoding is used with a beam width of 5. Prior to reinforcement learning (RL), the model is pretrained using supervised learning with cross-entropy loss. All experiments are conducted on a single NVIDIA Tesla V100 GPU.

Training Workflow

The training consists of two stages: 1. Supervised Pretraining: The model is trained using cross-entropy loss to learn the mapping between images and captions. 2. Reinforcement Learning Fine-tuning: The pretrained model is further optimized using the CIDEr score as a reward signal. The REINFORCE algorithm [28] with a baseline strategy is used to stabilize training.

4.2. Quantitative Results

The quantitative evaluation results on the MS-COCO dataset are shown in Table 1. The proposed DIRM achieves superior performance across all evaluation metrics, demonstrating its ability to generate captions that are both semantically accurate and contextually appropriate. Compared to baseline models, DIRM exhibits significant improvements, particularly in CIDEr and SPICE, which reflect semantic richness and content relevance.

Improved Semantic Understanding

The SPICE score of DIRM highlights its effectiveness in capturing semantic structures. This improvement stems from the visual-concept refinement module, which dynamically adjusts the semantic representations based on image content and linguistic context.

4.3. Ablation Study

To evaluate the contribution of key components in DIRM, we perform an ablation study. Table 2 presents the results of removing the visual-concept refinement module and scatter-connected mapping. Both components significantly impact performance, especially in CIDEr and SPICE scores, emphasizing their importance in enhancing semantic understanding.

4.4. Qualitative Analysis

Contextual Accuracy

DIRM generates captions with high contextual relevance. For instance, in images containing multiple objects or complex scenes, the model accurately describes key interactions, avoiding common errors such as object misidentification or redundant phrases.

Enhanced Semantic Detail

Compared to baseline methods, DIRM demonstrates a superior ability to capture fine-grained details. For example, it can correctly describe background elements or subtle visual cues, such as "a person holding an umbrella near a bustling market," which are often overlooked by simpler models.

Generalizability to Diverse Scenes

The implicit visual-concept modeling in DIRM enhances its generalization capability. It effectively generates accurate captions even for images containing rare objects or previously unseen combinations of objects.

4.5. Error Analysis and Future Directions

While DIRM achieves state-of-the-art results, there are some limitations. For example, the model occasionally generates captions with minor grammatical errors or overly verbose descriptions. Future work could explore incorporating syntactic constraints and more advanced language modeling techniques to address these challenges.

5. Conclusions and Future Directions

In this work, we introduced a novel image captioning framework, referred to as the Dynamic Iterative Refinement Model (DIRM), designed to generate semantically rich and contextually appropriate descriptive sentences by dynamically integrating visual concepts and linguistic features. Our approach leverages a combination of visual-concept refinement and scatter-connected mappings, enabling more precise alignment between visual content and generated text. The experimental results on the MS-COCO dataset demonstrated the superiority of DIRM over existing state-of-the-art methods, achieving significant improvements across multiple evaluation metrics, including BLEU, CIDEr, and SPICE.

The key contributions of this work can be summarized as follows:

-

We proposed a dynamic refinement mechanism that integrates visual concepts into the captioning process, allowing the model to focus on semantically important visual elements.

-

A scatter-connected mapping strategy was introduced, effectively aligning the visual-concept vocabulary with the decoder’s linguistic output, resulting in enhanced semantic accuracy.

-

Extensive experiments validated the effectiveness of DIRM, highlighting its ability to generate more descriptive, accurate, and contextually relevant captions compared to baseline models.

5.1. Future Directions

Despite the promising results achieved by DIRM, there remain several areas for improvement and exploration. These include:

1. Enhancing Generalization to Diverse Datasets

While DIRM performs exceptionally well on the MS-COCO dataset, future research could explore its generalizability to other datasets with varying domain-specific challenges, such as Flickr 30k [30] or domain-specific datasets like VizWiz [31]. Adapting the model to diverse visual styles and less-structured annotations could provide further insights into its robustness.

2. Incorporating Multimodal Information

DIRM currently focuses on static image-to-text translation. Expanding this framework to handle multimodal inputs, such as video sequences or audio-visual data, could significantly broaden its applicability. Temporal dynamics and audio cues could enhance the descriptive quality of generated captions in complex scenarios.

3. Refining Linguistic Coherence and Style

While the scatter mapping mechanism effectively improves semantic alignment, future work could explore methods to further refine linguistic coherence and stylistic diversity in generated captions. Techniques such as controllable text generation or large-scale pretraining on diverse corpora may help achieve this goal.

4. Real-Time and Low-Resource Adaptation

Improving the computational efficiency of DIRM for real-time applications is a practical direction for future work. Additionally, investigating methods to reduce the reliance on large annotated datasets, such as semi-supervised or unsupervised learning approaches, could make the model more accessible for resource-constrained applications.

5. Explainable and Trustworthy Image Captioning

As the demand for transparent AI systems increases, integrating explainability into the DIRM framework could provide users with better insights into how captions are generated. Visualizing attention distributions or identifying key features responsible for specific caption elements could enhance trust and interpretability.

In conclusion, the proposed DIRM framework represents a significant step forward in bridging the gap between visual content and textual descriptions. Its dynamic refinement capabilities, coupled with its robust performance, pave the way for future advancements in image captioning and related multimodal tasks.