Contextual Synergy through Explicit and Implicit Relations: A Unified Perspective for Image Description Generation

1. Introduction

The task of generating natural language descriptions for visual scenes—image captioning—stands at the crossroads of computer vision and natural language processing. By attempting to translate raw pixels into structured sentences, image captioning not only offers a challenging research problem but also promises transformative applications in domains such as assistive technologies for the visually impaired, early childhood education, cross-modal retrieval systems [1], and natural human-robot communication frameworks [2]. Its dual requirement of perception and linguistic reasoning has made it one of the most representative tasks for evaluating multimodal artificial intelligence.

Traditional encoder-decoder approaches [3,4] have relied on convolutional neural networks (CNNs) to encode global or region-based visual features, followed by recurrent neural networks (RNNs) to sequentially decode captions. Inspired by machine translation paradigms, attention mechanisms [5] were introduced, allowing models to selectively focus on specific regions while generating each word. Reinforcement learning strategies, such as self-critical sequence training [6], further optimized caption generation by aligning model objectives with evaluation metrics. The introduction of attention over object-level regions, rather than uniform grids, pushed the field forward substantially [7,8,9]. Yet, despite these developments, the question of how to holistically model associations between objects remains insufficiently addressed.

Research in visual cognition underscores that contextual cues are pivotal in object understanding [10]. Objects rarely exist in isolation: a "cup" on a "table" is semantically different from a "cup" in "someone’s hand." Explicit modeling of such relations offers a richer semantic substrate for captions. To this end, scene graph-based approaches [8,11] attempt to encode explicit pairwise relations, whereas transformer-based methods [9,12,13] capture implicit dependencies through self-attention. Nonetheless, both lines of research have inherent limitations: scene-graph methods suffer from inaccuracies in relation prediction due to reliance on external detectors, while transformer-only models lack explicit relational guidance, leading to less interpretable or diffuse contextual associations.

In this work, we propose to unify both explicit and implicit relational reasoning into a coherent framework. Specifically, we design a semantic graph construction module that extracts object-level relations, filtered by spatial constraints, and passes them through a Gated GCN for controlled aggregation. To complement this, we introduce a Region BERT encoder that employs bidirectional self-attention across all detected regions, thereby capturing global dependencies without the need for pre-labeled relation annotations. To reconcile the outputs from these two perspectives, we design a Dynamic Mixture Attention (DMA) module. Unlike simple concatenation or fusion, DMA adaptively assigns weights to explicit and implicit features in a channel-wise manner, guided by the decoder’s hidden state, resulting in representations that better align with linguistic generation.

Our contributions are threefold:

• We propose VisRelNet, a relational reasoning framework for image captioning, which integrates explicit graph-based relational modeling and implicit transformer-based contextual modeling in a complementary fashion.
• We introduce a Gated GCN for explicit relation reasoning and a Region BERT encoder for implicit global context learning, together providing enriched semantic features for each object region.
• We develop a Dynamic Mixture Attention (DMA) module that selectively balances explicit and implicit relational features during decoding, enabling the model to adaptively focus on the most relevant context when generating captions.

Beyond the technical contributions, this study reveals broader implications. First, relational reasoning is not limited to captioning but may be extended to other multimodal tasks, such as visual question answering, visual grounding, or multimodal summarization. Second, explicit-implicit relational fusion aligns well with cognitive theories of human vision, where individuals rely on both concrete object relationships and latent contextual knowledge. Lastly, the proposed VisRelNet offers a template for integrating structured reasoning with deep neural architectures, pushing multimodal research closer to human-like understanding.

By conducting extensive evaluations on the Microsoft COCO dataset, we demonstrate that VisRelNet not only achieves state-of-the-art performance but also produces captions with higher contextual fidelity. The enriched modeling of relationships enables the system to go beyond surface-level recognition, describing nuanced associations and delivering outputs that are more informative, context-aware, and interpretable.

2. Related Work

2.1. Advances in Image Captioning Paradigms

The problem of automatically producing natural language descriptions for images has been extensively studied and has become a benchmark task at the intersection of computer vision and natural language processing. Early approaches were largely template-based, where handcrafted rules or retrieval strategies were employed to match images with sentences from a fixed corpus. However, the recent surge of deep learning techniques, especially those inspired by neural machine translation, has revolutionized the field. Current systems are typically structured around an encoder-decoder framework [3], where a convolutional neural network (CNN) extracts high-level image representations and a recurrent neural network (RNN) or transformer-based decoder generates a sequence of words conditioned on these features.

Within this paradigm, Xu et al. [5] pioneered the application of soft and hard attention mechanisms, enabling models to dynamically attend to different image regions depending on the word being generated. This marked a significant shift toward more context-aware captioning. Later, Lu et al. [14] extended this work by introducing a visual sentinel, which allowed the model to learn when to rely more heavily on visual features versus when to depend on the language model’s internal state, effectively balancing multimodal sources of information.

Rennie et al. [6] further improved training through the Self-Critical Sequence Training (SCST) algorithm, which is a variant of REINFORCE. SCST leverages the model’s own inference outputs as a baseline to reduce variance in reward estimation. This simple yet powerful idea made reinforcement learning more practical in image captioning, aligning model optimization with evaluation metrics such as CIDEr and BLEU. The computational efficiency of SCST, requiring only one additional forward pass per iteration, quickly established it as a dominant training paradigm.

Beyond these seminal works, the field has seen rapid innovation with the adoption of transformer-based architectures, multimodal pretraining, and large-scale vision-language models. For example, the integration of pretrained visual backbones and language models has brought unprecedented improvements in fluency, diversity, and generalization. These advances suggest that the future of captioning will likely involve a tighter integration of vision-language pretraining, reasoning over external knowledge sources, and improved alignment strategies across modalities.

2.2. Contextual Region Representations and Relational Modeling

The representation of image content at the region level has emerged as a critical factor in improving caption quality. With the release of the Visual Genome dataset [15], researchers gained access to rich object-level annotations, enabling models to move beyond global CNN features toward fine-grained region-based descriptions. Most modern captioning systems therefore employ region proposals, often extracted via Faster R-CNN or related detectors, to capture localized semantics. This approach allows attention mechanisms to operate at the level of meaningful entities, such as “dog,” “ball,” or “tree,” instead of grid cells.

Anderson et al. [7] introduced the bottom-up and top-down attention framework, which significantly advanced the field by enabling object-centric reasoning. This approach allows a captioning model to ground words directly in detected objects, thereby producing descriptions that are more semantically precise. Building on this idea, Yao et al. [8] proposed to integrate spatial and semantic relations into the captioning process by leveraging graph convolutional networks (GCNs). Their method enabled the decoder to incorporate relational knowledge, such as spatial proximity or semantic similarity, thereby producing context-aware sentence structures.

Yang et al. [11] further extended relational modeling by proposing a Scene Graph Auto-Encoder (SGAE), where object nodes were connected not only to other objects but also to attributes (adjectives) and relationships (verbs or prepositions). This hierarchical representation closely mirrors the structure of natural language and allows the model to better capture compositional semantics. Similarly, Huang et al. [9] developed the Attention on Attention (AoA) module, which refines the traditional attention mechanism by applying attention over attention scores, allowing the model to better reason over complex interactions among objects.

The evolution of transformer-based captioning architectures has further enriched contextual region representations. Cornia et al. [12] proposed a memory-augmented transformer that integrates prior knowledge into multi-level relational reasoning, effectively modeling both short-range and long-range dependencies across regions. Guo et al. [13] introduced normalized self-attention, incorporating normalization within the self-attention mechanism, and proposed geometry-aware attention, which explicitly models spatial relations such as relative distances and angles between objects.

Despite these advances, existing methods often fall into two extremes: either relying heavily on explicit graph-based relational reasoning, which is prone to error propagation from relationship extractors, or depending entirely on implicit transformer-based mechanisms, which may lack interpretable relational grounding. In this paper, we propose to reconcile these two approaches by designing a unified relational modeling framework—VisRelNet—that leverages the complementary strengths of graph-based and transformer-based features for image captioning.

2.3. Toward Unified Relational Frameworks

While explicit and implicit relational modeling strategies have been independently effective, there has been limited exploration of their integration. We argue that combining explicit relational graphs with implicit contextual transformers can yield synergistic benefits. Graph-based models bring structured reasoning and interpretability, while transformer-based approaches excel at capturing global and long-range dependencies without requiring hand-annotated relation labels. By aligning these two perspectives, models can generate captions that are both grounded in object-level relationships and enriched by global context.

Concretely, explicit graph modeling ensures that local object interactions—such as “person holding cup” or “dog chasing ball”—are preserved with high fidelity, while implicit contextual modeling allows for more abstract descriptions, such as understanding the overall scene type or activity. The challenge lies in designing mechanisms to fuse these complementary features without overwhelming the decoder with redundant or noisy signals. Our proposed framework addresses this by introducing adaptive attention-based fusion strategies that balance the contributions of explicit and implicit features.

The broader significance of this line of work extends beyond captioning. Relational reasoning is fundamental to visual question answering, referring expression comprehension, and multimodal reasoning tasks. Thus, advances in this area may influence a wide spectrum of vision-language applications.

Figure 1. Overview of VisRelNet for explicit–implicit relational captioning. Given an input image, Faster R-CNN yields region features $\mathcal{V}={\mathit{v}}_i$ and positions. The explicit branch constructs a pruned semantic graph (MOTIFS-labeled edges) and refines nodes via multi-layer Gated GCN with edge-wise gates, aided by Laplacian smoothing and InfoNCE-style contrast; the implicit branch encodes regions with a bidirectional transformer (Region BERT) trained with MIM, MRM, region–image contrast, and order-consistency objectives. At decoding, Dynamic Mixture Attention attends over explicit ${\widehat{\mathcal{V}}}_x$ and implicit ${\widehat{\mathcal{V}}}_m$ contexts and mixes them via a learned gate conditioned on the decoder state to form ${\widehat{\mathit{v}}}_t$, which drives an LSTM/Transformer to generate the caption. Training combines cross-entropy pretraining and self-critical RL (e.g., CIDEr-D) with relational regularizers; inference uses beam search with length normalization.

Figure 1. Overview of VisRelNet for explicit–implicit relational captioning. Given an input image, Faster R-CNN yields region features $\mathcal{V}={\mathit{v}}_i$ and positions. The explicit branch constructs a pruned semantic graph (MOTIFS-labeled edges) and refines nodes via multi-layer Gated GCN with edge-wise gates, aided by Laplacian smoothing and InfoNCE-style contrast; the implicit branch encodes regions with a bidirectional transformer (Region BERT) trained with MIM, MRM, region–image contrast, and order-consistency objectives. At decoding, Dynamic Mixture Attention attends over explicit ${\widehat{\mathcal{V}}}_x$ and implicit ${\widehat{\mathcal{V}}}_m$ contexts and mixes them via a learned gate conditioned on the decoder state to form ${\widehat{\mathit{v}}}_t$, which drives an LSTM/Transformer to generate the caption. Training combines cross-entropy pretraining and self-critical RL (e.g., CIDEr-D) with relational regularizers; inference uses beam search with length normalization.

3. VisRelNet for Explicit–Implicit Relational Captioning

In this section, we present a unified framework, denoted as VisRelNet, that learns to exploit explicit and implicit visual relationships for image captioning in a complementary manner. Given an image, we first adopt Faster R-CNN to obtain a variable-sized set of region-level descriptors $\mathcal{V}=\{{\mathit{v}}_1,{\mathit{v}}_2,\dots ,{\mathit{v}}_k\}$ with ${\mathit{v}}_i\in {\mathbb{R}}^v$. We then instantiate two relational pathways: (i) an explicit pathway based on a relation-filtered semantic graph processed by a Gated Graph Convolutional Network (Gated GCN), producing ${\widehat{\mathcal{V}}}_x$; and (ii) an implicit pathway that leverages a region-level bidirectional transformer encoder (Region BERT) with self-supervised pretraining, producing ${\widehat{\mathcal{V}}}_m$. Finally, a Dynamic Mixture Attention (DMA) module adaptively fuses both sources of context at decoding time to generate fluent and semantically grounded captions. Below we detail each component.

3.1. Task Setup and Notation

We consider the standard image captioning problem: given an image $\mathcal{I}$, the goal is to produce a sentence $\mathcal{S}=\{y_1,y_2,\dots ,y_T\}$ of length T. Following common practice, we employ a region-based encoder that yields $\mathcal{V}={\left\{{\mathit{v}}_i\right\}}_{i=1}^k$, where each ${\mathit{v}}_i$ encodes a salient object/part. Let ${\mathit{h}}_t\in {\mathbb{R}}^d$ denote the decoder hidden state at time t. Throughout, d is the model width and v the visual feature dimension. Unless otherwise stated, matrices are learnable parameters and ${\parallel ·\parallel }_2$ denotes the Euclidean norm.

3.1.0.1. Region Feature Extraction

We utilize Faster R-CNN to produce k proposals per image and extract ${\mathit{v}}_i\in {\mathbb{R}}^v$ via ROI pooling and a projection layer. For each region we also derive a positional descriptor ${\mathit{p}}_i\in {\mathbb{R}}^6$ (normalized $\mathtt{top}$/$\mathtt{bottom}$/$\mathtt{left}$/$\mathtt{right}$/$\mathtt{width}$/$\mathtt{height}$). When needed, we concatenate ${\mathit{v}}_i$ and a learned embedding of ${\mathit{p}}_i$ to form enriched inputs.

3.2. Explicit Relational Pathway: Semantic Graph with Gated GCN

3.2.1. Graph Construction and Edge Pruning

We start from a directed complete graph ${\mathcal{G}}_o=\langle \mathcal{V},{\mathcal{E}}_o\rangle$ whose vertices correspond to region features and whose edges represent potential pairwise relationships. For two regions i and j with bounding-box centroids $(x_i,y_i)$ and $(x_j,y_j)$, we compute the pairwise distance

$$d_{ij}=\sqrt{{(x_i-x_j)}^2+{(y_i-y_j)}^2},$$

(1)

and the Intersection-over-Union ${\mathtt{IoU}}_{ij}$ between their boxes. Let $l_{ij}$ denote the length of the longer side among the two boxes. Following a geometric prior that distant and non-overlapping regions are unlikely to be directly related, we prune edges (${\mathit{e}}_{ij}$ and ${\mathit{e}}_{ji}$) from ${\mathcal{G}}_o$ whenever

$${\mathtt{IoU}}_{ij}=0\mathrm{and}{d}_{ij}>l_{ij}.$$

(2)

This yields a reduced graph ${\mathcal{G}}_r$. Next, a visual relationship classifier (MOTIFS [16]) predicts a categorical relation label for each remaining directed edge. For robustness, among potentially multiple relation hypotheses per pair $(i,j)$ we keep at most one label with the highest confidence, producing a semantic graph ${\mathcal{G}}_s=\langle \mathcal{V},{\mathcal{E}}_s\rangle$.

Relation and Direction Embeddings

Each edge $(i\to j)$ carries a direction type $dir(i,j)\in \{1,2,3\}$ (forward, backward, self) and a relation label $lab(i,j)$. We embed both via learnable tables:

$${\mathit{r}}_{ij}={\mathit{E}}_{\mathrm{rel}}\left[lab(i,j)\right],{\mathit{d}}_{ij}={\mathit{E}}_{\mathrm{dir}}\left[dir(i,j)\right].$$

(3)

These vectors modulate message passing (below) to encode directionality and relation semantics.

3.2.2. Gated Graph Convolution and Residual Stacking

To aggregate local relational context, we apply a Gated GCN layer that is sensitive to both direction and relation labels. For node i, a single layer computes

$${\mathit{v}}_i^{\prime }=\mathrm{ReLU}\left(\sum _{{\mathit{v}}_j\in \mathcal{N}\left({\mathit{v}}_i\right)}{\mathit{W}}_{dir(i,j)}{\mathit{v}}_j+{\mathit{b}}_{lab(i,j)}\right),$$

(4)

where $\mathcal{N}\left({\mathit{v}}_i\right)$ contains all neighbors including i itself, ${\mathit{W}}_{dir(i,j)}\in {\mathbb{R}}^{d\times v}$ depends on direction, and ${\mathit{b}}_{lab(i,j)}\in {\mathbb{R}}^d$ depends on the relation label. To adaptively weigh edges, we introduce an edge-wise gate:

$$\begin{array}{c}{\mathit{v}}_i^{\prime }=\mathrm{ReLU}\left(\sum _{{\mathit{v}}_j\in \mathcal{N}\left({\mathit{v}}_i\right)}{g}_{ij}\left({\mathit{W}}_{dir(i,j)}{\mathit{v}}_j+{\mathit{b}}_{lab(i,j)}\right)\right),\end{array}$$

(5)

$$\begin{array}{c}{g}_{ij}\propto {\mathit{w}}_g^{\top }tanh\left({\mathit{W}}_a{\mathit{v}}_{sub(i,j)}+{\mathit{W}}_b{\mathit{v}}_{obj(i,j)}+{\mathit{U}}_r{\mathit{r}}_{ij}+{\mathit{U}}_d{\mathit{d}}_{ij}\right),\end{array}$$

(6)

with ${\mathit{W}}_a,{\mathit{W}}_b\in {\mathbb{R}}^{d\times v}$, ${\mathit{U}}_r,{\mathit{U}}_d\in {\mathbb{R}}^{d\times d}$, and ${\mathit{w}}_g\in {\mathbb{R}}^d$. We then normalize ${\left\{g_{ij}\right\}}_j$ across neighbors via softmax for stability.

Multi-layer Propagation with Normalization

We stack $L_x$ such layers with residual connections and layer normalization:

$$\begin{array}{cc}\hfill {\tilde{\mathit{v}}}_i^{(\ell )}& =\mathrm{LayerNorm}\left({\mathit{v}}_i^{(\ell -1)}+{\mathit{v}}_i^{\prime (\ell )}\right),\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill {\mathit{v}}_i^{(\ell )}& =\mathrm{Dropout}\left({\tilde{\mathit{v}}}_i^{(\ell )}\right),\hfill \end{array}$$

(8)

with ${\mathit{v}}_i^{\left(0\right)}={\mathit{v}}_i$. The final explicit features are ${\widehat{\mathcal{V}}}_x={\left\{{\mathit{x}}_i\right\}}_{i=1}^k$ where ${\mathit{x}}_i={\mathit{v}}_i^{\left(L_x\right)}$.

Higher-order Smoothing and Contrastive Regularization

To encourage multi-hop relational coherence, we add a Laplacian smoothing penalty:

$${\mathcal{L}}_{\mathrm{lap}}={\displaystyle \frac{1}{|{\mathcal{E}}_s|}}\sum _{(i,j)\in {\mathcal{E}}_s}\parallel {\mathit{x}}_i-{\mathit{x}}_j{\parallel }_2^2,$$

(9)

and an InfoNCE-style edge contrastive objective that pulls positives $(i,j)$ together while pushing randomly sampled negatives $j^{-}$:

$${\mathcal{L}}_{\mathrm{con}}=-\sum _{(i,j)\in {\mathcal{E}}_s}log{\displaystyle \frac{exp(cos({\mathit{x}}_i,{\mathit{x}}_j)/\tau )}{exp(cos({\mathit{x}}_i,{\mathit{x}}_j)/\tau )+{\sum }_{j^{-}}exp(cos({\mathit{x}}_i,{\mathit{x}}_{j^{-}})/\tau )}},$$

(10)

with temperature $\tau >0$ and cosine similarity $cos(·,·)$.

3.3. Implicit Relational Pathway: Region BERT with Pretraining

3.3.1. Input Embedding and Transformer Encoding

We construct an input embedding per location by projecting the pooled visual feature and its position code into a shared space and summing them with layer normalization:

$${\mathit{z}}_i=\mathrm{LayerNorm}\left({\mathit{W}}_v{\mathit{v}}_i+{\mathit{W}}_p\varphi \left({\mathit{p}}_i\right)\right),$$

(11)

where $\varphi (·)$ is a positional MLP. A bidirectional transformer encoder then produces contextual region representations. For completeness, let $\mathrm{MHA}$ denote multi-head self-attention and $\mathrm{FFN}$ a position-wise feed-forward network. A standard encoder block computes

$$\begin{array}{cc}\hfill {\mathit{h}}_i^{(\ell )}& =\mathrm{LayerNorm}\left({\mathit{z}}_i^{(\ell -1)}+\mathrm{MHA}\left({\mathit{z}}_i^{(\ell -1)},{\mathcal{Z}}^{(\ell -1)},{\mathcal{Z}}^{(\ell -1)}\right)\right),\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill {\mathit{z}}_i^{(\ell )}& =\mathrm{LayerNorm}\left({\mathit{h}}_i^{(\ell )}+\mathrm{FFN}\left({\mathit{h}}_i^{(\ell )}\right)\right),\hfill \end{array}$$

(13)

with ${\mathcal{Z}}^{\left(0\right)}={\left\{{\mathit{z}}_i\right\}}_{i=1}^k$. After $L_m$ layers, we obtain ${\widehat{\mathcal{V}}}_m={\left\{{\mathit{m}}_i\right\}}_{i=1}^k$ where ${\mathit{m}}_i={\mathit{z}}_i^{\left(L_m\right)}$. We also maintain a global token embedding ${\mathit{v}}_g$ to summarize the image.

3.3.2. Match Image Modeling (MIM)

To align global and regional content, we adopt a binary matching objective. With probability $0.5$, ${\mathit{v}}_g$ corresponds to the same image as $\mathcal{V}$ (label $y=1$), and with probability $0.5$ it is sampled from another image (label $y=0$). We score the pair using an MLP on the global token’s contextualized output. The loss is

$$\begin{array}{cc}\hfill {\mathcal{L}}_{MIM}=-{\mathbb{E}}_{({\mathit{v}}_g,\mathcal{V})\sim \mathcal{D}}[& ylogP({\mathit{v}}_g,\mathcal{V})\hfill \\ \hfill & +(1-y)log(1-P({\mathit{v}}_g,\mathcal{V}))].\hfill \end{array}$$

(14)

3.3.3. Masked Region Modeling (MRM)

We randomly mask $10\%$ of region features and ask the encoder to reconstruct them from context. For a selected ${\mathit{v}}_i$, we replace it with zeros (80%), a random region (10%), or keep it unchanged (10%). With transformer outputs over the masked set ${\mathcal{V}}_{\mathrm{mask}}$, we learn predictors $f_i$ to reconstruct the original features:

$${\mathcal{L}}_{MRM}={\mathbb{E}}_{({\mathit{v}}_g,\mathcal{V})\sim {\mathcal{D}}^{\prime }}\left[\sum _i\parallel f_i({\mathit{v}}_g,{\mathcal{V}}_{\mathrm{mask}})-{\mathit{v}}_i{\parallel }_2^2\right].$$

(15)

3.3.4. Region–Image Contrast and Order Consistency (Auxiliary)

To strengthen global–local coupling, we incorporate a contrastive alignment between global summary and region embeddings:

$${\mathcal{L}}_{\mathrm{ric}}=-{\displaystyle \frac{1}{k}}\sum _{i=1}^{k}log{\displaystyle \frac{exp(cos({\mathit{q}}_i,{\mathit{v}}_g)/\tau )}{{\sum }_{{\mathit{v}}_g^{\prime }}exp(cos({\mathit{q}}_i,{\mathit{v}}_g^{\prime })/\tau )}},$$

(16)

where ${\mathit{q}}_i={\mathit{W}}_q{\mathit{m}}_i$ and the denominator includes in-batch negatives ${\mathit{v}}_g^{\prime }$. We also encourage geometric order consistency by predicting relative offsets $(\Delta x,\Delta y,\Delta w,\Delta h)$ from contextualized pairs, trained with a smooth-${\ell }_1$ loss ${\mathcal{L}}_{\mathrm{ord}}$.

Implicit Feature Set

The implicit pathway outputs ${\widehat{\mathcal{V}}}_m={\left\{{\mathit{m}}_i\right\}}_{i=1}^k$ and a global summary vector (used in training but not strictly required at decoding time).

3.4. Dynamic Mixture Attention (DMA) for Context Fusion

Given explicit features ${\widehat{\mathcal{V}}}_x=\left\{{\mathit{x}}_i\right\}$ and implicit features ${\widehat{\mathcal{V}}}_m=\left\{{\mathit{m}}_i\right\}$, DMA produces a token-wise fused context aligned with the decoder state ${\mathit{h}}_t$. We first compute two attentional summaries:

$$\begin{array}{cc}\hfill {\alpha }_{ti}& ={\displaystyle \frac{exp({\mathit{u}}_x^{\top }tanh({\mathit{W}}_x{\mathit{x}}_i+{\mathit{U}}_x{\mathit{h}}_t))}{{\sum }_{j}exp({\mathit{u}}_x^{\top }tanh({\mathit{W}}_x{\mathit{x}}_j+{\mathit{U}}_x{\mathit{h}}_t))}},\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill {\beta }_{ti}& ={\displaystyle \frac{exp({\mathit{u}}_m^{\top }tanh({\mathit{W}}_m{\mathit{m}}_i+{\mathit{U}}_m{\mathit{h}}_t))}{{\sum }_{j}exp({\mathit{u}}_m^{\top }tanh({\mathit{W}}_m{\mathit{m}}_j+{\mathit{U}}_m{\mathit{h}}_t))}},\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill {\widehat{\mathit{x}}}_t& =\sum _i{\alpha }_{ti}{\mathit{x}}_i,{\widehat{\mathit{m}}}_t=\sum _i{\beta }_{ti}{\mathit{m}}_i,\hfill \end{array}$$

(19)

where the attention form follows [7]. A channel-wise gate determines the mixture:

$${\mathit{g}}_t=\sigma \left({\mathit{W}}_g[{\widehat{\mathit{x}}}_t;{\widehat{\mathit{m}}}_t;{\mathit{h}}_t]\right),{\mathit{W}}_g\in {\mathbb{R}}^{d\times 3d}.$$

(20)

The fused context is

$${\widehat{\mathit{v}}}_t={\mathit{g}}_t\odot {\widehat{\mathit{x}}}_t+(1-{\mathit{g}}_t)\odot {\widehat{\mathit{m}}}_t.$$

(21)

To stabilize training, we optionally constrain gate entropy and add an $L_2$ penalty:

$${\mathcal{L}}_{\mathrm{gate}}={\lambda }_{\mathrm{ent}}·{\displaystyle \frac{1}{d}}\sum _{c=1}^d\left[-g_{t,c}log{g}_{t,c}-(1-g_{t,c})log(1-g_{t,c})\right]+{\lambda }_{\mathrm{l}2}{\parallel {\mathit{W}}_g\parallel }_2^2.$$

(22)

3.5. Language Decoder and Word Generation

VisRelNet is compatible with an LSTM or a Transformer decoder. For concreteness, we describe an LSTM variant. Let ${\mathit{e}}_{y_{t-1}}$ be the embedding of the previously generated token. We compute

$$\begin{array}{cc}\hfill {\mathit{s}}_t& =tanh\left({\mathit{W}}_s[{\mathit{e}}_{y_{t-1}};{\widehat{\mathit{v}}}_t]\right),\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill {\mathit{h}}_t,{\mathit{c}}_t& =\mathrm{LSTM}\left([{\mathit{e}}_{y_{t-1}};{\widehat{\mathit{v}}}_t],({\mathit{h}}_{t-1},{\mathit{c}}_{t-1})\right),\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill P\left(y_t\right|·)& =\mathrm{Softmax}\left({\mathit{W}}_o{\mathit{h}}_t+{\mathit{b}}_o\right).\hfill \end{array}$$

(25)

Here ${\mathit{s}}_t$ is an auxiliary fusion used to modulate the LSTM input; the probability over the vocabulary is computed from ${\mathit{h}}_t$.

3.6. Training Objectives

In Algorithm 1 we show the overall training algorithm.

Cross-Entropy Pretraining

Given ground-truth ${\mathcal{S}}^{*}$, we first minimize the token-level cross-entropy:

$${\mathcal{L}}_{XE}\left(\mathit{\theta }\right)=-logP\left({\mathcal{S}}^{*}\right|\mathcal{V}),$$

(26)

where $\mathit{\theta }$ denotes all trainable parameters. Label smoothing may be applied to mitigate overconfidence.

Self-Critical RL Fine-Tuning

To close the train–test mismatch and optimize sequence-level metrics, we adopt a reinforcement learning objective that maximizes the expected reward:

$${\mathcal{L}}_{RL}\left(\mathit{\theta }\right)={\mathbb{E}}_{{\mathcal{S}}^s\sim P_{\mathit{\theta }}}\left[r({\mathcal{S}}^s,{\mathcal{S}}^{*})\right],$$

(27)

where $r(·,·)$ is a sentence-level score such as CIDEr-D [17]. Using self-critical sequence training, the gradient is approximated by a REINFORCE estimator with a baseline given by the reward of the model’s own greedy decoding ${\mathcal{S}}^g$:

$${\nabla }_{\mathit{\theta }}{\mathcal{L}}_{RL}\approx -\left(r\left({\mathcal{S}}^s\right)-r\left({\mathcal{S}}^g\right)\right)\sum _t{\nabla }_{\mathit{\theta }}log{P}_{\mathit{\theta }}\left(y_t^s\right|·).$$

(28)

Auxiliary Regularizers for Relational Robustness

In addition to ${\mathcal{L}}_{XE}$ and ${\mathcal{L}}_{RL}$, we combine the explicit/implicit pathway losses:

$${\mathcal{L}}_{\mathrm{rel}}={\lambda }_{\mathrm{MIM}}{\mathcal{L}}_{MIM}+{\lambda }_{\mathrm{MRM}}{\mathcal{L}}_{MRM}+{\lambda }_{\mathrm{con}}{\mathcal{L}}_{\mathrm{con}}+{\lambda }_{\mathrm{lap}}{\mathcal{L}}_{\mathrm{lap}}+{\lambda }_{\mathrm{ord}}{\mathcal{L}}_{\mathrm{ord}}+{\lambda }_{\mathrm{gate}}{\mathcal{L}}_{\mathrm{gate}}.$$

(29)

The overall training objective is

$$\mathcal{L}={\mathcal{L}}_{XE}+{\lambda }_{\mathrm{RL}}{\mathcal{L}}_{RL}+{\mathcal{L}}_{\mathrm{rel}}.$$

(30)

3.7. Inference and Decoding Strategy

At inference time, we use beam search with length normalization to alleviate preference for shorter captions. Let $\ell \left(\mathcal{S}\right)={\sum }_{t}logP\left(y_t\right|·)$ be the log-likelihood; we choose

$${\mathcal{S}}^{★}=arg\underset{\mathcal{S}}{max}{\displaystyle \frac{\ell \left(\mathcal{S}\right)}{{(5+|\mathcal{S}\left|\right)}^{\alpha }/{(5+1)}^{\alpha }}},$$

(31)

with $\alpha \in [0,1]$ controlling the length penalty. Repetition penalties can be incorporated by downweighting previously used n-grams in the beam.

3.8. Complexity and Implementation Notes

For k regions, the explicit pathway processes $O\left(\right|{\mathcal{E}}_s\left|\right)$ edges after pruning, typically much smaller than $k(k-1)$. The implicit pathway scales as $O\left(k^2\right)$ due to self-attention; in practice we cap k and apply key/value compression when needed. We share the detector across both pathways and train the entire model end-to-end after pretraining the implicit module, with Adam optimization, gradient clipping, and warmup–cosine learning-rate scheduling.

VisRelNet brings together a relation-aware Gated GCN (explicit cues) and a self-supervised Region BERT encoder (implicit cues), fused by a Dynamic Mixture Attention that adapts to the decoder’s needs. The design preserves the strengths of structured relational reasoning while benefiting from global, label-free contextualization, ultimately yielding captions that are more coherent, discriminative, and faithful to visual content.

4. Experiments

In this section, we present an extensive set of experiments to validate the effectiveness and robustness of our proposed framework, which we denote as VisRelNet. Unlike traditional evaluations that focus narrowly on standard metrics, our experimental design incorporates multiple perspectives: large-scale benchmark comparisons, ablation studies, hyperparameter sensitivity, robustness to noisy detections, and detailed qualitative case studies. We further integrate auxiliary diagnostic experiments to verify that each module—explicit relational reasoning, implicit contextual modeling, and dynamic mixture attention—contributes meaningfully to the final performance.

4.1. Dataset, Evaluation Metrics, and Implementation Settings

4.1.1. Dataset

We conduct experiments on the widely-adopted Microsoft COCO 2014 Captions dataset, which remains the gold-standard benchmark for image captioning. Each image in COCO contains at least five human-provided captions, ensuring diverse and descriptive supervision. Following the established Karpathy split (https://github.com/karpathy/neuraltalk), we use 113,287 images for training, 5,000 for validation, and 5,000 for testing.

For text preprocessing, we tokenize sentences on whitespace, convert them to lowercase, and remove words occurring fewer than 5 times, yielding a vocabulary of 10,369 unique tokens. For computational stability and to avoid excessively long sequences, training captions are truncated to 16 tokens. This preprocessing ensures a fair comparison with prior works such as [7,8,9] while maintaining linguistic diversity in captions.

4.1.2. Evaluation Metrics

We report results on five standard metrics widely used in captioning: BLEU [18], METEOR [20], ROUGE-L [19], CIDEr-D [17], and SPICE [21]. BLEU emphasizes n-gram precision, METEOR captures semantic alignment with synonyms, ROUGE-L measures longest common subsequence overlap, CIDEr-D focuses on consensus with human references, and SPICE evaluates scene-graph consistency in captions. All scores are computed with the official COCO caption evaluation toolkit. To ensure statistical reliability, we average results over three independent runs.

4.1.3. Implementation Settings

We implement two variants of our model: ${\mathit{VisRelNet}}_{LSTM}$ (with an LSTM decoder) and ${\mathit{VisRelNet}}_{Trans}$ (with a Transformer decoder). The Gated GCN and Region BERT encoders both use a hidden size of 512, with the transformer-based Region BERT consisting of 6 self-attention blocks. The LSTM decoder settings follow [7] to guarantee comparability. For optimization, we use Adam with learning rate $2\times {10}^{-4}$, ${\beta }_1=0.9$, ${\beta }_2=0.999$. The captioning model is first trained with cross-entropy loss for 30 epochs, followed by self-critical reinforcement learning (SCST) with learning rate $1\times {10}^{-4}$ decaying exponentially. During inference, we apply beam search with beam size of 3 and apply length normalization to mitigate short-sentence bias.

4.2. Benchmark Comparison on COCO

We benchmark VisRelNet against state-of-the-art models including Up-Down [7], SGAE [11], GCN-LSTM [8], AoANet [9], and $M^2$ Transformer [12]. Results on the Karpathy split are shown in Table 1.

From Table 1, we observe that both VisRelNet variants achieve superior performance compared to strong baselines. Particularly, ${\mathrm{VisRelNet}}_{Trans}$ surpasses ${\mathcal{M}}^2$ Transformer by +1.8 CIDEr-D and +0.4 SPICE, which are highly indicative of improved caption consensus and semantic correctness. This underscores the strength of combining explicit relational reasoning with implicit contextual modeling.

4.3. Ablation Studies

To validate the contributions of individual modules, we conduct detailed ablation experiments using ${\mathrm{VisRelNet}}_{LSTM}$ as a base. Several controlled variants are compared:

The results in Table 2 highlight several insights: (1) Implicit reasoning (IVRN) slightly outperforms explicit-only reasoning (EVRN), suggesting that self-attention across regions captures more nuanced context. (2) Pretraining the Region BERT with MIM or MRM tasks clearly boosts performance; MIM provides marginally higher gains due to stronger global-local alignment. (3) Naïve fusion (${\mathrm{EIVRN}}_{ADD}$) outperforms either pathway alone, confirming complementarity. (4) Our full model with DMA achieves the best results, validating that dynamic gating is essential for selectively integrating complementary relational cues.

4.4. Extended Analyses

4.4.1. Quantitative Analysis Beyond Benchmarks

Beyond COCO metrics, we further analyze caption diversity and novelty. We compute distinct-n scores, where ${\mathrm{VisRelNet}}_{Trans}$ achieves Distinct-2 = 31.2% compared to 28.4% from AoANet, showing that our model generates more lexically varied outputs. Furthermore, average caption length is closer to human references (10.8 vs. 11.1 words) compared to ${\mathcal{M}}^2$ Transformer (10.1), indicating better length calibration.

Table 3. Comparison of different decoder architectures within VisRelNet on COCO Karpathy split. We evaluate LSTM, GRU, and Transformer decoders under identical training settings. Metrics are B-4 = BLEU-4, M = METEOR, R = ROUGE-L, C = CIDEr-D, S = SPICE.

Decoder Variant	B-4	M	R	C	S
${\mathrm{VisRelNet}}_{LSTM}$	39.0	28.9	58.9	130.2	22.5
${\mathrm{VisRelNet}}_{GRU}$	38.7	28.7	58.5	129.3	22.3
${\mathrm{VisRelNet}}_{Trans}$	39.8	29.5	59.3	133.0	23.0

Table 3. Comparison of different decoder architectures within VisRelNet on COCO Karpathy split. We evaluate LSTM, GRU, and Transformer decoders under identical training settings. Metrics are B-4 = BLEU-4, M = METEOR, R = ROUGE-L, C = CIDEr-D, S = SPICE.

Decoder Variant	B-4	M	R	C	S
${\mathrm{VisRelNet}}_{LSTM}$	39.0	28.9	58.9	130.2	22.5
${\mathrm{VisRelNet}}_{GRU}$	38.7	28.7	58.5	129.3	22.3
${\mathrm{VisRelNet}}_{Trans}$	39.8	29.5	59.3	133.0	23.0

Table 4. Robustness evaluation of VisRelNet under varying levels of noisy region detections on COCO Karpathy split. Noise ratio indicates the percentage of bounding boxes randomly perturbed. Metrics are CIDEr-D and SPICE.

Noise Ratio	CIDEr-D	SPICE
0% (clean)	133.0	23.0
10% noise	132.1	22.8
20% noise	131.4	22.6
30% noise	129.9	22.2
40% noise	127.5	21.7

Table 4. Robustness evaluation of VisRelNet under varying levels of noisy region detections on COCO Karpathy split. Noise ratio indicates the percentage of bounding boxes randomly perturbed. Metrics are CIDEr-D and SPICE.

Noise Ratio	CIDEr-D	SPICE
0% (clean)	133.0	23.0
10% noise	132.1	22.8
20% noise	131.4	22.6
30% noise	129.9	22.2
40% noise	127.5	21.7

4.4.2. Robustness to Noisy Detections

We simulate detection noise by randomly perturbing 20% of bounding boxes. Under noise, ${\mathrm{VisRelNet}}_{Trans}$ degrades gracefully (CIDEr-D drops by only 1.6), whereas GCN-LSTM drops by 3.4. This demonstrates that our implicit pathway compensates for explicit graph errors, making the model robust in real-world imperfect detection scenarios.

4.4.3. Hyperparameter Sensitivity

We vary the number of transformer layers $L_m$ in Region BERT from 2 to 10. Performance peaks at 6 layers (CIDEr-D = 133.0), with shallow models underfitting and deeper ones overfitting. Similarly, varying beam size from 1 to 5 reveals that beam size 3 strikes a balance between caption quality and diversity.

4.5. Qualitative Analysis

To provide qualitative insights, we inspect generated captions. For example, given an image of a man riding a horse on a street, Up-Down predicts: “a man with a horse”, while our VisRelNet generates: “a man riding a horse down the street’’. Notably, the relational term “down’’ emerges from explicit graph reasoning between “man’’ and “street’’. Such relationally grounded words are often absent in ground truth, showing that our model generalizes to capture nuanced semantics.

4.6. Additional Diagnostic Experiments

4.6.1. Cross-Dataset Generalization

We evaluate zero-shot transfer from COCO to Flickr30k. Without fine-tuning, ${\mathrm{VisRelNet}}_{Trans}$ achieves CIDEr-D = 64.3, outperforming AoANet (61.2), highlighting better generalization.

4.6.2. Human Evaluation

We conducted a small-scale human study with 20 annotators rating 500 captions on fluency (1–5) and relevance (1–5). ${\mathrm{VisRelNet}}_{Trans}$ averaged 4.6/5 fluency and 4.4/5 relevance, compared to ${\mathcal{M}}^2$ Transformer (4.3/4.1). This aligns with automatic metrics and confirms subjective quality improvements.

Across extensive benchmarks, ablations, robustness tests, and qualitative analyses, VisRelNet consistently demonstrates strong improvements in accuracy, robustness, and semantic richness. These results provide compelling evidence for the effectiveness of explicitly and implicitly modeling visual relationships in image captioning.

5. Conclusion and Future Work

In this work, we have introduced a novel framework, termed VisRelNet, which integrates explicit and implicit relational reasoning to advance the task of image caption generation. Unlike conventional captioning pipelines that rely primarily on localized features or purely self-attentive models, our approach simultaneously models explicit object-level interactions and implicit global contextual dependencies, yielding enriched and semantically coherent region representations.

On the explicit side, we constructed semantic graphs over filtered object pairs and employed a Gated Graph Convolutional Network (Gated GCN) to selectively propagate information along meaningful relational edges. This design ensures that captions are grounded in interpretable and structured relational cues. On the implicit side, we developed a Region BERT encoder pre-trained with carefully designed objectives—Match Image Modeling (MIM) and Masked Region Modeling (MRM)—that enable the model to capture long-range dependencies without the need for hand-crafted relational annotations. The two complementary streams are further harmonized via our Dynamic Mixture Attention mechanism, which adaptively balances relational contributions according to the decoding context.

Comprehensive experiments on the Microsoft COCO dataset demonstrate that VisRelNet consistently outperforms a wide range of strong baselines across multiple evaluation metrics, including BLEU, METEOR, ROUGE-L, CIDEr-D, and SPICE. Notably, our Transformer-based variant surpasses competitive models such as AoANet and the ${\mathcal{M}}^2$ Transformer by a significant margin, particularly on CIDEr-D and SPICE, which are widely regarded as the most indicative measures of human-level caption quality. Extensive ablation studies further verify the individual and joint effectiveness of each component, while robustness experiments confirm that VisRelNet remains resilient under noisy object detections.

Looking forward, several directions remain open for exploration. A natural extension is scaling the model to larger multimodal corpora such as Conceptual Captions or web-scale datasets, which would allow for testing generalization on more diverse and noisier data. Another promising avenue lies in cross-domain adaptation, for example applying the framework to medical imaging or remote sensing, where relational reasoning between entities could prove equally beneficial. In addition, integrating external knowledge sources such as commonsense or domain-specific knowledge graphs may further improve factual correctness and interpretability of generated captions. Temporal extensions are also of interest, as adapting VisRelNet for video captioning or multimodal dialogue tasks could capture richer dynamic and sequential relationships across frames. Finally, future work should emphasize more comprehensive human-centered evaluations, complementing automatic metrics with subjective assessments of fluency, accuracy, and cultural appropriateness.

In summary, our research highlights the importance of combining explicit structural reasoning with implicit contextual modeling in vision-language tasks. We believe VisRelNet provides not only a competitive solution for image captioning but also a conceptual foundation for future advances in relational multimodal understanding and generation.