Enhancing Caption Fidelity via Explanation-Guided Captioning with Vision-Language Fine-Tuning

1. Introduction

Image captioning—the task of automatically generating natural language descriptions from visual data—has garnered increasing attention in the field of vision-language understanding. The standard modeling paradigm typically comprises a convolutional neural network (CNN) as an image encoder and a recurrent neural network (RNN), often an LSTM or Transformer, as a text decoder [1,2,3]. A key advancement in this area is the introduction of attention mechanisms, which allow the decoder to selectively focus on relevant regions of the image when generating each word [4,5,8,11,12,15].

While attention heatmaps are often used as visual explanations of the model’s focus, they primarily reflect a heuristic alignment between image regions and textual tokens. However, such attention maps fall short in explicitly distinguishing the respective contributions of the image input and the sequential textual history. In particular, they do not provide clarity on whether the model is actually using the image evidence or merely relying on learned language priors—a common issue in captioning tasks.

To address these limitations, we adapt several explanation techniques from the interpretability literature—specifically, Layer-wise Relevance Propagation (LRP) [23,24], Grad-CAM, and Guided Grad-CAM [21,22]—to the image captioning domain. These methods enable high-resolution attribution across both visual and textual modalities. In particular, LRP provides signed relevance maps, indicating which features contribute positively or negatively to the model’s decisions, for both image pixels and prior generated tokens.

Through qualitative and quantitative analyses, we demonstrate that these methods offer a more faithful and granular decomposition of model decisions compared to conventional attention maps. Moreover, they uncover previously hidden model behaviors—such as over-reliance on frequent textual patterns or incorrect associations between objects and words—that are often responsible for hallucinated outputs. This is aligned with prior observations on hallucination issues in captioning systems [27,29,31].

Recognizing the power of explanation-based feedback, we design a novel fine-tuning strategy—CAPEV—which uses LRP-derived relevance scores to modulate the context representation during inference-time learning. Unlike traditional fine-tuning approaches that adjust model weights via backpropagation with global gradients, CAPEV focuses on local relevance signals. Specifically, features with high positive relevance are amplified, while those with negative contributions are suppressed, enabling the model to align its output more precisely with the true visual evidence. Our proposed modulation mechanism is lightweight, plug-and-play, and does not require external supervision or additional annotations—unlike prior approaches that rely on curated segmentation maps or human relevance annotations [28,29].

Importantly, CAPEV also addresses the phenomenon of "semantic shortcutting", where models predict plausible but visually unsupported tokens due to dataset biases. By emphasizing grounded evidence, our method enhances both interpretability and accuracy. Experiments on MSCOCO and Flickr30K validate the effectiveness of CAPEV in improving mean average precision (mAP) of object-centric predictions while maintaining BLEU, METEOR, ROUGE, CIDEr, and SPICE scores at competitive levels [16,17,19].

To summarize, our key contributions include:

• We adapt and extend explanation methods to generate multimodal relevance for image captioning models, revealing detailed attribution over both image and language inputs.
• We perform a rigorous quantitative analysis of these explanations in terms of grounding accuracy, interpretability, and their capacity to expose hallucinations and misaligned predictions.
• We propose CAPEV, a relevance-guided fine-tuning framework that reduces hallucinated object descriptions without compromising caption fluency or requiring extra supervision.

2. Related Work

2.1. Advances in Image Captioning Architectures

Image captioning has long been a benchmark task for evaluating cross-modal understanding between vision and language. The conventional modeling paradigm follows the encoder-decoder architecture, where a convolutional neural network (CNN) is utilized to extract dense visual features and a recurrent neural network (RNN), typically an LSTM or GRU, is employed to decode these features into coherent natural language sequences [1,2,3]. While this pipeline captures the basic semantics of the visual scene, it struggles with fine-grained object interactions and contextual reasoning.

To alleviate this limitation, attention mechanisms have been widely adopted to selectively focus on informative regions of the image while generating each word in the caption. Early variants such as soft and hard attention models [4] laid the groundwork for later attention-based advancements. Semantic attention [6], adaptive attention [7], bottom-up and top-down attention [8], and hierarchical attention mechanisms [10] introduced refined control over region-to-word mappings.

Recent breakthroughs in sequence modeling led to the rise of Transformer-based attention [11], where self-attention mechanisms facilitate rich cross-token interactions. These include Attention-on-Attention (AoA) networks [12], Entangled Transformer structures [13], and Meshed-Memory Transformers [15]. These architectures incorporate multi-head attention layers that simultaneously attend to multiple semantic aspects across the image and caption tokens.

Despite their success, attention-based models still face challenges in capturing object-level relationships and contextual attributes. To address this, graph-based scene modeling has emerged as a complementary approach. Scene graphs [28,37], attribute-based graphs [38], and hierarchical relational modeling frameworks [39,40] have been proposed to enrich image representations with structured semantics. Visual Relation Graphs (VRG) [41] capture inter-object relationships and spatial dependencies, leading to more context-aware caption generation.

Beyond static object modeling, a line of work focuses on local-global representation fusion. Techniques like visual-linguistic distillation [42], noun-chunk parsing [44], and policy-based gradual representation learning [43] aim to align local details with global context. Pretrained vision-language models such as Unified VLP [45], OSCAR [48], and VIVO [47] have pushed the boundaries further by leveraging external knowledge sources, e.g., image-tag pairs, to build universal multimodal embeddings akin to BERT [46].

Parallel to the mainstream task, several challenging extensions of image captioning have been studied. Notably, novel object captioning (NOC) [50,51,52,53,54,55] targets the out-of-distribution generalization problem by enabling models to describe previously unseen objects. Other directions include controllable style transfer [56], sentiment-aware captioning [57], and human-centered captioning [49], all aimed at enhancing the expressiveness and personalization of generated language.

2.2. Mitigating Bias in Vision-Language Systems

Multimodal models are inherently susceptible to dataset-induced biases due to the co-occurrence patterns in visual and textual modalities. In particular, vision-language models often exploit superficial correlations or dominant priors during inference, leading to biased or hallucinated outputs. This issue has been widely studied in both the visual question answering (VQA) and image captioning domains.

In the VQA domain, RUBi [30] proposes to down-weight biased language features during training to force models to rely more on visual signals. Similar efforts such as [58] disentangle question types to better isolate language-induced biases and introduce visual grounding constraints.

In the context of image captioning, gender bias and object priors are among the most prominent problems. Hendricks et al. [29] introduced appearance confusion and confidence loss terms to reduce gender stereotypes, using segmentation-based annotations to supervise the de-biasing process. HINT [31] adopts human-annotated attention maps to realign model predictions with visually grounded regions, thus guiding the model to "look before it talks."

However, these methods typically require costly external annotations, such as segmentation maps or patch rankings. In contrast, the CAPEV framework introduced in our work leverages internal explanation signals generated by LRP, which inherently reflect the contribution of both visual and textual features without extra supervision. This design allows for a scalable and annotation-free approach to mitigating hallucination caused by language bias.

2.3. Interpretability and Explanation for Captioning Models

As deep neural networks become increasingly opaque, interpretability techniques offer an essential lens through which model decisions can be understood and debugged. In the domain of image captioning, the challenge is further exacerbated by the sequential and multimodal nature of the task, making attribution analysis more intricate than in single-modal tasks like image classification.

Explanation techniques can broadly be categorized into three types: (1) gradient-based methods such as saliency maps [59], Guided Backpropagation [22], Integrated Gradients [60], and Grad-CAM [21]; (2) decomposition-based methods such as Layer-wise Relevance Propagation (LRP) [23], DeepLIFT [62], and Contextual Decomposition [65]; and (3) perturbation- and sampling-based methods such as LIME [66], RISE [69], and Occlusion-based techniques [67].

These methods have been extended to various neural architectures, including CNNs, RNNs, GNNs [72,73], and clustering models [77]. However, few works have attempted to adapt these methods for image captioning models, despite their complex multimodal dependencies. Early efforts such as [78] treated static images as videos to apply temporal explanation tools, while Grad-CAM has been applied in limited non-attention settings [21].

While attention heatmaps are often used as proxy explanations in captioning models, their interpretability has been critically questioned in NLP contexts [79,81]. Attention maps typically highlight spatial regions, but fail to convey signed relevance or disentangle modality-specific contributions. In this work, we instead adopt and extend LRP and gradient-based explanation techniques to generate pixel-level and token-level attribution signals, offering deeper insights into the behavior of captioning models.

2.4. Using Explanations to Guide Training

Recent research has explored the intersection between explainability and learning, showing that explanation signals can be used as auxiliary supervision to guide model optimization. Grad-CAM has been employed to design saliency-aware cross-entropy losses in classification [82], leading to improved robustness and visual grounding.

In image captioning, HINT [31] introduced a loss function that ranks image patches based on human annotations and explanation saliency, helping the model focus on correct visual regions during training. Similarly, Sun et al. [83] used LRP explanations in few-shot learning to identify transferable features and enhance generalization.

Building on this idea, our CAPEV framework leverages LRP-generated relevance signals not just for interpretation, but as actionable feedback to drive inference-time fine-tuning. This approach serves as a bridge between model transparency and practical performance gains, aligning the decision-making process with human-understandable evidence without introducing any external supervision cost.

3. Preliminary to Image Descriptions Captioning

3.1. Notational Framework and Pipeline Overview

We begin by formalizing the key components of typical image captioning systems, which follow an encoder-decoder architecture comprising three essential modules: a visual encoder, a language decoder, and a fusion-based word predictor.

Given an input image, a visual encoder—such as a CNN or a region-based detector like Faster R-CNN—is applied to extract spatial or region-level features denoted by $\mathit{I}\in {\mathbb{R}}^{n_v\times d_v}$, where $n_v$ is the number of regions or spatial locations, and $d_v$ is the dimensionality of the feature vectors. For detectors like Faster R-CNN, $n_v$ corresponds to the number of object proposals; for CNN feature maps, $n_v$ corresponds to the number of flattened spatial patches.

At each decoding step t, a hidden representation ${\mathit{h}}_t$ is computed via an LSTM conditioned on the previous word and a global visual summary. The decoder state evolves as:

$$\begin{array}{cc}\hfill {\mathit{x}}_t& =[{\mathit{E}}_w\left(w_{t-1}\right),{\mathit{I}}_g]\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill {\mathit{h}}_t,{\mathit{m}}_t& =\mathrm{LSTM}({\mathit{x}}_t,{\mathit{h}}_{t-1},{\mathit{m}}_{t-1})\hfill \end{array}$$

(2)

Here, ${\mathit{E}}_w(·)$ denotes the word embedding function, and ${\mathit{I}}_g={\displaystyle \frac{1}{n_v}}{\sum }_{k=1}^{n_v}{\mathit{I}}_{\left(k\right)}$ is a global average-pooled visual descriptor. The LSTM output state ${\mathit{h}}_t$ is further used to generate attention-guided context features ${\mathit{c}}_t$ via an attention module:

$$\begin{array}{cc}\hfill {\mathit{c}}_t& =\mathrm{ATT}({\mathit{h}}_t,\mathit{I})\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill {\mathit{p}}_t& =\mathrm{Predictor}({\mathit{h}}_t,{\mathit{c}}_t)\hfill \end{array}$$

(4)

The resulting score vector ${\mathit{p}}_t$ yields a distribution over vocabulary words at time t. The attention function $\mathrm{ATT}(·)$ and predictor module can be instantiated in various ways, leading to different model families.

3.2. Dynamic Visual-Linguistic Integration via Attention Modules

We investigate two widely adopted attention paradigms: adaptive attention, which leverages a gating sentinel to control visual-textual flow, and multi-head attention, a Transformer-based design that facilitates parallel attention to diverse semantic cues.

3.2.1. Adaptive Attention: Sentinel-Guided Integration

Adaptive attention introduces an auxiliary memory cell ${\mathit{s}}_t$—termed the sentinel vector—that selectively captures textual memory separate from the visual stream. At each time step, the sentinel is updated as:

$$\begin{array}{cc}\hfill {\mathit{s}}_t& =\sigma ({\mathit{W}}_x{\mathit{x}}_t+{\mathit{W}}_h{\mathit{h}}_{t-1})\odot tanh\left({\mathit{m}}_t\right)\hfill \end{array}$$

(5)

where ${\mathit{W}}_x\in {\mathbb{R}}^{d_h\times d_x}$ and ${\mathit{W}}_h\in {\mathbb{R}}^{d_h\times d_h}$ are learned projection matrices. The sigmoid gate modulates memory exposure. Subsequently, attention scores over both visual features and sentinel are computed:

$$\begin{array}{cc}\hfill \mathit{a}& ={\mathit{w}}_a^{\top }tanh(\mathit{I}{\mathit{W}}_I+{\mathit{W}}_g{\mathit{h}}_t)\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill b& ={\mathit{w}}_a^{\top }tanh({\mathit{W}}_s{\mathit{s}}_t+{\mathit{W}}_g{\mathit{h}}_t)\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill {\mathit{\alpha }}_t& =\mathrm{softmax}\left(\mathit{a}\right)\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill {\beta }_t& =\mathrm{softmax}{\left([\mathit{a};b]\right)}_{(n_v+1)}\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill {\mathit{c}}_t& =(1-{\beta }_t)\sum _{k=1}^{n_v}{\alpha }_{t_k}{\mathit{I}}_{\left(k\right)}+{\beta }_t{\mathit{s}}_t\hfill \end{array}$$

(10)

Here, ${\beta }_t$ balances visual versus textual contribution in the final attended vector. We define the adaptive attention operator compactly as:

$${\mathit{c}}_t={\mathrm{ATT}}_{\mathrm{ada}}({\mathit{h}}_t,{\mathit{s}}_t,\mathit{I})$$

(11)

3.2.2. Multi-Head Attention: Parallel Contextual Projections

In contrast, multi-head attention utilizes multiple linear projections of queries, keys, and values to allow the model to jointly attend to different aspects of the image. The formulation proceeds as:

$$\begin{array}{cc}\hfill \mathit{Q}& ={\mathit{h}}_t,\mathit{K}=\mathit{I}{\mathit{W}}_K,\mathit{V}=\mathit{I}{\mathit{W}}_V\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill {\mathit{\alpha }}^{\left(i\right)}& =\mathrm{softmax}\left({\displaystyle \frac{{\mathit{Q}}^{\left(i\right)}{\mathit{K}}^{\left(i\right)\top }}{\sqrt{d_h/n_h}}}\right)\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill {\mathit{v}}^{\left(i\right)}& =\sum _{k=1}^{n_v}{\alpha }_k^{\left(i\right)}{\mathit{V}}_k^{\left(i\right)}\hfill \end{array}$$

(14)

The outputs from each attention head are concatenated and transformed:

$$\begin{array}{cc}\hfill \mathit{v}& =[{\mathit{v}}^{\left(1\right)},\dots ,{\mathit{v}}^{\left(n_h\right)}]\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill \widehat{\mathit{v}}& ={\mathit{W}}_v\mathit{v}+{\mathit{b}}_v\hfill \end{array}$$

(16)

To model visual contribution uncertainty, we apply a gating mechanism to form the final context:

$${\mathit{c}}_t=\sigma ({\mathit{W}}_{mh}{\mathit{h}}_t+{\mathit{b}}_{mh})\odot \widehat{\mathit{v}}={\mathrm{ATT}}_{\mathrm{mha}}({\mathit{h}}_t,\mathit{I})$$

(17)

3.3. Unified Model Architectures for Evaluation

To provide a controlled analysis, we instantiate two representative captioning models:

• Ada-LSTM: Incorporates the adaptive attention module alongside an LSTM decoder and a fully connected prediction head.
• MH-FC: Uses Transformer-style multi-head attention with a feedforward predictor directly on attention output.

These models reflect the design patterns of prior works such as [7,8,12,43,44], making them representative baselines.

3.4. Training Objectives and Optimization Strategies

In the initial training phase, models are typically optimized via cross-entropy loss:

$${\mathcal{L}}_{\mathrm{ce}}=-\sum _{t=1}^{l}logp(w_t^{*}\mid w_{<t},\mathit{I})$$

(18)

where $w_t^{*}$ is the ground-truth word at position t, and $p(·)$ is the output distribution from the predictor.

To improve alignment with non-differentiable metrics such as CIDEr, a second-phase reinforcement learning strategy is often applied via Self-Critical Sequence Training (SCST) [84]:

$${\mathcal{L}}_{\mathrm{scst}}=-R\sum _{t=1}^{l}logp\left(w_t^s\right)$$

(19)

where $R=\mathrm{CIDEr}(S^s,S^{gt})-\mathrm{CIDEr}(S^{greedy},S^{gt})$ denotes the reward computed from a sampled caption $S^s$ and a greedy-decoded baseline $S^{greedy}$. This reward guides the model to maximize CIDEr alignment:

$$\underset{\theta }{max}{\mathbb{E}}_{S^s\sim p_{\theta }}\left[R\left(S^s\right)\right]$$

(20)

3.5. Extended Modules: Gated Aggregation and Regularized Context Refinement

To further enhance model flexibility and robustness, we optionally explore two modules often adopted in modern captioning systems:

Gated Context Aggregation:

Instead of relying solely on a hard switch (${\beta }_t$) or sigmoid gate, a soft mixture-of-experts strategy can be introduced:

$${\mathit{c}}_t=\sum _{i=1}^M{\gamma }_i{\mathit{c}}_t^{\left(i\right)},{\gamma }_i={\displaystyle \frac{e^{s_i}}{{\sum }_{j=1}^M{e}^{s_j}}}$$

(21)

where each ${\mathit{c}}_t^{\left(i\right)}$ corresponds to a different context pathway (e.g., visual-only, language-only, or hybrid), and ${\gamma }_i$ are learned mixing weights.

Context Regularization:

We introduce a context alignment loss to encourage consistency between visual context and ground-truth word embeddings:

$${\mathcal{L}}_{\mathrm{align}}=\sum _{t=1}^l{\parallel {\mathit{c}}_t-{\mathit{E}}_w\left(w_t^{*}\right)\parallel }_2^2$$

(22)

This auxiliary loss promotes semantic compatibility and stabilizes attention learning in early training epochs.

Figure 1. Overview of the proposed framework.

Figure 1. Overview of the proposed framework.

4. Proposed Methodology

In this section, we present a unified framework for generating explanation signals in image captioning models. We specifically focus on three representative families of explanation methods: Layer-wise Relevance Propagation (LRP) [23], Grad-CAM and its guided variant [21,22]. These methods allow us to produce pixel-wise and token-wise attribution maps, revealing both visual and linguistic evidence contributing to each generated word. All these mechanisms are integrated into our proposed framework, CAPEV (Caption Alignment via Pertinent Explanation-based Verification), which will later be extended for training-time or inference-time adjustments.

4.1. Gradient-Based Explanation: Grad-CAM and Guided Grad-CAM

Grad-CAM and its enhanced version, Guided Grad-CAM (denoted jointly as Grad*), belong to gradient-based saliency attribution methods. Their core principle is to compute the gradient of the output with respect to intermediate activations—in our case, the visual feature tensor $\mathit{I}\in {\mathbb{R}}^{n_v\times d_v}$—to identify salient regions.

Formally, given a predicted score $s_y$ corresponding to the target token $w_T$, we first backpropagate:

$$g\left(\mathit{I}\right)={\displaystyle \frac{\partial s_y}{\partial \mathit{I}}}\in {\mathbb{R}}^{n_v\times d_v}$$

(23)

From $g\left(\mathit{I}\right)$, channel-wise importance weights are computed as spatially averaged gradients:

$${\mathit{w}}_I=\sum _{k=1}^{n_v}g{\left(\mathit{I}\right)}_{\left(k\right)}\in {\mathbb{R}}^{d_v}$$

(24)

These weights are then linearly combined with the feature maps and passed through a ReLU to obtain the coarse class activation map:

$$\mathrm{CAM}=\mathrm{ReLU}\left(\sum _{j=1}^{d_v}{w}_{I_j}·{\mathit{I}}_{(:,j)}\right)\in {\mathbb{R}}^{n_v}$$

(25)

For fine-grained localization, Grad-CAM is often combined with Guided Backpropagation, which preserves local gradient patterns. Let ${\mathit{G}}_{gbp}$ denote the fine-grained gradient map, the final attribution map is:

$${\mathit{A}}_{\mathrm{guided}}={\mathit{G}}_{gbp}\odot {\mathrm{CAM}}_{\mathrm{upsampled}}$$

(26)

In addition, the linguistic relevance is calculated as:

$${\mathit{r}}_{\mathrm{text}}^{\left(t\right)}={\displaystyle \frac{\partial s_y}{\partial {\mathit{E}}_w\left(w_t\right)}}\in {\mathbb{R}}^{d_w}$$

(27)

which quantifies the influence of previous tokens $w_1,\dots ,w_{T-1}$ on the generation of $w_T$.

4.2. Relevance Propagation via LRP

LRP, unlike Grad*, operates by decomposing the prediction score backward through the network using conservation rules. For each neuron j with input neurons i and output:

$$\begin{array}{cc}\hfill z_j& =\sum _i{w}_{ij}{y}_i+b_j\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill {\widehat{z}}_j& =f\left(z_j\right)\hfill \end{array}$$

(29)

LRP redistributes a relevance score $R\left({\widehat{z}}_j\right)$ to the inputs using two canonical rules:

• $ϵ$-Rule:

  $$R_{i\leftarrow j}=R\left({\widehat{z}}_j\right)·{\displaystyle \frac{y_i{w}_{ij}}{z_j+ϵ·\mathrm{sign}\left(z_j\right)}}$$

  (30)
• $\alpha$-Rule:

  $$R_{i\leftarrow j}=R\left({\widehat{z}}_j\right)\left[(1+\alpha ){\displaystyle \frac{{\left(y_i{w}_{ij}\right)}^{+}}{z_j^{+}}}-\alpha {\displaystyle \frac{{\left(y_i{w}_{ij}\right)}^{-}}{z_j^{-}}}\right]$$

  (31)

  with ${(·)}^{+}=max(·,0)$, ${(·)}^{-}=min(·,0)$. These rules ensure that the decomposition is conservative and interpretable.
  Relevance is propagated through the network by recursively applying:

  $$R\left(y_i\right)=\sum _j{R}_{i\leftarrow j}$$

  (32)

4.3. Adapting LRP to Attention-Guided Captioning Models

Image captioning models typically apply attention-based feature selection mechanisms, which complicate LRP due to their nonlinearity and mixed modality inputs. However, attention operations—being soft weightings—can be treated as linear combinations of features with fixed weights during inference. Following prior practice [89], we assume attention operations are relevance-transparent and redistribute relevance proportionally to attention weights.

Let ${\mathit{c}}_t$ be the context vector at time t:

$${\mathit{c}}_t=\sum _{k=1}^{n_v}{\alpha }_{tk}{\mathit{I}}_{\left(k\right)}\Rightarrow R\left({\mathit{I}}_{\left(k\right)}\right)\propto {\alpha }_{tk}$$

(33)

For adaptive attention, the sentinel vector ${\mathit{s}}_t$ is also integrated:

$$R\left({\mathit{I}}_{\left(k\right)}\right)=(1-{\beta }_t)·{\alpha }_{tk}·R\left({\mathit{c}}_t\right),R\left({\mathit{s}}_t\right)={\beta }_t·R\left({\mathit{c}}_t\right)$$

(34)

4.4. Relevance Tracing in the CAPEV Framework

We now define the full pipeline for propagating explanation in the Ada-LSTM model under CAPEV. Starting from the final prediction $w_T$, we trace the relevance flow back through:

• Final fc layer (logits)
• LSTM layer 2 (language decoder)
• Attention context combination (${\mathit{c}}_t+{\mathit{h}}_t^2$)
• Attention module ${\mathrm{ATT}}_{\mathrm{ada}}$
• LSTM layer 1 (encoder-aware decoder)
• Word embedding layer
• CNN or detection backbone

The full propagation algorithm is detailed, which implements LRP rules at each stage. This process yields three relevance maps:

• $R_{\mathrm{img}}$: Pixel-level image attribution
• $R_{\mathrm{text}}$: Token-level linguistic attribution
• $R_{\mathrm{global}}$: Sentence-level summary score

These outputs form the basis for downstream caption verification and fine-tuning, as described later in our training approach.

4.5. Enhancing Explanation Quality: Regularization and Smoothing

To further improve the interpretability of explanation maps, CAPEV optionally applies two enhancement techniques:

Gaussian Smoothing:

Relevance scores are smoothed over spatial neighborhoods using a 2D Gaussian kernel $\mathcal{G}\left(\sigma \right)$:

$$\tilde{R}(x,y)=\sum _{i,j}R(i,j)·{\mathcal{G}}_{\sigma }(x-i,y-j)$$

(35)

Relevance Normalization:

We normalize relevance maps to sum to 1:

$${\widehat{R}}_{\left(k\right)}={\displaystyle \frac{R_{\left(k\right)}}{{\sum }_k{R}_{\left(k\right)}}}$$

(36)

These refinements produce attribution heatmaps that are more visually coherent and numerically stable across different architectures.

4.6. Interpretability-Guided Control for Inference Adaptation

As a prelude to the CAPEV fine-tuning strategy, we introduce a relevance-weighted residual mechanism to re-inject high-confidence attribution signals into the decoding process. Specifically, we define a relevance-modulated context vector:

$${\tilde{\mathit{c}}}_t={\mathit{c}}_t\odot (1+\lambda ·tanh\left(R\left({\mathit{c}}_t\right)\right))$$

(37)

where $\lambda$ is a hyperparameter controlling the influence of the relevance signal. This idea motivates our later design of explanation-aware fine-tuning in Section 5.

5. Experiments

5.1. Experimental Setup and Protocols

To thoroughly evaluate the effectiveness of our proposed model CAPEV, we conduct extensive experiments on two widely used benchmark datasets for image captioning and caption explanation: MSCOCO and Flickr30K. These datasets consist of diverse real-world images paired with multiple human-annotated captions. We follow standard data preprocessing procedures, including image resizing to $256\times 256$, tokenization using byte-pair encoding, and vocabulary filtering for rare words. We limit the maximum caption length to 20 tokens.

For all our experiments, we utilize two representative backbone captioning architectures as evaluation baselines: (1) a standard attention-based LSTM decoder (i.e., Ada-LSTM), and (2) a Transformer decoder with multi-head attention (i.e., MH-FC). Both architectures are augmented with our CAPEV explanation-enhancement module for inference-time interpretability.

All models are trained using Adam optimizer with a learning rate of $5\times {10}^{-4}$, batch size of 64, and early stopping based on the CIDEr score on the validation set. We also apply scheduled sampling with a decay rate of 0.95 to reduce exposure bias.

5.2. Quantitative Evaluation of Explanation Faithfulness

We first assess the faithfulness of generated explanations to the underlying image captioning model predictions. Following standard practice, we use the Deletion and Insertion metrics as quantitative proxies for explanation reliability. In the Deletion test, we progressively mask pixels in descending order of explanation relevance and measure the drop in predicted word confidence. In the Insertion test, we do the reverse. A steeper drop (for Deletion) and steeper rise (for Insertion) imply more faithful attribution.

We compare CAPEV against baseline explanation methods including Grad-CAM, Guided Grad-CAM, and vanilla LRP. As shown in Table 1, CAPEV consistently outperforms all baselines across both metrics and models, demonstrating its superior capability to identify truly causal visual evidence for caption words.

5.3. Caption Consistency Under Explanation Refinement

In this section, we investigate whether integrating CAPEV-enhanced explanations improves the stability and consistency of caption generation. Specifically, we introduce visual perturbations guided by explanation maps and evaluate the variation in generated captions. For a given image, we mask the top-20% most relevant pixels and observe the change in CIDEr score and BLEU-4 relative to the original caption.

Table 2 reports the average variation. CAPEV yields the smallest drop in CIDEr and BLEU-4 scores, indicating its robustness and alignment with essential visual regions for accurate caption generation.

Figure 2. Results of the word ablation analysis conducted on the MSCOCO2017 test set. For the Ada-LSTM model, 3,710 object words and 11,686 stop words were examined, while the MH-FC model was evaluated on 3,359 object words and 11,512 stop words. A higher average probability drop and greater drop frequency indicate stronger attribution of the word to the model’s predictive confidence.

Figure 2. Results of the word ablation analysis conducted on the MSCOCO2017 test set. For the Ada-LSTM model, 3,710 object words and 11,686 stop words were examined, while the MH-FC model was evaluated on 3,359 object words and 11,512 stop words. A higher average probability drop and greater drop frequency indicate stronger attribution of the word to the model’s predictive confidence.

5.4. Human Evaluation of Interpretability

To further assess the practical utility of CAPEV, we conduct a user study involving 50 participants with AI background. For each sample, participants are shown the input image, predicted caption, and heatmaps from different methods. They are asked to rate the explanation quality (0–5) in terms of clarity, alignment, and justification.

As summarized in Table 3, CAPEV significantly outperforms other methods with a mean rating of 4.38, highlighting its superior visual interpretability and human preference.

5.5. Cross-Model Transferability of Explanations

To investigate the generalization capacity of explanation methods, we perform a cross-model transfer experiment, where relevance maps generated from one captioning model are reused to guide or interpret another distinct model. Specifically, we utilize the CAPEV-generated relevance maps on Ada-LSTM to analyze prediction behaviors on MH-FC, and vice versa. The motivation is to assess whether attribution patterns are model-specific or contain transferable semantic grounding.

We define a transfer consistency score $T_{ij}$ from model i to model j as the cosine similarity between attribution heatmaps after L2 normalization:

$$T_{ij}={\displaystyle \frac{{\sum }_k{R}_k^i·R_k^j}{{\parallel R^i\parallel }_2{\parallel R^j\parallel }_2}}$$

(38)

where $R^i$ is the normalized relevance vector generated by model i on a shared image input. The averaged transfer scores are reported in Table 4.

These results indicate that relevance distributions have a degree of semantic consistency, but are still tailored by model-specific architecture and attention flows. CAPEV enables more transferable attributions compared to vanilla Grad-CAM.

5.6. Granularity Sensitivity on Visual Regions

In this section, we analyze how the resolution of input visual features impacts the quality and granularity of the generated explanations. We experiment with two types of visual input: grid-based CNN features with 49 spatial elements ($7\times 7$) and region-based Faster R-CNN proposals with 36 ROI features.

We define a visual granularity index (VGI) as follows:

$$\mathrm{VGI}=\sum _k\left|\Delta R_k\right|,\Delta R_k=R_k^{\mathrm{high}-\mathrm{res}}-R_k^{\mathrm{low}-\mathrm{res}}$$

(39)

where $R_k$ represents the normalized relevance score for region k.

Our experiments reveal that the region-based input yields higher fidelity in relevance maps for object-centric queries (e.g., “a man playing tennis”), while grid-based features provide smoother gradients over background-sensitive queries (e.g., “on the beach”). Quantitatively, CAPEV’s VGI score is 34.2% higher than Grad-CAM, indicating its improved granularity sensitivity.

5.7. Multilingual Caption Robustness

To evaluate the language robustness of our method, we extend CAPEV explanations to multilingual captioning settings. We use the XM3600 dataset with aligned captions in English, German, and Chinese. We apply CAPEV on multilingual Ada-LSTM models trained on each language.

To quantify alignment of relevance scores across languages, we define a multilingual consistency score (MCS):

$$\mathrm{MCS}={\displaystyle \frac{1}{L(L-1)}}\sum _{i<j}\mathrm{JSD}(R^i\Vert R^j)$$

(40)

where $\mathrm{JSD}(·)$ is Jensen-Shannon divergence between relevance maps, and $L=3$.

Our findings in Table 5 show CAPEV maintains better multilingual attribution stability than baseline methods:

These experiments demonstrate that CAPEV not only enhances explanation interpretability but also ensures cross-lingual robustness, making it suitable for global-scale vision-language applications.

6. Conclusion and Future Perspectives

In this work, we presented CAPEV, a novel framework that leverages Layer-wise Relevance Propagation (LRP) and gradient-based visual attribution techniques to enhance the interpretability and controllability of attention-based image captioning models. Going beyond the limitations of conventional attention heatmaps, CAPEV is designed to generate more faithful, fine-grained, and semantically coherent explanations that bridge visual evidence and linguistic outputs.

Through a comprehensive set of qualitative and quantitative evaluations, we demonstrated that CAPEV explanations provide insightful decompositions of the captioning process. Notably, they enable a finer understanding of how different visual regions and textual priors influence the generation of specific words. Our analysis shows that CAPEV explanations outperform traditional attention maps in terms of clarity, alignment, and faithfulness. Moreover, we highlight that these attribution maps are not only informative but also diagnostic — allowing us to identify potential failure cases, such as hallucinated objects, and trace back the erroneous reasoning steps within the captioning model.

Building upon these interpretability insights, we further proposed a relevance-guided inference-time fine-tuning (RIFT) strategy under the CAPEV framework, which aims to alleviate the well-known object hallucination problem in image captioning without requiring additional supervision or architectural changes. RIFT works by reinforcing the relevance signal from visual inputs during inference and integrating it back into model decisions through residual refinement. Our experiments validate that CAPEV-RIFT significantly reduces hallucination rates while preserving the core semantics of generated descriptions.

Despite these advantages, our ablation studies reveal that CAPEV-RIFT does not consistently improve sentence-level generation metrics such as CIDEr or METEOR. This discrepancy suggests that while hallucination is mitigated, some fine-grained linguistic qualities are not always preserved. To investigate this phenomenon, we performed sample-level analysis and found that CAPEV-RIFT excels especially on examples involving novel or rare object references, where reliance on visual grounding becomes crucial. These findings motivate further exploration of how explanation-enhanced fine-tuning can interact with language fluency objectives.

Inspired by this observation, we posit that CAPEV has significant potential in the realm of Novel Object Captioning (NOC) — a challenging task where models must generate accurate captions for images containing unseen or rare objects. Since standard training data often lacks sufficient coverage of such objects, their recognition and naming rely heavily on auxiliary signals such as object detectors or semantic priors. For instance, prior work such as [52] proposed a dynamic pointing mechanism that selects object mentions from detector outputs based on sentence context. To conclude, CAPEV bridges the gap between interpretability and generative control in vision-language models. By augmenting captioning with relevance-based reasoning, it lays the foundation for more transparent, robust, and accountable AI systems in multimodal understanding.