A Multi-Task Intervention Framework for Semantically Faithful Image Captioning

1. Introduction

Image captioning serves as a crucial bridge between visual perception and linguistic abstraction, enabling machines to produce human-like descriptions of complex scenes Chen et al. (2015). Over the past decade, the problem has evolved from early template-based methods to modern deep architectures that combine convolutional and sequential models. A widely adopted paradigm, known as extract-then-generate, first detects salient objects or regions Anderson et al. (2018) and then formulates caption generation as a sequence-to-sequence (seq2seq) problem implemented via LSTM or Transformer decoders Hochreiter and Schmidhuber (1997); Vaswani et al. (2017). This approach has yielded high-quality results across standard benchmarks, benefiting from powerful pre-trained object detectors and large-scale paired datasets.

Despite its success, the paradigm suffers from two persistent deficiencies that fundamentally limit caption quality. First, captions often exhibit content inconsistency, where the generated text contradicts the visual evidence. For example, given an image of a man with long hair, a model may incorrectly describe it as a woman, reflecting biased correlations in the training data rather than genuine understanding. Second, captions are frequently under-informative, tending to replace fine-grained details with generic descriptions—for instance, referring to a “Wii game” simply as a “video game.” While such errors may still yield plausible sentences, they undermine the informativeness and precision desired for downstream tasks such as visual question answering or retrieval.

From a causal perspective, these phenomena arise from spurious associations that the model inadvertently learns. Instead of capturing the causal relationship between scene content and corresponding linguistic expression, the model relies on surface-level statistical regularities. For instance, the visual feature “long hair” becomes spuriously linked with the word “woman” due to a latent confounder, “female,” that is unobserved yet systematically biases the mapping. This bias propagates through the learning process, distorting how the model attributes semantics. Conceptually, such relationships can be formalized through a directed acyclic graph (DAG) where confounders introduce non-causal dependencies between visual features and linguistic tokens. Without intervention, conventional training objectives minimize empirical risk while amplifying these spurious pathways, resulting in biased caption generation.

To fundamentally resolve these issues, we reimagine image captioning as a dependent multi-task learning problem. Our proposed framework, DEPICT, integrates three key innovations.

(1) Intermediate semantic mediation. We introduce a bag-of-categories (BOC) prediction task preceding caption generation. By learning to identify object categories and high-level semantic groups, the model forms a structured representation of the scene. This mediator provides an explicit semantic scaffold that constrains the captioning process, effectively reducing contradictions between visual content and textual output. The BOC supervision is lightweight and can leverage existing annotations from datasets such as MSCOCO Li et al. (2020); Wang et al. (2020).

(2) Causal intervention with proxy confounders. Real-world confounders such as gender, scene bias, or cultural priors are rarely annotated. To address this, DEPICT estimates latent confounders through proxy confounders—a set of high-frequency predicates and fine-grained visual concepts that statistically correlate with these latent variables. Using the principles of Pearl’s do-calculus, we intervene on the confounded causal graph by performing do(X), thereby breaking the undesired dependencies between visual attributes and biased linguistic predictions. The true confounder distribution is approximated via variational inference Kingma and Welling (2014), enabling the model to sample representations that reflect causal, rather than correlational, dependencies.

(3) Multi-agent reinforcement learning optimization. To ensure smooth cooperation between the BOC and captioning sub-tasks, we employ a multi-agent reinforcement learning (MARL) framework. Each sub-task corresponds to an autonomous agent: one focuses on visual categorization and the other on linguistic realization. A shared reward structure encourages mutual adaptation, aligning semantic precision and linguistic fluency. This design mitigates error accumulation across tasks—a common issue in cascaded multi-task systems—and facilitates stable end-to-end optimization.

Through extensive experiments, we demonstrate that DEPICT substantially improves both quantitative metrics (e.g., CIDEr, BLEU, SPICE) and qualitative aspects such as factual accuracy and detail richness. Beyond performance numbers, the model’s causal formulation provides interpretability and robustness: DEPICT consistently avoids misleading gender or object biases, generates nuanced descriptions, and maintains coherence across diverse visual scenes. Our contributions can be summarized as follows:

• We reformulate image captioning as a dependent multi-task problem, introducing intermediate semantic mediation to enhance consistency and informativeness.
• We develop a causal intervention mechanism that leverages proxy confounders and variational inference to remove spurious visual-linguistic dependencies.
• We integrate a multi-agent reinforcement learning framework to jointly optimize sub-tasks, minimizing error propagation and maximizing semantic alignment.
• We empirically validate that DEPICT achieves superior results and stronger interpretability compared with prior captioning architectures.

In summary, DEPICT represents a principled step toward causally faithful image captioning. By uniting causal reasoning, multi-task learning, and reinforcement optimization, it bridges the gap between visual perception and semantic understanding, offering a robust pathway for building interpretable and reliable vision–language systems.

2. Related Work

In this section, we review the recent literature related to image captioning and divide it into several major research directions. Compared with earlier overviews that broadly categorize the field, we emphasize the progressive conceptual shifts that have shaped contemporary captioning models. Specifically, we extend the taxonomy into five detailed subcategories: (1) architectural innovation, (2) representation learning and decoding strategies, (3) knowledge enhancement and external semantics, (4) causal intervention and bias correction, and (5) emerging multimodal foundation trends. The following subsections provide a comprehensive discussion of each category, highlighting how prior works motivate and contrast with our proposed framework.

2.1. Architectural Innovation

Early image captioning models were primarily grounded in encoder–decoder architectures inspired by neural machine translation. The visual encoder, typically a convolutional neural network (CNN), extracted a global image representation that was subsequently fed into a recurrent neural network (RNN) decoder such as long short-term memory (LSTM) Donahue et al. (2015); Vinyals et al. (2015). These models, though simple, struggled to align fine-grained visual features with linguistic tokens, resulting in coarse and sometimes inconsistent captions. Xu et al. Xu et al. (2015) introduced the attention mechanism, marking a paradigm shift by enabling adaptive focus on salient regions at each decoding step. The attention-based methods significantly improved visual grounding and interpretability, serving as a foundation for subsequent advancements.

Later research incorporated object-level information through region detectors Fang et al. (2015); Karpathy and Li (2015), transforming the captioning process into a two-stage pipeline: region extraction and sentence generation. Anderson et al. Anderson et al. (2018) introduced bottom-up and top-down attention, integrating object detection with dynamic language modeling and achieving substantial performance gains. These ideas inspired hybrid architectures that leveraged relational reasoning through self-attention mechanisms over object nodes Huang et al. (2019) or graph convolutional networks (GCNs) Yang et al. (2019); Yao et al. (2018). More recently, Transformer-based architectures Vaswani et al. (2017) have dominated the field with improvements in layer normalization, multi-head contextualization, and positional embeddings Guo et al. (2020). Notable innovations include meshed attention models Cornia et al. (2020) that fuse hierarchical cross-modal dependencies and higher-order attentional modules Pan et al. (2020) that better capture inter-object relations.

Another crucial component is the reinforcement learning-based optimization paradigm. The self-critical reinforcement learning (SCRL) framework Rennie et al. (2017) has become a near-universal training strategy to bridge the gap between maximum likelihood estimation (MLE) objectives and non-differentiable evaluation metrics like CIDEr and SPICE. By directly optimizing task-specific rewards, these methods have produced captions that better align with human judgment. Beyond SCRL, newer works explore actor–critic formulations and entropy-regularized policy optimization to stabilize gradient updates, further improving diversity and semantic coverage.

2.2. Representation Learning and Decoding Strategies

Beyond architectural design, significant efforts have been directed toward improving the learned representations and decoding dynamics. Several studies proposed leveraging spatial and semantic embeddings to enhance context modeling. For instance, some variants introduced relational graph modules that learn structured interactions between detected objects and contextual cues. Others designed hierarchical decoders that first predict coarse scene semantics before generating word-level descriptions, providing multi-level linguistic granularity.

Moreover, multimodal fusion strategies evolved from simple concatenation to sophisticated cross-attention mechanisms, where visual and textual tokens mutually guide representation refinement. Dual-stream and memory-augmented networks have been proposed to address long-range dependencies in both modalities. Some recent works even integrated masked language modeling objectives or contrastive pre-training schemes, aligning visual and textual embeddings through large-scale self-supervised data. These hybrid decoding frameworks paved the way for scalable captioning systems capable of transferring to unseen domains with minimal supervision.

2.3. Knowledge Enhancement and Semantic Priors

The integration of external knowledge has emerged as a powerful approach to enrich semantic understanding. Kim et al. Kim et al. (2019) introduced a relationship-aware model that leverages part-of-speech (POS) tagging, allowing the caption generator to explicitly encode syntactic roles such as subject, predicate, and object. This design introduces inductive linguistic bias and enhances syntactic fluency. Building upon this, Shi et al. Shi et al. (2020) exploited textual scene graphs extracted from captions using the parser proposed by Schuster et al. Schuster et al. (2015). By representing captions as relational triples—(subject, predicate, object)—these methods incorporate graph-structured semantics that guide visual reasoning and facilitate better compositional generalization.

Subsequent works have extended knowledge augmentation to include external commonsense knowledge bases, such as ConceptNet and Visual Genome. These systems align visual entities with their conceptual properties and infer implicit relationships (e.g., “a man holds a bat” → “the man is playing baseball”). Moreover, embedding-level knowledge distillation and attribute prediction networks have been adopted to pre-train models on large-scale textual corpora, transferring linguistic coherence to visual-language tasks. Such approaches effectively bridge the symbolic–neural gap, enabling more contextually grounded and interpretable captioning.

2.4. Causal Intervention and Bias Correction

A growing body of literature has recognized that conventional captioning models often encode unintended dataset biases and spurious correlations. To address this, causal reasoning frameworks have been introduced. Wang et al. Wang et al. (2020) first applied causal intervention to visual recognition, learning unbiased region representations via the do-calculus. Specifically, when modeling the dependency between two objects, such as “toilet” and “person,” they treated all other co-occurring objects as confounders and optimized for the interventional probability $p(\mathrm{toilet}\mid \mathrm{do}(\mathrm{person}\left)\right)$, effectively removing confounding influences. Similarly, Yang et al. proposed viewing dataset biases as confounders and reparameterizing concept representations (e.g., “apple,” “green”) to adjust their causal effects.

In addition to direct intervention, other studies have introduced counterfactual reasoning frameworks, generating hypothetical visual scenarios to assess the sensitivity of model predictions. By comparing outputs under factual and counterfactual contexts, these methods identify spurious dependencies and regularize models toward invariant representations. Causal debiasing has also been explored at the feature level, where latent confounders are disentangled from causal variables using variational inference and mutual information minimization. Collectively, these efforts emphasize that robust captioning systems should reason about cause–effect relations rather than mere statistical co-occurrence.

2.5. Emerging Multimodal Foundation Trends

Recently, the evolution of large-scale multimodal foundation models has profoundly impacted image captioning research. Pre-trained models such as CLIP, BLIP, and Flamingo serve as generic vision–language encoders that provide strong priors for downstream generation. By fine-tuning these backbones or incorporating adapter modules, captioning models can leverage vast cross-domain knowledge with minimal supervision. Moreover, instruction-tuned multimodal large language models (MLLMs) further push the boundary, enabling open-ended captioning that aligns with human conversational intent rather than fixed datasets.

Parallel developments in causal and self-supervised learning paradigms have started converging: models now integrate counterfactual data augmentation, interventional fine-tuning, and human feedback alignment to achieve causal interpretability at scale. The convergence of causal reasoning and foundation model adaptation opens promising directions for future captioning frameworks, providing a theoretical bridge between symbolic reasoning and neural representation learning.

In summary, prior studies have contributed along multiple complementary dimensions—architectural advances for stronger visual–linguistic alignment, knowledge integration for semantic depth, and causal modeling for bias mitigation. However, despite these achievements, most existing frameworks treat these aspects independently. Our proposed framework, DEPICT, builds upon this landscape by unifying these dimensions under a single causal multi-task paradigm. It explicitly disentangles spurious dependencies, leverages intermediate semantics, and utilizes interventional learning to generate captions that are both faithful and causally grounded.

Figure 1. Overview of the proposed interventional captioning framework. The system begins with a self-attentional visual encoder that extracts region features from the input image. A mediator head predicts bag-of-category semantics, while a proxy VAE models latent confounders to mitigate spurious correlations. These representations jointly inform an interventional caption generator that decodes proxy-invariant textual descriptions. Two reinforcement learning agents collaboratively optimize the mediator and captioner through alternating self-critical updates under a mixed MLE–RL training scheme, producing interpretable and causally disentangled captions.

Figure 1. Overview of the proposed interventional captioning framework. The system begins with a self-attentional visual encoder that extracts region features from the input image. A mediator head predicts bag-of-category semantics, while a proxy VAE models latent confounders to mitigate spurious correlations. These representations jointly inform an interventional caption generator that decodes proxy-invariant textual descriptions. Two reinforcement learning agents collaboratively optimize the mediator and captioner through alternating self-critical updates under a mixed MLE–RL training scheme, producing interpretable and causally disentangled captions.

3. Methodology

In this section, we cast image captioning as a dependent multi-task problem with explicit semantic mediation and principled de-confounding. We first formalize the learning objective (Section 3.2) and then reinterpret it through the lens of mediation and intervention (Section 3.3). Next, we introduce a latent de-confounding mechanism grounded in proxy variables and variational inference (Section 3.4). We subsequently present the parameterization of our captioner—renamed throughout as DEPICT (DEpendent multi-task framework with Proxy-confounder Intervened Captioning Transformer)—including encoders and generators (Section 3.5). Finally, we describe a cooperative optimization scheme based on multi-agent reinforcement learning (Section 3.6). In addition to the core pipeline, we enrich the method with further implementation details, regularization strategies, and training protocols to improve robustness and stability.

3.1. Problem Setup and Notation

Let $(\mathit{x},\mathit{y})$ denote an image and its ground-truth caption. A pre-trained Faster R-CNN detector Anderson et al. (2018) extracts a set of region features $\mathit{R}\in {\mathbb{R}}^{L_r\times d}$, with $L_r$ object proposals and d-dimensional descriptors. We adopt a mediator $\mathit{m}$—a bag-of-categories (BOC) representation—that summarizes salient semantics useful for captioning. We also consider a proxy concept set $\mathit{c}$ (e.g., frequent predicates and fine-grained object categories) and a latent confounder ${\mathit{z}}_c$ which captures biasing factors not annotated in the dataset. Throughout, bold lowercase letters denote vectors, bold uppercase letters denote matrices/sequences, and calligraphic fonts denote losses or sets. Unless stated otherwise, expectations are over the indicated sampling distributions.

3.2. Dependent Multi-Task Learning

Conventional captioners directly maximize $p(\mathit{y}\mid \mathit{R})$. Instead, we introduce a dependent multi-task (DMT) factorization that exposes a mediator:

$$logp(\mathit{y},\mathit{m}\mid \mathit{R})=logp(\mathit{m}\mid \mathit{R})+logp(\mathit{y}\mid \mathit{R},\mathit{m}).$$

(1)

Here $\mathit{m}$ is instantiated as BOC labels that are natively available in MSCOCO and widely used in recent work Li et al. (2020); Wang et al. (2020). Given detector features $\mathit{R}$, we first predict $\mathit{m}$ and then condition caption generation on both $\mathit{R}$ and $\mathit{m}$. This setting explicitly couples the tasks in a way that encourages semantically grounded decoding.

Stochastic Mediation for Train–Test Alignment.

Because $\mathit{m}$ is not observed at inference, we adopt a stochastic surrogate during training to reduce the exposure bias between tasks. Marginalizing $\mathit{m}$ yields

$$\begin{array}{cc}\hfill p(\mathit{y}\mid \mathit{R})& =\sum _{\mathit{m}}p(\mathit{y}\mid \mathit{R},\mathit{m})p(\mathit{m}\mid \mathit{R})\hfill \\ \hfill & ={\mathbb{E}}_{\tilde{\mathit{m}}\sim p(\mathit{m}\mid \mathit{R})}\left[p(\mathit{y}\mid \mathit{R},\tilde{\mathit{m}})\right].\hfill \end{array}$$

(2)

The joint objective then becomes

$$logp(\mathit{m}\mid \mathit{R})+{\mathbb{E}}_{\tilde{\mathit{m}}\sim p(\mathit{m}\mid \mathit{R})}\left[logp(\mathit{y}\mid \mathit{R},\tilde{\mathit{m}})\right].$$

(3)

To allow gradients to flow across tasks under discrete BOC variables, we employ the Gumbel–Softmax estimator Jang et al. (2017). Given class logits $\alpha$ and i.i.d. Gumbel noise $g_k\sim \mathrm{Gumbel}(0,1)$, a differentiable draw $\tilde{\mathit{m}}$ at temperature $\tau$ is

$${\tilde{m}}_k={\displaystyle \frac{exp(({\alpha }_k+g_k)/\tau )}{{\sum }_{j}exp(({\alpha }_j+g_j)/\tau )}}\mathrm{for}\mathrm{each}\mathrm{category}k.$$

We adopt a cosine annealing schedule for $\tau$ to balance exploration and stability (details in Appendix ).

3.3. Mediated Learning Under Intervention

The above factorization admits a causal reading: $\mathit{R}\to \mathit{m}\to \mathit{y}$, where $\mathit{m}$ acts as a mediator that transmits causal signals from vision to language. Under intervention via do-calculus Pearl (2010), when we force the input to an observed value (i.e., $\mathrm{do}(X=\mathit{x})$ with $X\mapsto \mathit{R}$), arrows entering $\mathit{R}$ are severed, while outgoing influences remain intact. In a purely mediated graph (without confounding), interventional and observational conditionals coincide, thus $p(\mathit{y}\mid \mathrm{do}(\mathit{R}\left)\right)=p(\mathit{y}\mid \mathit{R})$—consistent with (2).

3.4. Latent De-Confounding via Proxy Variables

In realistic datasets, spurious correlations arise because unobserved confounders affect both visual features and lexical choices. Let $\mathit{c}$ denote observable proxy concepts and ${\mathit{z}}_c$ denote the latent confounder. Under intervention on $\mathit{R}$, proxies no longer depend on $\mathit{R}$, yielding

$$\begin{array}{cc}\hfill p(\mathit{y}\mid \mathrm{do}(\mathit{R}\left)\right)& =\sum _{\mathit{c}}p(\mathit{y}\mid \mathit{R},\mathit{c})p\left(\mathit{c}\right)\hfill \\ \hfill & ={\mathbb{E}}_{\tilde{\mathit{c}}\sim p\left(\mathit{c}\right)}\left[p(\mathit{y}\mid \mathit{R},\tilde{\mathit{c}})\right],\hfill \end{array}$$

(4)

which differs from $p(\mathit{y}\mid \mathit{R})$ whenever proxies encode bias. Considering both mediator and proxies jointly, the interventional quantity becomes

$$\begin{array}{cc}\hfill p(\mathit{y}\mid \mathrm{do}(\mathit{R}\left)\right)& =\sum _{\mathit{c}}\sum _{\mathit{m}}p(\mathit{y}\mid \mathit{R},\mathit{m},\mathit{c})p(\mathit{m}\mid \mathit{R})p\left(\mathit{c}\right)\hfill \\ \hfill & ={\mathbb{E}}_{\tilde{\mathit{c}},\tilde{\mathit{m}}}\left[p(\mathit{y}\mid \mathit{R},\tilde{\mathit{m}},\tilde{\mathit{c}})\right],\hfill \end{array}$$

(5)

with $\tilde{\mathit{c}}\sim p\left(\mathit{c}\right)$ and $\tilde{\mathit{m}}\sim p(\mathit{m}\mid \mathit{R})$.

Variational Estimation of Latent Confounders.

We posit ${\mathit{z}}_c$ as a continuous latent that generates proxies $\mathit{c}$ and encodes nuisance factors. We estimate ${\mathit{z}}_c$ via a variational posterior $q_{\varphi }({\mathit{z}}_c\mid \mathit{c})$ and a simple prior $p\left({\mathit{z}}_c\right)$ (standard Gaussian), maximizing the ELBO

$$\begin{array}{cc}\hfill log{p}_{\vartheta ,\varphi }\left(\mathit{c}\right)& =log\int p_{\vartheta }(\mathit{c}\mid {\mathit{z}}_c)p\left({\mathit{z}}_c\right)d{\mathit{z}}_c\hfill \\ \hfill & \ge {\mathbb{E}}_{{\mathit{z}}_c\sim q_{\varphi }({\mathit{z}}_c\mid \mathit{c})}\left[log{p}_{\vartheta }(\mathit{c}\mid {\mathit{z}}_c)\right]\hfill \\ & -\mathrm{KL}\left(q_{\varphi }({\mathit{z}}_c\mid \mathit{c})\parallel p\left({\mathit{z}}_c\right)\right),\hfill \end{array}$$

(6)

where $\vartheta$ parameterizes the proxy likelihood and $\varphi$ the encoder. Substituting ${\mathit{z}}_c$ into the interventional objective leads to

$$\begin{array}{cc}\hfill log{p}_{\theta ,\varphi }(\mathit{y}\mid \mathrm{do}\left(\mathit{R}\right))& ={\mathbb{E}}_{\begin{array}{c}\tilde{\mathit{c}}\sim p\left(\mathit{c}\right)\\ {\mathit{z}}_c\sim q_{\varphi }({\mathit{z}}_c\mid \tilde{\mathit{c}})\\ \tilde{\mathit{m}}\sim p_{\phi }(\mathit{m}\mid \mathit{R})\end{array}}\left[log{p}_{\theta }(\mathit{y}\mid \mathit{R},\tilde{\mathit{m}},{\mathit{z}}_c,\tilde{\mathit{c}})\right],\hfill \end{array}$$

(7)

which encourages captions to depend on visual evidence and mediator semantics while rendering them insensitive to spurious proxy-driven shortcuts.

Global Objective.

The overall learning target combines mediator prediction, interventional captioning, and proxy modeling:

$$\underset{\theta ,\varphi ,\vartheta ,\phi }{max}log{p}_{\phi }(\mathit{m}\mid \mathit{R})+log{p}_{\theta ,\varphi }(\mathit{y}\mid \mathrm{do}\left(\mathit{R}\right))+log{p}_{\vartheta ,\varphi }\left(\mathit{c}\right),$$

with the second and third terms instantiated via (7) and (6).

3.5. Parameterization of DEPICT

Hierarchical Visual Encoder.

Given $\mathit{R}$, we employ a n-layer self-attentional encoder to refine region representations:

$${\mathit{R}}^{(i+1)}=\mathrm{Add}\&\mathrm{Norm}\left({\mathrm{Attn}}^{\left(i\right)}\left({\mathit{R}}^{\left(i\right)},{\mathit{R}}^{\left(i\right)},{\mathit{R}}^{\left(i\right)}\right)\right),$$

with ${\mathit{R}}^{\left(0\right)}=\mathit{R}$ and multi-head attention as in Vaswani et al. (2017). Intermediate features ${\mathit{R}}^{\left(i_m\right)}$ drive BOC prediction, while the final layer ${\mathit{R}}^{\left(n\right)}$ conditions captioning.

BOC Generator (Mediator Head).

We formulate mediator prediction as multi-category counting. Let ${\mathit{Y}}^m\in {\mathbb{R}}^{L_m\times N_m}$ be one-hot labels, where $L_m$ is the number of categories and $N_m$ the maximal count per category. Category-specific queries ${\mathit{Q}}^c\in {\mathbb{R}}^{L_m\times d}$ attend to ${\mathit{R}}^{\left(i_m\right)}$:

$${\mathit{H}}^c=\mathrm{Attn}\left({\mathit{Q}}^c,{\mathit{R}}^{\left(i_m\right)},{\mathit{R}}^{\left(i_m\right)}\right),$$

followed by a category-wise MLP and softmax to produce ${\tilde{\mathit{Y}}}^m$. Cross-entropy implements $log{p}_{\phi }(\mathit{m}\mid \mathit{R})$:

$${\mathcal{L}}^m=-\sum _{j=1}^{L_m}{\mathit{Y}}_j^{m}log{\tilde{\mathit{y}}}_j^m.$$

In addition, we apply an entropy regularizer $\mathcal{H}\left({\tilde{\mathit{Y}}}^m\right)$ to avoid overconfident BOC estimates at early epochs:

$${\mathcal{L}}_m^{\mathrm{ent}}=-\gamma \sum _j\sum _{u=1}^{N_m}{\tilde{y}}_{j,u}^{m}log{\tilde{y}}_{j,u}^m.$$

Caption Generator (Interventional Head).

We embed category names and counts to form ${\mathit{E}}^m={\left\{{\mathit{e}}_j^m\right\}}_{j=1}^{L_m}$:

$${\mathit{h}}_{j,t}^m=\mathrm{LSTM}\left(\mathrm{OH}\left(w_{j,t}^m\right){\mathit{W}}_w,{\mathit{h}}_{j,t-1}^m\right),$$

$${\mathit{e}}_j^m=\left({\mathit{h}}_{j,T_m}^m+\mathrm{OH}\left(i{d}_j^m\right){\mathit{W}}_o\right)\odot \sigma \left(\mathrm{OH}\left(m_j\right){\mathit{W}}_m\right),$$

max-pooled to ${\mathit{z}}_m={max}_j{\mathit{e}}_j^m$.

For proxies, we embed $\mathit{c}$ to ${\mathit{E}}^c\in {\mathbb{R}}^{L_c\times d}$, process with a 2-layer Transformer without positional encodings to respect set structure, mean-pool to ${\overline{\mathit{h}}}^z$, then map to $(\mu ,log{\sigma }^2)$ via two MLPs. The latent confounder is sampled by reparameterization

$${\mathit{z}}_c=\mu +\sigma \odot ϵ,ϵ\sim \mathcal{N}(\mathit{0},\mathit{I}),$$

and fed into the decoder alongside ${\mathit{z}}_m$ and ${\mathit{R}}^{\left(n\right)}$. A top-down attention LSTM Anderson et al. (2018) generates tokens with teacher forcing during MLE and with sampling during RL:

$${\mathcal{L}}^g=-\sum _{t=1}^{T}log{p}_{\theta }\left(y_t\mid {\mathit{y}}_{<t},{\mathit{R}}^{\left(n\right)},{\mathit{z}}_m,{\mathit{z}}_c\right).$$

To further encourage invariance to proxies, we add a decorrelation penalty between ${\mathit{z}}_c$ and ${\mathit{R}}^{\left(n\right)}$:

$${\mathcal{L}}^{\mathrm{dec}}=\eta \parallel \mathrm{Cov}({\mathit{z}}_c,\mathrm{Proj}\left({\mathit{R}}^{\left(n\right)}\right)){\parallel }_F^2,$$

where $\mathrm{Proj}$ is a linear map and ${\parallel ·\parallel }_F$ is the Frobenius norm.

Total MLE Objective.

During teacher-forced training, the aggregate loss is

$${\mathcal{L}}_{\mathrm{MLE}}={\mathcal{L}}^g+{\mathcal{L}}^m+{\mathcal{L}}_m^{\mathrm{ent}}+\beta \mathrm{KL}\left(q_{\varphi }({\mathit{z}}_c\mid \mathit{c})\parallel p\left({\mathit{z}}_c\right)\right)+{\mathcal{L}}^{\mathrm{dec}},$$

with a monotone schedule for $\beta$ (annealed KL) to stabilize early optimization.

Algorithm 1: Training DEPICT: Dependent Multi-Task Captioning with Causal Intervention

3.6. Optimization via Multi-Agent Reinforcement Learning

Standard RL-based captioners reduce train–test mismatch by directly optimizing sequence-level rewards Ranzato et al. (2016). In our dependent setting, we must also address inter-task mismatch: errors in BOC propagate to captions. We therefore adopt a multi-agent self-critical RL scheme Rennie et al. (2017) in which the mediator and captioner are separate agents trained with alternating updates and shared signals.

Agent 1: Mediator (BOC) Policy.

We sample $\tilde{\mathit{m}}\sim p_{\phi }(\mathit{m}\mid \mathit{R})$ (Monte Carlo) and form a greedy baseline $\overline{\mathit{m}}$. The policy gradient is

$${\nabla }_{\phi }\mathcal{L}\left(\phi \right)\approx -\left(r_1\left(\tilde{\mathit{m}}\right)-r_1\left(\overline{\mathit{m}}\right)\right){\nabla }_{\phi }log{p}_{\phi }(\tilde{\mathit{m}}\mid \mathit{R}),$$

with a composite reward

$$r_1\left(\tilde{\mathit{m}}\right)={\lambda }_1{s}_1(\tilde{\mathit{m}},\mathit{m})+{\lambda }_2{s}_2\left({\overline{\mathit{y}}}_{\tilde{\mathit{m}}},\mathit{y}\right),$$

where $s_1$ measures BOC agreement (e.g., F1 over counts) and $s_2$ is CIDEr-D Vedantam et al. (2015) computed on a greedy caption ${\overline{\mathit{y}}}_{\tilde{\mathit{m}}}$ produced by Agent 2 when conditioned on $\tilde{\mathit{m}}$.

Agent 2: Caption Policy.

Conditioned on the baseline mediator $\overline{\mathit{m}}$ and a sampled ${\mathit{z}}_c\sim p\left({\mathit{z}}_c\right)$or${\mathit{z}}_c\sim q_{\varphi }({\mathit{z}}_c\mid \tilde{\mathit{c}})$ with $\tilde{\mathit{c}}\sim p\left(\mathit{c}\right)$, the captioner samples $\tilde{\mathit{y}}$ and forms a greedy baseline $\overline{\mathit{y}}$:

$${\nabla }_{\theta }\mathcal{L}\left(\theta \right)\approx -\left(r_2\left(\tilde{\mathit{y}}\right)-r_2\left(\overline{\mathit{y}}\right)\right){\nabla }_{\theta }log{p}_{\theta }\left(\tilde{\mathit{y}}\mid \mathit{R},\overline{\mathit{m}},{\mathit{z}}_c\right),$$

where $r_2(·)=s_2(·,\mathit{y})$. We additionally use an entropy bonus $\mathcal{H}\left({\pi }_{\theta }\right)$ to sustain lexical diversity:

$${\mathcal{L}}_{\mathrm{cap}}^{\mathrm{ent}}=-\delta \sum _t\sum _v{p}_{\theta }(v\mid ·)log{p}_{\theta }(v\mid ·).$$

Alternating Max–Max Updates.

Following Xiao et al. (2020), we alternate updates that maximize each agent’s reward while holding the other fixed. Practically, we interleave $K_1$ mediator steps and $K_2$ caption steps per epoch and maintain an exponential moving average baseline to reduce variance.

Unified Training Schedule.

Training proceeds in three phases: (i) warm-up with ${\mathcal{L}}_{\mathrm{MLE}}$ (annealed $\beta$, high Gumbel temperature); (ii) mixed MLE+RL with gradually increased RL weight $\rho$; (iii) full RL with entropy regularization and proxy-invariance penalties. The full objective in phase (ii) is

$$min(1-\rho ){\mathcal{L}}_{\mathrm{MLE}}+\rho \left(\mathbb{E}\left[{\mathcal{L}}_{\mathrm{RL}}^m\right]+\mathbb{E}\left[{\mathcal{L}}_{\mathrm{RL}}^y\right]\right),$$

where expectations are over the respective sampling procedures.

3.7. Complexity, Stability, and Calibration

Computational Footprint.

Let N denote the number of encoder layers and $L_r$ the number of regions. The encoder scales as $\mathcal{O}\left(N{L}_r^{2}d\right)$ due to self-attention, while mediator attention adds $\mathcal{O}\left(L_m{L}_{r}d\right)$. The proxy VAE scales linearly in $L_c$ and d. Our MARL alternation is compute-efficient because both agents reuse shared encodings.

Stability Enhancements.

We adopt: (i) label smoothing for caption tokens (0.1), (ii) gradient clipping (${\ell }_2\le 1.0$), (iii) KL warm-up for $q_{\varphi }$, and (iv) temperature annealing for Gumbel–Softmax ($\tau :1.0\to 0.3$). To attenuate sensitivity to proxies, we also impose a mutual-information-style regularizer between the caption distribution and proxies:

$${\mathcal{L}}^{\mathrm{MI}}\approx \kappa |\mathrm{Cov}\left(f\left(\tilde{\mathit{c}}\right),g\left(\tilde{\mathit{y}}\right)\right){|}_1,$$

where f and g are learned projections; this surrogate discourages direct pathways from proxies to lexical choices.

Calibration of Interventions.

To prevent overcorrection, we introduce a Lagrangian penalty that softly enforces proximity between interventional and observational predictions when proxies are uninformative:

$${\mathcal{L}}^{\mathrm{cal}}=\zeta \parallel \mathrm{StopGrad}\left(p(\mathit{y}\mid \mathit{R})\right)-p(\mathit{y}\mid \mathrm{do}\left(\mathit{R}\right)){\parallel }_2^2.$$

This keeps DEPICT faithful to visual evidence while still neutralizing spurious influences.

3.8. DEPICT Pipeline

• DMT factorization: predict mediator BOC and condition captioning on it, cf. (1)–(3).
• Intervention + proxies: compute $p(\mathit{y}\mid \mathrm{do}(\mathit{R}\left)\right)$ with proxy sampling and latent confounders via VI, cf. (4)–(7).
• Parameterization: hierarchical encoder, attention-based mediator head, and top-down caption decoder with ${\mathit{z}}_m$ and ${\mathit{z}}_c$.
• Training: MLE warm-up, followed by alternating multi-agent SCRL with entropy, decorrelation, MI penalties, and calibration.

4. Experiments

In this section, we present a comprehensive empirical study of our framework, hereafter referred to as DEPICT (DEpendent multi-task framework with Proxy-confounder Intervened Captioning Transformer). We consolidate all experimental content under one section and substantially expand the evaluation along multiple dimensions: benchmarks and protocols, single-model and online server results, extensive ablations, causal stress tests, mediator quality analysis, diversity and faithfulness, human judgments, efficiency, and qualitative error analysis. Unless noted, we follow standard practices for evaluation and reporting and keep all referenced implementations faithful to their original settings.

4.1. Benchmarks, Metrics, and Protocols

Datasets. We evaluate on MSCOCO1, the de facto benchmark for image captioning. The dataset includes 82,783 training and 40,504 validation images. Following convention, we adopt the “Karpathy” split Karpathy and Li (2015): 113k images for training (train+rest-val), 5k for validation, and 5k for testing. We also submit to the online test server to obtain c5/c40 results. The vocabulary is built from tokens appearing at least 5 times, yielding $10,369$ unique entries. We cap decoding length at 16, covering $\approx 95\%$ of references.

Metrics. We report BLEU-n (B1/B4) Papineni et al. (2020), METEOR (ME) Banerjee and Lavie (2005), ROUGE-L (RG) Lin (2004), CIDEr-D (CD) Vedantam et al. (2015), and SPICE (SP) Anderson et al. (2016). Online server numbers are presented under both c5 and c40 settings. Evaluations are computed with the public COCO-caption toolkit.2

Compared Systems. We compare against SCST Rennie et al. (2017), Up-Down Anderson et al. (2018), SGAE Huang et al. (2019), AoANet Huang et al. (2019), Up-Down+VC (causal intervention) Wang et al. (2020), WeakVRD Shi et al. (2020), ${\mathcal{M}}^2$ Transformer Cornia et al. (2020), and XLAN Pan et al. (2020). A strong in-house baseline (TransLSTM) uses a Transformer encoder and a top-down LSTM decoder. For XLAN we report official results (‡) and our faithful reproduction (†). We remark that OSCAR Li et al. (2020) leverages large-scale vision–language pretraining and ground-truth labels at inference, and thus is not directly comparable to single-dataset captioners.

Training Hyperparameters. We employ Faster R-CNN features Anderson et al. (2018) with adaptive proposals $n\in [10,100]$. Visual features (2048-d) are projected to 1024-d; word embeddings, attention hidden sizes, and the LSTM decoder are all 1024-d. The encoder has 4 layers; we use layer 2 for BOC prediction and layer 4 for captioning. We train with Adam, starting at $1\times {10}^{-4}$, halving on CIDEr-D plateaus (2 epochs patience), minimum $5\times {10}^{-6}$, batch size 50. We pretrain with MLE for 30 epochs (enable Gumbel after epoch 15) and then fine-tune with MARL for 35 epochs. As a sampling-based extension, we generate 20 candidates via different ${\mathit{z}}_c$ draws and employ a learned image–text selector to choose the best candidate at test time (details in Appendix).

4.2. Single-Model Results on Karpathy Split

Discussion. Table 1 shows that DEPICT improves CIDEr-D over TransLSTM by $+3.3$ (MLE) and $+3.3$ (RL) on average. With a trained selector over 20 samples (varying ${\mathit{z}}_c$), gains reach $+6.6$ (MLE) and $+3.6$ (RL). Using gold BOC pushes the ceiling to $135.2$ CIDEr-D in RL, emphasizing the importance of accurate mediation. The “Optimum Selector” approximates an oracle by picking the best of 20 candidates post hoc, revealing an MLE upper bound of $\approx 141.7$ CIDEr-D, slightly decreasing under RL—suggesting that some reinforcement objectives may prune rare but correct lexical variants.

4.3. Online Test Server Results

Discussion. Online server results (Table 2) corroborate the Karpathy findings: DEPICT yields competitive BLEU and ROUGE and the best CIDEr-D under c40, indicating strong consensus with multiple references.

4.4. Ablations: Mediation and De-Confounding Components

Discussion. Removing either mediation or de-confounding degrades CIDEr-D by $\approx 1.0$–$2.0$ points. Sequential training (Pipeline) underperforms due to compounding exposure bias between tasks. DEPICT-Full consistently yields the best scores, and gains persist under RL.

Table 3. Ablation results on the Karpathy split (MLE and RL). “Pipeline” trains BOC then captioning sequentially. “-M” drops de-confounding; “-LDC” drops mediator. The full model provides the best trade-off.

Period	Ablated Models	B1	B4	ME	RG	CD	SP
MLE	TransLSTM	77.1	37.1	28.2	57.3	117.3	21.2
	Pipeline	77.0	36.7	27.9	56.9	116.2	20.9
	DEPICT-M	77.4	37.9	28.3	57.8	119.7	21.8
	DEPICT-LDC	77.5	37.2	28.1	57.5	118.9	21.6
	DEPICT-Full (Avg.)	78.6	38.0	28.5	58.1	120.6	22.1
	DEPICT-Full (Best)	79.2	38.3	28.8	58.3	123.9	22.3
RL	TransLSTM	80.2	38.6	28.5	58.3	128.7	22.2
	Pipeline	79.9	38.1	28.3	58.0	126.7	22.1
	DEPICT-M	81.2	39.3	28.9	59.0	131.5	22.8
	DEPICT-LDC	80.8	38.7	28.7	58.7	130.2	22.4
	DEPICT-Full (Avg.)	81.4	39.7	29.1	59.2	132.0	23.0
	DEPICT-Full (Best)	81.5	39.8	29.2	59.3	132.3	23.0

Table 3. Ablation results on the Karpathy split (MLE and RL). “Pipeline” trains BOC then captioning sequentially. “-M” drops de-confounding; “-LDC” drops mediator. The full model provides the best trade-off.

Period	Ablated Models	B1	B4	ME	RG	CD	SP
MLE	TransLSTM	77.1	37.1	28.2	57.3	117.3	21.2
	Pipeline	77.0	36.7	27.9	56.9	116.2	20.9
	DEPICT-M	77.4	37.9	28.3	57.8	119.7	21.8
	DEPICT-LDC	77.5	37.2	28.1	57.5	118.9	21.6
	DEPICT-Full (Avg.)	78.6	38.0	28.5	58.1	120.6	22.1
	DEPICT-Full (Best)	79.2	38.3	28.8	58.3	123.9	22.3
RL	TransLSTM	80.2	38.6	28.5	58.3	128.7	22.2
	Pipeline	79.9	38.1	28.3	58.0	126.7	22.1
	DEPICT-M	81.2	39.3	28.9	59.0	131.5	22.8
	DEPICT-LDC	80.8	38.7	28.7	58.7	130.2	22.4
	DEPICT-Full (Avg.)	81.4	39.7	29.1	59.2	132.0	23.0
	DEPICT-Full (Best)	81.5	39.8	29.2	59.3	132.3	23.0

4.5. Causal Stress Tests and Proxy Sensitivity

Findings. Table 4 demonstrates that naive or random proxies can slightly hurt performance, while principled high-frequency proxies together with a learned ${\mathit{z}}_c$ improve ME and CD. This aligns with the hypothesis that carefully modeled proxies approximate latent biases and benefit interventional decoding.

4.6. Mediator Quality and Oracle Analyses

Observation. Improvements with gold BOC quantify potential headroom if mediator prediction is further enhanced (e.g., stronger detectors/segmenters, category normalization). Concretely, replacing predicted $\mathit{m}$ with oracle labels tightens the semantic interface between vision and language, thereby reducing ambiguity in object presence and cardinality and yielding consistent gains in ME, RG, and especially CD. This phenomenon indicates that errors in $\mathit{m}$ propagate nonlinearly through $p(\mathit{y}\mid \mathit{R},\mathit{m})$, amplifying small miscounts or misclassifications into lexical substitutions (e.g., generic nouns replacing fine-grained ones). From an information-theoretic angle, the oracle mediator increases mutual information $I(\mathit{m};\mathit{y}\mid \mathit{R})$, effectively sharpening the conditional support of the decoder. Practically, this suggests straightforward avenues for improvement: (i) upgrading region proposals with open-vocabulary detection and segmentation, (ii) calibrating category priors to mitigate long-tail collapse, and (iii) standardizing label spaces via synonym clustering and category normalization. Together, these steps narrow the gap between predicted and gold BOC, translating into measurable downstream caption quality improvements.

Table 5. Effect of mediator quality. Gold BOC narrows the gap to the oracle. The learned selector exploits ${\mathit{z}}_c$ diversity to realize additional gains.

BOC Source	B4	ME	CD	SP
Predicted BOC (Avg.)	38.0	28.5	120.6	22.1
Predicted BOC + Selector (20)	38.3	28.8	123.9	22.3
Gold BOC (Avg.)	38.3	29.1	123.1	22.3
Gold BOC + Selector (20)	38.8	29.5	126.4	22.7

Table 5. Effect of mediator quality. Gold BOC narrows the gap to the oracle. The learned selector exploits ${\mathit{z}}_c$ diversity to realize additional gains.

BOC Source	B4	ME	CD	SP
Predicted BOC (Avg.)	38.0	28.5	120.6	22.1
Predicted BOC + Selector (20)	38.3	28.8	123.9	22.3
Gold BOC (Avg.)	38.3	29.1	123.1	22.3
Gold BOC + Selector (20)	38.8	29.5	126.4	22.7

4.7. Diversity and Faithfulness

Remark. Interventional sampling through ${\mathit{z}}_c$ encourages a broader lexical space while maintaining semantic grounding via $\mathit{m}$ and $\mathit{R}$. By drawing ${\mathit{z}}_c\sim q_{\varphi }({\mathit{z}}_c\mid \tilde{\mathit{c}})$ and decoding under $p_{\theta }(\mathit{y}\mid \mathit{R},{\mathit{z}}_m,{\mathit{z}}_c)$, the model explores alternative but still plausible linguistic realizations, increasing distinct-n without drifting into hallucination. Crucially, ${\mathit{z}}_c$ perturbs nuisance factors correlated with phrasing—such as stylistic preference for hypernyms vs. hyponyms—while ${\mathit{z}}_m$ anchors object identity and count, and $\mathit{R}$ preserves visual evidence. This separation of concerns regularizes the generation manifold: lexical variety arises from intervening on confounding style variables rather than destabilizing content variables. Empirically, this yields higher diversity scores with stable or improved CD/SP, indicating that variety is gained without sacrificing factual precision.

Table 6. Lexical diversity (distinct-n) and fraction of novel n-grams (Novel-n) on Karpathy test. DEPICT produces more varied yet faithful captions; the selector further promotes diversity without sacrificing accuracy.

Model	Div-1↑	Div-2↑	Novel-n (%)↑
TransLSTM	0.63	0.79	7.2
XLAN †	0.64	0.80	7.0
DEPICT (Avg.)	0.66	0.82	8.5
DEPICT + Selector (20)	0.68	0.84	9.4

Table 6. Lexical diversity (distinct-n) and fraction of novel n-grams (Novel-n) on Karpathy test. DEPICT produces more varied yet faithful captions; the selector further promotes diversity without sacrificing accuracy.

Model	Div-1↑	Div-2↑	Novel-n (%)↑
TransLSTM	0.63	0.79	7.2
XLAN †	0.64	0.80	7.0
DEPICT (Avg.)	0.66	0.82	8.5
DEPICT + Selector (20)	0.68	0.84	9.4

4.8. Human Evaluation at Scale

Interpretation. The edge in faithfulness is consistent with the causal design: mediation reduces contradictions, while de-confounding tempers dataset biases. The mediator $\mathit{m}$ forces the decoder to condition on explicitly recognized entities and their multiplicities, which curbs common mistakes such as gender or attribute flips and discourages unsupported object insertions. Simultaneously, the latent ${\mathit{z}}_c$ absorbs spurious co-occurrence patterns (e.g., long hair→ woman) by modeling them as nuisance variables to be intervened upon, thus weakening shortcuts that traditional maximum likelihood objectives tend to exploit. As a result, the conditional distribution concentrates on captions that are both visually warranted and linguistically specific, improving human-judged faithfulness alongside automatic metrics like SPICE, which are sensitive to semantic propositional content. In short, the causal factorization operationalizes “say what you see, not what the dataset expects,” yielding robust gains in factual consistency.

Table 7. Human A/B testing on 1k images (five annotators/image). Annotators favored DEPICT particularly for faithfulness.

Pairwise Preference (%)	Fluency	Faithfulness	Overall
DEPICT vs. TransLSTM	58.7	63.9	61.2
DEPICT vs. XLAN†	51.9	55.6	53.4
DEPICT+Selector vs. XLAN†	55.1	60.4	57.3

Table 7. Human A/B testing on 1k images (five annotators/image). Annotators favored DEPICT particularly for faithfulness.

Pairwise Preference (%)	Fluency	Faithfulness	Overall
DEPICT vs. TransLSTM	58.7	63.9	61.2
DEPICT vs. XLAN†	51.9	55.6	53.4
DEPICT+Selector vs. XLAN†	55.1	60.4	57.3

4.9. Efficiency and Complexity

Note. The bulk of added cost stems from mediator and VAE heads; both are lightweight compared to the visual encoder. The hierarchical self-attention over $\mathit{R}$ dominates FLOPs ($\mathcal{O}\left(L_r^{2}d\right)$), whereas the BOC head adds roughly $\mathcal{O}\left(L_m{L}_{r}d\right)$ attention and a small per-category classifier, and the proxy VAE introduces a shallow Transformer without positional encodings plus two small projection heads for $(\mu ,log{\sigma }^2)$. In training, KL annealing and Gumbel–Softmax sampling contribute negligible overhead relative to encoder–decoder passes; at inference, the optional k-sample selector scales linearly with k but remains bounded by the shared visual encoding reused across samples. Consequently, the method achieves a favorable accuracy–efficiency trade-off: noticeable boosts in CD/SP with only modest increases in parameter count and latency, preserving practicality for large-scale deployment.

Table 8. Model size and throughput on a single V100 (batch size 50). DEPICT adds modest overhead relative to TransLSTM; the selector adds small inference latency when generating $k=20$ samples.

Model	Params (M)	Train it/s↑	Inference ms/img↓
TransLSTM	60.8	13.1	21.3
XLAN †	71.5	10.9	23.8
DEPICT (Avg.)	66.2	12.2	22.5
DEPICT + Selector (20)	66.2	11.9	24.8

Table 8. Model size and throughput on a single V100 (batch size 50). DEPICT adds modest overhead relative to TransLSTM; the selector adds small inference latency when generating $k=20$ samples.

Model	Params (M)	Train it/s↑	Inference ms/img↓
TransLSTM	60.8	13.1	21.3
XLAN †	71.5	10.9	23.8
DEPICT (Avg.)	66.2	12.2	22.5
DEPICT + Selector (20)	66.2	11.9	24.8

4.10. Qualitative Analysis and Case Study

We further analyze error modes: (i) attribute flips (e.g., gender, color), (ii) object hallucination (mentioning missing entities), and (iii) under-specification (choosing generic nouns over fine-grained ones). The mediator reduces (i) and (iii) by anchoring object presence and counts; the de-confounder notably curbs (ii) by weakening shortcuts induced by high-frequency proxies. Representative cases illustrate that words previously underlined in red (contradictions) become corrected, and tokens previously marked as uninformative (gray) are replaced with precise, blue-highlighted phrases when DEPICT is applied.

Figure 2. Case study. The underlined red words lead to inconsistent information, and the words with gray background are not informative enough. The blue words represent consistent and informative expressions.

Figure 2. Case study. The underlined red words lead to inconsistent information, and the words with gray background are not informative enough. The blue words represent consistent and informative expressions.

While DEPICT is robust under moderate domain shift, extremely sparse categories remain challenging for mediator prediction; leveraging stronger detection/segmentation backbones or open-vocabulary detectors may close this gap. In addition, our RL objective focuses on CIDEr-D; multi-objective rewards incorporating SPICE and factuality estimators could further enhance semantic faithfulness. Finally, structured proxies derived from curated knowledge graphs could offer more controllable confounder modeling.

Across automatic metrics, human judgments, and stress tests, DEPICT consistently yields captions that are more faithful, informative, and diverse than baselines, validating the effectiveness of explicit mediation and principled causal intervention.

5. Conclusion and Future Work

In this paper, we proposed DEPICT, a dependent multi-task learning framework with causal intervention for image captioning. The central idea of our work is to integrate causal reasoning into the image captioning pipeline to make the generated captions more faithful, consistent, and informative. Traditional captioning models often rely heavily on statistical correlations between visual features and language patterns, which can lead to spurious reasoning and errors such as incorrect object descriptions or missing critical details. Our framework tackles this by explicitly modeling the causal dependencies between visual inputs, intermediate object concepts, and linguistic outputs.

We first introduced the use of a bag-of-categories (BOC) mediator that serves as an interpretable intermediate representation connecting the vision and language components. This mediator allows the model to focus on the actual content of the image and reduces inconsistencies in generated captions. Next, we proposed a latent de-confounding approach to remove spurious correlations caused by dataset biases. Through variational inference, we modeled latent confounders in a continuous space and applied causal intervention to ensure that the generated captions reflect true causal relations rather than dataset artifacts. Finally, we developed a multi-agent reinforcement learning strategy that optimizes both the mediator and caption generator collaboratively. This strategy helps align the two tasks in an end-to-end fashion and mitigates inter-task discrepancies during training.

Experimental results on the MSCOCO benchmark demonstrated that DEPICT achieves strong quantitative performance and produces captions that are more semantically coherent and visually grounded than previous models. The results also show that our causal intervention effectively reduces contradictions and hallucinations, while the multi-agent learning mechanism improves robustness and generalization across diverse visual scenes.

Although our method brings clear improvements, it still faces certain limitations. The mediator’s accuracy depends on pre-trained object detectors, which might fail in complex or cluttered environments. The latent confounder modeling, while powerful, relies on proxies extracted from the dataset and may not capture higher-level contextual confounders. In addition, our framework introduces modest computational overhead due to the extra modules for mediation and variational inference, although the trade-off is acceptable given the significant performance gains.

For future work, we plan to extend DEPICT toward open-vocabulary and cross-domain captioning, integrating modern vision-language foundation models for more flexible mediator prediction. Another direction is to develop hierarchical causal graphs that represent relationships among multiple levels of visual and linguistic concepts, allowing the model to reason about spatial, temporal, and compositional aspects of scenes. Furthermore, we aim to explore adaptive intervention strategies that dynamically adjust causal modeling based on visual uncertainty, as well as human-in-the-loop learning approaches that align causal reasoning with human judgments of relevance and factuality.

Overall, this research highlights the importance of causal thinking in multimodal understanding and generation. By bridging dependent multi-task learning with causal intervention, DEPICT offers a principled way to enhance both interpretability and accuracy in image captioning. We believe that such causally grounded models will be key to building future multimodal systems that not only describe what they see but also understand the underlying structure and reasoning of the visual world.