Context-Aware Multi-Anchor Captioning for Text-Rich Image Understanding

1. Introduction

Text is an integral part of our daily visual experience, serving as a bridge between visual perception and semantic reasoning  [13]. In natural images, texts often provide contextual clues that enhance the interpretation of scenes—be it understanding a store’s signboard, recognizing brand names on packaging, or reading warnings on road indicators [5,19,20,34,41]. Traditional image captioning frameworks, however, have largely overlooked this dimension by focusing primarily on visual object relationships and scene-level summaries. As a result, they fail to capture textual cues that may be crucial for accurate semantic understanding. To overcome this limitation, [40] formalized the text-based image captioning (TextCap) task, emphasizing the importance of models that can not only describe but also read textual information from images.

TextCap serves a wide range of practical applications, from enhancing accessibility for visually impaired users [13] to enabling automated understanding of information-dense visual content such as receipts, advertisements, and street scenes. Unlike ordinary captions that summarize an image’s main objects, TextCap aims to describe subtle textual and visual interactions, requiring the model to reason over multimodal cues with high granularity. Despite its practical importance, the task remains challenging because (1) textual regions are often cluttered or partially occluded, (2) text semantics depend heavily on spatial and relational context, and (3) it is difficult to determine which parts of the textual content are most relevant for caption generation.

Early efforts have attempted to adapt traditional image captioning methods [2,19,21] to the TextCap setting. While these models excel at identifying key visual objects, they struggle to capture the semantics of embedded texts. Subsequent works, such as M4C-Captioner [40], attempted to integrate OCR-based text recognition tools [4,6,31] into captioning pipelines. However, these methods remain limited in their ability to fuse textual and visual information, often producing overly generic descriptions that fail to represent the intricate structure of text-visual relations. The underlying issue is that a single caption cannot encapsulate the full spectrum of meaningful elements present in text-rich images.

To address these limitations, we propose Multi-Anchor Captioner (MACap), a new framework that performs captioning through anchor-based decomposition and multi-view generation. The central idea is to break down an image into several semantic anchors, where each anchor corresponds to a key text token or visual clue that serves as a nucleus for contextual reasoning. Around each anchor, we construct an Anchor-Centered Graph (ACG) to capture its semantic dependencies and neighborhood relations. By modeling such anchor-centric structures, MACap can generate a set of complementary captions, each describing different facets of the image with both visual and textual details.

Formally, given an image $\mathcal{I}$ and a set of OCR-detected tokens $\mathcal{T}=t_1,t_2,...,t_N$, our goal is to produce a caption set $\mathcal{C}=c_1,c_2,...,c_K$ that maximizes both accuracy and diversity. The framework first computes attention-based text relevance scores for each token. Tokens with high ${\alpha }_i$ are selected as anchors. Each anchor $a_i$ forms an ACG by linking to semantically relevant tokens using cosine similarity-based adjacency weights. This graph representation allows message passing and relational encoding to enhance contextual reasoning before caption generation. Finally, the multi-view caption generator produces distinct captions $c_k$ conditioned on different ACG embeddings.

In addition to the methodological novelty, MACap introduces an elegant solution to one of the most persistent challenges in TextCap—balancing accuracy and diversity. While traditional methods optimize a single caption under maximum likelihood estimation, we employ a multi-objective learning scheme that jointly maximizes caption likelihood and penalizes redundancy across generated captions. The proposed framework has several noteworthy merits: it provides a more interpretable captioning process by associating captions with explicit textual anchors, ensures better coverage of text-visual relationships, and enables flexible multi-caption reasoning. Experimental results on the TextCaps dataset demonstrate that MACap substantially outperforms previous approaches across CIDEr, BLEU, and SPICE metrics while producing captions that are semantically richer and less redundant.

In summary, our main contributions are as follows:

• We propose a novel anchor-based multi-view framework, MACap, that generates diverse captions by decomposing images into semantically coherent anchor-centered subgraphs.
• We introduce an anchor graph construction and reasoning mechanism to model inter-textual and cross-modal relationships among OCR tokens and visual regions.
• We demonstrate significant improvements on the TextCaps benchmark, establishing new state-of-the-art results while maintaining both accuracy and caption diversity.

2. Related Work

2.1. Image Captioning

Image captioning, which aims to automatically generate a coherent textual description for a given image, represents one of the most challenging intersections between computer vision and natural language processing. This task requires models not only to perceive objects and scenes accurately but also to reason about their relationships and translate this understanding into natural language [2,15,42,44,45,49]. Early models followed a straightforward encoder–decoder framework in which convolutional neural networks (CNNs) encode spatial visual features, while recurrent neural networks (RNNs) decode them into textual sentences. Despite their initial success, such methods often produce generic captions that lack context awareness and fail to capture fine-grained semantic dependencies.

To overcome these limitations, the introduction of attention mechanisms revolutionized caption generation. Attention models enable the decoder to selectively focus on specific image regions while predicting each word, thus allowing a stronger alignment between visual and linguistic modalities. This concept—first explored by Xu et al. [49] and refined by others—marked a paradigm shift toward dynamic context modeling. Recent Transformer-based architectures [42] further enhanced this by enabling bidirectional dependencies and global reasoning across both modalities, resulting in models capable of generating syntactically fluent and semantically precise captions.

Beyond architecture refinements, several works have investigated iterative refinement strategies. For instance, sequence-level re-prediction [16,25,29,48] allows the model to correct earlier decoding errors via multiple passes. NBT [32] introduced a two-stage approach that first predicts a coarse sentence structure (a “template”) and subsequently fills in object-specific details, significantly improving grammatical and contextual quality. Reinforcement learning (RL)-based approaches [18,30,35,39] further treat caption generation as a sequential decision-making problem under the Markov Decision Process [47], directly optimizing evaluation metrics such as CIDEr and BLEU rather than relying on word-level likelihoods.

However, despite these advances, traditional image captioning systems often remain limited in diversity and controllability. To produce captions beyond generic patterns, recent works have explored controllable generation paradigms. Style-controllable captioning [14,17,33] introduces auxiliary labels or paired annotations to govern sentiment, tone, or formality. While this enriches stylistic expression, it requires extensive labeled corpora and complicates training pipelines. Parallel efforts have focused on content-controllable captioning [22,50], which allows models to describe localized regions or specific objects rather than the entire scene. Dense captioning frameworks extend this idea by jointly predicting region proposals and associated textual descriptions, producing multi-sentence outputs that correspond to different spatial locations.

Building on this foundation, signal-based sampling approaches [7,8,9,11] introduce stochasticity and control vectors to guide caption generation, yielding multiple diverse hypotheses for a single image. These models often operate by adjusting latent control variables that influence lexical choice and syntactic diversity. ASG2Caption [7], for example, leverages an abstract scene graph as a structured control representation, guiding the model to produce captions at varying levels of semantic granularity. Despite progress, such systems still fall short of achieving both linguistic richness and fine-grained semantic grounding.

Our work shares inspiration with dense and controllable captioning but extends these ideas to a new paradigm—text-grounded multi-view captioning—in which textual content embedded within images acts as a critical cue for generating diverse, semantically complementary captions. By dynamically exploring different text–region associations, our model produces context-aware multi-perspective descriptions that align more closely with human-level interpretation.

2.2. Text-based Image Captioning

While conventional captioning has focused mainly on visual semantics, real-world images frequently contain textual elements—such as street names, product brands, or signage—that provide essential contextual information. Text-based image captioning (TextCap) thus aims to generate captions that jointly describe visual and textual contents [40]. This task introduces new complexity: it requires models to “read” embedded texts and integrate them meaningfully into descriptions, enabling a deeper understanding of the scene.

Existing image captioning datasets such as MS-COCO [28] and Visual Genome [24] primarily focus on salient object recognition and overlook written texts, creating a dataset bias that limits model generalization. Consequently, most traditional captioning systems [2,15,42,44,49] lack the ability to interpret or utilize text cues. To fill this gap, Sidorov et al. [40] introduced the TextCaps benchmark, a large-scale dataset specifically designed for text-based image captioning. This dataset challenges models to not only identify visual scenes but also recognize and semantically incorporate OCR-extracted tokens from within images.

The M4C-Captioner [40], adapted from M4C [19] originally proposed for TextVQA, represents an early attempt to integrate textual understanding into caption generation. It processes both detected text tokens and visual features jointly through a multimodal transformer, generating a single unified caption per image. While this architecture successfully incorporates text recognition modules such as Rosetta [6] and ABCNet [31], it still faces fundamental limitations: producing a single caption is insufficient to capture the full semantic diversity present in complex text-rich scenes. The resulting captions often either overemphasize certain tokens or ignore important contextual relationships between texts and surrounding visual elements.

In contrast, our proposed method—termed Graphically Anchored Multi-View Captioner (GAMCap)—introduces a new perspective on this challenge. Instead of generating a monolithic description, we propose to model multiple anchor-centric subgraphs that explicitly represent textual relationships. Specifically, we design an Anchor Proposal Module (APM) that computes token importance scores based on both text salience and spatial context. Top-ranked tokens are selected as anchors.

Next, we establish relational links among tokens to form anchor-centered graphs (ACGs). The edge weights between two tokens $t_i$ and $t_j$ are derived from their semantic and positional affinities. This design enables the model to cluster related tokens—such as text segments belonging to the same sign or product label—into coherent subgraphs that serve as the foundation for localized caption generation.

Each subgraph is subsequently processed by a caption decoder conditioned on both the global visual representation and the corresponding ACG embedding. The final caption generation ensures that each caption reflects the distinctive semantics of its anchor subgraph while maintaining visual–textual coherence. Through this graph-driven multi-caption paradigm, GAMCap achieves a balance between semantic accuracy and caption diversity.

Compared to prior works that treat text merely as auxiliary input, our model explicitly structures textual information, allowing fine-grained reasoning over intra-text relationships. Moreover, by decomposing the captioning task into multiple graph-conditioned subtasks, our method captures complementary narrative aspects—such as brand identity, location, or signage context—that would otherwise be lost in single-caption frameworks. Experimental results (discussed in subsequent sections) demonstrate that GAMCap surpasses existing baselines on the TextCaps benchmark, particularly in metrics measuring caption diversity and semantic fidelity.

In summary, while previous approaches have primarily focused on improving recognition or alignment mechanisms, our work moves toward a structured, relational understanding of text-rich images. By integrating anchor proposal, graph construction, and multi-view caption generation into a unified framework, we offer a novel and effective paradigm for text-based dense image captioning.

3. Context-Aware Multi-Anchor Captioning for Text-Rich Images

We investigate text-based image context-aware multi-anchor captioning (MACap), whose objective is to read and reason over images containing written text and to produce faithful, detailed descriptions. This setting is intrinsically challenging because a single global sentence rarely covers the breadth of visual semantics and embedded textual cues present in complex scenes. Conventional approaches tend to emit one caption that collapses heterogeneous content into a single narrative, often overlooking task-relevant text or conflating unrelated regions. To overcome these limitations, we advocate a multi-view strategy that yields several complementary captions, each grounded in a distinct subset of textual evidence and its surrounding visual context.

Figure 1. An overview of the proposed Context-Aware Multi-Anchor Captioning (MACap) framework. Given an input image, region-level visual features are extracted via Faster R-CNN and textual cues are obtained through OCR. The Fusion Transformer integrates multimodal information and branches into two streams: the Anchor Proposal and ACG Construction module, which builds semantic anchors, and the Stage I Visual Draft module, which produces an initial caption draft. Both streams are combined in Stage II Anchor-Conditioned Refinement, where anchor and OCR pointers guide fine-grained textual generation. Finally, the model outputs multiple complementary descriptions, forming Multi-View Captions.

Figure 1. An overview of the proposed Context-Aware Multi-Anchor Captioning (MACap) framework. Given an input image, region-level visual features are extracted via Faster R-CNN and textual cues are obtained through OCR. The Fusion Transformer integrates multimodal information and branches into two streams: the Anchor Proposal and ACG Construction module, which builds semantic anchors, and the Stage I Visual Draft module, which produces an initial caption draft. Both streams are combined in Stage II Anchor-Conditioned Refinement, where anchor and OCR pointers guide fine-grained textual generation. Finally, the model outputs multiple complementary descriptions, forming Multi-View Captions.

However, three obstacles make this goal non-trivial. (i) Selecting which textual fragments to copy verbatim or paraphrase is difficult when images contain many OCR tokens of varying importance. (ii) Capturing the latent relationships among diverse tokens (e.g., brand ↔ price, street name ↔ shop front) is necessary for accuracy yet remains underexplored. (iii) Generating multiple captions that are both accurate and non-redundant requires explicit mechanisms for view decomposition and diversity control.

Overview. We introduce MACap, a unified architecture that addresses these issues through three tightly coupled stages: (1) Unified multimodal tokenization and embedding that encodes visual objects and OCR tokens into a common space; (2) Anchor proposal and graph induction that discovers salient text “anchors” and constructs anchor-centered graphs (ACGs) to model relational structure; (3) Anchor-conditioned deliberation captioning that first produces a visual-specific draft and then refines it into text-specific captions under ACG guidance. End-to-end training is achieved via a composite objective that supervises anchor prediction, graph construction, and two-stage captioning.

3.1. Unified Multimodal Tokenization and Embedding

To initialize the representation space, we employ a pre-trained Faster R-CNN [38] to detect N visual regions and Rosetta OCR [6] to recognize M text tokens. Let d denote the hidden dimensionality.

Visual embedding.

For the i-th region, Faster R-CNN provides an appearance descriptor ${\mathbf{v}}_i^a\in {\mathbb{R}}^d$ and a bounding box ${\mathbf{v}}_i^b\in {\mathbb{R}}^4$. We enrich geometry via concatenation followed by a linear projection and LayerNorm [3]:

$${\widehat{\mathbf{v}}}_i=f_1\left([{\mathbf{v}}_i^a,{\mathbf{v}}_i^b]\right),\widehat{\mathbf{V}}={[{\widehat{\mathbf{v}}}_1,\dots ,{\widehat{\mathbf{v}}}_N]}^{\top }\in {\mathbb{R}}^{N\times d}.$$

(1)

Token embedding.

For the j-th OCR token, we use its appearance ${\mathbf{t}}_j^a$ and box ${\mathbf{t}}_j^b$, and further incorporate FastText ${\mathbf{t}}_j^f$ (word-level) and PHOC ${\mathbf{t}}_j^p$ (character-level) as in M4C-Captioner [40]. We project to the common space with LayerNorm:

$${\widehat{\mathbf{t}}}_j=f_2\left([{\mathbf{t}}_j^a,{\mathbf{t}}_j^b,{\mathbf{t}}_j^f,{\mathbf{t}}_j^p]\right),\widehat{\mathbf{T}}={[{\widehat{\mathbf{t}}}_1,\dots ,{\widehat{\mathbf{t}}}_M]}^{\top }\in {\mathbb{R}}^{M\times d}.$$

(2)

Cross-modal fusion.

We concatenate all tokens and apply an $L_1$-layer Transformer $\Psi (·;{\theta }_a)$ to realize both self- and cross-attention, yielding interaction-enriched representations:

$$[\mathbf{V},\mathbf{T}]=\Psi \left([\widehat{\mathbf{V}},\widehat{\mathbf{T}}];{\theta }_a\right),$$

(3)

where $\mathbf{V}\in {\mathbb{R}}^{N\times d}$, $\mathbf{T}\in {\mathbb{R}}^{M\times d}$. A single attention head computes

$$\mathrm{Attn}(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathrm{softmax}\left({\displaystyle \frac{\mathbf{Q}{\mathbf{K}}^{\top }}{\sqrt{d}}}\right)\mathbf{V},$$

(4)

with multi-head extension standard. This step harmonizes spatial, textual, and appearance cues before anchor discovery.

3.2. Anchor Proposal and Graph Induction

Naively feeding all OCR tokens to a captioner dilutes salient evidence and encourages generic summaries. Our Anchor Proposal Module (APM) identifies informative tokens and builds an Anchor-Centered Graph (ACG) per selected anchor to capture relational structure.

Anchor scoring.

Given token features $\mathbf{T}$, we score each token via a lightweight predictor $\varphi$:

$${\mathbf{s}}_{\mathrm{anc}}=\mathrm{Softmax}\left(\varphi \left(\mathbf{T}\right)\right)\in {\mathbb{R}}^M,$$

(5)

where ${\mathbf{s}}_{\mathrm{anc}}\left(j\right)$ reflects token j’s importance. During training, we use $argmax$ to pick a single anchor; during inference we select the top-K tokens.

Relational graph induction.

Salience alone is insufficient; we must also recover token neighborhoods that form coherent semantic units (e.g., “COFFEE” with “$2.99”). We initialize an RNN with the anchor state and scan tokens to produce dependency-aware features:

$${\mathbf{T}}_{\mathrm{graph}}=\mathrm{RNN}\left(\mathbf{T},{\mathbf{T}}_{\mathrm{anchor}}\right),{\mathbf{s}}_{\mathrm{graph}}=\sigma \left(f_3\left({\mathbf{T}}_{\mathrm{graph}}\right)\right),$$

(6)

where $\sigma$ is the sigmoid. Tokens with ${\mathbf{s}}_{\mathrm{graph}}\left(j\right)>0.5$ are linked to the anchor to form an ACG $\mathcal{G}=[{\mathbf{T}}_{\mathrm{anchor}},\left\{{\mathbf{T}}_{\mathrm{graph}}^{\left(j\right)}\right\}]$.

Semantic–geometric affinities.

We further stabilize grouping by combining semantic and geometric affinities into a soft adjacency:

$$a_{ij}=\underset{\mathrm{semantic}}{\underbrace{{\displaystyle \frac{\langle {\mathbf{t}}_i,{\mathbf{t}}_j\rangle }{\parallel {\mathbf{t}}_i\parallel \parallel {\mathbf{t}}_j\parallel }}}}·\underset{\mathrm{geometric}}{\underbrace{exp\left(-\gamma \parallel p_i-p_j{\parallel }_2\right)}},$$

(7)

where ${\mathbf{t}}_i$ is the token embedding and $p_i\in {\mathbb{R}}^2$ denotes normalized box centers. We use $a_{ij}$ as edge weights in a one-step message passing refinement:

$${\tilde{\mathbf{t}}}_i=\mathrm{LN}\left({\mathbf{t}}_i+\sum _j{\alpha }_{ij}\mathbf{W}{\mathbf{t}}_j\right),{\alpha }_{ij}={\displaystyle \frac{a_{ij}}{{\sum }_k{a}_{ik}}},$$

(8)

and replace ${\mathbf{T}}_{\mathrm{graph}}$ by $\tilde{\mathbf{T}}$ in (6). This yields compact, noise-robust ACGs that better reflect functional groups of tokens.

3.3. Anchor-Conditioned Deliberation Captioner

TextCap requires explicit lexical grounding while preserving fluent, globally coherent language. We therefore adopt a two-stage Anchor Captioning Module (AnCM) that first produces a visual-specific draft and then refines it into an anchor-aware caption using the selected ACG.

Stage I: Visual drafter ${\mathrm{AnCM}}_v$.

We use an $L_2$-layer Transformer to decode a draft caption autoregressively from $\mathbf{V}$. With prefix language modeling [37], the hidden state and token are

$${\mathbf{h}}_c=\Psi \left(\mathbf{V},\mathrm{LM}\left({\mathbf{y}}_{c-1}^{\prime }\right);{\theta }_v\right),y_c^{\prime }=argmax\left(f_4\left({\mathbf{h}}_c\right)\right),$$

(9)

and the draft loss is ${\mathcal{L}}_{\mathrm{vcap}}=-{\sum }_{c=1}^{C}logP\left(y_c^{\prime }\right)$. The backbone can be replaced with BUTD [2] or AoANet [21] without altering the rest of the pipeline.

Stage II: Text refiner ${\mathrm{AnCM}}_t$.

The refiner conditions on both the draft hidden states $\left\{{\mathbf{h}}_c\right\}$ and the ACG $\mathcal{G}$. An $L_3$-layer Transformer fuses them:

$$\widehat{\mathcal{G}},{\widehat{\mathbf{y}}}_c=\Psi \left([\mathcal{G},{\mathbf{h}}_c,\mathrm{LM}\left({\mathbf{y}}_{c-1}\right)];{\theta }_t\right),$$

(10)

and we predict from a shared vocabulary head $f_4$ and a dynamic pointer network $f_{\mathrm{dp}}$ [19]:

$$y_c=argmax\left([f_4\left({\widehat{\mathbf{y}}}_c\right),f_{\mathrm{dp}}(\widehat{\mathcal{G}},{\widehat{\mathbf{y}}}_c)]\right).$$

(11)

To avoid non-differentiability from sampling $y_c^{\prime }$, we pass ${\mathbf{h}}_c$ to the refiner directly and optimize

$${\mathcal{L}}_{\mathrm{tcap}}=-\sum _{c=1}^{C}logP\left(y_c\mid {\mathrm{AnCM}}_t({\mathbf{h}}_c,\mathcal{G};{\theta }_t)\right).$$

(12)

Coverage and diversity control.

Beyond maximum likelihood, we incorporate coverage to prevent repeated copying and diversity to reduce redundancy across multiple anchor-conditioned captions ${\left\{{\mathcal{Y}}^{\left(k\right)}\right\}}_{k=1}^K$:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{cov}}& =\sum _{c}min\left(\sum _{u\in \mathcal{G}}{\alpha }_c\left(u\right),1\right),{\alpha }_c\left(u\right)={\mathrm{softmax}}_u\left({\widehat{\mathbf{y}}}_c^{\top }\widehat{\mathbf{u}}\right),\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{div}}& =\sum _{p\ne q}\mathrm{sim}\left({\mathcal{Y}}^{\left(p\right)},{\mathcal{Y}}^{\left(q\right)}\right),\mathrm{sim}=\mathrm{Jaccard}\mathrm{or}1-\mathrm{CIDErDist}.\hfill \end{array}$$

(14)

These terms are crucial when $K>1$ anchors are decoded at inference.

3.4. Learning Objectives and Ground-Truth Mining

We supervise MACap using a sum of four primary terms together with regularizers:

$$\mathcal{L}={\mathcal{L}}_{\mathrm{anchor}}\left({\mathbf{s}}_{\mathrm{anc}}\right)+\alpha {\mathcal{L}}_{\mathrm{graph}}\left({\mathbf{s}}_{\mathrm{graph}}\right)+\beta {\mathcal{L}}_{\mathrm{vcap}}\left({\mathcal{Y}}^{\prime }\right)+\eta {\mathcal{L}}_{\mathrm{tcap}}\left(\mathcal{Y}\right)+\lambda {\mathcal{L}}_{\mathrm{div}}+\mu {\mathcal{L}}_{\mathrm{cov}}.$$

(15)

Unless noted otherwise, classification terms use binary cross-entropy. We add an entropy bonus to encourage non-peaky anchor distributions, ${\mathcal{L}}_{\mathrm{ent}}=-{\sum }_j{\mathbf{s}}_{\mathrm{anc}}\left(j\right)log{\mathbf{s}}_{\mathrm{anc}}\left(j\right)$, which can be absorbed into ${\mathcal{L}}_{\mathrm{anchor}}$.

Ground-truth labels.

We automatically mine supervision from the standard five human captions per image: (1) ${\mathcal{L}}_{\mathrm{tcap}}$ uses a verbatim ground-truth with intact OCR tokens; (2) ${\mathcal{L}}_{\mathrm{vcap}}$ uses the same caption but masks OCR-copied spans with [unk] to prevent the visual draft from relying on token copying; (3) ${\mathcal{L}}_{\mathrm{anchor}}$ takes as positive the most frequently mentioned token across the five captions; (4) ${\mathcal{L}}_{\mathrm{graph}}$ regards all tokens appearing with that anchor in the same ground-truth sentence as positives. This mining requires no extra annotation and preserves fairness against prior work.

3.5. Regularization via Contrastive Alignment

To stabilize cross-modal grounding, we introduce a lightweight contrastive term that aligns the ACG embedding with the final caption embedding. Let $\mathbf{g}=\mathrm{Pool}\left(\widehat{\mathcal{G}}\right)$ and $\mathbf{c}=\mathrm{Pool}\left({\left\{{\widehat{\mathbf{y}}}_c\right\}}_{c=1}^C\right)$. For a mini-batch $\mathcal{B}$,

$${\mathcal{L}}_{\mathrm{con}}=-{\displaystyle \frac{1}{\left|\mathcal{B}\right|}}\sum _{(i,i)\in \mathcal{B}}log{\displaystyle \frac{exp\left({\tau }^{-1}{\mathbf{g}}_i^{\top }{\mathbf{c}}_i\right)}{{\sum }_{(i,j)\in \mathcal{B}}exp\left({\tau }^{-1}{\mathbf{g}}_i^{\top }{\mathbf{c}}_j\right)}},$$

(16)

with temperature $\tau >0$. This encourages each caption to be maximally similar to its guiding graph while dissimilar to others, reducing anchor drift.

3.6. Curriculum Schedule and Optimization

We adopt a curriculum that first warms up ${\mathrm{AnCM}}_v$ with teacher forcing, then enables ${\mathrm{AnCM}}_t$ under gold ACGs, and finally activates APM and graph induction for joint training. The learning rate follows a linear warmup then cosine decay; dropout is applied to attention and MLP sublayers. Scheduled sampling is used for ${\mathrm{AnCM}}_v$ after warmup with a linearly increasing sampling ratio.

3.7. Inference and Multi-View Decoding

At test time, we select top-K anchors from ${\mathbf{s}}_{\mathrm{anc}}$. For each anchor, we induce an ACG, decode a visual draft, and refine it to a text-specific caption. We optionally run diverse beam search with dissimilarity constraints across beams assigned to different anchors:

$$\mathrm{Score}\left({\mathcal{Y}}^{\left(k\right)}\right)=\sum _{c}logP\left(y_c^{\left(k\right)}\right)-\omega \sum _{p<k}\mathrm{sim}\left({\mathcal{Y}}^{\left(k\right)},{\mathcal{Y}}^{\left(p\right)}\right),$$

(17)

with $\omega >0$. This procedure returns a set of complementary captions that collectively cover image content.

3.8. Complexity, Efficiency, and Memory Footprint

Let N and M be region and token counts, and K the number of anchors. The fusion Transformer costs $\mathcal{O}\left({(N+M)}^{2}d\right)$, while the per-anchor refiner scales as $\mathcal{O}(C{d}^2+|\mathcal{G}{|}^{2}d)$, where $\left|\mathcal{G}\right|$ is typically small. Because anchors are processed independently, decoding is embarrassingly parallel across K, enabling practical multi-view generation without prohibitive latency.

3.9. Robustness Enhancements

To mitigate OCR noise, we (i) apply label smoothing on token-copy logits; (ii) add a small KL penalty between vocabulary and pointer distributions to discourage overconfident copying:

$${\mathcal{L}}_{\mathrm{kl}}=\sum _c\mathrm{KL}\left(\mathrm{softmax}{f}_4\left({\widehat{\mathbf{y}}}_c\right)\parallel \mathrm{softmax}{f}_{\mathrm{dp}}(\widehat{\mathcal{G}},{\widehat{\mathbf{y}}}_c)\right),$$

(18)

and (iii) introduce geometric jittering of OCR boxes during training to improve spatial generalization.

3.10. Training Details

Formally, we train our MACap by minimizing the composite objective:

$$\mathcal{L}={\mathcal{L}}_{\mathrm{anchor}}\left({\mathbf{s}}_{\mathrm{anc}}\right)+\alpha {\mathcal{L}}_{\mathrm{graph}}\left({\mathbf{s}}_{\mathrm{graph}}\right)+\beta {\mathcal{L}}_{\mathrm{vcap}}\left({\mathcal{Y}}^{\prime }\right)+\eta {\mathcal{L}}_{\mathrm{tcap}}\left(\mathcal{Y}\right)+\lambda {\mathcal{L}}_{\mathrm{div}}+\mu {\mathcal{L}}_{\mathrm{cov}}+\rho {\mathcal{L}}_{\mathrm{con}}+\zeta {\mathcal{L}}_{\mathrm{kl}},$$

(19)

where ${\mathcal{Y}}^{\prime }=\left\{y_c^{\prime }\right\}$ and $\mathcal{Y}=\left\{y_c\right\}$ are the visual-specific and text-specific captions derived from (11). Hyperparameters $\{\alpha ,\beta ,\eta ,\lambda ,\mu ,\rho ,\zeta \}$ balance losses. Unless otherwise stated, classification terms use binary cross-entropy. All components are trained end-to-end.

Ground-truth labels. We mine supervision signals as follows. (1) For ${\mathcal{L}}_{\mathrm{tcap}}$, we use a human caption that preserves OCR tokens. (2) For ${\mathcal{L}}_{\mathrm{vcap}}$, we mask OCR spans with [unk] in the same caption to prevent leakage of token identity into the visual draft. (3) For ${\mathcal{L}}_{\mathrm{anchor}}$, we select the most frequently referenced token across the five annotations as the positive anchor. (4) For ${\mathcal{L}}_{\mathrm{graph}}$, tokens co-occurring with that anchor within a caption are positives; others are negatives. These labels are automatically induced from the training split without any extra annotation, ensuring fair comparison with prior art.

Compared with single-caption pipelines, MACap provides an explicit route to (i) selection (anchor discovery), (ii) structuring (ACG induction), and (iii) generation (deliberation with copy). This modular yet end-to-end design translates to improved textual grounding, higher descriptive fidelity, and controlled diversity across multiple views.

4. Experiments

We conduct extensive empirical studies to validate the proposed MACap on the TextCaps benchmark [40]. All experiments strictly follow the official splits and evaluation protocol. We first summarize the dataset and comparison protocol (Section 4.1), then describe implementation details (Section 4.2). Next, we present main results and diversity analysis (Section 4.3), followed by ablations on anchor discovery, graph induction and the two-stage captioner (Section 4.4). We further provide qualitative analyses and additional diagnostics (e.g., sensitivity to the number of anchors, efficiency, error breakdown) to thoroughly characterize the behavior of MACap (Section 4.5).

4.1. Dataset, Protocol, and Metrics

TextCaps. TextCaps [40] is curated from Open Images V3 and comprises 142,040 captions over 28,408 images, each verified to contain legible text using Rosetta OCR [6] and human inspection. Every image is annotated with five independent captions; the test split additionally contains a separate reference caption to approximate human performance. Many captions require reasoning that goes beyond verbatim copying, e.g., inferring attributes from brand names or prices [40]. On average, captions contain 12.4 tokens and reference two or more OCR tokens.

Evaluation metrics. We report standard captioning metrics BLEU (B) [36], METEOR (M) [10], ROUGE_L (R) [27], SPICE (S) [1], and CIDEr (C) [43]. Following [40], CIDEr is emphasized due to its sensitivity to informative n-grams. To quantify diversity, we compute Div-n [26] (token-level diversity) and SelfCIDEr [46] (semantic diversity). We further measure Cover Ratio (CR): the proportion of unique OCR tokens included across generated captions per image. We omit percent signs for brevity in reported numbers.

Compared methods and test protocol. We compare MACap with SOTA captioners: BUTD [2], AoANet [21], and M4C-Captioner [40]. We also include a recent variant MMA-SR [45]. All methods share the same detection and OCR backbones (Faster R-CNN, Rosetta) to isolate the effect of the captioner. Results on the test split are obtained via the official TextCaps server TextCaps: https://textvqa.org/textcaps; ablations are performed on the validation split due to submission limits.

4.2. Implementation Details

Backbone and optimization. Unless stated, the embedding size is $d=768$. Each $f_{*}$ is a linear layer followed by LayerNorm [3]. We train for $12,000$ iterations with batch size 128 using Adamax [23] (lr $2\times {10}^{-4}$). The multimodal Transformer $\Psi$ adopts BERT-BASE hyperparameters [12] (12 heads), with $L_1=2$ fusion layers, and $L_2=L_3=4$ layers for the two-stage captioner. We sample $N=100$ visual regions and $M=50$ OCR tokens per image; maximum caption length $C=30$. Loss weights use $\alpha =\beta =\gamma =1$ unless otherwise specified.

Reproducibility. We rely on the official feature extractors and the fixed vocabulary provided by the benchmark [40]. Our open-source reference implementation remains available for inspection .https://github.com/guanghuixu/AnchorCaptioner.

4.3. Main Results and Diversity Analyses

Overall accuracy. We first compare MACap against strong baselines under the standard single-caption setting (top-1 anchor at inference for MACap to match single-output baselines). Table 1 and Table 2 summarize accuracy and diversity. Standard captioners (BUTD, AoANet) underperform on TextCaps due to the inability to read text, whereas M4C-Captioner benefits from OCR-pointer mechanisms. MACap further improves upon M4C-Captioner by explicitly structuring text via anchors and ACGs and by refining the visual draft with anchor-aware copying.

Diversity and coverage. We next quantify multi-view generation. Baselines (BUTD, M4C) use beam search (beam=5). For MACap, we sample five anchors (top-K) and decode five captions. MACap improves token-level and semantic diversity and substantially raises OCR coverage (CR), demonstrating that anchor conditioning increases content breadth beyond generic rephrasings.

4.4. Ablations: Anchors, Graphs, and Captioner

Anchor proposal strategies. We compare independent (FC), multi-head (Transformer), and sequence (RNN) projections within the Anchor Proposal Module (APM). The sequence model, which conditions on prior selections, yields the highest CIDEr and the best A/F1 for anchor and ACG prediction. This validates modeling inter-token dependencies during grouping.

Table 3. Ablations on APM projection heads. Sequence modeling provides consistent gains across caption metrics and A/ F1 diagnostics.

#	Projection	B	M	R	S	C	A	F1
1	Single (FC)	24.0	22.3	46.8	15.7	90.9	48.7	69.1
2	Multiple (Transformer)	23.8	22.4	46.5	16.0	91.2	49.3	69.2
3	Sequence (RNN)	24.9	22.7	47.3	16.1	96.4	49.5	71.9

Table 3. Ablations on APM projection heads. Sequence modeling provides consistent gains across caption metrics and A/ F1 diagnostics.

#	Projection	B	M	R	S	C	A	F1
1	Single (FC)	24.0	22.3	46.8	15.7	90.9	48.7	69.1
2	Multiple (Transformer)	23.8	22.4	46.5	16.0	91.2	49.3	69.2
3	Sequence (RNN)	24.9	22.7	47.3	16.1	96.4	49.5	71.9

Rule-based vs. learned ACGs. We contrast APM with rule-based heuristics for anchor selection (Large, Centre, None) and neighborhood construction (All, Around, Random). Heuristics relying on size or location underperform the learned APM even when supplied with GT anchors, underscoring the need for semantic–geometric reasoning during grouping.

Table 4. Rule-based ACG construction vs. learned APM. Learned grouping substantially outperforms heuristics, even with oracle anchors.

#	Anchor	ACG	B	M	R	S	C
1	Large	All	21.3	21.1	44.9	14.5	77.1
2		Around	21.6	21.2	45.0	14.5	78.0
3		Random	20.9	20.8	44.5	14.2	73.1
4	Centre	All	21.3	21.1	44.9	14.5	77.0
5		Around	21.7	21.3	45.1	14.5	78.6
6		Random	20.8	20.9	44.6	14.2	73.5
7	-	All	21.2	21.2	44.8	14.6	76.9
8		Random	20.5	20.6	44.2	14.0	70.7
9	GT	All	23.6	22.5	46.4	15.8	91.0
10		Around	22.3	22.0	45.7	15.3	84.4
11		Random	21.5	21.3	45.0	14.8	79.3
12	APM (learned)	APM	24.9	22.7	47.3	16.1	96.4
13	GT	GT	25.8	23.5	48.2	17.0	105.3

Table 4. Rule-based ACG construction vs. learned APM. Learned grouping substantially outperforms heuristics, even with oracle anchors.

#	Anchor	ACG	B	M	R	S	C
1	Large	All	21.3	21.1	44.9	14.5	77.1
2		Around	21.6	21.2	45.0	14.5	78.0
3		Random	20.9	20.8	44.5	14.2	73.1
4	Centre	All	21.3	21.1	44.9	14.5	77.0
5		Around	21.7	21.3	45.1	14.5	78.6
6		Random	20.8	20.9	44.6	14.2	73.5
7	-	All	21.2	21.2	44.8	14.6	76.9
8		Random	20.5	20.6	44.2	14.0	70.7
9	GT	All	23.6	22.5	46.4	15.8	91.0
10		Around	22.3	22.0	45.7	15.3	84.4
11		Random	21.5	21.3	45.0	14.8	79.3
12	APM (learned)	APM	24.9	22.7	47.3	16.1	96.4
13	GT	GT	25.8	23.5	48.2	17.0	105.3

Two-stage captioner (AnCM). We assess the contribution of the visual draft and anchor-conditioned refinement and also feed ACGs into a strong baseline (M4C-Captioner) for reference. Using predicted ACGs (†) or oracle ACGs (*) improves both M4C and MACap; the latter remains superior under matched inputs, indicating the effectiveness of anchor-aware deliberation and the pointer-copy integration.

Table 5. Ablations for the two-stage captioner. †: predicted ACGs from APM. *: oracle ACGs. MACap benefits from structured ACG inputs and remains stronger than M4C under matched conditions.

#	Method	B	M	R	S	C
1	M4C-Captioner	23.5	22.1	46.4	15.7	90.2
2	${\mathrm{M4C-Captioner}}^{†}$	24.3	22.7	46.9	15.8	94.4
3	${\mathrm{M4C-Captioner}}^{*}$	24.6	22.7	47.1	15.9	100.1
4	${\mathrm{AnCM}}_v$ + ${\mathrm{AnCM}}_t^{†}$ (MACap)	24.9	22.7	47.3	16.1	96.4
5	${\mathrm{AnCM}}_v$ + ${\mathrm{AnCM}}_t^{*}$ (MACap)	25.8	23.5	48.2	17.0	105.3

Table 5. Ablations for the two-stage captioner. †: predicted ACGs from APM. *: oracle ACGs. MACap benefits from structured ACG inputs and remains stronger than M4C under matched conditions.

#	Method	B	M	R	S	C
1	M4C-Captioner	23.5	22.1	46.4	15.7	90.2
2	${\mathrm{M4C-Captioner}}^{†}$	24.3	22.7	46.9	15.8	94.4
3	${\mathrm{M4C-Captioner}}^{*}$	24.6	22.7	47.1	15.9	100.1
4	${\mathrm{AnCM}}_v$ + ${\mathrm{AnCM}}_t^{†}$ (MACap)	24.9	22.7	47.3	16.1	96.4
5	${\mathrm{AnCM}}_v$ + ${\mathrm{AnCM}}_t^{*}$ (MACap)	25.8	23.5	48.2	17.0	105.3

4.5. Additional Diagnostics and Qualitative Analyses

Qualitative behavior. We observe that the visual draft often captures global semantics (e.g., “a storefront with products”) but omits specific text. The anchor-conditioned refiner substitutes unknown slots with copied OCR tokens and adjusts surrounding syntax to maintain fluency, not merely replacing tokens. In our validation set, $66.4\%$ of drafts contain at least one $<$unk\$>$ (avg. $1.24$ per caption). The refiner modifies $26.9\%$ of words on average and lifts CIDEr from $29.7$ (draft-only) to $96.4$ after refinement (cf. Table 1 row 5 vs. row 6).

Sensitivity to number of anchors K. We evaluate top-K decoding (validation set) to study coverage–redundancy trade-offs. Increasing K improves CR and SelfCIDEr with mild drop of per-caption CIDEr, as expected; the set-level CIDEr (max over K) increases.

Table 6. Effect of the number of anchors K on validation. Larger K yields better coverage and semantic diversity; the best single caption (max CIDEr) also improves.

K	CIDEr (avg)	CIDEr (max)	Div-2	SelfCIDEr	CR
1	96.4	96.4	44.0	58.4	38.6
3	94.9	98.7	46.8	61.2	44.3
5	93.8	99.5	47.9	63.0	48.7

Table 6. Effect of the number of anchors K on validation. Larger K yields better coverage and semantic diversity; the best single caption (max CIDEr) also improves.

K	CIDEr (avg)	CIDEr (max)	Div-2	SelfCIDEr	CR
1	96.4	96.4	44.0	58.4	38.6
3	94.9	98.7	46.8	61.2	44.3
5	93.8	99.5	47.9	63.0	48.7

Runtime and efficiency. We report average inference time per image on a single V100 (batch size 1). Parallelizing anchors keeps latency practical.

Table 7. Efficiency on validation: MACap introduces modest overhead for anchor reasoning and refinement; cost scales gently with K.

Method	K	Time (ms/img)	Memory (MB)
M4C-Captioner	1	42	780
MACap	1	55	900
MACap	3	71	970
MACap	5	86	1040

Table 7. Efficiency on validation: MACap introduces modest overhead for anchor reasoning and refinement; cost scales gently with K.

Method	K	Time (ms/img)	Memory (MB)
M4C-Captioner	1	42	780
MACap	1	55	900
MACap	3	71	970
MACap	5	86	1040

Error breakdown. Typical failure modes include: (i) OCR noise leading to near-duplicate tokens (mitigated by label smoothing and KL regularization); (ii) anchors focusing on salient but semantically redundant words (alleviated by the diversity loss); (iii) long-range associations (brand → product category) occasionally missed when the anchor neighborhood is too small—expanding neighborhood radius slightly improves SPICE by $\sim 0.2$ with negligible speed cost.

Ablation: Coverage and diversity losses. Removing ${\mathcal{L}}_{\mathrm{cov}}$ increases token repetition and reduces CR by $-3.2$. Dropping ${\mathcal{L}}_{\mathrm{div}}$ reduces Div-2 by $-2.6$ and SelfCIDEr by $-1.8$. Both losses jointly improve multi-caption quality without harming single-caption CIDEr.

Table 8. Impact of coverage/diversity regularizers under $K=5$ on validation.

Config	C (avg)	C (max)	Div-2	SelfCIDEr	CR
MACap (full)	93.8	99.5	47.9	63.0	48.7
w/o ${\mathcal{L}}_{\mathrm{cov}}$	93.9	99.2	46.5	62.2	45.5
w/o ${\mathcal{L}}_{\mathrm{div}}$	94.1	99.0	45.3	61.2	46.8

Table 8. Impact of coverage/diversity regularizers under $K=5$ on validation.

Config	C (avg)	C (max)	Div-2	SelfCIDEr	CR
MACap (full)	93.8	99.5	47.9	63.0	48.7
w/o ${\mathcal{L}}_{\mathrm{cov}}$	93.9	99.2	46.5	62.2	45.5
w/o ${\mathcal{L}}_{\mathrm{div}}$	94.1	99.0	45.3	61.2	46.8

Human comparison. Human captions achieve higher SPICE and CIDEr than all models but lower CR, reflecting a preference for salient, coherent narratives over exhaustive token coverage. MACap narrows the CIDEr gap while substantially improving coverage versus single-caption baselines.

Qualitative notes. Representative examples indicate that MACap (i) correctly copies numbers, prices, and brand names; (ii) rewrites draft sentences to integrate copied tokens with proper grammar; (iii) produces complementary captions under different anchors (e.g., one focusing on brand/price, another on location/signage), consistent with the quantitative diversity gains.

Across accuracy, diversity, coverage, and efficiency, MACap delivers consistent gains over strong baselines. Learned anchors and ACGs are essential; the two-stage captioner further converts structured textual evidence into fluent, content-rich multi-view descriptions.

4.6. Cross-Dataset Transfer and Zero-Shot Generalization

To examine whether MACap merely overfits TextCaps [40] or learns transferable text-grounded reasoning, we evaluate the pretrained model without any additional fine-tuning on two text-heavy benchmarks: (i) ST-VQA-style caption splits derived from public ST-VQA images (where available captions focus on scene text); and (ii) an internal collection of signboard and storefront photos curated from open-license sources with five human references per image. Across both corpora, MACap preserves a strong balance between lexical grounding and fluency: qualitatively, anchors gravitate toward salient tokens (brand names, prices, street names), and the refiner integrates them into coherent sentences.

We further probe compositionality by constructing cross-domain test sets with shifted token distributions (e.g., non-English scripts mixed with Arabic numerals, rare brand names, and long alphanumeric strings). Zero-shot performance degrades gracefully compared to single-caption baselines: while BLEU and ROUGE_L drop slightly due to vocabulary mismatch, SPICE and CIDEr remain comparatively robust, suggesting that anchor-centered structuring helps the model capture higher-level relations even when exact lexical matches are scarce. Interestingly, the set-level metrics (taking the best among K captions) improve more markedly than single-caption scores, indicating that multi-view decoding partially compensates for domain shifts by covering diverse hypotheses.

To disentangle the contribution of anchors vs. the two-stage decoder in transfer, we perform an oracle study: replacing APM’s anchors with heuristics (largest or central text) reduces semantic adequacy in out-of-domain scenes, where geometric prominence correlates weakly with semantic salience. In contrast, feeding MACap with oracle anchors derived from a lightweight keyword detector (trained on just a few dozen phrases) recovers most of the lost performance, underscoring that anchor quality is the principal driver of generalization. Overall, these findings support the claim that explicit anchor–graph structuring produces representations that are less brittle to distribution shifts than monolithic single-caption decoders.

4.7. Robustness Under OCR Noise and Visual Perturbations

Practical deployment faces imperfect OCR outputs and non-ideal imaging conditions. We therefore run controlled stress tests to evaluate the resilience of MACap to (i) token-level corruption and (ii) image-level perturbations. For token-level noise, we simulate character substitutions, insertions, and deletions at rates $ϵ\in \{5\%,10\%,20\%\}$, as well as space removal and case randomization. For image-level noise, we apply moderate blur, contrast shifts, JPEG compression, and geometric jitter to region proposals. Across both regimes, single-caption baselines exhibit rapid degradation, particularly when exact copying is required (prices, serial numbers). In contrast, MACap degrades more gracefully: the APM and ACG stages redistribute attention toward more reliable tokens (e.g., backing off from corrupted brand strings to nearby category or slogan text), and the refiner preserves sentence scaffolding even when specific numbers become uncertain.

Ablating the robustness components reveals their complementary roles. Removing label smoothing increases overconfidence in the pointer logits, leading to brittle copying behavior under character noise; omitting the KL consistency penalty causes the vocabulary head and pointer head to diverge, which manifests as grammatical artifacts (e.g., tense or number mismatches) when copying fails. Geometric jitter during training improves tolerance to proposal misalignment and OCR box offsets by encouraging the fusion module to integrate multiple partially overlapping cues. Together, these regularizers reduce the gap between clean and perturbed conditions and maintain higher coverage of valid OCR tokens across anchors. Qualitative inspection confirms that the model often substitutes corrupted tokens with semantically proximate alternatives (e.g., “$2.9?” → “two dollars and ninety cents”) while preserving factual plausibility, demonstrating the benefit of anchor-conditioned deliberation.

4.8. Training Data Ablation and Anchor Budget Sensitivity

We next study how MACap scales with supervision volume and with the number of anchors decoded at test time. For data ablation, we train models with $25\%$, $50\%$, $75\%$, and $100\%$ of the original training split (stratified by image). Performance grows near-logarithmically with data, but we observe disproportionately larger gains in SPICE and CIDEr when moving from $50\%$ to $75\%$, coinciding with better anchor reliability: the APM benefits from more examples of rare token patterns and long-tail co-occurrences (e.g., “lotto + jackpot”, “pharmacy + 24h”). Notably, diversity metrics (SelfCIDEr, Div-2) improve even in low-data regimes when decoding multiple anchors ($K\ge 3$), suggesting that the multi-view mechanism can partially offset data scarcity by exploring different textual neighborhoods.

For anchor budget sensitivity, we fix the trained model and vary K at inference. As discussed earlier, larger K increases coverage (CR) and set-level semantic diversity with only modest compute overhead due to per-anchor parallelism. However, we also find a diminishing return beyond $K=5$ on TextCaps: many additional anchors become semantically redundant (e.g., repeated store slogans). A simple heuristic—non-maximum suppression in embedding space for candidate anchors—removes near-duplicates before decoding and yields a cleaner caption set with similar or better set-level CIDEr. Finally, we verify that re-ranking generated captions with a length-normalized CIDEr plus a light lexical penalty improves user-facing selection without retraining, providing a practical knob for applications that must present a single caption while retaining the benefits of multi-view generation.

5. Conclusion and Future Work

In this work, we presented MACap, a comprehensive and context-aware multi-anchor captioning framework designed for the challenging TextCap task, which requires fine-grained reasoning over both visual and textual cues in images. Unlike prior methods that tend to produce a single global caption summarizing only the most salient parts of a scene, MACap explicitly models multiple localized perspectives, enabling the generation of diverse captions that cover different regions and textual entities within an image.

Existing captioning frameworks such as M4C-Captioner and AoANet often overlook the subtle and context-dependent relationships among textual elements detected by OCR systems. These models generally output one caption reflecting only the global semantics, which can cause them to miss critical textual information like brand names, numbers, or scene-specific text. In contrast, MACap introduces an anchor-based mechanism that identifies salient text tokens, groups them into anchor-centered graphs (ACGs), and leverages these structures to generate multiple semantically complementary captions.

The framework consists of two major components: the Anchor Proposal Module (APM) and the Anchor Captioning Module (AnCM). The APM dynamically selects meaningful anchor points by analyzing OCR tokens and visual objects, constructing local semantic graphs that connect related regions and textual cues. This process allows the model to reason contextually about spatial proximity and semantic relevance, rather than relying on raw OCR outputs alone. The AnCM then generates a two-stage description: a first draft caption based on visual evidence and a refined caption that incorporates textual understanding via the corresponding ACGs. Through this refinement process, the model can reinterpret and augment visual captions with text-grounded semantics, producing captions that are both accurate and contextually informative.

The proposed MACap significantly advances the expressiveness and robustness of multimodal captioning systems. Experimental results on the TextCaps benchmark demonstrate consistent improvements across all standard evaluation metrics, particularly in CIDEr, where the model surpasses the previous state of the art by over six points. Beyond numeric gains, qualitative analyses reveal that MACap captures richer contextual details—such as brand labels, signboard texts, and numerical references—that are typically overlooked by conventional models. These multi-perspective captions provide a more comprehensive understanding of the visual scene and closely mirror human descriptions.

Moreover, extensive ablation studies confirm that each module of MACap contributes meaningfully to the final performance. The Anchor Proposal Module effectively localizes text entities that are semantically relevant, while the Anchor Captioning Module demonstrates strong refinement capability, generating captions that reflect not only visual patterns but also embedded linguistic meaning. The modular nature of the system also facilitates interpretability, as each anchor-centered graph can be traced to a specific caption segment, allowing clearer analysis of model behavior and output justification.

Looking ahead, we envision several promising research directions building upon MACap. First, while our model is designed for image captioning with embedded text, its underlying anchor-graph reasoning can naturally extend to other vision–language tasks such as visual question answering, document layout understanding, and scene text interpretation. Second, integrating MACap with large-scale multimodal pretrained models (e.g., BLIP-2, GPT-4V) could further enhance generalization and zero-shot reasoning ability, allowing it to handle more open-domain tasks. Third, we plan to explore reinforcement-based training objectives that optimize for human-centric criteria, including factual correctness, coverage diversity, and linguistic coherence, thereby aligning model outputs more closely with human preferences.

Additionally, the anchor-conditioned caption generation process can be extended toward interactive and controllable captioning, where users can select or modify anchors to influence the focus of generated captions. This opens the door to user-driven and adaptive captioning systems for applications in accessibility, education, and assistive technologies. Beyond that, the interpretability of anchor graphs may enable integration with symbolic reasoning or structured knowledge retrieval systems, offering new opportunities for bridging visual perception with high-level semantic reasoning.

In conclusion, MACap provides a principled step toward fine-grained, text-aware image captioning. By explicitly modeling anchors, contextual graphs, and multi-view caption generation, it achieves a balance between visual understanding, textual grounding, and linguistic fluency. Our work highlights that capturing structured semantic relationships within text-rich images is crucial for comprehensive multimodal understanding. In future research, we aim to expand this anchor-based framework into a broader foundation for unified multimodal intelligence—capable of seamlessly integrating perception, reasoning, and language generation across diverse real-world scenarios.