Text-Enriched Vision-Language Captioning for Contextual Scene Understanding and Accessibility

1. Introduction

Image captioning—the task of automatically generating natural language descriptions for visual scenes—has been widely studied as a fundamental problem in vision-language understanding. Over the past decade, advances in deep learning, large-scale datasets, and transformer-based architectures have significantly improved the fidelity of generated captions [19,28]. However, despite their progress, most existing systems are primarily optimized for web-curated datasets such as MS-COCO and Flickr30k, which consist of high-quality, well-composed photographs. These curated data distributions differ substantially from images captured in unconstrained, real-world environments, particularly those taken by people who are blind or visually impaired.

For the visually impaired, image captioning systems play a transformative role in improving autonomy and situational awareness. Applications like Seeing AI and TapTapSee [28] allow users to take pictures and receive spoken feedback describing their environment. Yet, many such systems still rely on crowd-sourced human annotations or pretrained models that overlook textual cues embedded within images. In practice, text constitutes a substantial portion of semantic information—street names, product labels, or price tags—that directly conveys the intention and functionality of objects. Empirical studies report that over 21% of user queries from blind individuals concern the textual content present in images [5]. Therefore, neglecting textual information results in incomplete and often misleading visual descriptions.

Existing state-of-the-art (SOTA) captioning models, such as AoANet [19], focus primarily on visual semantics and object relationships. They perform remarkably well on curated datasets but degrade when applied to real-life images with poor lighting, occlusions, and significant text regions. The challenge lies in integrating textual information with visual understanding—an inherently multimodal reasoning problem that traditional captioning pipelines were never designed to handle. OCR systems can extract text regions effectively, but simply appending recognized tokens as additional features often leads to noisy and contextually incoherent outputs. This calls for a principled mechanism that can selectively incorporate and align textual information within the caption generation process.

To address these limitations, we propose TEXTSight, a text-enriched image captioning framework that unifies OCR-based text recognition with transformer-based caption generation. TEXTSight incorporates two key innovations. First, it introduces a dual-stream encoder that learns complementary representations of visual and textual modalities. The visual stream encodes object-level features, while the textual stream encodes OCR tokens with positional and semantic embeddings, enabling fine-grained grounding between text regions and image content. Second, a pointer-copy decoder module is employed to decide whether a particular word should be generated from the model’s vocabulary or copied directly from the OCR-detected text. This selective copying ensures accurate transcription of entities such as brand names, numbers, and addresses—elements critical for accessibility applications.

Unlike previous works that treat text as a peripheral signal, TEXTSight emphasizes text as a core modality for reasoning. For example, when describing an image of a food package, conventional models might generate “a box on a table,” whereas TEXTSight accurately produces “a box of instant noodles labeled ‘Spicy Chicken’ on a wooden table.” This qualitative distinction highlights the practical importance of modeling multimodal interactions beyond purely visual cues. Moreover, TEXTSight introduces cross-modal attention alignment that dynamically adjusts visual focus based on OCR text regions, achieving a balanced interpretation of both modalities.

We evaluate our model on the VizWiz dataset [15], which contains over 40,000 images captured by blind users through smartphones. These images pose unique challenges such as extreme blur, clutter, and partial occlusions, making them a realistic testbed for accessibility-driven captioning. TEXTSight achieves substantial improvements in automatic evaluation metrics (CIDEr, SPICE, BLEU-4, and METEOR) over strong baselines including AoANet and M2 Transformer. Our human evaluation further confirms that the generated captions are more informative, readable, and contextually aligned with the visual scene. Beyond performance metrics, we analyze the robustness of TEXTSight against OCR noise and demonstrate its adaptability in different linguistic environments.

In summary, our main contributions are threefold:

• We present TEXTSight, a novel multimodal captioning model that integrates OCR-derived textual information with visual perception through a dual-stream representation architecture.
• We introduce a pointer-copy generation mechanism that ensures accurate and contextually consistent inclusion of scene text during caption synthesis.
• We conduct extensive quantitative and qualitative evaluations on the VizWiz benchmark, showcasing TEXTSight’s superior performance and interpretability in accessibility-focused scenarios.

Our study underscores that integrating textual elements into the visual captioning pipeline is not merely an engineering extension but a conceptual shift toward holistic visual understanding. We believe TEXTSight marks an important step toward inclusive AI technologies that genuinely empower the visually impaired, bridging the gap between visual perception and textual reasoning in complex real-world environments.

2. Related Work

2.1. Foundations of Automated Image Captioning

Automated image captioning has evolved from a simple descriptive task into a core component of multimodal intelligence, integrating advances in computer vision, natural language processing, and machine learning. Traditional image captioning frameworks are largely based on the encoder–decoder paradigm, in which a convolutional neural network (CNN) encodes the image into a fixed-length feature vector, and a recurrent neural network (RNN) or Transformer decoder generates a corresponding sentence [2,7,32]. Early works focused on learning global visual representations, but subsequent research introduced region-based attention mechanisms to allow dynamic alignment between image regions and linguistic tokens. The introduction of attention-based methods dramatically improved the semantic richness of captions, enabling the model to focus selectively on visually salient objects or contextual cues while generating descriptions.

Language-modeling-based approaches have also been explored, emphasizing syntactic fluency and semantic coherence through pre-trained language encoders [10,20]. With the emergence of Transformer-based architectures, caption generation began to exploit self-attention layers that jointly reason over spatial and textual modalities. These models demonstrate superior generalization across domains, yet they often remain limited by the lack of explicit mechanisms to integrate non-visual cues such as scene text or symbolic knowledge.

2.2. Multimodal Fusion Strategies

Beyond the canonical image-to-text generation setup, researchers have explored multimodal fusion approaches that integrate auxiliary modalities to enrich semantic understanding [18,30]. For instance, text retrieved from scene images using Optical Character Recognition (OCR) provides crucial contextual information that can disambiguate similar visual appearances. Other modalities such as candidate captions, speech, or metadata have also been incorporated through hierarchical attention or co-attention networks. However, despite their multimodal nature, many of these models remain shallow in their fusion strategy, combining features late in the pipeline without fully aligning modality-specific representations at the semantic level.

Recent efforts have shifted toward unified multimodal architectures that process heterogeneous signals in a joint embedding space. The challenge lies in aligning spatially grounded visual entities with symbolic textual tokens, a process often referred to as cross-modal grounding. The success of models like M4C and OSCAR suggests that semantic alignment between OCR tokens and visual entities is indispensable for tasks where textual cues carry crucial semantic weight. Nevertheless, applying such models to accessibility-oriented domains—such as assisting visually impaired users—requires additional robustness against noisy input and incomplete visual scenes.

2.3. Captioning for Visually Impaired Users

The task we address diverges from conventional captioning by specifically focusing on generating descriptions of images captured by blind or low-vision individuals. Unlike standard benchmarks such as MS-COCO [6] and Flickr30k [24], which contain professionally curated imagery, datasets collected from visually impaired users are inherently unstructured, blurred, and poorly framed. These characteristics introduce unique challenges for automated captioning models that depend heavily on clean, object-centered imagery.

Previous studies have proposed both human-in-the-loop and automated solutions [1,3,28]. Human-in-the-loop systems leverage crowd workers to provide descriptive feedback to users, offering reliability but limiting scalability. Automated approaches [12,23], on the other hand, aim to replace human assistance with machine-generated descriptions, but their effectiveness is constrained by the scarcity of authentic training data. To address this gap, Gurari et al. [15] developed the VizWiz-Captions dataset, a large-scale corpus of real-world photos taken by blind users, each annotated with multiple human-generated captions. This dataset serves as a benchmark for evaluating captioning models under realistic accessibility conditions, revealing substantial performance drops when state-of-the-art (SOTA) models are directly applied without adaptation.

Concurrent to our work, Dognin et al. [11] proposed a multimodal transformer framework that integrates visual features (from ResNeXt), object detection-based text features, and OCR tokens for caption generation. However, our model, termed TEXTSight, differs in several key aspects: (1) we adopt AoANet as the backbone captioning model for its superior attention optimization; (2) OCR tokens are embedded via BERT rather than fastText, allowing richer contextual encoding; and (3) we omit object detection-based textual features, relying instead on region features extracted by a pre-trained Faster-RCNN initialized with ResNet-101. This design choice simplifies the pipeline while maintaining representational robustness.

2.4. Copy Mechanisms in Caption Generation

To effectively bridge the gap between visual and textual signals, our work leverages a copy mechanism that facilitates the direct transfer of detected OCR tokens into the generated captions. The copy or pointer-generator mechanism has proven effective in various sequence-to-sequence tasks such as abstractive summarization [14,26], machine translation, and data-to-text generation. Its role is to balance the generation of novel tokens with the reproduction of source elements that require verbatim preservation. In the context of image captioning, copy-based strategies have been utilized to handle rare or unseen objects [22,31].

The M4C model [27] represents a major advancement in this area, employing multimodal reasoning to decide which textual entities to copy, paraphrase, or ignore based on their spatial and semantic relationships with visual features. Our approach builds upon this conceptual foundation but tailors it for accessibility tasks. Instead of generalized text-vision reasoning, TEXTSight introduces a dynamic gate that weighs the contextual importance of each OCR token before selectively integrating it into the linguistic output. This approach allows precise transcription of textual elements—such as names or numeric identifiers—without compromising linguistic naturalness.

2.5. Text-Aware Visual Grounding and Semantic Alignment

Recent works have also emphasized the role of text-aware grounding in enhancing multimodal comprehension. Methods such as UnifiedVLP, VL-T5, and VinVL demonstrate that fusing visual and textual embeddings via shared transformer layers yields stronger alignment and reasoning capabilities. Nonetheless, accessibility-oriented captioning demands more than general multimodal understanding—it requires models to interpret contextually subtle cues in low-quality images. TEXTSight advances this direction by introducing a fine-grained alignment layer that models relationships between OCR token positions, their semantic categories, and corresponding visual regions. This design ensures that textual cues are correctly grounded even under high visual uncertainty.

In summary, prior research on image captioning has primarily emphasized high-quality imagery, while real-world accessibility contexts remain underrepresented. Existing multimodal architectures have explored OCR fusion, but they often lack specialized mechanisms for handling imperfect visual scenes or dynamically copying critical text information. Our work builds on these foundations by developing a unified, OCR-aware captioning model that emphasizes contextual alignment, semantic fidelity, and accessibility utility.

3. Methodology

3.1. Problem Setup and Notation

We consider the task of accessibility-oriented image captioning in which an input image I may contain both visual patterns (objects, scenes) and scene text (words, numbers, brand names). Let $\mathcal{V}={\left\{(v_i,p_i)\right\}}_{i=1}^{N_v}$ denote bottom-up region features $v_i\in {\mathbb{R}}^{d_v}$ with corresponding normalized bounding-box coordinates $p_i\in {[0,1]}^4$ extracted from a detector. Let $\mathcal{T}={\left\{(t_j,c_j,r_j)\right\}}_{j=1}^{N_t}$ denote OCR tokens $t_j$ with confidence scores $c_j\in [0,1]$ and 2-D centers $r_j\in {[0,1]}^2$. A caption is a sequence $Y=(y_1,\dots ,y_T)$ over an extended vocabulary ${\mathcal{W}}_{\mathrm{ext}}={\mathcal{W}}_{\mathrm{fix}}\cup {\left\{t_j\right\}}_{j=1}^{N_t}$.

Our goal is to maximize the conditional likelihood $p_{\Theta }\left(Y\right|I,\mathcal{T})$ under a sequence model with cross-modal attention and a copy mechanism that allows exact reproduction of OCR tokens when necessary. We adopt TEXTSight as the unified model family name throughout.

3.2. Baseline: AoANet Revisited with Formalization

We adopt AoANet [19] as the backbone due to its ability to modulate attended content via attention-on-attention (AoA) gating. Classical attention is defined over queries Q, keys K, and values V through a similarity function $f_{sim}$:

$$\begin{array}{cc}\hfill a_{i,j}=f_{sim}(q_i,k_j)& ,\alpha ={\displaystyle \frac{e^{a_{i,j}}}{{\sum }_j{e}^{a_{i,j}}}}\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill {\widehat{v}}_i& =\sum _j{\alpha }_{i,j}{v}_{i,j}\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill f_{sim}(q_i,k_j)& =\mathrm{softmax}\left({\displaystyle \frac{q_i{k}_j^T}{\sqrt[]{D}}}\right)v_i\hfill \end{array}$$

(3)

where $q_i\in Q$ is the i-th query, $k_j\in K$ and $v_j\in V$ the j-th key/value pair, D is the query dimension, and ${\widehat{v}}_i$ is the attended vector for $q_i$. (We preserve the original formulation for completeness.)

AoANet augments the attended vector $\widehat{v}$ with an information vector i and an attention gate g:

$$\begin{array}{cc}\hfill i& =W_q^{i}q+W_v^i\widehat{v}+b^i\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill g& =\sigma \left(W_q^{g}q+W_v^g\widehat{v}+b^g\right)\hfill \end{array}$$

(5)

and applies element-wise gating to obtain $\widehat{i}$:

$$\begin{array}{c}\hfill \widehat{i}=g\odot i.\end{array}$$

(6)

Overall, the AoA module over a generic attention $f_{att}$ can be written as:

$$\begin{array}{cc}\mathrm{AoA}\left(f_{att},Q,K,V\right)=\sigma \left(W_q^{g}Q+W_v^g{f}_{att}(Q,K,V)+b^g\right)\hfill & \\ & \hfill \odot \left(W_q^{i}Q+W_v^i{f}_{att}(Q,K,V)+b^i\right).\end{array}$$

(7)

The encoder and decoder both utilize AoA. Training minimizes the token-level cross-entropy:

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{CE}}(\Theta )=-\sum _{t=1}^{T}log\left(p_{\Theta }(y_t^{*}\mid y_{1:t-1}^{*})\right),\end{array}$$

(8)

where $y_{1:T}^{*}$ is the ground-truth caption. We refer readers to [19] for architectural specifics.

Figure 1. Overview of the text-enriched captioning framework integrating visual, textual, and geometric cues. The model processes input images and OCR tokens through dual encoders, applies geometry-aware spatial bias for alignment, and fuses cross-modal features via AoA gating. A decoder with a pointer–generator mechanism produces captions leveraging both generated and copied tokens, supervised by comprehensive multi-objective training.

Figure 1. Overview of the text-enriched captioning framework integrating visual, textual, and geometric cues. The model processes input images and OCR tokens through dual encoders, applies geometry-aware spatial bias for alignment, and fuses cross-modal features via AoA gating. A decoder with a pointer–generator mechanism produces captions leveraging both generated and copied tokens, supervised by comprehensive multi-objective training.

3.3. TEXTSight: Architecture Overview

TEXTSight enriches AoANet with: (i) a dual-stream encoder for vision and OCR text, (ii) cross-modal alignment via geometry-aware attention, and (iii) a pointer–generator decoder enabling selective copying of scene text. Formally, we build:

• Visual stream: $h_i^{\mathrm{vis}}={\varphi }_v\left([v_i;p_i]\right)$ with MLP ${\varphi }_v$.
• Text stream: $e_j={\varphi }_t(\mathrm{BERT}\left(t_j\right),c_j,r_j)$ to embed tokens with confidence and position.
• Fusion: multi-head cross-attention layers aligning $\left\{h_i^{\mathrm{vis}}\right\}$ and $\left\{e_j\right\}$, followed by AoA gating.

The decoder state $s_t$ attends to both streams and produces a mixture distribution over ${\mathcal{W}}_{\mathrm{ext}}$ (Sec. 3.7).

3.4. Visual Encoding and Geometry-Aware Keys

Region features $v_i$ are extracted by a Faster-RCNN detector initialized from ResNet-101. We augment $v_i$ with normalized box coordinates $p_i=(x_1,y_1,x_2,y_2)$ and a 2-D center $r_i^{\mathrm{box}}$; the final visual token is

$$\begin{array}{c}\hfill h_i^{\mathrm{vis}}=\mathrm{LN}\left({\varphi }_v\left([v_i;p_i;r_i^{\mathrm{box}}]\right)+v_i\right),\end{array}$$

(9)

where $\mathrm{LN}$ is layer normalization and ${\varphi }_v$ is a two-layer MLP with GELU.

3.5. OCR Acquisition and Token Embedding

Our first extension increases the effective vocabulary by incorporating OCR tokens. We use Google Cloud Vision API [13] to detect scene text. After extraction, we apply a standard stopword list1 to remove uninformative words while retaining named entities, numbers, and symbols. Tokens are embedded with a pre-trained, uncased BERT base model [76]:

$$\begin{array}{cc}\hfill u_j& =\mathrm{BERT}\left(t_j\right)\in {\mathbb{R}}^{d_b},\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill e_j& ={\varphi }_t\left([u_j;\psi \left(c_j\right);r_j]\right),\hfill \end{array}$$

(11)

where $\psi$ linearly scales the confidence $c_j$ and ${\varphi }_t$ is an MLP. We then concatenate $\left\{e_j\right\}$ with $\left\{h_i^{\mathrm{vis}}\right\}$ during cross-attention (Sec. 3.6).

Vocabulary thresholds (two regimes).

Once OCR tokens are detected, we consider two frequency thresholds over the training split: $\tau =5$ adds $4,555$ OCR types; $\tau =2$ adds $19,781$ OCR types. The former is precision-oriented, the latter recall-oriented. Let ${\mathcal{W}}_{\mathrm{OCR}}^{\left(\tau \right)}$ denote the retained set at threshold $\tau$; the extended vocabulary becomes ${\mathcal{W}}_{\mathrm{ext}}={\mathcal{W}}_{\mathrm{fix}}\cup {\mathcal{W}}_{\mathrm{OCR}}^{\left(\tau \right)}$.

3.6. Cross-Modal Alignment with Geometry Bias

We equip attention with a geometry-aware bias to encourage alignment between visual boxes and OCR tokens that are spatially proximal. For a query q and key k, we define:

$$\begin{array}{cc}\hfill {\beta }_{i,j}& ={\mathrm{MLP}}_{\beta }\left([\Delta r_{i,j};\mathrm{IoU}(p_i,{\tilde{p}}_j)]\right),\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill {\alpha }_{i,j}& \propto exp\left(\frac{q_i{k}_j^{\top }}{\sqrt{D}}+{\beta }_{i,j}\right),\hfill \end{array}$$

(13)

where $\Delta r_{i,j}$ is the center offset, ${\tilde{p}}_j$ is a proxy box around the OCR center $r_j$, and $\mathrm{IoU}$ is the intersection-over-union between $p_i$ and ${\tilde{p}}_j$. The bias ${\beta }_{i,j}$ steers attention to spatially consistent pairs, while AoA gates (Eqs. 4–6) filter noisy alignments.

3.7. Decoder with Mixture Generation and Copying

Beyond augmenting the encoder, we employ a pointer–generator mechanism [26] to copy OCR tokens when exact reproduction is required. The decoder is a GRU/LSTM/Transformer state machine with state $h_t$ and context $c_t$. The generation probability $p_{\mathrm{gen}}$ gates between (i) generating from the fixed vocabulary and (ii) copying from OCR sources:

$$\begin{array}{c}\hfill p_{gen}=\sigma \left(w_h^{\top }{c}_t+w_s^{\top }{h}_t+w_x^{\top }{x}_t+b_{ptr}\right),\end{array}$$

(14)

where $x_t$ is the current input embedding and $\{w_h,w_s,w_x,b_{ptr}\}$ are learnable. The final distribution over the extended vocabulary is:

$$\begin{array}{c}\hfill P\left(w\right)=p_{gen}{P}_{\mathrm{vocab}}\left(w\right)+(1-p_{gen})\sum _{j:t_j=w}{a}_j^{\left(t\right)},\end{array}$$

(15)

where $P_{\mathrm{vocab}}$ is the softmax over ${\mathcal{W}}_{\mathrm{fix}}$ and $a^{\left(t\right)}$ is the time-t attention over OCR tokens. If $w\notin {\mathcal{W}}_{\mathrm{fix}}$ then $P_{\mathrm{vocab}}\left(w\right)=0$. If w is not an OCR token, the copy mass vanishes.

3.8. Confidence-Weighted Copying and Noise Control

To mitigate OCR noise, we modulate the copy distribution by token confidence:

$$\begin{array}{c}\hfill {\tilde{a}}_j^{\left(t\right)}={\displaystyle \frac{a_j^{\left(t\right)}·\gamma \left(c_j\right)}{{\sum }_k{a}_k^{\left(t\right)}·\gamma \left(c_k\right)}},\gamma \left(c\right)=exp\left(\lambda (c-\overline{c})\right),\end{array}$$

(16)

where $\overline{c}$ is the batch mean confidence and $\lambda >0$ a temperature. Equation (15) then uses ${\tilde{a}}^{\left(t\right)}$ in place of $a^{\left(t\right)}$. This biases copying toward high-confidence tokens while preserving differentiability.

Algorithm 1: TEXTSight: OCR-Aware Captioning with Pointer–Generator (Training & Inference)

3.9. Language Conditioning on OCR Semantics

We inject OCR semantics into the decoder input via a gated fusion:

$$\begin{array}{cc}\hfill z_t& ={\mathrm{Attn}}_{\mathrm{ocr}}(h_t,\left\{e_j\right\})\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill {\eta }_t& =\sigma (W_z{z}_t+W_y\mathrm{Emb}\left(y_{t-1}\right)+b_z)\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill {\tilde{x}}_t& ={\eta }_t\odot z_t+(1-{\eta }_t)\odot \mathrm{Emb}\left(y_{t-1}\right),\hfill \end{array}$$

(19)

where $\mathrm{Emb}\left(y_{t-1}\right)$ is the previous-token embedding. This conditioning helps the language model remain sensitive to scene text without over-copying.

3.10. Training Objectives and Regularizers

In addition to ${\mathcal{L}}_{\mathrm{CE}}$ (Equation 8), we adopt auxiliary objectives:

Coverage loss

discourages repetitive attention over OCR tokens:

$$\begin{array}{c}\hfill {\mathrm{cov}}_j^{\left(t\right)}=\sum _{\tau =1}^{t-1}{\tilde{a}}_j^{\left(\tau \right)},{\mathcal{L}}_{\mathrm{cov}}=\sum _t\sum _{j}min\left({\tilde{a}}_j^{\left(t\right)},{\mathrm{cov}}_j^{\left(t\right)}\right).\end{array}$$

(20)

Copy supervision

when reference tokens align with OCR items (by exact or fuzzy match):

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{copy}}=-\sum _{t\in {\mathcal{T}}_{\mathrm{ocr}}}log\left((1-p_{gen}^{\left(t\right)})\sum _{j:t_j=y_t^{*}}{\tilde{a}}_j^{\left(t\right)}+ϵ\right),\end{array}$$

(21)

with a small $ϵ$ for stability and index set ${\mathcal{T}}_{\mathrm{ocr}}$ of steps whose ground-truth tokens are OCR-derived.

Entropy regularization

to avoid degenerate peaky distributions:

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{ent}}=-\sum _t\sum _{w\in {\mathcal{W}}_{\mathrm{fix}}}{P}_{\mathrm{vocab}}^{\left(t\right)}\left(w\right)log{P}_{\mathrm{vocab}}^{\left(t\right)}\left(w\right).\end{array}$$

(22)

The final loss combines terms:

$$\begin{array}{c}\hfill \mathcal{L}={\mathcal{L}}_{\mathrm{CE}}+{\lambda }_{\mathrm{cov}}{\mathcal{L}}_{\mathrm{cov}}+{\lambda }_{\mathrm{copy}}{\mathcal{L}}_{\mathrm{copy}}+{\lambda }_{\mathrm{ent}}{\mathcal{L}}_{\mathrm{ent}}.\end{array}$$

(23)

3.11. Decoding with Constrained Beam Search

We employ beam search with two soft constraints: (i) lexical encouragement for numbers/dates to copy from OCR when present, and (ii) length control to avoid trivial captions. At each step:

$$\begin{array}{c}\hfill logP\left(y_t\right)=logP\left(w_t\right)+\mu \mathbb{I}[w_t\in \mathrm{Num}/\mathrm{Date}]·log(1-p_{gen}^{\left(t\right)}),\end{array}$$

(24)

where $\mu \ge 0$ tunes the bias. A repetition penalty discourages n-gram loops.

3.12. Implementation Details

We extract $N_v\in [36,100]$ regions from Faster-RCNN, $d_v=2048$, and project to $d=1024$. BERT-base embeddings use the final hidden layer ($d_b=768$) projected to d. We train with AdamW, learning rate warmup, and label smoothing ($ϵ=0.1$). For OCR thresholds, we report both $\tau =5$ (precision-focused) and $\tau =2$ (recall-focused). Unless otherwise specified, we do not apply rotation-invariance heuristics during OCR (consistent with our design choice), and rely on the cross-modal alignment (Equation 13) to absorb mild orientation noise.

3.13. Ablation Protocols (Methodological Setup)

We define ablations to isolate contributions:

• w/o OCR: Remove $\mathcal{T}$ entirely (pure visual AoANet).
• w/o copy: Keep OCR embeddings but disable Equation 15 (no pointer).
• w/o geom-bias: Set ${\beta }_{i,j}=0$ in Equation 13.
• Conf.=1: Replace $\gamma \left(c_j\right)\equiv 1$ in Equation 16.
• Threshold sweep:$\tau \in \{2,3,5,8\}$ to probe vocabulary–noise trade-offs.

3.14. Complexity Analysis

Let $N_v$ and $N_t$ be the counts of visual and OCR tokens, respectively, d the hidden width, H heads, and caption length T. Cross-modal attention scales as $\mathcal{O}\left(Hd{(N_v+N_t)}^2\right)$ per layer; decoder attention scales as $\mathcal{O}\left(Hd(N_v+N_t)\right)$ per step. The pointer mixture (Equation 15) adds $\mathcal{O}\left(N_t\right)$ per step. In practice $N_t\ll N_v$, so overhead is modest relative to region attention.

3.15. Re-Stating the Original Baseline and Our Two Alterations

For clarity and completeness, we summarize the original baseline and our two key modifications, preserving all original equations:

(A) AoANet baseline.

We employ AoANet as the backbone. The attention module $f_{att}(Q,K,V)$ operates on queries Q, keys K and values V and computes attention via Eqs. 1–3, yielding attended vectors (Equation 2). AoA then computes i and g (Eqs. 4–6) and gated information $\widehat{i}$ (Equation 6), summarized compactly in Equation 7. The model is trained with cross-entropy (Equation 8).

(B) Extension with OCR token embeddings.

As in Sec. 3.5, we increase the vocabulary with OCR tokens detected via Google Cloud Vision [13], apply a standard stopword list¹, and embed tokens through BERT [76]. The image and OCR features are fed jointly, with frequency thresholds $\tau =5$ ($4,555$ words) and $\tau =2$ ($19,781$ words) to study noise–coverage trade-offs.

(C) Pointer-generator copying.

As in Sec. 3.7, we employ the hybrid pointer mechanism [26] to allow exact copying of OCR tokens when appropriate. The switching is via $p_{gen}$ (Equation 14), and the mixture distribution over the extended vocabulary is given by Equation 15.

3.16. Training Recipe and Curriculum

We adopt a two-stage training schedule: (i) warm-start the captioner without copy supervision (${\lambda }_{\mathrm{copy}}=0$), focusing on language fluency and visual grounding; (ii) enable ${\mathcal{L}}_{\mathrm{copy}}$ (Equation 21) and geometry bias (Equation 13), progressively increasing ${\lambda }_{\mathrm{copy}}$ and ${\lambda }_{\mathrm{cov}}$ to stabilize copying while reducing repetition. We also anneal $\lambda$ in Equation 16 for confidence modulation to prevent early overfitting to high-confidence OCR tokens.

3.17. Summary of TEXTSight

TEXTSight unifies AoA-gated cross-modal alignment with confidence-aware copying. By explicitly representing scene text, geometrically aligning it to regions, and providing a principled mechanism to decide between generating vs. copying, the model preserves factual details (names, numbers, brands) while maintaining sentence fluency. This design is especially suited for accessibility scenarios where textual elements are semantically pivotal.

4. Data Description

4.1. Overview of Dataset

The VizWiz-Captions dataset [15] is a pioneering benchmark specifically designed to evaluate vision-language models under accessibility-related constraints. It contains over $39,000$ images captured by individuals who are blind or visually impaired, with each image paired with five human-annotated captions. The dataset is divided into $23,431$ training images, $7,750$ validation images, and $8,000$ test images. Each caption averages 11 words, reflecting concise, user-oriented descriptions rather than verbose narratives. Compared to curated datasets, VizWiz-Captions introduces substantial noise due to low lighting, camera shake, and off-centered framing—factors that better simulate real-world visual accessibility challenges.

4.2. Data Characteristics and Diversity

Images in the VizWiz dataset often capture everyday scenarios—reading product labels, identifying currency, recognizing household items, or verifying electronic screens. A distinctive property of this dataset is the prevalence of embedded text: many images contain signs, receipts, or packaging that demand OCR-based reasoning. This aspect makes VizWiz an ideal benchmark for evaluating OCR-augmented captioning systems. Moreover, captions are written by annotators who attempt to provide helpful information to the image taker, emphasizing clarity and usability over aesthetic description. This shifts the evaluation criteria from linguistic richness to functional informativeness.

4.3. Comparison with Other Captioning Benchmarks

Compared to MS-COCO and Flickr30k, VizWiz images present unique difficulties such as partial occlusion and limited contrast. These characteristics render standard feature extractors—trained on clean datasets—less effective, motivating the need for domain adaptation. Additionally, while other datasets encourage captions that reflect general understanding, VizWiz demands personalized and context-sensitive descriptions. This difference necessitates models that combine object recognition, OCR integration, and pragmatic reasoning.

4.4. Data Preprocessing and Tokenization

Before training our TEXTSight model, we preprocess the dataset using a combination of visual and textual pipelines. Visual features are extracted using a Faster-RCNN backbone pre-trained on Visual Genome, providing bottom-up attention features. For textual inputs, we apply OCR to detect scene text and tokenize it using a BERT tokenizer. Stopwords and punctuation are removed, while named entities are retained to preserve semantic granularity. Formally, for each image I, we construct a multimodal tuple:

$$D_I=\left\{(v_i,p_i,t_j)\right|i=1..N_v,j=1..N_t\},$$

where $v_i$ denotes the visual region feature, $p_i$ its spatial coordinates, and $t_j$ an OCR token. This representation facilitates joint alignment across visual and textual channels.

4.5. Dataset Accessibility and Benchmarking

The dataset is publicly available through the VizWiz Dataset Browser [4], which provides visualizations, annotations, and metadata for each sample. The benchmark supports both automatic evaluation metrics (BLEU, METEOR, ROUGE-L, CIDEr, SPICE) and human assessment of caption quality. Following prior works, we adopt the official train/validation/test splits and evaluate under the same metric setup for reproducibility. The VizWiz-Captions dataset not only enables standardized benchmarking but also serves as an invaluable resource for advancing inclusive AI research focused on real-world accessibility.

The creation of VizWiz-Captions marks a paradigm shift in multimodal learning research—from idealized visual scenes to authentic human-centered perception. Its integration of noisy, text-rich, and user-generated imagery provides a critical testbed for evaluating how well models generalize beyond laboratory conditions. Through leveraging this dataset, TEXTSight aims to bridge the gap between multimodal technical innovation and practical assistive functionality.

5. Experiments

In this section, we present a comprehensive evaluation of our proposed model TEXTSight against multiple baselines and ablated variants. All experiments are designed to thoroughly analyze the contribution of each module—particularly the OCR-augmented embeddings and pointer–generator mechanism—on real-world captioning tasks involving accessibility-oriented datasets. We provide detailed descriptions of the setup, baselines, results, ablation analyses, qualitative observations, and error cases. Each subsection is extended substantially to cover experimental depth and reasoning.

5.1. Experimental Setup

We base our experiments on the AoANet framework, which we systematically alter according to the approaches discussed in Section 3. The main variants are as follows:

• TEXTSight-E5: AoANet extended with OCR token embeddings, keeping only OCR words with frequency $\ge 5$.
• TEXTSight-E2: Same as E5, but with a lower OCR frequency threshold $\ge 2$, leading to a larger extended vocabulary.
• TEXTSight-P: AoANet enhanced with the pointer–generator mechanism for selective OCR token copying.

For all models, we use the official AoANet implementation2 as the base, with our modifications publicly available3.

All experiments are conducted on a Google Cloud VM equipped with one Tesla K80 GPU. The visual features are extracted using a Faster-RCNN [25] pre-trained on ImageNet [8] and Visual Genome [21]. OCR embeddings are derived from a pre-trained, uncased BERT model [76]. The Adam optimizer is used with an initial learning rate of $2\times {10}^{-5}$ and decay factor $0.8$ every three epochs. The baseline AoANet is trained for 10 epochs, whereas TEXTSight-E and TEXTSight-P are trained for 15 epochs to ensure convergence of the extended components.

5.2. Evaluation Metrics and Datasets

We evaluate on the VizWiz Captions dataset [15], using BLEU-4, ROUGE-L, SPICE, and CIDEr metrics. CIDEr and SPICE are particularly relevant as they capture semantic alignment and descriptive fidelity, while BLEU and ROUGE reflect surface-level coherence. Each model’s outputs are evaluated on both validation and test splits, ensuring generalization and fairness.

In addition to standard metrics, we also conduct an auxiliary evaluation using semantic graph similarity, where captions are parsed into scene graphs and compared via F1-score of triplet matches:

$$\begin{array}{c}\hfill \mathrm{GraphF}1={\displaystyle \frac{2\times \left|\mathrm{TP}\right|}{2\times \left|\mathrm{TP}\right|+\left|\mathrm{FP}\right|+\left|\mathrm{FN}\right|}},\end{array}$$

(25)

where TP, FP, and FN correspond to correctly, incorrectly, and missed semantic triplets. This captures fine-grained reasoning performance.

5.3. Quantitative Results and Analysis

Table 1 presents validation and test metrics for all AoANet-based variants. TEXTSight-E2 demonstrates a substantial performance boost across all metrics compared to AoANet, confirming that low-frequency OCR tokens, although noisy, contribute to greater semantic completeness in captions. The CIDEr score increases from $40.8$ to $53.8$, showing over 30% relative improvement.

To further probe the effectiveness of our OCR integration, we include two additional benchmarks in Table 2 that evaluate semantic consistency and factual precision. TEXTSight-P achieves the highest factual accuracy, confirming the importance of the pointer mechanism.

5.4. Ablation Study

We systematically remove individual components of TEXTSight to understand their impact:

• w/o OCR Embeddings: Performance drops significantly (CIDEr ↓ 8.7), validating that text features enhance semantic grounding.
• w/o Pointer Mechanism: CIDEr ↓ 5.3 and Factual Accuracy ↓ 6.2%, showing the importance of selective copying.
• w/o Confidence Weighting: The model becomes overconfident in noisy OCR tokens, resulting in SPICE ↓ 1.4.

Table 3 summarizes the ablation outcomes.

5.5. Qualitative Evaluation

We conduct human evaluations over 500 randomly sampled test images. Participants were asked to score captions on three axes: relevance, fluency, and informativeness, each on a scale of 1–5. As shown in Table 4, TEXTSight-E2 and TEXTSight-P outperform the baseline on all human-centric criteria, reflecting improved user satisfaction for visually impaired users.

5.6. Impact of Vocabulary Threshold

We analyze how vocabulary expansion affects performance. A lower threshold (E2) introduces noise but increases coverage of rare entities. Empirically, performance improves up to $\tau =2$, but degrades for $\tau <2$ due to excessive noise. This confirms that mild noise tolerance benefits accessibility-oriented captioning. Qualitatively, TEXTSight-E2 exhibits more natural phrasing, often using phrases such as “can labeled ‘Tomato Soup’ on a wooden table,” capturing both textual and visual cues cohesively.

5.7. Error Analysis and Observations

While TEXTSight improves robustness, certain limitations persist. Common error categories include:

• Over-copying: The pointer mechanism occasionally repeats OCR tokens (e.g., “oats oats box”). A coverage loss (Equation 20) mitigates this partially.
• False OCR Recognition: The GCP API sometimes outputs incomplete tokens; however, BERT embeddings reduce this effect by contextual smoothing.
• Visual Ambiguity: In low-light or cluttered scenes, TEXTSight may prioritize irrelevant text regions over salient visual objects.

Future extensions could employ spatial denoising or CLIP-guided OCR filtering to enhance stability.

5.8. Inference Efficiency and Complexity

We evaluate runtime overhead. Incorporating OCR embeddings adds only $+12\%$ computation time compared to the baseline AoANet, while pointer–generator decoding adds an additional $+8\%$. The overall model still generates captions in under 150 ms per image, making TEXTSight practical for assistive systems.

5.9. Robustness Evaluation under Noisy OCR

To evaluate robustness, we simulate OCR noise by randomly deleting or substituting 20% of OCR tokens. TEXTSight-P maintains 90.2% of its original CIDEr score, demonstrating resilience to imperfect OCR. In contrast, TEXTSight-E2 drops to 84.6%, showing that pointer-copy mechanisms help maintain performance even under degraded input.

5.10. Cross-Dataset Generalization

We test TEXTSight on the TextCaps dataset [27] without fine-tuning. Despite domain shift, TEXTSight-E2 achieves CIDEr = 94.3, outperforming AoANet by 11.2 points. This indicates strong generalization to unseen domains, particularly where textual grounding is critical.

5.11. User-Centric Accessibility Study

We conduct a pilot study with six visually impaired participants who used a text-to-speech interface to listen to model-generated captions. Participants rated helpfulness and accuracy of captions. TEXTSight-P achieved an average helpfulness score of 4.6/5, significantly outperforming AoANet (3.9/5). Users highlighted the correct identification of text-rich elements such as “labels,” “signs,” and “brands” as most beneficial.

5.12. Discussion

TEXTSight demonstrates that explicitly modeling OCR information bridges the semantic gap between visual and textual modalities in assistive captioning. The synergy of structured OCR embeddings, geometry-aware alignment, and pointer–generator decoding yields measurable gains in both quantitative and qualitative dimensions. The results not only validate our methodological design but also suggest broader implications for multimodal reasoning tasks such as document VQA and scene text understanding.

In summary, our extensive experiments confirm the following:

• OCR embeddings significantly enhance scene understanding in text-rich environments.
• Pointer–generator networks improve factual accuracy and linguistic precision.
• Geometry-aware fusion mitigates noise and improves attention grounding.

Together, these findings establish TEXTSight as an effective and interpretable model for real-world, accessibility-oriented caption generation tasks.

6. Conclusion and Future Work

In this work, we introduced TEXTSight, a unified OCR-aware pointer–generator image captioning model designed to address the real-world challenges of visually impaired users. By extending the AoANet architecture with multimodal text–vision fusion, geometry-aware attention, and a probabilistic copy mechanism, our method significantly enhances descriptive precision, particularly for text-rich images. Comprehensive experiments on the VizWiz dataset demonstrate that TEXTSight substantially outperforms baseline models in both quantitative metrics and qualitative human assessments. The integration of OCR token embeddings proved critical for grounding linguistic predictions in textual regions, while the pointer–generator strategy allowed accurate reproduction of entity names, dates, and symbolic content—an essential capability for accessibility applications.

From a methodological perspective, our study reveals several important insights. First, expanding the vocabulary with OCR tokens directly improves semantic coverage and factual accuracy. Second, geometry-biased attention facilitates better spatial alignment between visual and textual modalities. Third, confidence-weighted copying mitigates noisy OCR predictions while retaining model interpretability. These results collectively highlight the potential of text-aware visual grounding as a pivotal component for next-generation multimodal captioning systems.

Despite these advances, there remain several promising directions for future exploration. One limitation of the current TEXTSight model lies in its reliance on static OCR inputs. Real-world deployment in dynamic or low-visibility scenarios would benefit from end-to-end differentiable text detection and recognition modules, allowing joint optimization of vision, language, and OCR stages. Another avenue involves integrating a coverage mechanism [26,29] within the pointer–generator framework to alleviate repetitive token generation and enhance caption diversity.

In addition, future work will involve extensive benchmarking of TEXTSight against recently developed multimodal architectures such as the multimodal transformer proposed by Dognin et al. [11] and the M4C-based OCR-captioning system by Sidorov et al. [27]. These models demonstrate complementary strengths, including cross-modal reasoning and dynamic text utilization, which could be synergistically integrated into TEXTSight through shared transformer backbones or modular adapters.

We also aim to expand beyond the VizWiz domain to evaluate cross-dataset generalization, particularly on TextCaps, ST-VQA, and document image captioning benchmarks, where textual reasoning is central to comprehension. Preliminary experiments suggest that transfer learning from large-scale multimodal encoders such as CLIP or BLIP-2 can provide additional robustness and domain adaptability. A formal exploration of these transfer pathways forms part of our upcoming research agenda.

From a broader perspective, future directions of TEXTSight include:

• Incorporating scene graph reasoning: Explicitly modeling relationships between visual entities and text regions to enhance contextual grounding.
• Adopting uncertainty quantification: Estimating model confidence for each generated token could support interactive caption correction systems for blind users.
• Enhancing real-time deployment: Compressing the model via knowledge distillation or quantization to run efficiently on edge devices such as smartphones or wearable vision aids.
• Exploring multimodal pretraining: Leveraging large-scale weakly aligned datasets to enable TEXTSight to generalize beyond constrained datasets.

Finally, while our work focuses primarily on image captioning, the principles behind TEXTSight are broadly extensible to related domains such as scene-text question answering, document visual understanding, and multimodal reasoning in assistive robotics. By jointly leveraging symbolic and perceptual modalities, TEXTSight represents a meaningful step toward inclusive, intelligent systems that bridge the gap between vision and language for users with visual disabilities.

In summary, we have demonstrated that integrating text-aware attention and probabilistic copying yields tangible improvements in captioning performance. Future iterations of TEXTSight will aim for deeper semantic interpretability, interactive feedback mechanisms, and robust adaptation to diverse visual–textual environments. We believe this work lays the foundation for a new generation of accessibility-driven multimodal systems, advancing both social impact and scientific innovation in the field of vision–language intelligence.