Conditioned Visual Captioning with Spatially-Aware Multimodal Modeling

1. Introduction

Scene text is an essential source of information, often providing critical details such as product labels, book titles, and road signs. Recognizing and describing such textual content is crucial for a wide range of applications, particularly in assistive technologies for visually impaired users [2,10,14,18,23,28,32]. Recent advancements in text-aware image captioning [24,29,31,33,35] have significantly improved the integration of scene text into captions. However, these models typically generate static descriptions without adapting to user-specific information needs.

In scenarios where an image contains multiple scene texts, summarizing all textual content in a single caption can be cumbersome and inefficient. Prior research [19] indicates that visually impaired users prefer a progressive interaction paradigm, where they receive an initial high-level summary followed by detailed captions based on their specific queries. For example, when encountering an image of a book, users might ask targeted questions such as "Who is the author?" or "What is the title?" instead of receiving a verbose caption listing all textual elements indiscriminately.

To address this limitation, we introduce the Question-driven Text-aware Image Captioning (Q-TAG) task, wherein users pose scene-text-related questions following an initial general caption, prompting the model to generate more personalized and informative captions. This differs from the Text-based Visual Question Answering (TextVQA) [4,13,15,25], which typically produces brief textual answers to single questions without incorporating scene understanding. Instead, Q-TAG requires the model to generate coherent, natural language descriptions enriched with textual and visual context. Compared to prior controllable captioning methods [5,6,34], our approach is more intuitive and user-friendly, leveraging natural language queries instead of pre-selected object tokens or predefined visual regions.

Given the lack of existing datasets for this task, we develop an automated framework to generate Q-TAG datasets from TextCaps [24] and VizWiz-Captions [11]. Our pipeline extracts relevant scene text from images, constructs initial coarse descriptions, and generates diverse question-answer pairs to simulate real-world user interactions.

To tackle the Q-TAG task, we introduce a novel Question-driven Text-aware Captioning Generator (Q-TAG), which consists of:

• A Spatially-aware Multimodal Encoder, which integrates region-based object features and text-based scene representations while encoding spatial relationships using geometric modeling.
• A Question-driven Feature Selector, which dynamically attends to relevant scene-text-visual features based on user queries, filtering extraneous information.
• A Multimodal Fusion Decoder, which synthesizes the retrieved information to generate fluent and context-aware captions tailored to user queries.

Our extensive experiments on the datasets demonstrate that Q-TAG generates more diverse, informative, and user-centric captions than prior text-aware captioning models.

Key contributions of our work:

• We introduce the Q-TAG task, pioneering question-controlled text-aware image captioning to enhance accessibility for visually impaired individuals.
• We develop a novel model, Q-TAG, which integrates spatial-aware encoding, question-guided feature selection, and multimodal fusion for enhanced caption generation.
• Our model achieves superior performance against state-of-the-art baselines, demonstrating improved contextual awareness, linguistic diversity, and user adaptability.

By leveraging natural language interactions, our approach facilitates intuitive user engagement, enabling more effective comprehension of visually rich environments. Future research directions include incorporating reinforcement learning for adaptive caption refinement and exploring multimodal pretraining for further generalization.

2. Related Work

A plethora of deep learning-based frameworks [2,10,12,14,18,23,28,32] have been developed to address the challenge of general image captioning, aiming to automatically generate detailed and semantically rich textual descriptions of images. Among these, AoANet [14] achieves state-of-the-art performance by leveraging an advanced attention-on-attention mechanism, which significantly enhances context comprehension.

Fishch et al. [10] introduced an innovative training paradigm, wherein question-answering accuracy serves as an auxiliary reward to reinforce information richness in generated captions. However, despite their effectiveness, these methods primarily produce static captions, failing to dynamically adapt to user preferences or highlight scene texts in a manner that aligns with individual user needs.

Text-aware image captioning focuses on interpreting embedded scene text in images while integrating it with surrounding visual objects for coherent descriptions. Notable datasets supporting this research direction include TextCaps [24], derived from Open Images V3, and VizWiz-Captions [11], comprising images captured by visually impaired users. Approximately 63% of VizWiz images contain readable text; however, real-world capturing conditions introduce challenges such as poor lighting, blur, and overexposure, making the dataset highly representative of practical scenarios.

Leveraging these datasets, various models [29,31,33,35] have been proposed to refine text-aware captioning. Wang et al. [29] incorporated spatial reasoning into OCR token processing to better contextualize scene text. Zhu et al. [35] formulated a multi-attention-based strong baseline model. Wang et al. [31] enhanced text selection with confidence embeddings, filtering out irrelevant content, while Yang et al. [33] designed pre-training tasks specifically tailored for text-centric scene understanding. Although these methods contribute significantly, they do not address user-driven personalized captioning, which remains a crucial gap in this field.

TextVQA models [4,13,15,25] are designed to locate and interpret scene text to answer specific questions. The primary distinction between the TextVQA task and the proposed Q-TAG framework lies in two aspects. First, Q-TAG demands the simultaneous processing of multiple questions, necessitating advanced multi-query comprehension mechanisms. Second, rather than outputting fragmented one-word or short-phrase answers, Q-TAG systems must construct complete, naturally flowing captions that fluently incorporate the relevant scene text and associated visual elements, offering a richer, more holistic textual representation.

Controllable Image Captioning endeavors to generate diverse descriptions by emphasizing different aspects of an image [5,6,34]. Zheng et al. [34] introduced a method that guides captioning via object-focused tokens, allowing the system to tailor descriptions based on predefined object categories. Cornia et al. [6] proposed region-based control signals, enabling caption generation for specific localized areas within an image. Additionally, Chen et al. [5] advanced controllability by employing abstract scene graphs to encode user intentions at a finer granularity.

While these approaches offer precise user-driven control, they rely on explicit, structured signals, such as object selections or bounding-box specifications, which are impractical for visually impaired users. Unlike these rigid control methods, our Q-TAG model introduces a language-driven interactive framework, where users can simply pose natural language queries to obtain personalized captions. This method is more intuitive, removing the prerequisite of visual perception while maintaining high expressiveness, making it an ideal solution for assistive technologies catering to visually impaired individuals.

3. Our Methodology

In this section, we present the Q-TAG framework, a novel architecture designed for the Question-Controlled Text-Aware Captioning (Qc-TextCap) task. Our model consists of three major components: the Spatially-Aware Multimodal Encoder, the Question-Guided Feature Selector, and the Multimodal Fusion Decoder. These modules work in synergy to analyze image regions, process textual questions, and generate rich, personalized captions that integrate scene text information with object relationships.

3.1. Spatially-Aware Multimodal Encoder

To effectively capture the spatial relationships between objects and scene text, our Spatially-Aware Multimodal Encoder fuses object features with scene text features while leveraging geometric properties. Given an input image I, an initial caption $C^{ini}$, and a set of user queries $\mathcal{Q}$, we apply an object detection model [2] to identify bounding boxes $B^{obj}$ and an OCR engine to extract scene text bounding boxes $B^{ocr}$. We then extract feature representations $V^{obj}$ and $V^{ocr}$ for these regions.

To model the spatial relationships, we define each bounding box $b_i$ using its geometric attributes:

$$b_i=(c_i^x,c_i^y,w_i,h_i)$$

(1)

where $c^x$ and $c^y$ denote the center coordinates, while w and h represent the width and height. The spatial relationships between two regions i and j are encoded as:

$$s_{ij}^g=log\left({\displaystyle \frac{|c_i^x-c_j^x|}{w_i}},{\displaystyle \frac{|c_i^y-c_j^y|}{h_i}},{\displaystyle \frac{w_j}{w_i}},{\displaystyle \frac{h_j}{h_i}}\right)W^g$$

(2)

These spatial features are incorporated into the self-attention mechanism, ensuring that the encoder prioritizes regions based on spatial relevance. The attended visual feature is then computed as:

$${\widehat{v}}_i=\sum _j{s}_{ij}{V}_j$$

(3)

where $s_{ij}$ is the spatially weighted attention score.

3.2. Question-Guided Feature Selector

To address multiple user queries in a single caption, the Question-Guided Feature Selector dynamically extracts relevant visual-textual information. Instead of treating questions holistically, we perform word-level alignment between question tokens and image features. Given the token embeddings $T^{que}$, we compute attention scores to identify the most relevant visual features for each word:

$$s_{ij}^q=t_i^{que}{W}_Q^q{\left({\widehat{v}}_j{W}_K^q\right)}^T$$

(4)

where $W_Q^q,W_K^q$ are learned projection matrices. The refined feature representation for each question token is then computed as:

$${\widehat{t}}_i^{que}=t_i^{que}{W}_T^q+\sum _j{s}_{ij}^q{\widehat{v}}_j{W}_V^q$$

(5)

This selective feature extraction ensures that each generated caption explicitly answers the posed questions while maintaining linguistic coherence.

3.3. Multimodal Fusion Decoder

The Multimodal Fusion Decoder synthesizes the extracted features into a fluent and meaningful caption. At each decoding step t, the decoder integrates multiple feature sources:

$$Y_t^{dec}=\mathrm{Transformer}\left([{\widehat{V}}^{obj},{\widehat{V}}^{ocr},{\widehat{T}}^{que},T^{ini},Y_{t-1}^{dec}]\right)$$

(6)

where ${\widehat{V}}^{obj},{\widehat{V}}^{ocr}$ are the encoded object and text features, ${\widehat{T}}^{que}$ are the question-guided features, and $T^{ini}$ represents the initial caption.

To generate the next token $y_t$, we employ a pointer network that selects between vocabulary words and OCR tokens:

$${\widehat{y}}_t=\mathrm{Softmax}(W^{voc}{z}_{t-1}^{dec}+W^{ocr}{z}^{ocr})$$

(7)

where $z_{t-1}^{dec}$ represents the decoder state from the previous step, and $z^{ocr}$ denotes the OCR-based contextual embedding.

3.4. Training Strategy

To enhance the model’s ability to incorporate scene text into captions, we experiment with different training objectives. The primary loss function is defined as:

$$\mathcal{L}=-\sum _{t=1}^l{y}_{t}log\left({\widehat{y}}_t\right)$$

(8)

where $y_t$ is the ground-truth token. We also explore a multi-task learning approach by incorporating a contrastive loss to refine text-image alignment.

3.5. Evaluation and Results

We evaluate Q-TAG on the ControlTextCaps and ControlVizWiz datasets, comparing it against state-of-the-art models. Table 1 presents our performance metrics, showing substantial improvements in BLEU, CIDEr, and SPICE scores.

Our results indicate that Q-TAG significantly enhances caption quality, providing more informative, user-adaptive descriptions. Future work will explore reinforcement learning for iterative refinement and multimodal pretraining to improve generalization across diverse image-text scenarios.

4. Experiments

In this section, we conduct extensive evaluations of our proposed Q-TAG framework on the benchmark datasets. We analyze the performance of Q-TAG under various training strategies, compare it against state-of-the-art baselines, and assess its effectiveness in generating personalized, text-aware image captions.

4.1. Experimental Setup

Baseline Methods. To evaluate Q-TAG comprehensively, we compare it against the following models:

1) M4C-Captioner [24]: A state-of-the-art text-aware image captioning model that fuses object and scene text features using multi-layer transformers and a pointer network. 2) ControlM4CC: An extension of M4C-Captioner adapted for question-controlled captioning, where questions and initial captions are concatenated with visual features. 3) Q-TAG w/o Spatial Encoder: A variant of our model that omits the Spatially-Aware Multimodal Encoder to analyze its contribution to overall performance.

Evaluation Metrics. We utilize the standard captioning evaluation metrics: BLEU [20], METEOR [7], ROUGE-L [16], CIDEr [27], and SPICE [1]. Since CIDEr emphasizes rare words, it is particularly suitable for text-aware captioning tasks. Additionally, we introduce Answer Recall (AnsRecall) to measure how accurately the generated captions capture key scene text details in response to given questions.

Implementation Details. We set the maximum sequence lengths for initial captions and concatenated questions to 20 tokens, while generated captions are capped at 30 tokens. Each image is associated with up to 100 object bounding boxes and 50 scene text bounding boxes. During training, the batch size is set to 50, with training steps of 10,000 for ControlVizWiz and 16,000 for ControlTextCaps. Greedy decoding is used for inference unless otherwise specified.

4.2. Evaluation of Q-TAG on Qc-TextCap Task

Comparison of Non-Controllable vs. Controllable Models.Table 1 shows the performance of various models. The question-controlled models outperform M4C-Captioner, demonstrating the benefits of question-guided caption generation. Our Q-TAG model surpasses ControlM4CC in both captioning and question-answering capabilities, validating the effectiveness of the overall architecture.

Ablation studies reveal that removing the Spatially-Aware Multimodal Encoder significantly impacts CIDEr scores (+14.1/+13.8), highlighting its role in fusing spatially aligned visual-textual information. Additionally, Q-TAG achieves higher AnsRecall scores, confirming that spatial reasoning enhances the incorporation of scene text in generated captions.

Impact of Training Strategies. To investigate how different training approaches influence performance, we compare three strategies: 1) Auto: Using only automatically generated initial captions. 2) Pseudo: Using pseudo captions lacking scene text. 3) Rand(auto, pseudo): Randomly selecting one of the two strategies for each training instance.

As seen in Table 2, models trained with the ‘rand(auto, pseudo)’ strategy outperform those trained with a fixed approach. This suggests that varying training inputs improves generalization, enabling better adaptation to diverse user queries.

4.3. Diversity Evaluation

Q-TAG is designed to generate diverse captions tailored to different user queries. To quantify this, we compute Diversity-1 (Div-1), Diversity-2 (Div-2) [3,5], and SelfCIDEr [30]. Table 3 shows that Q-TAG significantly outperforms M4C-Captioner across all diversity metrics, demonstrating its ability to generate more varied and informative captions.

4.4. Qualitative Evaluation and Human Assessment

We conduct human evaluations to compare Q-TAG against M4C-Captioner in terms of scene text accuracy (ST Info) and overall caption quality. Six evaluators rated captions based on how well they incorporated scene text and their fluency.

As seen in Table 4, Q-TAG is preferred over M4C-Captioner in most cases, reinforcing its ability to deliver superior, user-personalized text-aware captions. Our experimental findings establish that Q-TAG significantly improves upon existing text-aware captioning models by incorporating question-controlled guidance. The model enhances caption diversity, ensures more accurate scene text representation, and provides greater user adaptability. Future work will focus on integrating reinforcement learning for iterative caption refinement and extending the framework to handle multi-turn question interactions.

5. Conclusions and Future Directions

In this work, we introduce Question-Guided Text-Aware Image Captioning (Q-TAG), a novel and challenging task aimed at generating personalized, text-aware image captions tailored to the needs of visually impaired users. Unlike conventional text-aware captioning models, Q-TAG employs user-posed questions as control signals to dynamically focus on relevant scene text and visual elements, thereby enhancing both informativeness and interactivity in generated captions.

The Q-TAG task demands that models possess the capability to comprehend user queries, identify relevant scene text regions, and seamlessly integrate extracted information with an initial coarse-grained caption to generate a final, contextually enriched description. To facilitate research in this domain, we construct two benchmark datasets—ControlTextCaps and ControlVizWiz—by augmenting existing text-aware captioning datasets with question-controlled annotations. These datasets will be released publicly to support further advancements in the field.

To tackle this task, we propose the Q-TAG framework, which incorporates a Spatially-Aware Multimodal Encoder, a Question-Guided Feature Selector, and a Multimodal Fusion Decoder to progressively integrate relevant visual-textual features. Our extensive experimental results on both datasets demonstrate that Q-TAG significantly outperforms baseline methods in terms of both captioning quality and question-answering accuracy. The inclusion of explicit question-guided control signals enables the generation of more informative, diverse, and user-adaptive captions, making it a practical enhancement for assistive technologies.

Moving forward, several promising research directions emerge from our work:

• Multi-turn Question-Controlled Captioning: Extending Q-TAG to support interactive, multi-turn dialogues where users can refine or request additional details iteratively.
• Incorporating Commonsense Knowledge: Enhancing scene text interpretation by integrating external knowledge sources to infer implicit relationships and contextual meanings.
• Leveraging Reinforcement Learning: Employing reinforcement learning to optimize caption generation based on user engagement and feedback, ensuring continuously refined outputs.
• Multimodal Pretraining Strategies: Exploring large-scale multimodal pretraining approaches to improve generalization and robustness across diverse real-world datasets.
• Adaptive Decoding Mechanisms: Investigating more flexible decoding strategies, such as constrained beam search or neural-symbolic approaches, to ensure higher coherence in generated captions.

By advancing these directions, we envision Q-TAG evolving into a powerful, intelligent assistive system capable of delivering highly customized and meaningful textual descriptions for users with diverse accessibility needs.