Context-Guided Multi-Branch Fusion for Text-Dependent Visual Question Reasoning

1. Introduction

Deep neural networks have achieved remarkable progress in a variety of domains such as image recognition Guo et al. (2020b); Guo, Stutz, and Schiele (2022; Guo et al. (2019)), natural language modeling Dai and Callan (2019), and generative modeling Goodfellow et al. (2020); Guo et al. (2020a); Guo et al. (2022). Among the numerous multimodal tasks that bridge vision and language, Visual Question Answering (VQA) has emerged as a crucial testbed for studying joint perception, reasoning, and understanding. In a typical VQA task, a model is required to produce an accurate natural language answer to a question about a given image. Such a task lies at the intersection of computer vision and natural language processing, requiring not only robust feature extraction but also logical reasoning and compositional understanding across modalities. VQA has wide-ranging applications, including automated assistance for the visually impaired, medical image diagnosis, and document interpretation. However, despite the rapid advancement of transformer-based architectures, VQA remains a formidable challenge due to its inherent need for structured reasoning over heterogeneous inputs.

A central difficulty in VQA lies in constructing discriminative visual representations that capture both fine-grained object-level semantics and high-level contextual cues. Early approaches Hong et al. (2019); Ma et al. (2018); Yu et al. (2017) relied heavily on convolutional backbones, using global feature maps or logits from classification heads as image encodings. However, these representations are largely object-centric and insufficient for tasks involving relational or textual reasoning. Subsequent works such as VinVL Zhang et al. (2021) enhanced representation richness through large-scale pretraining and improved object detection, yet they still focus primarily on spatial and object attributes while overlooking embedded textual content. In many real-world scenes—street signs, advertisements, or manuals—the textual elements carry essential clues for answering questions. The lack of integration between textual cues and visual perception therefore forms a persistent bottleneck in modern VQA systems.

Figure 1. Illustration of the motivation behind the proposed Contextually-Guided Multi-Branch Fusion Network (CMFN). The example shows that conventional VQA systems fail to answer questions requiring text comprehension (e.g., identifying “bus number 24”), while CMFN employs an adaptive routing mechanism that dynamically selects between a visual reasoning branch and a text-aware reasoning branch. By integrating OCR-derived textual cues and visual semantics, CMFN accurately interprets text-dependent visual questions for robust and contextually grounded reasoning.

Figure 1. Illustration of the motivation behind the proposed Contextually-Guided Multi-Branch Fusion Network (CMFN). The example shows that conventional VQA systems fail to answer questions requiring text comprehension (e.g., identifying “bus number 24”), while CMFN employs an adaptive routing mechanism that dynamically selects between a visual reasoning branch and a text-aware reasoning branch. By integrating OCR-derived textual cues and visual semantics, CMFN accurately interprets text-dependent visual questions for robust and contextually grounded reasoning.

Another significant challenge lies in the design of the answer prediction mechanism. Most existing methods Anderson et al. (2018); Li et al. (2020); Zhang et al. (2021) formulate answer generation as a multi-label classification problem over a fixed vocabulary (e.g., the 3,000 most frequent answers). While efficient, this closed-set assumption severely restricts the model’s expressiveness and prevents it from handling open-ended or text-specific answers that do not appear in the training distribution. In particular, when a question refers to a text region—such as “What number is printed on the bus?”—the correct answer cannot be drawn from the static candidate set. This highlights the necessity of flexible, generative, and text-aware mechanisms that can adaptively form answer sequences from dynamic OCR tokens or contextual embeddings.

To overcome these limitations, we propose a new perspective on VQA that explicitly distinguishes between text-dependent and text-independent reasoning scenarios. Our proposed framework, named Contextually-Guided Multi-Branch Fusion Network (CMFN), employs an adaptive dual-branch architecture comprising a visual reasoning branch for general queries and a text-aware reasoning branch for questions involving scene text understanding. These two branches are harmonized by a contextual gating mechanism that determines, for each instance, which reasoning pathway to follow. The text-aware branch integrates OCR-derived embeddings, capturing character-level and word-level semantics and aligning them with visual and linguistic cues through a cross-modal fusion layer. This design allows CMFN to “read” and “reason” simultaneously—achieving both fine-grained perception and semantic flexibility.

Our approach contributes to VQA research in several important aspects. (1) We introduce an OCR-aware multimodal representation module that jointly learns from object-level regions and scene text, effectively bridging the gap between visual and linguistic information. (2) We develop a dual-branch answer generation network that dynamically routes each question-image pair through either a classification-based or a pointer-based decoding pathway, depending on its contextual relevance to textual cues. (3) A novel contextual alignment gate is incorporated to regulate the degree of fusion between the two branches, yielding more coherent and accurate answers under diverse reasoning conditions. (4) Comprehensive experiments on VQA v2.0 validate the superior performance of CMFN, which achieves over 4% improvement in accuracy on number-related and text-reading questions compared with strong baselines like VinVL Zhang et al. (2021). These results demonstrate that adaptive routing and text-sensitive reasoning provide a powerful new direction for visual-linguistic understanding.

Beyond benchmark improvements, our work underscores a broader insight: visual reasoning cannot be fully realized without textual comprehension. Many real-world VQA scenarios—such as reading road names in navigation, interpreting data from charts, or identifying medication labels—demand models that are not only visually grounded but also linguistically adaptive. CMFN thus represents a step towards cognitively-inspired multimodal intelligence, where reasoning flows flexibly between perception and reading comprehension. In future research, this principle could extend beyond VQA to unified frameworks for document understanding, scene-text-based grounding, and general vision-language reasoning across open environments.

2. Related Work

2.1. Text-Based Visual Question Answering

Text-based Visual Question Answering (Text-VQA) has become a rapidly emerging subfield in vision-language research Hu and Singh (2021); Li et al. (2020); Lu et al. (2019); Tan and Bansal (2019), aiming to bridge the gap between textual perception and visual reasoning. Unlike standard VQA tasks that primarily rely on recognizing objects, colours, or spatial relations, Text-VQA requires a model to “read” and reason over scene text (e.g., street signs, charts, advertisements), which introduces new complexities in both perception and reasoning. The text embedded in natural images is often distorted, occluded, or contextually dependent, making it challenging to align with the linguistic semantics of the question. This subproblem demands models to unify three modalities — visual features, textual embeddings from OCR, and linguistic representations from the question — in a coherent reasoning framework.

Early research, such as LoRRA Singh et al. (2019), extended the traditional attention mechanism to simultaneously attend to image regions and recognized text tokens. It selected answers either from a fixed-length candidate set or directly from OCR tokens, demonstrating the feasibility of integrating reading comprehension with visual perception. However, LoRRA’s single-step prediction process limited its ability to generate multi-token answers, such as numerical sequences or compound words. This was later addressed by M4C Hu et al. (2020), which introduced a flexible pointer network capable of iteratively generating multi-word answers by dynamically deciding between copying OCR tokens or selecting from a pre-defined vocabulary. M4C paved the way for more compositional reasoning and set a foundation for modern text-reading VQA systems.

Subsequent studies built upon M4C with specialized architectural refinements. LaAP-Net Han, Huang, and Han (2020) leveraged localization priors, incorporating text bounding box coordinates as evidence to better link spatial information and answer grounding. Zhu et. al Zhu et al. (2020) further designed dual-attention fusion streams, separately attending to visual and textual modalities before integrating them via a multimodal reasoning head. In contrast to these methods, our proposed framework introduces a broader multimodal alignment strategy, leveraging both object-level appearance cues and text-level semantic information to create richer scene understanding. By integrating a dual routing module for adaptive answer prediction, our model surpasses conventional VQA approaches in handling text-heavy environments and heterogeneous reasoning scenarios.

Beyond architectural advances, recent work has also explored improving textual grounding and representation. Models like TAP and OCR-VQA employ pretraining objectives that encourage joint understanding of text and visual layout, while Donut et. al directly processes images in a text-centric transformer space without explicit OCR postprocessing. Such methods collectively indicate a paradigm shift toward unified, OCR-agnostic text-reading VQA pipelines. Our model, while grounded in the traditional OCR-based paradigm, advances this direction by introducing a flexible, context-driven routing mechanism capable of dynamically balancing visual and text-centric reasoning pathways.

2.2. Pointer Networks and Generative Decoding

For questions requiring the reading of embedded text—such as numerical values, brand names, or product information—answers often originate from OCR token sequences within the image. Conventional classification-based VQA models fail to generalize to such cases due to their reliance on fixed candidate vocabularies. To address this limitation, Pointer Networks Vinyals, Fortunato, and Jaitly (2015) were introduced as an elegant mechanism to directly “point” to relevant tokens in the input sequence rather than predicting from a closed set. The pointer mechanism estimates a probability distribution over input positions, allowing the model to flexibly generate answers of variable length. This dynamic formulation inherently overcomes the out-of-vocabulary (OOV) issue common in closed-set VQA frameworks.

Pointer Networks have been widely applied to sequential reasoning tasks such as abstractive summarization Nallapati et al. (2016), neural machine translation Gu et al. (2016), and image captioning Lu et al. (2018). Their capacity for fine-grained token selection inspired several VQA models to integrate similar mechanisms. LoRRA Singh et al. (2019) extended the answer space by including OCR tokens, effectively enabling partial text copying. Building upon this, M4C Hu et al. (2020) used a multi-step pointer-decoder to iteratively refine answer sequences, combining candidate-based classification with token-level copying. Nevertheless, M4C’s decoding remains purely sequential and context-independent at each time step. In our model, we propose a contextually adaptive pointer network, which dynamically determines whether the model should copy from OCR tokens or select from the candidate vocabulary, enabling more fine-grained integration of multimodal cues.

Moreover, recent advancements such as TAP Hu et al. (2020), TextCaps Hu et al. (2020), and OCRFormer Hu et al. (2020) demonstrate that pointer-based generative decoding can be further enhanced by hierarchical attention, transformer-based contextualization, and implicit language priors. Our method extends this line by embedding the pointer mechanism within a dual-routing reasoning framework, balancing between visual and textual experts for adaptive decoding.

2.3. Dynamic Neural Networks and Mixture-of-Experts Routing

Dynamic neural networks have become an effective paradigm for achieving conditional computation and model specialization Han et al. (2021). Rather than processing all inputs through a static pipeline, these networks selectively activate specific modules (“experts”) tailored to different input types. This design not only enhances interpretability but also enables resource-efficient training by routing samples to the most suitable expert.

The Mixture-of-Experts (MoE) framework Shazeer et al. (2017) is one of the most well-known instances of this paradigm, achieving scalability across massive model architectures by activating only a subset of experts per input. In the vision-language community, Lioutas et. al Lioutas, Passalis, and Tefas (2018) utilized multiple attention-based experts to refine multimodal alignment, while Patro et. al Patro et al. (2020) fused heterogeneous cues through expert mixing for robust question generation. More recently, Wang et. al Wang et al. (2021) introduced VLMo, a modality-specific MoE transformer that learns cross-modal representations via specialized experts for vision, language, and joint embedding spaces.

In the context of VQA, dynamic routing offers a promising solution to the heterogeneity of question types. For example, numerical questions (“How many...?”) and textual questions (“What word is written on the sign?”) may require distinct reasoning mechanisms. Motivated by this observation, we propose a dual-expert routing framework consisting of two specialized modules: a classification-based expert for general answers and a dynamic pointer expert for text-based answers. A lightweight gating network learns to select between them based on the joint embedding of the question and visual context. This mechanism ensures efficient reasoning diversity without additional computational burden. Unlike conventional MoE models focused on load balancing, our gating mechanism emphasizes semantic discrimination, aligning routing decisions with cognitive reasoning types rather than computational efficiency.

2.4. Cross-Modal Fusion and Contextual Alignment

A crucial component in Text-VQA is the alignment between visual, textual, and linguistic embeddings. Standard cross-attention mechanisms Li et al. (2020); Lu et al. (2019); Tan and Bansal (2019) allow interaction between modalities, but often fail to maintain context consistency when fusing OCR-derived text and object-level features. To enhance fusion granularity, several recent works propose hierarchical or graph-based fusion strategies. ROSITA ? introduced a scene-graph integration mechanism that models relations between textual nodes and visual entities, while UniT Hu and Singh (2021) employed a shared transformer backbone across tasks to learn generalizable multimodal alignments.

Our proposed framework extends this fusion perspective by introducing a contextual alignment gate, which dynamically adjusts the contribution of textual and visual signals based on question semantics. This gate allows the model to prioritize the modality most relevant to the current question, improving interpretability and generalization. Through this adaptive alignment mechanism, our method effectively bridges the gap between perception and reasoning, enabling more reliable and context-aware answer generation across diverse multimodal environments.

Figure 2. Overview of the proposed CMFN (Contextually-Guided Multi-Branch Fusion Network) framework for visual question answering. The pipeline begins with an image–question input, which is processed through three parallel encoders: an object-level feature extractor, an OCR feature encoder, and a question encoder. Their outputs are integrated by a Cross-Modal Fusion Transformer, producing a unified multimodal representation. A learnable gating network then dynamically routes this representation to one of two reasoning experts — a Fixed-Set Classifier for structured answers or an OCR-Aware Pointer Decoder for open-ended textual outputs. The final predicted answer is generated based on the selected expert. The entire system is jointly optimized under a multi-objective loss combining classification, pointer, gating, and alignment terms.

Figure 2. Overview of the proposed CMFN (Contextually-Guided Multi-Branch Fusion Network) framework for visual question answering. The pipeline begins with an image–question input, which is processed through three parallel encoders: an object-level feature extractor, an OCR feature encoder, and a question encoder. Their outputs are integrated by a Cross-Modal Fusion Transformer, producing a unified multimodal representation. A learnable gating network then dynamically routes this representation to one of two reasoning experts — a Fixed-Set Classifier for structured answers or an OCR-Aware Pointer Decoder for open-ended textual outputs. The final predicted answer is generated based on the selected expert. The entire system is jointly optimized under a multi-objective loss combining classification, pointer, gating, and alignment terms.

3. Methodology

In this work, we introduce a unified framework named CMFN (Contextually-Guided Multi-Branch Fusion Network) that enables intelligent routing between text-oriented and non-text-oriented reasoning pathways for visual question answering. The core design philosophy of CMFN lies in dynamically integrating multi-source visual, textual, and question embeddings under a Transformer-based architecture. Our method leverages enhanced object and OCR representations, a cross-modal encoder-decoder interaction mechanism, and a dual-expert reasoning module that determines optimal prediction pathways via a learnable gating system. This section provides a detailed explanation of each component, together with extended mathematical formulations and additional design insights.

3.1. Enhanced Visual and Textual Representations

We begin by constructing multimodal feature representations from the visual and textual content of the input image. Given an image-question pair $(I,Q)$, our objective is to embed its heterogeneous components into a shared latent space that preserves semantic granularity and spatial consistency. We define three key feature sources: (1) object-level region embeddings, (2) OCR token embeddings, and (3) linguistic question embeddings. These features are subsequently aligned and fused via cross-modal attention layers.

3.1.1. Object-Level Feature Embedding

To capture the visual content, we employ the object detector from VinVL Zhang et al. (2021), which yields a set of E region proposals ${\left\{r_e\right\}}_{e=1}^E$, each represented by a d-dimensional feature vector ${\mathbf{x}}_e^{obj}$. These features encode appearance, shape, and contextual attributes. We project them into a unified latent space using a learned transformation:

$${\mathbf{h}}_e^{obj}=LN(W^{obj}{\mathbf{x}}_e^{obj}+{\mathbf{b}}^{obj}),$$

(1)

where $LN(·)$ denotes layer normalization. To enhance alignment between object regions and linguistic tokens, we concatenate the corresponding object tags detected by VinVL to the input question words as auxiliary context tokens, yielding enriched semantic grounding across modalities. Following Li et al. (2020), this augmentation improves visual-semantic alignment and alleviates ambiguity in cross-modal interactions.

Furthermore, to capture relationships between visual regions, we incorporate a self-attention refinement mechanism:

$${\tilde{\mathbf{h}}}_e^{obj}=\sum _{e^{\prime }=1}^E{\alpha }_{e{e}^{\prime }}\left({\mathbf{W}}_V{\mathbf{h}}_{e^{\prime }}^{obj}\right),$$

(2)

where ${\alpha }_{e{e}^{\prime }}=\mathrm{softmax}\left({\left({\mathbf{W}}_Q{\mathbf{h}}_e^{obj}\right)}^{\top }\left({\mathbf{W}}_K{\mathbf{h}}_{e^{\prime }}^{obj}\right)\right)$ computes the attention correlation between regions. This design ensures that object embeddings are aware of contextual dependencies among different visual entities.

3.1.2. OCR Feature Representation

Scene text often carries key cues for answering textual questions. To accurately interpret such text, we design a multi-level OCR representation that integrates semantic, spatial, and visual descriptors for each OCR token $o_m$ ($m=1,\dots ,M$).

The semantic feature ${\mathbf{x}}_m^{smt}$ is formed by concatenating subword and character-level representations:

$${\mathbf{x}}_m^{smt}=[{\mathbf{x}}_m^{ft};{\mathbf{x}}_m^{phoc};{\mathbf{x}}_m^{vis}],$$

(3)

where ${\mathbf{x}}_m^{ft}\in {\mathbb{R}}^{300}$ is a FastText embedding Bojanowski et al. (2017), ${\mathbf{x}}_m^{phoc}\in {\mathbb{R}}^{604}$ encodes character histograms Almazán et al. (2014), and ${\mathbf{x}}_m^{vis}\in {\mathbb{R}}^{2048}$ captures visual texture features extracted via Faster R-CNN Ren et al. (2015). These representations together provide both orthographic and visual context.

To encode spatial positioning, we define a geometric embedding:

$${\mathbf{x}}_m^{spt}=[x_{min}/W,y_{min}/H,x_{max}/W,y_{max}/H],$$

(4)

where $(W,H)$ are the image dimensions. The final OCR embedding is computed as:

$${\mathbf{h}}_m^{ocr}=LN(W^{smt}{\mathbf{x}}_m^{smt}+W^{spt}{\mathbf{x}}_m^{spt}).$$

(5)

We also introduce a contrastive pre-alignment loss ${\mathcal{L}}_{align}$ between OCR features and question embeddings to encourage semantic coherence:

$${\mathcal{L}}_{align}=-\sum _{m}log{\displaystyle \frac{exp({\mathbf{h}}_m^{ocr}·{\mathbf{h}}_{pos}^Q/\tau )}{{\sum }_{n}exp({\mathbf{h}}_m^{ocr}·{\mathbf{h}}_n^Q/\tau )}},$$

(6)

where $\tau$ is a temperature coefficient.

3.1.3. Question Encoding and Semantic Conditioning

For the question $Q={\left\{q_t\right\}}_{t=1}^{L_Q}$, we adopt a pre-trained Transformer encoder (e.g., BERT Kenton and Toutanova (2019)) to obtain contextualized embeddings ${\mathbf{h}}_t^Q$. To align with visual modalities, we map all question tokens into the same d-dimensional space and use a global question summary token ${\mathbf{h}}_{cls}^Q$ to condition subsequent reasoning modules. The representation is further enhanced via question-guided attention over the visual features:

$${\widehat{\mathbf{h}}}_e^{obj}=\sum _{t=1}^{L_Q}{\beta }_{et}{\mathbf{h}}_t^Q,$$

(7)

where ${\beta }_{et}$ measures semantic affinity between visual entity e and textual token t.

3.2. Cross-Modal Fusion Transformer

After obtaining ${\mathbf{h}}_e^{obj}$, ${\mathbf{h}}_m^{ocr}$, and ${\mathbf{h}}_t^Q$, we feed all features into a multimodal Transformer encoder-decoder. The encoder jointly models intra- and inter-modal interactions, while the decoder focuses on contextual reasoning and iterative answer generation. Given the concatenated sequence:

$$\mathbf{X}=[{\mathbf{h}}_1^Q,\dots ,{\mathbf{h}}_{L_Q}^Q,{\mathbf{h}}_1^{obj},\dots ,{\mathbf{h}}_E^{obj},{\mathbf{h}}_1^{ocr},\dots ,{\mathbf{h}}_M^{ocr}],$$

the self-attention mechanism computes:

$$\mathrm{Attn}(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathrm{softmax}\left({\displaystyle \frac{{\mathbf{QK}}^{\top }}{\sqrt{d}}}\right)\mathbf{V}.$$

(8)

This operation is repeated across multiple layers to integrate multi-source features into a common representation $\mathbf{Z}$. The decoder operates autoregressively, predicting the next answer token conditioned on previously generated tokens and multimodal context.

3.3. Dual-Expert Answer Reasoning Module

We now describe the core innovation of CMFN—the dual-expert reasoning module, which unifies two complementary experts under a dynamic routing mechanism. One expert (the classification head) predicts fixed-set answers, while the other (the pointer decoder) generates open-ended textual sequences.

3.3.1. Expert I: Classification-Based Prediction

For standard questions, the model predicts from a fixed vocabulary ${\mathcal{V}}_C$ of C frequent answers. Given the pooled multimodal representation ${\mathbf{z}}^{\left[CLS\right]}$, the prediction is:

$$\widehat{\mathbf{s}}=\sigma (W_{cls}{\mathbf{z}}^{\left[CLS\right]}+{\mathbf{b}}_{cls}),$$

(9)

where $\sigma (·)$ is the sigmoid function. The training objective is a binary cross-entropy loss:

$${\mathcal{L}}_{cls}=-{\displaystyle \frac{1}{N}}\sum _{i=1}^N\sum _{c=1}^C\left[s_{i,c}log{\widehat{s}}_{i,c}+(1-s_{i,c})log(1-{\widehat{s}}_{i,c})\right].$$

(10)

To prevent overconfidence on high-frequency answers, we apply a temperature-scaled calibration term during inference:

$${\widehat{s}}_{i,c}^{\prime }={\displaystyle \frac{exp({\widehat{s}}_{i,c}/\tau )}{{\sum }_{j}exp({\widehat{s}}_{i,j}/\tau )}}.$$

(11)

3.3.2. Expert II: Dynamic Pointer Decoding

For textual answers composed of OCR tokens, we design a generative pointer network. At each decoding step t, the model computes an attention distribution over OCR embeddings:

$$y_m^t=\mathrm{softmax}\left({\left(W^{dec}{\mathbf{z}}_t^{dec}\right)}^{\top }\left(W^{ocr}{\mathbf{z}}_m^{ocr}\right)\right),$$

(12)

and selects the most probable OCR token as the next output. To encourage sequential coherence, we introduce a coverage constraint:

$${\mathcal{L}}_{cov}=\sum _t\sum _{m}min(y_m^t,c_m^{t-1}),$$

(13)

where $c_m^{t-1}$ accumulates attention weights from previous steps, reducing repetitive selection.

The overall pointer loss combines cross-entropy and coverage penalties:

$${\mathcal{L}}_{ptr}={\mathcal{L}}_{ce}+\lambda {\mathcal{L}}_{cov}.$$

(14)

3.3.3. Expert Coordination via Gating Network

The gating network determines which expert should be activated for a given input instance. It processes ${\mathbf{z}}^{\left[CLS\right]}$ through a two-layer MLP:

$$\widehat{g}=\sigma (W_2\mathrm{ReLU}(W_1{\mathbf{z}}^{\left[CLS\right]}+{\mathbf{b}}_1)+{\mathbf{b}}_2).$$

(15)

During training, we supervise the gating decision using a binary flag $g_i\in \{0,1\}$ with loss:

$${\mathcal{L}}_{gate}=-{\displaystyle \frac{1}{N}}\sum _i[g_{i}log{\widehat{g}}_i+(1-g_i)log(1-{\widehat{g}}_i)].$$

(16)

At inference, $\widehat{g}>0.5$ activates the pointer branch; otherwise, the classifier branch is used.

3.3.4. Joint Optimization Objective

The total loss integrates all three learning signals:

$${\mathcal{L}}_{total}={\mathcal{L}}_{cls}+{\mathcal{L}}_{ptr}+{\mathcal{L}}_{gate}+\alpha {\mathcal{L}}_{align},$$

(17)

where $\alpha$ balances the alignment regularization term. This unified training strategy allows CMFN to simultaneously learn robust visual-textual fusion and adaptive reasoning capabilities.

3.4. Inference Strategy and Answer Decoding

During inference, the model first computes $\widehat{g}$ to determine the routing pathway. If $\widehat{g}<0.5$, the classifier provides a direct prediction; otherwise, the pointer decoder generates answers token-by-token. For stability, we adopt temperature-based sampling with beam search width $k=5$:

$$P\left(y_{1:T}\right|I,Q)=\prod _{t=1}^T\mathrm{softmax}\left({\displaystyle \frac{y^t}{\tau }}\right).$$

(18)

This approach balances exploration and accuracy, improving generalization on unseen question types.

Overall, CMFN presents a fully dynamic, multimodal reasoning framework that unifies discriminative and generative paradigms within a single Transformer backbone, delivering robust text-aware understanding and context-sensitive answer generation.

4. Experiments

In this section, we conduct a comprehensive empirical study of the proposed CMFN (Contextually-Guided Multi-Branch Fusion Network). Unless otherwise stated, we report results on VQA v2.0 Goyal et al. (2017) with the standard splits and official evaluation protocol. We (i) describe datasets and metrics, (ii) present implementation and training details, (iii) compare against strong state-of-the-art (SoTA) systems, and (iv) provide a thorough ablation and analysis on the contributions of OCR features, dual-branch routing, gating behaviour, robustness to OCR quality, computational efficiency, calibration, and qualitative error patterns. All results are averaged over three runs with different random seeds.

4.1. Benchmarks and Data Protocols

The VQA v2.0 dataset Goyal et al. (2017) contains 265,016 images sourced from MS-COCO and abstract scenes. Each image is associated with at least three open-ended questions, and every question has ten human-provided answers. The dataset is intentionally balanced—pairs of visually similar images yield distinct answers for the same question—to reduce annotation bias and promote genuine visual-language reasoning. The task spans diverse question families (e.g., Yes/No, Number, Other), and approximately $43\%$ of images include salient scene text. Consequently, success on VQA v2.0 requires both conventional visual reasoning and robust text-reading capability.

4.2. Evaluation Criteria

We adopt the official accuracy metric Antol et al. (2015), which is resilient to inter-annotator variance:

$$Acc\left(\mathtt{ans}\right)=min\left\{{\displaystyle \frac{\#\mathrm{humans}\mathrm{that}\mathrm{said}\mathtt{ans}}{3}},1\right\}.$$

(19)

Prior to scoring, standard answer normalisation is applied (lowercasing, digit canonicalisation, and removal of punctuation/articles). We report accuracy on test-dev and test-std, and include type-wise breakdowns (Yes/No, Number, Other) as commonly practised.

4.3. Training Setup and Hyperparameters

Our implementation is based on PyTorch with Adam optimisation. We fine-tune BERT-base and BERT-large backbones with initial learning rates $5\times {10}^{-5}$ and $1\times {10}^{-5}$, respectively. The maximum tokenised question length is $V=128$. For each image, we retain up to $E=50$ object regions and $M=50$ OCR tokens. The maximum decoding horizon is $T=12$. Embedding dimensionality is $d=768$, the candidate answer set size is $C=3129$, dropout is 0.3, weight decay is 0.05, and we use 12 attention heads in the Transformer layers; other settings follow BERT Kenton and Toutanova (2019). We train for 35 epochs with batch size 48. At inference, we use temperature scaling and (for the pointer branch) a small beam of width 5 unless stated otherwise.

4.4. Main Results vs. State-of-the-Art

We compare CMFN to strong baselines including UNITER Chen et al. (2019), VILLA Gan et al. (2020), ERNIE-VIL Yu et al. (2021), Oscar Li et al. (2020), and VinVL Zhang et al. (2021). Results on test-dev and test-std are summarised in Table 1. CMFN consistently outperforms prior art across both backbone sizes. Notably, CMFN-Large improves over VinVL-Large by ∼0.7–0.9% absolute on both splits under identical settings, indicating that adaptive routing between classification and OCR-pointer decoding yields measurable gains without sacrificing performance on general (non-text) cases.

To further dissect performance by answer type, we compare against VinVL in Table 2. CMFN delivers substantial gains on Number—a category tightly coupled with text-reading and fine-grained counting—while maintaining or slightly improving Yes/No and Other.

4.5. Ablation on OCR Feature Integration

We isolate the effect of OCR features by comparing CMFN without OCR (classification only), CMFN with OCR features but classification-only decoding, and full CMFN with OCR + dual routing. Table 2 (rows 1–3) shows an ∼1.0–1.3% boost in overall accuracy when OCR features are injected into the visual-textual stream, with the largest improvements on Number. This confirms that fine-grained textual embeddings (FastText/PHOC/appearance + box geometry) contribute complementary information that is underrepresented in purely object-centric features.

4.6. Analysis of Dual-Branch Routing

We partition the validation samples by answer source, i.e., whether the ground truth is composed from OCR tokens or lies in the fixed candidate set. Table 3 indicates that CMFN sharply improves OCR-sourced answers ($+36$ points over VinVL), while preserving performance on candidate-set answers.

4.7. Gating Behaviour and Calibration

The gating module performs a binary decision to select the appropriate reasoning expert. We measure its quality via classification accuracy and F1. CMFN attains $98.6\%$ accuracy and $0.82$ F1 on the validation routing labels. We also assess confidence calibration using Expected Calibration Error (ECE; lower is better) with temperature scaling at inference. Table 4 shows that CMFN’s dual-branch design slightly improves calibration compared to a single-branch classifier, likely because the routing reduces mode collapse on heterogeneous question types.

4.8. Sensitivity to OCR Quality

We bucket validation questions by OCR recognition confidence (low: $<0.6$, medium: $[0.6,0.8)$, high: $\ge 0.8$; averaged per-example). Table 5 shows that CMFN degrades gracefully with poorer OCR; the pointer decoder remains robust due to contextual fusion and coverage penalties that discourage repeated or spurious token selections.

4.9. Decoding Strategy and Answer Lengths

We probe the pointer branch with greedy vs. beam search and report accuracy stratified by target length. Short spans ( $\le 2$ ) benefit little from beam search, while longer spans see noticeable gains (Table 6). We thus adopt beam width 5 as default.

4.10. Computational Efficiency

We report average per-example latency on a single A100 (batch size 64). Despite the dual-branch design, CMFN’s overhead is modest; gating is a single MLP pass, and the pointer branch decodes only when needed.

Table 7. Efficiency comparison on test-dev. CMFN adds small overhead for routing/pointer decoding.

Model	#FLOPs (G)	Latency (ms)
VinVL-Base	78.2	35.1
Classifier-only (w/ OCR)	80.6	36.4
CMFN-Base (ours)	82.1	38.2

Table 7. Efficiency comparison on test-dev. CMFN adds small overhead for routing/pointer decoding.

Model	#FLOPs (G)	Latency (ms)
VinVL-Base	78.2	35.1
Classifier-only (w/ OCR)	80.6	36.4
CMFN-Base (ours)	82.1	38.2

4.11. Robustness to Noisy Text and Perturbations

We evaluate CMFN under synthetic text perturbations (random character flips, affine transforms) applied to OCR crops prior to recognition. Table 8 shows modest degradation relative to the baseline, reflecting benefits from PHOC and visual appearance channels in the OCR embedding, as well as question-conditioned fusion.

4.12. Ablation of Architectural Components

We ablate main components: (i) removing PHOC, (ii) removing appearance features, (iii) removing spatial boxes, (iv) dropping coverage loss, (v) disabling alignment loss. The results in Table 9 highlight that PHOC and spatial cues are particularly important for Number and Other, while ${\mathcal{L}}_{cov}$ and ${\mathcal{L}}_{align}$ contribute to stable decoding and better cross-modal grounding.

Figure 3. Case study with an example of model prediction.

Figure 3. Case study with an example of model prediction.

4.13. Human Study and Qualitative Inspection

We performed a small-scale human audit (n=200 randomly sampled validation questions; mixed Number/Other). Annotators judged whether model answers were (i) correct, (ii) partially correct (under/over-specified), or (iii) incorrect due to text misreading or wrong grounding. CMFN reduced “text-misread’’ errors by 28% relative to VinVL and produced fewer partially correct answers on multi-token responses (e.g., street names). Typical CMFN successes involve accurately composing multi-word OCR sequences or mixed alphanumeric strings, aligning well with the intended routing design.

4.14. Limitations and Reproducibility Notes

While CMFN improves text-heavy questions, performance still depends on OCR quality in extreme cases (low resolution, severe blur). Further, our pointer branch relies on detected tokens and cannot generate characters absent from OCR outputs. Future extensions could integrate OCR-free recognisers or masked image-to-text decoders. We release code, configs, and scripts to reproduce all reported numbers, including seed control and data pre-processing for the exact candidate set of size $C=3129$.

4.15. Effect of OCR Feature

As discussed, injecting OCR embeddings into the encoder improves Number and certain Other categories that reference written text (e.g., brand names). Compared to VinVL, the classifier-only CMFN variant (with OCR features fused but no pointer decoding) sees an overall gain (Table 2, row 2 vs. row 1), confirming that textual cues enhance cross-modal alignment even when the final decoder is a fixed-vocabulary classifier. The gains are further amplified when activating the dual routing pathway (row 3), demonstrating a complementary effect between perceptual enrichment (OCR features) and decoding flexibility (pointer branch).

4.16. Effect of Dual Routing Prediction Module

The dual-branch design targets heterogeneous question types. In practice, many questions can be solved reliably from a frequent-answer set, whereas a significant minority require copying/sequencing OCR tokens. CMFN’s gating decision separates these regimes effectively (Sec. 4.7), yielding +2% on Number vs. classifier-only CMFN and ∼+4% vs. VinVL (Table 2). The source-wise analysis (Table 3) further indicates that CMFN’s pointer branch closes a long-standing gap on OCR-derived answers, without materially harming candidate-set accuracy.

4.17. Qualitative Analysis

We observe that failures of prior art often stem from (i) selecting a frequent but semantically adjacent candidate (e.g., “main” for a specific street name), or (ii) producing an incomplete numeric string. CMFN reduces such errors by attending jointly to character-level PHOC and visual cues, then composing the sequence via coverage-regularised pointer decoding. Typical improvements include multi-token street names, bus route numbers, jersey numbers with leading zeros, and product codes—cases where pure classification is ill-suited.

5. Conclusion and Future Work

In this work, we presented the Contextually-Guided Multi-Branch Fusion Network (CMFN), a unified framework designed to address the challenges of visual question answering (VQA) in complex multimodal environments. Unlike traditional models that rely solely on object-level features or rigid classification-based decoding, CMFN introduces a context-driven reasoning paradigm capable of flexibly understanding both visual and textual information embedded in images.

Our model begins by constructing comprehensive multimodal representations that combine object-level region features with fine-grained OCR embeddings. These OCR features capture the semantic richness of text regions, enabling the model to understand contextual cues such as numbers, street names, or labels that are often crucial to accurate question answering. This enhanced representation forms the foundation upon which CMFN performs its reasoning process.

To handle the diverse nature of VQA questions, CMFN incorporates a dual-branch reasoning mechanism that dynamically routes each instance through one of two specialized branches. The first branch functions as a classifier, suitable for general questions whose answers are drawn from a predefined candidate set. The second branch employs a dynamic pointer network, which generates answers by composing tokens directly from detected OCR text when the question requires reading or understanding textual content. A gating network oversees this process, learning to identify which branch should be activated for each input instance based on its contextual features. Through this adaptive routing design, CMFN achieves a flexible balance between efficiency and reasoning precision.

Extensive experiments on the VQA v2.0 dataset demonstrate that CMFN consistently outperforms strong baselines, including UNITER, VinVL, and Oscar, across all question types. Notably, CMFN delivers substantial gains on text-related and “number” questions, which are typically considered challenging due to their dependency on accurate reading and contextual understanding. The results confirm that equipping the model with a text-aware routing mechanism enables a more human-like reasoning process and a deeper comprehension of scene semantics.

Beyond quantitative improvements, CMFN contributes conceptually to the field of multimodal reasoning. It exemplifies how dynamic, context-sensitive routing can bridge structured and unstructured modalities in a single end-to-end system. This perspective opens a promising research direction toward multimodal intelligence systems that can reason with similar adaptability and selectivity as humans.

Looking forward, we identify several avenues for future research. First, extending CMFN beyond static image-text understanding to video and temporal reasoning tasks could further enrich its contextual awareness and enable fine-grained spatio-temporal reasoning. Second, exploring token-level routing, where individual elements of visual or textual input are dynamically assigned to specific experts, could yield even more interpretable and efficient multimodal reasoning systems. Third, integrating CMFN into large multimodal language models (MLLMs) may allow it to operate in interactive, dialogue-based environments—facilitating question answering with multi-turn understanding, visual grounding, and natural language explanations.

In summary, CMFN offers a new perspective on how multimodal models can dynamically adapt their reasoning process to diverse types of visual-linguistic queries. By unifying visual perception, textual comprehension, and adaptive decision-making, CMFN provides both practical improvements and conceptual advances toward building more intelligent, context-aware multimodal systems. We believe that the proposed framework not only establishes a strong foundation for robust VQA but also paves the way for future research at the intersection of cognition-inspired reasoning and large-scale multimodal learning.