Contextual Knowledge Infusion via Iterative Semantic Tracing for Vision–Language Understanding

1. Introduction

Visual question answering (VQA) [3] has become an influential benchmark for assessing multimodal intelligence, prompting numerous advances in computational vision–language modeling [3,11,14]. Despite this progress, a fundamental limitation remains: most established models primarily rely on the visual signal and lack the ability to integrate the extensive external knowledge that many questions inherently require. For instance, when a question refers to taxonomic or functional categories, as in the example asking about the entity belonging to the class of eukaryotes, visual content alone is insufficient. Humans, however, naturally fuse perceptual information with background knowledge, enabling seamless transitions between what is seen and what must be recalled from memory.

Motivated by this discrepancy between human and model capabilities, several knowledge-oriented VQA datasets have emerged to provide a more grounded testing environment. The FVQA dataset [32] introduced a pipeline that integrates fact triplets from external sources such as ConceptNet [26], WebChild [28], and DBPedia [4]. More recently, the KRVQR dataset [7] expanded the scope of knowledge-based VQA by incorporating a richer and more diverse knowledge base alongside the questions. These datasets highlight the need for systems that do not merely identify objects but must retrieve, align, and reason over external information sources. Other datasets such as OK-VQA [16] and SelectSS [12] extend this idea by requiring external knowledge searches, but the absence of a predefined knowledge base differentiates their objectives from the setting considered in this work.

In parallel, graph-based reasoning approaches have gained traction in knowledge-augmented VQA [18,40]. These methods typically structure both visual and knowledge modalities into graph forms, enabling cross-graph alignment or multi-hop relational inference. Yet, they often assume that the reasoning dynamics of knowledge and images are symmetric. This assumption overlooks a crucial distinction: knowledge triplets contain explicit relational facts, whereas visual graphs implicitly encode spatial or contextual cues that may be noisy or incomplete. Treating these modalities uniformly can thus obscure the complementary nature of their contributions.

Figure 1. Motivational illustration showing why knowledge-based reasoning is essential in vision–language question answering. While the image provides only partial visual cues, answering the question requires connecting these cues with external factual knowledge (e.g., linking “cucumber” to its biological class “eukaryotes”). Humans naturally integrate perceptual understanding and stored knowledge, whereas models must learn to perform this multi-step fusion. This illustrates the core challenge addressed in this work: bridging implicit visual evidence with explicit knowledge representations through iterative reasoning.

Figure 1. Motivational illustration showing why knowledge-based reasoning is essential in vision–language question answering. While the image provides only partial visual cues, answering the question requires connecting these cues with external factual knowledge (e.g., linking “cucumber” to its biological class “eukaryotes”). Humans naturally integrate perceptual understanding and stored knowledge, whereas models must learn to perform this multi-step fusion. This illustrates the core challenge addressed in this work: bridging implicit visual evidence with explicit knowledge representations through iterative reasoning.

To address these limitations, we explore a more differentiated formulation of multimodal reasoning. In this work, we introduce KV-Trace, a model that separates explicit reasoning over structured knowledge from implicit reasoning over spatial visual entities. Instead of adopting predefined key–value formulations [17,34], our model constructs a dynamic memory architecture that progressively adapts its internal representations to the question. This approach allows KV-Trace to interpret knowledge triplets in a multi-faceted manner, jointly capturing the semantics of subjects, relations, and objects. Meanwhile, the visual modality is processed through a spatial-aware image graph whose nodes and edges encode object embeddings and relational geometry derived from Faster-RCNN detections [25]. Unlike traditional scene graph formulations, our graph remains lightweight while retaining sufficient structural cues for relational reasoning.

The interplay between explicit and implicit reasoning becomes particularly powerful when the question itself guides their interaction. Inspired by works that leverage common-sense priors for scene understanding [9,39], we enable the question-aware knowledge representation to inform the traversal of the spatial image graph. This design allows the model to sharpen object–relation associations in the image based on the most relevant external knowledge signals. Through iterative refinement, KV-Trace alternates between knowledge-centric interpretation and visually grounded inference, ultimately producing a multi-step reasoning trajectory that adapts to the question at hand.

Our contributions can thus be summarized in three main aspects. First, we design a dynamic knowledge memory module that constructs progressively updated representations of relevant fact triplets. Second, we introduce a guided graph reasoning mechanism that uses knowledge-aware representations to navigate a sparse spatial relational graph. Third, we conduct comprehensive experiments and ablations to assess interpretability and effectiveness, demonstrating that the explicit separation of knowledge and visual reasoning yields more faithful and robust behaviors. Collectively, these components offer a refined perspective on knowledge-based VQA and provide a pathway toward more human-like multimodal inference systems.

2. Related Work

2.1. Foundations of Visual Question Answering

The task of visual question answering (VQA), first introduced by Antol et al. [3], aims to evaluate a system’s capability to jointly reason over visual content and natural language. Early approaches [2,3,5,8,14,15] primarily adopted a CNN–RNN architecture in which the CNN encodes the image and an RNN (often an LSTM or GRU) encodes the question. These models typically follow a late-fusion paradigm, merging the representations through concatenation, element-wise product, or bilinear pooling to predict answers. Although simple, this line of research demonstrated the importance of effective cross-modal fusion. The emergence of attention mechanisms further advanced the field, enabling the model to selectively focus on image regions relevant to the question [1,14,35]. Attention-based models alleviated the bottleneck of compressing the entire image into a single vector, improving both interpretability and performance across standard VQA benchmarks.

Building on these foundations, graph-based reasoning began to attract interest due to its ability to represent relational structures explicitly. For example, Teney, Liu, and van den Hengel [29] encoded both the question and image as graph structures and used graph attention layers to propagate information across nodes. Subsequent efforts such as Hu et al. [10] and Wang et al. [31] implemented question-conditioned image graphs, allowing relationships between entities to be modulated by linguistic cues. In another direction, Norcliffe-Brown, Vafeias, and Parisot [20] explored constructing a fully connected graph over region proposals, generating question-dependent graph connectivity and applying graph convolutions to aggregate relational cues. These developments reflect a growing recognition that VQA benefits not only from local image patterns but also from structured reasoning capabilities.

Despite these advances, conventional VQA approaches remain limited when the question requires factual, commonsense, or domain-specific knowledge unseen in the image. The inability of standard architectures to incorporate external world knowledge motivates the exploration of knowledge-centric VQA, which forms the core focus of this paper.

2.2. Knowledge-Based VQA and External Reasoning

Knowledge-based VQA (KVQA) extends the standard VQA setting by requiring models to integrate visual content with external knowledge sources. Early works such as Wang et al. [32] and Marino et al. [16] formalized this problem by associating questions with structured or unstructured knowledge retrieved from sources such as ConceptNet [26], WebChild [28], and DBPedia [4]. Some initial systems [28,32] relied on template-based question parsing, translating questions into predefined slots used to retrieve relevant triplets from a knowledge base. However, these approaches often struggled to generalize beyond template coverage and were sensitive to linguistic variability.

Graph-based models offered a more flexible framework. Narasimhan, Lazebnik, and Schwing [18] constructed a fact graph and performed graph convolution to integrate information from relevant knowledge triplets. Later, Ziaeefard and Lécué [41] proposed role-aware attention mechanisms across fact graphs and image graphs, enabling reasoning over multimodal structures simultaneously. A more comprehensive system, MUCKO [40], represented the image using three complementary graphs—semantic, fact, and spatial—and iteratively propagated question-guided signals across all graphs. While these models showed promising results, many of them still relied on fixed representations of knowledge triplets and lacked the ability to dynamically adapt their reasoning process to question semantics.

In contrast, our method KV-Trace employs a dynamic key-value mechanism to encode triplets more flexibly, allowing the model to capture interactions between subjects, objects, and relations in a question-aware manner. Moreover, whereas many prior works leverage dense graph structures, KV-Trace utilizes a sparse spatial-aware graph, improving efficiency and reducing noise introduced by irrelevant visual relations. As an extension, we also explored integrating semantic graphs derived from dense captioning, examining how such high-level descriptions complement structured knowledge representation.

2.3. Key-Value Memory Architectures

Key-value memory networks [17], derived from early memory network formulations [27,33], represent a powerful mechanism for reasoning over large sets of symbolic facts. In traditional designs, a knowledge triplet $<subject,relation,object>$ is stored as a key-value pair where the key contains the subject and relation, and the value corresponds to the object. These architectures perform reasoning by first addressing relevant keys based on the query and then retrieving associated values. Key-value memories have been particularly impactful in knowledge-based question answering [17,34] due to their structured representation of factual information.

However, conventional key-value formulations impose rigid assumptions about what constitutes the “key” and the “value” in a knowledge triplet, often oversimplifying the semantic interdependence between triplet components. To address these limitations, our model KV-Trace introduces a dynamic memory design in which each triplet is encoded with all three components jointly and updated in a question-aware manner. This allows the model to reason symmetrically about subjects, relations, and objects rather than treating them asymmetrically. The enhanced flexibility of this design significantly improves the ability to capture nuanced knowledge dependencies required for multi-step reasoning in complex VQA scenarios.

2.4. Graph Neural Networks for Semantic and Spatial Reasoning

Graph neural networks (GNNs) have played a significant role in vision–language research due to their ability to capture multi-hop relational dependencies. Beyond VQA, GNNs have been used for scene graph generation, object grounding, and visual explanation tasks. Early methods applied spectral or spatial graph convolutions over object-level features, but recent work has shifted toward attention-based GNNs that dynamically adjust edges based on linguistic guidance. Techniques such as relational graph attention, neighborhood pooling, and hierarchical propagation have demonstrated superior expressiveness for capturing long-range dependencies across image regions.

In the context of knowledge-based reasoning, GNNs offer a natural way to merge symbolic triplets with visually grounded relational graphs. Systems that combine fact graphs with scene graphs have shown that cross-graph attention can propagate semantic cues across modalities efficiently. Our approach builds upon these insights by designing a sparse spatial-aware graph where nodes represent object category embeddings and edges encode relative geometric relations. This design balances expressiveness with computational tractability and avoids the noise inherent in dense graph construction.

2.5. Neural-Symbolic Models and Hybrid Reasoning

Recent years have witnessed renewed interest in neural-symbolic reasoning, which integrates differentiable neural networks with structured symbolic representations. Such models aim to combine the statistical generalization ability of deep networks with the interpretability and precision of symbolic logic. In VQA, neural-symbolic systems have attempted to parse questions into structured programs, match symbolic predicates to visual entities, or utilize differentiable logic operators to perform multi-step reasoning. Although these approaches often require supervised program annotations, they highlight the broader challenge of representing reasoning processes explicitly rather than relying solely on implicit feature fusion.

The ideas behind neural-symbolic reasoning naturally motivate our work. The dynamic key-value mechanism of KV-Trace can be interpreted as a form of differentiable symbolic storage, whereas the spatial-aware graph corresponds to a structured representation of relational visual knowledge. By interleaving updates across these two components, our model performs a hybrid form of reasoning that merges implicit neural inference with explicit knowledge manipulation.

Figure 2. Overview of the KV-Trace architecture. The model integrates dynamic key-value memory reasoning with spatial-aware visual graph inference. Symbolic cues from retrieved knowledge triplets interact iteratively with visual graph representations across multiple reasoning hops, culminating in a unified answer prediction.

Figure 2. Overview of the KV-Trace architecture. The model integrates dynamic key-value memory reasoning with spatial-aware visual graph inference. Symbolic cues from retrieved knowledge triplets interact iteratively with visual graph representations across multiple reasoning hops, culminating in a unified answer prediction.

2.6. Large Language Models and Knowledge Retrieval for VQA

With the emergence of large language models (LLMs), several recent studies have incorporated them into VQA pipelines for generating or retrieving external knowledge. LLMs have been used to reformulate questions, retrieve commonsense facts, or generate pseudo-rationales that guide visual encoders. However, LLM-driven retrieval often introduces noisy or hallucinated content, and integrating such unstructured text with visual reasoning remains challenging. Unlike retrieval-heavy pipelines, our model maintains a structured symbolic backbone grounded in well-defined triplets, ensuring consistency and interpretability during reasoning.

Multistep reasoning has long been recognized as a key challenge in VQA, particularly in tasks requiring relational comparison, compositional logic, or stepwise inference. Existing approaches include recurrent attention mechanisms, iterative graph refinement, memory-based controllers, and multi-hop reasoning layers. Although these models demonstrate impressive performance on compositional benchmarks, they often lack the capacity to integrate symbolic world knowledge. In contrast, KV-Trace explicitly builds iterative interactions between dynamic memory (symbolic reasoning) and a spatial-aware graph (visual reasoning), yielding a synergistic multi-hop reasoning process.

Commonsense knowledge plays a crucial role in bridging the gap between perception and abstract understanding. Several works incorporate commonsense relations, taxonomies, affordances, or functional attributes to enhance visual reasoning. Systems such as those leveraging ConceptNet or WordNet provide rich relational structures, but their integration into differentiable architectures remains nontrivial. By encoding relations directly into dynamic memory and propagating their effects into the visual graph, KV-Trace provides a principled way to enrich visual reasoning with structured commonsense cues.

3. Methodology

Given a question Q, an image I, and a structured knowledge base $K=\{f_1,f_2,\dots ,f_n\}$ composed of RDF triplets, the objective of a knowledge-based VQA system is to derive the correct answer A through a coherent sequence of symbolic and perceptual reasoning operations. Unlike conventional visual question answering, where answers can often be derived purely from image content, knowledge-based VQA requires the integration of structured factual information with visual and linguistic cues. In this work, we propose KV-Trace, a comprehensive multi-stage reasoning architecture that leverages symbolic key-value memory networks and spatial-aware image graph reasoning to achieve flexible and interpretable multi-hop inference.

The overall pipeline of KV-Trace consists of two tightly coupled reasoning processes: (1) an explicit symbolic reasoning module operating over a dynamically constructed key-value memory bank, and (2) an implicit visual reasoning module operating over a spatial-aware graph derived from detected objects in the image. Through iterative refinement, these two components influence and correct each other, resulting in richer multi-step inference behavior.

Below we expand the entire reasoning framework in detail, elaborating significantly on each technical component, introducing additional mathematical layers, additional submodules, and refining the conceptual flow far beyond the original description.

3.1. Dynamic Key-Value Memory Construction and Semantic Encoding

The objective of this stage is to extract a compact but semantically expressive set of knowledge triplets that are most relevant to the question–image pair, and then encode them into a differentiable key-value memory structure. Unlike traditional key-value memory networks [17,34], which store subject–relation pairs as keys and their objects as values, KV-Trace stores full triplet semantics in both keys and values, allowing the model to reason holistically over all triplet components.

3.1.1. Retrieval of Relevant Knowledge Facts

To identify the subset of the knowledge base most relevant to the current question and visual scene, we perform a multimodal matching procedure. The nouns mentioned in Q are extracted using Stanza [24], whereas Faster-RCNN [25] detects $\mathcal{O}=\{o_1,\dots ,o_r\}$ object proposals from the image. Each detected object label and each extracted noun is projected into the GloVe embedding space using $\varphi (·)$ [22].

For a given triplet $f_i=(e_1,r,e_2)$, we compute a multimodal compatibility score:

$$\begin{array}{cc}\hfill S_i& =\alpha \underset{a\in Q_{\mathrm{noun}}}{max}cos(\varphi \left(a\right),\varphi \left(e_1\right))+\beta \underset{b\in \mathcal{O}}{max}cos(\varphi \left(b\right),\varphi \left(e_1\right))\hfill \\ \hfill & +\gamma \underset{c\in Q_{\mathrm{noun}}\cup \mathcal{O}}{max}cos(\varphi \left(c\right),\varphi \left(e_2\right)),\hfill \end{array}$$

(1)

where $\alpha$, $\beta$, and $\gamma$ are learnable coefficients. We sort triplets by $S_i$ and retain the top $k=5$.

Additionally, we introduce a semantic filtering criterion:

$$\mathrm{retain}\left(f_i\right)=\mathbb{I}\left(max(cos(\varphi \left(e_1\right),\varphi \left(Q\right)),cos(\varphi \left(e_2\right),\varphi \left(Q\right)))>\tau \right)$$

where $\tau$ is a trainable adaptive threshold. This improves retrieval quality by removing noisy matches.

3.1.2. Embedding and Structuring of Knowledge Triplets

Let $F_{s_i}$, $F_{r_i}$, $F_{o_i}$ denote the embeddings of the subject, relation, and object of triplet $f_i$. We construct a unified semantic embedding:

$$\begin{array}{cc}\hfill F_i& =[F_{s_i};F_{r_i};F_{o_i}]\in {\mathbb{R}}^{3d_w},\hfill \end{array}$$

(2)

where $d_w$ is the GloVe dimension.

The memory key is computed as:

$$\begin{array}{c}\hfill k_i=\sigma (W_k{F}_i+b_k),\end{array}$$

(3)

and the value representation stores the decomposed embeddings:

$$\begin{array}{c}\hfill v_i=[F_{s_i},F_{r_i},F_{o_i}].\end{array}$$

(4)

To strengthen the relational consistency, we incorporate an auxiliary triplet factorization objective:

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{KG}}=\sum _i{∥F_{s_i}+F_{r_i}-F_{o_i}∥}_2^2,\end{array}$$

(5)

similar to translational knowledge graph embeddings, which stabilizes the structure and improves symbolic reasoning.

3.2. Spatial-Aware Visual Graph Construction and Geometric Reasoning

The second major component of KV-Trace is a spatial-relational graph that models interactions among detected objects. This structure allows relational reasoning on the visual domain, complementing the symbolic knowledge stored in the memory.

3.2.1. Node Initialization and Feature Composition

Each detected object $o_i$ contributes:

$$v_i=\varphi \left(\mathrm{label}\left(o_i\right)\right),b_i=[x_i^1,y_i^1,x_i^2,y_i^2].$$

In addition to GloVe embeddings, we enhance nodes using a learned visual projection:

$${\tilde{v}}_i=W_v[v_i;b_i]+b_v.$$

3.2.2. Enhanced Geometric Edge Encoding

We expand the original spatial encoding with additional normalized geometric terms:

$$\begin{array}{cc}\hfill r_{ij}^{\mathrm{geo}}=[& {\displaystyle \frac{x_i^c-x_j^c}{\sqrt{w_i{h}_i}}},{\displaystyle \frac{y_i^c-y_j^c}{\sqrt{w_i{h}_i}}},log{\displaystyle \frac{w_j}{w_i}},log{\displaystyle \frac{h_j}{h_i}},\hfill \\ \hfill & {\displaystyle \frac{w_j{h}_j}{w_i{h}_i}},{\displaystyle \frac{{\parallel b_i-b_j\parallel }_2}{\sqrt{w_i{h}_i}}}].\hfill \end{array}$$

(6)

We further incorporate a semantic interaction vector:

$$r_{ij}^{\mathrm{sem}}=v_i\odot v_j.$$

The final edge vector is:

$$e_{ij}=[r_{ij}^{\mathrm{geo}};r_{ij}^{\mathrm{sem}}].$$

3.2.3. Sparse Graph Construction via Mutual Proximity

We build a sparse graph by connecting $o_i$ to its mutual K nearest neighbors. This reduces noise from distant objects and significantly improves computational scalability.

3.3. Iterative Multi-Hop Reasoning Module

The iterative multi-hop reasoning mechanism in KV-Trace is designed to emulate a structured human-like inference process, where symbolic cues and visual evidence interact in alternating phases. Instead of treating reasoning as a single forward computation, the model maintains an evolving internal state that is refined over T iterative updates. At each iteration, symbolic attention, memory lookup, graph reasoning, and cross-modal fusion cooperatively update the latent belief representation. This iterative procedure offers two major benefits: (i) it enables the model to revisit earlier interpretations of the question or scene as more contextual evidence becomes available, and (ii) it naturally supports multi-hop reasoning chains that require progressive deduction through interdependent symbolic and visual cues.

Formally, the iterative reasoning procedure produces a sequence of tuples:

$$(q^1,m^1,R^1,I^1),\dots ,(q^T,m^T,R^T,I^T),$$

where each component is gradually refined, and the coupling among them allows the model to recursively update its interpretation of the question, the memory, and the visual graph. Below, we elaborate every constituent process and describe additional computational intuitions, intermediate latent structures, and expanded mathematical foundations that enable robust iterative reasoning in KV-Trace.

3.3.1. Question Encoding with Dynamic Reasoning Context

The question understanding module aims to extract a contextualized semantic representation that evolves with the iterative reasoning cycles. Unlike static encoders that summarize the question once, our formulation acknowledges that different reasoning iterations may require focusing on different parts of the question. For example, early iterations may require identifying key entities, while later iterations may prioritize relational terms or hidden logical constraints. To accommodate this need, KV-Trace dynamically reinterprets the semantic content of the question.

We first convert the tokenized question $Q=\{w_1,\dots ,w_S\}$ into a sequence of GloVe embeddings, which are then processed with a bidirectional LSTM:

$$\begin{array}{c}\hfill [h_1,h_2,\dots ,h_S]=\mathrm{BiLSTM}\left(Q\right).\end{array}$$

(1)

Each $h_s$ contains both future and past context, allowing the model to reason about long-range linguistic dependencies.

For the first reasoning step, we initialize the context vector using the terminal states of the BiLSTM:

$$c^1=[{\overrightarrow{h}}_S;{\overleftarrow{h}}_1].$$

This initialization captures both the global summary and directional contextual cues of the question.

From the second iteration onward, the context vector becomes an adaptive representation influenced by the symbolic and visual reasoning outputs from the previous step:

$$\begin{array}{c}\hfill c^t=W_4^t\mathrm{ReLU}\left(W_5[R^{t-1};I^{t-1}]\right).\end{array}$$

(4)

This update rule introduces a recurrent dependency across reasoning iterations, enabling the reasoning chain to revise its semantic focus as new evidence accumulates.

Next, attention over question words is computed:

$$\begin{array}{cc}\hfill {\alpha }_s^t& =\mathrm{softmax}\left(W_1\left(h_s\odot \left(W_2^t\mathrm{ReLU}\left(W_3{c}^t\right)\right)\right)\right),\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill q^t& =\sum _{s=1}^S{\alpha }_s^t{h}_s.\hfill \end{array}$$

(3)

Here, the dot-product interaction $h_s\odot \mathrm{ReLU}\left(W_3{c}^t\right)$ ensures that attention weights adapt to the evolving context vector. As $c^t$ changes, the model may switch its attention to different parts of the question, implicitly implementing multi-hop textual reasoning.

To further enhance reasoning depth, KV-Trace introduces an auxiliary semantic refinement update:

$${\tilde{q}}^t=\mathrm{LayerNorm}(q^t+W_q{c}^t),$$

which stabilizes the interaction between the previous context and the current attention-weighted summary. This representation ${\tilde{q}}^t$ is used in the symbolic memory lookup, enabling deeper textual reasoning at later iterations.

3.3.2. Symbolic Key Addressing and Value Reading

This module performs explicit symbolic reasoning by selecting relevant knowledge triplets from the dynamic key-value memory. The key addressing stage determines which stored triplets are relevant, while the value reading stage aggregates semantically meaningful components (subject, relation, object) conditioned on the question representation.

Hierarchical projections:

$$\begin{array}{cc}\hfill \widehat{q}& =\mathrm{ReLU}\left(W_6\mathrm{ReLU}\left(W_{7}q\right)\right),\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill {\widehat{k}}_i& =\mathrm{ReLU}\left(W_8\mathrm{ReLU}\left(W_9{k}_i\right)\right),\hfill \end{array}$$

(6)

map both the question and memory keys into a shared semantic space, allowing the model to compute relevance weights:

$$p_i=\mathrm{softmax}\left(\widehat{q}{\widehat{k}}_i^T\right).$$

(7)

To encourage sharper symbolic reasoning, we include a sparsity-inspired auxiliary constraint:

$$p_i^{\prime }={\displaystyle \frac{exp(\widehat{q}{\widehat{k}}_i^T/\tau )}{{\sum }_{j}exp(\widehat{q}{\widehat{k}}_j^T/\tau )}}$$

with a temperature parameter $\tau <1$. This gradually sharpens the distribution during training, enabling crisp multi-hop inference over factual knowledge.

Next, each triplet element undergoes a local nonlinear transformation:

$$\begin{array}{cc}\hfill {\widehat{t}}_{ij}& =\mathrm{ReLU}\left(W_{10}\mathrm{ReLU}\left(W_{11}{t}_{ij}\right)\right),\hfill \end{array}$$

(8)

and the model computes the element-wise importance weights:

$$\begin{array}{c}\hfill s_{ij}={\displaystyle \frac{1-\mathrm{softmax}\left(\widehat{q}{\widehat{t}}_{ij}^T\right)}{2}}.\end{array}$$

(9)

The symbolic embedding for triplet i is:

$${\widehat{t}}_i=\sum _{j=1}^3{s}_{ij}{\widehat{t}}_{ij}.$$

Finally, symbolic memory readout:

$$m^t=\sum _i{p}_i{\widehat{t}}_i,$$

(11)

acts as an explicit multi-hop symbolic reasoning vector. This representation evolves across iterations and plays a critical role in guiding the visual reasoning process, allowing high-order associations such as chaining relational facts or refining object-level hypotheses.

We further enhance symbolic reasoning by introducing a consistency-preserving transformation:

$${\tilde{m}}^t=\mathrm{LayerNorm}(m^t+W_m{q}^t),$$

which reinforces alignment between linguistic meaning and retrieved knowledge.

3.3.3. Cross-Modal Knowledge-Aware Question Fusion

Following the symbolic memory read step, the model fuses linguistic semantics and symbolic cues to form the cross-modal representation:

$$\begin{array}{c}\hfill R^t=W_{11}^t\mathrm{ELU}\left(W_{12}[q^t;m^t]\right).\end{array}$$

(12)

This fusion produces a latent vector that serves as a bridge between the symbolic memory and the visual graph reasoning module. The ELU activation increases representational stability, preventing vanishing gradients while allowing negative values to persist, which improves reasoning robustness in multi-hop scenarios.

To further strengthen cross-modal alignment, we introduce an auxiliary “semantic alignment regularizer”:

$${\mathcal{L}}_{\mathrm{align}}^t={\parallel q^t-W_a{m}^t\parallel }_2^2,$$

encouraging $q^t$ and $m^t$ to encode mutually consistent aspects of the question and relevant knowledge.

3.3.4. Node and Edge Attention for Visual Graph Reasoning

The spatial-aware graph constructed earlier now participates in multi-hop visual reasoning. Each iteration computes attention at both the node and edge levels, conditioned on the fused representation $R^t$.

Node attention:

$${\alpha }_i=\mathrm{softmax}({\omega }_{v}tanh(W_{13}{v}_i+W_{14}{R}^t)).$$

(13)

Edge attention:

$${\beta }_{ij}=\mathrm{softmax}({\omega }_{e}tanh(W_{15}{e}_{ij}+W_{16}{R}^t)).$$

(14)

These attentions selectively highlight visually important regions and spatial relations. Because $R^t$ evolves over iterations, the model’s visual focus naturally shifts across different reasoning hops—for example, zooming in on different object clusters or different relational patterns.

Additionally, we include a regularization term to encourage sharper edge dependencies:

$${\mathcal{L}}_{\mathrm{edge}}^t=\sum _{i,j}{\beta }_{ij}(1-{\beta }_{ij}).$$

This favors high-confidence relational decisions and prevents overly diffuse attention over the visual graph.

3.3.5. Multi-Head Graph Attention Aggregation

Visual graph reasoning is executed using multi-head attention, enabling the model to capture diverse relational patterns.

Message passing:

$$\begin{array}{cc}\hfill m_i^k& =\sum _{j\in {\mathcal{N}}_i}[{\alpha }_j{W}_{17}{v}_j;{\beta }_{ij}{W}_{18}{e}_{ij}],\hfill \end{array}$$

(15)

Node update:

$$\begin{array}{cc}\hfill h_i^k& ={\alpha }_i\mathrm{ReLU}\left(W_{19}[m_i^k;W_{20}{v}_i]\right).\hfill \end{array}$$

(16)

Multiple heads capture heterogeneous relational cues such as semantic similarity, geometric alignment, or relational consistency. The aggregated node representation:

$$\begin{array}{c}\hfill {\widehat{v}}_i=\mathrm{LayerNorm}\left(\mathrm{ELU}\left(W_{21}[h_i^1;...;h_i^H]\right)\right)\end{array}$$

(17)

provides a stable and expressive descriptor of each object after multi-hop attention.

Pooling:

$$I^t=\mathrm{MaxPooling}\left(\left\{v_i\right\}\right).$$

(18)

This yields the visual summary for iteration t. Over successive hops, $I^t$ becomes increasingly aligned with relevant symbolic knowledge extracted earlier.

3.4. Final Prediction and Optimization

After T rounds of symbolic and visual refinement, the final combined representation:

$$z=W_f[R^T;I^T],\widehat{y}=\mathrm{softmax}\left(z\right),$$

encodes both the multi-hop symbolic reasoning chain and the multi-step visual interpretation of the scene. This fusion allows the model to answer questions requiring complex deductions such as multi-entity relations, implicit inference, and relational chaining grounded in both factual and perceptual evidence.

The primary training loss is the cross-entropy objective:

$$\begin{array}{c}\hfill L=-{\displaystyle \frac{1}{N}}\sum _{i,c}{y}_{c}log{\widehat{y}}_c,\end{array}$$

(19)

which guides the model toward correct answer predictions.

To stabilize multi-hop reasoning, we incorporate additional regularizers:

1) Attention Regularization

$$L_{\mathrm{attn}}={\lambda }_1\sum _t{\parallel {\alpha }^t\parallel }_2^2,$$

encouraging structured and non-overly noisy attention patterns over the visual nodes.

2) Symbolic Entropy Regularization

$$L_{\mathrm{entropy}}={\lambda }_2\sum _{t}H\left(p_i\right),$$

which prevents symbolic key distributions from collapsing too early during training, supporting robust multi-hop inference.

3) Knowledge Graph Structural Regularizer

$${\mathcal{L}}_{\mathrm{KG}}=\sum _{f_i}{\parallel F_{s_i}+F_{r_i}-F_{o_i}\parallel }_2^2,$$

preserving relational consistency in the memory representations.

4) Cross-Modal Consistency Regularizer

$${\mathcal{L}}_{\mathrm{cross}}={\parallel R^T-W_c{I}^T\parallel }_2^2,$$

encouraging final symbolic and visual summaries to remain mutually coherent.

The complete training objective:

$$L_{\mathrm{total}}=L+L_{\mathrm{attn}}+L_{\mathrm{entropy}}+{\mathcal{L}}_{\mathrm{KG}}+{\mathcal{L}}_{\mathrm{cross}}$$

ensures that the model learns to perform iterative multi-hop reasoning in a stable, interpretable, and semantically consistent manner.

4. Experiments

In this section, we present a comprehensive empirical study of the proposed KV-Trace model. We first describe the datasets and task setup, followed by the evaluation metrics and implementation details. We then summarize the baseline methods used for comparison, report main quantitative results on two benchmark datasets (KRVQR and FVQA), and provide a series of ablation studies to understand the contribution of each architectural component. Finally, we offer qualitative and error analyses to shed light on how KV-Trace performs multi-hop reasoning in practice. All experiments are conducted under a unified protocol to ensure fair and reproducible comparisons.

4.1. Datasets and Task Setup

• KRVQR. In this paper we mainly focus on the KRVQR dataset [7], which is a large-scale and challenging benchmark specifically designed for knowledge-routed visual question answering. The dataset contains $32,910$ images paired with $157,201$ question–answer pairs. Following Cao et al. [7], we adopt the official partition, where the data are split into training, validation, and test sets with proportions of $60\%$, $20\%$, and $20\%$, respectively. Each instance is accompanied by a set of knowledge triplets drawn from an external knowledge base (KB), and each question is annotated with its answer grounded in the combination of image content and KB facts.

A distinctive property of KRVQR is that the questions are explicitly categorized by reasoning complexity. Roughly $43.5\%$ of the questions require one-step reasoning, while the remaining $56.5\%$ require two-step reasoning. For one-step reasoning questions, the answer can be derived by using a single relation, where the relation may be found in the KB and/or in the visual scene. In contrast, two-step reasoning questions cannot be answered by looking at one triplet alone; the model must infer over a composition of two relations, which may involve combining visual relations with KB relations, or chaining two KB relations in a multi-hop manner. This mixture of one- and two-step questions makes KRVQR an ideal testbed for evaluating the ability of KV-Trace to perform iterative, multi-hop reasoning over both symbolic and visual structures.

• FVQA. We also evaluate our model on the FVQA dataset [32], which is an earlier benchmark for fact-based visual question answering. FVQA consists of $2,190$ images and $5,826$ questions. The standard split divides these into $2,927$ training questions and $2,899$ test questions. Each question is accompanied by a set of corresponding KB triplets, and the answer is directly supported by at least one fact in the KB. In contrast to KRVQR, all questions in FVQA are designed to be solvable with one-step reasoning [32], i.e., they can be answered by retrieving and using a single fact triplet from the knowledge base together with the image. As a result, FVQA provides a complementary evaluation scenario where the primary challenge is the correct retrieval and grounding of knowledge, rather than multi-hop reasoning depth.

By considering both KRVQR and FVQA, our experimental setup covers a spectrum of reasoning requirements: from single-hop KB retrieval with visual grounding, to more complex multi-hop inference that must combine multiple relational cues across modalities.

4.2. Evaluation Metrics

Following the literature on knowledge-based VQA [7,32], we adopt top-1 and top-3 accuracy as our main quantitative evaluation metrics. For the KRVQR dataset, we follow Cao et al. [7] and report top-1 accuracy, i.e., the percentage of questions for which the most probable predicted answer matches the ground truth answer:

$$\mathrm{Acc}@1={\displaystyle \frac{1}{N}}\sum _{i=1}^N\mathbb{I}\left(arg\underset{c}{max}{\widehat{y}}_{ic}=y_i\right).$$

For the FVQA dataset, we report both top-1 and top-3 accuracies, denoted as Acc@1 and Acc@3, respectively. Top-3 accuracy assesses whether the ground truth answer appears in the top three predictions produced by the model:

$$\mathrm{Acc}@3={\displaystyle \frac{1}{N}}\sum _{i=1}^N\mathbb{I}\left(y_i\in \mathrm{Top}3\left({\widehat{y}}_i\right)\right).$$

These metrics capture not only the model’s best guess but also its ability to produce a small set of plausible alternatives, which is especially relevant for knowledge-based settings where multiple semantically related answers might be close competitors.

4.3. Implementation Details and Training Protocol

• Model Configuration. We implement KV-Trace using the PyTorch deep learning framework [21]. The question encoder is a two-layer bidirectional LSTM with hidden size 512 in each direction, resulting in 1024-dimensional contextual token embeddings. We apply a dropout rate of $0.1$ to the LSTM outputs and intermediate fully connected layers to mitigate overfitting. GloVe embeddings [22] are used to initialize word vectors for both question and knowledge base entities, and these embeddings are fine-tuned during training.

The dynamic key-value memory module maintains embeddings of dimension 300 for keys (triplet-level representations) and stores decomposed subject/relation/object vectors as values. For the spatial-aware image graph, node and edge feature dimensions are set to 1024, and we use $H=4$ attention heads in the graph attention network. Unless otherwise stated, the number of reasoning steps T in the iterative module (Section 3.3) is set to 2, which is consistent with the maximum number of reasoning hops required by the KRVQR dataset. We empirically verify this choice through ablation studies.

• Optimization. The model is trained end-to-end using the Adam optimizer [13], with a base learning rate of $1\times {10}^{-4}$. We apply a warm-up strategy in the first two epochs: the learning rate is linearly increased from 0 to the base rate, which stabilizes early training when gradients can be noisy. Starting from epoch 20, we decay the learning rate by a factor of $0.1$ at fixed intervals based on validation performance. The batch size is set to 128, and we train for approximately 40 epochs, choosing the checkpoint with the best validation accuracy for final evaluation on the test sets.
• Reasoning Steps and Hyperparameters. The choice of $T=2$ reasoning steps aligns with the composition depth in KRVQR. We evaluate different values of T in the ablation section and find that deeper iterative reasoning tends to overfit or propagate noise when the dataset does not require more hops. All hyperparameters such as dropout, hidden sizes, and memory dimensions are tuned on the validation split of KRVQR and then reused for FVQA, yielding a fair cross-dataset comparison.

4.4. Iterative Reasoning Algorithm

For clarity and reproducibility, we summarize the iterative reasoning procedure of KV-Trace in Algorithm 1. This algorithm describes how question encoding, key-value memory reading, cross-modal fusion, graph attention, and iterative context updates are combined into a coherent step-by-step inference pipeline.

The above procedure makes explicit how multi-hop reasoning is realized in practice: the model repeatedly re-encodes the question conditioned on the evolving context, performs symbolic retrieval from the memory, and uses the retrieved knowledge to steer graph-based visual reasoning.

4.5. Baselines and Comparative Systems

We compare KV-Trace against a diverse set of baseline methods, including both general VQA models and dedicated knowledge-based VQA systems.

On KRVQR, we report results for:

Algorithm 1: Iterative Reasoning Module of KV-Trace

• Q-type [7]: a question-type prior baseline that predicts answers using only the question type distribution and ignores the image and KB.
• LSTM [7]: an LSTM-based model that encodes the question and image features without explicit knowledge reasoning.
• FiLM [23]: a feature-wise linear modulation model that conditions visual features on language representations.
• MFH [38]: a multimodal factorized high-order pooling model that learns high-capacity interactions between image and question features.
• UpDown [1]: a bottom-up and top-down attention model using object-level features for VQA.
• MCAN [37]: a state-of-the-art modular co-attention network for general VQA.
• Mucko [40]: a knowledge-based VQA approach that uses multiple graphs (semantic, fact, and visual) and inter-graph attention.
• KM-net [6]: a key–value memory network for reasoning over knowledge bases.

These baselines were implemented and evaluated on KRVQR in Cao et al. [7], except Mucko which we reimplement using the publicly available descriptions due to the absence of released code.

On FVQA, we additionally consider:

• FVQA (Ensemble) [32]: the original FVQA ensemble model.
• $STT{F}^1$ and $O{B}^2$: two baselines from Narasimhan, Lazebnik, and Schwing [18], Narasimhan and Schwing [19].
• GRUC [36]: the state-of-the-art model on FVQA that integrates dense captioning and a multi-graph reasoning architecture.
• GRUC (w/o Semantic graph): an ablated version of GRUC without the semantic graph.

All results for KV-Trace are averaged over 5 independent runs with different random seeds, and we report the mean accuracy.

4.6. Main Results on KRVQR

Table 1 compares the performance of KV-Trace with prior methods on the KRVQR dataset. We also report a variant of KV-Trace that incorporates additional dense captioning information for image regions.

From Table 1, we observe that KV-Trace substantially outperforms all baselines, including specialized knowledge-based models such as Mucko and KM-net. In particular, KV-Trace achieves a gain of more than 6 percentage points over KM-net, highlighting the benefit of combining dynamic key–value memory with spatial-aware graph reasoning and iterative multi-hop inference. When augmented with dense captioning (KV-Trace + Dense Captioning), the performance improves slightly further, although a paired t-test over multiple runs shows that this improvement is only marginally significant, indicating that most of the gains come from the core architecture of KV-Trace.

4.7. Main Results on FVQA

Table 2 reports top-1 and top-3 accuracies on the FVQA dataset. We compare KV-Trace against the FVQA ensemble model, several baselines from Narasimhan, Lazebnik, and Schwing [18], Narasimhan and Schwing [19], Mucko, and GRUC variants.

On FVQA, KV-Trace already matches or slightly surpasses the performance of GRUC without leveraging dense captions. When dense captioning is incorporated, KV-Trace + Dense Captioning achieves new state-of-the-art performance, improving top-1 accuracy by approximately $1.6$ percentage points and top-3 accuracy by over 4 points compared to GRUC. Interestingly, dense captioning plays a more pronounced role on FVQA than on KRVQR, which may be attributed to FVQA’s bias toward single-hop reasoning, where local textual descriptions of image regions can more directly support factual grounding.

4.8. Ablation: Number of Reasoning Steps

To investigate the effect of the number of reasoning steps T in the iterative module, we vary T from 1 to 4 while keeping all other hyperparameters fixed. Table 3 summarizes the results on the KRVQR dataset.

The results indicate that the best performance is obtained with two reasoning steps, which aligns well with the dataset’s annotation that includes up to two-hop questions. Using only one step leads to slightly lower performance, suggesting that a single-pass inference is insufficient to capture multi-hop dependencies. In contrast, using more than two steps (three or four) causes a noticeable degradation in performance, likely due to overfitting and accumulated noise from unnecessary iterative updates when the data do not require deeper reasoning. This confirms that the iterative design of KV-Trace is effective when properly aligned with the underlying reasoning depth of the dataset.

4.9. Ablation: Memory Module Variants

We next examine the contribution of the proposed dynamic key–value memory module by replacing it with alternative memory structures and measuring performance on KRVQR. We consider three variants: (i) a simple average-embedding memory, (ii) a standard key–value memory [17], and (iii) the proposed dynamic key–value memory. Table 4 presents the results.

The comparison demonstrates that the proposed dynamic key–value memory substantially outperforms the other two variants. The conventional key–value memory, which uses subject–relation pairs as keys and the object as value, falls short in this setting because many questions in KRVQR involve reasoning about different elements of a triplet (e.g., subject or relation, rather than only the object). A simple average-embedding memory performs slightly better than the standard key–value memory, as it does not impose a rigid subject–relation/object decomposition, but it still lacks the flexibility and expressivity of our dynamic triplet-level memory representation. These observations confirm the importance of designing a memory structure that can symmetrically reason about all parts of a knowledge triplet.

4.10. Ablation: Knowledge-Guided Graph Reasoning

Finally, we analyze the impact of injecting external knowledge into the visual graph reasoning module. To this end, we compare the full KV-Trace model with a variant that disables knowledge-guided reasoning on the spatial-aware image graph, i.e., the graph attention is conditioned only on the question encoding but not on the retrieved knowledge triplets. Results are shown in Table 5.

We observe that removing knowledge guidance from the visual graph leads to a drop of nearly $1.8$ percentage points in top-1 accuracy. This indicates that symbolic knowledge not only supports answer prediction directly but also helps the model attend to more relevant regions and relations in the image. In other words, knowledge-guided graph reasoning provides an important synergy: the retrieved triplets refine the visual attention patterns, and the structured visual context, in turn, disambiguates which knowledge facts are most useful for answering the question.

4.11. Additional Analyses and Discussion

Beyond the standard ablations, we also explore the effect of varying the number of retrieved facts and the maximum number of detected objects used to construct the spatial graph. Table 6 illustrates the influence of different values of k, the number of top-ranked knowledge triplets stored in the memory.

The results suggest that retrieving too few facts may omit important information, whereas retrieving too many facts introduces noise that can distract the reasoning process. Empirically, using $k=5$ retrieved facts provides a good balance between coverage and noise, which is consistent with the setting described in the methodology.

In a similar manner, we control the number of detected objects r used to build the spatial-aware graph. Using very small r can cause the model to miss relevant visual entities, while very large r increases computational cost and may lead to an overly dense or noisy graph. We find that $r=36$ yields a favorable trade-off, echoing prior work in object-based VQA models.

4.12. Qualitative and Error Analysis

To further understand how KV-Trace performs multi-hop reasoning, we conduct qualitative analyses on randomly selected examples from the KRVQR test set. For correctly answered cases, we observe that the model tends to (i) assign high attention scores to the fact triplets that are semantically closest to the question entities and relations, and (ii) focus visual attention on object regions that are strongly linked to these facts. For example, when asked about the biological category of an object and its relation to another object, the model first retrieves the relevant taxonomic triplet from the KB and then attends to the corresponding visual object before generating the final answer.

For incorrectly answered questions, common failure patterns include: (1) ambiguity in the visual scene, such as multiple similar objects (e.g., several persons or tools) where the model attends to the wrong instance; (2) incomplete or noisy knowledge retrieval, where the relevant fact is not among the top-k retrieved triplets; and (3) compounding errors in iterative reasoning when the initial attention is misaligned, leading later iterations to reinforce an incorrect hypothesis.

These observations suggest that future work could focus on improving ambiguity resolution in crowded scenes, designing more robust knowledge retrieval mechanisms, and exploring uncertainty-aware iterative reasoning strategies that can detect and correct misaligned intermediate steps rather than amplifying them.

Overall, the experimental results demonstrate that KV-Trace consistently outperforms strong baselines across two benchmarks, while the detailed ablations confirm the importance of each architectural component, including the dynamic key–value memory, knowledge-guided graph reasoning, and the iterative multi-hop inference module.

5. Conclusions

In this work, we introduced KV-Trace, a comprehensive multi-step reasoning framework that unifies explicit symbolic inference and implicit visual relational reasoning through a dynamically structured key-value memory and a spatially grounded graph-based visual encoder. Our approach departs from conventional knowledge-based VQA pipelines by tightly integrating knowledge retrieval, memory-augmented reasoning, and graph-structured visual understanding into a single, iterative architecture capable of refining its internal representations over multiple reasoning cycles.

Across the reasoning process, KV-Trace first performs explicit knowledge grounding by addressing relevant triplets stored in a dynamic memory bank, where each key-value slot encodes the semantic components of an RDF fact. This symbolic inference is then complemented by implicit visual graph reasoning, where spatially related objects and their relational cues are propagated through multi-head graph attention, enabling the model to incorporate both conceptual knowledge and visual regularities. The iterative design ensures that these two streams of reasoning—linguistic-symbolic and relational-visual—reinforce one another, allowing the model to progressively converge toward a coherent, knowledge-consistent interpretation of the question and image.

Extensive experiments on two challenging knowledge-based visual question answering benchmarks, KRVQR and FVQA, demonstrate the effectiveness of the proposed architecture. KV-Trace achieves new state-of-the-art results on both datasets, benefiting from its ability to reason over multi-hop knowledge chains and spatial configurations in complex scenes. The improvements are consistent across one-step and two-step reasoning questions, indicating that the design is inherently flexible and well-suited for tasks requiring diverse types of inferential behavior, ranging from attribute lookup to relational chaining to cross-modal grounding.

Furthermore, the modular design of KV-Trace opens several promising research directions. For instance, the dynamic memory component could be extended to handle richer forms of structured knowledge, including hierarchical knowledge graphs, probabilistic rules, or large-scale encyclopedic corpora. Similarly, the graph reasoning module could benefit from more advanced geometric or causal relational modeling to capture fine-grained scene dynamics or deeper causal dependencies. Another fruitful direction lies in exploring reinforcement learning–based training schemes that allow the model to autonomously select reasoning strategies, thereby enabling adaptive multi-hop inference beyond fixed-step architectures.

In summary, KV-Trace offers a unified, interpretable, and extensible framework for knowledge-grounded multimodal reasoning. By integrating structured symbolic memories with spatially aware graph attention, it provides a principled approach toward bridging visual perception and knowledge-based inference. We believe that this work not only contributes a strong method for KVQA but also lays the foundation for future progress in multimodal reasoning systems that require richer forms of symbolic–visual integration, such as embodied QA, scientific diagram understanding, and real-world decision-making agents.