Relation-Sensitive VQA with A Unified Tri-Modal Graph Framework

1. Introduction

Visual question answering (VQA) tasks demand an intelligent agent capable of interpreting a natural-language question and producing answers grounded in image content. A predominant line of research has adopted scene graphs (SGs) to represent visual semantics, encoding objects, attributes, and relationships as graph entities that can be exploited for structured reasoning [1]. By translating an image into a symbolic yet visually grounded representation, SGs allow models to interact with semantically organized information, bridging the gap between linguistic queries and visual evidence. These structured representations further exhibit advantages over classic feature-based pipelines due to their interpretability and modular decomposition of visual scenes [2,3].

Figure 1. A motivational illustration showing why tri-modal reasoning is necessary for complex VQA. Conventional scene-graph models often bias toward object features and therefore fail to correctly integrate attributes and relations. Our framework conceptually decomposes a scene graph into object-centric and relation-centric modalities while explicitly preserving attribute cues, enabling TriUnity-GNN to construct a unified semantic space that better aligns visual structure with question intent.

Figure 1. A motivational illustration showing why tri-modal reasoning is necessary for complex VQA. Conventional scene-graph models often bias toward object features and therefore fail to correctly integrate attributes and relations. Our framework conceptually decomposes a scene graph into object-centric and relation-centric modalities while explicitly preserving attribute cues, enabling TriUnity-GNN to construct a unified semantic space that better aligns visual structure with question intent.

A wide range of efforts has attempted to incorporate scene graphs into reasoning mechanisms. Several approaches treat SGs as probabilistic structures, progressively updating node states from question-conditioned instructions [4]. Other lines of research integrate graph neural networks (GNNs) into scene graph learning [5,6], enabling joint encoding of object nodes and relational edges before mapping these encodings to the answer space. These works demonstrate the clear benefits of structuring visual semantics. Nevertheless, despite their progress, a persistent performance gap remains when models face complex reasoning problems involving rich attribute hierarchies or relational dependencies, and these limitations hinder the applicability of SG-based VQA systems in real-world settings [7].

A central challenge arises from the uneven emphasis placed on distinct semantic dimensions. Many conventional models inadvertently prioritize object features, while underutilizing the nuanced roles of attributes or the structural significance of relations. For example, in questions requiring attribute–relation interactions, prior methods often misidentify attributes because the relational context is not sufficiently integrated into the object representation. Such false attribute inference commonly occurs when the reasoning pipeline cannot connect contextual relational cues with fine-grained attributes. Similarly, in scenarios that require identifying spatial or functional relations, many models fail to capture the governing relational patterns, resulting in missing relational cues. These limitations stem from insufficiently balanced information propagation within the scene graph and inadequate representational alignment across different semantic types. Although GNN-based models aggregate neighboring information [8], they frequently fail to maintain attribute specificity. Meanwhile, soft-instruction reasoning techniques [4] tend to treat attributes as secondary modifiers, leading to suboptimal multi-type reasoning.

Motivated by these challenges, we propose a comprehensive scene-graph reasoning paradigm termed TriUnity-GNN, designed to address the tri-modal imbalance inherent in existing VQA methods. Our framework explicitly restructures the scene graph into two complementary modal views: an object-centric view where nodes represent objects and edges encode relations, and a relation-centric view where the roles are inverted, assigning relations as nodes and objects as edges. This dual construction yields richer multi-scale perspectives that existing pipelines seldom explore. By leveraging dual encoders tailored for these two modalities, TriUnity-GNN captures semantic complementarities between object-level and relation-level structures, enabling deeper structural reasoning across the scene.

Beyond this structural redesign, our model introduces an enhanced message-passing mechanism that significantly improves semantic interaction. Unlike conventional GGNN variants that primarily accumulate node information, our formulation emphasizes bi-directional, attribute-aware propagation pathways that allow attributes, objects, and relations to exchange information more holistically. Through this process, the model produces a tri-modal, unified representation sensitive to all knowledge types encoded in the scene graph. Importantly, the attribute signals are explicitly fused after the graph encoders, ensuring that their contributions are preserved rather than overwhelmed by object-dominant features. To align the unified visual representation with linguistic intent, we incorporate a multi-head attention fusion driven by question embeddings, ensuring that the final answer is grounded in both the semantic structure of the scene graph and the contextual semantics of the question.

Our contributions are threefold. First, we diagnose and articulate the fundamental limitations of existing SG-based reasoning frameworks, emphasizing the need for tri-modal balance among objects, relations, and attributes. Second, we develop TriUnity-GNN, a novel architecture that enhances reasoning capability by integrating a dual-modality encoding design with a tri-modal message-passing structure. Third, extensive experiments on challenging datasets demonstrate that our method substantially improves the accuracy and interpretability of VQA models, confirming the necessity of unified multi-semantic graph reasoning for complex question understanding.

2. Related Work

In this section, we provide a comprehensive review of prior research that forms the conceptual foundation of our TriUnity-GNN framework. To present a clearer map of the research landscape, we reorganize the literature into several thematic directions, each addressing a different aspect of visual understanding, graph-based semantic modeling, or multimodal reasoning. Compared with prior summaries, our discussion greatly expands the scope and depth, aiming to uncover the fundamental motivations behind unifying objects, attributes, and relations for structured VQA reasoning.

2.1. Visual Question Answering: Classical Architectures and Limitations

Earlier VQA systems rely heavily on sequential language encoders, such as LSTMs and later BERT-like pre-trained transformers [9], combined with CNN-based visual backbones [10]. These pipelines typically fuse modalities through attention mechanisms [11,12], yielding performance gains by aligning spatial visual regions with textual cues. Despite effectiveness, these methods inherently lack structured visual reasoning: they operate on dense feature maps, making relational dependencies implicit and often difficult to interpret or manipulate. Transformer-based VQA systems [22] further push performance but suffer from high computational cost, limited transparency, and insufficient control over disentangled semantic types within images [23]. A growing body of research suggests that feature-only VQA pipelines struggle with tasks requiring multi-hop reasoning, attribute disambiguation, or fine-grained relational behavior—core limitations that structured scene-graph approaches attempt to address.

2.2. Visual–Semantic Structures and Scene Graphs

Scene graphs have emerged as a powerful intermediate representation for capturing the semantic structure of images. Scene Graph Generation (SGG) systems [21] extract objects, attributes, and relations from visual regions, converting pixel-level evidence into symbolic structures that explicitly express semantic constraints. The appeal of SGs lies in their interpretability and modularity, qualities desirable for high-level reasoning tasks such as VQA, captioning, and visual commonsense inference [3]. Early reasoning methods such as NSM [4] adopt soft traversal strategies, updating node-level probabilities according to question-derived instructions. Other methods utilize GGNN-style propagation to encode semantic structure [5,6,24]. Yet most existing pipelines underestimate the importance of attribute signals and treat relations merely as auxiliary edges rather than key semantic entities. This imbalance causes incorrect reasoning in attribute-heavy or relation-dominated queries, motivating a rethinking of SG usage for multimodal reasoning.

2.3. Scene Graph Reasoning for VQA and Its Inherent Bottlenecks

Although scene graphs provide structured semantics, many SG-based VQA systems inherit a structural bottleneck: they frequently encode SGs in a uni-directional or object-dominant manner. Attributes often collapse into weak descriptors attached to nodes, while relations are treated as pairwise edges without deeper functional semantics. Sequential reasoning models [4] capture some multi-step processes but struggle to jointly track attribute-object interactions and relational dependencies. GNN-based methods [6] improve representational quality but still process nodes and edges asymmetrically, which limits their ability to maintain balance across the tri-modal components of SGs. Our TriUnity-GNN addresses these issues by disentangling and reconstructing SGs into two complementary modalities—object-centric and relation-centric—and by amplifying attribute signals via explicit fusion mechanisms.

2.4. Graph Neural Networks for Structured Reasoning

The development of GNNs [14] has significantly influenced structured reasoning across domains such as knowledge graphs [15,16], molecular analysis, and 3D scene understanding. Different GNN variants—GCN, GAT, and GGNN—offer various trade-offs between aggregation expressiveness, computational efficiency, and inductive biases. However, classical GNNs cannot easily encode graphs with heterogeneous node/edge types or complex label schemas, both essential properties of scene graphs. Many GNNs treat attributes as passive features rather than semantic entities, causing loss of fine-grained detail during propagation. Moreover, relation embeddings are often dominated by object embeddings due to message imbalance. Our model overcomes these challenges by introducing a tri-directional message-passing paradigm that ensures balanced semantic flow among objects, attributes, and relations.

2.5. Multimodal Transformers and Their Interaction with Graph Structures

Transformers have reshaped multimodal learning by enabling large-scale cross-attention between text and vision. Models such as ViLBERT, VisualBERT, and UNITER [22] demonstrate strong representation learning capabilities but operate on dense region features rather than structured graph entities. Several hybrid approaches integrate transformers with scene graphs, yet they typically treat SGs as side information or shallow priors rather than a full-fledged reasoning substrate. Additionally, transformers often overshadow graph-based components due to their massive capacity, making it difficult to enforce interpretability. To contrast, TriUnity-GNN embraces structure first, treating SG components as primary reasoning elements and leveraging attention only for alignment with textual queries.

2.6. Attribute Modeling in Multimodal Reasoning

Attributes frequently serve as crucial discriminative cues for answering fine-grained questions (e.g., color, material, state). However, many VQA and SG-based pipelines treat attributes as marginal modifiers attached to objects, leading to representation collapse and weaker attribute sensitivity. Recent works attempt to incorporate attributes more explicitly, yet they often focus only on visual appearance without reasoning about attribute–relation dependencies. Our approach explicitly models attribute features at multiple stages—pre-encoding, mid-level fusion, and post-aggregation—to ensure attributes contribute equally alongside objects and relations.

2.7. Relation-Centric and Functional Reasoning

Understanding relations such as spatial alignment, functional interaction, and temporal continuity is essential for compositional reasoning. Recent studies highlight the difficulty of encoding relations when models depend primarily on object-centric embeddings. Relation-dominated instances—such as “Where is A relative to B?” or “Why is C affecting D?”—often require multi-hop propagation and structural inversion. We adopt a fundamentally different viewpoint: by constructing a relation-centric modality in which relations become nodes and objects become edges, TriUnity-GNN enables explicit modeling of relational structures, enhancing the interpretability and depth of reasoning.

2.8. Explainability and Structured Interpretability in VQA

Interpretability has become increasingly important for trustworthy AI. SGs are inherently interpretable, but many existing SG-based reasoning frameworks dilute the interpretive power by collapsing heterogeneous signals into homogeneous embeddings. In contrast, our model preserves explicit structural pathways for attributes and relations, making the reasoning trace visible and semantically grounded. This aligns with ongoing research efforts emphasizing structured explanations, symbolic grounding, and reasoning transparency.

While prior work contributes valuable insights into VQA, SG generation, and GNN-based reasoning, existing pipelines do not provide balanced tri-modal integration. Our TriUnity-GNN differs fundamentally in three aspects: (1) it restructures SGs into dual complementary modalities rather than encoding them monolithically; (2) it introduces tri-directional message-passing to ensure balanced semantic propagation; (3) it integrates attributes through an explicit fusion mechanism instead of treating them as peripheral features. These design choices allow our model to capture multi-level semantics in a more holistic and interpretable manner.

Figure 2. Overview of the proposed TriUnity-GNN framework for visual question answering. The model begins with an image–question pair that branches into two processing streams: (1) a Scene Graph Generator that extracts objects, relations, and attributes from the image, and (2) a Question Encoder that produces a semantic vector representation of the question. The extracted scene graph is reorganized into two complementary semantic views, an Object-Significant Graph ($G_{obj}$) and a Relation-Significant Graph ($G_{rel}$), each processed by its own GGNN Encoder with Gated Propagation. Attribute features are aggregated and injected into both graph views. A Cross-View Alignment Head enforces consistency between the object-centric and relation-centric embeddings. The outputs from both views are fused with the question representation through a Cross-Modal Attention module, yielding a reasoning vector that captures multimodal semantic dependencies. Finally, an Answer Classifier predicts the most likely answer.

Figure 2. Overview of the proposed TriUnity-GNN framework for visual question answering. The model begins with an image–question pair that branches into two processing streams: (1) a Scene Graph Generator that extracts objects, relations, and attributes from the image, and (2) a Question Encoder that produces a semantic vector representation of the question. The extracted scene graph is reorganized into two complementary semantic views, an Object-Significant Graph ($G_{obj}$) and a Relation-Significant Graph ($G_{rel}$), each processed by its own GGNN Encoder with Gated Propagation. Attribute features are aggregated and injected into both graph views. A Cross-View Alignment Head enforces consistency between the object-centric and relation-centric embeddings. The outputs from both views are fused with the question representation through a Cross-Modal Attention module, yielding a reasoning vector that captures multimodal semantic dependencies. Finally, an Answer Classifier predicts the most likely answer.

3. Methodology

In this section, we present the technical details of our proposed TriUnity-GNN framework for visual question answering. The core idea is to explicitly disentangle and then re-unify three complementary semantic facets that naturally arise in scene graphs: objects, relations, and attributes. Instead of treating the scene graph as a monolithic structure, we build two complementary graph views (object-significant and relation-significant) and perform message passing over both of them with a carefully designed gated propagation scheme. The resulting node- and edge-level representations are then fused with attribute information and aligned with the linguistic representation of the question by a cross-modal attention module, before being fed into an answer classifier.

Formally, let an input image be denoted by I and its associated natural-language question by a token sequence $Q=(w_1,w_2,\dots ,w_T)$. Our scene graph generator [19] produces a scene graph $\mathcal{G}$ consisting of a set of object nodes $\mathcal{N}$, a set of directed relation edges $\mathcal{E}$, and attribute annotations for each node. In what follows, we first describe how we encode the question and construct dual-view scene graphs, and then explain the structure of the TriUnity-GNN encoders, the fusion module, and the answer prediction head. Finally, we introduce the training objective and several auxiliary regularization terms that further stabilize learning.

3.1. Question Representation with Recurrent Encoding

We start by encoding the question into a dense vector representation that captures its global semantics as well as fine-grained compositional structure. Each token $w_t$ is first mapped to a pre-trained GloVe embedding [17]. Let $\mathbf{E}\in {\mathbb{R}}^{\left|\mathcal{V}\right|\times d_e}$ be the embedding matrix, where $\left|\mathcal{V}\right|$ is the vocabulary size and $d_e$ is the embedding dimension. The embedding of the t-th token is

$${\mathbf{x}}_t=\mathbf{E}\left[w_t\right]\in {\mathbb{R}}^{d_e}.$$

(1)

We then feed the sequence $({\mathbf{x}}_1,\dots ,{\mathbf{x}}_T)$ into a uni-directional or bi-directional LSTM, which we denote generically as $\mathrm{LSTM}(·)$. The recurrent update at step t is expressed as

$${\mathbf{h}}_t^Q=\mathrm{LSTM}({\mathbf{x}}_t,{\mathbf{h}}_{t-1}^Q),$$

(2)

where ${\mathbf{h}}_t^Q\in {\mathbb{R}}^{d_h}$ is the hidden state at time t. We use the final hidden state as the global question encoding,

$$q={\mathbf{h}}_T^Q\in {\mathbb{R}}^{\dim },$$

(3)

where dim is the dimensionality of the question representation. To better preserve token-level information and facilitate fine-grained attention, we also retain all intermediate hidden states ${\left\{{\mathbf{h}}_t^Q\right\}}_{t=1}^T$ for later use in cross-modal alignment if needed.

In practice, we often apply layer normalization and a residual projection on top of q to obtain a stabilized representation:

$$q^{\prime }=\mathrm{LN}(W_{q}q+b_q)+q,$$

(4)

where $W_q$ and $b_q$ are trainable parameters. For clarity, we use q to denote the final normalized question vector in the rest of this section.

3.2. Dual Semantic Views of the Scene Graph

We next describe how we reorganize the scene graph into two complementary semantic views that emphasize objects and relations, respectively. This design is crucial for achieving a balanced treatment of object-centric and relation-centric information in TriUnity-GNN.

Object-Significant Graph. We denote the object-significant graph by $G_{\mathrm{obj}}=(\mathcal{N},\mathcal{E})$, where each node in $\mathcal{N}$ corresponds to an object instance detected in the image and each directed edge in $\mathcal{E}$ represents a relation between two objects. Let $\mathcal{N}={\left\{n_i\right\}}_{i=1}^{\left|\mathcal{N}\right|}$ and $\mathcal{E}={\left\{e_k\right\}}_{k=1}^{\left|\mathcal{E}\right|}$. For $n_i,n_j\in \mathcal{N}$ and $e_k\in \mathcal{E}$, the triplet $\langle n_i-e_k-n_j\rangle$ encodes a semantic relation $e_k$ directed from object $n_i$ to object $n_j$. This view preserves the intuitive notion of “objects as nodes, relations as edges” that has been widely adopted in previous scene graph literature.

Relation-Significant Graph. To compensate for the object-dominant nature of $G_{\mathrm{obj}}$, we construct a relation-significant graph $G_{\mathrm{rel}}$ in which relations become first-class nodes and objects act as edges. Formally, we define $G_{\mathrm{rel}}=(\mathcal{E},\tilde{\mathcal{N}})$, where each node corresponds to a relation $e_i\in \mathcal{E}$, and each edge corresponds to an object $n_k\in \mathcal{N}$ that is shared by two relations. For $e_i,e_j\in \mathcal{E}$ and $n_k\in \mathcal{N}$, the pattern $\langle e_i-n_k-e_j\rangle$ indicates that the two relations $e_i$ and $e_j$ are connected through the common object $n_k$. This inversion yields a complementary structural perspective in which relational dependencies become easier to track, especially when answering questions that are predominantly relation-centric (e.g., spatial or functional reasoning).

Attribute Types and Node Properties. Attributes form the third pillar of our tri-modal representation. We define a set of attribute types $\mathcal{L}$, and let $L=\left|\mathcal{L}\right|$ denote the number of attribute categories (e.g., color, material, state). For each node $n_i\in \mathcal{N}$, we maintain a collection of $(L+1)$ property vectors

$${\left\{n_i^l\right\}}_{l=0}^L,$$

where $n_i^0$ encodes the object name (category) embedding and $n_i^l$ ($l\ge 1$) denotes the embedding of the l-th attribute of $n_i$. These embeddings can be obtained either from pre-trained word embeddings or from a learned attribute embedding table. To aggregate the attributes into a single vector, we often use an attention-based pooling:

$${\alpha }_i^l={\displaystyle \frac{exp\left({\mathbf{u}}_a^{\top }{n}_i^l\right)}{{\sum }_{m=0}^{L}exp\left({\mathbf{u}}_a^{\top }{n}_i^m\right)}},a_i=\sum _{l=0}^L{\alpha }_i^l{n}_i^l,$$

(5)

where ${\mathbf{u}}_a$ is a learnable attention vector and $a_i$ is the attribute-aware representation of node $n_i$. This attribute summary is later integrated into the graph encoder and fusion module, ensuring that attributes contribute explicitly to the final reasoning representation.

3.3. TriUnity-GNN Encoders and Message Passing

To encode both $G_{\mathrm{obj}}$ and $G_{\mathrm{rel}}$, we instantiate two GGNN-based encoders that share the same architectural template but operate on different graph views. The object-view encoder focuses on learning rich object-centric representations, while the relation-view encoder emphasizes relational semantics. The dual structure of TriUnity-GNN ensures that the final representation allocates comparable capacity to objects and relations.

Graph Representation as Input Tuples. Before encoding, each scene graph is transformed into an information tuple $(\mathcal{N},\mathcal{E},A_{\mathrm{in}},A_{\mathrm{out}})$:

• $\mathcal{N}$ is the set of node embeddings (initially derived from object names and attributes or relation labels).
• $\mathcal{E}$ is the set of directed edges specifying valid connections.
• $A_{\mathrm{in}}$ is the adjacency matrix that describes incident (incoming) edges for each node.
• $A_{\mathrm{out}}$ is the adjacency matrix that describes outgoing edges for each node.

For node $n_i$, its hidden state at time step t is denoted by $h_i^t$. At $t=0$, we initialize $h_i^0$ using the corresponding name and attribute embeddings, e.g.,

$$h_i^0=[n_i^0;a_i]\in {\mathbb{R}}^{d_h},$$

(6)

where $[·;·]$ denotes vector concatenation and zero-padding can be used to match dimensions when necessary.

Message-Passing Module. To enhance information transfer between nodes and incident edges, we design a message-passing (MP) module that replaces the simple linear transformations used in standard GNNs. Consider the local structure $\langle n_i,e_k,n_j\rangle$, where $e_k$ is a directed edge from node $n_i$ to node $n_j$. We denote the embedding of edge $e_k$ by $e_k$ and the hidden state of node $n_j$ by $h_j$. We inject the pair $[e_k,h_j]$ as a sequence into a bidirectional GRU, and the initial hidden state of the GRU is set to $h_i$. The GRU thus summarizes the influence of neighbor node $n_j$ and edge $e_k$ on node $n_i$.

Formally, the aggregated incident and outgoing information gains for node $n_i$ at a given iteration are

$$\begin{array}{c}M{P}_i\left(A_{\mathrm{in}}\right)={\sum }_{k,j}^{\langle n_i,e_k,n_j\rangle \in A_{\mathrm{in}}}\mathrm{GRU}([e_k,h_j],h_i),\end{array}$$

(7)

$$\begin{array}{c}M{P}_i\left(A_{\mathrm{out}}\right)={\sum }_{k,j}^{\langle n_j,e_k,n_i\rangle \in A_{\mathrm{out}}}\mathrm{GRU}([e_k,h_j],h_i),\end{array}$$

(8)

where $M{P}_i\left(A_{\mathrm{in}}\right)$ and $M{P}_i\left(A_{\mathrm{out}}\right)$ correspond to the incident and outgoing message summaries, respectively. Intuitively, this module learns how much information from each neighboring configuration should be propagated toward $n_i$, conditioned on both the current state of $n_i$ and the attributes of its neighbors.

Gated Propagation Module. Having obtained the messages from incoming and outgoing edges, we concatenate them to form the overall context vector for node $n_i$:

$$\begin{array}{c}{k}_i^t=[M{P}_i^t\left(A_{\mathrm{in}}\right),M{P}_i^t\left(A_{\mathrm{out}}\right)],\end{array}$$

(9)

which collects information from all adjacent nodes and edges at time t. We then combine $k_i^{t-1}$ with the previous hidden state $h_i^{t-1}$ to compute a candidate update via a GRU-like gating mechanism:

$$\begin{array}{c}{c}_i^t=[h_i^{(t-1)},k_i^{(t-1)}]W+b,\end{array}$$

(10)

$$\begin{array}{c}{z}_i^t=\sigma \left(U^z{c}_i^t\right),r_i^t=\sigma \left(U^r{c}_i^t\right),\end{array}$$

(11)

where W, $U^z$, and $U^r$ are trainable weight matrices, and $\sigma (·)$ denotes a non-linear activation function (we use ReLU in practice). The update and reset gates, $z_i^t$ and $r_i^t$, adaptively control the information flow between the newly aggregated neighborhood messages and the previous state of the node.

The candidate hidden state and the final update are then given by

$$\begin{array}{c}{\tilde{h}}_i^t=tanh(U_1{k}_i^{(t-1)}+U_2(r_i^t\odot h_i^{(t-1)})),\end{array}$$

(12)

$$\begin{array}{c}{h}_i^t=(1-z_i^t)\odot h_i^{(t-1)}+z_i^t\odot {\tilde{h}}_i^t,\end{array}$$

(13)

where $U_1$ and $U_2$ are trainable parameters, and ⊙ denotes element-wise multiplication. After T propagation steps, we obtain the final hidden states ${\left\{h_i^T\right\}}_i$ for all nodes. For each node $n_i$, a graph-aware embedding $g_i$ is computed as

$$g_i=\sigma \left(f(h_i^T,n_i)\right),$$

(14)

where $f(·)$ is a multi-layer perceptron (MLP) that takes the concatenation of $h_i^T$ and the initial node embedding $n_i$ as input. We apply this encoder both on $G_{\mathrm{obj}}$ and $G_{\mathrm{rel}}$, resulting in dual sets of embeddings $\left\{g_i^N\right\}$ and $\left\{g_j^E\right\}$ that describe object-centric and relation-centric semantics, respectively.

To encourage consistency between these two views, we further introduce a simple contrastive alignment objective at the graph level. Let $z_{\mathrm{obj}}$ and $z_{\mathrm{rel}}$ be the mean-pooled embeddings from the two encoders. We define

$${\mathcal{L}}_{\mathrm{align}}={∥{\displaystyle \frac{z_{\mathrm{obj}}}{\parallel z_{\mathrm{obj}}{\parallel }_2}}-{\displaystyle \frac{z_{\mathrm{rel}}}{\parallel z_{\mathrm{rel}}{\parallel }_2}}∥}_2^2,$$

(15)

which encourages the two modalities to agree in a shared latent space while still retaining their complementary characteristics.

3.4. Cross-Modal Fusion with Attribute-Enriched Representations

Once the dual encoders produce node and relation features, we incorporate attribute information and perform cross-modal fusion with the question representation. Let $G^N$ and $G^E$ denote the feature maps from the object and relation encoders, respectively. For each object node i, we fuse its encoder output $g_i^N$ with all attribute embeddings ${\left\{n_i^l\right\}}_{l=0}^L$ to obtain the fused node representation $F_i^N$; for each relation edge j, we merge $g_j^E$ with its original edge embedding $e_j$ to obtain $F_j^E$:

$$F_i^N=\left\{\begin{array}{c}[g_i^N,n_i^0]\hfill \\ \dots \hfill \\ [g_i^N,n_i^L]\hfill \end{array}\right.,F_j^E=[g_j^E,e_j],F=[F^N,F^E],$$

(16)

where $F^N$ and $F^E$ stand for the sets of fused node and edge features, respectively, and F is the concatenation of them into a unified full-scale feature map. In practice, we further apply a learned projection to map all elements in F into a common dimensionality, followed by layer normalization and optional dropout.

To combine the visual graph representation with the textual question representation, we use a multi-head attention mechanism. Given the feature map F as the key-value store and the question vector q as the query, we compute the reasoning vector r as

$$\begin{array}{c}r=\mathrm{Attention}(F,q).\end{array}$$

(17)

Here, the attention operation can be instantiated as scaled dot-product attention with multiple heads:

$$\begin{array}{cc}\hfill \alpha & =\mathrm{softmax}\left({\displaystyle \frac{\left(W_{q}q\right){\left(W_{k}F\right)}^{\top }}{\sqrt{d_k}}}\right),\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill r& =W_o\left(\alpha W_{v}F\right),\hfill \end{array}$$

(19)

where $W_q$, $W_k$, $W_v$, and $W_o$ are trainable matrices and $d_k$ is the key dimension. This attention mechanism allows TriUnity-GNN to selectively focus on the most relevant objects, relations, and attributes given the question context, thereby enabling more precise reasoning.

3.5. Answer Prediction and Training Objective

The final stage of our model is the answer prediction head, which maps the fused reasoning representation to a distribution over the candidate answer set. We first concatenate the question vector q and the reasoning vector r to form the joint representation:

$$u=[q,r].$$

(20)

This vector is then fed into a two-layer MLP $f(·)$ with non-linear activation and dropout:

$$\begin{array}{cc}\hfill u^{\prime }& =\varphi (W_{1}u+b_1),\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill s& =W_2{u}^{\prime }+b_2,\hfill \end{array}$$

(22)

where $\varphi (·)$ is typically a ReLU or GELU activation, and $s\in {\mathbb{R}}^{\left|\mathcal{A}\right|}$ are the unnormalized scores for all candidate answers $\mathcal{A}$. We then obtain the predicted answer distribution via a softmax:

$$p(a\mid I,Q)=\mathrm{softmax}\left(s\right),$$

(23)

and take the answer with maximum probability as the model prediction:

$$\begin{array}{c}\widehat{a}=arg\underset{a\in \mathcal{A}}{max}p(a\mid I,Q)=argmax\left(\mathrm{softmax}\left(f\left((q,r)\right)\right)\right).\end{array}$$

(24)

Given the ground-truth answer $a^{*}$, we use the standard cross-entropy loss as the main supervision signal:

$${\mathcal{L}}_{\mathrm{vqa}}=-logp(a^{*}\mid I,Q).$$

(25)

To further regularize the model and encourage consistent representations across the dual graph views, we incorporate the alignment loss ${\mathcal{L}}_{\mathrm{align}}$ defined earlier and an ${\ell }_2$ weight decay term ${\mathcal{L}}_{\mathrm{reg}}$. The overall training objective is

$$\mathcal{L}={\mathcal{L}}_{\mathrm{vqa}}+{\lambda }_{\mathrm{align}}{\mathcal{L}}_{\mathrm{align}}+{\lambda }_{\mathrm{reg}}{\mathcal{L}}_{\mathrm{reg}},$$

(26)

where ${\lambda }_{\mathrm{align}}$ and ${\lambda }_{\mathrm{reg}}$ are hyperparameters that trade off the contribution of additional regularizers. This composite objective encourages TriUnity-GNN to learn not only accurate answer predictions but also coherent and balanced tri-modal graph representations.

4. Experiments

In this section, we conduct an extensive empirical study to validate the effectiveness of the proposed TriUnity-GNN framework. All experiments are performed on large-scale visual question answering benchmarks that provide scene-graph annotations or allow reliable scene-graph generation, including VG-GroundTruth, Motif-VG, and GQA. Unless otherwise stated, we use the same scene-graph generator [19] for all graph-based methods in order to isolate the effect of the reasoning architectures themselves. We first describe the overall setup and implementation details, and then report results on each dataset, followed by a series of in-depth analyses such as question-type sensitivity, error-rate decomposition, ablation studies, robustness evaluation, and computational efficiency comparison.

4.1. Experimental Setup and Implementation Details

We follow the standard data splits for VG-GroundTruth, Motif-VG and GQA. For a fair comparison, all graph-based baselines and our method share the same scene-graph generator [19]; only the reasoning modules differ. Images are resized to a fixed resolution, and object proposals are obtained via a Faster-RCNN backbone pre-trained on Visual Genome. We extract appearance features for each object region and relation features for each pair of objects, which are then combined with category and attribute embeddings to form the initial node and edge representations.

All models are optimized with Adam using an initial learning rate of $1{\mathrm{e}}^{-4}$, a batch size of 64, and an exponential learning rate decay schedule. For TriUnity-GNN, we set the hidden dimensionality of each GGNN encoder to 512, the number of message-passing steps to $T=3$, and the number of attention heads in the cross-modal fusion module to 4. Unless otherwise specified, we train each model for 20 epochs and select the best checkpoint on the validation set according to overall accuracy. To mitigate randomness, every experiment is repeated three times with different random seeds, and we report the average performance.

4.2. Overall Performance on VG Benchmarks

Table 1 summarizes the quantitative results on VG-GroundTruth and Motif-VG, broken down by question type. Across both datasets, TriUnity-GNN clearly surpasses all graph-based baselines. On VG-GroundTruth, the overall accuracy improves from 71.2% (ReGAT) to 76.1%, yielding an absolute gain of nearly 5 points. On Motif-VG, which uses automatically generated scene graphs and is therefore more challenging, the improvement is even more pronounced: we achieve 73.5% overall accuracy compared to 69.9% of the strongest baseline.

The per-type breakdown reveals that the gains are not limited to a particular category. For “What” questions, which focus mostly on object categories and attributes, TriUnity-GNN reaches 76.8% and 80.5% accuracy on VG-GroundTruth and Motif-VG, respectively, showing that explicit attribute modeling and dual-view aggregation significantly benefit object-centric queries. For “Where” questions, which rely heavily on spatial relations, the relation-significant graph allows the model to better capture complex spatial layouts, leading to improvements of 9.8 points on VG-GroundTruth and 5.8 points on Motif-VG compared to NSM. Meanwhile, the “Who” questions often involve human-centric entity recognition; our method again attains the best scores, suggesting that the tri-modal representation can robustly disambiguate people and their contextual roles.

The most striking improvements appear in the “Why” category. Such questions require multi-step reasoning over objects, relations and attributes simultaneously (for example, understanding that a monitor is dark because a light is off behind it). Our method improves the accuracy from 92.3% (ReGAT) to 98.2% on VG-GroundTruth and from 91.8% to 96.8% on Motif-VG. This confirms that unifying object-centric and relation-centric reasoning with attribute-aware fusion is particularly beneficial for high-level causal or explanatory queries.

4.3. Evaluation on the GQA Benchmark

We now turn to the GQA dataset, which provides a more diverse set of compositional questions and richer structural annotations. The results in Table 2 show that TriUnity-GNN achieves the highest overall accuracy (72.30%) among all compared models, including strong transformer-based systems such as LXRT and graph-centric models such as NSM and ReGAT. The gain over ReGAT is about 1.8 points in overall accuracy, despite our model being trained under essentially the same supervision regime.

A closer inspection of the metrics reveals that the advantage of TriUnity-GNN is especially apparent in open-ended questions. On the “Open” subset, our method outperforms the second-best baseline by almost 11 points (73.52% vs. 62.58%). These questions typically require combining multiple visual cues—such as attribute states, spatial relationships, and object identities—to arrive at the correct answer, which aligns well with the strengths of our tri-modal scene-graph representation.

Moreover, the “Distribution” metric shows that our predicted answer distribution remains well-aligned with the ground-truth distribution, with one of the lowest discrepancy scores (3.65). This indicates that TriUnity-GNN does not simply memorize frequent answers but instead maintains a balanced prediction pattern across classes. For binary (yes/no) questions, our performance is comparable to ReGAT and slightly higher than NSM, but the margin is smaller. This observation is consistent with the fact that yes/no questions often require subtle semantic judgments that are less directly grounded in explicit scene-graph structures, whereas our model is particularly strong when the answer can be traced back to specific objects, relations, and attributes.

4.4. Question-Type Sensitivity and Robustness

To better understand how robust TriUnity-GNN is with respect to linguistic complexity, we group questions by their token length and report accuracy in Table 6. Across all datasets, the model maintains strong performance as the question length increases from short (1–5 words) to very long (>15 words). The degradation is gradual rather than abrupt; for example, on VG-GroundTruth, the accuracy drops from 78.2% to 73.8% when moving from the shortest to the longest group, indicating that the question encoder and cross-modal fusion module can cope well with long and compositional queries.

Qualitatively, shorter questions tend to involve simple attribute queries (e.g., “What color is the car?”), while longer questions involve multiple relational constraints (e.g., “What is the man holding who is standing behind the car near the tree?”). The dual encoders of TriUnity-GNN help maintain reasoning performance by allowing the model to separately track object-centric and relation-centric chains, which are then fused via attention.

4.5. Error-Rate Decomposition on Motif-VG

To examine in which aspects TriUnity-GNN improves over previous work, we perform an error-rate analysis on Motif-VG and categorize incorrect predictions into three types: relation errors (where the primary failure lies in misinterpreting relations), object errors (incorrect object grounding or classification), and attribute errors (incorrect attribute assignment such as color or state). Table 4 presents the results.

Compared to FSTT and ReGAT, TriUnity-GNN drastically reduces the error rates in all three categories. The relation error rate drops from 44.6% to 32.9%, reflecting the strength of the relation-significant graph and the message-passing mechanism that explicitly propagate relational cues. Object errors are more than halved relative to FSTT (25.2% vs. 58.6%), which demonstrates that jointly modeling attributes and relations helps disambiguate visually similar entities by exploiting their contextual semantics. Finally, attribute errors are reduced from 49.6% (FSTT) and 22.8% (ReGAT) down to 16.8%, corroborating that our explicit attribute fusion is effective in preventing false attribute selection.

4.6. Ablation Study on Architectural Components

Table 3 reports a detailed ablation study on VG-GroundTruth. We start from a raw GGNN network without our proposed enhancements, denoted as Base. When the base model is applied to only the object-significant graph (Base-Obj) or only the relation-significant graph (Base-Rel), the accuracy remains low (35.4% and 35.2%, respectively), indicating that focusing solely on one modality is insufficient.

Table 3. Ablation analysis on VG-GroundTruth. “MP” denotes the message-passing module, “Dual” indicates the dual encoder structure, “attr” stands for explicit attribute modeling, “rela” for the relation-significant branch, and “QF” for cross-modal question fusion.

Models	Acc. (%)
Base	35.4
Base-Obj	35.4
+MP	39.3(+3.9)
+Dual	67.9(+32.7)
Base-Rel	35.2
+MP	38.8(+3.6)
+Dual	67.7(+32.5)
TriUnity-GNN(full)	76.4
+ (w/o attr)	72.5(−3.9)
+ (w/o rela)	75.1(−1.3)
+ (w/o QF)	55.7(−20.7)

Table 3. Ablation analysis on VG-GroundTruth. “MP” denotes the message-passing module, “Dual” indicates the dual encoder structure, “attr” stands for explicit attribute modeling, “rela” for the relation-significant branch, and “QF” for cross-modal question fusion.

Models	Acc. (%)
Base	35.4
Base-Obj	35.4
+MP	39.3(+3.9)
+Dual	67.9(+32.7)
Base-Rel	35.2
+MP	38.8(+3.6)
+Dual	67.7(+32.5)
TriUnity-GNN(full)	76.4
+ (w/o attr)	72.5(−3.9)
+ (w/o rela)	75.1(−1.3)
+ (w/o QF)	55.7(−20.7)

Table 4. Error-rate decomposition on Motif-VG, categorized by whether the incorrect answer is primarily attributed to relation recognition, object grounding, or attribute identification. TriUnity-GNN substantially reduces errors in all three categories.

Models	Relation Err.	Object Err.	Attribute Err.	#Samples
FSTT [5]	45.9%	58.6%	49.6%	27,810
ReGAT [6]	44.6%	47.3%	22.8%	27,810
TriUnity-GNN (ours)	32.9%	25.2%	16.8%	27,810

Table 4. Error-rate decomposition on Motif-VG, categorized by whether the incorrect answer is primarily attributed to relation recognition, object grounding, or attribute identification. TriUnity-GNN substantially reduces errors in all three categories.

Models	Relation Err.	Object Err.	Attribute Err.	#Samples
FSTT [5]	45.9%	58.6%	49.6%	27,810
ReGAT [6]	44.6%	47.3%	22.8%	27,810
TriUnity-GNN (ours)	32.9%	25.2%	16.8%	27,810

Adding the message-passing (MP) module yields consistent improvements of about 3–4 points for both object-based and relation-based variants. This confirms that injecting a GRU-based message-passing mechanism enables more expressive aggregation of edge and neighbor information than simple linear updates. When we further introduce the dual encoder structure (Dual) that jointly encodes both graph views, the performance jumps to about 68%. This large gain (over 30 points) illustrates that balancing object and relation information is essential for high-quality scene-graph representations.

The full TriUnity-GNN model, which additionally incorporates explicit attribute fusion and cross-modal question fusion (QF), reaches 76.4% accuracy. Removing attribute modeling (w/o attr) leads to a 3.9-point drop, showing that attributes cannot be treated merely as side information but should be explicitly fused with node features. Removing the relation-significant branch (w/o rela) causes a smaller but still noticeable decrease (1.3 points), suggesting that relations are particularly important for a subset of questions, such as spatial reasoning. Finally, discarding question fusion (w/o QF) severely degrades performance (a decrease of about 20.7 points), which highlights that tri-modal graph reasoning must be tightly guided by the textual query to provide meaningful answers.

4.7. Robustness and Generalization Analysis

Beyond standard accuracy, we also investigate how well TriUnity-GNN generalizes across datasets and question distributions. First, we observe that the relative ranking of our model with respect to baselines remains consistent between VG-GroundTruth and Motif-VG, despite the latter relying on automatically generated scene graphs that inevitably contain noise. This suggests that the dual encoders and message-passing modules are robust to moderate graph imperfections and can still extract reliable reasoning signals.

Second, the “Distribution” metric in Table 2 indicates that our predictions are not overly biased toward frequent answers. Combined with the error-rate analysis in Table 4, this implies that TriUnity-GNN not only improves average accuracy but also reduces systematic biases, for instance by better handling rare attribute combinations or uncommon relational patterns.

4.8. Qualitative Reasoning Behaviour

To gain qualitative insight, we manually inspect a diverse set of question–image pairs across datasets. For object-centric questions, the attention weights in the fusion module tend to focus on nodes whose attributes are most relevant to the queried property, such as color or material. For relation-centric questions (e.g., “What is in front of the bus?”), the relation-significant graph helps the model first identify the key relation nodes and then backtrack to associated objects via the dual-view mapping. For explanatory questions (“Why is the monitor dark?”), TriUnity-GNN typically attends to a small subgraph involving a chain of relations and attributes (e.g., the monitor, the light source, and the state of the light), supporting consistent reasoning decisions.

Although visualizations are omitted here to save space, we note that the qualitative patterns align well with the quantitative findings: nodes that participate in correct reasoning paths receive higher attention, while irrelevant nodes are largely suppressed. This behaviour demonstrates that the tri-modal graph representation indeed leads to interpretable and localized reasoning trajectories.

4.9. Computational Efficiency and Model Complexity

Finally, we compare model sizes and computational costs in Table 5. TriUnity-GNN has a moderate increase in the number of parameters compared with F-GN and ReGAT, owing to the presence of dual encoders and the attribute fusion module. Nevertheless, the theoretical complexity (measured in GFLOPs) and empirical inference time per image remain comparable to other strong baselines. The additional cost is therefore modest relative to the substantial accuracy improvements demonstrated across all benchmarks.

Table 5. Model complexity comparison on VG-GroundTruth. Despite having a slightly larger parameter count due to the dual encoders, TriUnity-GNN maintains competitive inference time.

Models	#Params (M)	GFLOPs	Time / Image (ms)
F-GN [3]	53.1	46.8	41.2
ReGAT [6]	61.4	52.3	47.9
NSM [4]	58.7	49.2	50.6
TriUnity-GNN (ours)	64.2	55.6	49.8

Table 5. Model complexity comparison on VG-GroundTruth. Despite having a slightly larger parameter count due to the dual encoders, TriUnity-GNN maintains competitive inference time.

Models	#Params (M)	GFLOPs	Time / Image (ms)
F-GN [3]	53.1	46.8	41.2
ReGAT [6]	61.4	52.3	47.9
NSM [4]	58.7	49.2	50.6
TriUnity-GNN (ours)	64.2	55.6	49.8

Table 6. Robustness of TriUnity-GNN to question length on different datasets. Accuracy decreases only mildly as questions become longer and more compositional.

Question Length	VG Acc.	Motif-VG Acc.	GQA Acc.
1 ∼ 5 words	78.2	74.3	71.1
6 ∼ 10 words	77.0	73.5	72.3
11 ∼ 15 words	75.6	71.9	71.8
> 15 words	73.8	69.7	69.4

Table 6. Robustness of TriUnity-GNN to question length on different datasets. Accuracy decreases only mildly as questions become longer and more compositional.

Question Length	VG Acc.	Motif-VG Acc.	GQA Acc.
1 ∼ 5 words	78.2	74.3	71.1
6 ∼ 10 words	77.0	73.5	72.3
11 ∼ 15 words	75.6	71.9	71.8
> 15 words	73.8	69.7	69.4

In summary, the experimental evidence shows that TriUnity-GNN delivers consistent gains over previous scene-graph-based and GNN-based approaches on multiple datasets, while maintaining reasonable computational requirements and offering interpretable tri-modal reasoning behaviour.

5. Conclusions

In this work, we introduced TriUnity-GNN, a comprehensive scene-graph–driven reasoning framework that reformulates visual question answering through a unified tri-modal perspective, explicitly integrating objects, relations, and attributes into a single balanced representation space. Departing from conventional VQA systems that implicitly rely on object-biased or relation-biased graph structures, our approach leverages a dual-encoder architecture—one object-significant and one relation-significant—enhanced by a dedicated message-passing mechanism to propagate structural semantics more effectively across heterogeneous graph components. By fusing these dual-view structural embeddings with attribute-aware representations and aligning them with the linguistic cues extracted from the question, TriUnity-GNN yields a full-scale, context-aware scene graph representation capable of addressing complex reasoning demands.

Beyond simply reproducing existing modeling patterns in SG-based VQA, our framework establishes a new form of semantic equilibrium: objects, relations, and attributes contribute proportionally and interactively throughout the reasoning pipeline. This balance is empirically validated across benchmarks such as GQA, VG, and Motif-VG, where TriUnity-GNN consistently surpasses strong baselines under various evaluation protocols. Particularly noteworthy is its robustness in question categories requiring high-level inferential skills—including compositional “what,” fine-grained “where,” entity-centric “who,” and causality-oriented “why” queries. The improved performance in these challenging categories highlights the importance of modeling multi-scale dependencies and demonstrates the advantage gained by explicitly structuring scene graphs into dual semantic modalities.

Furthermore, the ablation analyses confirm the essential contributions of our architectural components. Removing the message-passing mechanism, attribute integration, or relation-significant pathway substantially decreases accuracy, illustrating that each module plays a critical role in constructing a coherent and interpretable reasoning substrate. These findings provide actionable evidence that multi-perspective scene graph decomposition—supported by principled cross-modal fusion—is a promising direction for next-generation VQA frameworks.

Looking ahead, several extensions emerge naturally from this work. One promising direction involves enabling TriUnity-GNN to dynamically modulate its reliance on different SG components based on the question type, difficulty, or ambiguity level. Such an adaptive mechanism could be implemented through question-conditioned gating functions or reinforcement learning policies that learn to allocate attention to structural components most relevant to the query. Another direction involves scaling the framework toward more complex and diverse multimodal reasoning tasks, such as temporal VQA, video scene graph understanding, multi-image reasoning, and interactive dialog-style VQA. The modular design of TriUnity-GNN makes it well-suited for integrating temporal edges, long-range dependencies, or dynamic scene graphs that evolve over time.

In summary, TriUnity-GNN provides a unified and interpretable solution to long-standing challenges in scene-graph–based VQA by explicitly harmonizing object-level, relation-level, and attribute-level semantics. Its strong empirical performance and architectural flexibility offer a new foundation upon which future studies can build more dynamic, adaptive, and generalizable multimodal reasoning systems.