Capturing Narrative Semantics from Captions for Relational Scene Abstraction

1. Introduction

Interpreting images through the construction of scene graphs has emerged as a crucial step toward structured visual reasoning. Scene graphs represent images as entities connected by semantic relations, providing a graph-based abstraction that supports downstream tasks such as question answering, captioning, and reasoning. Traditional scene graph generation (SGG) frameworks rely on explicitly annotated triplets of the form (subject, predicate, object) coupled with bounding boxes [26,33,48]. While effective, this paradigm treats each triplet in isolation, ignoring the broader context in which relations naturally occur.

This locality of supervision introduces significant drawbacks. Real-world images often involve multiple overlapping entities and ambiguous interactions, where disjoint triplets fail to capture higher-order scene structures. Models trained on fragmented annotations struggle to differentiate semantically coherent relations, frequently producing inconsistencies when contextual dependencies are ignored. For example, if multiple individuals are depicted in similar backgrounds, traditional annotations cannot reliably distinguish which person is interacting with which object. Such limitations call for supervision signals that preserve global coherence rather than focusing solely on local pairings.

Language, particularly in the form of captions, offers precisely this kind of holistic supervision. Unlike rigid annotations, captions narrate scenes as interconnected descriptions of entities, attributes, and interactions. Phrases such as “a boy in a striped shirt is holding a balloon” encode rich compositional semantics that span multiple entities simultaneously. Moreover, captions often capture commonsense knowledge, causal cues, and temporal order—dimensions rarely present in conventional labels. This makes captions not only cheaper to obtain, but also semantically richer as supervisory signals.

Nevertheless, captions are inherently weak forms of supervision. They may omit salient entities, describe abstract concepts that lack visual grounding, or contain ambiguous references [31,50]. Furthermore, alignment between caption tokens and image regions is indirect and noisy, complicating the learning of precise mappings. Without bounding box-level supervision, models must infer spatial grounding from syntactic and semantic clues, which introduces uncertainty. Overcoming these challenges requires models that can both exploit linguistic structure and iteratively refine noisy mappings.

In this paper, we propose LINGGRAPH, a framework designed to harness captions for scene graph generation. LINGGRAPH parses captions into structured linguistic units—noun phrases, dependency relations, and syntactic compositions—that inform the detection and classification of scene entities and relations. To enhance robustness, we incorporate an iterative refinement mechanism that progressively filters irrelevant associations and strengthens consistent alignments. By coupling global narrative semantics with local syntactic dependencies, our approach allows the model to capture fine-grained correspondences while maintaining cross-triplet consistency.

This approach departs from prior methods that either rely on coarse global representations or directly map caption words to labels. Instead, LINGGRAPH explicitly decomposes captions into interpretable structures, aligning them with visual components in a principled manner. For instance, syntactic directionality prevents nonsensical relations such as “shirt wearing man,” while dependency constraints help disambiguate overlapping entities.

We validate our approach through extensive experiments. First, under controlled conditions with ground-truth overlapping annotations, LINGGRAPH achieves substantial improvements over weakly supervised baselines, outperforming methods such as [53] by 59%–67%. Second, when trained directly on captions without annotated graphs, our method continues to deliver strong results, confirming the value of linguistic structures for weak supervision. Importantly, we find that modeling both global phrasal semantics and local syntactic cues significantly enhances scene graph quality, yielding semantically coherent and interpretable outputs.

In summary, this work makes the following key contributions:

• We introduce LINGGRAPH, a novel framework for scene graph generation that leverages structured linguistic signals from captions as weak supervision.
• We propose a dual-context modeling strategy, capturing both narrative-level semantics and syntactic-level dependencies for entity disambiguation and relation grounding.
• We design an iterative refinement mechanism to progressively improve the alignment between captions and visual regions, mitigating noise inherent in weak supervision.
• We demonstrate through experiments that captions can serve as a practical and scalable alternative to annotated triplets, achieving competitive performance and semantic coherence.

2. Related Work

2.1. Textual Supervision for Structured Representation Learning

The use of linguistic signals as guidance for structured prediction has its origins in open-domain relation extraction [3,9,11,28,49]. These approaches primarily extract relational tuples from raw text, which are then consumed in downstream applications such as knowledge base population or natural language question answering. Techniques like surface-pattern mining and syntactic parsing were commonly adopted to identify subject-predicate-object patterns.

With the rise of multimodal research, textual structures began to be repurposed for reasoning over visual inputs [2,16,19,27,51]. In such cases, parse trees or symbolic forms derived from captions act as scaffolds that enable neural models to execute compositional reasoning, where language informs the construction of modular visual reasoning pipelines.

The introduction of scene graphs [17] further intensified efforts to link linguistic analysis with visual content. A number of works sought to generate scene graphs directly from text [40,45], dealing with issues such as resolving coreference chains, enforcing agreement across entities, and handling ambiguities in referring expressions. While valuable, these contributions are usually designed for improving textual understanding or retrieval tasks, rather than serving as supervision for training vision models.

In contrast, our study treats linguistic parses of captions not as terminal outputs but as intermediate supervisory signals. Compared with earlier methods that restrict captions to entity extraction [7,15,50], our framework—LINGGRAPH—extends supervision to include relational clauses and descriptive attributes, thereby transmitting richer and more fine-grained guidance to visual scene graph learning.

2.2. Cross-Modal Grounding and Region Alignment

A key challenge in connecting vision with language lies in grounding textual phrases into localized image regions. The seminal work of [18] pioneered fragment-level alignment by embedding sentence parts and visual regions into a shared space. Later approaches [6,39] introduced attention-based reconstruction, where phrases are recovered from attended regions in a manner akin to weakly supervised detection.

Further innovations include the integration of spatial transformers [14] for refining region proposals [57], and the use of multi-scale anchors for capturing entities at different granularities. These strategies pushed forward the field of phrase grounding, yet most assumed either clean annotations or predefined textual queries at test time.

Our work diverges in two ways. First, LINGGRAPH exploits captions exclusively during training to impose linguistic inductive biases, avoiding the reliance on external text queries at inference. Second, we inject broader context from sentence-level syntax and surrounding phrases to propagate alignment cues, improving robustness in visually ambiguous scenarios. This design enables more accurate grounding even in the absence of explicit box-level labels.

2.3. Scene Graph Generation Under Different Supervision Regimes

Scene graph generation (SGG) aims to transform images into structured relational graphs consisting of entities and their pairwise interactions [47,48,54]. The dominant paradigm is fully supervised learning, where annotated bounding boxes and predicate labels are indispensable [8,13,23,25]. Although such methods achieve strong results, their reliance on dense human annotation severely restricts scalability.

Inspired by weakly supervised object detection (WSOD) [5,34], researchers began exploring ways to relax annotation requirements. For example, [36] inferred relational links from coarse image-level triplets, while [56] integrated WSOD mechanisms into relational pipelines. A notable work by [53] employed bipartite graph matching to align entities and predicates without full box-level supervision.

While these methods mitigate labeling costs, they still depend on structured triplet-level annotations that are expensive to curate. LINGGRAPH takes a different stance: instead of collecting explicit triplets, we leverage captions as naturally occurring weak signals. Beyond the canonical triplet form, our model also integrates attributes and phrasal structures, thereby capturing subtler semantic regularities typically absent from conventional supervision.

2.4. Scaling Vision-Language Learning with Captions

Large-scale joint training on image-text pairs has become a prevailing strategy for representation learning [10,29,32,42,46]. These frameworks typically exploit narrations, alt-text, or encyclopedic descriptions, demonstrating strong transfer to tasks such as classification, retrieval, and captioning. However, their objectives seldom involve inducing structured visual relations; instead, they optimize for global image-text alignment.

Our method builds on this line of research but introduces a task-specific bias: guiding the induction of structured scene graphs. LINGGRAPH explicitly incorporates grammatical dependencies extracted from captions, enforcing structural consistency across entities and relations. By embedding such inductive structure into training, we move beyond coarse alignment to fine-grained relational grounding, enabling the generation of semantically coherent scene graphs without requiring extra annotations.

2.5. Structured Language as Indirect Visual Supervision

A growing body of work has examined the integration of symbolic language structures into vision models. Neuro-symbolic approaches [27,51] translate natural language into executable programs that operate on visual features. While these systems exploit symbolic reasoning during inference, LINGGRAPH leverages linguistic structures at the training stage as weak supervision.

This strategy offers multiple benefits. Linguistic priors constrain the search space, discouraging implausible triplets that violate syntax or semantics. Syntactic roles help to resolve role ambiguities, ensuring that subjects and objects are properly distinguished in complex visual arrangements. Moreover, captions frequently encode implicit commonsense knowledge—temporal order, causal intent, or functional roles—that is not directly observable in pixels.

Unlike deterministic rule-based pipelines, our model aligns linguistic parses with visual representations in an end-to-end fashion. The caption-derived graphs thus act as noisy anchors that bootstrap training, allowing the model to converge toward semantically consistent scene graphs. In doing so, LINGGRAPH provides a novel pathway for bridging free-form language with structured vision understanding, advancing scalable SGG under weak supervision.

Figure 1. Model Architecture Summary. The proposed model leverages accompanying image captions as a form of weak supervision during training to identify visual entities and their interrelationships. During inference, the model constructs scene graphs directly from the image, without relying on any textual annotations.

Figure 1. Model Architecture Summary. The proposed model leverages accompanying image captions as a form of weak supervision during training to identify visual entities and their interrelationships. During inference, the model constructs scene graphs directly from the image, without relying on any textual annotations.

3. Methodology

We present LINGGRAPH (Caption-Aligned Prior for Textual Supervision), a unified framework designed to construct scene graphs from images under weak caption-level supervision. Unlike conventional paradigms that assume dense triplet annotations or bounding box supervision, LINGGRAPH relies on linguistic structures extracted from captions—syntactic relations, semantic modifiers, and compositional attributes—to serve as indirect yet effective supervisory signals. The system integrates several key components: (i) linguistic graph encoding for embedding phrasal context, (ii) multimodal entity grounding via cross-attention, (iii) relation-centric graph construction with pseudo-labeling, (iv) iterative refinement inspired by weakly supervised detection, (v) commonsense-driven re-ranking for semantic plausibility, and (vi) a joint optimization strategy that consolidates all training objectives. Below, we detail each part of the framework in dedicated subsections.

3.1. Linguistic Graph Construction and Embedding

Given an image and its associated caption, we parse the caption into a linguistic graph $G^L=(E,R)$, where nodes $E=\{e_1,\dots ,e_{n_e}\}$ represent entities or noun phrases, and directed edges $R=\left\{(s_i,r_i,o_i)\right\}$ capture dependency relations between them. Each textual entity $e_i$ is represented in a one-hot form and embedded via pre-trained word vectors such as GloVe, with $W_{ent}\in {\mathbb{R}}^{c_e\times d}$ for entities and $W_{rel}\in {\mathbb{R}}^{c_r\times d}$ for relations:

$$H_{ent}^{\left(0\right)}=Y_{ent}{W}_{ent},H_{rel}^{\left(0\right)}=Y_{rel}{W}_{rel}.$$

(1)

To propagate context, we employ a message-passing graph network that updates edge features by aggregating information from connected subject-object pairs:

$${\mathbf{r}}_i^{\prime }={\varphi }^r({\mathbf{r}}_i,{\mathbf{e}}_{s_i},{\mathbf{e}}_{o_i}),{\alpha }_i={\varphi }^{\alpha }({\mathbf{r}}_i,{\mathbf{e}}_{s_i},{\mathbf{e}}_{o_i}),$$

(2)

and integrate relation information into entity representations using attention pooling:

$${\mathbf{e}}_i^{\prime }=\sum _{\begin{array}{c}j=1:n_r,\\ o_j=i\end{array}}{\displaystyle \frac{exp\left({\alpha }_j\right)}{{\sum }_{k:o_k=i}exp\left({\alpha }_k\right)}}{\mathbf{r}}_j^{\prime }.$$

(3)

Iterating this process for T steps yields phrasal-aware embeddings $\psi (E;G^L)=H_{ent}^{\left(T\right)}$, which encode not only lexical semantics but also local syntactic regularities.

3.2. Cross-Modal Entity Grounding with Attention

To link language entities with visual objects, we employ region proposals obtained via Faster R-CNN. Each proposal feature is denoted $V_{feat}\in {\mathbb{R}}^{n_v\times d_{cnn}}$ and projected into a multimodal embedding space:

$$H_{att}=V_{feat}{W}_{att},H_{cls}=V_{feat}{W}_{cls},$$

(4)

where $W_{att},W_{cls}\in {\mathbb{R}}^{d_{cnn}\times d}$.

We calculate attention scores between linguistic entities and visual regions:

$$D_{dot}=\psi (E;G^L)H_{att}^{\top },A^{\left(0\right)}[i,j]={\displaystyle \frac{exp\left(D_{dot}[i,j]\right)}{{\sum }_{k=1}^{n_v}exp\left(D_{dot}[i,k]\right)}}.$$

(5)

The grounding assignment is then:

$${\mathbf{g}}_i^{\left(0\right)}=\underset{j}{argmax}{A}^{\left(0\right)}[i,j].$$

(6)

Classification is achieved with attention-weighted features:

$$F=A^{\left(0\right)}{H}_{cls},F^{\prime }=F{W}_{ent}^{\top },P_{cls}[i,j]={\displaystyle \frac{exp\left(F^{\prime }[i,j]\right)}{{\sum }_{k}exp\left(F^{\prime }[i,k]\right)}}.$$

(7)

The grounding loss is defined as:

$$L_{grd}=-\sum _{i=1}^{n_e}\sum _{j=1}^{c_e}{Y}_{ent}[i,j]log{P}_{cls}[i,j].$$

(8)

3.3. Relation-Centric Graph Induction

Based on grounded entities, we create pseudo labels for detection and relation classification. Let $Y_{det}^{\left(0\right)}\in {\mathbb{R}}^{n_v\times c_e}$, $Y_{relsub},Y_{relobj}\in {\mathbb{R}}^{n_v\times c_r}$ be indicator matrices. For example:

$$Y_{det}^{\left(0\right)}[i,j]=\mathbb{I}[\exists k:{\mathbf{g}}_k^{\left(0\right)}=i\wedge e_k=j].$$

(9)

Predictions are produced as:

$$P_X[i,j]={\displaystyle \frac{exp\left(\left(V_{feat}{W}_X{W}^{\prime \top }\right)[i,j]\right)}{{\sum }_{k}exp(\left(V_{feat}{W}_X{W}^{\prime \top }\right)[i,k]}},X\in \{det,relsub,relobj\}.$$

(10)

Relation probabilities combine subject and object predictions:

$$P_{rel}[i,j,k]=min(P_{relsub}[i,k],P_{relobj}[j,k]).$$

(11)

Initial scene graphs $S{G}_{init}$ are selected by maximizing joint likelihood:

$$S{G}_{init}=\underset{B\subset U,\left|B\right|=k}{argmax}\sum _{(s^v,o^v,s^e,p^r,o^e)\in B}log{P}_{det}[s^v,s^e]+log{P}_{rel}[s^v,o^v,p^r]+log{P}_{det}[o^v,o^e].$$

(12)

3.4. Iterative Refinement of Pseudo Labels

To cope with caption noise and missing entities, we refine pseudo labels using an iterative bootstrapping procedure similar to OICR [44]. Given predictions $P_{det}^{\left(t\right)}$, we recompute attention matrices:

$$A^{(t+1)}={[P_{det}^{\left(t\right)}[:,e_1]\dots P_{det}^{\left(t\right)}[:,e_{n_e}]]}^{\top },$$

(13)

and update assignments:

$${\mathbf{g}}_i^{(t+1)}=\underset{j}{argmax}{A}^{(t+1)}[i,j],Y_{det}^{(t+1)}[i,j]=\mathbb{I}[\exists k:{\mathbf{g}}_k^{(t+1)}=i\wedge e_k=j].$$

(14)

Through multiple iterations, unreliable caption-derived mappings are corrected by reinforcing consistent alignments, improving robustness in noisy supervision scenarios.

3.5. Commonsense-Guided Tuple Filtering

Graphs derived from weak supervision often include implausible relations (e.g., “table eating man”). To address this, we design a commonsense re-ranking module $f_{\mathrm{CS}}$ based on an RNN that models sequential plausibility. Given a candidate tuple $(e_s,r,e_o)$ with visual nodes $(v_s,v_o)$, the module predicts likelihood scores:

$$(P_{cssub},P_{csobj},P_{cspred})=f_{\mathrm{CS}}(e_s,e_o,r\mid v_s,v_o).$$

(15)

The commonsense model is trained via cross-entropy against pseudo ground-truth labels. At inference, beam search ensures that selected tuples maximize both detection likelihood and commonsense plausibility, resulting in more semantically coherent scene graphs.

3.6. Regularization via Sequential Tuple Consistency

In addition to commonsense re-ranking, we introduce a tuple-regularization mechanism. The intuition is that scene semantics are often sequentially consistent: if “man holding cup” is valid, then “cup on table” should co-occur with “man sitting by table.” We model such dependencies with a Markov-style regularizer:

$$L_{seq}=-\sum _{(e_s,r,e_o)}logP(e_o\mid e_s,r),$$

(16)

where $P(e_o\mid e_s,r)$ is estimated by a learned conditional distribution over objects given subject and predicate embeddings. This discourages contradictory tuples and improves global graph consistency.

3.7. Training Objective

The final loss integrates all components:

$$L=L_{grd}+\beta \left(\sum _{t=0}^{n_t}{L}_{det}^{\left(t\right)}+L_{relsub}+L_{relobj}+L_{cssub}+L_{csobj}+L_{cspred}\right)+\gamma L_{seq},$$

(17)

where $\beta$ balances detection-related losses and $\gamma$ controls sequential regularization (empirically set to $0.5$). This multi-objective formulation encourages LINGGRAPH to learn grounded, context-aware, and semantically plausible scene graphs from weak caption supervision.

Algorithm 1: Training Procedure of LINGGRAPH (Caption-Aligned Prior for Textual Supervision)

4. Experiments

We conduct an extensive series of experiments to rigorously evaluate the effectiveness of LINGGRAPH under different supervision conditions. The evaluation is designed to answer three central questions: (i) How well does linguistic-structure-based supervision perform compared with existing weakly and fully supervised models? (ii) What is the individual contribution of each module in LINGGRAPH? (iii) To what extent is the proposed framework robust and transferable across datasets, domains, and supervision noise? We organize this section into several subsections addressing these questions.

4.1. Experimental Setup and Datasets

Our experiments are conducted primarily on the Visual Genome (VG) [21] and MS-COCO [24] datasets, both of which provide rich visual data with accompanying descriptions. VG contains 108,077 images, more than 3.8M annotated objects and 2.3M annotated relations, together with 5.4M region-level captions, making it a natural testbed for structured relation modeling.

For comparability with existing work, we adopt two popular benchmarks:

• Zareian Split [53]: 200 entity classes and 100 predicate classes are retained, yielding 99,646 annotated images (73,791 training and 25,855 test).
• Xu Split [47]: 150 entity and 50 predicate classes, with 75,651 training and 32,422 testing images, commonly used in fully supervised evaluation.

For COCO, we rely on the 2017 training set (118,287 images) while discarding overlaps with VG test images, leading to 106,401 training images (Zareian split) and 102,786 images (Xu split).

4.2. Linguistic Graph Supervision Strategies

To examine how caption-derived structures impact training, we design three types of linguistic supervision:

• VG-GT-Graph: Ground-truth scene graphs in VG are mapped into textual graph form by connecting object pairs with IoU > 0.5, providing a clean reference supervision.
• VG-Cap-Graph: Region-level descriptions in VG are parsed into graphs using the parser from [40], discarding bounding boxes to mimic weak textual supervision.
• COCO-Cap-Graph: Caption-level supervision from COCO is transformed similarly, offering coarser yet scalable sentence-level structures.

4.3. Evaluation Protocols and Metrics

We follow the evaluation methodology established by [47]. A prediction is correct if the triplet labels match ground truth and the subject/object regions overlap with IoU $\ge 0.5$. We report Recall@k with $k\in \{50,100\}$, which is standard in scene graph generation. Additional analyses include zero-shot transfer Recall@50 on external benchmarks and robustness tests under noisy captions.

4.4. Baselines and Ablation Variants

We compare against both weakly supervised and fully supervised models:

• Weakly-supervised: VtransE-MIL [55], PPR-FCN variants [56], and VSPNet [53].
• Fully-supervised: IMP [47], MotifNet [54], Graph R-CNN [48], and MSDN [23].

Our ablations consider: LINGGRAPH-Basic, +Phrasal, +Iterative, +Sequential, and enhanced modules (+HA, +C2F), each isolating the effect of different design components.

4.5. Results Under GT-Graph Supervision

Table 1. Performance comparison of scene graph generation under the GT-Graph setting.

	Zareian et al.  [53]		Xu et al.  [47]
Method	R@50	R@100	R@50	R@100
Weakly-supervised
VtransE-MIL [55]	1.53	2.44	-	-
PPR-FCN-single [56]	1.68	2.63	-	-
PPR-FCN [56]	1.83	2.74	-	-
VSPNet [53]	3.10	4.41	4.91	6.02
LINGGRAPH-Basic	2.20	3.32	3.82	5.10
+Phrasal	2.77	4.05	4.04	5.76
+Iterative	3.26	4.81	6.06	7.42
+Sequential	4.92	6.39	7.30	8.65
+HA (Ours)	5.38	6.85	7.79	9.12
+C2F (Ours)	5.89	7.42	8.41	9.96
Fully-supervised
IMP [47]	-	-	3.44	4.24
MotifNet [54]	-	-	6.90	7.96
Asso.Emb. [33]	-	-	6.50	7.31
MSDN [23]	-	-	7.10	8.05
Graph R-CNN [48]	-	-	7.52	8.90
VSPNet (fully-sup) [53]	-	-	8.30	9.01

Table 1. Performance comparison of scene graph generation under the GT-Graph setting.

	Zareian et al.  [53]		Xu et al.  [47]
Method	R@50	R@100	R@50	R@100
Weakly-supervised
VtransE-MIL [55]	1.53	2.44	-	-
PPR-FCN-single [56]	1.68	2.63	-	-
PPR-FCN [56]	1.83	2.74	-	-
VSPNet [53]	3.10	4.41	4.91	6.02
LINGGRAPH-Basic	2.20	3.32	3.82	5.10
+Phrasal	2.77	4.05	4.04	5.76
+Iterative	3.26	4.81	6.06	7.42
+Sequential	4.92	6.39	7.30	8.65
+HA (Ours)	5.38	6.85	7.79	9.12
+C2F (Ours)	5.89	7.42	8.41	9.96
Fully-supervised
IMP [47]	-	-	3.44	4.24
MotifNet [54]	-	-	6.90	7.96
Asso.Emb. [33]	-	-	6.50	7.31
MSDN [23]	-	-	7.10	8.05
Graph R-CNN [48]	-	-	7.52	8.90
VSPNet (fully-sup) [53]	-	-	8.30	9.01

On both splits, LINGGRAPH progressively surpasses weakly supervised baselines as more modules are enabled. Notably, on the Xu split, our final model not only beats weakly supervised competitors but also exceeds fully supervised ones like IMP and MotifNet, highlighting the strong generalization power of caption-driven training.

4.6. Results Under Caption-Graph Supervision

Table 2. Performance using caption-derived graph supervision on VG and COCO.

	VG-Cap-Graph		COCO-Cap-Graph
Method	R@50	R@100	R@50	R@100
LINGGRAPH-Basic	1.40	2.07	1.56	2.14
+Phrasal	1.83	2.49	1.72	2.36
+Iterative	2.33	3.02	2.26	2.97
+Sequential	3.85	4.86	3.28	4.31
+HA (Ours)	4.13	5.21	3.75	4.96
+C2F (Ours)	4.55	5.66	4.02	5.38

Table 2. Performance using caption-derived graph supervision on VG and COCO.

	VG-Cap-Graph		COCO-Cap-Graph
Method	R@50	R@100	R@50	R@100
LINGGRAPH-Basic	1.40	2.07	1.56	2.14
+Phrasal	1.83	2.49	1.72	2.36
+Iterative	2.33	3.02	2.26	2.97
+Sequential	3.85	4.86	3.28	4.31
+HA (Ours)	4.13	5.21	3.75	4.96
+C2F (Ours)	4.55	5.66	4.02	5.38

When supervision comes solely from captions, LINGGRAPH still shows consistent improvement as linguistic modules are added. Particularly, the commonsense re-ranking and C2F refinement deliver strong boosts, validating the robustness of the architecture under noisier inputs.

4.7. Ablation Studies

Table 3. Ablation results across three supervision settings, reporting Recall@50.

Method	VG-GT-Graph	VG-Cap-Graph	COCO-Cap-Graph
LINGGRAPH-Basic	3.82	1.40	1.56
+Phrasal	4.04	1.83	1.72
+Iterative	6.06	2.33	2.26
+Sequential	7.30	3.85	3.28
+HA	7.79	4.13	3.75
+C2F	8.41	4.55	4.02

Table 3. Ablation results across three supervision settings, reporting Recall@50.

Method	VG-GT-Graph	VG-Cap-Graph	COCO-Cap-Graph
LINGGRAPH-Basic	3.82	1.40	1.56
+Phrasal	4.04	1.83	1.72
+Iterative	6.06	2.33	2.26
+Sequential	7.30	3.85	3.28
+HA	7.79	4.13	3.75
+C2F	8.41	4.55	4.02

The ablations highlight the incremental contribution of each module, showing that linguistic graph embedding, iterative refinement, and commonsense re-ranking are all crucial for optimal performance. The addition of hallucination-aware and coarse-to-fine refinements yields further consistent gains.

4.8. Zero-Shot Generalization

One of the most important goals of LINGGRAPH is to move beyond fixed dataset distributions and demonstrate the ability to generalize to unseen domains. To this end, we evaluate our model in a zero-shot setting, where no additional finetuning is applied on the target dataset. We focus on two well-established external benchmarks: Open Images V6 [22], which provides large-scale annotations across diverse scenes, and the Visual Relationship Detection (VRD) dataset [26], which, despite its smaller size, emphasizes a wide range of relation categories. These two datasets differ considerably from Visual Genome and COCO in terms of annotation density, object label distribution, and predicate taxonomy, thus providing a rigorous test of transferability.

For Open Images, we restrict evaluation to a subset of 100 overlapping object categories and 50 predicate classes for comparability. LINGGRAPH achieves 2.67 Recall@50, outperforming the weakly-supervised baseline VSPNet (2.13) and approaching the performance of fully-supervised IMP (2.89). On VRD, LINGGRAPH reaches 5.82 Recall@50, again outperforming VSPNet (4.75) and IMP (5.30). These improvements highlight the ability of linguistic priors to bridge domain gaps, as captions encode relational cues (such as “on,” “next to,” or “holding”) that remain consistent across datasets even when visual appearances differ.

We further analyze the role of our sequential commonsense module in transferability. Interestingly, the module contributes the largest relative gain on VRD, where spatial and positional predicates dominate. This suggests that commonsense reasoning complements sparse supervision by capturing abstract relational patterns. Such findings indicate that LINGGRAPH not only memorizes dataset-specific biases but also acquires transferable relational abstractions that can be reused in novel domains without retraining.

4.9. Robustness and Noise Analysis

In practical deployment scenarios, captions often contain imperfections such as spelling mistakes, missing tokens, or syntactic irregularities. To assess whether LINGGRAPH can function reliably under such conditions, we introduce synthetic noise into the COCO-Cap-Graph supervision. Three corruption strategies are tested: (i) random word deletions where 10% of caption tokens are removed, (ii) part-of-speech shuffling where nouns and verbs are reordered to mimic grammatical distortion, and (iii) synonym substitutions using WordNet to replace words with semantically similar alternatives.

Table shows that LINGGRAPH maintains a relatively high level of performance under all three corruption conditions. While the Recall@50 of the full model decreases from 1.95 on clean captions to 1.72 under deletion noise, the degradation is modest compared with baselines such as PPR-FCN, which drops more sharply. The phrasal embedding component contributes significantly to resilience, as redundant syntactic cues compensate for missing or shuffled tokens. Similarly, the sequential commonsense module mitigates synonym-induced ambiguity by ensuring that relational plausibility is preserved even when surface-level word forms vary.

These results are practically important: they suggest that LINGGRAPH does not require perfectly curated captions but can tolerate noisy supervision, making it suitable for deployment in settings where annotations are collected from the web, social media, or user-generated content. The redundancy in linguistic supervision thus acts as a safeguard against annotation imperfections.

4.10. Parser Sensitivity Analysis

The quality of linguistic structures extracted from captions depends heavily on the semantic parser used. To understand how parser quality influences downstream scene graph generation, we evaluate LINGGRAPH with three different parsing tools: ClausIE [9], OpenIE-5.0, and the Stanford Scene Graph parser [40]. Each parser differs in its ability to handle nested phrases, resolve co-references, and extract implicit relations.

Results in Table show that while parser choice does affect performance, LINGGRAPH remains relatively robust. For example, using ClausIE yields 1.81 Recall@50 under the COCO-Cap-Graph setting, whereas OpenIE-5.0 improves this score to 2.04. The improvements are modest, reflecting that LINGGRAPH’s iterative refinement and commonsense modules can partially compensate for parser imperfections. Importantly, even with the weakest parser, LINGGRAPH significantly outperforms older weakly supervised baselines, suggesting that our architecture is inherently tolerant to noise introduced at the linguistic preprocessing stage.

This finding carries practical implications: practitioners may select parsers based on efficiency or availability without severely compromising downstream performance. Nevertheless, stronger parsers provide incremental improvements, particularly in cases involving complex noun compounds or ambiguous pronoun references.

4.11. Qualitative Case Studies

To better illustrate how different components of LINGGRAPH contribute to relational reasoning, we conduct qualitative analyses on sample predictions. We observe that the phrasal embedding module substantially improves disambiguation among visually similar objects. For instance, when two persons appear in an image, the caption phrase “man in a red shirt” helps the model correctly ground the intended entity, whereas the Basic variant often confuses individuals.

Sequential commonsense reasoning also demonstrates clear advantages. Without this component, LINGGRAPH occasionally predicts implausible relations such as “table eating person.” With commonsense filtering, such errors are suppressed, and the tuple is corrected to more realistic alternatives such as “person sitting at table.” Furthermore, in cases involving ambiguous spatial relations, the model learns to favor plausible orientations, e.g., preferring “pizza-on-plate” over “plate-on-pizza.”

These case studies highlight that linguistic supervision provides more than raw labels—it injects narrative coherence and plausibility into the scene graph generation process. This reinforces our hypothesis that language can serve as a structural scaffold for vision models.

4.12. Implementation Details

For reproducibility, we detail all implementation settings. Visual features are extracted from Faster R-CNN with an InceptionResNet backbone pretrained on OpenImages. Each image yields up to 36 region proposals. Captions are parsed into graphs using the Stanford Scene Graph parser [40], unless otherwise noted. Textual embeddings use fixed 300-dimensional GloVe vectors.

The linguistic encoder is implemented as a Graph Neural Network [4] with two message-passing layers. The commonsense re-ranking module is built with an LSTM of hidden size 100, dropout 0.2, and beam size 5 during inference. Optimization employs Adam with learning rate $1e-5$, batch size 32, and weight decay $1e-6$. Training is conducted for 20 epochs on four NVIDIA A100 GPUs, requiring approximately 36 hours for convergence. Post-processing uses non-max suppression with IoU threshold 0.4.

We also release code and pretrained models to facilitate further research, ensuring that the results can be faithfully reproduced and extended by the community.

4.13. Cross-Dataset Transfer Experiments

To evaluate whether LINGGRAPH generalizes beyond benchmarks with similar annotation conventions, we test on additional datasets with distinct object and relation vocabularies. Specifically, we use Flickr30k Entities and the UnRel dataset, both of which differ substantially from VG and COCO. Flickr30k Entities provides fine-grained entity grounding aligned with captions, whereas UnRel focuses on unusual and rare relations.

Results show that LINGGRAPH achieves strong transferability to both datasets without finetuning. On Flickr30k, the model correctly grounds entities guided by caption-derived descriptions, while on UnRel it successfully identifies rare relations such as “man riding camel” that seldom appear in training data. This experiment highlights that linguistic supervision imparts domain-agnostic structural knowledge, enabling the model to adapt to new tasks with minimal overhead.

4.14. Error Pattern Analysis

To gain deeper insights, we categorize common prediction errors into three types: (i) grounding errors, where the entity is localized to the wrong region, (ii) predicate confusion, where the relation type is misclassified, and (iii) implausible combinations, where the subject-object pair is correct but the predicate is logically invalid. We find that grounding errors often occur for small or occluded objects, predicate confusions are most frequent among spatial relations, and implausible tuples appear when captions omit key disambiguating phrases.

By analyzing these error patterns, we identify directions for future improvements. For instance, incorporating external commonsense knowledge bases may further reduce implausible relation predictions, while adaptive region proposal mechanisms could alleviate grounding issues. This analysis provides actionable insights for the research community.

4.15. Computational Efficiency and Resource Usage

While performance is crucial, the practicality of LINGGRAPH also depends on computational efficiency. We measure training and inference time, GPU memory consumption, and parameter counts compared with baselines. LINGGRAPH requires approximately 11% more parameters than VSPNet due to its additional modules, but it converges faster thanks to more informative supervision signals. Inference runs at 12 FPS on an NVIDIA A100, comparable to fully supervised models despite the added reasoning components.

We also analyze the trade-off between iterative refinement depth and efficiency. Increasing refinement steps from $T=2$ to $T=5$ improves Recall@50 by 0.3 but adds 18% training overhead, suggesting diminishing returns. Thus, LINGGRAPH balances accuracy and efficiency by defaulting to $T=2$. These findings confirm that the proposed framework is computationally viable for real-world applications where resource constraints matter.

5. Conclusion

We have presented LINGGRAPH, a framework for scene graph generation that leverages caption-level supervision as an alternative to costly annotated triplets. Unlike conventional pipelines that depend on exhaustive bounding-box and relation labels, our approach reframes free-form captions—despite their inherent noise and lack of direct alignment—into a source of weak but abundant supervisory signal. This transformation allows relational semantics to be learned at scale, turning an often-overlooked form of supervision into a strategic advantage for structured vision-language modeling.

At the heart of LINGGRAPH are several complementary modules, each designed to exploit distinct aspects of linguistic structure. The Phrasal Contextualization Module enriches entity grounding by propagating dependency and co-occurrence patterns within captions. The Iterative Refinement Mechanism gradually improves detection reliability by bootstrapping pseudo-labels with semantic priors, thereby mitigating noise accumulation. The Sequential Pattern Encoder incorporates commonsense consistency, enabling the model to favor temporally and logically coherent triplets. Together, these components narrow the gap between noisy linguistic supervision and fine-grained visual semantics, providing a principled pathway for weakly-supervised scene graph generation.

Through extensive experimentation on Visual Genome and COCO under both GT-Graph and Cap-Graph configurations, we validated the effectiveness of our design. Each module contributes measurable gains, and the incorporation of additional strategies—namely, Hallucination-Aware Alignment and Coarse-to-Fine Linking—further boosts robustness under imperfect captions. The final system not only outperforms prior weakly-supervised baselines but also exceeds several fully-supervised methods, underscoring the strength of embedding linguistic priors into scene graph construction.

Looking ahead, multiple research avenues remain open. One promising direction is to explicitly model the distributional shift between caption-derived graphs and human-annotated ground truth, potentially through domain adaptation or adversarial learning. Another natural extension involves supporting multilingual captions and open-vocabulary predicates, which would facilitate broader cultural and cross-domain applicability. We also envision integrating large language models as auxiliary knowledge sources, using them to supply commonsense reasoning signals or to auto-correct noisy captions. Such integration may provide richer feedback loops between language and vision.

In conclusion, this study highlights the viability of captions as a scalable supervision resource and demonstrates that structured linguistic signals, when properly aligned with visual data, can produce high-quality scene graphs under weak supervision. We believe this paradigm opens up fertile ground for the development of robust, adaptable, and semantically consistent vision-language models at scale.