Emergent Semantics from Disjoint Modalities: Unsupervised Cross-Domain Vision-Language Grounding

1. Introduction

In recent years, the unification of visual and linguistic modalities has profoundly reshaped the research landscape of artificial intelligence. The capacity to integrate heterogeneous streams of information has enabled transformative advances in tasks such as visual question answering, multimodal retrieval, referring expression resolution, and grounded caption generation. Central to this transformation is the advent of large-scale pre-trained vision-and-language (V&L) models, including ViLBERT [35], LXMERT, VisualBERT [31], VLBERT, and UNITER [7], which rely on transformer-based architectures to produce joint embeddings. These joint embeddings allow visual features and textual tokens to be projected into a unified semantic space, thereby supporting cross-modal reasoning and generation at an unprecedented level of accuracy. As a result, these models dominate benchmarks and have become the de facto standard in modern multimodal research.

Yet, despite the compelling progress, a critical weakness continues to constrain the scalability and generalizability of current V&L frameworks. Unlike their unimodal counterparts such as BERT [42], which thrive on vast amounts of raw and uncurated textual data, multimodal models remain highly dependent on large-scale parallel corpora consisting of aligned image-caption pairs. Constructing these corpora is costly, requiring extensive human annotation or laborious filtering. For example, MS-COCO [6], a widely used benchmark, demands manual captioning for hundreds of thousands of images. Similarly, Conceptual Captions [3], although sourced from the web, involved intensive filtering procedures, retaining only a few million pairs from billions of crawled candidates. Such dependency on curated pairs fundamentally limits the scalability of multimodal models to domains where aligned resources are scarce, such as medical imaging or low-resource languages, and thereby undermines the vision of truly universal multimodal intelligence.

This observation raises a fundamental question: must effective multimodal representations always be grounded in parallel supervision, or can cross-modal structure emerge even in the absence of alignment? In this work, we argue for the latter. Specifically, we address the task of unsupervised pre-training for vision-and-language models, where the learning signal is derived exclusively from disjoint corpora of images and texts. Our formulation represents a paradigm shift: rather than relying on costly annotations to align modalities, we investigate whether latent semantic structures in each modality can be independently acquired and subsequently reconciled through implicit anchors. This perspective resonates with broader trends in machine learning that aim to reduce the dependence on direct supervision and instead harness the abundance of unannotated data naturally available at scale.

The intellectual roots of this idea lie in several adjacent research domains. A prominent parallel can be drawn with the development of unsupervised machine translation [28], which demonstrated that bilingual alignment can emerge without parallel corpora, provided that monolingual data is sufficiently exploited. Similarly, work on unsupervised caption generation [12] shows that linguistic regularities can bootstrap image-to-text mappings even without aligned supervision. These findings collectively suggest that semantic alignment is not solely a product of annotated data, but can also emerge as an emergent property of distributional regularities across domains. Extending this intuition to multimodality, we hypothesize that the structures governing visual and textual data exhibit a degree of latent compatibility, which can be uncovered through carefully designed unsupervised objectives.

Further inspiration is drawn from multilingual representation learning [41], where models like mBERT [9] are trained on heterogeneous language corpora without relying on direct translations. Remarkably, these models achieve semantically coherent embeddings across languages purely through masked token prediction. Subsequent studies [8] reveal that these embeddings, despite lacking explicit cross-lingual supervision, are linearly mappable into a shared semantic space, exposing universal regularities in human language. By analogy, if we treat images as compositions of discrete visual tokens [11], multimodal learning can be reinterpreted through the lens of multilingual learning: modalities are analogous to languages, and cross-modal alignment can arise from shared semantic pivots. This analogy guides our central hypothesis that vision and language, although semantically distinct, may converge in representation space if provided with a sufficient set of semantic anchors.

Building upon this hypothesis, we propose VISTRA (VISion-Text Representation Alignment), an unsupervised framework that discards the need for parallel image-caption pairs altogether. At the core of VISTRA is a dual-modality masked prediction scheme, which extends the architecture of VisualBERT [31] into a setting where modalities are processed independently. During training, the model encounters either textual documents or visual inputs, never paired simultaneously. For texts, randomly masked tokens are reconstructed; for images, masked regions are predicted in terms of semantic content or positional structure. The critical innovation lies in the incorporation of semantic anchors extracted from object detectors [32]. These anchors serve as pseudo-shared vocabulary items that recur across modalities—for instance, an object tag such as “cake” may appear both as a detected region label in an image and as a noun in text. Through such anchors, VISTRA implicitly aligns visual and linguistic signals in a shared embedding space without requiring explicit supervision.

The practical implications of this approach are manifold. Consider real-world applications where parallel multimodal datasets are prohibitively expensive or unavailable. In social media analysis, for instance, detecting hateful memes [22] often involves mismatched modalities, where text and image are loosely related or not captioned at all. In medical contexts, image-text pairs such as radiology images and diagnostic reports [33] are scarce and domain-specific, making supervised pre-training infeasible. In such cases, unsupervised frameworks like VISTRA offer a scalable alternative, leveraging abundant unpaired data while still yielding transferable multimodal representations.

To validate this paradigm, we conduct extensive experiments across multiple benchmarks, including Visual Question Answering (VQA) [15], Natural Language for Visual Reasoning (NLVR2), Flickr30K image retrieval, and RefCOCO+. To mimic the absence of parallel corpora, we deliberately discard alignment information from standard datasets, simulating a genuinely unsupervised environment. Remarkably, VISTRA achieves competitive performance relative to supervised baselines, even rivaling models trained with full access to paired data. Furthermore, when pre-trained on entirely disjoint sources—such as large-scale web images and BookCorpus—VISTRA continues to deliver robust performance, underscoring the feasibility of learning from unpaired multimodal resources.

Finally, we complement these findings with detailed ablation studies to dissect the contribution of semantic anchors. Quantitative results reveal that object tags substantially enhance cross-modal alignment, while qualitative case analyses illustrate how anchors foster emergent grounding between vision and language. We also explore a semi-supervised extension, demonstrating that integrating a small fraction of aligned data with abundant unpaired corpora not only preserves performance but also introduces regularization benefits and semantic diversity. Collectively, these results challenge the entrenched view that parallel datasets are indispensable for V&L pre-training, and instead highlight unsupervised learning as a promising frontier for building scalable, general-purpose multimodal systems.

2. Related Work

• Pre-trained V&L Transformers

One of the most dominant research directions in multimodal learning centers on large-scale pre-trained vision-and-language (V&L) transformers, most of which adopt the “mask-and-predict” strategy. Within this paradigm, models are optimized to recover hidden visual regions or textual tokens based on their surrounding context, thereby encouraging the fusion of heterogeneous information sources [7,13,19,30,31,35]. A key characteristic of these frameworks is their reliance on parallel image-text datasets, which serve as the foundation for learning fine-grained associations. Architecturally, two major paradigms have emerged. In the dual-stream family, visual features and textual embeddings are encoded separately through modality-specific transformers before being merged by a dedicated fusion layer. ViLBERT [35], LXMERT, and ERNIE-ViL are well-known representatives of this line. By contrast, the single-stream design concatenates both modalities from the start and feeds them directly into a unified transformer, as exemplified by VisualBERT [31], VL-BERT, and UNITER [7]. Both paradigms have established strong baselines across benchmarks such as VQA, retrieval, and multimodal reasoning [22], and probing analyses [2] confirm their ability to capture subtle inter-modal dependencies and contextual relations after pre-training.

A number of extensions have further enriched these transformer backbones by introducing auxiliary semantic signals. A notable example is Oscar [32], which injects object tags as additional tokens into the input, thereby enhancing visual-linguistic grounding under the assumption of well-annotated image-caption data. Our approach draws inspiration from this idea but advances it to an unsupervised regime, where object tags function not as aligned supervision but as indirect semantic anchors linking disjoint corpora. In this way, tags acquire heightened importance because they supply the only cross-modal bridge when paired supervision is unavailable. VIVO [18] also leverages curated image-tag pairs for novel object captioning, combining them later with supervised datasets. By contrast, our method eliminates manual annotation entirely; tags are extracted automatically by frozen detection models, making the pipeline more scalable and less prone to domain-specific overfitting.

• Self-supervised Representation Learning

Self-supervised learning (SSL) has emerged as a unifying principle across domains, where useful representations are distilled from unlabeled data through proxy prediction tasks. In natural language processing, pioneering models such as ELMo [40], BERT [42], and RoBERTa [34] demonstrated that masked language modeling provides robust contextual encoders transferable to a variety of downstream applications. In the visual domain, early SSL approaches were largely based on low-level pretext tasks, such as predicting image color channels, ordering shuffled patches, or reconstructing spatial contexts [10,37,39]. More recently, contrastive learning frameworks like SimCLR [5] and MoCo have shown that maximizing agreement between augmented views yields state-of-the-art visual encoders.

In line with this trajectory, our framework formulates multimodal pre-training as a self-supervised problem over unimodal datasets. Specifically, masked token modeling is independently applied to large corpora of language and image data, without requiring any explicit alignment. Unlike traditional SSL in vision, which often emphasizes pixel-level patterns, we base our visual encoder on object-centric features derived from a pre-trained detector, thus shifting the focus toward higher-level semantics. Conceptually, our design represents a hybrid SSL scheme: the masked modeling idea is shared across both modalities, while semantic anchors guide the implicit cross-modal alignment. This differs from most previous V&L SSL approaches, which generally assume some form of weak alignment such as noisy captions or paired meta-information.

• Unsupervised Multilingual Language Models

Another key source of inspiration for our framework is the progress in unsupervised multilingual language modeling. Models such as mBERT [9] have convincingly demonstrated that cross-lingual representations can be learned in the absence of parallel translation pairs. These models, trained over diverse monolingual corpora with a shared encoder and vocabulary, display strong zero-shot and few-shot transfer capabilities across languages. Empirical studies [8] further reveal that shared tokens—numbers, named entities, and universal symbols—play an outsized role in facilitating transfer, while architectural features like shared positional embeddings also enhance cross-lingual alignment.

Our proposed method extends these principles to the multimodal domain. We adopt a unified transformer that processes both visual and linguistic sequences, and introduce semantic anchors, derived from object detection, as cross-modal equivalents of overlapping vocabulary. These anchors act as bridge tokens, making it possible to align otherwise unpaired modalities. The analogy between multilingual transfer and multimodal alignment has been highlighted in recent efforts to establish universal representational spaces across domains [4]. Building upon this insight, our framework introduces a concrete, unsupervised bridging mechanism that plays the same role as shared lexical tokens in multilingual corpora.

• Unsupervised Grounding and Captioning

A related area of investigation concerns unsupervised grounding and caption generation. The overarching goal in these works is to discover correspondences between visual regions and linguistic expressions without paired annotation. Earlier methods relied on co-occurrence statistics or weak signals to link textual phrases with visual entities. More recent works in unsupervised captioning [12,16,27] adopt generative objectives, combining separately collected image and text corpora through adversarial learning, reinforcement-based optimization, or back-translation strategies. These models attempt to generate plausible captions for images even in the absence of aligned training pairs.

While conceptually relevant, our approach departs in two important ways. First, rather than producing captions, our objective is to construct aligned multimodal representations, which can subsequently serve multiple downstream applications beyond captioning. Second, unlike captioning pipelines that still depend on sentence-level descriptive corpora, we broaden the textual modality to include unconstrained, domain-general sources such as books and articles. This flexibility demonstrates that representation alignment can emerge even when the textual data is not curated to describe images, thereby increasing the applicability of our framework in real-world large-scale repositories.

• Cross-modal Alignment without Pairing

Recent research has started to question the necessity of strong alignment for multimodal training. CLIP-style frameworks, for instance, rely on large-scale web-harvested image-text pairs where alignment is weak or noisy. These methods confirm that loose supervision, when combined with scale, is sufficient to learn transferable representations. However, they still presuppose some form of co-occurrence between images and texts. Our approach is more radical: we remove the assumption of co-occurrence entirely, and instead rely on a small set of semantically meaningful anchors to achieve alignment. This property makes our framework particularly relevant for specialized domains, including medical imaging or security-critical environments, where images are sensitive and textual annotations are infeasible. In such contexts, pairing is either impossible or prohibitively expensive, underscoring the unique suitability of our method.

• Multimodal Learning under Resource Constraints

An additional dimension of our contribution lies in addressing multimodal learning for under-resourced contexts. Collecting annotated multimodal corpora is a privilege available only in a handful of high-resource languages and domains. For the majority of communities, especially those involving low-resource languages or culturally specific imagery, parallel datasets are absent. This inequity motivates research into models that can generalize from heterogeneous, unpaired corpora. By demonstrating that our framework performs competitively under such conditions, we provide evidence that resource-efficient multimodal pre-training is achievable, offering a more equitable path for future AI systems that must operate globally across diverse cultural and linguistic settings.

• Vision-Language Models with Weak Supervision

Finally, several approaches attempt to relax the dependence on strong supervision by incorporating semi-supervised or weakly supervised signals. These include methods that bootstrap alignment with pseudo-labeled captions, utilize noisy alt-text from web or social media as supervisory signals, or apply contrastive consistency across modalities. Our model differs in that it bypasses alignment altogether and instead introduces structured semantic cues in the form of detector-generated tags. This shift in perspective opens up new possibilities for combining pre-existing visual priors (e.g., object graphs, region-based detectors) with transformer-based multimodal pre-training, thereby offering a scalable alternative that retains efficiency while eliminating annotation overhead.

Figure 1. An overview of pre-training using unpaired data.

Figure 1. An overview of pre-training using unpaired data.

3. Proposed Framework

In this section, we begin by revisiting the conventional recipe of supervised vision-and-language (V&L) pre-training, using a modified VisualBERT as an illustrative baseline. This background serves to highlight the dependency of existing approaches on paired data. We then introduce our unsupervised framework, VISTRA, which is designed to overcome this dependency by exploiting two non-overlapping corpora of texts and images. The framework introduces several innovations: (1) a dual-stream masking and reconstruction paradigm, (2) the use of detector-based tags as semantic anchors, (3) a unified representation space that integrates both modalities with shared embeddings, and (4) an additional alignment regularizer that enforces semantic consistency across heterogeneous signals.

3.1. Supervised Pre-Training with Aligned Pairs

The majority of early V&L pre-training schemes are grounded in the assumption that training data consist of paired image-caption examples. To make this concrete, we adopt VisualBERT as a reference point and construct a supervised baseline, denoted as Supervised-VB, which is also evaluated in our experiments. Specifically, Supervised-VB extends VisualBERT by combining three elements: (i) feature regression and classification losses over detected regions, inspired by LXMERT; (ii) reconstruction of detector-derived object tags following Oscar [32]; and (iii) an image-text matching signal. The model accepts an aligned input pair $(T,I)$, where T denotes a tokenized sentence and I represents the image, decomposed into m region vectors and l detector tags. Textual tokens are embedded through word, segment, and positional encodings, while visual inputs are represented by region-level features with associated spatial encodings.

Formally, an aligned pair $(T,I)$ is encoded as:

$$\mathbf{T}=[{\mathbf{w}}_1,{\mathbf{w}}_2,\dots ,{\mathbf{w}}_n],\mathbf{I}=[{\mathbf{r}}_1,\dots ,{\mathbf{r}}_m;{\mathbf{d}}_1,\dots ,{\mathbf{d}}_l],$$

(1)

where ${\mathbf{w}}_i$ is the embedding of the i-th subword, ${\mathbf{r}}_j$ is the feature vector of region j with positional encoding, and ${\mathbf{d}}_k$ corresponds to the embedding of the k-th detector tag. The concatenated sequence $[\mathbf{T};\mathbf{I}]$ is passed to a transformer encoder $f_{\theta }$, which produces contextualized multimodal representations.

Training objectives in this supervised setting combine four terms:

$${\mathcal{L}}_{\mathrm{supervised}}={\mathcal{L}}_{\mathrm{MLM}}+{\mathcal{L}}_{\mathrm{V}-\mathrm{Pred}}+{\mathcal{L}}_{\mathrm{Tag}-\mathrm{Recon}}+{\mathcal{L}}_{\mathrm{Match}}.$$

(2)

Here, ${\mathcal{L}}_{\mathrm{MLM}}$ masks and predicts language tokens, ${\mathcal{L}}_{\mathrm{V}-\mathrm{Pred}}$ covers region regression and classification, ${\mathcal{L}}_{\mathrm{Tag}-\mathrm{Recon}}$ reconstructs masked tags, and ${\mathcal{L}}_{\mathrm{Match}}$ ensures correct image-text alignment.

The visual prediction component itself is decomposed as:

$${\mathcal{L}}_{\mathrm{V}-\mathrm{Pred}}={\mathcal{L}}_{\mathrm{reg}}+{\mathcal{L}}_{\mathrm{cls}},$$

(3)

where ${\mathcal{L}}_{\mathrm{reg}}={\displaystyle \frac{1}{m}}{\sum }_{j=1}^m{\parallel {\mathbf{r}}_j-{\widehat{\mathbf{r}}}_j\parallel }_2^2$ enforces regression to the detector features, and ${\mathcal{L}}_{\mathrm{cls}}$ is a categorical prediction loss over region labels. In aggregate, Supervised-VB exemplifies the strengths but also the limitations of supervised multimodal pre-training: it achieves strong downstream results but relies critically on costly paired data.

Algorithm 1:Unsupervised Pre-training Procedure of VISTRA

3.2. Unsupervised Pre-Training on Disjoint Modalities

To address the bottleneck of annotation, we propose VISTRA, which completely discards the assumption of aligned supervision. The framework is premised on the hypothesis that latent alignment can emerge from independent corpora if the model is properly regularized with semantic anchors and shared representations. VISTRA comprises four interlocking modules: (1) dual-stream masked modeling for intra-modality learning, (2) detector-based tag anchoring, (3) unified multimodal embeddings with shared vocabulary, and (4) contrastive alignment for fine-grained consistency.

3.2.1. Dual-Stream Masked Modeling

Unlike supervised approaches, VISTRA learns from two separate data sources: a textual corpus $D_T$ (e.g., Wikipedia, BookCorpus) and an image corpus $D_I$ (e.g., OpenImages). At each step, the model samples from one corpus, masks part of the input, and predicts the masked content using the same transformer encoder $f_{\theta }$. For a text sequence T, a subset of tokens is masked to form $\tilde{T}$, and for an image I, regions or tags are masked to form $\tilde{I}$.

Formally:

$$\tilde{T}=\mathrm{Mask}\left(T\right),\tilde{I}=\mathrm{Mask}\left(I\right),$$

(4)

where masked tokens or visual elements are replaced by a [MASK] symbol. The shared transformer $f_{\theta }$ is then trained to reconstruct the missing components. The overall unsupervised objective is:

$${\mathcal{L}}_{\mathrm{unsup}}={\mathbb{E}}_{T\sim D_T}{\mathcal{L}}_{\mathrm{MLM}}(f_{\theta }\left(\tilde{T}\right),T)+{\mathbb{E}}_{I\sim D_I}\left[{\mathcal{L}}_{\mathrm{V}-\mathrm{Pred}}(f_{\theta }\left(\tilde{I}\right),I)+{\mathcal{L}}_{\mathrm{Tag}-\mathrm{Recon}}\right].$$

(5)

This design enables the transformer to share parameters across both text-only and image-only inputs, ensuring that representational patterns learned in one modality inform the other. Such parameter tying implicitly encourages structural regularities to be discovered and reused across domains.

3.2.2. Semantic Anchoring via Detector Tags

To provide a bridge between modalities, VISTRA incorporates semantic anchors derived from object detection models. Let $\mathcal{D}$ denote a frozen detector that maps image I to a set of bounding box–tag pairs:

$$\mathcal{D}\left(I\right)={\left\{(b_j,t_j)\right\}}_{j=1}^m,$$

(6)

where $b_j$ is the bounding box and $t_j$ is the textual label. Each tag token is embedded as:

$${\mathbf{d}}_j^{\left(i\right)}=\mathbf{e}\left(t_j^{\left(i\right)}\right)+\mathbf{p}\left(b_j\right),$$

(7)

where $\mathbf{e}(·)$ is the subword embedding and $\mathbf{p}(·)$ encodes spatial position. A subset of tag tokens is masked during pre-training, and the model learns to reconstruct them:

$${\mathcal{L}}_{\mathrm{Tag}-\mathrm{Recon}}=-\sum _{k\in \mathcal{M}}logP(t_k\mid \tilde{I};\theta ).$$

(8)

This mechanism equates detector tags with natural language tokens by placing them into a shared embedding layer and prediction head. The consequence is that words and tags become interchangeable anchors, allowing implicit alignment even without co-occurring training examples. This setup mirrors multilingual pre-training, where overlapping lexical tokens such as numbers or names enforce cross-lingual consistency.

3.2.3. Unified Embedding and Multimodal Projection

To further integrate modalities, VISTRA builds a shared subword vocabulary via byte pair encoding (BPE). Both textual tokens and detector tags are represented using the same token embedding matrix, augmented with position and segment embeddings. For an input sequence S of modality m, the embedding is:

$${\mathbf{X}}_S={\mathbf{E}}_{\mathrm{token}}\left(S\right)+{\mathbf{E}}_{\mathrm{position}}\left(S\right)+{\mathbf{E}}_{\mathrm{segment}}\left(m\right).$$

(9)

Beyond embedding, VISTRA employs a projection head $g_{\varphi }$ and contrastive alignment to refine semantic consistency. For a region feature ${\mathbf{r}}_j$ and its tag ${\mathbf{d}}_j$, we enforce:

$${\mathcal{L}}_{\mathrm{align}}=-log{\displaystyle \frac{exp(\mathrm{sim}(g_{\varphi }\left({\mathbf{r}}_j\right),g_{\varphi }\left({\mathbf{d}}_j\right))/\tau )}{{\sum }_{k}exp(\mathrm{sim}(g_{\varphi }\left({\mathbf{r}}_j\right),g_{\varphi }\left({\mathbf{d}}_k\right))/\tau )}},$$

(10)

where $\mathrm{sim}(·,·)$ is cosine similarity and $\tau$ is a temperature parameter. This objective pulls semantically matched pairs closer while pushing apart unrelated instances, thereby strengthening modality invariance.

3.2.4. Final Training Objective

The complete pre-training loss of VISTRA integrates reconstruction and alignment components:

$${\mathcal{L}}_{\mathrm{VISTRA}}={\mathcal{L}}_{\mathrm{unsup}}+\lambda ·{\mathcal{L}}_{\mathrm{align}},$$

(11)

with $\lambda$ weighting the alignment term. After pre-training, VISTRA can be adapted to downstream V&L tasks such as VQA or retrieval by simply providing paired inputs, leveraging its already aligned embedding space without architectural changes.

4. Experiments

To rigorously assess the effectiveness of our unsupervised framework and to establish its practical utility, we conduct a wide range of experiments across several representative settings. The experimental design serves multiple purposes: (i) to compare the gap between models trained with paired supervision and our unsupervised alternative, (ii) to examine the adaptability of the proposed method in scenarios where aligned corpora are missing, and (iii) to analyze the contributions of individual components such as detector tags, corpus selection, and hybrid training strategies. Through these systematic studies, we aim to demonstrate not only the competitiveness of VISTRA, but also its robustness and versatility under different data availability conditions.

4.1. Evaluation with Simulated Alignment Removal

We first adopt a controlled simulation protocol, which has been a common practice in unsupervised learning literature, by intentionally discarding alignment information from datasets that originally contain paired annotations. Specifically, we repurpose Conceptual Captions (CC), which consists of millions of loosely associated image-text pairs, into a pseudo-unaligned setup. This allows us to conduct a fair comparison between supervised and unsupervised models trained on exactly the same raw corpus, isolating the effect of paired supervision. The central goal here is to investigate whether VISTRA can recover strong cross-modal representations purely from unpaired instances.

Overall Benchmark Results.

Table 1 summarizes performance across four widely used benchmarks: VQA 2.0 for question answering, NLVR2 for compositional reasoning, Flickr30K for retrieval, and RefCOCO+ for referring expression grounding. Across the board, our unsupervised model surpasses the Base model (VisualBERT initialized only from BERT, without multimodal pre-training), confirming the effectiveness of unsupervised representation learning. While the supervised VisualBERT baseline has marginally higher scores, the differences are surprisingly small, given that VISTRA never observes any explicit alignment during training. This result supports the central claim that cross-modal grounding can emerge from disjoint corpora with the right architectural and objective choices.

Case Study: Flickr30K Retrieval.

A notable highlight is observed on Flickr30K. Here, VISTRA achieves Recall@1 of 55.37, dramatically higher than the baseline (42.86) and approaching the supervised score of 61.19. Considering that no alignment signal is available in our setting, this performance suggests that semantic anchors and contextual prediction objectives are sufficient to construct image-text mappings of competitive quality. The result demonstrates the surprising resilience of unsupervised objectives in capturing semantic structures that transfer well to retrieval.

Efficiency Considerations.

An additional advantage of our framework is its resource efficiency. All experiments are trained from BERT_base initialization using 4 NVIDIA V100 GPUs (16GB memory) for 10 epochs. Full unsupervised training completes in roughly 72 hours. This low computational overhead indicates that VISTRA is suitable for groups without access to massive compute clusters, offering a practical route to high-quality multimodal encoders without costly annotation.

4.2. Experiments with Realistic Unaligned Sources

We next move beyond controlled simulations to evaluate more realistic situations where corpora are entirely disjoint and text resources are unrelated to image descriptions. This setup is intended to approximate the conditions encountered in real-world applications, where photographs and large text corpora coexist but are rarely aligned.

Training on Disjoint Text and Image Corpora.

We construct VISTRA-disjoint by using CC for images, SBU Captions [38] for textual descriptions, and BookCorpus for additional general-domain text. Results indicate that this configuration yields the best performance on three out of four benchmarks. This demonstrates that heterogeneous corpora, though unaligned, provide complementary signals that improve the diversity of learned features and support stronger generalization than homogeneous caption-only sources.

Training on Purely General-Domain Text.

We further challenge our model with VISTRA-nocaption, which replaces all caption-style data with BookCorpus only. Even under this extreme setting, the model sustains strong accuracy (e.g., 71.47 on NLVR2 Dev and 64.25 on RefCOCO+ TestB). This outcome suggests that generic corpora, despite being unrelated to images, contain sufficient overlap in common nouns and object words (e.g., “car,” “man,” “tree”) to align effectively with detector tags. Such findings highlight the applicability of VISTRA in contexts where no caption annotations are available at all.

Broader Implications.

Together, these findings establish that VISTRA can thrive in natural data conditions where multimodal resources are heterogeneous and uncurated. This widens the scope of multimodal pre-training to specialized domains such as medicine or law, where paired annotations are difficult or impossible to obtain, but large collections of images and documents exist separately.

Table 2. VISTRA maintains strong performance when trained on disjoint or non-caption corpora. Disjoint caption sources (VISTRA-disjoint) and general-domain text (VISTRA-nocaption) still yield robust results across tasks.

Model	Text Type		VQA	NLVR		Flickr30K			RefCOCO+
	Caption	General	Test-Dev	Dev	Test-P	R@1	R@5	R@10	Dev	TestA	TestB
Base	-	-	69.26	68.40	68.65	42.86	73.62	83.28	70.66	77.06	61.43
VISTRA	CC	BC	70.74	71.74	71.02	55.37	82.93	89.84	72.42	79.11	64.19
VISTRA-disjoint	SBU	BC	70.70	71.97	72.11	56.12	82.82	90.12	73.05	79.48	64.19
VISTRA-nocaption	-	BC	70.47	71.47	71.19	54.36	82.22	89.24	72.96	79.30	64.25

Table 2. VISTRA maintains strong performance when trained on disjoint or non-caption corpora. Disjoint caption sources (VISTRA-disjoint) and general-domain text (VISTRA-nocaption) still yield robust results across tasks.

Model	Text Type		VQA	NLVR		Flickr30K			RefCOCO+
	Caption	General	Test-Dev	Dev	Test-P	R@1	R@5	R@10	Dev	TestA	TestB
Base	-	-	69.26	68.40	68.65	42.86	73.62	83.28	70.66	77.06	61.43
VISTRA	CC	BC	70.74	71.74	71.02	55.37	82.93	89.84	72.42	79.11	64.19
VISTRA-disjoint	SBU	BC	70.70	71.97	72.11	56.12	82.82	90.12	73.05	79.48	64.19
VISTRA-nocaption	-	BC	70.47	71.47	71.19	54.36	82.22	89.24	72.96	79.30	64.25

4.3. Impact of Detector Tags and Semi-supervised Variants

We next examine the contribution of semantic anchors by ablating detector tags. Table 3 shows that excluding tags consistently reduces performance, with the most pronounced drops observed in the unsupervised configuration. For example, Flickr30K Recall@1 decreases from 55.37 to 50.56, and NLVR2 Test-P accuracy drops by over two points. These results empirically confirm that tags act as indispensable structural pivots that tether textual and visual spaces when alignment is missing.

Detector Tags as Alignment Catalysts.

The larger performance gap in the unsupervised case compared to the supervised baseline highlights the catalytic role of tags: without explicit supervision, anchors become the dominant mechanism that facilitates cross-modal consistency. In their absence, the model struggles to discover alignment signals, underscoring the necessity of structured semantic bridges.

Hybrid Training as a Synergistic Strategy.

We also experiment with a semi-supervised regime that integrates a small amount of aligned data with large unaligned corpora. This hybrid configuration yields the strongest overall performance, surpassing both purely supervised and unsupervised models. For instance, on NLVR2 Test-P the hybrid model attains 74.82, higher than either baseline. This observation indicates that combining curated annotations with abundant unpaired data can lead to synergistic improvements, suggesting an attractive direction for future multimodal pre-training paradigms.

4.4. Consolidated Insights

Our experimental exploration leads to several overarching insights:

• VISTRA performs competitively with supervised methods on diverse benchmarks, even though it relies solely on unpaired corpora.
• Detector-generated tags are indispensable for unsupervised grounding, providing the structural anchors needed to link modalities in the absence of alignment.
• The framework generalizes well across challenging setups, including training on disjoint corpora and text-only datasets devoid of captions.
• Incorporating hybrid supervision further enhances performance, pointing to a promising avenue for blending annotated and unannotated resources in future research.

Collectively, these findings reinforce our central thesis: scalable and high-quality multimodal representations can be obtained without reliance on expensive parallel data. By exploiting semantic anchors and carefully designed objectives, VISTRA presents a practical path forward for building annotation-efficient V&L models.

5. Conclusion and Future Work

This paper has introduced a new perspective on vision-and-language (V&L) representation learning by demonstrating that competitive multimodal encoders can be obtained without any reliance on explicitly paired image-text annotations. Our framework, named VISTRA (Vision-Text Representation Alignment), rethinks pre-training by decoupling the learning of visual and textual corpora while using detector-generated tags as implicit semantic bridges. Through this design, the heavy requirement for curated parallel datasets is substantially relaxed, showing that even loosely grounded signals from disjoint corpora can suffice to construct strong multimodal representations.

We validated the effectiveness of this paradigm through extensive experiments on four standard V&L benchmarks: VQA, NLVR2, Flickr30K Retrieval, and RefCOCO+. The results consistently indicate that VISTRA is capable of matching or even surpassing a number of supervised and weakly supervised baselines. Particularly noteworthy is its robustness in more challenging settings—such as training with fully disjoint caption sources or with general-domain text devoid of explicit visual descriptions—where the model continues to achieve strong results. Ablation analyses further confirmed that detector tags are indispensable, as they provide soft anchors that tie linguistic symbols to visual regions, thereby enabling cross-modal grounding under weak or noisy supervision.

At the methodological level, the essence of VISTRA lies in its modality-agnostic Transformer backbone, unified embedding vocabulary shared across textual tokens and detector tags, and its reconstruction-driven pre-training losses. Together, these design elements foster the emergence of semantically meaningful alignment across modalities without the need for explicit pairwise annotation. The addition of tag-level masked modeling augments this process by introducing lightweight yet highly effective visual-linguistic cues, thereby improving the quality of grounding and enhancing transferability to downstream tasks.

Beyond its empirical validation, the framework also points toward broader implications. In real-world or low-resource scenarios where curated parallel corpora are costly or infeasible—such as medical imaging, surveillance footage, or industrial inspection—VISTRA provides a scalable solution that leverages the natural abundance of unpaired corpora. The ability to extract value from such unstructured data streams could democratize access to multimodal pre-training, extending it to domains and communities that have historically been excluded due to annotation bottlenecks. Moreover, the approach encourages reconsideration of how multimodal systems are built, shifting the focus from annotation-heavy pipelines to annotation-light but semantically rich alternatives.

Looking forward, several natural extensions arise. One promising direction is to expand VISTRA into the temporal domain, enabling unsupervised pre-training for video-language tasks through frame-level anchoring and sequence-aware modeling. Another avenue involves exploring stronger alignment induction methods—such as adversarial alignment or cross-modal contrastive learning—to further tighten the cohesion between modalities. In addition, integrating higher-order symbolic signals, including scene graphs or structured semantic parses, with detector tags may unlock compositional reasoning capabilities and improve generalization to complex multimodal inference tasks. Finally, studying how unaligned multimodal pre-training interacts with downstream adaptation techniques such as prompt tuning or lightweight finetuning could provide practical strategies for task-specific deployments.

In conclusion, VISTRA establishes an alternative path toward scalable multimodal pre-training that is efficient, annotation-light, and robust across domains. By proving that strong performance can be achieved without aligned corpora, our study underscores the viability of unsupervised pre-training as a foundation for generalizable and interpretable V&L models, and paves the way for future research into broader, more inclusive multimodal intelligence.