Decoupled Yet Aligned Transformer for Semantic Image-Text Retrieval

1. Introduction

The fusion of computer vision and natural language processing has catalyzed significant advances in multimodal learning, particularly in tasks requiring semantic understanding across different data modalities. One prominent and widely studied problem is image-text retrieval, where the objective is to identify visually grounded content that corresponds semantically to a textual description, or vice versa. Applications of this task include image search, automated media tagging, human-computer interaction, and content-based recommendation systems. Despite its utility, the semantic and structural divergence between image pixels and natural language makes retrieval inherently difficult, especially when faced with large-scale datasets and time-critical response demands.

This paper explores the development of a robust and scalable multimodal retrieval framework. Our core objective is to learn compact yet semantically expressive representations of images and text that are directly comparable within a shared latent space. Toward this goal, we propose DUET, a dual-stream transformer encoder architecture that maintains early-layer modality specialization and enables semantic fusion through deep-layer parameter sharing. This architecture ensures that modality-specific characteristics are preserved, while high-level abstraction is performed within a semantically consistent embedding space.

Our design philosophy stems from the recognition of shortcomings in previous work [1,2,3,4,5]. While many methods integrate modalities early through mutual attention or fusion modules, this tightly coupled interaction prevents independent feature encoding—a critical bottleneck for real-time retrieval systems. In contrast, DUET facilitates decoupled feature extraction, enabling each modality to be indexed separately and searched independently. Semantic consistency is still achieved via aligned transformation in the shared transformer layers, leveraging the contextual modeling power of self-attention networks [8].

The challenge of aligning vision and language stems from their inherent representational mismatch: images encode object-centric visual patterns and spatial dependencies, whereas text represents abstracted, sequential information. Bridging this modality gap demands not only fine-grained object detection and grounding, but also the modeling of complex relationships and contextual cues. For instance, retrieving a relevant image for the sentence “A boy is kicking a soccer ball” necessitates identifying not just the visual entities (boy, ball), but also their interactions and the specific action involved (kicking).

Legacy models based on convolutional backbones [4] often yield coarse global representations inadequate for such nuanced reasoning. Region-based methods attempt to resolve this but often fall short in capturing semantic dependencies among visual regions. Likewise, early recurrent language models struggle with capturing long-range dependencies and global context. Transformer-based architectures [3,37], with their inherent self-attention mechanism, offer a promising alternative by modeling fine-grained dependencies across and within modalities.

Nevertheless, most existing transformer-based frameworks utilize unified encoding pipelines that inhibit independent precomputation—a major drawback when scaling to large candidate pools. Real-world systems require that a query vector $\varphi \left(C\right)$ can be matched against a massive image database $\varphi \left(I\right)$ via similarity scoring $s=\varphi (I,C)$, without evaluating pairwise interactions for every image-text pair [6]. DUET circumvents this bottleneck by introducing a decoupled transformer backbone where vision and language features are encoded separately, but fused via shared deep-layer weights, ensuring semantically aligned embeddings amenable to efficient approximate nearest-neighbor search.

In parallel, existing evaluation methodologies predominantly adopt strict metrics such as Recall@K, which assume exact ground-truth pairings between images and captions. Such rigid criteria overlook the presence of semantically plausible alternatives—e.g., multiple captions describing the same scene differently. To reflect real-world relevance more faithfully, we adopt a semantic-aware evaluation strategy by employing a caption-similarity-weighted version of discounted cumulative gain (DCG) [2]. This method enables a more graded notion of relevance, rewarding near-miss results in accordance with their semantic proximity to the ground truth.

To summarize, our contributions are three-fold:

• We introduce DUET, a novel dual-stream transformer framework that disentangles early-stage modality encoding and achieves semantic alignment through parameter sharing in deeper layers.
• We propose a caption-similarity-weighted DCG metric that accounts for graded semantic relevance, addressing the limitations of traditional binary evaluation criteria.
• We provide extensive experiments on the MS-COCO benchmark, showing that DUET sets a new performance standard under both exact match and semantic retrieval metrics.

By reconceptualizing image-text representation learning through a modular and semantically grounded architecture, DUET advances the frontier of efficient and intelligent cross-modal retrieval.

2. Related Work

To contextualize our contributions, we examine three primary lines of research that inform our approach: (1) methods for constructing cross-modal embedding spaces for image-text alignment, (2) neural reasoning frameworks that support high-order relational modeling, and (3) evaluation strategies that accommodate semantic fuzziness in retrieval tasks.

2.1. Cross-Modal Embedding Strategies for Alignment

Establishing a joint semantic space for vision and language lies at the heart of image-text retrieval. The majority of traditional methods approach this problem by learning embedding functions that map images and textual descriptions into a shared vector space, where semantically related pairs are drawn closer under a similarity measure—such as cosine similarity, dot product, or contrastive losses.

In earlier paradigms, visual features were extracted using convolutional neural networks (CNNs) pretrained on large-scale classification datasets. Commonly used backbones included VGG [9,10,11,12,13] and ResNet [1,14,15,16], whose penultimate layer activations served as global image descriptors. However, these global encodings often overlooked fine-grained details and object relationships, limiting their efficacy in grounding sentence semantics.

To improve localization and semantic grounding, attention shifted toward region-level visual features. The bottom-up attention mechanism introduced by [18] enabled the extraction of object-centric features using Faster R-CNN, providing a basis for localized representation learning. Subsequent efforts such as [5,7] built upon this by incorporating cross-modal attention, selectively integrating salient image regions with sentence tokens—thus filtering visual noise and aligning relevant entities.

On the language side, early systems utilized recurrent neural networks (RNNs), including GRUs and LSTMs, to encode sequential sentence inputs [1,4,7,16]. While effective for capturing local dependencies, RNNs struggled with modeling long-range relations and global sentence semantics due to their inherent memory constraints.

The introduction of transformer architectures [8] marked a turning point. Models like BERT [37] demonstrated strong capacity for context-aware encoding via self-attention, inspiring a wave of cross-modal transformers such as ViLBERT [3] and ImageBERT [6]. These models extended transformer-based reasoning to visual inputs by integrating region features and token embeddings into a unified attention space.

Despite their strong performance, unified transformers often entangle visual and textual inputs from early layers, precluding modular feature extraction. This tight coupling poses a bottleneck for retrieval systems that demand scalability. Systems that rely on pairwise interaction scores $s=\varphi (I,C)$ become computationally prohibitive when deployed over millions of candidates [6].

To alleviate this, dual-encoder designs—where visual and textual inputs are encoded independently as ${\psi }_v\left(I\right)$ and ${\psi }_t\left(C\right)$ into a shared space—have gained traction. These allow for decoupled feature precomputation and efficient retrieval via approximate nearest neighbor search. Works like [7] explored this line by using GCNs for visual reasoning and GRUs for sentence modeling, paired with auxiliary sentence reconstruction to stabilize training. Our approach extends this idea, substituting both modality pipelines with transformer-based architectures to enhance abstraction and enable end-to-end semantic fusion.

2.2. Relational Reasoning Mechanisms in Neural Architectures

Beyond embedding alignment, advanced retrieval models must reason over object interactions, contextual cues, and structured semantic relationships. A seminal contribution in this direction was the Relation Network (RN) by [19], which explicitly separated perception and reasoning. RNs operated by applying relational functions over all object pairs, conditioned on a question representation from an LSTM. While effective in visual QA, the framework lacked generalization to complex multimodal alignment tasks due to its rigid pairwise formulation.

Efforts to adapt RN-style architectures for retrieval, such as [20,21], focused on aggregating relational cues into compact feature representations. These models introduced various relation encoding mechanisms but were still confined to visual-only pipelines and did not generalize well to cross-modal contexts. Symbolic reasoning approaches [22,23] introduced more interpretable mechanisms by representing the reasoning process as sequences of programmatic operations. These include neural module networks and differentiable execution graphs. While powerful, they typically require detailed structured annotations, which limits their applicability in large-scale weakly supervised settings.

Graph-based reasoning models offer a middle ground, where structured representations are learned without requiring strict symbolic supervision. Graph convolutional networks (GCNs) have been widely adopted for visual reasoning tasks [24,25,26], modeling relationships among detected objects or scene entities. Other works construct scene graphs from images [27,28] and use them to perform structured reasoning over visual scenes. These methods provide strong relational inductive biases and facilitate deeper semantic abstraction, paving the way for unified scene-text understanding.

2.3. Beyond Binary: Semantic-Aware Retrieval Evaluation

The final axis of comparison concerns evaluation metrics. Standard evaluation in image-text retrieval typically employs Recall@K [1,3,6,7,29], which assesses whether a ground-truth item is retrieved within the top K ranked results. While this measure provides a clear-cut benchmark for exact-match retrieval, it lacks sensitivity to semantic approximation. Consider the case where the target caption is “a boy playing football,” and the retrieved image depicts “a child kicking a soccer ball.” Though semantically similar, traditional metrics would penalize such retrievals. This motivates the adoption of evaluation protocols that reflect graded semantic relevance.

To this end, [2] proposed a DCG-based evaluation scheme where each retrieved item is weighted by its semantic similarity to the query, measured via caption embeddings. This approach assigns partial credit to near-miss retrievals and aligns better with human perception. Building on this, our work introduces a similarity-weighted DCG protocol tailored for cross-modal scenarios, which factors in semantic variability and recognizes soft alignments in real-world applications. As image-text retrieval systems mature and expand in scale, it becomes increasingly critical to evaluate them not only on exact correctness, but also on their ability to retrieve content that is semantically aligned, contextually relevant, and diverse in linguistic expression.

3. Preliminary

3.1. Pipeline Formalization and Notation

We begin by establishing the notational conventions and core components of standard image captioning frameworks, which predominantly follow an encoder-decoder paradigm consisting of three integral modules: a visual encoder, a language decoder, and a context-aware word predictor.

Given an image input, the visual encoder—implemented via either CNN backbones or region-based detectors like Faster R-CNN—generates feature representations denoted by $\mathit{I}\in {\mathbb{R}}^{n_v\times d_v}$, where $n_v$ is the number of extracted spatial locations or detected regions, and $d_v$ denotes the feature dimensionality. In region-based settings, $n_v$ corresponds to object proposals, while in CNNs it reflects flattened spatial grids.

At each decoding timestep t, the language decoder (typically an LSTM) evolves its hidden state ${\mathit{h}}_t$ based on prior word information and a global visual summary. The decoder input ${\mathit{x}}_t$ and recurrent updates follow:

$$\begin{array}{cc}\hfill {\mathit{x}}_t& =[{\mathit{E}}_w\left(w_{t-1}\right),{\mathit{I}}_g]\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill {\mathit{h}}_t,{\mathit{m}}_t& =\mathrm{LSTM}({\mathit{x}}_t,{\mathit{h}}_{t-1},{\mathit{m}}_{t-1})\hfill \end{array}$$

(2)

where ${\mathit{E}}_w(·)$ denotes the word embedding lookup, and ${\mathit{I}}_g={\displaystyle \frac{1}{n_v}}{\sum }_{k=1}^{n_v}{\mathit{I}}_{\left(k\right)}$ represents a global average of image features. The decoder output ${\mathit{h}}_t$ is passed into an attention module to produce a context vector ${\mathit{c}}_t$, followed by vocabulary prediction:

$$\begin{array}{cc}\hfill {\mathit{c}}_t& =\mathrm{ATT}({\mathit{h}}_t,\mathit{I})\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill {\mathit{p}}_t& =\mathrm{Predictor}({\mathit{h}}_t,{\mathit{c}}_t)\hfill \end{array}$$

(4)

The predicted score vector ${\mathit{p}}_t$ defines a probability distribution over the vocabulary at step t. The implementation of the attention function and prediction layer defines the model variant.

3.2. Visual-Linguistic Fusion via Attention Mechanisms

We explore two prominent attention designs for integrating visual information during decoding: (1) adaptive attention mechanisms incorporating a sentinel gate, and (2) multi-head attention frameworks based on Transformer formulations.

3.2.1. Adaptive Attention with Sentinel Mechanism

Adaptive attention introduces a dynamically learned sentinel vector ${\mathit{s}}_t$ that selectively retains non-visual (linguistic or memory) information. This vector is computed as:

$$\begin{array}{cc}\hfill {\mathit{s}}_t& =\sigma ({\mathit{W}}_x{\mathit{x}}_t+{\mathit{W}}_h{\mathit{h}}_{t-1})\odot tanh\left({\mathit{m}}_t\right)\hfill \end{array}$$

(5)

where ${\mathit{W}}_x\in {\mathbb{R}}^{d_h\times d_x}$ and ${\mathit{W}}_h\in {\mathbb{R}}^{d_h\times d_h}$ are learned projections, and $\sigma (·)$ is a sigmoid gating function. This sentinel serves as an additional source for attention, parallel to the visual features.

Attention scores are then computed for both the visual regions and the sentinel:

$$\begin{array}{cc}\hfill \mathit{a}& ={\mathit{w}}_a^{\top }tanh(\mathit{I}{\mathit{W}}_I+{\mathit{W}}_g{\mathit{h}}_t)\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill b& ={\mathit{w}}_a^{\top }tanh({\mathit{W}}_s{\mathit{s}}_t+{\mathit{W}}_g{\mathit{h}}_t)\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill {\mathit{\alpha }}_t& =\mathrm{softmax}\left(\mathit{a}\right)\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill {\beta }_t& =\mathrm{softmax}{\left([\mathit{a};b]\right)}_{(n_v+1)}\hfill \end{array}$$

(9)

The final context vector is a weighted interpolation between visual features and the sentinel:

$$\begin{array}{cc}\hfill {\mathit{c}}_t& =(1-{\beta }_t)\sum _{k=1}^{n_v}{\alpha }_{t_k}{\mathit{I}}_{\left(k\right)}+{\beta }_t{\mathit{s}}_t\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill \Rightarrow {\mathit{c}}_t& ={\mathrm{ATT}}_{\mathrm{ada}}({\mathit{h}}_t,{\mathit{s}}_t,\mathit{I})\hfill \end{array}$$

(11)

The scalar ${\beta }_t$ controls the extent to which non-visual context influences the final representation.

3.2.2. Transformer-Based Multi-Head Attention

An alternative approach adopts multi-head attention (MHA), enabling the model to simultaneously attend to various semantic dimensions of the input via parallel projections. The mechanism proceeds as:

$$\begin{array}{cc}\hfill \mathit{Q}& ={\mathit{h}}_t,\mathit{K}=\mathit{I}{\mathit{W}}_K,\mathit{V}=\mathit{I}{\mathit{W}}_V\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill {\mathit{\alpha }}^{\left(i\right)}& =\mathrm{softmax}\left({\displaystyle \frac{{\mathit{Q}}^{\left(i\right)}{\mathit{K}}^{\left(i\right)\top }}{\sqrt{d_h/n_h}}}\right)\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill {\mathit{v}}^{\left(i\right)}& =\sum _{k=1}^{n_v}{\alpha }_k^{\left(i\right)}{\mathit{V}}_k^{\left(i\right)}\hfill \end{array}$$

(14)

The context vectors from each head are concatenated and linearly transformed:

$$\begin{array}{cc}\hfill \mathit{v}& =[{\mathit{v}}^{\left(1\right)},\dots ,{\mathit{v}}^{\left(n_h\right)}],\widehat{\mathit{v}}={\mathit{W}}_v\mathit{v}+{\mathit{b}}_v\hfill \end{array}$$

(15)

Finally, a gated fusion is applied to regulate visual contributions:

$${\mathit{c}}_t=\sigma ({\mathit{W}}_{mh}{\mathit{h}}_t+{\mathit{b}}_{mh})\odot \widehat{\mathit{v}}={\mathrm{ATT}}_{\mathrm{mha}}({\mathit{h}}_t,\mathit{I})$$

(16)

This formulation allows the decoder to dynamically scale visual influence based on linguistic context.

3.3. Instantiating Representative Architectures

For comparative analysis and ablation, we instantiate two prototypical image captioning systems based on the above attention mechanisms:

• Ada-LSTM: A hybrid model combining adaptive attention with an LSTM-based decoder and a standard prediction head.
• MH-FC: A multi-head attention variant employing transformer-style attention and a feedforward classifier over fused features.

3.4. Training Losses and Learning Objectives

Initial model training is conducted under the standard cross-entropy (XE) objective, encouraging likelihood maximization of ground-truth sequences:

$${\mathcal{L}}_{\mathrm{ce}}=-\sum _{t=1}^{l}logp(w_t^{*}\mid w_{<t},\mathit{I})$$

(17)

where $w_t^{*}$ is the target token at step t and $p(·)$ denotes the decoder’s softmax distribution.

To align training with task-specific reward metrics like CIDEr, we subsequently apply reinforcement learning via Self-Critical Sequence Training (SCST) [?]:

$${\mathcal{L}}_{\mathrm{scst}}=-R\sum _{t=1}^{l}logp\left(w_t^s\right)$$

(18)

Here, the reward R is computed as the CIDEr score differential between a sampled caption $S^s$ and its greedy counterpart $S^{greedy}$:

$$R=\mathrm{CIDEr}(S^s,S^{gt})-\mathrm{CIDEr}(S^{greedy},S^{gt})$$

(19)

This encourages sampled captions to surpass greedy baselines in evaluation score:

$$\underset{\theta }{max}{\mathbb{E}}_{S^s\sim p_{\theta }}\left[R\left(S^s\right)\right]$$

(20)

3.5. Extended Design: Context Aggregation and Alignment Regularization

To further enhance model flexibility and performance, we incorporate two optional modules commonly used in advanced captioning systems:

Soft Gated Aggregation

We generalize hard attention gating by introducing a mixture-of-experts formulation over multiple context pathways:

$${\mathit{c}}_t=\sum _{i=1}^M{\gamma }_i{\mathit{c}}_t^{\left(i\right)},{\gamma }_i={\displaystyle \frac{e^{s_i}}{{\sum }_{j=1}^M{e}^{s_j}}}$$

(21)

Each ${\mathit{c}}_t^{\left(i\right)}$ corresponds to a specific feature channel (e.g., visual-only, sentinel, language-derived), with learned gating weights ${\gamma }_i$ indicating their contribution.

Context Alignment Loss

To promote semantic alignment between the attended visual context and language embeddings, we adopt an auxiliary supervision term:

$${\mathcal{L}}_{\mathrm{align}}=\sum _{t=1}^l{\parallel {\mathit{c}}_t-{\mathit{E}}_w\left(w_t^{*}\right)\parallel }_2^2$$

(22)

This regularization encourages the visual context to remain compatible with ground-truth word semantics, especially in early training stages.

Figure 1. Overview of the proposed framework.

Figure 1. Overview of the proposed framework.

4. Unified Multimodal Reasoning with Dual Transformers

We propose DUET (Dual-Stream Encoder for Unified Embedding and Translation), a unified framework for cross-modal semantic reasoning built entirely upon Transformer Encoder (TE) backbones. DUET processes visual and textual modalities through distinct yet partially shared encoder streams, designed to preserve modality-specific nuances while facilitating joint abstraction in a shared semantic space. This section details our model design, including input representations, dual encoder architecture, contrastive alignment objective, and regularization strategies.

4.1. Representation of Multimodal Inputs

Let a paired training sample consist of an image I and its corresponding caption C. We represent $I=\{r_0,r_1,...,r_n\}$ as n region-level visual embeddings, and $C=\{w_0,w_1,...,w_m\}$ as m tokenized word embeddings. Each modality is prepended with a [CLS]-style summary token—$r_0^{\left(\mathrm{I}\text{-}\mathrm{CLS}\right)}$ for image and $w_0^{\left(\mathrm{T}\text{-}\mathrm{CLS}\right)}$ for text—to aggregate global semantic cues during encoding. These special tokens are subsequently used to compute cross-modal matching scores.

4.1.1. Spatially-Aware Visual Embedding

For visual input, we adopt bottom-up attention features as in [31], using a pre-trained Faster R-CNN [30] detector on Visual Genome [32]. Each region $r_i$ is associated with a high-dimensional feature ${\mathbf{r}}_i\in {\mathbb{R}}^{d_v}$ and a normalized spatial box ${\mathbf{b}}_i$ defined as:

$${\mathbf{b}}_i=\left[{\displaystyle \frac{x_1}{W}},{\displaystyle \frac{y_1}{H}},{\displaystyle \frac{x_2}{W}},{\displaystyle \frac{y_2}{H}},{\displaystyle \frac{(x_2-x_1)(y_2-y_1)}{WH}}\right].$$

(23)

The combined visual-spatial descriptor $[{\mathbf{r}}_i;{\mathbf{b}}_i]$ is projected through a two-layer MLP:

$${\widehat{\mathbf{r}}}_i={\mathrm{MLP}}_{\mathrm{vis}}\left([{\mathbf{r}}_i;{\mathbf{b}}_i]\right)={\mathbf{W}}_2·\mathrm{ReLU}({\mathbf{W}}_1[{\mathbf{r}}_i;{\mathbf{b}}_i]+{\mathbf{b}}_1)+{\mathbf{b}}_2,$$

(24)

yielding spatially enriched region features ${\widehat{\mathbf{r}}}_i\in {\mathbb{R}}^d$.

4.1.2. Contextualized Language Encoding via BERT

The textual component is encoded using BERT [37], which provides context-sensitive token embeddings. For a caption C of m words:

$${\widehat{\mathbf{w}}}_j=\mathrm{BERT}{\left(C\right)}_j\in {\mathbb{R}}^d,\forall j\in [1,m],$$

(25)

where ${\widehat{\mathit{w}}}_0$ refers to the [T-CLS] embedding summarizing sentence-level semantics. Since BERT already encodes positional information, no explicit sequence modeling is required.

4.2. Dual Transformer Encoder Design

DUET consists of two Transformer Encoder branches—${\mathrm{TE}}_{\mathrm{vis}}$ for images and ${\mathrm{TE}}_{\mathrm{text}}$ for text—each processing its respective inputs independently in early layers and partially sharing parameters in the upper layers. The attention mechanism follows the canonical formulation:

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^{\top }}{\sqrt{d_k}}}\right)V,$$

(26)

with $Q=X{W}^Q$, $K=X{W}^K$, and $V=X{W}^V$ denoting learned projections.

Stacking L such layers yields contextualized representations ${\mathbf{h}}_{\mathrm{vis}}^{\left(L\right)}\in {\mathbb{R}}^{n+1\times d}$ and ${\mathbf{h}}_{\mathrm{text}}^{\left(L\right)}\in {\mathbb{R}}^{m+1\times d}$. We extract final image and caption embeddings via their respective [CLS] tokens:

$${\mathbf{v}}_I={\mathbf{h}}_{\mathrm{vis},0}^{\left(L\right)},{\mathbf{v}}_C={\mathbf{h}}_{\mathrm{text},0}^{\left(L\right)}.$$

(27)

To enforce high-level semantic alignment, the final k encoder layers are shared across both streams, encouraging convergence in the abstract feature space.

4.3. Cross-Modal Contrastive Alignment

To train DUET for retrieval, we adopt a bidirectional contrastive objective that maximizes the similarity between matched pairs and penalizes mismatched ones. Given cosine similarity $S(i,c)=cos({\mathbf{v}}_I,{\mathbf{v}}_C)$, the margin-based loss is:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{match}}(i,c)& =\underset{c^{\prime }}{max}{[\alpha +S(i,c^{\prime })-S(i,c)]}_{+}\hfill \\ \hfill & +\underset{i^{\prime }}{max}{[\alpha +S(i^{\prime },c)-S(i,c)]}_{+},\hfill \end{array}$$

(28)

where ${[·]}_{+}$ is the hinge function and $\alpha$ is the contrastive margin. The hard negatives $i^{\prime }$ and $c^{\prime }$ are mined from the current mini-batch:

$$i^{\prime }=arg\underset{j\ne i}{max}S(j,c),c^{\prime }=arg\underset{k\ne c}{max}S(i,k).$$

(29)

This encourages the model to be discriminative over subtle mismatches and robust to distractors.

4.4. Auxiliary Learning Signals and Regularization

To further enhance alignment quality and model stability, we incorporate two auxiliary losses:

(i) Spatial Coordinate Regression.

From an intermediate transformer layer $l<L$, we reconstruct region coordinates using a lightweight regression head:

$${\widehat{\mathbf{b}}}_i={\mathrm{MLP}}_{\mathrm{pos}}\left({\mathbf{h}}_i^{\left(l\right)}\right),$$

(30)

and define the coordinate loss as:

$${\mathcal{L}}_{\mathrm{bbox}}={\displaystyle \frac{1}{n}}\sum _{i=1}^n{∥{\widehat{\mathbf{b}}}_i-{\mathbf{b}}_i∥}^2.$$

(31)

(ii) Embedding Norm Stability.

To ensure consistent scale and enhance angular similarity, we regularize embedding magnitudes:

$${\mathcal{L}}_{\mathrm{norm}}=\left|\parallel {\mathbf{v}}_I{\parallel }_2-1\right|+\left|\parallel {\mathbf{v}}_C{\parallel }_2-1\right|.$$

(32)

This constraint mitigates representational drift during training.

4.5. Overall Optimization Objective

The final objective combines all components with tunable weights ${\lambda }_1$ and ${\lambda }_2$:

$${\mathcal{L}}_{\mathrm{total}}={\mathcal{L}}_{\mathrm{match}}+{\lambda }_1{\mathcal{L}}_{\mathrm{bbox}}+{\lambda }_2{\mathcal{L}}_{\mathrm{norm}}.$$

(33)

Empirically, we observe that these auxiliary objectives improve both training convergence and retrieval performance, particularly in scenarios involving fine-grained spatial reasoning.

5. Experiment and Evaluation

We perform an extensive empirical study to evaluate the effectiveness of our proposed DUET framework on the image-text retrieval task. Experiments are conducted on the MS-COCO benchmark [36] following standard settings. This section addresses the following central questions:

• How does DUET compare to existing state-of-the-art methods under both hard (Recall@K) and soft (NDCG) evaluation protocols?
• Can DUET better capture semantic correspondence beyond literal matches?
• What impact do individual architectural components have on performance?

5.1. Dataset and Evaluation Metrics

We evaluate DUET using the MS-COCO dataset [36], which includes 123,287 images, each annotated with five diverse human-written captions. Following the widely accepted Karpathy split [1,4,7], we use 113,287 images for training, 5,000 for validation, and 5,000 for testing. All reported metrics are computed over both the 5K test set and the 1K subset averaged across five folds.

Evaluation comprises two distinct metric types:

• Recall@K: Measures the percentage of queries for which the correct result is among the top K retrieved items.
• NDCG@25: Captures ranking quality with soft relevance, computed using textual similarity functions including ROUGE-L [34] and SPICE [35].

5.2. Implementation Configuration

Textual embeddings are obtained via a pre-trained BERT encoder using the HuggingFace transformers library1, yielding 768-dimensional representations. For image inputs, we adopt the Bottom-Up Attention detector2 to extract the top 36 region features (2048-D) per image.

On the visual side, we employ four dedicated Transformer Encoder layers. For textual inputs, the BERT backbone is fine-tuned without adding extra layers. The top two Transformer layers are shared between both modalities, projecting into a joint 1024-dimensional embedding space.

Training proceeds for 30 epochs using Adam optimizer with a learning rate of $2\times {10}^{-5}$. The contrastive loss margin $\alpha$ is set to 0.2. We use a mini-batch size of 90, balancing performance and GPU memory constraints.

5.3. Comparison with State-of-the-Art Methods

Table 1 demonstrates that DUET consistently surpasses existing state-of-the-art methods across both strict (Recall@K) and soft (NDCG) retrieval metrics. The improvements are observed on both the smaller 1K subset and the larger 5K full test set.

In particular, DUET achieves the best R@1 and ${\mathrm{NDCG}}_{\mathtt{SPICE}}$ scores, indicating its superior precision and ability to capture high-level semantic similarity. Despite its simple design—eschewing ensemble learning or test-time augmentations—DUET significantly outperforms baselines like VSE++ and VSRN, attesting to the efficacy of its dual-stream transformer backbone and shared semantic reasoning layers.

The most pronounced gains occur under the soft NDCG metric, which rewards partial matches and semantic closeness. This indicates DUET’s ability to retrieve relevant content even in cases of lexical variance, which Recall@K alone fails to capture. These findings emphasize the necessity of incorporating semantic-aware metrics for realistic system evaluation.

Notable observations include:

• On the 1K set, DUET achieves +0.7 improvement in R@1 and +3.3 in ${\mathrm{NDCG}}_{\mathtt{SPICE}}$ over VSRN, highlighting its enhanced semantic alignment.
• On the full 5K test set, DUET retains an advantage across all metrics, showing scalability and robustness.
• Improvements in SPICE-based NDCG reflect DUET’s stronger grasp of conceptual similarity and abstract reasoning.

5.4. Qualitative Examples: Generalization Beyond Lexical Overlap

DUET is capable of retrieving images with semantically coherent but lexically divergent descriptions. For instance, given the query “a child leaping across water,” DUET correctly retrieves “a young girl jumping over a puddle.” Such cases exemplify its strength in capturing abstract semantic equivalence—a key requirement for practical applications. These results further justify the adoption of metrics like NDCG to assess generalization capacity.

5.5. Ablation Analysis

We conduct ablation studies to isolate the contributions of critical components in DUET’s architecture. Table 2 reports performance under four variants:

• DUET w/o shared layers: Disables transformer weight sharing.
• DUET w/o spatial features: Removes bounding-box coordinate input.
• DUET (no norm reg): Drops embedding norm regularization.
• Full DUET: The complete model as proposed.

Findings reveal that shared transformer layers contribute significantly to alignment performance, especially under NDCG. Spatial features are also essential, particularly for modeling grounding and inter-object relationships. Norm regularization, while less critical, helps stabilize training and marginally boosts precision.

5.6. Inference Efficiency and Scalability

Beyond accuracy, latency and scalability are critical in real-world systems. We evaluate DUET’s retrieval runtime on a 10K image corpus using precomputed embeddings and a cosine-based nearest neighbor index.

We utilize FAISS3 with a flat index structure for ANN search. DUET retrieves top results in ∼15 ms per query, including embedding lookup and ranking. This low-latency behavior stems from DUET’s decoupled modality encoders, enabling offline indexing and avoiding pairwise computation during inference.

DUET’s shared-layer design also produces compact, semantically rich embeddings, reducing storage footprint and increasing retrieval throughput. Its simplicity makes it well-suited for deployment across GPU and CPU environments with minimal engineering overhead. In summary, DUET achieves a strong balance between semantic interpretability, accuracy, and system-level efficiency—making it a practical choice for scalable image-text retrieval applications.

6. Conclusion and Future Work

This work addressed the central problem of efficient and scalable image-text retrieval by rethinking how multimodal representations are constructed, aligned, and utilized. A key observation driving our motivation is that many prior models intertwine vision and language from early stages, yielding entangled representations that hinder independent indexing and inference—two crucial requirements for real-world large-scale retrieval systems.

To overcome this challenge, we proposed DUET, a Dual-Stream Encoder for Unified Embedding and Translation, grounded in the Transformer Encoder (TE) architecture. DUET explicitly decouples visual and textual processing in early layers, preserving modality-specific structures, while progressively enabling shared semantic abstraction through weight-tied layers at higher depths. This design not only supports modular encoding and fast inference but also fosters richer cross-modal alignment by promoting structured interaction at the conceptual level. Additionally, DUET incorporates relational reasoning mechanisms to capture not only object-level semantics but also spatial configurations and intra-modality context.

Recognizing the inadequacy of binary relevance metrics for capturing semantic alignment, we introduced evaluation based on NDCG, which reflects graded relevance through similarity scores computed using ROUGE-L and SPICE. These soft metrics enable a more faithful assessment of semantic retrieval quality, particularly when retrieved items are paraphrased or contextually aligned but lexically different from the reference.

Our experiments on MS-COCO demonstrate that DUET establishes new state-of-the-art performance under both strict (Recall@K) and soft (NDCG) protocols, significantly outperforming strong baselines like VSE++ and VSRN. In particular, the gains under ${\mathrm{NDCG}}_{\mathtt{SPICE}}$ highlight DUET’s capability to model deeper relational semantics and its robustness to linguistic variability. These findings validate the efficacy of DUET’s dual-stream design in bridging the modality gap between vision and language while preserving scalability and deployment readiness.

Looking ahead, several extensions present themselves for exploration. First, we envision incorporating bidirectional reconstruction objectives that more explicitly structure the shared embedding space. For example, enabling the model to reconstruct sentence-level descriptions from visual embeddings—or vice versa, generate region-level visual features from text—may enhance the semantic granularity and controllability of learned representations.

Second, while the current hinge-based contrastive loss is effective for ranking, it imposes sharp margins that do not fully capture nuanced semantic proximity. We plan to investigate alternative alignment objectives such as soft contrastive losses or distribution-aware formulations, which account for relevance gradients and semantic fuzziness inherent in natural language.

Third, we aim to extend DUET beyond pairwise retrieval tasks. The modular transformer backbone can serve as a flexible substrate for broader multimodal tasks such as visual question answering, multimodal summarization, and commonsense reasoning. By enriching DUET with symbolic or hierarchical reasoning modules and adapting it to other data types—such as audio, video, and structured knowledge—we anticipate broader applicability in open-domain multimodal understanding.

In summary, DUET represents a principled and practical advancement toward semantically grounded multimodal retrieval. Through dual-stream reasoning, spatial-aware design, and contrastive training within a transformer framework, DUET bridges effectiveness with efficiency. We hope this work inspires future research into structured multimodal alignment and motivates the development of evaluation protocols that better reflect real-world relevance and user intent.