FashionIQ-Based Explainable Composed Image Retrieval Using Advanced Multilingual LMM’s

1. Introduction

In the fields of e-commerce and digital fashion, the ability to retrieve products using both visual and linguistic cues is growing increasingly important. Traditional image retrieval systems have two inherent limitations: they either rely solely on visual similarity matching, or only support text keyword queries. Neither can capture users’ nuanced need to “find target products that are similar to a reference image but include specific modifications.” Composed Image Retrieval (CIR) fills this gap: it allows users to submit a reference image and natural language modification queries at the same time, to match target products that meet the semantic requirements of both types of input.

The fashion sector is one of the most commercially valuable and practically viable application scenarios for CIR. Current text-based retrieval methods cannot enable users to clearly articulate their visual preferences, while purely visual search also fails to capture demand for apparel design modifications. Systems that can simultaneously understand visual context and linguistic instructions can accommodate needs such as changing the floral pattern on a skirt or adjusting the sleeve length of a shirt, which aligns with the logic of natural human communication.

While existing CIR methods boast considerable real-world application prospects, they still face two core limitations that hinder their widespread adoption: first, nearly all systems only support English queries, which restricts technical accessibility to this technology for the vast majority of non-English users across the globe; second, their black-box operation makes their retrieval logic impossible to explain, which weakens user trust and limits their practical real-world promotion.

1.1. Problem Statement

In recent years, multimodal learning and vision-language models have achieved notable progress, yet existing compositional image retrieval frameworks still face core limitations: they only support English, failing to reach hundreds of millions of non-English users speaking languages such as Urdu and Chinese; they operate as black-box systems lacking interpretability, and cannot output the visual and linguistic basis for their retrieval decisions; and the short modification captions of the widely used benchmark FashionIQ are semantically sparse, which is insufficient to support accurate retrieval. These are exactly the core problems this study seeks to solve.

1.2. Research Objectives

This study explicitly lists the following research objectives:

• Develop a multilingual CIR framework that supports English, Urdu, and Chinese queries on the FashionIQ benchmark dataset.
• Develop a projection head to align multilingual text embeddings output by cross-language encoders to the visual feature space of OpenAI’s CLIP.
• Integrate LLaVA to generate target image descriptions that supplement short modification queries.
• Evaluate the proposed framework through comprehensive quantitative and ablation experiments on the FashionIQ validation set.

1.3. NLP and Multimodal Approaches for CIR

Large-scale vision-language pre-training has reshaped the development landscape of the image retrieval field. The joint embedding space of CLIP, which is trained on hundreds of millions of paired image-text data, forms the core foundation of modern CIR systems. However, its text encoder natively only supports a single language, which restricts its application in multilingual scenarios.

Widely used multilingual text encoders in the field, such as multilingual-E5 and mBERT, possess cross-lingual representation capabilities, but their embedding spaces are incompatible with visual encoders trained with English as their core training language. The cross-language cross-modal retrieval system proposed in this research bridges this gap by training a projection head, and introduces large multimodal models such as LLaVA to implement query augmentation, which solves the semantic sparsity problem of short modified descriptions in benchmark datasets. This study uses the neural machine translation model M2M100 to complete its multilingual processing pipeline.

2. Related Work

The foundational work in the field of composed image retrieval (CIR) was completed by Vo et al. [1] in 2019: this team formally defined the research task, released the TIRG model and the FashionIQ benchmark, and established the core terminology and evaluation protocol that have been adopted across the entire field ever since. Subsequent work by Baldrati et al. [2,3] significantly elevated the field’s performance baseline through CLIP adaptation methods, but all existing CIR systems still lack multilingual support and visual interpretability.

2.1. Composed Image Retrieval: Methods and Benchmarks

In 2019, Vo et al. [1] first formally proposed the official academic task of Composed Image Retrieval. In the same period, they released the Text Image Residual Gating (TIRG) model and the FashionIQ benchmark dataset. TIRG achieves selective modulation of visual features by fusing two types of embeddings through element-wise multiplication and addition. However, its shallow gating mechanism cannot capture complex compositional semantics.

The FashionIQ dataset released by Wu et al. [4] in 2021 has become a standard benchmark for synthetic image retrieval in the field of fashion research. Each sample in this dataset is structured as a triplet, consisting of a reference image, two natural language modification descriptions, and a target image. The dataset covers three categories of apparel: dresses, shirts, and tops including T-shirts. It focuses on four types of fine-grained differences: pattern, color, neckline style, and sleeve style, and evaluates retrieval accuracy using three metrics: R@1, R@5, and R@10.

All current cross-modal retrieval (CIR) methods in the field of fashion image retrieval share a core limitation: they only support English queries. FashionIQ, the field’s authoritative benchmark, only provides English modification descriptions, and no mature multilingual solutions have been developed to date. All current cross-modal image retrieval (CIR) systems also lack interpretability mechanisms. This study integrates Grad-CAM saliency maps into a self-developed framework to fill this research gap.

2.2. Vision-Language Pretraining and CLIP

In 2021, Radford et al. [11] proposed CLIP (Contrastive Language-Image Pre-Training), a multimodal pre-training model that serves as a turning point in the field of vision-language research. Trained on 400 million web-scraped image-text pairs, the model applies contrastive learning to map semantically similar images and text to adjacent positions in a joint embedding space. Baldrati et al.’s [2] team proposed the CLIP4CIR method equipped with a dual-core module, which delivered performance far surpassing prior approaches on the FashionIQ and CIRR benchmarks, establishing CLIP as the de facto backbone for the entire field.

In the multilingual cross-modal image-text retrieval scenario, the mainstream baseline model CLIP has a core defect: although its visual encoder is designed to be language-agnostic, its text encoder is trained solely on English captions, so it cannot adapt to non-English queries. This study introduces multilingual-E5-base as its multilingual text encoder, and trains a projection head to map cross-lingual text embeddings to the CLIP visual feature space, so as to preserve both the quality of the original visual representations and the capability to handle multilingual queries.

2.3. Multilingual and Cross-Lingual Text Embeddings

The mainstream multilingual dense passage retrieval model multilingual-E5, proposed by Wang et al. [24], is fine-tuned on multilingual text pairs with a contrastive objective. Its base version outputs 768-dimensional embeddings, delivers outstanding performance on the MIRACL and mMARCO benchmarks, supports more than 100 languages, and fits the multilingual CIR framework of this study.

This study proposes a cross-modal retrieval framework that sets Urdu as its core target language. Urdu uses a right-to-left writing system, has complex morphology, contains a large number of Persian-Arabic loanwords, and has more than 200 million native speakers. This language is severely underrepresented in existing multilingual pre-trained corpora, and no cross-modal image retrieval system currently supports it.

2.4. Neural Machine Translation for Query Expansion

The multilingual neural machine translation model M2M100, proposed by Fan et al. [33] in 2021, removes the key constraint of earlier models that required English to serve as a pivot language. This model supports mutual translation between 100 languages and avoids the extra errors that come with English-pivot translation. Its 418 million-parameter version strikes a solid balance between translation quality and computing power costs, so this study selects it as the core backbone translation model.

2.5. Large Multimodal Models for Visual Understanding

The first-generation LLaVA, proposed by Liu et al. [39] in 2023, connects a CLIP visual encoder and a LLaMA language model via a lightweight linear projection layer. Trained on visual instruction tuning data composed of instruction-following image-text pairs, this model can generate image descriptions with rich details.

The fashion image retrieval framework proposed in this study targets the inherent pain point of the public dataset FashionIQ: its crowdsourced modification descriptions only consist of 1–2 short sentences, such as “change to off-white” and “extend the skirt length”, which are semantically sparse and lack core visual details, making them unable to support the operation of retrieval models. The framework integrates the large model LLaVA-1.6, which takes reference images and original descriptions as input to generate fine-grained descriptions; the average length of these new descriptions rises from 5.21 words to 63.98 words, marking a 12.28-fold expansion.

2.6. Visual Explainability in Deep Learning

Deep neural networks suffer from a black-box (opacity) flaw, as they cannot provide human-interpretable justifications for their outputs. In 2017, Selvaraju et al. [47] proposed Gradient-weighted Class Activation Mapping (Grad-CAM), which calculates weights using the gradients of class scores with respect to the final convolutional feature maps. This approach is compatible with any network architecture, and can generate coarse saliency maps that, after upsampling and overlaying, output heatmaps that mark the core regions driving a model’s decision-making. To the best of the authors’ knowledge, the framework proposed in this study is the first CIR system that incorporates Grad-CAM saliency maps and weighted fused query embeddings into the retrieval workflow.

2.7. Research Gaps and Motivation

A core research gap exists in the field of Composed Image Retrieval (CIR) systems: existing methods only support English, and all existing CIR systems entirely lack multilingual support. This is far from a mere technical limitation; it is also a barrier to user access. More than 1 billion Chinese users and over 200 million Urdu users cannot use these systems, which establishes the core rationale for carrying out the present study.

The second core research gap in the CIR field is the complete lack of visual interpretability in this domain. Methods such as Grad-CAM can be implemented to generate visual explanations for retrieval decisions, which offers notable value for consumer scenarios; however, none of the existing CIR frameworks have integrated this module.

A third hidden problem in fashion cross-modal retrieval benchmark datasets is semantic sparsity of modified descriptive texts: the crowdsourced short descriptions in FashionIQ average only 5.21 words, and universally omit key attributes such as neckline style, sleeve length, fabric texture, and silhouette. No prior studies have explored using large multimodal models to enrich such queries, which is the core innovation direction of this study.

Table 1. Comparative Summary of Related Works in CIR and Multimodal Retrieval.

Reference	Model/Method	Dataset	Key Contribution	Limitation
Vo et al. [1] (2019)	TIRG	Fashion200K, MIT-States	First end-to-end CIR model using gating mechanism	English-only, no explainability
Baldrati et al. [2] (2022)	CLIP4CIR	FashionIQ, CIRR	Leverages CLIP for zero-shot CIR	Monolingual, black-box
Wu et al. [4] (2021)	FashionIQ Dataset	FashionIQ	Standard CIR benchmark with fine-grained queries	English only
Gu et al. [5] (2023)	ARTEMIS	FashionIQ, CIRR	Attention-based relevance mechanism	No multilingual support
Radford et al. [11] (2021)	CLIP	WIT-400M	Large-scale vision-language pretraining	English-only text encoder
Wang et al. [24] (2022)	multilingual-E5	MIRACL, mMARCO	Cross-lingual text embeddings for retrieval	Not tested for CIR
Fan et al. [33] (2021)	M2M100	100-language pairs	Direct multilingual translation without English pivot	Translation quality varies
Liu et al. [39] (2023)	LLaVA	Visual Instruction Tuning	Multimodal instruction tuning with LLaMA	Not designed for retrieval
Selvaraju et al. [47] (2017)	Grad-CAM	ImageNet, VQA	Gradient-based visual saliency for any CNN	Post-hoc, not native to retrieval
Proposed Work	CLIP+E5+LLaVA+M2M100+Grad-CAM	FashionIQ (EN/UR/ZH)	Unified multilingual + explainable CIR with LMM query enrichment	—

Table 1. Comparative Summary of Related Works in CIR and Multimodal Retrieval.

Reference	Model/Method	Dataset	Key Contribution	Limitation
Vo et al. [1] (2019)	TIRG	Fashion200K, MIT-States	First end-to-end CIR model using gating mechanism	English-only, no explainability
Baldrati et al. [2] (2022)	CLIP4CIR	FashionIQ, CIRR	Leverages CLIP for zero-shot CIR	Monolingual, black-box
Wu et al. [4] (2021)	FashionIQ Dataset	FashionIQ	Standard CIR benchmark with fine-grained queries	English only
Gu et al. [5] (2023)	ARTEMIS	FashionIQ, CIRR	Attention-based relevance mechanism	No multilingual support
Radford et al. [11] (2021)	CLIP	WIT-400M	Large-scale vision-language pretraining	English-only text encoder
Wang et al. [24] (2022)	multilingual-E5	MIRACL, mMARCO	Cross-lingual text embeddings for retrieval	Not tested for CIR
Fan et al. [33] (2021)	M2M100	100-language pairs	Direct multilingual translation without English pivot	Translation quality varies
Liu et al. [39] (2023)	LLaVA	Visual Instruction Tuning	Multimodal instruction tuning with LLaMA	Not designed for retrieval
Selvaraju et al. [47] (2017)	Grad-CAM	ImageNet, VQA	Gradient-based visual saliency for any CNN	Post-hoc, not native to retrieval
Proposed Work	CLIP+E5+LLaVA+M2M100+Grad-CAM	FashionIQ (EN/UR/ZH)	Unified multilingual + explainable CIR with LMM query enrichment	—

3. Methodology

This study proposes a multilingual, interpretable Composed Image Retrieval (CIR) method tailored to the FashionIQ benchmark. This task requires users to input a candidate query image and a natural language modification description, and the system returns the target image that matches the stated modification from a large-scale image gallery. Existing CIR systems can only support English queries and cannot generalize across languages, which forms a critical limitation for real-world applications targeting global consumers.

This study proposes an integrated progressive pipeline multimodal system, whose core consists of five components: (1) a CLIP-based visual encoder that generates high-quality image embeddings; (2) the multilingual text encoder multilingual-E5-base, which supports generating semantically aligned embeddings for English, Urdu, and Chinese; (3) an automatic translation pipeline built using Facebook AI’s M2M100-418M to produce a trilingual caption dataset; (4) a learnable projection head that solves the cross-modal embedding space mismatch problem; (5) LLaVA-1.6, which converts sparse 5-word captions into 64-word detailed descriptions. The system is additionally equipped with a weighted fusion module with learnable scalars and a Grad-CAM interpretability module.

The technical framework proposed in this paper is built on three core design principles. First, it is modular: all its components can be independently trained, evaluated, and replaced, and it supports ablation studies with controlled variables. Second, it is efficient: pre-stored frozen model embeddings mean a single epoch of training on one consumer-grade GPU only takes 10 to 20 minutes. Third, it is scalable: its projection head is language-agnostic, and it currently supports Chinese, English, and Urdu.

Table 2. Component-Level Summary of the Proposed Framework.

Stage	Component	Input	Output	Role
1	CLIP ViT-B/32	RGB image (224×224)	512-dim embedding	Visual representation
2	M2M100-418M	English caption	UR / ZH caption	Multilingual translation
2	multilingual-E5-base	Text caption (any lang)	768-dim embedding	Multilingual text encoding
3	Projection Head (MLP)	768-dim text embedding	512-dim embedding	Cross-modal alignment
4	Weighted Fusion (α, β)	Image + text embeddings	512-dim query	Modality-adaptive fusion
5	LLaVA-1.6	Image + short caption	~64-word description	Query enrichment
—	Grad-CAM	Candidate image + query	Saliency heatmap	Visual explainability

Table 2. Component-Level Summary of the Proposed Framework.

Stage	Component	Input	Output	Role
1	CLIP ViT-B/32	RGB image (224×224)	512-dim embedding	Visual representation
2	M2M100-418M	English caption	UR / ZH caption	Multilingual translation
2	multilingual-E5-base	Text caption (any lang)	768-dim embedding	Multilingual text encoding
3	Projection Head (MLP)	768-dim text embedding	512-dim embedding	Cross-modal alignment
4	Weighted Fusion (α, β)	Image + text embeddings	512-dim query	Modality-adaptive fusion
5	LLaVA-1.6	Image + short caption	~64-word description	Query enrichment
—	Grad-CAM	Candidate image + query	Saliency heatmap	Visual explainability

3.1. Dataset: FashionIQ

All experiments conducted in this study adopt the publicly available FashionIQ dataset as the core benchmark. This dataset is purpose-built for Composed Image Retrieval (CIR) in the fashion domain, and covers three product categories: dress, shirt, and toptee. The FashionIQ dataset provides two crowdsourced natural language captions for each group of image triplets. These captions describe the required visual modifications to transform the reference image into the target image. The captions were collected via Amazon Mechanical Turk; they are colloquial and aligned with real user search behavior, but they have notable flaws: they are generally short, and their writing styles vary widely.

This study constructs a unified image gallery for the fashion image retrieval task by merging all unique target images from the training splits of the three categories, to simulate the real-world deployment scenario of users conducting cross-category searches. The generated unified gallery contains approximately 22,200 images.

Table 3. FashionIQ Dataset Statistics.

Category	Train Pairs	Val Pairs	Train Targets	Val Targets
Dress	11,466	1,497	≈7,860	≈2,032
Shirt	9,645	1,210	≈6,400	≈1,663
TopTee	11,817	1,477	≈7,940	≈2,095
Total	32,928	4,184	≈22,200	≈5,790

Table 3. FashionIQ Dataset Statistics.

Category	Train Pairs	Val Pairs	Train Targets	Val Targets
Dress	11,466	1,497	≈7,860	≈2,032
Shirt	9,645	1,210	≈6,400	≈1,663
TopTee	11,817	1,477	≈7,940	≈2,095
Total	32,928	4,184	≈22,200	≈5,790

This study expands the monolingual FashionIQ benchmark to multilingual scenarios by applying the M2M100-418M model to translate all captions from the original dataset’s training and validation splits, generating a parallel caption dataset covering English, Urdu, and Chinese. The dataset is stored in a unified JSON format supporting language-agnostic evaluation, allowing any of the three caption languages to be selected during inference without modifying any other part of the pipeline.

3.2. System Environment and Experimental Setup

All experiments are deployed on a hardware platform compatible with standard consumer-grade hardware. The training experiments adopt a core design of precomputed embedding cache: rather than recalculating CLIP image embeddings and multilingual-E5-base text embeddings during every training iteration, all embeddings are computed and stored in Google Drive before training officially commences. This strategy cuts the training duration for a single epoch from 5–6 hours to only 10–20 minutes.

Table 4. Experimental Environment Specifications.

Component	Specification
GPU	NVIDIA T4 (15 GB VRAM)
Platform	Google Colab
Deep Learning Framework	PyTorch 2.x
Transformer Library	Hugging Face Transformers 4.44.2
Text Embedding	Sentence-Transformers
Storage	Google Drive (persistent cache)
Programming Language	Python 3.10
Visual Backbone	openai/clip-vit-base-patch32
Text Encoder	intfloat/multilingual-e5-base
Translation Model	facebook/m2m100_418M
LMM for Enrichment	llava-hf/llava-v1.6-mistral-7b-hf

Table 4. Experimental Environment Specifications.

Component	Specification
GPU	NVIDIA T4 (15 GB VRAM)
Platform	Google Colab
Deep Learning Framework	PyTorch 2.x
Transformer Library	Hugging Face Transformers 4.44.2
Text Embedding	Sentence-Transformers
Storage	Google Drive (persistent cache)
Programming Language	Python 3.10
Visual Backbone	openai/clip-vit-base-patch32
Text Encoder	intfloat/multilingual-e5-base
Translation Model	facebook/m2m100_418M
LMM for Enrichment	llava-hf/llava-v1.6-mistral-7b-hf

Table 5. Hyperparameter Configuration for All Training Stages.

Hyperparameter	Projection Head (Step 2A)	Weighted Fusion (Step 3A)	Rich Caption Head (Sec M)
Optimizer	AdamW	AdamW	AdamW
Learning Rate	1e-4	5e-5	1e-4
Weight Decay	1e-4	1e-4	1e-4
Batch Size	32	32	64
Epochs	3	3	3
Dropout	0.1	0.1	0.1
Loss Function	1 - cosine sim	1 - cosine sim	1 - cosine sim
Best Val Loss	0.1608	0.1580	0.1716

Table 5. Hyperparameter Configuration for All Training Stages.

Hyperparameter	Projection Head (Step 2A)	Weighted Fusion (Step 3A)	Rich Caption Head (Sec M)
Optimizer	AdamW	AdamW	AdamW
Learning Rate	1e-4	5e-5	1e-4
Weight Decay	1e-4	1e-4	1e-4
Batch Size	32	32	64
Epochs	3	3	3
Dropout	0.1	0.1	0.1
Loss Function	1 - cosine sim	1 - cosine sim	1 - cosine sim
Best Val Loss	0.1608	0.1580	0.1716

3.3. Multilingual Caption Generation via M2M100

This study adopts the 418M-parameter variant of the M2M100 model developed by Facebook AI. This model supports direct mutual translation between one hundred languages, with no requirement to use English as a pivot language. In the experiments, beam search decoding with num_beams=4 is applied, with a generation length limit of max_new_tokens=64. After comparing multiple translation tools including Google Translate and the OPUS-MT series, M2M100 was selected because it is openly available for academic use, its performance in Urdu and Chinese has been validated, and it supports direct mutual translation between any pair of languages.

Before formally launching model training, the multilingual-E5-base sentence Transformer was used to verify the quality of translated captions generated by M2M100, using average cosine similarity and top-K cross-language recall rate across the full set of 5,657 validation samples.

3.4. CLIP Visual Encoder

This study adopts the CLIP variant openai/clip-vit-base-patch32 released by OpenAI. This model is pre-trained on 400 million image-text pairs, and outputs 512-dimensional normalized image and text embeddings. During training, all core parameters of the CLIP model are frozen; only the projection head and fusion weights are permitted to participate in the training process, to avoid catastrophic forgetting on the comparatively small FashionIQ training set of 32,928 triplets.

3.5. Multilingual Text Projection Head

The 768-dimensional text embeddings generated by the multilingual-E5-base text model and the 512-dimensional image embeddings output by the CLIP image encoder are mismatched in core attributes including dimensionality. A trainable projection head is introduced to bridge the gap between the two feature spaces. The projection head is a compact two-layer MLP with approximately 786,432 trainable parameters: a linear layer (768→512), ReLU activation, dropout (0.1), and a second linear layer (512→512), with the output L2-normalized to match CLIP’s unit-norm embedding convention.

Training data is drawn from 16,810 training-split triplets. A mixed-language sampling strategy randomly selects one of the three languages (English, Urdu, or Chinese) per sample at each training step, forcing the projection head to produce language-agnostic query representations. Training ran for 3 epochs using AdamW (lr=1e-4, weight_decay=1e-4). Best checkpoint: val_loss = 0.1608.

3.6. Weighted Fusion with Learnable Scalar Weights

The current mainstream training scheme for cross-modal projection heads applies symmetric equal-weight summation to image and text embeddings. This study introduces learnable scalars — the image weight alpha and the text weight beta — which dynamically adjust the weight proportion of the two modalities based on the level of detail in their paired captions. Both are initialized to 1.0, corresponding to the symmetric equal-weight baseline, and are jointly trained with the projection head weights during model fine-tuning.

3.7. LLaVA-Based Query Enrichment

The modified descriptions of the FashionIQ dataset average only 5.21 words, and omit core visual attributes, which limits the discriminative power of text query embeddings. This study uses LLaVA-1.6 to generate new descriptions that average 63.98 words, expanding information volume 12.28-fold. The multimodal large model adopted is llava-hf/llava-v1.6-mistral-7b-hf, with Mistral-7B as its base model, using 4-bit NF4 double quantization via BitsAndBytes to reduce VRAM demand from approximately 14 GB to around 4 GB. A two-stage workflow is adopted: LLaVA generates detailed English descriptions first, then M2M100-418M translates these into Urdu and Chinese.

3.8. Explainability via Grad-CAM

Gradient-weighted Class Activation Mapping (Grad-CAM) provides visual interpretability for image retrieval tasks. The gradients of the composite query-target similarity score with respect to the convolutional feature maps of candidate images are calculated to generate a spatial saliency heatmap that marks high-impact image regions. This heatmap is upsampled to match input resolution, overlaid on the candidate image, and displayed alongside the retrieved top-1 target image.

3.9. Evaluation Methodology

Recall@K (R@K) refers to the proportion of queries for which the correct target image appears in the top K retrieval results. In this study, K is set to 1, 5, and 10, with R@10 adopted as the primary comparative indicator. A six-configuration ablation study isolates the contribution of each proposed component: C1 (Image Only), C2 (CLIP Baseline, zero-shot), C3 (Multilingual, No Projection), C4 (Projection Head, Fixed Weights), C5 (Full Model, Short Captions), and C6 (Full Model + LMM Enrichment).

4. Experiments and Results

All experiments are run on the NVIDIA T4 GPU hosted on the Google Colab platform. The default evaluation adopts the FashionIQ validation set containing 5,657 samples, which remains unchanged unless otherwise specified. For the ablation study, 1,855 apparel samples equipped with both short captions and LLaVA-augmented data are used to guarantee fair comparison.

4.1. CLIP Zero-Shot Baseline

The standard combinatorial retrieval scheme that adopts frozen CLIP features and no task-specific training is designated Configuration C2, and serves as the core comparative baseline for all new research components added in subsequent work.

Table 6. CLIP Zero-Shot Baseline — English Recall@K (%).

Configuration	R@1 (%)	R@5 (%)	R@10 (%)
C2: CLIP Baseline	3.33	12.15	17.00

Table 6. CLIP Zero-Shot Baseline — English Recall@K (%).

Configuration	R@1 (%)	R@5 (%)	R@10 (%)
C2: CLIP Baseline	3.33	12.15	17.00

The R@10 score for English queries of the CLIP zero-shot baseline reaches 17.00%. For comparison, the corresponding R@10 score is 18.72% for Urdu and 19.79% for Chinese queries.

4.2. Multilingual Caption Quality Verification

Cross-language caption translation quality was verified during the pre-training phase using the M2M100 translation model and the multilingual-E5-base verification model, over the full validation set of 5,657 samples, using two metrics: average pairwise cosine similarity and cross-language retrieval recall rate.

Table 7. Cross-Lingual Semantic Alignment — Cosine Similarity and Recall@K. 

Language Pair	Avg Cosine Sim	R@1 (%)	R@5 (%)	R@10 (%)
EN → UR	0.8556	30.37	47.32	55.14
EN → ZH	0.8587	24.96	44.65	54.45
UR → EN	0.8294	22.77	38.02	45.38
ZH → EN	–	26.06	44.69	53.81

Table 7. Cross-Lingual Semantic Alignment — Cosine Similarity and Recall@K. 

Language Pair	Avg Cosine Sim	R@1 (%)	R@5 (%)	R@10 (%)
EN → UR	0.8556	30.37	47.32	55.14
EN → ZH	0.8587	24.96	44.65	54.45
UR → EN	0.8294	22.77	38.02	45.38
ZH → EN	–	26.06	44.69	53.81

Average cosine similarity of approximately 0.855 confirms that M2M100 translations faithfully preserve semantic content. R@10 = 55.14% for EN to UR and 54.45% for EN to ZH indicate the ground-truth translation ranks within the top 10 for more than half of all validation queries, validating the translation pipeline for multilingual CIR training.

4.3. Projection Head Training Results

The projection head completed three rounds of training using 16,810 training split triplets and mixed-language sampling. The optimal model checkpoint from the third round has a validation loss of 0.1608, which is 27.2% lower than the initial validation loss.

Table 8. Step 2A — Projection Head Recall@K by Query Language (%).

Query Language	R@1 (%)	R@5 (%)	R@10 (%)
English	0.88	3.82	5.43
Urdu	0.88	3.59	5.16
Chinese	1.01	3.55	5.41

Table 8. Step 2A — Projection Head Recall@K by Query Language (%).

Query Language	R@1 (%)	R@5 (%)	R@10 (%)
English	0.88	3.82	5.43
Urdu	0.88	3.59	5.16
Chinese	1.01	3.55	5.41

Using the 5,657 validation samples from the unified gallery, the mixed-language training projection head achieves comparable R@10 scores across English, Urdu, and Chinese (5.16–5.43%), in contrast with the CLIP baseline where English, Urdu, and Chinese differ (17.00%, 18.72%, and 19.79% respectively).

Table 9. Section M — Rich Caption Projection Head Recall@K by Query Language (%).

Query Language	R@1 (%)	R@5 (%)	R@10 (%)
English	0.94	3.86	5.43
Urdu	0.92	3.86	5.18
Chinese	0.92	3.75	5.24

Table 9. Section M — Rich Caption Projection Head Recall@K by Query Language (%).

Query Language	R@1 (%)	R@5 (%)	R@10 (%)
English	0.94	3.86	5.43
Urdu	0.92	3.86	5.18
Chinese	0.92	3.75	5.24

To address the distribution mismatch identified in the ablation discussion (Section 4.6), the projection head was retrained from scratch using LLaVA-augmented descriptions for 4,771 formal dress samples, producing a total cache of 12,303 embeddings, and converging to an optimal training loss of 0.1716. The rich-caption projection head attains R@10 scores highly consistent with the short-caption projection head, confirming that the mixed-language training strategy generalizes to longer descriptions.

4.4. Weighted Fusion Training and Evaluation

Fine-tuning experiments on the learnable weight parameters converged with alpha (image weight) reaching 0.9295 and beta (text weight) reaching 1.0685, indicating that descriptive text serves as the core directional signal, with the average text weight 14.9% higher than the image weight.

Table 10. Learned Fusion Weights — Initial vs Converged Values.

Parameter	Initial Value	Learned Value
alpha (image weight)	1.0000	0.9295
beta (text weight)	1.0000	1.0685

Table 10. Learned Fusion Weights — Initial vs Converged Values.

Parameter	Initial Value	Learned Value
alpha (image weight)	1.0000	0.9295
beta (text weight)	1.0000	1.0685

Table 11. Step 3A — Weighted Fusion Recall@K by Query Language (%).

Query Language	R@1 (%)	R@5 (%)	R@10 (%)
English	0.97	3.89	6.10
Urdu	0.88	3.64	5.69
Chinese	0.99	3.57	5.71

Table 11. Step 3A — Weighted Fusion Recall@K by Query Language (%).

Query Language	R@1 (%)	R@5 (%)	R@10 (%)
English	0.97	3.89	6.10
Urdu	0.88	3.64	5.69
Chinese	0.99	3.57	5.71

Weighted fusion consistently improves over the fixed-weight projection head across all languages, with English R@10 improving from 5.43% to 6.10% (a 12.3% relative gain), Urdu improving from 5.16% to 5.69% (+5.8% relative), and Chinese improving from 5.41% to 5.71%.

4.5. LLaVA Query Enrichment Summary

Table 12. LLaVA Query Enrichment Statistics.

Metric	Value
LLaVA Model	LLaVA-1.6 (Mistral-7B backbone)
Translation Model	facebook/m2m100_418M
Total Enriched Samples	4,771
Category	Dress
Avg. Short Caption Length	5.21 words
Avg. Rich Caption Length	63.98 words
Enrichment Factor	12.28x
Languages Generated	English, Urdu, Chinese

Table 12. LLaVA Query Enrichment Statistics.

Metric	Value
LLaVA Model	LLaVA-1.6 (Mistral-7B backbone)
Translation Model	facebook/m2m100_418M
Total Enriched Samples	4,771
Category	Dress
Avg. Short Caption Length	5.21 words
Avg. Rich Caption Length	63.98 words
Enrichment Factor	12.28x
Languages Generated	English, Urdu, Chinese

4.6. Ablation Study

Six test configurations were evaluated on 1,855 apparel category validation samples present in both the short-caption and LLaVA-augmented datasets, sharing a unified category gallery, across Recall@1, Recall@5, and Recall@10 for three query languages.

Table 13. Ablation Study — Complete Recall@10 (%) Summary.

Configuration	EN R@10	UR R@10	ZH R@10
C1: Image Only	1.89	1.89	1.89
C2: CLIP Baseline	3.72	1.89	1.99
C3: Multilingual No Proj	2.10	1.99	1.83
C4: Proj Head (α=β=1)	1.73	1.73	1.83
C5: Full Model (Short)	1.89	1.83	1.83
C6: Full Model + LMM (Rich)	1.35	1.19	1.46

Table 13. Ablation Study — Complete Recall@10 (%) Summary.

Configuration	EN R@10	UR R@10	ZH R@10
C1: Image Only	1.89	1.89	1.89
C2: CLIP Baseline	3.72	1.89	1.99
C3: Multilingual No Proj	2.10	1.99	1.83
C4: Proj Head (α=β=1)	1.73	1.73	1.83
C5: Full Model (Short)	1.89	1.83	1.83
C6: Full Model + LMM (Rich)	1.35	1.19	1.46

Finding 1 — Text modality contribution: the R@10 score of the image-only C1 scenario is 1.89%. The CLIP baseline C2 English scenario, which uses text-corrected captions, reaches an R@10 of 3.72% on this 1,855-sample subset.

Finding 2 — CLIP text encoder for non-English queries: for C2, the R@10 scores for English, Urdu, and Chinese on this subset are 3.72%, 1.89%, and 1.99% respectively, showing C2’s non-English performance close to the image-only baseline C1.

Finding 3 — Direct embedding addition without projection: the projection-free multilingual model C3 underperforms C1 for Chinese (1.83% vs 1.89%), indicating that naive concatenation of incompatible embedding spaces can be harmful.

Finding 4 — Projection head parity: configuration C4 registers R@10 scores ranging from 1.73% to 1.83% across English, Urdu, and Chinese, indicating balanced performance achieved through mixed-language training.

Finding 5 — Learned fusion weights: C5 improves over C4 for English (1.89% vs 1.73%, +9.2% relative) and Urdu (1.83% vs 1.73%, +5.8% relative).

Finding 6 — LLaVA enrichment and projection head retraining: within the E5 embedding space, the recall rate of configuration C6 is lower than C5 across all languages on this subset. The root cause is a distribution mismatch, since the projection head used for C5/C6 in this ablation was trained on 5-word short captions while C6’s inference receives 64-word long descriptions. The retraining experiment in Section 4.3 (Table 9), which retrained the projection head directly on LLaVA-augmented captions, shows that this mismatch is learnable and resolvable.

4.7. Explainability Results

An interpretability verification experiment was conducted using a shirt-category sample, candidate image ID B0059BG2SE, and the query instruction “brighten the image and add plaid patterns.” Grad-CAM heatmaps verified that the model identified the core attributes of fabric pattern, color, and texture, with a top-1 cosine similarity score of 0.9285.

Table 14. Explainability Demonstration Sample Details.

Field	Value
Category	Shirt
Candidate Image ID	B0059BG2SE
True Target ID	B0056FTFB4
Query Text (EN)	Brighter and checked
Retrieved Top-1 ID	B00DN6JHNE
Top-1 Similarity Score	0.9285
Input Language	English

Table 14. Explainability Demonstration Sample Details.

Field	Value
Category	Shirt
Candidate Image ID	B0059BG2SE
True Target ID	B0056FTFB4
Query Text (EN)	Brighter and checked
Retrieved Top-1 ID	B00DN6JHNE
Top-1 Similarity Score	0.9285
Input Language	English

4.8. Consolidated Results Summary

Table 15. Consolidated Results Summary — Key Metrics Across All Experimental Steps.

Step / Configuration	EN R@10	UR R@10	ZH R@10	Best Val Loss
C2: CLIP Baseline	17.00	18.72	19.79	–
Step 2A: Proj Head	5.43	5.16	5.41	0.1608
Step 2B: Proj Head (cat)	6.15	5.87	5.94	–
Step 3A: Weighted Fusion	6.10	5.69	5.71	0.1580
C5: Full Model (ablation)	1.89	1.83	1.83	–
Sec M: Rich Proj Head	5.43	5.18	5.24	0.1716

Table 15. Consolidated Results Summary — Key Metrics Across All Experimental Steps.

Step / Configuration	EN R@10	UR R@10	ZH R@10	Best Val Loss
C2: CLIP Baseline	17.00	18.72	19.79	–
Step 2A: Proj Head	5.43	5.16	5.41	0.1608
Step 2B: Proj Head (cat)	6.15	5.87	5.94	–
Step 3A: Weighted Fusion	6.10	5.69	5.71	0.1580
C5: Full Model (ablation)	1.89	1.83	1.83	–
Sec M: Rich Proj Head	5.43	5.18	5.24	0.1716

5. Discussion

The multimodal cross-lingual combinatorial image retrieval framework proposed in this study evaluates retrieval performance across English, Urdu, and Chinese. On the CLIP zero-shot baseline, R@10 for English reaches 17.00%, while Urdu and Chinese reach 18.72% and 19.79% respectively. The framework developed in this work trains a multilingual projection head via mixed-language sampling, mapping multilingual text embeddings to the CLIP visual space.

The R@10 of steps 2A, 2B, and 3A is in the 5–6% range on the large-scale unified retrieval gallery of 22,200 entries, reflecting the difficulty of retrieval at this gallery scale; in the ablation experiment conducted on a smaller category-restricted gallery of 1,855 entries, configuration C5 reached an R@10 of approximately 1.83–1.89%, while configuration C6 with LLaVA enrichment scored lower on that same subset. The Section M retraining experiment, which retrained the projection head directly on 4,771 LLaVA-augmented samples across the three languages, achieved a loss of 0.1716 (comparable to the 0.1608 loss of the short-caption head) with matching Recall@10 performance, indicating that the original gap between C5 and C6 stemmed from an input-length mismatch between the training and inference captions used in the ablation setting, rather than an inherent limitation of the rich-caption approach.

6. Conclusion and Future Work

6.1. Conclusion

Composed image retrieval is a highly challenging multimodal task that requires systems to integrate candidate images and natural language modification descriptions to retrieve matching target images from large-scale image repositories. This study proposes a multilingual interpretable framework and evaluates it on the FashionIQ benchmark. Existing CIR systems rely on English-centric CLIP encoders and typically show reduced adequacy for non-English queries; this framework targets that cross-lingual gap and evaluates retrieval for English, Urdu, and Chinese.

The core of the cross-lingual multimodal retrieval model proposed in this research adopts a two-layer MLP projection head, which maps the 768-dimensional text embeddings generated by multilingual-e5-base to the 512-dimensional visual embedding space of CLIP, to realize cross-lingual image-text semantic fusion. During training, English, Urdu, and Chinese captions are randomly sampled for each batch, and a weighted fusion module with an image weight α = 0.9295 and a text weight β = 1.0685 is introduced.

This study also integrates LLaVA-1.6 (Mistral-7B) for query augmentation. The original captions averaged 5.21 words, while the generated target descriptions averaged 63.98 words, reaching an augmentation factor of 12.28×. Six ablation configurations (C1–C6) were used to verify the value of each system component, and Grad-CAM saliency visualization confirmed that the system focuses on semantically associated image regions when processing modified queries, supporting the interpretability of its retrieval decisions.

Key findings of this research are summarized below:

• The CLIP text encoder records R@10 scores of 17.00% for English, 18.72% for Urdu, and 19.79% for Chinese queries on the zero-shot baseline.
• The multilingual projection head trained via mixed-language sampling achieves stable, comparable recall rates across all three languages within a single unified model.
• Stacking an untrained projection head with multilingual-e5-base and CLIP embeddings can impair model performance relative to an image-only baseline, confirming the necessity of a trained projection head to resolve spatial misalignment.
• The fusion weights obtained through training, α=0.9295 and β=1.0685, outperform equal fixed weights, indicating that fusion weights benefit from training rather than a fixed symmetric assumption.
• The query descriptions generated by LLaVA-1.6 are 12.28 times the length of the original captions; results show the projection head must be retrained on rich-description embeddings to avoid distribution mismatch between training and inference caption lengths.
• Grad-CAM explainable visual verification confirms that when processing modified queries, the fashion retrieval system focuses on semantically related visual regions — pattern, color, and fabric — which can support the establishment of user trust.

6.2. Limitations

This study has four core limitations: first, the sample size for model training is limited, and training epochs are constrained by available compute, which affects the stability of the model’s performance; second, the dataset used in this study is sourced from a single domain, and the model’s cross-scenario generalization ability has not been verified; third, the interpretability assessment only covers the feature weight dimension, while other dimensions are not included in the evaluation; fourth, translation quality is verified via automatic metrics without supplementary human evaluation, leaving this gap to be addressed in future research.

6.3. Future Work

Future work includes: (1) full-scale LMM query enrichment across the complete FashionIQ training and validation sets, with projection head retraining on the resulting rich-caption embeddings; (2) expansion to additional languages such as Arabic, Hindi, French, and Spanish, leveraging the low-cost extensibility of multilingual-e5-base and M2M100; (3) fine-tuning the multilingual text encoder jointly with the projection head under a carefully controlled learning rate; (4) advanced fusion architectures such as cross-attention between image-text embeddings and dynamic, input-conditioned weight prediction; (5) systematic, quantitative explainability evaluation paired with standardized test sets and human-subject usability studies; (6) extension to other CIR benchmarks such as CIRR and Fashion200K to verify cross-domain generalization; and (7) deployment as a web-based multilingual fashion search interface supporting native-language queries, enabling collection of real-world user feedback for iterative optimization.