Vision–Language Foundation Models and Multimodal Large Language Models: A Comprehensive Survey of Architectures, Benchmarks, and Open Challenges

1. Introduction

With the rapid advancement of artificial intelligence and machine learning, multimodal fusion techniques and vision-language models (VLMs) have emerged as critical components driving innovation across diverse sectors. By integrating visual and linguistic modalities, these models enable richer semantic understanding, enhanced interaction, and improved automation in complex tasks. In the healthcare domain, VLMs and Multimodal Large Language Models (MLLMs) have been utilized to interpret medical imagery such as X-rays, CT, and MRI scans in conjunction with radiology reports, supporting automated diagnosis and clinical decision-making [1]. In agriculture, multimodal fusion approaches are applied to precision farming tasks including crop health monitoring, disease detection, and yield estimation by combining visual crop data with textual or sensor-based contextual information [2]. In the retail and manufacturing industries, VLMs facilitate visual product search, automated tagging, and defect detection by aligning product images with descriptive textual data, improving quality control and recommendation systems [3]. In education, these models enhance multimodal learning environments by generating visual explanations and captions that aid accessibility and support learners with visual impairments [4]. Furthermore, in robotics, vision-language and vision-language-action models empower autonomous systems to jointly process visual scenes and natural language instructions, thereby advancing navigation, manipulation, and human-robot interaction [5]. Collectively, these developments demonstrate that VLMs and MLLMs are not only transforming traditional visual understanding tasks but are also establishing the foundation for cross-domain, context-aware intelligent systems that bridge perception, reasoning, and interaction in real-world environments.

Building on this foundation, the exponential rise of vision-language models (VLMs) and multimodal large language models (MLLMs) has significantly transformed the paradigm of multimodal fusion in recent years. Large-scale pretrained VLMs such as CLIP [6], ALIGN [7], and BLIP-2 [8] possess remarkable cross-modal alignment and generalization capabilities, enabling robust performance in zero-shot image classification and retrieval tasks [6,7]. These models further demonstrate strong potential in instruction-following scenarios, where natural language prompts are seamlessly mapped to visual tasks through multimodal reasoning [8,9]. Similarly, in visual question answering (VQA), transformer-based architectures such as LXMERT [10] and ViLBERT [11] have achieved state-of-the-art results in understanding fine-grained relationships between vision and language [10,11]. This evolution signifies a paradigm shift in robotic vision systems from passive perception toward proactive, semantically aware, and linguistically interactive agents capable of understanding and reasoning about their environment [12,13].

Despite these advancements, several practical challenges persist in deploying multimodal fusion for robotic applications. First, effectively integrating heterogeneous data across visual, textual, and sensory modalities remains a fundamental obstacle, particularly regarding modality alignment, unified feature representation, and spatiotemporal synchronization [14,15]. Second, robotic systems impose stringent constraints on real-time processing and computational efficiency, demanding lightweight yet accurate fusion architectures that balance inference speed and performance [15,16]. Third, although pretrained VLMs exhibit strong generalization capabilities, their adaptability to task-specific robotic environments such as dynamic scene understanding, manipulation, and embodied reasoning remains limited [17]. Addressing these challenges necessitates future research focusing on self-supervised multimodal pretraining, adaptive attention mechanisms for efficient fusion, and domain-specific fine-tuning strategies to enhance robotic perception and interaction in real-world contexts [18,19].

2. Background / Theoretical Foundation

2.1. What Is Multimodality?

The concept of multimodality fundamentally refers to the integration and processing of multiple types of data or semiotic resources, known as modalities, to communicate or process information [20,21]. In the domain of communication and semiotics, Multimodal Discourse is defined as "the combination of different semiotic modes for example, language and music in a communicative artifact or event" [22]. Human communication is inherently multimodal, involving not only language but also other modes such as gesture, gaze, and facial expression. A mode itself is characterized as "a socially and culturally given semiotic resource for making meaning" [20]. Historically, fields like academia favored monomodality (text-only documents), but this has reversed with the rise of modern media and digital tools that incorporate color illustrations, sophisticated layout, and typography. In the realm of Artificial Intelligence (AI), multimodality specifically refers to systems that integrate and process diverse data streams, such as text, audio, images, or video, to achieve a more holistic understanding of complex inputs [21,23]. Multimodal models, especially large multimodal models (MLLMs), enhance AI capabilities by integrating visual and textual data, mimicking human learning processes. These systems gain richer context and better reasoning skills by combining different forms of information, allowing them to perform complex tasks like Visual Question Answering (VQA), image captioning, and visual dialogue [21,23]. For instance, a multimodal AI system must accurately and efficiently manage different types of information, such as finding relevant images based on a text query or explaining an image’s content in natural language. This approach is crucial in applications like robotics, where machines combine inputs from cameras (vision), microphones (sound), and force sensors (touch) to interact effectively with the environment [24]. Recently we also worked on similar interests like [25,26]. We have also worked on using Artificial Intelligence and Multi-Modality in storytelling too [27,28]

2.2. Different Types of Fusion

Multimodal fusion techniques aim to combine data from multiple sources or modalities to generate more accurate and insightful representations [29]. The two fundamental strategies are Early Fusion and Late Fusion, differentiated by the stage at which data integration occurs [30,31]. Early Fusion, also referred to as feature-level fusion [29,30], combines raw data or low-level features from different modalities into a single feature set before inputting them into a single machine learning model [29,31]. This approach captures intricate relationships between modalities and yields rich feature representations [29,31,32]. However, early fusion can result in high-dimensional feature spaces, inflexibility, and significant challenges when dealing with heterogeneous or asynchronous data [29,31,32]. In contrast, Late Fusion, or decision-level fusion, processes each modality independently using separate models, combining the final predictions or outputs only at the decision stage, often using techniques like averaging or voting [29,30,31]. This method offers modularity and avoids the high dimensionality associated with early fusion [29,31], but it risks missing critical cross-modal interactions that are crucial for complex tasks, as the input modalities are processed separately [29,32]. In the context of vision-language transformers, early fusion may be related to Merged Attention, where unimodal representations are simply concatenated along the sequence dimension [33]. Hybrid Fusion (sometimes called intermediate fusion) combines aspects of both early and late strategies to mitigate their trade-offs [30,31]. Hybrid methods often apply feature-level fusion to modalities that are synchronous in time while using decision-level fusion for the remaining asynchronous modalities [30]. For example, intermediate fusion often utilizes cross-attention layers to dynamically integrate modality-specific representations [34]. Modern examples of advanced fusion include Progressive Fusion, which utilizes backward connections to feed late-stage fused representations back to the early layers of the unimodal feature generators, allowing progressive refinement and bridging the gap between early and late fusion advantages [32]. Another technique is Compound Tokens, which is generated via channel fusion concatenating the output of a cross-attention layer with the original query tokens along the feature (channel) dimension thus avoiding increased token length while benefiting from cross-attention [33]. Similarly, Depth-Breadth Fusion (DBFusion) is a novel feature-fusion architecture that concatenates visual features extracted from different depths (layers) and breadths (prompts) along the channel dimension, serving as a simple yet effective strategy [35].

2.3. Architecture Overview

3. General Transformer Architectures (LLMs)

These models primarily use variants of the original Transformer architecture [36].

3.1. Encoder–Decoder Architecture

This structure processes inputs through an encoder and feeds the representation to a decoder for output generation [37]. The encoder uses self-attention across the full input sequence, while the decoder uses cross-attention and generates tokens sequentially [37].

T5 [38] is a prime example, using a unified text-to-text paradigm for all NLP tasks [38]. Another example is AlexaTM [39]. Some research suggests encoder–decoder models may be advantageous, though scaling decoder-only models can close this performance gap [37].

3.2. Causal Decoder Architecture

This architecture, often used for Natural Language Generation (NLG), lacks an encoder and relies solely on a decoder for output [37]. It employs causal attention, where the prediction of a token depends only on previous time steps [37]. Examples include PaLM [40], GPT-3 [41], BLOOM [42], and LLaMA [43].

3.3. Prefix Decoder Architecture (Non-Causal Decoder)

In this variant, the attention calculation is bidirectional and not strictly dependent only on prior context [37]. An example is U-PaLM, which is described as a non-causal decoder model [44].

3.4. Mixture-of-Experts (MoE)

This is an efficient sparse variation of the Transformer [37]. It incorporates parallel independent experts (typically feed-forward layers) and a router that directs tokens to specific experts [37].

Mathematically, the MoE layer can be expressed as:

$$y=\sum _{i=1}^N{g}_i\left(x\right)·E_i\left(x\right)$$

where: - $E_i\left(x\right)$ denotes the output of the $i^{th}$ expert, - $g_i\left(x\right)$ represents the gating function (probability of routing to each expert), - and only a sparse subset of experts is activated per input.

MoE architectures, like PanGu-Σ (1.085 trillion parameters), allow for massive model scaling without proportionate increases in computational cost, as only a fraction of experts are activated per input [37,45].

4. Multimodal Architectures (VLMs)

Vision-Language Models (VLMs), or Multimodal LLMs (MLLMs), combine vision encoders and LLMs to handle both image and text inputs [46]. They are primarily differentiated by their mechanism for multimodal fusion [46].

4.1. Classification by Fusion Mechanism

4.1.1. Dual Encoder Architectures

These models process modalities independently using dedicated encoders before interaction occurs. Fusion often happens via similarity comparison between global feature vectors [46].

CLIP (Contrastive Language–Image Pretraining) is a dual encoder that uses a contrastive objective [46,47], where fusion is achieved via the dot product between global image and text embeddings [47]:

$$s(I,T)={\displaystyle \frac{f_I\left(I\right)·f_T\left(T\right)}{\parallel f_I\left(I\right)\parallel \parallel f_T\left(T\right)\parallel }}$$

where $f_I$ and $f_T$ are the image and text encoders, respectively.

4.1.2. Fusion Encoders (Single-Stream)

These architectures perform multimodal interaction early by directly concatenating or summing image and text embeddings and feeding them into shared Transformer layers [48]. Examples include UniTER [48].

4.1.3. Hybrid Methods

These models mix aspects of dual and fusion encoders. ViLBERT (Vision-and-Language BERT) uses a co-attention Transformer with two parallel streams that interact through cross-attentional layers [49].

CoCa (Contrastive Captioner) employs a minimalist encoder–decoder design pretrained jointly with a contrastive loss (as in CLIP) and a generative captioning loss [50]. The decoder layers omit cross-attention in the first half to maintain unimodal text representation before later layers cross-attend to the image encoder for multimodal representations [50].

4.2. Cross-Modal Interaction Mechanisms (Attention Variants)

Multimodal Transformers use various methods to integrate inputs $X_A$ and $X_B$ across modalities, represented by token embeddings $Z^{\left(A\right)}$ and $Z^{\left(B\right)}$ [51].

4.2.1. Early Summation

Weighted embeddings from different modalities are summed before entering the Transformer layers $Tf(·)$:

$$Z=\alpha Z^{\left(A\right)}\oplus \beta Z^{\left(B\right)}$$

[51]

4.2.2. Early Concatenation

Token embedding sequences are concatenated and processed by subsequent Transformer layers:

$$Z=C(Z^{\left(A\right)},Z^{\left(B\right)})$$

[51]

4.2.3. Cross-Attention (Co-Attention)

Used mainly in multi-stream or hybrid models, where queries from one modality attend to keys and values from another modality:

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V$$

[51,52]

4.2.4. Hierarchical Attention (Multi-Stream to One-Stream)

A late fusion method where independent Transformer streams encode inputs $T{f}_1,T{f}_2$, and their outputs are concatenated and fused by another Transformer:

$$Z=T{f}_3\left([T{f}_1\left(X_A\right);T{f}_2\left(X_B\right)]\right)$$

[51]

4.2.5. Hierarchical Attention (One-Stream to Multi-Stream)

An early interaction method where concatenated inputs pass through a shared Transformer before splitting into separate streams [53].

4.2.6. Cross-Attention to Concatenation

Combines outputs from cross-attention streams, concatenates them, and processes with a final Transformer layer:

$$Z=Tf\left([Z_{CA}^{\left(A\right)};Z_{CA}^{\left(B\right)}]\right)$$

[51]

5. Specific Advanced VLM Architectures

5.1. Flamingo Architecture

Flamingo is designed for visually conditioned autoregressive text generation, capable of handling interleaved images/videos and text prompts [54]. It uses a Vision Encoder, a Perceiver Resampler (to produce fixed visual tokens), and pre-trained, frozen Language Model blocks interleaved with gated cross-attention dense blocks [54]. The cross-attention layers are trained from scratch and attend to the Perceiver Resampler outputs [54].

5.2. LLaVA Architecture

LLaVA models (e.g., LLaVA-v1.5 [55]) efficiently connect pre-trained LLMs (like Vicuna) and visual encoders (like CLIP) via a linear projection layer that maps image features into the LLM word embedding space [55].

6. Multimodal Datasets and Benchmarks

6.1. General and Comprehensive Multimodal Language Model (MLLM) Benchmarks

These benchmarks are designed to evaluate the broad perception, cognition, and integrated capabilities of MLLMs.

Table 1. Overview of General and Comprehensive MLLM Benchmarks.

Benchmark Name	Citation	Key Details & Data Sources
MMBench	[56]	A novel multi-modality benchmark utilizing a meticulously curated dataset and the CircularEval strategy with ChatGPT for robust evaluation.
MME	[57,58,59]	Measures both perception and cognition abilities across subtasks. It uses the MSCOCO dataset.
MM-Vet	[60]	Devised to study integrated vision-language capabilities, offering insights beyond overall model rankings. It covers 200 items in total.
SEED-Bench	[61]	A comprehensive benchmark featuring multiple-choice questions covering various evaluation dimensions for both image and video modalities.
SEED-Bench-2	[62]	Categorized MLLMs’ capabilities into hierarchical levels from L0 to L4.
SEED-Bench-H	[62]	A comprehensive integration of previous SEED-Bench series (SEED-Bench, SEED-Bench-2, SEED-Bench-2-Plus) with 28,000 multiple-choice questions spanning 34 dimensions.
LLaVA-Bench	[62]	Constructed to examine a variety of MLLM capabilities.
LAMM	[63]	Provides a comprehensive assessment of MLLMs’ capabilities, particularly in understanding visual prompting instructions.
MDVP-Bench	[64]	Created to provide a comprehensive assessment of MLLMs’ capabilities, particularly in understanding visual prompting instructions.
ChEF	[65]	Constructed as a standardized and holistic evaluation framework.
UniBench	[66]	Constructed as a standardized and holistic evaluation framework.
TouchStone	[67]	Proposed to support open-ended answers, although its small scale introduces instability.
Open-VQA	[68]	Proposed to support open-ended answers.
VLUE	[69,70]	The first multi-task benchmark focusing on vision-language understanding, covering image-text retrieval, visual question answering, visual reasoning, and visual grounding, and includes a newly annotated private out-of-distribution (OOD) test set using images from MaRVL.

Table 1. Overview of General and Comprehensive MLLM Benchmarks.

Benchmark Name	Citation	Key Details & Data Sources
MMBench	[56]	A novel multi-modality benchmark utilizing a meticulously curated dataset and the CircularEval strategy with ChatGPT for robust evaluation.
MME	[57,58,59]	Measures both perception and cognition abilities across subtasks. It uses the MSCOCO dataset.
MM-Vet	[60]	Devised to study integrated vision-language capabilities, offering insights beyond overall model rankings. It covers 200 items in total.
SEED-Bench	[61]	A comprehensive benchmark featuring multiple-choice questions covering various evaluation dimensions for both image and video modalities.
SEED-Bench-2	[62]	Categorized MLLMs’ capabilities into hierarchical levels from L0 to L4.
SEED-Bench-H	[62]	A comprehensive integration of previous SEED-Bench series (SEED-Bench, SEED-Bench-2, SEED-Bench-2-Plus) with 28,000 multiple-choice questions spanning 34 dimensions.
LLaVA-Bench	[62]	Constructed to examine a variety of MLLM capabilities.
LAMM	[63]	Provides a comprehensive assessment of MLLMs’ capabilities, particularly in understanding visual prompting instructions.
MDVP-Bench	[64]	Created to provide a comprehensive assessment of MLLMs’ capabilities, particularly in understanding visual prompting instructions.
ChEF	[65]	Constructed as a standardized and holistic evaluation framework.
UniBench	[66]	Constructed as a standardized and holistic evaluation framework.
TouchStone	[67]	Proposed to support open-ended answers, although its small scale introduces instability.
Open-VQA	[68]	Proposed to support open-ended answers.
VLUE	[69,70]	The first multi-task benchmark focusing on vision-language understanding, covering image-text retrieval, visual question answering, visual reasoning, and visual grounding, and includes a newly annotated private out-of-distribution (OOD) test set using images from MaRVL.

6.2. Hallucination Evaluation Benchmarks

These benchmarks specifically target assessing hallucinations in Image-to-Text (I2T) and Text-to-Image (T2I) generation tasks.

A. I2T (Image-to-Text) Hallucination Benchmarks

Table 2. Overview of I2T (Image-to-Text) Hallucination Benchmarks.

Benchmark Name	Citation	Key Details & Data Sources
POPE	[71]	Discriminative task benchmark using MSCOCO [59]. Targets faithfulness hallucinations, specifically object hallucinations.
HallusionBench	[72]	Discriminative benchmark sourced from a website [72], targeting both faithfulness and factuality.
CHAIR	[73]	Generative task benchmark focusing on object hallucinations in image captioning, sourced from MSCOCO [59].
AMBER	[74,75]	Comprehensive, LLM-free multi-dimensional benchmark evaluating object existence, attributes, and relations using manually collected images.
MERLIM	[76]	Evaluates existence, relation, and counting hallucinations using edited and original images from MSCOCO [59].
HaELM	[77]	First benchmark to utilize LLMs for hallucination evaluation within MLLMs, sourced from MSCOCO [59].
R-Bench	[78]	Discriminative benchmark evaluating relationship hallucinations, using MSCOCO [59].
Hal-Eval	[79]	Comprehensive benchmark including both in-domain (MSCOCO [59]) and out-of-domain datasets to assess potential data leakage.
VHtest	[80]	Uses MSCOCO [59] and DALL-E-3 generated data to construct synthetic datasets.
LongHalQA	[81]	Discriminative benchmark using Visual Genome [82] and Object365 [83].
PhD	[84]	Discriminative benchmark using TDIUC [85] to evaluate faithfulness and factuality.
HallucinaGen	[86]	Generative benchmark using MSCOCO [59] and NIH Chest X-ray [87].
FactCheXcker	[88]	Pipeline detecting object and measurement hallucinations in radiology reports, leveraging the MIMIC-CXR dataset.
NOPE	[89]	Generative benchmark sourced from OpenImages [90].
CIEM	[91]	Discriminative benchmark leveraging LLMs for automated question generation, sourced from MSCOCO [59].
RAH-Bench	[92]	Discriminative benchmark leveraging LLMs for automated question generation, sourced from MSCOCO [59].
ROPE	[93]	Discriminative benchmark using MSCOCO [59] and ADE20K [94].
VisDiaHalBench	[95]	Discriminative benchmark sourced from GQA [96].
CC-Eval	[97]	Generative benchmark sourced from Visual Genome [82].
GAVIE	[98]	Generative benchmark sourced from Visual Genome [82].
MMHal-Bench	[99]	Generative benchmark sourced from OpenImages [90].
FGHE	[100]	Discriminative benchmark sourced from MSCOCO [59].
VHILT	[101]	Generative task benchmark sourced from a website.
Med-HallMark	[102]	Comprehensive medical benchmark sourced from Slake [103] and others.
AutoHallusion	[104]	Discriminative benchmark establishing automated pipelines, sourced from MSCOCO [59] and DALL-E-2 [105].

Table 2. Overview of I2T (Image-to-Text) Hallucination Benchmarks.

Benchmark Name	Citation	Key Details & Data Sources
POPE	[71]	Discriminative task benchmark using MSCOCO [59]. Targets faithfulness hallucinations, specifically object hallucinations.
HallusionBench	[72]	Discriminative benchmark sourced from a website [72], targeting both faithfulness and factuality.
CHAIR	[73]	Generative task benchmark focusing on object hallucinations in image captioning, sourced from MSCOCO [59].
AMBER	[74,75]	Comprehensive, LLM-free multi-dimensional benchmark evaluating object existence, attributes, and relations using manually collected images.
MERLIM	[76]	Evaluates existence, relation, and counting hallucinations using edited and original images from MSCOCO [59].
HaELM	[77]	First benchmark to utilize LLMs for hallucination evaluation within MLLMs, sourced from MSCOCO [59].
R-Bench	[78]	Discriminative benchmark evaluating relationship hallucinations, using MSCOCO [59].
Hal-Eval	[79]	Comprehensive benchmark including both in-domain (MSCOCO [59]) and out-of-domain datasets to assess potential data leakage.
VHtest	[80]	Uses MSCOCO [59] and DALL-E-3 generated data to construct synthetic datasets.
LongHalQA	[81]	Discriminative benchmark using Visual Genome [82] and Object365 [83].
PhD	[84]	Discriminative benchmark using TDIUC [85] to evaluate faithfulness and factuality.
HallucinaGen	[86]	Generative benchmark using MSCOCO [59] and NIH Chest X-ray [87].
FactCheXcker	[88]	Pipeline detecting object and measurement hallucinations in radiology reports, leveraging the MIMIC-CXR dataset.
NOPE	[89]	Generative benchmark sourced from OpenImages [90].
CIEM	[91]	Discriminative benchmark leveraging LLMs for automated question generation, sourced from MSCOCO [59].
RAH-Bench	[92]	Discriminative benchmark leveraging LLMs for automated question generation, sourced from MSCOCO [59].
ROPE	[93]	Discriminative benchmark using MSCOCO [59] and ADE20K [94].
VisDiaHalBench	[95]	Discriminative benchmark sourced from GQA [96].
CC-Eval	[97]	Generative benchmark sourced from Visual Genome [82].
GAVIE	[98]	Generative benchmark sourced from Visual Genome [82].
MMHal-Bench	[99]	Generative benchmark sourced from OpenImages [90].
FGHE	[100]	Discriminative benchmark sourced from MSCOCO [59].
VHILT	[101]	Generative task benchmark sourced from a website.
Med-HallMark	[102]	Comprehensive medical benchmark sourced from Slake [103] and others.
AutoHallusion	[104]	Discriminative benchmark establishing automated pipelines, sourced from MSCOCO [59] and DALL-E-2 [105].

B. T2I (Text-to-Image) Hallucination Benchmarks

Benchmark Name	Citation	Key Details & Data Sources
TIFA v1.0	[106]	Generative task benchmark sourced from MSCOCO [59].
T2I-FactualBench	[107]	Generative task benchmark evaluating factuality hallucinations, sourced from GPT.
T2I-CompBench	[108]	A comprehensive open-world benchmark for evaluating compositional T2I generation, sourced from MSCOCO [59], Template, and GPT.
WISE	[109]	Designed to evaluate factuality hallucinations through complex prompts across natural sciences, spatiotemporal reasoning, and cultural knowledge, sourced from LLM-Constructed data.
SR 2D	[110]	Generative task benchmark sourced from MSCOCO [59].
DrawBench	[111]	Generative task benchmark involving human evaluation, sourced from Human and DALL-E [105].
ABC-6K & CC-500	[112]	Generative task benchmark sourced from MSCOCO [59].
PaintSkills	[113]	Generative task benchmark sourced from Template.
HRS-Bench	[114]	Generative task benchmark sourced from GPT.
GenAI-Bench	[115]	Generative task benchmark sourced from Human input.
I-HallA v1.0	[116]	Generative task benchmark focusing on factuality hallucinations, sourced from Textbook data.
OpenCHAIR	[117]	Generative task benchmark using Stable Diffusion.
ODE	[118]	Comprehensive benchmark utilizing Stable Diffusion to construct synthetic datasets.

6.3. Domain-Specific and Focused Benchmarks

These benchmarks evaluate capabilities in specialized fields (e.g., medical, finance, robotics) or focused tasks (e.g., visual reasoning, long context).

A. Expert-Level and Reasoning Benchmarks

Benchmark Name	Citation	Key Details & Data Sources
MMMU	[119,119]	Massive Multi-discipline Multimodal Understanding and Reasoning benchmark, featuring 11.5K college-level questions across 6 disciplines, sourced from Textbooks and the Internet.
MMMU-Pro	[119]	A more robust version of the MMMU benchmark, introduced in September 2024.
MathVista	[120]	Evaluates mathematical reasoning in visual contexts, limited exclusively to the mathematical domain.
SCIENCEQA	[121]	Assesses multimodal reasoning via thought chains for science question answering.
GAIA	[122]	A benchmark testing fundamental abilities such as reasoning, multimodality handling, or tool use.
Visual CoT	[123]	Constructed with visual chain-of-thought prompts, requiring comprehensive recognition and understanding of image text content.
MMStar	[124]	A vision-indispensable benchmark covering a wide range of tasks and difficulty levels.
CLEVR	[125]	A diagnostic dataset for compositional language and elementary visual reasoning, relying on synthetic images.

B. Medical and Healthcare Benchmarks

Benchmark Name	Citation	Key Details & Data Sources
CARES	[126]	A benchmark for evaluating the trustworthiness of medical vision-language models (Med-LVLMs) across five dimensions (trustfulness, fairness, safety, privacy, robustness).
OmniMedVQA	[127]	A large-scale comprehensive evaluation benchmark for medical LVLM, collected from 73 different medical datasets and 12 modalities, used as a source for CARES.
MIMIC-CXR	[128]	A large publicly available database of labeled chest radiographs. Used to construct CARES.
IU-Xray	[129]	A dataset including chest X-ray images and corresponding diagnostic reports, used to construct CARES.
Harvard-FairVLMed	[130]	Focuses on fairness in multimodal fundus images, used to construct CARES.
PMC-OA	[131,132]	Contains biomedical images extracted from open-access publications, used to construct CARES.
HAM10000	[133]	A dataset of dermatoscopic images of skin lesions for classification, used to construct CARES.
OL3I	[134]	A multimodal dataset for opportunistic CT prediction of ischemic heart disease (IHD), used to construct CARES.
VQA-RAD	[135]	An early-released VQA dataset, generally avoided in new medical benchmarks like CARES to prevent data leakage.
SLAKE	[103]	A semantically-labeled knowledge-enhanced dataset for medical VQA, generally avoided in new medical benchmarks like CARES to prevent data leakage.

C. Long Context and Document Understanding Benchmarks

Benchmark Name	Citation	Key Details & Data Sources
Document Haystack	[136]	A novel benchmark evaluating VLMs’ ability to retrieve key multimodal information from long, visually complex documents (5 to 200 pages).
MM-NIAH (Multimodal Needle in a Haystack)	[137]	Benchmarking long-context capability, although its prompt length limitations make it less suitable for very long documents.
M-LongDoc	[138]	Benchmark for multimodal super-long document understanding, featuring documents spanning hundreds of pages.
Needle in a Haystack	[139]	Tests models’ ability to retrieve information (the "needle") embedded within an extended context window (the "haystack").
LongBench	[140]	The first bilingual, multi-task framework for assessing long-form text understanding.
MileBench	[141]	Benchmarking MLLMs in long context.
DUDE	[142]	Document Understanding Dataset and Evaluation benchmark, attempting to tackle multi-page document comprehension.
Loong		Benchmark dealing with extended multi-document question answering.
SlideVQA	[143]	A dataset for document visual question answering on multiple images.
MMLongBench-Doc	[144]	Benchmarking long-context document understanding with visualizations.

D. Specialized Datasets/Benchmarks (Perception, Retrieval, etc.)

Dataset/Benchmark Name	Citation	Key Details & Data Sources
MS COCO (Common Objects in Context)	[59]	Widely used dataset (330,000+ images) for object detection, segmentation, VQA, and captioning.
Visual Genome	[82]	Provides dense annotations (3.8M objects, 2.3M relationships) to bridge images and language, enabling reasoning tasks.
Flickr30K Entities	[145]	Extends Flickr30K with bounding box annotations and coreference chains for phrase grounding.
ImageBind (Meta AI)	[146]	Large-scale dataset linking images with six modalities (text, audio, depth, thermal, IMU) for unified multimodal embeddings.
LAION-5B	[147]	One of the largest open multimodal datasets (5.85 billion image-text pairs) for training foundation models.
Conceptual Captions (CC3M)	[148]	Contains ∼3.3 million image-caption pairs extracted and filtered from the web, designed for automatic image captioning.
VizWiz	[149]	Benchmark consisting of visual questions originating from blind people.
GQA	[96]	Developed to address the limitations of VQAv2, offering rich semantic and visual complexity for real-world visual reasoning.
VQAv2	[150]	A benchmark using pairs of similar images leading to different answers to compel models to prioritize visual data.
OCRBench	[151]	Focuses on Optical Character Recognition tasks.
TallyQA	(Contextual citation)	A Visual Question Answering dataset specifically designed to address counting questions in images.
RF100-VL (Roboflow100-VL)	[152]	Large-scale multimodal benchmark evaluating VLMs on out-of-distribution object detection, covering seven domains.
NLVR	[153]	A corpus for reasoning about natural language grounded in photographs (NLVR2 is the related task in VLUE [69]).
Massive Multitask Language Understanding (MMLU)		Crucial benchmark for evaluating general knowledge and reasoning across 57 diverse subjects.

6.4. Other Modalities (Video, Audio, 3D)

Dataset/Benchmark Name	Citation	Key Details & Data Sources
MVBench	[154]	A comprehensive multi-modal video understanding benchmark focusing on temporal perception.
Perception Test	[155]	A diagnostic benchmark for multimodal video models, covering Memory, Abstraction, Physics, and Semantics.
MSR VTT	[156]	A large video captioning dataset (10,000 video clips, 200,000 clip–sentence pairs) bridging video content and natural language.
VaTeX (Video And Text)	[157]	A multilingual video captioning dataset (English and Chinese) with 41,250 videos and 825,000 captions.
Dynamic-SUPERB	[158]	A benchmark assessing MLLMs’ ability to follow instructions in the audio domain, focusing on human speech processing.
AIR-Bench	[159]	A comprehensive benchmark designed to evaluate MLLMs’ ability to comprehend various audio signals (speech, natural sounds, music) and interact according to instructions.
MuChoMusic	[160]	The first benchmark for evaluating music understanding in audio MLLMs.
MCUB (Multimodal Commonality Understanding Benchmark)	[161]	Includes four modalities image, audio, video, and point cloud measuring the model’s ability to identify commonalities among input entities.
M3DBench	[162]	Focuses on 3D instruction following.
ScanQA	[163]	3D question answering for spatial scene understanding.
AVQA	[164]	Designed for audio-visual question answering on general videos of real-life scenarios.
MMT-Bench	[165]	A comprehensive benchmark assessing MLLMs across massive multimodal tasks toward multitask AGI.

6.5. Text-to-Audio Generation

Text-to-Audio (TTA) generation has emerged as a significant cross-modal research direction that extends the vision-language paradigm to the auditory domain. Unlike Text-to-Speech (TTS), which focuses exclusively on synthesizing spoken words from text, TTA encompasses the broader generation of general sound effects, music, and environmental audio from natural language descriptions [166]. This capability is central to achieving holistic multimodal understanding, as auditory signals provide complementary semantic information to visual and textual modalities. The rapid development of TTA systems has been catalyzed by advances in Large Language Models (LLMs), diffusion-based generative frameworks, and contrastive audio-language pretraining, mirroring the architectural trends observed in vision-language models discussed in preceding sections.

6.5.1. Architectural Taxonomy of TTA Models

Current TTA approaches can be broadly categorized into three architectural families.

Diffusion-based models constitute the dominant paradigm, employing probabilistic generative processes that learn to reverse a gradual noise-adding procedure to produce realistic audio samples [167]. Notable examples include AudioLDM [168], which operates in a compressed latent space using latent diffusion models to generate diverse audio types including sound effects, music, and speech from natural language prompts. Make-An-Audio [169] addresses data scarcity through pseudo prompt enhancement via expert distillation and dynamic reprogramming, while operating on spectrogram autoencoders that predict self-supervised representations. PicoAudio2 [170] introduces fine-grained temporal control by combining coarse text descriptions with temporal matrices through a Diffusion Transformer backbone, enabling precise timing of audio events. DualSpec [171] extends TTA to the spatial audio domain, generating 3D sound from text descriptions by leveraging dual spectrogram representations (Mel and STFT) to balance sound quality with directional accuracy.

Transformer-based models adapt transformer architectures to handle audio tokens conditioned on text inputs. VinTAGe [172] exemplifies this approach as a flow-based transformer that leverages a Visual-Text Encoder for cross-modal interaction, enabling holistic audio generation that is both temporally synchronized with video and semantically aligned with text.

Audio Language Models represent a third category, utilizing transformer-based language models to predict audio tokens autoregressively or non-autoregressively. This paradigm includes AudioLM [173], SPEAR-TTS [174], MusicLM [175], and MusicGen [176], which treat audio generation as a sequence modeling task analogous to text generation. VALL-E [177] further demonstrates this approach by framing TTS as a conditional language modeling task.

A related emerging direction involves joint video-audio conditioning models such as T2AV [178], which generates audio from text while maintaining temporal alignment with video content, and ReWaS [179], which uses energy patterns as a bridge between video and audio modalities.

6.5.2. TTA Datasets and Benchmarks

Several datasets have been developed to support TTA research, though the field still faces significant data scarcity compared to vision-language tasks. Table 9 summarizes the key datasets used in current TTA research.

6.5.3. Comparative Analysis of TTA Models

Table 10 presents a comparative overview of representative TTA models across key evaluation dimensions. The CLAP (Contrastive Language-Audio Pretraining) score measures audio-text alignment, while the Fréchet Audio Distance (FAD) quantifies the distributional similarity between generated and reference audio, with lower values indicating higher quality.

As shown in Table 10, diffusion-based architectures generally demonstrate superior generation quality (lower FAD), with DreamAudio [183] achieving the best FAD score of 0.46. Hybrid models such as VinTAGe [172] achieve the highest CLAP alignment score (0.86), suggesting that incorporating visual conditioning alongside text improves semantic alignment. The progression from pure transformer-based autoregressive models (AudioGen) to latent diffusion and hybrid architectures mirrors the broader trend observed in vision-language models, where increasingly sophisticated fusion mechanisms yield improved cross-modal alignment.

6.5.4. Evaluation Metrics for TTA

TTA evaluation employs both objective and subjective metrics, many of which parallel those used in vision-language evaluation. The CLAP Score measures audio-text alignment analogous to CLIP-based image-text alignment. The Fréchet Audio Distance (FAD) quantifies distributional similarity between generated and reference audio features (typically extracted via VGGish), serving a role comparable to FID in image generation. Kullback-Leibler (KL) Divergence measures the logarithmic difference between probability distributions of audio features extracted via PANNs [184]. Mean Opinion Score (MOS) provides subjective evaluation across dimensions including overall quality (MOS-Q), faithfulness to text description (MOS-F), and temporal alignment (MOS-T). Additional task-specific metrics include the Inception Score (IS) for assessing generation quality and diversity, defined as:

$$\mathrm{IS}=exp\left({\mathbb{E}}_{\mathbf{x}}\left[\mathrm{KL}\left(p\left(y\right|\mathbf{x})\parallel p(y)\right)\right]\right)$$

(1)

and Segment-F1 for evaluating temporal accuracy of audio event detection, which compares predicted audio event segments with ground-truth segments and computes F1 based on segment-level matches. The Structural Similarity Index (SSIM) is also employed for spectrogram-level comparison:

$$\mathrm{SSIM}={\displaystyle \frac{(2{\mu }_x{\mu }_y+c_1)(2{\sigma }_{xy}+c_2)}{({\mu }_x^2+{\mu }_y^2+c_1)({\sigma }_x^2+{\sigma }_y^2+c_2)}}$$

(2)

6.5.5. Open Challenges and Connections to Vision-Language Research

Several open challenges in TTA research directly parallel those identified in vision-language modeling. First, semantic alignment between text and audio remains difficult, as text encoders often fail to capture precise temporal and acoustic cues from natural language descriptions, echoing the fine-grained cross-modal alignment challenges discussed in Section 8. Second, data scarcity persists as a fundamental limitation; general audio datasets lack the fine-grained temporal annotations necessary for training models with precise timing control, and there exists a significant distribution gap between simulated and real-world audio data. Third, computational efficiency for latent diffusion models and the difficulty of modeling long continuous waveforms impose constraints analogous to those faced by large-scale VLMs. Fourth, connecting video information to audio generation models across different representation spaces presents a cross-modal bridging challenge that mirrors the modality alignment problems observed in vision-language fusion (Section ??). Finally, balancing sound quality with directional accuracy in spatial audio generation [171] exemplifies the broader trade-off between generation fidelity and semantic precision that pervades multimodal generative systems.

Future directions for TTA include developing unified Audio-LLMs with zero-shot generalization across understanding and generation tasks, expanding instruction-tuning datasets such as Audio-FLAN [182], integrating conversational capabilities for real-time dialogue, and improving temporal control for overlapping sound events. These directions are closely aligned with the broader research trajectories for multimodal foundation models outlined in Section 8, particularly regarding self-supervised pretraining, adaptive fusion mechanisms, and efficient cross-modal interaction.

7. Evolution of Multimodal Vision Models

The evolution of Multimodal Vision Models (VLM/MLLM) can be systematically categorized into three major eras, moving from early systems focused on task-specific feature engineering to modern, large-scale foundational models that leverage generalized pre-training and transformer architectures.

Early Models (2007–2015) [185,186,187]

This initial phase saw the introduction of foundational tasks for combining vision and language, primarily relying on convolutional neural networks (CNNs) for vision and recurrent neural networks (RNNs) or long short-term memory (LSTM) for language. This era predates the widespread adoption of large-scale, unified vision-language pre-training (VLP).

Key Models and Architectures

• DeViSE (Deep Visual-Semantic Embedding Model) [187]
  Architecture & Training: Introduced in 2013, DeViSE focused on learning a shared embedding space between visual and semantic modalities.
  Unique Contributions: This approach enabled zero-shot classification, allowing the model to detect unseen object classes by leveraging purely textual descriptions.
• VQA (Visual Question Answering) [185,188]
  Unique Contributions: While VQA refers primarily to the task and dataset (introduced in 2015 by Antol et al.), it drove the development of early VLM architectures, defining the goal of answering questions based on visual input.
  Architecture & Training (Early Methods): The earliest deep learning approaches for VQA relied on CNN–RNN pairs. For vision feature extraction, models like VGGNet [189,189] and GoogLeNet [190,190] were commonly used, often employing transfer learning by leveraging knowledge learned on large vision datasets like ImageNet [191,191]. The fused output was then typically passed to a classifier or generator.
• NeuralTalk / Neural-Image-QA [186]
  Architecture & Training: Neural-Image-QA (2015) was one of the first deep learning-based approaches for image question answering. It often used components like GoogLeNet for the image encoder and LSTM for the text encoder.
  Unique Contributions: These models marked the shift towards deep learning for image understanding and question answering tasks.

Transformer Revolution (2016–2020) [36,192,193,194]

This period is defined by the proliferation of the Transformer architecture [36], leading to the emergence of Vision-Language Pre-training (VLP) techniques that treat vision and language jointly, often pre-trained on large image-text pair datasets.

Key Models and Architectures

• VisualBERT [192,192]
  Architecture: A single-stream model that processes both vision and language sequences jointly within a single encoder, usually based on BERT. The visual features were typically extracted using Faster R-CNN (FR-CNN) [195,195].
  Training & Contributions: Served as a highly performant and relatively simple baseline for vision and language tasks.
• ViLBERT (Pretraining Task-Agnostic Visiolinguistic Representations) [193,193]
  Architecture: A dual-stream model architecture that encodes the visual and textual sequences separately before joining them in a Cross-Modal Transformer for fusion. It used BERT for the text encoder and FR-CNN for the visual encoder.
  Unique Contributions: ViLBERT was an early example of dual-stream models, proposed to account for the differences in abstraction levels between the two modalities. It aimed to pre-train task-agnostic representations for vision-and-language tasks.
• LXMERT (Learning Cross-Modality Encoder Representations from Transformers) [194,194]
  Architecture: A dual-stream framework based on Transformer encoders, featuring three components: a language encoder, an object relationship encoder, and a dedicated cross-modality encoder. It uses Cross-Modal Transformer technology.
  Training & Contributions: LXMERT utilized a comprehensive pre-training strategy involving five diverse tasks, including masked language modeling, masked object prediction (feature regression and label classification), cross-modality matching, and image question answering. This resulted in strong generalization capabilities across multiple visual reasoning tasks.

Recent Large-Scale MLLMs (2021–2025) [47,54,196,197,198]

This era is characterized by the convergence of massive, pre-trained Large Language Models (LLMs) and advanced vision encoders, resulting in Multimodal Large Language Models (MLLMs). These models often utilize frozen LLMs as a backbone and focus on efficient alignment strategies.

Key Models and Architectures

• CLIP (Contrastive Language-Image Pre-training) [47,47]
  Year: 2021.
  Architecture: Encoder–decoder model, using Vision Transformers (ViT) [199,200] or ResNets as the vision encoder.
  Training & Contributions: Trained using a contrastive learning objective on 400M image-text pairs [47], aligning vision and language encoders into a shared representation space. This training method enables remarkable transferability and strong zero-shot classification capabilities, surpassing classical single-modality models.
• Flamingo [54]
  Year: 2022.
  Architecture: Decoder-only structure, designed to bridge powerful pretrained vision-only models (like NFNet) and language-only models (like Chinchilla-70B). It incorporates Cross-Attention (XAttn LLM) modules within the language model layers to fuse visual features.
  Training & Contributions: Flamingo was the first VLM to explore in-context few-shot learning at scale. It introduced architectural innovations to handle interleaved visual and textual data sequences. The model uses a resampling strategy to fix the number of visual tokens presented to the LLM.
• BLIP and BLIP-2 [198,201]
  Year: BLIP (2022), BLIP-2 (2023).
  Architecture: BLIP used an Encoder–decoder architecture trained from scratch. BLIP-2 introduced the Q-Former (Querying Transformer). The Q-Former acts as a flexible, trainable adapter module between a frozen visual encoder (like EVA ViT-g) and a frozen LLM (like FlanT5).
  Training & Contributions: BLIP used bootstrapping for unified V–L understanding and generation. BLIP-2 revolutionized VLM training by decoupling the visual encoder and the LLM, enabling the leverage of powerful, frozen pre-trained LLMs to bootstrap language-image pre-training.
• LLaVA-1.5 [202]
  Year: 2023.
  Architecture: Decoder-only model, typically using a frozen CLIP ViT-L/14 visual encoder and a Vicuna LLM backbone. It uses a simple MLP projection (a two-layer multilayer perceptron) to connect visual features to the textual embedding space.
  Training & Contributions: A primary example of utilizing visual instruction tuning (VIT) to enhance multimodal capabilities and promote conversation skills.
• GPT-4V (GPT-4 Vision) [196,203]
  Year: 2023.
  Architecture & Training: Details are undisclosed.
  Unique Contributions: Recognized as a pioneering MLLM [196,197], GPT-4V demonstrates strong reasoning and understanding across visual and textual data. Qualitatively, it is noted for its precision and succinctness in responses compared to competitors.
• Gemini [197,204]
  Year: 2023.
  Architecture & Training: A family of models utilizing a decoder-only architecture [197,197]. Details are undisclosed.
  Unique Contributions: Gemini excels in providing detailed, expansive answers, often incorporating relevant imagery and links, showcasing sophisticated multimodal capabilities [204].
• CogVLM [205?]
  Year: 2023.
  Architecture: Encoder–decoder model, utilizing a visual expert (CLIP ViT-L/14) and combining projection (MLP) with a modality experts fusion strategy.
  Training: It is visually instructed tuned. CogVLM is designed as a visual expert for pretrained language models.

8. Conclusion

Vision-based multimodal learning has undergone a profound transformation over the past decade, evolving from early visual–semantic embedding approaches to large-scale vision-language models (VLMs) and instruction-tuned multimodal large language models (MLLMs). In this review, we presented a comprehensive, task-oriented, and historically grounded analysis of this evolution, systematically categorizing multimodal vision models across major architectural paradigms, including dual-encoder contrastive frameworks, fusion-based transformer architectures, and unified generative models. By examining representative models and their applications across core vision-centric tasks such as image captioning, visual question answering, visual grounding, and cross-modal generation, we highlighted how advances in multimodal pretraining and transformer-based design have reshaped the capabilities of visual-language systems.

Our analysis demonstrates that large-scale multimodal pretraining has fundamentally shifted the field from task-specific multimodal fusion toward more generalizable, instruction-following, and zero-shot capable models. Compared to traditional early, late, and hybrid fusion strategies, modern VLMs and MLLMs benefit from stronger cross-modal alignment, emergent multimodal reasoning abilities, and improved transfer across downstream tasks. However, this progress has also introduced new challenges related to computational cost, data efficiency, interpretability, and robustness in real-world scenarios. Through a detailed examination of widely used multimodal datasets and benchmarks, we further revealed limitations in dataset diversity, annotation bias, and representativeness, which continue to constrain the generalization and evaluation of multimodal models. Furthermore, our analysis extends beyond the vision-language boundary to examine Text-to-Audio (TTA) generation as an increasingly important cross-modal research direction. However, TTA research faces its own distinct challenges, including limited large-scale annotated datasets, difficulty in capturing precise temporal cues from natural language, and the inherent trade-off between generation fidelity and semantic alignment.

Despite their impressive performance, current VLMs and MLLMs still struggle with fine-grained cross-modal reasoning, reliable grounding between visual entities and linguistic concepts, and adaptation to dynamic or low-resource environments. These challenges are particularly evident in embodied and robotic settings, where real-time constraints, multimodal synchronization, and domain-specific variability demand more efficient and adaptive fusion mechanisms. Addressing these limitations requires future research efforts that move beyond scale alone, emphasizing self-supervised and weakly supervised multimodal learning, dynamic and task-aware fusion strategies, and more principled approaches to multimodal reasoning and generalization.

Looking forward, promising research directions include self-supervised multimodal pretraining to reduce reliance on large annotated datasets, adaptive attention and routing mechanisms for efficient cross-modal interaction, and the integration of symbolic reasoning and world knowledge into multimodal foundation models. Furthermore, as multimodal systems become increasingly deployed in real-world applications, incorporating ethical, human-centered, and safety-aware design principles will be critical to ensuring responsible and trustworthy multimodal AI. By consolidating past developments, clarifying current limitations, and outlining future research trajectories, this survey aims to serve as a unified reference framework for advancing vision-language understanding and multimodal intelligence in the next generation of AI systems.