Continuous or Discrete, That Is the Question: A Survey on Large Multi-Modal Models from the Perspective of Input-Output Space Extension

1. Introduction

The goal of AI research is to build versatile intelligent systems capable of fulfilling tasks across diverse scenarios. Recently, the generalization and interactivity demonstrated by large language models (LLMs), which leverage language instructions as the interface between users and machines, have significantly advanced the progress towards general-purpose AI [1,2,3,4]. To extend these capabilities to multi-modal contexts, research on large multi-modal models (LMMs) is emerging, aiming to expand the input and output space of the language-based interface to more modalities. As shown in Figure 1, to extend the input space, existing methods introduce discretely or continuously encoded modality representations into the text input and learn cross-modal alignment from multi-modal intertwined data, enabling LMMs to understand multi-modal information [5,6,7,8]. Similarly, the output space can be divided into multiple subspaces of different modalities, which are further aligned with corresponding modality decoders to generate multi-modal content [9,10,11,12].

Although there are several surveys that detail the current progress in constructing LMMs, most of these works are limited to specific perspectives. (1) Some studies merely introduce the input-side extension, lacking discussion on the extension of outputs [13,14]. (2) Certain studies solely discuss specific sub-problems in the construction of LMMs, such as applications in specific modalities [15,16] and scenarios [17,18], evaluation [19,20], and data [21]. (3) Meanwhile, most existing reviews focus on a specific type of model framework: encoding information from other modalities in a continuous manner and aligning them with text embeddings through connection modules, neglecting related research on other architectures, such as unified discretely represented LMMs [12,22]. These limitations prevent existing reviews from adequately covering research problems in LMM construction and limit their applicability to a broader scope.

To this end, this survey aims to summarize related works from a more general perspective: the extension of the input-output space. As illustrated in Figure 1, existing LMMs can be systematically summarized from this perspective, encompassing various modalities, scenarios, and model architectures, while also leaving room for further exploration to more modalities and scenarios.

To conduct a comprehensive survey, we follow a top-down logic to break down the construction of LMMs into several sub-problems, providing detailed discussions to offer insights to readers. Particularly, we try to answer the following questions. (i) How can modality signals be encoded using discrete or continuous representations, and how to construct multi-modal input-output spaces? (§Section 3) (ii) How to design model architectures and corresponding training strategies to align the constructed multi-modal representation space? (§Section 4) (iii) How to comprehensively evaluate LMMs based on the expanded input-output space? (§Section 5, Section 6) The content of this paper focuses primarily on the extension to vision and naturally extends to audio and arbitrary-to-arbitrary modality interactions. In addition to modality extension, §Section 7 introduces how to extend the input-output space to embodied scenarios, further demonstrating the extensibility of LMMs from the perspective discussed in this paper. In §Section 8, we summarize the discussion on the questions raised above, providing readers with key take-home messages and an outlook on future research.

In summary, our contributions are threefold:

• Going beyond specific scenarios and model framework, we review the current LMMs from a general perspective of input-output space extension. We hope that such a broad and comprehensive survey can provide an intuitive overview to related researchers and inspire future work.
• Based on the structure of input-output spaces, we systematically review the existing models, including mainstream models based on discrete-continuous hybrid spaces and models with unified multi-modal discrete representations. Additionally, we introduce how to align the constructed multi-modal representations and conduct evaluations according to the extended input and output.
• We elaborate on how to extend LMMs to embodied scenarios to highlight the extensibility of LMMs from the input-output extension perspective. To our knowledge, this is the first article to summarize embodied LMMs.

2. Preliminary

Before introducing LMMs, we briefly outline the evolution of multi-modal research paradigms in this section to highlight the key difference and advancements of LMMs, compared to earlier multi-modal models. As presented in Figure 2, we focus on the vision-language domain, diving the development of related research into three stages.

• Task-Oriented Paradigm

Early multi-modal research focuses on specific scenarios and tasks. The core research problems are how to define multi-modal tasks, construct benchmarks, and design models to address these tasks. The most commonly explored tasks include VQA [23,24], image-text retrieval [25,26], image captioning [27,28], visual grounding [29], visual reasoning [30,31,32,33], and so on. Models with various architectures have been proposed for specific tasks [34,35,36,37,38,39,40]. The key characteristic of task-oriented methods is to define tasks through a large number of samples (training sets), limiting the generalizability and requiring high costs for transfer across tasks.

• Vision-Language Pre-training (VLP)

Since task-oriented methods may introduce task-specific inductive bias and lead to overfitting, researchers explore ways to construct unified architectures and learn generalized multi-modal representations. Inspired by the pre-training techniques introduced by BERT [41], vision-language pre-trained (VLP) models are built on multi-layer Transformer [42] and trained with self-supervised tasks on large amounts of image-text pairs [43,44,45,46]. Pre-trained models provide effective initial checkpoint for fine-tuning on various downstream tasks [47,48,49,50,51]. VLP methods make an important step towards generalization, but they still require specific parameters and fine-tuning samples to define tasks, failing to provide a unified interface for users.

• Large Multi-Modal Models (LMMs)

The success of LLMs has revealed the potential of using language-based instructions as a generalized and interactive interface [52]. Inspired by this, LMMs also leverage language as the interface between users and machines. By integrating and aligning other modalities into the input-output space, LMMs can understand multi-modal context and respond subsequent instructions from users, even in zero-shot scenarios [6,7,53]. Such generalizability and interactivity make LMMs highly applicable and versatile as multi-modal foundation models.

3. Input-Output Space Extension

In this section, we introduce prevalent solutions to construct multi-modal input-output space. As illustrated in Figure 3, existing methods can be categorized based on different input-output space structures, and the extension to other modalities can be summarized in a similar manner.

3.1. Encode Multi-Modal Input Representation

Regarding the input, the core research problems involve how to code the representations of each modality and how to integrate them into a multi-modal input space (illustrated in the lower part in Figure 3).

3.1.1. Textual Representation

As a discrete signal, text is a sequence composed of characters. Following the practice of LLMs, LMMs typically utilize tokenizers, such as BPE [54,55], WordPiece [56], and Unigram [57], to merge characters into sub-word tokens. Ultimately, texts are represented as sequences of discrete tokens.

3.1.2. Visual Representation

For visual signals with spatial-temporal information, LMMs mainly employ pre-trained visual encoders for representing images (videos) into continuous features or discrete codes. Figure 4 illustrates the evolution of existing visual encoders.

• Encoder Architecture

Commonly adopted architectures can be divided into two categories: convolution-based [58,59] and vision-Transformer-based models [60,61]. Both methods encode images into continuous 2D feature maps. These features can be further converted into discrete visual codes through vector quantization (VQ) by learning a fixed-size visual codebook [62,63]. In addition, models like Fuyu [64] do not rely on visual encoders and directly use pixel values of image patches as the visual representations.

• Encoder Training

Employed visual encoders are mainly pre-trained in supervised or self-supervised manner. Early exploration utilize image categories as supervision signals [65], while CLIP-like models [66,67,68] use language supervision to learn generalized representations. Additionally, SAM [69] leverages segmentation tasks as training objectives. In contrast, self-supervised learning only requires images for training. Contrastive self-supervised methods train models to distinguish representations between different images [70,71,72,73]; another line of approaches construct auto-encoders, where models are demanded to reconstruct images from the encoded visual representations, which is often used to support downstream image generation [62,63,74,75].

• Visual Representation Enhancement

Since most visual encoders are limited to fixed resolutions and capture specific aspects of visual features, existing LMMs proposes to enhance the input visual representations on two aspects: resolution enhancement and feature enhancement.

To support high-resolution input images, a line of methods directlty extends the visual encoder, including interpolating position embeddings in vision Transformers [7,76] and using CNN-based models to enhance the encoding efficiency of high-resolution images while compressing the size of encoded feature maps [77,78]. Another line of approaches propose to crop high-resolution images into multiple sub-images and input them into the low-resolution encoder along with the down-sampled full image [79,80,81,82,83]. Additionally, different sub-image partitioning templates help address issues caused by varying aspect ratios of images.

Regarding feature enhancement, common practices consider ensembing visual representations encoded by different encoders, such as combining encoders trained with different strategies [84,85], or integrating high-resolution and low-resolution encoders [86,87]. Specialized modules have been introduced to better fuse features from different encoders [87,88,89]

• Multi-Image Input

Based on the prevalent sequence modeling framework of current LMMs, multiple images can be intuitively arranged in the input sequence [90,91,92,93]. For videos, where images (frames) are temporally related, spatial-temporal encoders such as TimeSformer [94] and VideoSwin [95] can be further used for encoding [96,97].

3.1.3. Constructing Multi-Modal Input Space

As illustrated in the lower part Figure 3, there exist two mainstream types of multi-modal input space.

• Type A: Hybrid Input Space

Text are represented in a discrete form, while visual signals are encoded in continuous representations, preseving the complete visual information. However, due to the gap in the input space, connection modules are required to perform input-level cross-modal alignment, which is discussed in Section 4.

• Type B: Unified Discrete Input Space

Different from Type A, further quantizing visual representations into discrete visual codes facilitates the construction of a unified input space. A multi-model vocabulary can be intuitively integrated and directly used to support subsequent modeling.

3.1.4. Extension to More Modalities

Beyond the vision modality, signals from other modalities can be encoded and introduced into the input space following a similar paradigm. For example, various encoders can help encode audio into continuous [98,99,100] or discrete [101] representations. As a step further, an arbitrary-modality input space can be represented in either hybrid [11,102,103,104] or unified discrete forms [12].

3.2. Decode Multi-Modal Output Representation

Based on the input, backbones of LMMs present continuous multi-modal output representations which can be used to decode the output signals of different modalities. For example, with the commonly used causal modeling framework, the output representation can be leveraged to predict the signal at the next position in the sequence. Predicted token sequence can be converted to text with the tokenizer while different image generator can be adopted to decode images from output in different forms. In this section, we discuss the commonly adopted paradigms to partition the output space of different modalities and perform corresponding decoding, as shown in the upper part of Figure 3.

3.2.1. Type 1: Text-Only Output Space

If only text output is required, similar to LLMs, discrete tokens can be generated from the ouput representations through a classification-based language modeling (LM) head and specific decoding strategies [5,6].

Please note that models that first generate text descriptions and then use external tools like Stable Diffusion and CLIP to generate or retrieve content in other modalities, such as Visual ChatGPT [105], InternLM-XComposer series [106,107], and Mini-Gemini [87], are also classified as text-only output models because they are not in an end-to-end manner.

3.2.2. Type 2: Hybrid Multi-Modal Output Space

To support image generation, a series of methods first introduce special tokens, such as the start and end tokens for images, or a series of consecutive placeholder tokens to indicate where images should be generated. The continuous output representations at the corresponding positions are then connected to visual decoders (mainly Diffusion models [108]) through visual mapping modules [9,109,110,111]. Similar to the hybrid input space, visual mapping modules perform output-level alignment and requires further training which is described in Section 4.

3.2.3. Type 3: Unified Discrete Multi-Modal Output Space

Based on the joint multi-modal vocabulary constructed within Type B input space described in Section 3.1.3, image generation can be naturally integrated into the token decoding process. Image and text tokens in the vocabulary inherently divide the output space, and the predicted visual codes can be fed to the corresponding codebook detokenizer to generate the image [22,112].

3.2.4. Extension to More Modalities

The Type 2 and Type 3 output spaces can be further expanded to incorporate more modalities to support arbitrary-modality output. Next-GPT [11] and Codi-2 [103] further extends the hybrid output space, while AnyGPT [12] and UnifiedIO-2 [104] construct unified discrete spaces for all modalities.

3.3. Prevalent Input-Output Paradigms

Considering the input-output space structures introduced above, most existing LMMs can be categorized to three types: (1) Multi-modal understanding models that rely on Type A input and Type 1 output, these models are mainly designed for understanding tasks that can be fully expressed in language [7,84,113,114]; (2) Multi-modal generation models which comprise of Type A input and Type 2 output, such models excel in generating multi-modal interleaved responses based on the context [9,11,75]; (3) Unified multi-modal models that represent and generate multiple modalities in a unified discrete form [12,22,112].

Table 1 and Table 2 list the design paradigms of currently popular LMMs, grouped according to the aforementioned classification criteria. The model alignment architectures that will be discussed in Section 4 are also included.

4. Multi-Modal Alignment

Based on the multi-modal input-output spaces introduced in Section 3, the design of LMMs further needs to consider how to align representations across different modalities. The core research problems behind include: (1) How to design the corresponding model architecture to uniformly model multi-modal representations (Section 4.1); (2) How to train the model parameters to learn alignment and interaction across modalities (Section 4.2). Ultimately, through multi-modal alignment, LMMs can simultaneously comprehend multi-modal contexts and generate multi-modal responses.

4.1. Alignment Architecture

The current mainstream LMM architectures follow a similar paradigm: aligning inputs from all modalities to a unified multi-modal backbone for modeling, interaction, and generating multi-modal responses. To facilitate the unified modeling, additional modules are designed for (1) input-level alignment to unify the multi-modal inputs into a consistent form and space; (2) internal alignment of the backbone for complex cross-modal interactions; and (3) output-level alignment to map the outputs of the backbone to different modality decoders.

4.1.1. Multi-Modal Modeling Backbone

Typically, the backbone is based on a decoder-only architecture composed of multiple Transformer blocks [42]. To better understand language, the backbone is primarily initialized with a pre-trained LLM, such as LLaMA [2,177,178], Vicuna [179], Mistral [180], Qwen [3,181], and so on [143,182].

In addition, LMMs for edge devices are usually initialized with smaller language models, such as MobileLLaMA [183], Phi [148], etc. [184,185]. The backbone can also inherit MoE-based language models like Mixtral 8x7B [186].

Apart from the commonly used architecture mentioned above, some LMMs adopt encoder-decoder backbones [104,187,188,189,190]. Additionally, native LMMs like Chameleon [22] are not initialized with pre-trained LLMs and trained from scratch.

4.1.2. Input-level Alignment

As introduced in Section 3.1.3, there may exist gaps between modalities in the extended multi-modal input space. To enable the backbone to process multi-modal information uniformly, it is necessary to align the form and space of inputs across modalities at the input level.

Specifically, for Type B input space, since all modalities are represented in a unified discrete token form, input-level alignment can be achieved by directly merging the vocabularies of multiple modalities and learning the token representations through subsequent alignment training [12,22,112].

Regarding Type A hybrid input space, it is required to introduce a connection module to convert inputs from other modalities into a sequential representation that matches the dimension of textual token embeddings. Commonly adopted connection modules are summarized below.

• MLP Based

A typical connection module is implemented through one or more linear projection layers, connected by activation functions such as GeLU [191], resulting in a multi-layer perceptron (MLP). This approach directly aligns the dimension of representations from other modalities with text [6,128]. By further flattening the 2D or 3D features into 1D in a specific order, it allows for alignment with the text sequence [11,83,123]. The advantage of MLP-based modules lies in the simplicity, light weight, offering fast convergence during alignment training. However, MLP-based modules cannot compress redundant information, which may result in excessively long modality representation sequences (e.g., high-resolution images), reducing computational efficiency and requiring additional designs to compress the information [76,147,192].

• Attention Based

Another prevalent connection modules are based on attention mechanisms. This method typically introduces a fixed number of learnable vectors as queries, which retrieve relevant information from other-modality representations (serving as keys and values) through cross-attention modules. The output representations of the queries, enriched with information from other modalities, serves as the modality input to the backbone. Representative module architectures include Q-Former [5,113], abstractor [118,132], resampler [80,124], and so on [7,106]. The query-level representations obtained from attention mechanisms effectively compress and aggregate information from other modalities. Additionally, recent works have demonstrated further extensibility, including integrating representations from multiple encoders [87,88,193], incorporating local grounding information [194], and scaling up to an 8B Q-LLaMA [114]. However, these modules mainly involve more parameters and typically require additional training [5,194]. Yao et al. have found that attention-based modules may result in the loss of important information.

• Others

In additional to the mainstream structures mentioned above, several other connection modules have been proposed. CNN-based modules utilize the inductive bias of convolutional operations to model local information, further combined with pooling layers, the number of resulted tokens can be effectively reduced [136,140,156]. Adaptive pooling-based modules can compress features using spatial relationships without introducing additional parameters [149,154]. Furthermore, VL-Mamba explores to use vision selective scanning as connection to integrate representations across different modalities [145].

4.1.3. Internal Alignment

With the help of input-level alignment, vanilla Transformer based backbones can uniformly process multi-modal information. Furthermore, researchers have explored introducing additional parameter modules within the backbone to further enhance the modeling of internal interactions and alignment between modalities. In this section, we categorize and summarize commonly adopted methods for internal alignment.

• Cross-Attention Layer

Flamingo [115], as a pioneering work in LMM, is the first to propose inserting cross-attention layers between the original layers of the backbone, allowing text to perceive information from the visual context. Additionally, a tanh gating mechanism is introduced to control the degree of modality fusion. Subsequently, the Flamingo architecture has been adopted by recently proposed LMMs [121,126,127,195], CogAgent [86] further utilizes cross-attention to supplement high-resolution image information. Although effective, densely inserted cross-attention layers bring a large number of parameters. Ye et al. improve this by introducing sparsely inserted hyper attention, which significantly reduce extra parameters and facilitate model convergence through parallel self-attention and cross-attention calculation.

• Adaption Prompt

LLaMA-Adapter incorporates modality representations into lightweight learnable adaption prompts and feed the prompts as prefix contexts to the backbone [116]. LLaMA-Adapter V2 [119] improves this method with an early knowledge fusion strategy. ImageBind-LLM [102] further extends the adaption prompts to support more modalities.

• Visual Expert

To distinguish between visual and textual modeling, some LMMs introduce visual expert modules to process visual tokens specifically. CogVLM [130] adds additional attention and FFN layers to process visual tokens without compromising the original textual modeling capabilities of backbones. mPLUG-Owl2 [132] only introduces modality-specific parameter blocks in the normalization layers and the K and V mapping layers of the attention modules. InternLM-XComposer2 [107], on the other hand, designs a lightweight Partial LoRA module for additional modeling of visual tokens.

• Mixture of Experts (MoE)

Unlike previously discussed modules that are densely activated, the idea of MoE is to introduce “experts” modules in the backbone which can be sparsely activated according to different inputs through gating routers [196,197,198]. A typical solution of introducing MoE into LMMs is based on sparse upcycling [199] to sparsify a dense checkpoint. LLaVA-MoLE [138] considers LoRA as experts and incorporates them into the FFN layers, while MoE-LLaVA [139] directly extends the FFN layers of the base model. CuMo [155] expands the FFN layers of the visual encoder and connection module with co-upcycled MoE layers. All these models utilize Top-K gating routers.

Methods mentioned above introduce MoE through implicit knowledge modeling, explicit modality-specific knowledge can also be incorporated in the design of MoE in LMMs. Uni-MoE [163] extends FFN layers and allocate specific experts for each modality. Modality-specific data are utilized to train corresponding experts for further enhancement. During inference, only relevant modality experts are activated, allowing effective utilization of modality knowledge while maintaining efficiency. Similarly, Chameleon-MoMa [175] duplicates FFN layers in Chameleon and divides experts into modality-specific groups in which routers are independently learned.

4.1.4. Output-level Alignment

Regarding the multi-modal output space described in Section 3.2, both Type 1 and Type 3 are represented in a unified discrete token-based form, multi-modal content can be intuitively generated through a next-token prediction approach with the help of modality-specific de-tokenizer [12,22,104,112].

For the Type 2 hybrid output space, although modality-related tokens help divide the output space into different modalities, additional mapping modules are required to align the output space of LMM backbones with the input space of corresponding modality generators. Considering image generation, commonly used modules are built on linear projection [109] or the Transformer architecture [9,110]. Similar to Q-Former, Transformer-based modules learn a fixed number of queries to retrieve information from the LMM outputs through cross-attention, serving as the condition input of image diffusion models [108]. Next-GPT [11] further extends the Transformer-based mapping modules to fit more modality diffusion generators. Additionally, Emu series [75,111] replace the linear projection of cross-attention in diffusion models to perform dimensional conversion, achieve output-level alignment.

4.2. Multi-Modal Training

In this section, we discuss how to train the model constructed in Section 4.1 to learn cross-modal alignment and modeling, facilitating the understanding and generation of multi-modal content. We first introduce commonly adopted training data, followed by an elaboration of how to utilize the data to design multi-stage training frameworks for LMMs.

4.2.1. Training Data

We separate existing training data for multi-modal alignment into two categories: pre-training data and instruction tuning data. In the following section, we delve into these data categories, providing a detailed overview of their composition.

• Pre-training Data

The pre-training data mainly consists of sequences interspersed with multi-modal information, guiding LMMs to learn the multi-modal associations embedded within and align the representations. Common pre-training data typically exists in three forms, as described below.

X-text Pairs. The most typical format consists of paired X-modal data and the corresponding text, which is image-caption pair for the vision modality. Early captioning data are primarily curated by human annotators, including Flickr30K [26], COCO [203], SBU [241]. Although these datasets are of high quality, their scales are limited and the cost of further extension is affordable. To this end, subsequent works introduce methods for image-text pairs collection by crawling the web, followed by rigorous filtering and post-processing, which leads to the development of significantly larger datasets, such as CC3M [200], CC12M [242], LAION [206], COYO [202], and DataComp [209].

To process the web crawling data, filtering and deduplication are applied to ensure the datasets with high quality and broad coverage. Data filtering aims at removing undesirable content, focusing on text and image data separately. Text filtering includes language filtering, which eliminates documents below a certain language threshold [267,268,269], and content filtering, which keeps toxic or incomplete sentences out [270,271]. Image filtering discards low-resolution images, those with inappropriate aspect ratios, and images with unavailable URLs [213,259,261]. Deduplication is also essential, as redundant information can harm the model performance [272]. The exact deduplication method removes duplication through string matching [267,273], while URL-based deduplication identify redundant information from the same web pages [269]. In addition, locality sensitive hashing (LSH) methods can be adopted to perform approximate deduplication [274,275], and semantic-level deduplication can be managed by clustering semantic embeddings and retaining representative data [276,277,278]. Commonly used image deduplication methods involve removing by image URLs or pHash algorithms [259,279]. Although the aforementioned filtering methods are effective, these large-scale datasets are generally weakly labeled, suffering from noise and sub-optimal conditions for model training, with many captions being too simple or failing to accurately describe the images.

In response, recent methods resort to synthetic re-captioning, where original images are re-captioned by advanced models to generate concise textual descriptions, represented by LAION-COCO [207] and LAION-BLIP [48]. Further advancements are then attempted, applying more sophisticated captioning models and unique prompting stratagies, to generate detailed and high-quality image. For example, LaCLIP [343] utilizes LLM to rewrite raw captions, but resulting in severe hallucination, because of limited visual information included in low-quality raw captions. VeCap [344] uses LLaVA [251] to extract all possible visual clues and leverage an LLM to do ethical check and fuse the concepts from both AltText and visual clues to generate the final caption. Subsequently, Capsfusion [345] fine-tunes LLaMA-2 [2] with training data generated by ChatGPT, and the fine-tuned LLaMA-2 [2] organically fuses and harnesses raw and synthetic captions. Monkey [80] utilize a combination of several advanced systems to collect visual descriptions which are provided to ChatGPT. Different from aforementioned automatic generation pipelines, AS-1B [221] introduces a semi-automatic data engine that efficiently leverages various foundation models as annotators, significantly reducing the enormous labeling costs to a manageable level. Additionally, CogVLM2 [156], ImageInWords [218] and Densely Captioned Images (DCI) [219] also generate and refine detailed captions with human in the loop. Recently, it has become a trend to directly utilize GPT-4V’s visual perception capabilities to generate high-quality image descriptions [215,257]. Similar approaches are adopted in constructing pre-training data for Ovis [153] and LLaVA-OneVision [160]. Utilizing data generated by GPT-4V, ShareGPT-4V [131] trains captioning engines, while DenseFusion [256] further integrates visual experts as image priors to scale up hyper-detailed image-text data. In contrast to the image-to-text generation, SYNTH${}^2$ [214] leverages a text-to-image model to generate synthetic images.

Similarly, comparable datasets can be created for other modalities. For the video modality, early caption datasets were primarily composed of manual annotations, such as YouCook2 [233], VATEX [248], and Panda-70M [226]. Additionally, web-crawled datasets like HowTo100M [222], VideoCC3M [224], HD-VILA-100M [232], and WebVid-10M [225] are commonly used for video-language alignment, while synthetic captions such as Vript [243] and VIDAL [227] are generated by advanced GPT models. For the audio modality, audio-text pairs like Clotho [228], AudioCaps [229], and AudioSet [231] are widely utilized for pre-training, with synthetic captions provided by WavCaps [230], LAION-Audio-630K [244], and AF-AudioSet [247]. Please refer to Table 3 and Table 4 for the summarized commonly-adopted X-text pairs.

Multi-modal Interleaved Documents. Although X-text pairs have been demonstrated to be effective in pre-training for cross-modal alignment, these data are usually short in length and relatively simple in form. Additionally, Single X-text pairs cannot enable LMMs to learn in-context learning capabilities in multi-modal contexts [115]. To address this, researchers propose to constructed multi-modal documents, in which multiple information units of different modalities (images, sentences, speech, etc.) are distributed in an interleaved manner. Regarding the vision modality, MMC4 [259] is the first large-scale publicly available multi-modal interleaved dataset, which is an extension of the text-only C4 dataset by gathering images from WAT files and associating each image with a sentence. OBELICS [260] is constructed from HTML files obtained from Common Crawl dumps, and the resulting documents maintain the original linearity of images and texts as they appeared on the websites, while removing spam and ads. Exposing the model to a much wider distribution of texts, MINT-1T [261] curates more diverse sources of interleaved documents from HTML documents, PDFs and ArXiv papers, with a 10x scale-up from existing open-source datasets.

As for other modalities, InternVid-ICL [264] is a large-scale interleaved video-text dataset, established by arranging clips and their descriptions in sequences according to the chronological order, connecting the interlaced multi-modal items to create video-centric dialogues. Additionally, Howto-Interlink7M [265] is a high-quality interleaved video-text dataset derived from HowTo100M [222], and YT-Storyboard-1B [266] is curated for training Emu [111] and Emu2 [75]. Compared to X-text pairs, interleaved data is relatively limited scarce. Table 4 presents relevant multi-modal interleaved datasets.

Scenario-oriented Data. While X-text data and interleaved documents effectively aid cross-modal alignment, they are limited to semantic-level information about objects and scenes, and cannot offer LMMs extensive knowledge to handle demands of diverse scenarios. Therefore, according to specific scenarios, existing methods typically aggregate relevant data to help LMMs learn particular capabilities [156,158,374]. This type of data mainly pertains to the vision-language contexts and can be obtained through methods such as manual curating [23,292], re-formulating existing data [147,157], or automatic synthetic data generation [255,316]. In Table 5, we summarize several scenarios of interest and related datasets. General VQA [24,282,282] focuses on visual understanding of real-world images, including identifying people and objects, scene comprehension, counting, and color recognition, often necessitating complex reasoning about visual facts. General OCR data [300,303,304] is leveraged to enhance text-rich image understanding. Document/Chart/Screen data [304,307,312] empowers LMMs to interpret complex text and structural information, enhancing their understanding of documents, tables, and screen contents. Math/Science/Code [317,320,327] involves advanced mathematical reasoning and geometry tasks, as well as code visualization tasks. Detection and Grounding data [29,330] conveys spatial information of objects through annotations like bounding boxes, endowing capabilities of visual referring and fine-grained visual perception.

• Instruction-following Data

Based on the multi-modal representations aligned using pre-training data, LMMs further leverage instruction-following data to learn how to comprehend and follow instructions in multi-modal contexts, as well as develop the ability to solve diverse tasks. As revealed by the success of instruction-tuned LMMs [383], rich and comprehensive instruction-following data is the key to help models learn generalizable capabilities. Inspired by the prevalent methods to construct textual instruction data [52,384], researchers typically create multi-modal data through two approaches.

Reformulating Task-oriented Datasets. As discussed in Section 2, during the development of multi-modal research, a large amount of datasets have been established for various scenarios and tasks. These datasets are typically collected, annotated, and validated by humans according to corresponding requirements, ensuring high quality. To meet the demands of the tasks, this type of data generally exists in specific input-output formats [23,25,29] and requires additional processing to be reformulated into instruction-following data. Various datasets have been reformulated as listed in Table 6.

The most common solution is to introduce templates, providing an appropriate textual description of the task (i.e., instructions) as well as a question-and-answer format, and correctly placing the input and output of each sample in their respective positions [336,385,386]. In this case, specific templates need to be designed based on the corresponding task. Early works rely on annotators for template design [113,332,334,339,387], and certain specific tasks might require image processing, such as image concatenation [388] and object annotation [342]. Later, researchers also adopt tools like ChatGPT or Gemini-Pro to assist in template design and extension [121,150,389,390,391], and further rephrase brief and incomplete responses into longer, more complete sentences [392,393].

Furthermore, diverse reformulated datasets can be organically integrated for joint training to empower LMMs with a wide range of capabilities. For example, Cauldron [152] converts multiple samples of the same context (such as the same image) into multi-turn conversations. In MANTIS [333], each data item contains multiple images and multiple turns of QA pairs with a suitable text-image interleaving format. LEOPARD-INSTRUCT [377] spans key domains commonly encountered in real-world scenarios, such as multi-page documents, charts, tables, and webpage trajectories, tailored to text-rich, multi-image contexts, while Cambrian-10M [88] considers different types of data from the perspective of capabilities, balances their proportions, and enhances model performance in knowledge-intensive tasks.

Self-Instruction Although task-oriented datasets facilitates LMMs to learn diverse capabilities, the knowledge entailed is limited to the corresponding scenarios. At the same time, the template formats are also restricted to specific tasks and mainly differ from general human-machine interaction. To further enrich the instruction data, powerful proprietary models can be leveraged to generate data according to the provided information and requirements. The most typical approach is motivated by the self instruct method [394]. To prompt ChatGPT-like models to generate instruction data based on input multi-modal information, task descriptions and specific requirements are provided through system prompts and user queries. Additionally, in-context examples can be included to offer detailed guidance that cannot be well defined through language. Table 7 includes several datasets curated with similar methods.

For images, early works represented by LLaVA [6] provide visual information to ChatGPT via text descriptions including captions, objects and bounding boxes [122,347,395]. Similarly, for audio-text instruction-tuning data, audio captions and labels from the original dataset are fed into ChatGPT to generate QA pairs [166,370]. In particular, if the audio data contains talks or speeches, the original transcripts are also provided to ChatGPT as additional information [380,396]. Later, GPT-4V is accessible and mainly employed to directly perceive multi-modal inputs and generate fine-grained captions, diverse questions and detailed answers [351,352,359,363]. GPT-4o accepts any combination of text, audio, images, and video as input, allowing the generation based on more complex multi-modal information [156,348,354].

The general data generation process can be further optimized based on specific objectives. To acquire data of higher quality, GPT-4V and Gemini Pro can be employed to evaluate the generated data and discard samples containing hallucinatory content, meaningless questions, or erroneous answers [240,356,371]. In addition, the strong instruction-following capability of proprietary models can be utilized to generate data in specific formats such as negative instruction [360], complex questions [351], and multi-modal chain-of-thoughts [350,379,381]. Different from generating textual answers by LMMs, some multi-modal generative tools are adopted to convert textual descriptions into multi-modal elements, to meet the requirements of any-to-any LMMs [11,12].

Please notice that some datasets are constructed with both of the aforementioned methods, as presented in Table 8.

4.2.2. Training Stages

The training of current LMMs typically involve multiple stages, with each stage using different data to train specific parameters, gradually learning cross-modal alignment as well as multi-modal understanding and generation capabilities. Most LMMs undergo two main stages: pre-training and instruction fine-tuning. Some models also introduce additional training stages for learning specific capabilities.

• Pre-training

The primary goal of pre-training is to align and associate the input representations of various modalities within the multi-modal input space, enabling the backbone to uniformly model and understand inputs across modalities. Figure 5 illustrates the commonly applied settings in the pre-training phase which is described below.

Training data. As mentioned in the training data section, commonly used data include X-text pairs and multi-modal interleaved documents. To improve the pre-training efficiency, existing methods propose to further enhance the quality of pre-training data through filtering and sampling [141]. Additionally, scenario-oriented data can be considered to enhance specific capabilities of LMMs. Besides multi-modal data, text-only data can be adopted to maintain the language modeling capabilities of backbones [84,106,133].

Training objective. With the input-level alignment architecture, the training data can be constructed as multi-modal sequences. For (Output Type 1) LMMs with text-only output, models are typically required to generate textual tokens in the sequence based on information from other modalities, thereby learning semantic alignment [5,6]. For LMMs with multi-modal generation capabilities, the training objectives also include predicting information in other modalities in the sequence, whether in continuous (Output Type 2) [75,109,111] or discrete (Output Type 3) [22,112,172] formats.

Trainable parameters. For LMMs with hybrid input space of Type A, input-level alignment can be efficiently achieved by merely updating the connection module [5,6,128], while some methods further unlock the backbone [135,147] or modality encoder [7,118,143] to increase the trainable model capacity, thereby enhancing specific capabilities like in-context learning [135] and adapting to multi-modal contexts. Regarding models with unified Type B input space, most methods jointly train the extended multi-modal embeddings and the backbones [12], additional training strategies may be required to stabilize the training process [22,112]. As an exception, Ovis [153] treats the visual embedding as a separate module, training only the corresponding parameters during the pre-training phase. Meanwhile, if LMMs are designed with internal alignment modules or output mapping modules, the modules are typically updated during the pre-training stage [109,115,130,397]. When modality encoders and backbones are activated for training, LoRA [398] tuning can be adopted to retain the pre-trained knowledge and improve training stability [129,152,158].

Multi-stage pre-training. A line of studies have explored dividing pre-training into two stages, using different forms of data in each stage [84,109,152], introducing higher-quality data and task-specific data in the latter stage [7,129,130], gradually unlocking more trainable modules [7,84,153] and enhancing the input image resolution [130,143,152].

Specialized setup. Apart from general settings, some methods adopt special training strategies. InternLM-XComposer 2 [107] and mPLUG-Owl2 [132] utilize a layer-wise learning rate decay method to maintain pre-trained knowledge while updating the modality encoders. Unlike updating the entire modality encoders, ShareGPT4V [131] and TinyLLaVA [142] demonstrate that merely training the latter half of the layers is more effective and efficient. Considering the training stages, different from the aforementioned discussions, specialized pre-training stages and objectives can be used to train complex connection modules [5,114]. IDEFICS3 [158] even conducts three-stage pre-training to learn different levels of capabilities in a more refined manner.

• Instruction Fine-tuning

The instruction fine-tuning stage is a crucial step in developing LMMs as versatile AI systems. It enables the model to understand and follow instructions to generate appropriate responses, thereby enhancing the interactivity. Additionally, by fine-tuning with diverse instruction data, the stage improves the generalization capabilities of LMMs, facilitating models to handle unseen scenarios and tasks in a zero-shot manner to meet real-world requirements. Figure 6 provides a straightforward illustration for this stage.

Training data. As previously introduced in the data section, instruction-tuning data is required to be sufficiently rich. Therefore, most LMMs adopt different strategies to construct a mixed dataset based on different requirements, such as mixing task-oriented data with self-instructed data [128,152], combining general data with data from specific scenarios [122,397], unifying data from various modalities [11,12,363], integrating understanding and generation data [75,109], and blending multi-modal data and text-only data [135,144].

Training objective. Integrated with system prompts, instruction-following data is typically formatted in specific templates to form dialogue sequences. The training objective is to generate multi-modal content within the sequence. Unlike the pre-training stage, only the contents in responses are used for gradient calculation, while the user instructions and the system prompt are masked [6].

Trainable parameters. As discussed in [128], merely fine-tuning the connection module makes it difficult for LMMs to adapt to complex multi-modal contexts. Therefore, current LMMs typically update the backbones during the instruction fine-tuning stage, either through full-parameter tuning or using PEFT (parameter-efficient fine-tuning) methods like LoRA [398]. As for small-scale LMMs, researchers have found that utilizing LoRA can reduce the risk of catastrophic forgetting [141,151]. In addition, internal alignment and output mapping modules are jointly fine-tuned in this stage, if they are included in the LMM [9,109,115,130]. As for modality encoders, most methods tend to keep them frozen in this stage, while some specific models activate them to learn specific encoding knowledge [80,141,143,153].

Specialized setup. Different from the prevalent multi-stage setting, SPHINX-X [81] and PandaGPT [162] resort to a one-stage training framework and mainly utilize instruction data to jointly learn cross-modal alignment and the ability to follow multi-modal instructions. To enhance capabilities in specific scenarios, some instruction fine-tuned models are further trained with scenario-oriented data. For instance, IDEFICS2 [152] is further adapted to long conversations, while InternLM-XComposer series [107,157] mainly focus on the article composition.

• Additional Alignment Stages

In addition to the regular pre-training and instruction fine-tuning stages, some specialized models require additional training stages to achieve alignment for specific objectives.

Output-level alignment. To enable LMMs to generate multi-modal response, output-level alignment is required. Benefiting from unified multi-modal discrete representation and the pre-trained tokenizer and detokenizer for each modality, models with Type 3 output space can achieve output-level alignment directly through conventional pre-training and instruction fine-tuning [12,22,112,172]. For models with Type 2 hybrid output space, an additional alignment stage may be required. By rearranging the order of text and other-modality information in “text + X” pairs and interleaved sequences, the text-to-other-modalities generation ability can be learned in the autoregressive setting. A line of approaches keeps modality decoders frozen and train the output mapping modules through gradients passed from the decoder for alignment [103,109,110]. Since most modality decoders are originally conditioned on text for generation, the representations from the decoders’ corresponding text encoder can be utilized as supervision signal [9,11]. Another line of methods, represented by Emu series [75,111], propose to construct an autoencoder architecture between modality encoders and decoders. These methods first train LMMs to align the visual input and output spaces, then align the modality decoders to this space.

Sparse internal alignment. Specifically, LMMs integrated with MoE-based internal alignment modules typically undergo an additional sparsification stage, referred as sparse upcycling [199]. During this stage, MoE layers are added to an already established dense LMMs. The parameters of MoE layers are fine-tuned using multi-modal instruction datasets, which further facilitate modality alignment and fusion while mitigating conflicts [138,139,155,163].

We summarize several LMMs and the corresponding training settings in Table 9 and Table 10.

5. Evaluation and Benchmarks

Multimodal evaluation benchmarks are implemented to assess and compare the performance of different LMMs on various tasks. From the perspective of input-output space extension, we categorize existing benchmarks into three types. (1) Based on the input space extended to multiple modalities, modality comprehension benchmarks assess perception capabilities of LMMs across multi-modal signals, covering image-to-text, video-to-text, and audio-to-text tasks. (2) With the extension of the output space, modality generation benchmarks evaluate the abilities of LMMs to produce multi-modal outputs through images, videos, and audio generation tasks. (3) Hallucination benchmarks diagnose whether representations are well aligned between modalities. In this section, we introduce modality comprehension and generation benchmarks and Section 6 focuses on the benchmarks, tasks, and methods for hallucination diagnosis.

5.1. Modality Comprehension Benchmarks

Modality comprehension benchmarks assess the ability of LMMs to understand visual or audio inputs, typically in the form of Any-to-Text tasks, outputting with text. The general evaluation framework is presented in Figure 7.

5.1.1. Image-to-Text

Currently, a large amount of multi-modal benchmarks focus on images and text. Early benchmarks aims at solving problems in specific tasks with task-oriented forms and data, comprising visual question answering (VQA), referring expression comprehension and image captioning. Specifically, VQA tasks consist of general VQA [23,24,283,287,310], knowledge-based VQA [280,320], and text-oriented VQA focusing on text understanding capabilities in images [300,301,304,307,322]. Referring expression comprehension mainly includes RefCOCO, RefCOCO+, RefCOCOg [29] and GRIT [329], requiring models to localize object queries in images. Besides, COCO [399], NoCaps [27], TextCaps [28] are commonly evaluated for image captioning. These traditional Image-to-Text benchmarks determine whether the output is accurate by fully matching the output with the ground truth. For image captioning tasks, conventional metrics like BLEU [400], ROUGE [401], CIDEr [402], METEOR [403] are implemented to evaluate the quality of the model output, referring annotated captions. Additionally, these benchmarks may require specific output formats, such as single words or phrases, necessitating additional requirements for the model [128] or corresponding evaluation methods [404,405].

As LMMs achieve substantial progress across various downstream tasks, benchmarks specifically for their evaluation have been designed and proposed, which adapt to the free-form text output by these models, comprising multiple-choice questions and open-ended generation tasks. Concretely, input prompts for multiple-choice questions consist of designed instructions, questions, images, and options. The options select by LMMs can be detected from the generated responses through option symbols or text matching methods. MME [406] formalizes the questions into binary terms and evaluates the perceptual and cognitive capabilities of LMMs by asking the model to answer yes or no, while M3Exam [407], MMT-Bench [408], and SEED-Bench-1 [409] further expand the number of options in each question, posing greater challenges to LMMs. Additionally, MMBench [410] proposes to use LLM-based choice extractors to convert the free-form text into a specific choice (A, B, C, etc.), and MMStar [411] introduces new metrics for evaluating multi-modal gain and multi-modal leakage. Besides general scenarios, researchers have also constructed relevant benchmarks targeting specific capabilities and contexts [412,413,414,415].

Furthermore, beyond designing questions to guide LMMs to produce outputs in specific formats for evaluation, it is necessary to assess the model’s ability when responding with free-form texts in real-world scenarios (open-ended generation). Under this setting, rule-based or LLM-based evaluation pipelines are mainly employed. In rule-based cases, robust regular expressions are deployed to extract key phrases, such as numbers and conclusion phrases, from the responses for accurate answer matching, this method is adopted in MMMU [416], OCR-Bench [417], and MathVista [418]. Similarly, ReForm-Eval [405] and SciGraphQA [419] implement rule-based metrics to process free-form predictions. In addition, LLM-based evaluation methods are more suited for assessing long and complex outputs. These methods typically use powerful large models, such as ChatGPT, to score the model output based on the question description, instructions, visual information, and designed scoring ruless [251,378,420,421,422]. In addition to automated evaluation pipelines, LVLM-eHub [404] and OwlEval [118] resort to manual evaluation methods, which are directly in line with human preferences but significantly increase the evaluation cost.

5.1.2. Video-to-Text

According to the Any-to-Text paradigm, Video-to-Text comprehension benchmarks are also considered. Video reasoning [291,292,295,423] is a commonly adopted task for evaluation. In related benchmarks, QA pairs are typically generated from existing video descriptions, and the capabilities of LMMs are measured in terms of precision. Similar to image-to-text benchmarks, some benchmarks are designed in the form of multiple-choice questions for efficient and reliable evaluation [296,391,424,425,426,427]. Furthermore, researchers are also interested in the video captioning task which is evaluated using traditional metrics [233,248,428]. However, these traditional evaluation metrics rely on the exact matching between the generated and ground-truth captions, which are limited in capturing the richness of video content. Thus, the ChatGPT-assisted evaluation method is applied in VCGBench [123], VCGBench-Diverse [373], MSVC [429], and MLVU [430].

5.1.3. Audio-to-Text

Most existing Audio-to-Text comprehension benchmarks focus on task-oriented evaluation, including Automatic Speech Recognition (ASR), Automatic Speech Translation (AST), Speech Emotion Recognition (SER), Audio Question Answer (AQA), and Audio Captionning (AC). ASR tasks report a word error rate (WER) metric, where a lower number is better [254,431,432], and CoVoST2 [433] is commonly adopted for the translation task with a BLEU [400] score. Meld [434] evaluates SER and ClothoAQA [435] evaluates AQA with accuracy. AudioCaps [229] and Clotho [228] are widely used for the AC task, with CIDEr [402], SPICE [436], or SPIDEr [437] as the metric. Although these benchmarks reveal the capabilities of LMMs from multiple perspectives, they are limited to specific tasks and cannot adequately reflect performance in real-world scenarios. Therefore, AIR-Bench [438] is proposed to conduct assessment that aligns closely with the actual user interaction experience. In addition, AudioBench [439] is introduced as a comprehensive evaluation benchmark specifically designed for general instruction-following audio-language models, and Dynamic-SUPERB [440] is a benchmark that covers comprehensive diverse speech tasks, designed for building universal speech models.

5.2. Modality Generation Benchmarks

Based on the output spaces that are extended to multiple modalities, modality generation benchmarks evaluate the abilities of LMMs to generate multi-modal content. These benchmarks can be categorized into image, video, and audio generation tasks depending on the target modality.

We illustrate the general evaluation framework of generation benchmarks in Figure 8. According to different scenarios, various conditional generation tasks provide multi-modal contexts, and the generated outputs are assessed from different perspectives based on different reference information.

5.2.1. Image Generation

Conditional image synthesis involves generating visuals based on specific conditions or inputs, such as text descriptions or visual prompts, with the aim of creating high-quality, contextually accurate images that fulfill the given conditions. This process encompasses tasks such as text-to-image generation [441,442,443], image restoration [444,445], and image editing [446,447]. Furthermore, certain purely visual tasks, such as object detection, depth estimation, and segmentation, can also be framed within the context of image generation [448,449,450].

For the synthesized images, evaluations are conducted based on various forms of reference information. (1) When the target image is used as a reference, the evaluation can be carried out by measuring the differences between the generated and real images [25,451], often quantified by metrics such as FID [452], KID [453], SSIM [454], and CLIP-I [66]. MJHQ-30K [455] employs FID to evaluate the generation capability against a dataset of 30K high-quality images. Meanwhile, leveraging GPT-4V’s visual perception capabilities, VIESCORE [456] assesses the semantic alignment between reference and generated images, evaluating their adherence to input instructions. Similarly, in DREAMBENCH++[457], GPT-4o is prompted with the task definition, target image, and output image, then assigns a final score based on the alignment. (2) In cases where text serves as the reference, the alignment between generated images and the textual descriptions is measured through metrics like CLIPScore[458] and BLIP-CLIP [48]. Additionally, GenAI-Bench [459] employs VQAScore by inputting both images and formatted text questions into a VQA model, calculating the probability of the model predicting a "yes" answer. As widely used comprehensive benchmarks for image generation, T2I-Compbench [460] further utilizes MiniGPT-4 [117] with Chain-of-Thought reasoning for evaluating semantic alignment, and GENEval [461] utilizes an object detection model to identify objects in the image, then processes each bounding box through a classification model to determine the corresponding category, which is subsequently matched with the original text description. (3) When no explicit reference is available, human evaluation is often employed to assess the intrinsic diversity and clarity of synthetic images [462,463,464]. However, to mitigate the high cost of human evaluation, automated metrics like the Inception Score (IS) [465], Aesthetic Predictor’s Score (AP) are introduced for independent quality assessment. To further align with human preferences, some approaches leverage reward-model-based methods [466,467,468], while others automate the process using LMMs [469,470,471]. DPG-Bench employs mPLUG-large [472] as its adjudicator, evaluating the generated images based on specific questions and calculating scores in accordance with DSG [473]. (4) With reference information for specific dimensions such as segmentation maps, bounding boxes, key points, and depth maps, task-specific evaluations are also applicable. This process involves extracting the relevant information from the generated image and comparing it to the reference. Specifically, metrics such as mIoU [474], cIoU [475], and gIoU [474] are calculated in segmentation tasks [476,477,478]. Similarly, for detection tasks, including object detection and keypoint detection, AP and mAP serve as critical performance metrics [385,450,479]. In depth estimation, discrepancies are quantified using absolute error metrics [480,481], including Mean Absolute Error (MAE) and Root Mean Square Error (RMSE).

5.2.2. Video Generation

Similar to image generation, video generation can also be categorized into various tasks based on different generation conditions, including the commonly used text-to-video [225,428,482,483], image-to-video [484,485], and video editing [486] tasks. In addition, videos further enable temporal-related generation tasks such as future prediction, frame interpolation, and video looping.

The generated videos are evaluated from various perspectives with respect to different references. Since videos can be regarded as sequences of multiple frames, previously introduced image-level metrics, such as FID, IS, PSNR, SSIM, and CLIPSIM, can be used to assess the quality of video frames. Additionally, researchers have developed several video-level metrics. (1) Without considering the reference, the focus is on the perceptual quality of the video itself. With the help of video encoders like C3D [487], IS can be improved to Video Inception Score [488]. EvalCrafter [489] and VBench [490] respectively utilize the Dover [491] model and the LAION aesthetic predictor to assess video quality in terms of aesthetics. Similarly, Wu et al. mix several experts with different biases towards a comprehensive quality score. Meanwhile, temporal consistency is another crucial aspect of videos, which is characterized at the pixel level [489,493,494] and the semantic level [495,496]. Furthermore, several benchmarks [489,490,497,498] provide a more systematic assessment of temporal consistency, including aspects from the perspective of subjects, backgrounds, actions, and dynamics. (2) Given ground-truth target videos as references, researchers design FVD [499], adapting FID to video scenarios by utilizing the I3D video encoder [500]. There also exist several variants such as KVD [501] and UMT-FVD [483]. (3) When evaluating conditional generation, it is necessary to assess the alignment between conditions and generated videos. To measure the video-text alignment, CLIPScore [458] is widely used, while FETV [483] explores several variants of CLIPScore. EvalCrafter [489] additionally designs SD-Score and BLIP-BLEU. For fine-grained consistency evaluation, VBench [337] and EvalCrafter [489] utilize various tools for multiple-perspective assessment, whereas other researchers [483,492] build a series of QA pairs based on text descriptions for evaluation. Specifically, for edit tasks, Frame-ACC [496] can be used to determine whether effective editing has been made. Regarding the image condition, image-video conformity can be measure by PIC [502] (4) Although automated metrics are efficient, they may lose important information due to the complicated contents contained in videos, human evaluation is widely adopted for video generation evaluation [483,485,496,502]. Recently, GPT-4o is employed to reduce the need for manual intervention [498].

5.2.3. Audio Generation

Audio generation tasks involve crafting audio content either from text descriptions or existing audio recordings. This includes text-to-audio synthesis, wherein audio is generated based on textual prompts [503,504,505], as well as audio-to-audio generation, which focuses on manipulating audio by adding, removing, or replacing elements within the tracks [103,164].

Regarding the evaluation on generated audio: (1) When considering ground-truth audio as a reference, subjective evaluation is typically conducted through human scoring, which includes Mean Opinion Scores (MOS) to assess the overall quality of speech. Similarity MOS (SMOS) is used to evaluate speaker similarity between the speech prompt and the generated output, while Comparative MOS (CMOS) assesses the relative naturalness of synthesized speech compared to the original ground truth audio [506,507]. Given that subjective metrics often require considerable time and workload, various objective metrics are applied to streamline the process, such as FAD [508], KL [509], and CLAP Score [99,510], which can also be utilized for evaluating music generation quality [511,512,513]. For speech synthesis tasks, Speaker Similarity measures the consistency of timbre between the generated and prompt speech by comparing speaker embeddings, with similarity scores predicted by the speaker verification model WavLM-TDNN [514,515]. Additionally, [516] introduces the TDOA benchmark to assess spatial audio quality by computing TDOA distributions and comparing the generated audio to the ground truth using Mean Absolute Error (MAE). (2) In cases where reference text descriptions are provided, Word Error Rate (WER) and Character Error Rate (CER) are commonly used to evaluate the content accuracy of synthesized speech by calculating the distance between the transcription of the synthesized speech and the input text conditions [12,517,518]. By converting speech into text, ChatGPT Score is employed to evaluate the response quality [10], while models like LLAMA-OMNI [171] and EMOVA [519] utilize GPT-4o to evaluate transcription accuracy and score model responses. Diffsound [520] further adopts a pre-trained audio caption transformer (ACT) [521] to compute a sound-caption-based loss. (3) When only the generated audio is analyzed through discriminate models, besides the fundamental metric like IS [465], Liu et al. train a sound classifier to verify sample quality. Moreover, the UTMOS model [523] is specifically designed to predict the Mean Opinion Score (MOS) for speech, allowing for an assessment of its naturalness.

6. Diagnostics: Benchmarks for Hallucination Evaluation

Despite the powerful capabilities LMMs have demonstrated across various scenarios, there may still exists misalignment within the models. Pre-trained LMMs may not fully and accurately understand multi-modal information, leading to the risk of generating incorrect information, also known as hallucinations. Therefore, it is necessary to diagnose these symptoms and analyze the internal mechanisms of LMMs more deeply. In this section, we focus on the misalignment between texts and images, introducing benchmarks and methods utilized to diagnose hallucinations in LVLMs. Corresponding to the hallucination symptoms in description and judgement tasks, current hallucination detection methods can be classified into two major types: (1) evaluating LMMs’ capabilities of hallucination discrimination, and (2) assessing the model’s ability of non-hallucinatory content generation.

6.1. Evaluation on Hallucination Discrimination

Hallucination discrimination evaluation approach is designed to assess the ability of LMMs to distinguish between accurate and fabricated content. Methods following this approach typically adopt a question-answering format, where LMMs are asked questions based on descriptions that either align with or contradict the content of a given image, and their responses are evaluated accordingly. For instance, POPE [524] employs binary questions about the presence of objects in images to assess the hallucination discrimination capability of LMMs. CIEM [525], similar to POPE [524], automates the object selection process by utilizing ChatGPT for prompting. Another method, NOPE [526], is also VQA-based and specifically designed to evaluate the models’ ability to recognize the absence of objects in visual queries, with correct responses being negative statements.

6.2. Evaluation on Hallucination Generation

Evaluating hallucination generation aims to measure the proportion of hallucinated content in the outputs. Currently, there are primarily two main types: rule-based and model-based methods. Handcrafted rule-based methods are characterized by their strong interpretability, achieved by manually designing multiple evaluation steps with specific and clear objectives. Typical benchmarks include CHAIR [527], CCEval [528], and FAITHSCORE [529]. Additionally, AMBER [530] can be used to evaluate both generative and discriminative tasks through several rule-based metrics, enabling the detection of existence, attribute, and relation hallucinations. Model-based methods directly assess the performance of LMMs by evaluating their responses with the help of intelligent models. Based on the model used, these methods can be categorized into two types: LLM-based evaluation and hallucination-data-driven model evaluation. For LLM-based evaluation, GPT-4 is mainly employed to assess contents generated by LVLMs, focusing on hallucination levels, capitalizing on the robust natural language understanding and processing capabilities of advanced LLMs [360,531]. In practice, LLMs are prompted to evaluate and score these responses by comprehensively considering visual information with dense captions, object bounding boxes, user instructions, and model responses. Likewise, hallucination-data-driven model evaluation methods build labelled hallucination datasets for fine-tuning models to detect hallucinations [532,533].

7. Extension to Embodied Agents

Embodied AI is a rapidly advancing field of research that explores how agents develop intelligence through interaction with environments. This interaction encompasses not only the perception and understanding of the environment, but also the decision-making of future actions [534]. In this section, we will firstly introduce several categories of embodied tasks, then delve into how to adapt LVLMs to embodied tasks by extending the input-output spaces.

7.1. Embodied Tasks

Tasks are referred to as “embodied” because the agent needs to interact with a real or virtual environment. Based on the complexity of the interaction actions, we can categorize embodied tasks and corresponding datasets as follows: (1) Embodied Question Answering (EQA) [535,536]: In these tasks, the agent is required only to answer user questions. Broadly speaking, we can consider such action spaces as discrete vocabularies. (2) Vision-and-Language Navigation (VLN) [537,538]: These tasks involve navigation within an environment based on user instructions. However, this tasks does not require interactions with objects. Therefore, the action space is either discrete directional movements, such as forward, backward, left, and right, or it can involve continuous control parameters, such as speed and direction. Graphical User Interface (GUI) Navigation [539,540] is a specialized category of VLN tasks where the agent needs to operate on a computer or mobile phone based on user instructions. Action spaces are often discretized screen operations, such as clicking, swiping, and text-editing. (3) Vision-and-Language Manipulation (VLM) [541,542,543]: These tasks require the agent to not only engage in question-answer dialogues with the user, but also navigate the environment and interact with objects based on user instructions. This action space builds upon the action space of VLN tasks by adding object manipulation actions. (4) Open-World Robot Control (ORC) [544,545,546]: In these tasks, agents are equipped with high-degree-of-freedom robotic arms, capable of performing precise object manipulations, such as grasping and moving objects. Both indoor environments, such as household scenarios, and outdoor settings, such as material transport scenarios, are involved. The action space for ORC tasks is continuous, determined by the complexity of the robotic arm movements.

7.2. Input Extension: Environment Representation

Since embodied agents interact with the environment as the subject, the egocentric observation becomes an essential choice [413,537,542,547,548]. Under egocentric observations, the environment is often represented as a local image [549,550,551] corresponding to the current orientation or by rotating 360 degrees, which could be satisfactory for EQA tasks. However, VLN and VLM tasks require an integrated understanding of observed environments. When the agent operates from a single egocentric view, its understanding of the environment is inherently partial and localized. To obtain a complete picture, the agent must engage in thorough and repeated exploration of the environment [552,553]. Therefore, the ability to integrate temporal local information and transform it into a long-term global perspective is crucial for embodied agents. Several works utilize topological map [554,555] to record the spatial semantics during navigation, either for obtaining a better visual representation for the environment [556], or for constructing reasoning chains [557]. Others employ bird’s-eye-view grid maps to structure the visited environment [558,559,560]. For ORC tasks, a detailed 3D modeling of the environment is essential for executing precise actions with a robotic arm. For example, VoxPoser [561] take the 3D value map derived from interactions between a LLM and a vision-language model to enable exact and efficient object manipulations.

7.3. Output Extension: Action Representation

As stated in Section 7.1, different embodied tasks have distinct action spaces, necessitating the extensions to model outputs to accommodate the specific demands of each task.

• Discrete Action Space

For embodied tasks of VLN and VLM with discrete action spaces, embodied actions are divided into a fixed set of categories. The output of agents select one action category for execution. One line of work, i.e. LLaRP [562], utilizes an additional action prediction module specifically designed to decode discrete actions, which could generalize to unseen tasks better. Another line of work leverages the powerful language decoding capabilities of LLMs. For example, NavGPT [563] and NaviLLM [564] predict actions as plain-text, which is then parsed into specific action commands. This design simplifies the complexity of action decision, but limits the granularity of actions for complex operations like robotic arm control in ORC tasks. To mitigate this issue, RT-2 [565] introduce special action tokens into the vocabulary. These discrete tokens are then de-tokenized into continuous control signals of robot action.

• Continuous Action Space

To better adapt to ORC tasks, the extension to continuous actions is necessary. A continuous action space is represented by a set of continuous values, such as the joint angles or velocities of a robotic arm, allowing the agent to move or adjust freely within the control space. Since the direct outputs of LVLMs are discrete tokens, decoding continuous actions typically requires an extra action decoding head. RoboFlamingo [566] experiments with different action decoding head architectures (e.g., MLP, RNN, and Transformer) to enable language-conditioned robotic control. Octo [567] employs a modular framework, integrating diffusion model-based action policies to predict continuous actions. Unlike RoboFlamingo, the advantage of Octo lies in its ability to flexibly connect different task encoders, observation encoders, and action decoders, making it highly adaptable.

• Hierarchical Action Space

Hierarchical action space separates the level of action control into high-level task planning and low-level control policies (could be either discrete or continuous), each handled by separate modules or models. Specifically, PALM-E [551] and LEO [568] uses high-level instructions generated by LVLMs to guide low-level control policies in executing specific actions. LEO [568] further enhances its understanding on 3D world by utilizing a 3D encoder and crafting large-scale datasets for training.

7.4. Multi-Modal Alignment

• Input-level Alignment

To bridge the gap between the newly introduced environment representation and other modalities, SMNet [554], GridMM [560] and Trans4Map [558] employ end-to-end imitation learning, continuously adjusting the model parameters to optimize allocentric map generation and updating processes. However, these obtained map representations are highly dependent on the UNet and GRU modules nested within the model architecture, lacking the ability to transfer between different language backbones. To address this issue, Ego${}^2$-Map [556] takes a self-supervised contrastive learning strategy, comparing egocentric view features with their corresponding semantic maps. Such representations encompass rich spatial information from a map, exhibiting strong generalizable capability on various environments for both high-level and low-level action spaces.

• Output-level Alignment

Adapting the outputs to different action spaces is essential for agents to understand and execute complex tasks. There are two major strategies: (1) Direct Alignment: This approach maps instructions directly to executable actions in an end-to-end manner, as exemplified by RoboFlamingo [566] and Octo [567]. RoboFlamingo uses an MLP-based action decoder to convert hidden states into specific control signals, while Octo employs a conditional diffusion decoder as the action policy module. This decoder predicts continuous action distributions, transforming Gaussian noise into desired action outputs through a series of denoising steps. During training, both RoboFlamingo and Octo collect sequential actions covering various scenarios and tasks, enhancing the model’s generalization capability during pre-training. They also allow the policy module to be fine-tuned with a small amount of trajectory data so as to quickly adapt to new tasks. (2) Indirect Alignment: This method breaks down user instructions into language plans that can be understood by downstream models, with representative works as PALM-E [551] and LEO [568]. PALM-E pre-trains on large datasets of robotic manipulation planning, visual question answering and captioning, converting complex environmental perceptions into multi-step task planning. It integrates the task plans with SayCan [569] for specific action execution. While LEO adopts a two-stage training process involving 3D vision-language alignment and fine-tuning on 3D vision-language-action instructions, enhancing the agent’s adaptability to different action spaces.

7.5. Evaluation

• Task-Specific Benchmarks

Different embodied AI tasks involve distinct scenarios, each with specific evaluation datasets. These datasets fall under the category of modality comprehension datasets, as they involve interpreting multiple types of input, such as vision, language, and history actions. Based on the task categories in Section 7.1, task-specific datasets are: (1) For EQA tasks, representative datasets include EQA [535], IQUAD [536], and SQA3D [570]. (2) For VLN tasks, notable benchmarks include MultiON [571], R2R [537], R2R-CE [538], SOON [572], and REVERIE [548]. For GUI navigation tasks, relevant datasets involve MinoWeb++ [539], Mind2Web [573], AITZ [574], and GUI-Odyssey [575]. Specifically, there is a bilingual evaluation benchmark FunUI [576] to access the basic UI understandings of GUI agents. (3) For VLM tasks, representative datasets include ALFRED [541], TEACH [542], and HomeRobot OVMM [543]. (4) For ORC tasks, notable datasets include Franka Kitchen [544], Open X-Embodiment [546] and CALVIN [545]. Generally, for EQA tasks, the primary evaluation metric is accuracy, while for action prediction, i.e. VLN, VLM an ORC tasks, both single-step action prediction accuracy and episodic task execution success rate are used as evaluation metrics.

• Comprehensive Benchmarks

Comprehensive evaluation of agent performance across various embodied scenarios is challenging. To address this, Visual-AgentBench (VAB) [577] has been developed as a pioneering benchmark specifically designed to access LMMs as visually-grounded agents. VAB covers a wide range of environments, including embodied, graphical user interfaces, and visual design. It evaluates the understanding and interaction capabilities of LMMs through multi-task assessments and offers a hybrid trajectory training set built using methods such as program solvers, LMM agent bootstrapping, and human demonstrations to enhance the performance of LMMs in behavior cloning. Success rate is adopted as the evaluation metric.

8. Discussion and Outlook

8.0.1. How to construct multi-modal input-output spaces with discretely or continuously encoded modality signal?

Currently, mainstream LMMs follow the hybrid structure, where modality signals are continuously encoded and integrated into the text space. This method is simple yet effective, leveraging encoders like CLIP [66] and CLAP [99], which are aligned with text through large-scale pre-training, to achieve impressive performance in comprehension tasks. However, this approach introduce additional design costs for corresponding alignment modules for the input and output ends.

Meanwhile, hybrid input spaces cannot directly support multi-modal content generation. This necessitates the design of more complex output layers and decoding strategies for LMMs with multi-modal generation capabilities, leading to a significant gap between the input and output spaces.

On the other hand, the unified discrete space structure is more straightforward, supporting both comprehension and generation tasks through a unified approach (e.g., next-token prediction). However, they are currently limited by the absence of strong discrete encoders across various modalities, akin to CLIP, resulting in slightly weaker performance on comprehension tasks compared to hybrid models. Ovis [153], however, has shown that by carefully designing and expanding the visual vocabulary, discrete models can also perform well on comprehension tasks. Additionally, due to the competitive relationship between modalities, improving training stability is also a challenge that needs to be addressed for unified discrete representation models.

In conclusion, both approaches have their strengths and weaknesses, with significant room for optimization. At the same time, we believe that the current training strategies of discrete and continuous encoders are not mutually exclusive, the development and approaches of both methods can learn from each other. The research community eagerly anticipates an effective modality encoding method that unifies understanding and generation.

Furthermore, there is a noticeable granularity gap between textual and modal representations, whether the modality signals are encoded continuously or discretely. Text tokens carry explicit semantics, while individual modality tokens might only contain limited information. A single text token may correspond to multiple tokens in an image, leading to excessively long token sequences for modality signals in current LMMs. In the future, can we build modality representations that carry semantics at specific levels?

8.0.2. How to design model architectures and training strategies to align the constructed multi-modal space?

The architectures should to be designed based on the input and output space. Most LMMs are built on a backbone, usually initialized from a pre-trained LLM to gain better text understanding capabilities and initial representations. For hybrid spaces, additional design is required for input and output alignment modules. Although the LLM backbone can perform unified multi-modal modeling through training, relatively complex internal alignment modules can be introduced to model complex cross-modal interactions.

As introduced in Section 4.1, there is a variety of designs for each module, with different structures having trade-offs across various dimensions. No structure consistently performs better across different scenarios and requirements.

Regarding training strategies, most models undergo two stages: pre-training and instruction fine-tuning. The former aligns modal representations, while the latter enhances instruction-following capabilities in multi-modal scenarios. The scale and quality of training data are critical. Ensuring the data with a wide coverage and high-quality information is the key to effectively improving LMM’s performance as the data scale up. During this process, it is necessary to select appropriate parameter settings.

In summary, the design of current multi-modal alignment frameworks—including structure, data, and training parameters—remains a research problem with a vast feasible space. Since training LMMs requires extensive computation, empirical experiments demand significant computational resources. Finding ways to quickly validate the effectiveness of an optimization direction is essential. Additionally, there have already been relevant explorations to provide some general conclusions [144,152], helping researchers offer heuristic approaches to narrow down the model design space.

8.0.3. How to comprehensively evaluate LMMs based on the expanded input-output space?

Based on the extended input space, text instructions can be leveraged as the interface for evaluating the LMM’s capabilities of understanding multi-modal information through X-to-Text tasks. Specifically, depending on the task, language can be used to describe the corresponding requirements, guiding the model to respond to the questions in the desired form. Evaluation data can be constructed based on different scenarios. Combining data from multiple scenarios leads to a comprehensive evaluation of the model.

With output extension, the model can perform generation tasks in the target modality based on different multimodal contexts, such as text-to-image/video/audio generation and image/video/audio editing tasks. The generated images, videos, and audio are further evaluated according to different metrics with respect to the reference information.

Most current evaluation benchmarks aim to assess zero-shot capabilities, aligning with real-world application scenarios. Additionally, another effective way to define tasks is in-context learning (ICL), where examples can more effectively convey task requirements when it is difficult to describe them in text. However, this method has yet to be fully explored in multi-modal scenarios. At the same time, some studies have found a mutually exclusive phenomenon between zero-shot and ICL capabilities in LMMs [144].

Furthermore, two major limitations of current benchmarks are: (1) the gap between the evaluation tasks and realistic queries from users; and (2) the misalignment between the scores evaluated and human preferences from the real world, while extensive manual evaluation introduces higher costs. Defining appropriate questions and metrics can prevent LMMs from overfitting to certain scores. Moreover, it is also crucial to reveal the risks of LMMs in practical applications.

8.0.4. A promising way towards world models.

As demonstrated in Section 7, the perspective of expanding the input-output space is not limited to modalities. Any form of information and signals can be considered. By encoding them into the input-output space and aligning them through the design of model architecture and training strategies, the trained models an be applied and evaluated in downstream tasks. We believe that this framework further reveals the possibility of building models capable of understanding the physical world. The claim, “predicting the next token is to understand the world”, could be validated with a premise that the defined token space has been expanded to cover a sufficient amount of information and signals from the world.

9. Conclusion

In this paper, we summarize the current methods of large multi-modal model (LMM) construction from the perspective of input-output space extension. We further break down and provide detailed discussion of the key research problems in the construction process, including the structure of multi-modal input and output spaces, multi-modal representation alignment frameworks, and comprehensive evaluation of generated multi-modal content. Our summarizing framework is not only straightforward but also effectively encapsulates the mainstream approaches while offering potential for further extension. This paper will continue to be updated, and we hope it can provide an intuitive and comprehensive overview for related researchers and inspire future work.