Synthetic Data for Multimodal Large Language Models: A Lifecycle-Oriented Survey

1. Introduction

Multimodal large language models (MLLMs) have expanded language modeling beyond text-only interaction to visual perception, document understanding, chart reasoning, video analysis, GUI interaction, and multimodal decision making [1,2,3,4,5,6,7,8]. Their progress depends not only on model scale and architecture, but also on high-quality multimodal supervision [9,10,11,12]. Such supervision must connect language to non-textual source evidence, such as image regions, video events, document layouts, chart values, table cells, GUI states, or perceptual reasoning traces [6,7,8,13,14]. Building and verifying these signals is costly, uneven across domains, and difficult to scale. Synthetic multimodal data has therefore become an important resource for scaling MLLM training, including pretraining, supervised fine-tuning, preference optimization, and reinforcement learning [1,5,15,16,17,18,19,20].

Recent synthetic data pipelines differ from earlier augmentation techniques. Conventional augmentation usually applies local transformations while preserving inherited labels. In contrast, MLLM-oriented pipelines can construct instruction-following records, reasoning traces, alignment signals, and interaction trajectories [1,15,21,22,23,24,25,26]. Recent work spans image and video instruction data [1,5,15,16,19,26,27], document and text-rich image supervision [6,25,28], chart and table reasoning [7,13,29], and GUI or web trajectory construction [8,14,30,31,32,33]. These data are useful only when they preserve source grounding, can be reliably curated and verified, match the training objective, and remain stable after model updates. Synthetic multimodal data should therefore be studied as a lifecycle rather than as the isolated output of a generator.

Existing studies often treat this lifecycle in parts, including seed construction [6,7,15], data generation [1,5,30,34], quality control [35,36,37,38], training integration [3,4,17,18,39], and safety or evaluation risks [40,41,42,43,44]. This separation hides cross-stage failure modes: weak seeds can produce ungrounded supervision, uncalibrated curation can retain fluent but unsupported samples, poorly matched objectives can amplify shortcuts, and pipeline errors can affect downstream behavior. A unified view is needed to connect construction, generation, curation and verification, integration, and risk propagation within synthetic multimodal data pipelines.

This survey addresses this gap with a lifecycle framework organized around four production stages: Seed Construction and Condensation, Data Generation, Data Curation and Verification, and Training Integration. These stages explain how source-grounded data become candidate supervision, how candidate data is verified or repaired, and how accepted samples enter MLLM training. We further analyze Risk Propagation in Synthetic Multimodal Data Pipelines, where synthetic supervision can affect model behavior, robustness, grounding, and evaluation validity after downstream use. The overall structure of the survey is illustrated in Figure 1.

We introduce the Levels of Synthetic Data Autonomy (LSDA) as a forward-looking, pipeline-level perspective synthesized from autonomy trends in the reviewed literature. LSDA describes how data decisions across a complete pipeline are delegated from direct human control to automated components such as models, tools, verifiers, environments, and agents. It is used to analyze future directions for synthetic data pipelines, rather than to assign fixed levels to individual methods. It characterizes autonomy in pipeline execution, while reliability depends separately on grounding, verification, provenance, integration control, and downstream stability.

The main contributions of this survey are:

• An MLLM-centered lifecycle taxonomy. We organize synthetic multimodal data for MLLMs into four production stages plus a risk-propagation perspective, showing how seeds, generated supervision, curation signals, training objectives, and downstream integration jointly determine data value.
• A mechanism-level synthesis of synthetic supervision pipelines. We group existing works by their functional roles: how seeds preserve source information, how generation creates trainable supervision, how curation and verification calibrate trust, how integration changes optimization, and how risks propagate after integration.
• A risk-aware view of synthetic multimodal data pipelines. We connect data generation, curation, and integration to hallucination amplification, distribution shift, verifier bias, reward exploitation, contamination, pipeline-induced drift, and robustness degradation.

To complement these core contributions, Section 8 uses LSDA to summarize an autonomy-oriented roadmap for future synthetic multimodal data pipelines, focusing on reliable generation, curation, integration, provenance, and downstream stability.

The remainder of this survey is organized as follows. Section 2 defines the background, scope, related-survey positioning, and unified formulation. Section 3, Section 4, Section 5 and Section 6 review the four production stages. Section 7 analyzes risk propagation after synthetic supervision enters model updates, model selection, or evaluation-dependent decisions. Section 8 discusses grand challenges and future directions, and Section 9 concludes the survey.

2. Preliminaries

2.1. Background and Scope

Synthetic multimodal data for MLLMs refers to training-oriented supervision that is generated, transformed, verified, curated, or integrated through algorithmic or model-assisted processes, and that connects language to non-textual source information [9,10,21,22]. Such source information may come from visual, temporal, document, structured-data, or interaction contexts. This notion is broader than classical data augmentation. Traditional augmentation usually applies local transformations to existing samples while preserving inherited labels, whereas MLLM-oriented synthetic pipelines can construct new supervision records for instruction following, reasoning, alignment, or agentic interaction [1,7,8,15,17]. The object of analysis is therefore not a single generation method, but the lifecycle through which synthetic multimodal data becomes usable supervision for MLLM parameter updates.

• Definition. Synthetic multimodal data pipeline for MLLMs. A synthetic multimodal data pipeline is a lifecycle system that transforms seed data into training-ready supervision for MLLMs. This process encompasses four stages: seed construction and condensation, data generation, data curation and verification, and training integration.

We include a study when its synthetic or semi-synthetic output directly contributes to MLLM pretraining, continued pretraining, instruction tuning, preference optimization, reinforcement fine-tuning, trajectory training, or closely related parameter-updating procedures [3,4,17,18]. Studies centered only on benchmarks, standalone evaluation, model architecture, general media synthesis, or broad governance are discussed only when they directly affect this lifecycle through training data construction, cross-modal verification, provenance, contamination, risk propagation, or stability after integration.

This survey focuses on synthetic data for MLLMs from a lifecycle-oriented perspective, while allowing the data-generating tools to come from broader sources, including text-only LLMs, agentic systems, tool-assisted workflows, and multimodal models. Adjacent works are retained only when they clarify a mechanism needed for synthetic multimodal supervision, such as supervision bootstrapping, model-based filtering, process supervision, mixture design, trajectory construction, or synthetic-data-based training analysis. The central systems problem is how to increase data scale and automation while preserving grounding, verification, reliable integration, and downstream stability after synthetic supervision is used for model updates.

2.2. Unified Framework: Four-Stage Pipeline

As shown in Figure 1, our framework organizes synthetic multimodal data pipelines into four production stages:

1.

Seed Construction and Condensation, which builds and selects the source-bearing seeds from which synthetic expansion begins;

2.

Data Generation, which expands these seeds into candidate multimodal supervision through model-assisted, programmatic, rendered, simulated, or interaction-based generation;

3.

Data Curation and Verification, which filters, verifies, ranks, critiques, repairs, or rejects generated candidates according to quality, grounding, answerability, diversity, safety, and cross-modal consistency; and

4.

Training Integration, which determines how accepted synthetic multimodal data is serialized, mixed with human-origin data, assigned to objectives, scheduled, and used in downstream optimization.

This four-stage pipeline is the primary taxonomy of the survey because it follows the operational flow by which synthetic multimodal data is engineered for MLLMs. The stages specify where a method or design choice operates in the data lifecycle: seed construction, generation, curation, or training integration.

We also introduce the Levels of Synthetic Data Autonomy (LSDA) as a pipeline-level perspective on how synthetic data pipelines evolve as more decisions are delegated to rules, models, tools, environments, verifiers, rewards, or autonomous agents. LSDA describes the degree of delegation across a complete synthetic data pipeline, from direct human control toward more automated, externally grounded, and auditable workflows. Higher LSDA levels indicate stronger delegation; they do not by themselves imply higher data quality or greater trustworthiness, which depend on grounding preservation, independent verification, provenance, calibrated integration, and downstream stability. We use this perspective later to discuss autonomy trends, risk implications, and future research directions.

2.3. Differences from Related Surveys

Existing surveys adjacent to this topic can be grouped into several families, including general data-centric AI surveys [45,46], LLM synthetic-data and annotation surveys [21,22,23,47], MLLM data-centric surveys [9,10,11,12], model- and capability-centric MLLM surveys [48,49,50,51,52], and instruction-tuning or post-training surveys [53,54,55,56,57]. These perspectives provide useful foundations, but they usually organize the field around data-centric AI in general, text-oriented synthetic data, multimodal data resources, model architectures, capabilities, benchmarks, or downstream post-training objectives. In contrast, this survey takes synthetic multimodal data for MLLM parameter updates as the primary object of analysis and organizes related methods through a lifecycle-oriented framework. Table 1 summarizes representative surveys in each family and contrasts their organizing principles with our lifecycle-oriented perspective.

Our survey differs in three aspects. First, the primary organizing principle is the synthetic multimodal data lifecycle for MLLMs, rather than model architecture, benchmark design, capability taxonomy, or a single post-training stage. Second, the reviewed works are grouped by their functional roles in the pipeline: source-information-preserving seeds, candidate supervision generation, data curation and verification, training integration, and risk propagation. Third, the survey discusses autonomy trends in synthetic data pipelines through LSDA, describing how these pipelines move from human-curated workflows toward more automated, externally grounded, and auditable systems.

2.4. Synthetic-Data Pipeline Formulation

To make later discussions precise, we describe a synthetic multimodal data pipeline in abstract form. Let

$${\mathcal{D}}_H^{\mathrm{mm}}={\left\{(u_i,z_i,y_i,r_i)\right\}}_{i=1}^N$$

denote human-origin multimodal data, i.e., data collected from external human or real-world sources rather than generated by the synthetic pipeline. Here, N is the number of records, $u_i$ is a text-side instruction, query, or context; $z_i$ is non-textual source information such as an image, video, document, table, chart, GUI state, or scene representation; $y_i$ is the target response or label; and $r_i$ is optional metadata such as provenance, license, source, or modality information. The superscript $\mathrm{mm}$ denotes multimodal data throughout the survey.

A synthetic multimodal data pipeline starts from a seed set ${\mathcal{S}}_0^{\mathrm{mm}}$, which may be drawn from ${\mathcal{D}}_H^{\mathrm{mm}}$, manually authored, reconstructed from existing data, distilled from external corpora, or constructed by previous models. A generator G produces candidate synthetic multimodal samples:

$${\mathcal{C}}^{\mathrm{mm}}=G({\mathcal{S}}_0^{\mathrm{mm}},\mathcal{E},\mathcal{P}),$$

where $\mathcal{E}$ denotes optional external resources, environments, or tools, and $\mathcal{P}$ denotes prompts, schemas, formatting rules, task specifications, or modality constraints.

The candidates are then processed by a curation-and-verification operator V, which may implement filtering, verification, ranking, critique, debate, reward-based selection, or repair:

$${\mathcal{D}}_S^{\mathrm{mm}}=V\left({\mathcal{C}}^{\mathrm{mm}}\right).$$

In multimodal settings, V often needs to account for not only linguistic quality, but also visual grounding, image–text alignment, temporal consistency, factual support, action-state validity, safety, redundancy, and distributional coverage.

The final training set is obtained by integrating human-origin and accepted synthetic multimodal data:

$${\mathcal{D}}_{\mathrm{train}}=I({\mathcal{D}}_H^{\mathrm{mm}},{\mathcal{D}}_S^{\mathrm{mm}}),$$

where I denotes the integration procedure that covers mixing, scheduling, serialization, formatting, and objective assignment. A target MLLM M is then optimized on ${\mathcal{D}}_{\mathrm{train}}$.

This formulation highlights four recurring variables in synthetic multimodal data pipelines. The seed set determines the source support available for synthetic expansion. The generator determines how candidate data is produced beyond observed data while preserving grounding. The curation-and-verification operator determines which samples are selected, repaired, or rejected before downstream use. The integration procedure determines how accepted synthetic data is mixed, formatted, scheduled, and assigned to training objectives. These variables provide a common notation for the following sections, which review seed construction and condensation, data generation, data curation and verification, and training integration in sequence.

3. Seed Construction and Condensation

Reliable synthetic multimodal supervision begins with seeds that preserve source-side information and provide useful starting points for expansion. In the formulation of Section 2.4, this stage constructs the seed set ${\mathcal{S}}_0^{\mathrm{mm}}$ used by later generation. We consider two complementary sources: externally grounded seeds and model-assisted seeds. Seed condensation then selects compact, diverse, and high-value subsets for downstream generation.

We formulate this stage as

$${\mathcal{S}}_{\mathrm{ma}}^{\mathrm{mm}}=T_{\mathrm{ma}}\left({\mathcal{D}}_{\mathrm{src}}^{\mathrm{mm}}\right),{\mathcal{S}}_0^{\mathrm{mm}}=C\left({\mathcal{S}}_{\mathrm{ext}}^{\mathrm{mm}}\cup {\mathcal{S}}_{\mathrm{ma}}^{\mathrm{mm}}\right),$$

(1)

where ${\mathcal{D}}_{\mathrm{src}}^{\mathrm{mm}}$ denotes the multimodal source pool used for model-assisted seed construction, $T_{\mathrm{ma}}$ denotes a family of model-assisted seed-construction operations, ${\mathcal{S}}_{\mathrm{ma}}^{\mathrm{mm}}$ denotes the resulting model-assisted seeds, ${\mathcal{S}}_{\mathrm{ext}}^{\mathrm{mm}}$ denotes externally grounded seeds that preserve source-side information, and C denotes seed condensation for coverage preservation and value-aware selection.

The section follows this decomposition. Externally grounded seeds provide source information from images, documents, charts, text-rich data, and GUI interaction records (Section 3.1). Model-assisted seeds reconstruct existing sources, instantiate schema-based data, or prepare sources for task-oriented supervision forms (Section 3.2). Seed condensation selects compact and useful subsets from candidate seed pools by preserving coverage and estimating model-aware value (Section 3.3). The organization of this section is illustrated in Figure 2, followed by a summary of trade-offs and hybrid usage (Section 3.4).

3.1. Externally Grounded Seeds

Externally grounded seeds instantiate the ${\mathcal{S}}_{\mathrm{ext}}^{\mathrm{mm}}$ term in Eq. (1). Each seed preserves a source instance and the source-side information needed for later multimodal supervision:

$$s_i=(a_i,e_i),s_i\in {\mathcal{S}}_{\mathrm{ext}}^{\mathrm{mm}},$$

(2)

where $a_i$ denotes the source instance and $e_i$ denotes the preserved source information. In MLLM data construction, $e_i$ may include whole-image captions, regions, masks, object labels, document layouts, OCR text, chart tables, numerical values, GUI states, target elements, or action records. These seeds keep later captions, questions, instructions, reasoning traces, and interaction trajectories grounded in the original source.

We organize externally grounded seeds by the source information they preserve. Coarse-to-fine visual seeds preserve image-level, region-level, and dense visual information (Section 3.1.1). Document, chart, and text-rich seeds preserve layout, text, table, and numerical structure (Section 3.1.2). GUI interaction seeds preserve state–target–action information in screen-based environments (Section 3.1.3). This organization follows the information required by downstream supervision rather than dataset names. A single source may provide multiple types of source information, while its seed-stage role is to make the relevant information available before model-assisted expansion or generation.

3.1.1. Coarse-to-Fine Visual Seeds

Visual seeds are image-side starting points for synthetic multimodal data construction. This subsection organizes them by annotation granularity rather than by dataset name. Coarse image–text seeds describe the whole image and provide global scene information; region- and mask-level seeds connect language to specific image areas; dense annotation seeds further record objects, attributes, relations, text spans, masks, or dense captions. This distinction matters because later captions, questions, instructions, and reasoning examples are easier to construct and check when the seed records both what appears in the image and where the relevant content appears.

Coarse image–text seeds provide broad semantic coverage. They pair images with captions, descriptions, or global question–answer annotations, defining the visual scope from which later instructions or questions can be constructed. ShareGPT4V and PixelProse enrich image–text resources with detailed captions for caption-rich visual instruction construction [15,20,58]. Coarse seeds scale broad coverage but leave fine-grained localization, object relations, and visual details underspecified.

Region- and mask-level seeds reduce this limitation by linking language to localized image content. Kosmos-2 connects grounded text spans to visual locations, while Ferret supports referring and grounding through user-specified points, boxes, and free-form regions [59,60]. Osprey and RegionGPT provide mask- or region-based instruction and caption data for fine-grained region understanding, and URECA emphasizes unique and consistent region–caption mappings across multi-granularity regions [61,62,63]. Localized seeds expose the relevant visual area before supervision generation, reducing reliance on global image context alone.

Dense annotation seeds provide richer visual information for later data construction. Visual Genome supplies objects, attributes, relationships, region descriptions, and question–answer annotations, showing that an image seed can contain structured information beyond a single caption [64]. GLaMM introduces grounded conversations and dense segmentation-based annotations through GranD [65]. FullAnno generates fine-grained image annotations, including object categories and positions, region descriptions, text information, and dense captions [66]. DenseWorld-1M constructs dense grounded captions from entity-level masks, labels, and object-level captions [67]. Dense annotations move seed construction from whole-image descriptions to structured local records of objects, regions, masks, and relations.

3.1.2. Document, Chart, and Text-Rich Seeds

Document, chart, table, and text-rich data require seeds that preserve structure, not only visual appearance. This subsection groups such seeds into three types. Document-layout seeds retain reading order, layout regions, and OCR or textual source information. Chart-and-table seeds retain axes, legends, cells, values, and numerical relations. Text-rich image seeds retain embedded words and their connection to the surrounding visual content. The main seed-stage role is to keep these structural elements available so that later captions, questions, instructions, rationales, or training examples can be constructed from information present in the original data.

Document-layout seeds preserve page organization and text structure. Large-scale layout resources such as PubLayNet and DocLayNet provide document pages with explicit layout-element annotations [68,69]. mPLUG-DocOwl 1.5 converts public text-rich images into structure-aware text sequences and multi-grained text–box pairs, making layout and text localization available for document-oriented supervision [6]. LayoutLLM converts public document or layout datasets into structured textual or HTML-like representations [70]. Document seeds preserve both textual content and its arrangement across layout regions.

Chart-and-table seeds preserve the data structure behind visual marks. ChartInstruct anchors chart comprehension and reasoning in chart and table structure, while ChartLlama uses controlled tabular data to seed chart rendering and instruction generation [7,13]. MMC organizes large-scale chart instruction data around visual and structural chart information [29]. Chart and table seeds retain the values, labels, and structural relations that later questions and answers depend on.

Text-rich image seeds focus on images where embedded text is part of the visual content. LLaVAR uses OCR-recognized text and image captions from text-rich images to generate conversations for visual instruction tuning [28]. Text-rich seeds expose embedded words and their visual context, allowing later supervision to use readable text rather than visual appearance alone.

3.1.3. GUI Interaction Seeds

GUI interaction seeds provide starting records for synthetic data construction in screen-based environments. Their key feature is the state–target–action relation: a useful seed should record the screen state, the user instruction or intent, the relevant interface element, and the action associated with that state. This makes GUI seeds different from ordinary visual seeds. A screenshot alone only shows the interface, while a GUI interaction seed specifies what should be done on the interface and where the action should be applied.

Screen-grounding seeds connect user instructions to visible interface elements. SeeClick emphasizes GUI grounding, where models associate instructions with target elements on the screen [8]. OS-Atlas organizes GUI grounding data across operating-system interfaces and interface elements [14]. GUI grounding seeds pair screen states with target-element information, making screen images usable as interaction records rather than ordinary visual inputs.

Action-record seeds add the operation associated with a screen state. Large-scale mobile-control datasets such as Android in the Wild, AndroidControl, and AMEX pair screenshots or GUI states with natural-language instructions, target information, and action demonstrations or stepwise GUI-action chains [71,72,73]. ScreenAgent, ShowUI, and CogAgent use screenshots, visual histories, GUI grounding data, or action-related supervision for computer-control and GUI-agent settings [74,75,76]. These records encode state, instruction, target, and action as grounded units for trajectory construction, interaction instructions, and GUI-agent training examples.

3.2. Model-Assisted Seed Construction

Model-assisted seed construction instantiates the ${\mathcal{S}}_{\mathrm{ma}}^{\mathrm{mm}}$ term in Eq. (1). Given a multimodal source pool ${\mathcal{D}}_{\mathrm{src}}^{\mathrm{mm}}$, we write this process as

$${\mathcal{S}}_{\mathrm{ma}}^{\mathrm{mm}}=T_{\mathrm{ma}}\left({\mathcal{D}}_{\mathrm{src}}^{\mathrm{mm}}\right),$$

(3)

where $T_{\mathrm{ma}}$ denotes a family or composition of model-assisted seed-construction operations. A pipeline belongs to this category when learned models participate in a key seed-construction step, such as generating, organizing, annotating, converting, filtering, or verifying seed information. Auxiliary components such as parsers, code, renderers, and rule-based filters may instantiate or structure multimodal content within the pipeline. A work may contribute to more than one seed role: the source data can be externally grounded, while the construction pipeline can still use model-assisted operations to reconstruct, organize, or specialize that source information.

A structured seed unit in ${\mathcal{S}}_{\mathrm{ma}}^{\mathrm{mm}}$ contains a source instance together with explicit fields that later data construction can use, such as OCR text, text–box pairs, chart tables, layout information, region masks, object labels, dense captions, task templates, or question formats. We distinguish three recurring operation types. Reconstruction-based operation $T_{\mathrm{rec}}$ converts existing multimodal sources into clearer structured seed units. Schema-based operation $T_{\mathrm{sch}}$ instantiates controllable data from explicit construction schemas. Task-oriented operation $T_{\mathrm{task}}$ prepares source data for downstream supervision forms such as captioning, visual question answering, dialogue, reasoning, or interaction. A concrete $T_{\mathrm{ma}}$ pipeline may use one of these operation types or compose several of them. Model assistance increases scale or structure, while the resulting seed remains tied either to source information or to an explicit construction schema.

3.2.1. Reconstruction-Based Seeds

Reconstruction-based operation $T_{\mathrm{rec}}$ turns existing multimodal sources into structured seed units. It makes layout, OCR text, chart tables, text–box pairs, regions, masks, object labels, and dense captions explicit. Rather than creating new visual content, it parses, reorganizes, or enriches information already tied to the source data. This representation keeps details that plain image captions often omit.

For documents and text-rich images, $T_{\mathrm{rec}}$ exposes text and localization-related information through model-assisted conversion from visual-text sources to supervision-ready records. LLaVAR uses OCR-recognized text and image captions from text-rich images to prompt GPT-4 to generate conversations for visual instruction tuning [28]. Model-assisted reconstruction turns existing visual-text sources into structured seed units that preserve information ordinary captions may miss.

For charts and structured visual data, $T_{\mathrm{rec}}$ preserves the relation between the visual form and the underlying data. ChartInstruct generates chart-specific instruction data around chart images and chart/table structure [7]. Chart seeds retain the rendered figure together with the table values and structural information needed for later chart questions and reasoning examples.

For local visual source information, $T_{\mathrm{rec}}$ turns image regions and dense annotations into usable seed units. GLaMM builds GranD through an automated annotation pipeline, producing grounded concepts and segmentation masks for grounded conversation generation [65]. FullAnno uses a cascade annotation process with expert models and LLM prompts to generate fine-grained image annotations, including object categories and positions, region descriptions, text information, and dense captions [66]. DenseWorld-1M uses a multi-stage labeling pipeline and VLM models to obtain entity-level masks and labels, generate object-level captions, and merge them into spatial and relational dense captions [67]. Model-assisted annotation pipelines convert image content into structured records of regions, objects, masks, captions, and relations.

3.2.2. Schema-Based Seeds

Schema-based operation $T_{\mathrm{sch}}$ constructs seeds from explicit construction schemas. Here, a construction schema denotes a structured construction plan that defines a controllable multimodal generation space. Instead of starting only from existing data, the pipeline first defines planned visual or multimodal conditions, such as a chart type, table, visual encoding, rendered data type, layout, scene structure, or task setting. Learned models may generate or organize tables, layouts, scene graphs, task settings, code, or instructions, while code and renderers may execute the schema-guided instantiation, such as plotting charts, rendering layouts, or creating structured visual scenes. This mechanism is useful when the goal is to cover planned visual or multimodal conditions rather than only reuse naturally collected sources.

For rendered and structured visual data, Multimodal Self-Instruct uses language models and code capabilities to synthesize abstract visual scenarios, including charts, tables, maps, dashboards, flowcharts, relation graphs, layouts, and puzzles, before constructing visual reasoning instructions [34]. For charts, ChartLlama separates tabular data generation, chart rendering, and instruction-tuning data design [13]. ChartInstruct organizes chart instruction data around chart images and chart/table structure [7]. ECD modularizes synthetic chart generation, visual diversification, quality filtering, and QA-pair construction for chart understanding [77]. These pipelines use explicit tables, chart structures, construction schemas, and model-assisted generation or filtering to guide seed construction.

For 3D multimodal data, SceneVerse extends schema-based construction beyond 2D data by using scene-graph generation and object-relation structure to organize large-scale 3D vision-language pairs [78]. Object, relation, layout, and scene schemas give schema-based seeds controllable coverage across data types, visual encodings, spatial structures, and task forms. Grounding, answerability, and quality still depend on how the instantiated seeds are expanded and verified.

The main advantage of schema-based seeds lies in their controllability: they can systematically vary data types, layouts, visual encodings, scene structures, and task forms. However, their effectiveness still depends on whether the instantiated seeds remain grounded, answerable, and useful after being expanded into training samples.

3.2.3. Task-Oriented Seeds

Task-oriented operation $T_{\mathrm{task}}$ prepares a multimodal source for a specific supervision form. Unlike schema-based construction, which controls what data or structured setting is created, task-oriented construction controls how the data is used: captioning, visual question answering, multi-turn dialogue, reasoning, or interaction. The same image, chart, or rendered instance can therefore seed different supervision samples depending on the target task form.

For caption-oriented supervision, ShareGPT4V collects GPT-4V-generated high-quality captions and expands them into highly descriptive image captions for large multimodal models [15]. Caption-oriented seeds prepare image sources for caption-rich instruction construction.

For instruction-oriented supervision, ALLaVA uses strong proprietary models to generate fine-grained image annotations for vision-language alignment and complex reasoning VQA pairs for visual instruction fine-tuning [16]. Instructify converts available captions, bounding boxes, and QA metadata into visual instruction conversations with quality control [79]. Instruction-oriented seeds turn image metadata and annotations into task-ready visual instruction samples.

For reasoning-oriented supervision over generated data, Multimodal Self-Instruct pairs synthesized abstract visual data with visual reasoning instructions [34]. Task-oriented seeds define whether a source instance is described, questioned, discussed, reasoned over, or used in an interaction setting.

3.3. Seed Condensation

Seed condensation instantiates the C term in Eq. (1). It refines externally grounded and model-assisted seeds into the final seed set ${\mathcal{S}}_0^{\mathrm{mm}}$ used for downstream generation:

$${\mathcal{S}}_0^{\mathrm{mm}}=C\left({\mathcal{S}}_{\mathrm{ext}}^{\mathrm{mm}}\cup {\mathcal{S}}_{\mathrm{ma}}^{\mathrm{mm}}\right),|{\mathcal{S}}_0^{\mathrm{mm}}|\le |{\mathcal{S}}_{\mathrm{ext}}^{\mathrm{mm}}\cup {\mathcal{S}}_{\mathrm{ma}}^{\mathrm{mm}}|,$$

(4)

where ${\mathcal{S}}_{\mathrm{ext}}^{\mathrm{mm}}$ denotes externally grounded seeds, ${\mathcal{S}}_{\mathrm{ma}}^{\mathrm{mm}}$ denotes model-assisted seeds, and C denotes one or more condensation operations.

In this survey, we focus on two types of condensation. Coverage selection keeps a compact subset that represents the visual, semantic, task, or dialogue space of the original pool. Model-aware value selection estimates which seeds are useful for a target MLLM, task, or fine-tuning setting. Condensation reduces scale while controlling redundancy, coverage loss, and poorly targeted expansion in later synthetic supervision.

3.3.1. Coverage Selection

Coverage selection keeps a compact subset that still represents the intended seed space. A candidate pool may overrepresent common objects, simple layouts, short captions, frequent GUI actions, easy chart patterns, or common visual reasoning forms. Coverage-oriented condensation therefore reduces redundancy while preserving visual, semantic, task, and dialogue diversity.

For visual instruction tuning, several methods select compact subsets that preserve visual-language coverage. ScalSelect extracts instruction-relevant visual representations from the target VLM and selects samples that approximate the dominant subspace of the full dataset [80]. PRISM uses Pearson-correlation-based visual encoding signals to identify high-value visual instruction samples without proxy models or gradient optimization [81]. CoIDO formulates visual-instruction selection as a joint importance–diversity optimization problem [82]. Filter Images First selects representative unlabeled images before expensive instruction generation, reducing both instruction-generation and fine-tuning cost [83]. A condensed seed subset should preserve the visual and semantic space rather than only collect individually clean examples.

Coverage selection also applies to instruction and dialogue structure. DataTailor selects multimodal instruction data using informativeness, uniqueness, and representativeness [84]. MDS extends selection to multi-turn dialogue instruction tuning, where the unit of selection is a whole conversation and coverage includes user-query trajectories and dialogue structure [85]. Coverage therefore includes image diversity, task distribution, supervision format, and conversational structure.

3.3.2. Model-Aware Value Selection

Model-aware value selection asks which seeds are most likely to improve a target model under a specific training budget, task, or fine-tuning objective. This differs from coverage selection. A seed may be diverse and structurally valid, but still contribute little to the target MLLM. Conversely, a small number of selected seeds may be valuable if they expose weaknesses of the current model, match the desired capability, or provide difficult but learnable supervision.

For LVLM and MLLM fine-tuning, value-oriented selection can use task difficulty, necessity, informativeness, or model-dependent signals. TIVE combines task-level difficulty estimation with instance-level gradient influence, allowing a small portion of visual instruction data to match or exceed full-data fine-tuning in multiple LVLM settings [86]. MLLM-Selector combines necessity and diversity for visual instruction tuning by using a seed model to estimate samples important for MLLM performance [87]. DataTailor treats informativeness, uniqueness, and representativeness as joint signals for selecting multimodal instruction data [84]. Seed value depends on how a sample interacts with the target model and training objective, not only on its diversity.

Reasoning-oriented multimodal data ties value selection to the target capability. Truth in the Few selects high-value multimodal reasoning samples by identifying examples with strong reasoning activation potential rather than relying on generic visual-language diversity alone [88]. Seeds for visual reasoning, chart understanding, dialogue behavior, or GUI interaction should be selected according to the capability they are expected to improve. Model-aware value selection is most useful when generation and training budgets are limited and retained seeds must provide the greatest expected benefit for a specific MLLM capability.

3.4. Summary and Discussion

A reliable synthetic multimodal pipeline requires seeds that are both grounded and scalable. Externally grounded seeds provide source information fields such as image regions, document layouts, chart values, text-rich content, and GUI state–target–action records. Model-assisted seeds reconstruct existing sources, instantiate schema-based data, or prepare sources for task-oriented supervision. Seed condensation selects compact subsets by preserving coverage and estimating model-aware value. These three families address different bottlenecks and provide complementary forms of seed quality, as summarized in Table 2.

The main design trade-off is between fidelity, scalability, and coverage. Externally grounded seeds provide stronger source information but are more costly and may under-cover long-tail objects, rare layouts, multilingual text, or uncommon interaction patterns. Model-assisted seeds improve scale by parsing, filtering, rendering, formatting, or annotating source data, but they can inherit source errors and bias the seed pool toward selected task formats. Seed condensation reduces cost and redundancy, but aggressive selection can remove rare or difficult seeds that matter for generalization. Seed quality depends on whether retained seeds preserve the source information, structure, coverage, and target capabilities needed by the downstream MLLM pipeline.

A robust seed pipeline combines the three families. Externally grounded seeds provide the source-information base, model-assisted seeds make this information scalable and task-ready, and condensation selects the subset that best balances coverage, cost, and expected training value. The seed pool then serves as the starting point for generating candidate synthetic multimodal supervision.

4. Data Generation

Data generation expands the seed set ${\mathcal{S}}_0^{\mathrm{mm}}$ into candidate multimodal supervision for later curation and training integration. Following the formulation in Section 2.4, we write this stage as

$${\mathcal{C}}^{\mathrm{mm}}=G({\mathcal{S}}_0^{\mathrm{mm}},\mathcal{E},\mathcal{P}),$$

(5)

where G denotes one or more generation operations, $\mathcal{E}$ denotes optional external resources, environments, or tools, and $\mathcal{P}$ denotes prompts, schemas, formatting rules, task specifications, or modality constraints. The object of analysis is the trainable multimodal record produced by generation, such as an image–question pair, a region-grounded instruction, a temporal video QA example, a chart reasoning task, an interleaved multi-image dialogue, a GUI or web action trace, a tool-use trajectory, or a reasoning trace.

We organize G by the type of candidate supervision it produces: visual instruction generation (Section 4.1), temporal supervision generation (Section 4.2), structured visual data generation (Section 4.3), multi-image dialogue generation (Section 4.4), and agentic interaction trajectory generation (Section 4.5). The main design question is how to expand seeds at scale while keeping generated samples tied to their source information, such as image content, temporal events, document or chart structure, and GUI or web state–action information. The organization of this section is illustrated in Figure 3, and Figure 4 shows representative mechanisms for each generation type. Table 3 maps each generation operation to its generated samples and required source information.

4.1. Visual Instruction Generation

Visual instruction generation instantiates G in Eq. (5) as an image-centered expansion operation. In this subsection, ${\mathcal{S}}_0^{\mathrm{mm}}$ refers to image-side seeds such as images, captions, regions, grounded spans, scene representations, or auxiliary knowledge. The resulting ${\mathcal{C}}^{\mathrm{mm}}$ consists of candidate visual-language supervision records, including captions, visual questions, instructions, conversations, image–dialogue pairs, or reasoning examples. The key requirement is that the generated language remains tied to observable visual source information, such as objects, attributes, relations, regions, or grounded visual spans.

Teacher-generated visual instruction data provides the basic pattern. LLaVA uses GPT-4-generated language–image instruction-following data for visual instruction tuning, establishing a widely used recipe for converting image-side context into conversations and reasoning-oriented supervision [1]. StableLLaVA extends this pattern by synchronously synthesizing images and dialogues with ChatGPT and text-to-image generative models, showing that visual instruction data can be scaled through jointly generated image–dialogue pairs [89]. ShareGPT4V and ALLaVA further show that GPT-4V-style supervision can expand high-quality captions, fine-grained image annotations, and complex visual question-answering data for LMM/LVLM training [15,16,19]. MMEvol treats visual instruction data as evolvable supervision: starting from seed instructions, it increases instruction diversity, extends visual reasoning steps, and explores fine-grained image information [24]. Together, these pipelines expand image-side seeds into trainable language supervision while keeping the generated records connected to image content.

Programmatic and knowledge-augmented generation add more control to this process. ProVision uses scene graphs and human-written programs to synthesize vision-centric instruction data, making the generation process more interpretable and controllable [90]. SK-VQA generates large-scale image-matched external context and QA pairs for context-augmented multimodal question answering, showing that generated supervision can combine visual source information with auxiliary knowledge [91]. VisualWebInstruct similarly uses web search, HTML processing, filtering, and synthesis to construct reasoning-focused multimodal instruction data from image-side seeds and web sources [92]. Related visual-instruction and synthetic image-text efforts further explore synthetic pairing, image-only prompting, and richer visual-language supervision [93,94]. Across these pipelines, generated answers should depend on the image or explicitly linked visual source information.

The strength of visual instruction generation is scale: teacher models, programs, and synthetic pairing can quickly expand image seeds into diverse captions, questions, and conversations. Its main risk is grounding drift. A generated question may be fluent and useful-looking while relying on language priors or auxiliary text rather than on the image. This makes visual instruction generation dependent on grounded seeds before generation and answerability checks after generation.

4.2. Temporal Supervision Generation

Temporal supervision generation instantiates G in Eq. (5) as a temporal expansion operation. In this subsection, ${\mathcal{S}}_0^{\mathrm{mm}}$ refers to video, audio-visual, or long-form temporal seeds, including frames, clips, event sequences, dense video descriptions, or hierarchical temporal summaries. The resulting ${\mathcal{C}}^{\mathrm{mm}}$ consists of candidate temporal supervision records, such as video captions, temporal QA pairs, video dialogues, multiple-choice video QA, or timestamped event-centered supervision. Compared with single-image generation, this operation must preserve event order, state changes, object persistence, camera motion, and causal relations.

Caption-centered video pipelines illustrate the short- and medium-video case. ShareGPT4Video constructs dense video captions with GPT-4V and captioning strategies designed for temporal change, intra-frame detail, and variable video length [27]. LLaVA-Video builds a synthetic video instruction dataset containing detailed captioning, open-ended QA, and multiple-choice QA tasks [5]. Video-ChatGPT constructs video-instruction pairs through a manual and semi-automated pipeline for video-based conversation modeling [95]. These pipelines first summarize or describe raw video information and then use the resulting textual scaffold to construct candidate supervision.

Long-form video generation makes temporal compression more explicit. LongViTU constructs long-form video QA data with hierarchical video organization, self-revision, long-term context, and timestamp labels for relevant events [96]. ReWatch-R1 further introduces agentic synthesis for video-grounded reasoning traces, constructing caption, QA, and CoT-style temporal supervision through a multi-stage pipeline [26]. Other long-video instruction efforts similarly rely on intermediate temporal organization before QA construction [97]. The key issue is which clips, events, and relations are preserved as trainable supervision.

Temporal generation expands MLLM training beyond static perception, but it is fragile. If decisive frames, audio-visual cues, or state transitions are omitted from the intermediate summary, later questions may be plausible but weakly grounded. A useful temporal record should preserve the clip, event, or temporal relation that supports the generated answer.

4.3. Structured Visual Data Generation

Structured visual data generation instantiates G in Eq. (5) as a structure-preserving expansion operation. In this subsection, ${\mathcal{S}}_0^{\mathrm{mm}}$ refers to documents, charts, tables, rendered diagrams, dashboards, layouts, maps, OCR or layout records, table data, plotting scripts, or renderer specifications. The resulting ${\mathcal{C}}^{\mathrm{mm}}$ consists of candidate structured visual supervision records, such as document QA, chart reasoning tasks, chart-image–code pairs, plot QA, or rendered visual reasoning examples. These records depend on latent structure rather than visual appearance alone, so useful generation should preserve intermediate representations such as OCR text, layout structure, table values, plotting code, or renderer specifications.

Chart generation illustrates how intermediate structure links rendered appearance with trainable supervision. ChartGen starts from seed chart images, reconstructs them into executable plotting scripts with a VLM, uses a code-oriented LLM to augment the scripts, and then produces synthetic chart-image–code pairs with multiple chart types, plotting libraries, and data modalities [98]. PlotQA builds large-scale question-answering supervision over scientific plots, making chart structure and underlying values part of the supervision signal [99]. Together, these works show that structured intermediates, such as tables, plotting scripts, or data values, can preserve the link between visual marks and the source information needed for reasoning.

Rendered data follow the same principle. Multimodal Self-Instruct uses language models and code capabilities to synthesize abstract visual scenarios, such as charts, tables, maps, dashboards, flowcharts, relation graphs, layouts, and puzzles, and then constructs visual reasoning instructions from them [34]. CoSyn further shows that code-guided rendering can scale synthetic text-rich multimodal data and instruction-tuning records [25]. ChartCoder provides related source information from the chart-to-code direction, where large-scale chart-to-code data preserves dense chart information in executable code form [101]. Follow-Your-Instruction further extends structured generation toward world-data synthesis by using MLLM-based agents to construct 2D, 3D, and 4D data through asset collection, layout construction, semantic refinement, and temporal planning [100]. Related chart and rendered-data studies also use chart structure, generated plots, or refined visual representations to construct supervision for chart understanding and reasoning [77,99,102,103,104].

The strength of structured visual data generation is controllability: tables, code, layouts, and renderers expose the structure that later answers depend on. The main risk is template bias. A renderer or program generator may produce visually valid data with stronger regularities than those in human-origin data, causing the MLLM to learn artificial cues rather than robust multimodal source information use. Structured visual data generation is most useful when the generated output remains coupled to checkable structure.

4.4. Multi-Image Dialogue Generation

Multi-image dialogue generation instantiates G in Eq. (5) as a multi-context expansion operation. In this subsection, ${\mathcal{S}}_0^{\mathrm{mm}}$ refers to multiple visual inputs, correlated images, ordered image sets, or interleaved image–text seeds. The resulting ${\mathcal{C}}^{\mathrm{mm}}$ consists of candidate multi-image supervision records, such as interleaved conversations, comparison QA, co-reference tasks, multi-image reasoning examples, or multi-turn dialogues. This operation differs from single-image instruction generation because the answer may depend on relations among images, including comparison, co-reference, temporal ordering, image selection, or reference tracking across turns.

TextBind and MANTIS provide representative examples of this family. TextBind generates multi-turn interleaved multimodal instruction-response conversations from image-caption pairs [105]. MANTIS constructs multi-image instruction data for skills such as co-reference, comparison, reasoning, and temporal understanding [106]. SMIR adds a synthetic data-generation pipeline for multi-image reasoning, where correlated images and complex reasoning instructions are used to create synthetic training samples [107]. The generation target is a structured conversation or reasoning record over multiple visual contexts, rather than a larger collection of single-image captions.

Other interleaved, multi-image reasoning, and grounding efforts further extend visual conversations and multi-image supervision [108,109]. Across this family, the important information is not only the answer text, but also which image each turn refers to, how the images are ordered, and how the response depends on cross-image relations.

The strength of multi-image generation is realism: users often compare images, refer back to earlier visual source information, and reason across multiple views. Its main risk is serialization fragility. If image order, role markers, or reference spans are underspecified, the generated record may not teach the intended multimodal relation even when the answer text is fluent.

4.5. Agentic Interaction Trajectory Generation

Agentic interaction trajectory generation instantiates G in Eq. (5) as an interaction-centered expansion operation. In this subsection, ${\mathcal{S}}_0^{\mathrm{mm}}$ refers to GUI screens, web states, user intents, DOM or accessibility information, target elements, tool states, simulator states, or partial interaction traces. The resulting ${\mathcal{C}}^{\mathrm{mm}}$ consists of candidate interaction supervision records, such as GUI or web trajectories, screenshot–intent–action traces, tool-use records, task refinements, or outcome-labeled interaction records. This operation belongs to multimodal data generation because the training signal combines visual interface content, language instructions, structured state information, actions, and task outcomes.

Web and GUI trajectory pipelines illustrate this mechanism. Explorer synthesizes large-scale multimodal web trajectories through exploration and refinement, producing successful web trajectories with screenshots and web elements for training multimodal web agents [30]. OS-Genesis reverses the conventional GUI trajectory collection process: agents first perceive environments and perform step-wise interactions, and tasks are then retrospectively derived from these trajectories, with a trajectory reward model used for quality control [31]. Recent web-agent pipelines construct synthetic supervision through task refinement, trajectory refinement, or scalable WebUI trajectory synthesis [32,33]. Agentic interaction becomes a data-generation mechanism when it outputs screenshot–intent–action traces or web trajectories that can be reused for training.

Additional GUI and web pipelines show that task generation, browsing traces, tutorial-derived actions, and interaction data can serve as candidate supervision for MLLM-based agents [110,111,112,113,114,115]. These pipelines turn interactive behavior into records that connect state, instruction, action, and outcome.

The strength of agentic trajectory generation is that it exposes executable behavior rather than static perception alone. Its main risk is action-state mismatch. A trajectory may look plausible but contain infeasible actions, missing interface elements, or unreliable state transitions. Useful interaction records should preserve not only the final action, but also the source information showing that the action was possible in the observed environment.

4.6. Summary and Discussion

Data generation expands seed pools into candidate multimodal supervision for MLLM training. This section reviewed five generation families: visual instruction generation, temporal supervision generation, structured visual data generation, multi-image dialogue generation, and agentic interaction trajectory generation. As summarized in Table 3, these families differ not only in output format, but also in the source information that each generated sample must preserve. Visual instruction generation relies on image-side evidence; temporal supervision generation preserves events and temporal relations; structured visual data generation uses OCR, layout, tables, code, or renderers; multi-image dialogue generation depends on cross-image relations and reference tracking; and agentic interaction trajectory generation connects states, intents, actions, and outcomes.

This mapping also exposes a shared risk: as generation pipelines scale, generated samples may become fluent or diverse while losing source information, cross-modal consistency, or grounding. Image-centered generation may drift away from observable visual content. Temporal generation may omit decisive events during sampling or summarization. Structured visual data generation may introduce template or renderer bias. Multi-image generation is sensitive to image order, reference markers, and serialization. Agentic trajectory generation depends on action validity, state transitions, and environment coverage. These differences show that generated samples should be evaluated not only by fluency or diversity, but also by whether the source information needed for the target answer, action, or reasoning step remains available.

Data generation should therefore be treated as a middle stage of the synthetic multimodal data lifecycle rather than as an isolated capability. Seed construction determines what source information can be expanded; data generation converts this information into candidate supervision; data curation and verification checks whether the candidate is grounded, answerable, executable, or repairable; and training integration determines whether the accepted record improves the model after parameter updates. As generation pipelines scale, the key challenge shifts from producing more samples to preserving source information, maintaining cross-modal consistency, and ensuring grounding.

5. Data Curation and Verification

Data curation and verification converts candidate multimodal supervision into reusable signals for MLLM parameter updates. Following the formulation in Section 2.4, this stage processes the candidate pool ${\mathcal{C}}^{\mathrm{mm}}$ and produces curated synthetic data:

$${\mathcal{D}}_S^{\mathrm{mm}}=V\left({\mathcal{C}}^{\mathrm{mm}}\right),$$

(6)

where V denotes one or more curation and verification operations. The output may include retained samples, rejected candidates, ranked records, repaired responses, preference pairs, feedback records, judge scores, or validated interaction trajectories.

The organization of this section is illustrated in Figure 5. We organize V by the type of curation signal it produces: quality filtering and data selection (Section 5.1), grounding verification (Section 5.2), model-based judging (Section 5.3), preference and feedback construction (Section 5.4), critique and repair (Section 5.5), and GUI or web trajectory validation (Section 5.6). These operations correspond to six roles: selecting candidates, checking source support, judging open-ended quality, constructing feedback signals, repairing flawed records, and validating stateful interaction trajectories. The main design question is how to scale these curation signals while preserving grounding, reliability, and action validity before synthetic multimodal data enters training. Table 4 summarizes the curation and verification mechanisms and indexes the cited works.

5.1. Quality Filtering and Data Selection

Quality filtering and data selection instantiate V in Eq. (6) as low-cost selection operations over ${\mathcal{C}}^{\mathrm{mm}}$. They produce retention, rejection, ranking, or routing signals before more expensive grounding verification, model-based judging, repair, or human inspection. Their role is to remove obvious noise, reduce redundancy, and route uncertain or high-value samples to stronger curation mechanisms. We discuss three filtering signals: pair-level compatibility, dataset-scale filtering policy, and MLLM-oriented data-value selection.

Pair-Level Compatibility

Pair-level compatibility checks whether a candidate multimodal pair is broadly aligned before deeper verification. Image–text curation provides the clearest transferable template. CLIPScore provides a lightweight image–caption compatibility signal by using CLIP to score image–caption alignment without human-written reference captions [35]. Such compatibility scores support early rejection and routing for obvious mismatches in large candidate pools.

Dataset-Scale Filtering Policy

Dataset-scale filtering policy extends pair-level compatibility into corpus construction. DataComp studies multimodal dataset construction by fixing the CLIP training and evaluation procedure while comparing filtering techniques and data sources over a large image–text candidate pool [36]. MetaCLIP provides a metadata-guided view of curation by constructing a balanced subset from a raw image–text pool according to metadata distribution [116]. Data Filtering Networks study learned filtering models for selecting data from large uncurated image–text pools [117]. UniFilter extends learned filtering to caption and interleaved multimodal data by training a unified multimodal data-quality classifier with semi-synthetic quality labels [118]. Dataset-scale filtering shapes the quality, balance, and usefulness of the resulting multimodal corpus.

MLLM-Oriented Data-Value Selection

For generated MLLM supervision, filtering moves from generic pair quality to task-oriented data value. Visual-centric data selection with collaborative agents selects visual instruction data according to image quality and image–instruction relevance [119]. Pre-instruction data selection selects useful images before instruction generation, reducing the cost of visual instruction-tuning data construction [83]. CoIDO further shows that MLLM-oriented filtering can jointly consider sample usefulness and candidate-set coverage [82]. These signals estimate whether a sample is informative, relevant, diverse, or useful for instruction tuning or related training.

Overall, quality filtering and data selection form the lowest-cost component of V. Low-quality or redundant samples can be removed, clearly useful samples can be retained, and uncertain samples can be passed to grounding verification, model-based judging, feedback construction, or repair. Their residual risk is shallow acceptance: visually plausible but unsupported synthetic samples may still enter training if filtering scores are treated as correctness signals.

5.2. Grounding Verification

Grounding verification instantiates V in Eq. (6) as the source-support checking component over ${\mathcal{C}}^{\mathrm{mm}}$. It examines whether a generated candidate is supported by the source information it uses and produces answerability, consistency, or grounding signals for ${\mathcal{D}}_S^{\mathrm{mm}}$. In MLLM data curation, such source information may include image regions, object presence, visual attributes, OCR spans, document layouts, chart tables, numerical values, temporal events, or GUI states. We discuss three grounding signals: visual source support, structured source support, and attribution-style diagnosis.

Visual Source Support

Visual source support checks whether image content supports generated captions, answers, or rationales. TIFA provides a mechanism-level formulation from text-to-image evaluation by decomposing faithfulness into visual questions and checking whether the image supports the expected answers [120]. Hallucination-oriented methods address a closely related failure mode. HalluciDoctor and M-HalDetect study hallucination in visual-language outputs or visual instruction data [40,41]. POPE provides a diagnostic reference for object-hallucination evaluation in LVLMs, while LURE analyzes object hallucination in LVLM-generated descriptions and studies post-hoc rectification [42,121]. These works show that fluent but visually unsupported candidates should be detected before they become training targets.

Structured Source Support

Structured source support checks whether text-rich, document, chart, table, or numerical information supports the candidate. For charts, DePlot illustrates a tool-like route in which plots are translated into table-like representations, making some chart claims checkable through structured source information rather than free-form perception alone [122]. OCRBench provides diagnostic coverage for OCR-related capabilities, including text recognition, scene-text VQA, document-oriented VQA, key information extraction, and related tasks [123]. These works show that document and chart candidates should be checked against textual, tabular, or numerical source information before acceptance.

Attribution-Style Diagnosis

Attribution-style diagnosis provides a complementary grounding signal. Instead of checking only whether the final answer appears plausible, attribution examines whether the response or reasoning process is connected to the relevant source information during answer production [124]. In a curation pipeline, such attribution signals provide diagnostic support for detecting weak grounding, especially when language priors or shortcuts can produce fluent answers.

Overall, grounding verification checks source support within V. Visual source-support checks target image-grounded claims; structured source-support checks target OCR, document, chart, table, or numerical fields; and attribution-style diagnosis checks whether responses are connected to relevant source information. These signals are complementary, and robust curation combines modality-specific checks across visual, textual, structured, temporal, and interaction sources.

5.3. Model-Based Judging

Model-based judging instantiates V in Eq. (6) by using learned LLM, VLM, or MLLM judges to assign scores, critiques, rankings, or preference-like labels to generated multimodal candidates. It provides open-ended quality assessment for aspects such as helpfulness, faithfulness, completeness, instruction following, comparative quality, and usefulness as supervision. Judge outputs are most reliable when they are used together with source-support checks, calibration data, disagreement analysis, and audits of judge behavior.

Existing model-based judging methods can be organized into three forms. Rubric-based judging makes evaluation criteria explicit and scores candidates against those criteria. Critic-based judging trains a generalist multimodal critic to evaluate, compare, or explain candidate responses. Reasoning-enhanced judge construction improves the judging component through reasoning supervision, synthesized response candidates, or self-generated judge-training data. The first two forms describe how judgment signals are produced, while the third describes how the judge is constructed or improved.

Rubric-Based Judging

Rubric-based judging produces structured scores for V by making evaluation criteria explicit. This form is useful when generated multimodal candidates must be evaluated for helpfulness, faithfulness, instruction following, grounding, or rubric satisfaction. Prometheus-Vision provides a representative example by training a VLM evaluator with customized score rubrics for fine-grained evaluation of VLM responses [37]. In a synthetic-data curation pipeline, rubric-conditioned judging can help rank, route, or critique generated candidates. Its coverage depends on how well the rubric captures source-specific errors, such as OCR mistakes, chart-value errors, temporal inconsistencies, or fine-grained visual-grounding failures.

Critic-Based Judging

Critic-based judging produces broader critique, comparison, ranking, or reward-like signals for V. This form trains a multimodal critic to assess candidate responses across diverse tasks and criteria. LLaVA-Critic is relevant as a generalist LMM evaluator for multimodal responses, and it also shows how critic outputs can support preference learning [38]. In the curation stage, critic-based judging can help identify better responses, explain weaknesses, or produce signals that may later be converted into preference or reward data. A central concern is model dependence: the critic may inherit visual blind spots, preference bias, verbosity bias, or generator-specific artifacts, making calibration and source-support checks important.

Reasoning-Enhanced Judge Construction

Reasoning-enhanced judge construction improves the reliability and scalability of the judge used inside V. This form focuses on how the judging component is built or improved when expert annotations and human preference labels are limited. Flex-Judge, MR. Judge, and self-improving VLM judge work provide examples of reasoning-guided multimodal judge construction, synthesized judge-training data, and iterative judge improvement [125,126,127]. These works show that synthetic-data curation can benefit from scalable ways to construct and improve the component that produces judgment signals.

Overall, model-based judging expands V from fixed filtering and source-specific verification to open-ended quality assessment. Rubric-based judges provide criterion-based scores, critic-based judges provide broad critique and preference-like signals, and reasoning-enhanced judge construction improves the judging component itself. These signals support ranking, critique, routing, and later feedback construction before accepted samples enter ${\mathcal{D}}_S^{\mathrm{mm}}$.

5.4. Preference and Feedback Construction

Preference and feedback construction instantiates V in Eq. (6) as a signal-construction operation over ${\mathcal{C}}^{\mathrm{mm}}$. It converts curation judgments into reusable supervision records in ${\mathcal{D}}_S^{\mathrm{mm}}$, such as preference pairs, correction labels, critiques, reward scores, rejection reasons, or fine-grained feedback records. Existing works can be grouped by the type of feedback signal they construct: preference and reward signals, correctional feedback, and AI or self-feedback.

Preference and Reward Signals

Preference and reward signals encode relative or scalar judgments over multimodal responses. LLaVA-RLHF adapts RLHF to vision–language alignment by collecting human comparison feedback and training a factually augmented reward model for hallucination control [17]. Align2LLaVA curates machine-generated multimodal instructions through cascaded human and LLM preference alignment [128]. These pipelines produce ranked, reward-labeled, or preference-aligned records as reusable supervision signals.

Correctional Feedback

Correctional feedback localizes why a response is flawed. RLHF-V uses fine-grained correctional human feedback, including segment-level corrections on hallucinations, for trustworthy MLLM behavior alignment [18]. This signal is useful for multimodal curation because a scalar preference may hide whether the failure is a hallucinated object, an incorrect OCR span, an unsupported temporal relation, a wrong chart value, or an invalid GUI action. Localized feedback preserves more information about the failure mode and makes the curated output more useful for later training.

AI Feedback and Self-Feedback

AI feedback and self-feedback scale feedback construction when human annotation is expensive. VLFeedback constructs a large-scale AI feedback dataset for LVLM alignment, while RLAIF-V studies open-source AI feedback through feedback data generation for preference learning and self-feedback guidance [129,130]. These methods use model-based evaluators or self-feedback procedures to produce feedback records at scale. The main risk is judge dependence: if the feedback model shares the generator’s blind spots, the resulting preference or correction signal may reinforce hallucination or grounding errors.

Overall, preference and feedback construction contributes alignment-ready signals to ${\mathcal{D}}_S^{\mathrm{mm}}$. Candidates in ${\mathcal{C}}^{\mathrm{mm}}$ are transformed into preference pairs, correction labels, critiques, rejection reasons, and reward-like records. These signals are valuable because they preserve why a candidate is preferred, rejected, or corrected. Their main limitation is signal compression: if a complex multimodal failure is reduced to a single scalar preference or reward, later training may absorb the preference without learning the grounding reason behind it.

5.5. Critique and Repair

Critique and repair instantiates V in Eq. (6) as a revision operation over flawed but recoverable candidates. This operation transforms candidates in ${\mathcal{C}}^{\mathrm{mm}}$ into corrected samples, dehallucinated instruction records, or repair traces in ${\mathcal{D}}_S^{\mathrm{mm}}$. A typical repair loop localizes the error against source evidence, revises or regenerates the candidate, and re-verifies the repaired output before accepting, logging, or rejecting it. Existing works can be grouped into two repair mechanisms: data-level repair or regeneration of synthetic instruction data, and response-level repair of flawed multimodal outputs.

Data-Level Repair and Regeneration

Data-level repair operates on generated instruction data before it is reused for training. HalluciDoctor identifies and eliminates hallucinations in machine-generated visual instruction data, followed by counterfactual visual instruction expansion [40]. Visually Dehallucinative Instruction Generation constructs dehallucinated visual instructions by constraining generated question–answer pairs to image content, producing image-aligned instruction data such as CAP2QA-COCO [131]. Prescribing the Right Remedy provides a diagnosis-guided route: hallucination diagnosis is used to generate targeted instruction data tailored to model-specific hallucination patterns [132]. These methods revise, constrain, or regenerate synthetic visual instruction data according to detected hallucination patterns.

Response-Level Repair Mechanisms

Response-level repair corrects individual multimodal outputs when a candidate response is flawed but recoverable. Woodpecker identifies visual claims, validates them against image source information, and corrects hallucinated content [133]. Volcano provides a self-feedback-guided revision route in which the model generates visual feedback for its initial response and uses that feedback to revise hallucinated content [134]. These methods turn flawed candidates into corrected responses or repair traces when the error can be localized and the revision is checked against source information.

Repair signals should remain tied to source information. A revised response may become more fluent without becoming more grounded. Data-level repair, response-level correction, and self-feedback revision are most reliable when paired with source-grounded signals, such as visual knowledge validation, OCR text, chart structure, object grounding, temporal evidence, or environment feedback. The main risk of repair is unsupported revision: the repaired sample may look cleaner while preserving the same grounding error.

5.6. GUI or Web Trajectory Validation

GUI or web trajectory validation instantiates V in Eq. (6) as a stateful validity-checking operation for interaction data. A candidate trajectory usually contains an instruction, screenshot or interface state, target element, action, intermediate observation, and final outcome. This operation produces validity signals for ${\mathcal{D}}_S^{\mathrm{mm}}$ by checking whether the action sequence is consistent with the observed interface state and whether the final outcome follows from the executed steps. Existing works can be grouped into two validation signals: trajectory-level quality control and GUI grounding validation.

Trajectory-Level Quality Control

Trajectory-level quality control checks whether generated interaction records are coherent, executable, and useful for training. OS-Genesis, Explorer, Wang et al., and Auto-scaling Continuous Memory provide direct trajectory-data evidence: they synthesize tasks, roll out GUI or web trajectories, refine noisy records, or verify task success before reuse [30,31,32,135]. GUI-Critic-R1 provides a complementary pre-execution critic signal for diagnosing potential GUI automation errors before action execution [136]. These works show that GUI or web curation should check whether a retained trajectory remains consistent across instruction, state, action, observation, and outcome.

GUI Grounding Validation

GUI grounding validation checks whether the action target and action format are supported by the visible or serialized interface state. SeeClick enhances visual GUI agents through GUI grounding pre-training and automates the curation of GUI grounding data [8]. OS-ATLAS further supports cross-platform GUI grounding by synthesizing GUI grounding data across Windows, Linux, macOS, Android, and web interfaces [14]. These works show that a trajectory should preserve a valid link among language instruction, visible target element, action format, and current interface state. In the synthetic-data lifecycle, generation proposes tasks and trajectories, while curation validates which interaction records are reliable enough for training integration.

Overall, GUI or web trajectory validation contributes stateful action-validity signals to ${\mathcal{D}}_S^{\mathrm{mm}}$. It transforms candidate interaction records in ${\mathcal{C}}^{\mathrm{mm}}$ into validated, refined, or quality-labeled interaction records. Its main risk is environment dependence: a trajectory may be valid only under a specific interface state, browser condition, tool version, or serialization format.

5.7. Summary and Discussion

Data curation and verification transforms generated candidates into reusable multimodal supervision through V. Quality filtering and data selection provide low-cost filtering, ranking, selection, or routing signals before deeper checks are applied (Section 5.1). Grounding verification checks whether generated claims are supported by visual, textual, numerical, or structured source information (Section 5.2). Model-based judging supplies open-ended quality assessment through rubric scores, critic signals, rankings, critiques, or preference-like judgments (Section 5.3). Preference and feedback construction converts curation judgments into alignment-ready feedback records (Section 5.4). Critique and repair transforms flawed but recoverable candidates into corrected samples, dehallucinated instruction records, or repair traces (Section 5.5). GUI or web trajectory validation checks consistency among instructions, interface states, actions, observations, and outcomes (Section 5.6).

The main trade-off is among cost, independence, and failure coverage. Filtering and selection scale to large candidate pools, but their signals remain shallow. Grounding verification provides stronger source support, but its coverage depends on modality-specific tools such as VQA models, OCR systems, chart parsers, attribution methods, and execution environments. Model-based judges can assess open-ended qualities, but their judgments may inherit visual blind spots, preference bias, or rubric limitations. Feedback construction preserves why a candidate is preferred, rejected, or corrected, while overly compressed rewards or preferences may hide local grounding errors. Repair can recover useful synthetic supervision, but revised samples must remain tied to source information. GUI or web validators add action-state consistency checks, although their reliability depends on interface state, browser condition, tool version, and serialization format.

These mechanisms are complementary components of V. A practical curation pipeline can use low-cost filtering to reduce candidate volume, source-support verification to detect unsupported claims, model-based judging to assess open-ended quality, feedback construction to preserve preference or correction information, repair to recover flawed but useful samples, and trajectory validation to handle stateful interaction records. A remaining design challenge is how to compose these signals under realistic budget, coverage, and reliability constraints. As synthetic multimodal data pipelines scale, curated outputs need calibrated curation signals, clear provenance, verifier audits, and stable handoff to training integration, where curated samples, scores, feedback records, and repaired trajectories become parameter-updating supervision.

6. Training Integration

After seed construction, data generation, and curation, synthetic multimodal supervision becomes useful only when it is integrated into MLLM training in a controlled form. Training integration converts curated synthetic data into optimization-ready training inputs. Following the formulation in Section 2.4, this stage maps curated synthetic data ${\mathcal{D}}_S^{\mathrm{mm}}$ into the final training set:

$${\mathcal{D}}_{\mathrm{train}}=I({\mathcal{D}}_H^{\mathrm{mm}},{\mathcal{D}}_S^{\mathrm{mm}}),$$

(7)

where I denotes the integration procedure that covers formatting, mixture design, objective assignment, and scheduling. The resulting training set may contain serialized image–text records, interleaved documents, instruction-response conversations, visual reasoning traces, GUI state–action trajectories, preference pairs, reward-labeled samples, or domain-specific mixtures.

We organize I into four coupled decisions: micro-level formatting, macro-level composition, objective-level optimization, and temporal-level scheduling. Micro-level formatting determines how curated records expose image positions, document order, temporal context, GUI state, source information, actions, and response fields to the optimizer (Section 6.1). Macro-level composition determines how synthetic records are balanced with human-origin data, text-only data, modality-specific data, and domain-specific supervision (Section 6.2). Objective-level optimization determines whether integrated records are used for pretraining, supervised fine-tuning, preference optimization, reinforcement fine-tuning, or verifiable reward optimization (Section 6.3). Temporal-level scheduling determines when different supervision types enter training and how generated, corrected, failed, or trajectory-level records are incorporated into later training stages (Section 6.4). The main design question is how to preserve useful supervision while controlling distribution shift, modality imbalance, shortcut learning, and training-stage instability. Figure 6 illustrates the organization of this section, and Table 5 summarizes the integration mechanisms and cited works.

6.1. Micro-Level Formatting

Micro-level formatting determines how curated supervision in ${\mathcal{D}}_S^{\mathrm{mm}}$ is exposed to the optimizer as training-ready records in ${\mathcal{D}}_{\mathrm{train}}$. In MLLM training, formatting is more consequential than text-only role formatting because each record may need to preserve image positions, document order, OCR or chart source information, temporal scope, GUI state, action format, role boundaries, response fields, and the boundary between source information and language output. A synthetic sample can be useful at generation time but weak at training time if its modality order, task field, answer schema, or action representation conflicts with the model interface. This subsection summarizes three serialization forms: interleaved document serialization, instruction-response serialization, and state-action serialization.

Interleaved Document Serialization

Interleaved document serialization arranges images and text as document-like sequences, making image order, text order, and image–text grouping part of the training signal. This form is important for synthetic multi-image examples, long-context multimodal documents, and generated interleaved conversations, where supervision depends on relations among multiple visual and textual units.

TextBind, Sparkles, and MANTIS provide examples of interleaved or multi-image instruction integration. TextBind generates multi-turn interleaved multimodal instruction-response conversations from image-caption pairs; Sparkles constructs machine-generated dialogue data for word-level interleaved multi-image and text interactions; and MANTIS builds multi-image instruction data for instruction tuning across multiple visual inputs [105,106,108]. Large-scale pretraining corpora provide broader serialization precedents: MMC4 and OBELICS construct interleaved image–text web documents, while VILA and MM1 analyze how interleaved image–text data and its mixture with image-caption or text-only data affect multimodal pretraining and instruction tuning [3,4,137,138]. These works show that generated multimodal documents, multi-image examples, and long-context samples should preserve document order and image–text correspondence when integrated into ${\mathcal{D}}_{\mathrm{train}}$.

Instruction-Response Serialization

Instruction-response serialization converts heterogeneous multimodal tasks into trainable instruction units with explicit task, modality, and response fields. This form is central to visual instruction tuning because the same source image or document can support different target behaviors depending on whether the serialized unit asks for captioning, question answering, reasoning, grounding, OCR, chart reading, or dialogue.

Schema-oriented works provide the basic integration template. M3IT reformats diverse multimodal and multilingual tasks into a unified vision-to-text instruction structure [139]. MultiInstruct and InstructBLIP further transform multimodal tasks or public vision-language datasets into unified instruction-tuning formats [2,140]. Direct synthetic-data evidence comes from LLaVA and SVIT, which use GPT-based generation to construct visual instruction-tuning records such as conversations, detailed descriptions, complex reasoning samples, and referring question–answer pairs [1,141]. More recent large-scale instruction resources make this serialization issue explicit at scale. MMInstruct uses GPT-4V/GPT-3.5 generation and human correction to build diverse visual instruction data; Infinity-MM combines unified preprocessing with a tagging-guided synthetic instruction generation method; Leopard constructs text-rich multimodal instruction-tuning data for multi-image and high-resolution settings; and LLaVA-Video builds synthetic video instruction-following data covering detailed captioning, open-ended question answering, and multiple-choice question answering [5,142,143,145]. LLaVA-OneVision-1.5 and InternVL2.5 further connect large curated multimodal datasets with efficient training frameworks and multimodal data-packing strategies [144,146]. These works show that synthetic multimodal supervision becomes trainable when task fields, modality fields, response fields, and packing constraints are handled consistently.

State-Action Serialization

State-action serialization turns GUI, web, or tool-use trajectories into training records that couple observations with executable actions. For action-oriented synthetic data, the serialized record must preserve the observed state, action space, action arguments, and target behavior. The integration problem is to keep screenshots, interface elements, histories, action arguments, and outcomes aligned within the same training record.

TongUI constructs GUI trajectory data from multimodal web tutorials, making tutorial-derived screenshots and action steps consumable for GUI-agent training [147]. UIPro curates large-scale GUI understanding data and introduces a unified action space to combine heterogeneous GUI-agent datasets [148]. UI-TARS provides a related action-modeling precedent through screenshot-based GUI perception, unified action modeling, large-scale action traces, and reflective online traces [149]. These works show that GUI and tool-use supervision must serialize both the interface state and the action representation. Dropping either side weakens the training signal even when the trajectory was correctly generated or verified.

Overall, micro-level formatting defines the atomic training records produced by I. Interleaved document serialization preserves image–text order, instruction-response serialization exposes task and answer fields, and state-action serialization couples observations with executable actions. The main risk is interface information loss: if source information, modality order, action format, or task boundaries are dropped during serialization, later objectives may optimize response style while losing the multimodal signal that the synthetic record was intended to teach.

6.2. Macro-Level Composition

Macro-level composition determines how training records in ${\mathcal{D}}_{\mathrm{train}}$ are distributed during optimization. After micro-level formatting, each record is structurally compatible with the model interface, but the training mixture still needs to balance synthetic data, human-origin data, text-only data, image–text data, interleaved data, domain-specific supervision, and action-oriented data. This subsection summarizes works that integrate synthetic and curated multimodal data through the mixture component of I. The main design problem is to expand capability coverage while controlling distribution shift, modality imbalance, and shortcut learning. We organize this subsection in a coarse-to-fine order: global data balance, task-domain-format composition, and mixture ratio search.

Global Data Balance

Global data balance determines how multimodal data is mixed with text-only data and other base-distribution data. This decision is important because adding visual, interleaved, or synthetic supervision can improve multimodal capability while also changing the language distribution seen during training. VILA and MM1 provide representative evidence for this balance: VILA studies interleaved image–text pretraining and text-only instruction-data re-blending, while MM1 identifies a careful mix of image-caption, interleaved image–text, and text-only data as important for strong multimodal pretraining performance [3,4]. These works show that synthetic or curated multimodal data should expand visual, grounding, and task coverage while preserving language competence. Recent analysis of MLLM fine-tuning also shows that data-hybrid training can reduce task-specific overfitting and forgetting, reinforcing the need to balance task-specific multimodal supervision with broader training distributions [150].

Mixing Tasks, Domains, and Data Formats

This mechanism operates inside the multimodal portion of the training mixture, determining how synthetic and curated multimodal data are combined across task types, visual or interaction domains, and serialized data formats. Task composition controls the proportion of supervision targets, such as captioning, VQA, OCR, chart understanding, reasoning, GUI understanding, and multi-image reasoning. Domain composition controls coverage over natural images, documents, charts, webpages, GUI screenshots, text-rich images, and high-resolution inputs. Format composition controls the sources and serialized forms of supervision, including image–text pairs, interleaved image–text records, OCR or document records, GUI screenshot–action records, and multi-image or high-resolution records. Cambrian-1 provides a vision-centric MLLM recipe that emphasizes high-quality visual instruction-tuning data, data source balancing, and distribution ratios [151]. Infinity-MM and MMInstruct support large-scale expansion of task and domain coverage through diverse multimodal instruction data [142,143]. Open-Qwen2VL supports practical format composition through data filtering, multimodal data mixture strategies, sequence packing, and compute-aware pretraining [152]. LLaVA-MoLE adds a conflict-aware perspective by showing that instruction data from distinct domains can conflict and that expert-based routing can mitigate such conflicts during MLLM instruction fine-tuning [153]. Together, these works show that synthetic and curated multimodal data should be integrated as a structured mixture of task types, domains, and serialized formats.

Mixture Ratio Search

Mixture ratio search estimates effective proportions after candidate data pools have been defined. This decision reduces manual trial-and-error over data-mixture ratios, especially when full MLLM training is expensive. DaMo predicts downstream performance for candidate data-mixture ratios in mobile-phone-agent fine-tuning; Linear Model Merging estimates mixture efficacy through weighted combinations of domain-specific multimodal experts; and DataProphet estimates the usefulness of supervision datasets for target benchmarks before training [154,155,156]. These works show that mixture design can use predictive search or proxy evaluation before full training. The main risk is proxy mismatch: a candidate mixture ratio or dataset combination may appear effective according to a predictor, a merged-expert proxy, or a limited benchmark while still requiring final training validation.

Overall, macro-level composition defines the distributional role of I. Global data balance allocates the training budget between multimodal and base-distribution data, task-domain-format composition structures the internal coverage of the multimodal pool, and mixture ratio search reduces the cost of selecting effective proportions. The main risk is distributional distortion: if synthetic or domain-specific data is overweighted, the model may gain a narrow capability while losing language competence, grounding robustness, or performance on other modalities.

6.3. Objective-Level Optimization

Objective-level optimization determines how training records in ${\mathcal{D}}_{\mathrm{train}}$ update MLLM parameters. The same serialized record can be used by different objectives: pretraining treats it as distributional alignment data, SFT treats it as target behavior, preference optimization treats it as comparative supervision, and reinforcement fine-tuning treats it as reward- or interaction-conditioned feedback. This subsection summarizes three objective-level integration mechanisms within I: training with preferences and feedback, training GUI agents with reinforcement learning, and training with verifiable visual rewards.

Training with Preferences and Feedback

This mechanism turns comparison labels, correction signals, AI feedback, or task preferences into parameter-update targets. LLaVA-RLHF adapts RLHF to vision–language alignment with hallucination-aware human preferences and a factually augmented reward model [17]. RLHF-V uses fine-grained correctional human feedback and dense direct preference optimization to align MLLM behavior [18]. VLFeedback and RLAIF-V show how AI feedback or open-source model feedback can produce preference data for LVLM/MLLM alignment [129,130]. Task Preference Optimization integrates differentiable task preferences from fine-grained visual tasks into MLLM alignment [157]. These works show that curated preferences, corrections, and task feedback become training signals when the objective specifies which visual, grounding, or task behavior should be preferred.

Training GUI Agents with Reinforcement Learning

This mechanism consumes GUI, web, or mobile-agent supervision as state–action training signals. MagicGUI connects a scalable GUI data pipeline with reinforcement fine-tuning for mobile-use agents, while AgentCPM-GUI uses grounding-aware pretraining, trajectory-based SFT, and reinforcement fine-tuning to improve mobile GUI interaction [158,159]. These works show that trajectory data becomes useful for parameter updates when the objective reinforces valid actions, successful interaction outcomes, or improved state-conditioned behavior.

Training with Verifiable Visual Rewards

This mechanism uses task-specific or rule-based rewards to update multimodal reasoning behavior. MoDoMoDo studies multi-domain RLVR for MLLMs with verifiable vision–language problems and data-mixture-aware RL fine-tuning [39]. Ground-R1 supports grounded visual reasoning through reinforcement learning with evidence-region rollouts and answer-related rewards [160]. R1-VL uses dense step-wise reasoning rewards for multimodal reasoning, Visual-RFT uses visual perception verifiable rewards for reinforcement fine-tuning on visual tasks, and Perception-R1 introduces visual perception rewards to encourage accurate perception during multimodal reasoning [161,162,163]. Recent post-training work also shows that SFT-style labels and RLVR-style rollouts can be combined within one objective design [164]. These works show that visual reasoning traces, answer correctness, grounding formats, perception signals, and task-specific rewards can be integrated as optimization targets.

Overall, objective-level optimization defines the training objective attached to each signal in I. Preference and feedback optimization consumes comparison or correction signals, GUI-agent reinforcement learning consumes state–action trajectories, and verifiable visual reward optimization consumes reward-shaped visual or reasoning signals. The main risk is objective mismatch: if the objective rewards shortcuts, shallow preference patterns, or environment-specific artifacts, the model may improve the optimized metric while weakening grounding, robustness, or general multimodal behavior.

6.4. Temporal-Level Scheduling

Temporal-level scheduling determines when training records in ${\mathcal{D}}_{\mathrm{train}}$ enter optimization and how generated, corrected, failed, or trajectory-level records are incorporated into training. The order matters because multimodal signals differ in difficulty and stability: caption-style records, interleaved documents, dense visual reasoning traces, long videos, and GUI trajectories may require different model capabilities before they can be absorbed effectively. This subsection summarizes two temporal integration mechanisms within I: staged training of multimodal signals and experience-based supervision integration.

Staged Training of Multimodal Signals

This mechanism determines when easier, broader, or more reliable multimodal supervision enters training before harder interaction or reasoning signals. MobileGUI-RL synthesizes a curriculum of learnable mobile-GUI tasks through self-exploration and filtering before online reinforcement learning [165]. UI-TARS-2 provides a related scalable data-and-training route for GUI agents by combining large-scale data generation with a multi-turn reinforcement learning framework [166]. Continual multimodal instruction-tuning work further shows the need for curriculum- and modality-aware scheduling, since task order and modality imbalance can affect how new supervision is absorbed [167]. These works show that synthetic multimodal signals can be introduced through staged or curriculum-aware training.

Experience-Based Supervision Integration

This mechanism incorporates trajectory records, failed experiences, calibrated interactions, or structured experience memory into training supervision. Step-GUI converts model-generated GUI trajectories into reliable training signals through a calibrated step reward system and trajectory-level calibration [168]. UI-Voyager uses rejection fine-tuning and group-relative self-distillation to turn successful rollouts into dense corrective supervision for failed GUI trajectories [169]. UI-Mem accumulates structured experience memory, including workflows, subtask skills, and failure patterns, for online GUI reinforcement learning [170]. These works show that interaction data can serve as training supervision when experience records remain tied to interface states, actions, outcomes, and failure modes.

Overall, temporal-level scheduling defines the exposure policy of I. Staged training controls when each supervision type enters training, while experience-based supervision integration controls how generated or interaction-derived records are incorporated into scheduled training stages. The main risk is error amplification: trajectories, failed attempts, or model-generated supervision may introduce modality drift, environment-specific artifacts, or unverified feedback if they are integrated without calibration.

6.5. Summary and Discussion

Training integration converts curated multimodal data and supervision signals into optimization inputs for parameter updates. In Eq. (7), I maps ${\mathcal{D}}_H^{\mathrm{mm}}$ and ${\mathcal{D}}_S^{\mathrm{mm}}$ into the final training set ${\mathcal{D}}_{\mathrm{train}}$ through four coupled decisions. Micro-level formatting defines atomic training records by preserving modality order, task fields, response fields, and action formats (Section 6.1). Macro-level composition controls how text-only, multimodal, synthetic, human-origin, task-specific, and domain-specific data share the training budget (Section 6.2). Objective-level optimization determines how integrated signals are consumed by pretraining, SFT, preference optimization, reinforcement fine-tuning, or verifiable reward optimization (Section 6.3). Temporal-level scheduling determines when different supervision types enter training and how generated, corrected, failed, or trajectory-level records are incorporated into training stages (Section 6.4).

The central trade-off is signal preservation versus distributional disturbance. Serialization must expose the multimodal information that the synthetic record is intended to teach. Mixture design must expand task and modality coverage while preserving base language and visual distributions. Objective design must reward the intended behavior rather than shortcuts or environment-specific artifacts. Scheduling must introduce difficult signals when the model can absorb them, while experience-based supervision integration should avoid amplifying unverified synthetic or interaction-derived errors. These four decisions are tightly coupled: a well-curated sample becomes useful training signal only when it is properly serialized, mixed, assigned to an appropriate objective, and scheduled at a stable training stage.

Training integration also clarifies the handoff among lifecycle stages. Seed construction provides grounded sources and initial supervision pools; data generation produces candidate multimodal supervision; data curation and verification filters, verifies, scores, repairs, or labels those candidates; and training integration specifies how accepted records enter parameter updates. A chart, video, GUI trajectory, preference label, reward score, or reasoning trace becomes part of this stage when it is serialized, mixed, assigned to an objective, and scheduled for optimization.

A remaining challenge is incomplete reporting of integration controls. Synthetic MLLM papers often describe generated data in detail while giving less visibility into serialization schemas, human-origin–synthetic ratios, task and modality mixtures, objective choices, reward sources, reward granularity, curriculum design, experience-integration policy, and stopping rules. Without these details, gains are difficult to attribute and risks are difficult to audit.

7. Risk Propagation in Synthetic Multimodal Data Pipelines

Risks in synthetic multimodal data pipelines can originate at multiple lifecycle stages and propagate after accepted synthetic signals enter model updates, model selection, or evaluation-dependent decisions. Seed construction may under-cover long-tail source information; data generation may introduce weakly grounded or overly regular supervision; data curation may pass biased verifier, judge, or reward signals; and training integration may overweight synthetic formats or attach inappropriate objectives. After crossing this boundary, local data defects may influence the training distribution, model behavior, robustness, grounding, or evaluation validity.

In this section, synthetic multimodal data refers to trainable outputs of the synthetic-data lifecycle, including synthetic or curated multimodal examples and derived supervision signals such as instruction–response records, reasoning traces, GUI or web trajectories, preference pairs, correction labels, AI feedback, reward signals, and rollout feedback. Parameter-updating stages include pretraining or continued pretraining, supervised fine-tuning or instruction tuning, preference optimization, reinforcement fine-tuning, and parameter-efficient or distillation-style updates when synthetic or model-curated multimodal signals update full MLLM parameters or trainable adaptation modules.

The central issue is risk propagation across the update boundary. Once a biased sample, verifier score, reward signal, trajectory record, or contaminated example is accepted into downstream use, its error can affect the training distribution, objective pressure, model behavior, or reported capability. Following the notation in Section 2.4, we write this boundary as

$${\theta }^{+}=\mathrm{Update}\left(\theta ,{\mathcal{D}}_{\mathrm{train}}\right),{\mathcal{D}}_{\mathrm{train}}=I\left({\mathcal{D}}_H^{\mathrm{mm}},{\mathcal{D}}_S^{\mathrm{mm}}\right),$$

(8)

where $\theta$ and ${\theta }^{+}$ denote the model parameters before and after the update, ${\mathcal{D}}_S^{\mathrm{mm}}$ denotes curated synthetic data or derived supervision, and I denotes the training integration procedure that forms the final training set ${\mathcal{D}}_{\mathrm{train}}$.

The overall framework in Figure 1 places risk propagation as a cross-stage analytical layer of the synthetic multimodal data pipeline. As summarized in Table 6, we first define the risk boundary for synthetic-data-based MLLM development (Section 7.1). We then discuss four risk areas: coverage erosion and knowledge degradation (Section 7.2), distributional and multimodal drift (Section 7.3), verifier and reward feedback risks (Section 7.4), and contamination and robustness boundaries (Section 7.5). We close with stability implications for increasingly automated synthetic data pipelines (Section 7.6).

7.1. Problem Formulation and Risk Scope

The central object is the model-update and evaluation boundary in Eq. (8). Pipeline risks arise when synthetic examples, curated records, verifier scores, reward signals, feedback labels, or contaminated samples become optimization targets or influence model selection. These signals can change the effective training distribution, objective pressure, modality balance, grounding behavior, and evaluation validity. Synthetic data can improve coverage by adding rare visual states, difficult reasoning traces, long documents, GUI transitions, or preference supervision. It can also destabilize training when it replaces human-origin support, concentrates formats, weakens grounding, distorts objectives, or leaks evaluation content into training or model selection.

This framing separates the source of perturbation from the observed failure. Under-covered seeds affect support preservation; weakly grounded generation affects hallucination and knowledge behavior; high synthetic ratios and standardized templates affect diversity; imperfect verifiers and reward models reshape the accepted distribution; and contamination affects the validity of both training and evaluation. The following sections therefore treat data generators, verifiers, reward models, GUI agents, and integration recipes as components of a synthetic data pipeline whose outputs may become risks after they enter parameter-updating stages or evaluation-dependent decisions.

7.2. Coverage Erosion and Knowledge Degradation

Coverage erosion refers to the weakening of rare or low-probability regions when generated or model-selected data is used heavily without sufficient anchoring in human-origin data. Analyses of model collapse and generated-data degradation provide useful theoretical precedents, showing that training heavily on generated samples may erode low-probability regions, alter scaling behavior, and degrade generation quality [171,172,173,174]. In MLLM pipelines, this issue is especially relevant because rare visual states, document layouts, numerical patterns, GUI transitions, and long-tail concepts are often the targets of synthetic expansion. Recent studies on multimodal pretraining further indicate that downstream performance remains closely coupled with the prevalence of corresponding concepts in pretraining corpora, making long-tail support a central challenge for scalable synthetic-data construction [175].

Support preservation depends on the data-use policy and the available anchors. Stability analyses show that generated data can remain useful when it accumulates alongside human-origin data, original support remains accessible, maximum-likelihood assumptions are satisfied, or verification constrains accepted samples [176,177,178,179,180]. These findings point to a common design requirement: synthetic data use needs anchors. Support preservation, mixture control, and reliable screening jointly determine whether synthetic supervision expands or narrows the effective training distribution.

Knowledge degradation is a second form of pipeline risk. A model may preserve surface fluency while losing factual precision or rare knowledge under synthetic-data-based model updates [181]. In multimodal reasoning, the analogous risk is that coherent rationales survive while visual evidence receives less effective attention. Recent MLLM reasoning-risk evidence shows that longer reasoning can amplify hallucination or leave preference optimization biased toward response-level correctness rather than faithful reasoning traces [182,183]. Multimodal synthetic-data and fact-level studies further motivate checking whether accepted supervision remains grounded after it enters parameter updates [184,185]. These studies show that reasoning traces and preference signals need grounding checks before they become optimization targets.

7.3. Distributional and Multimodal Drift

Distributional drift appears when synthetic supervision becomes more regular than the target data distribution. Text synthesis studies show that high synthetic proportions can concentrate local patterns and produce distribution shift, while diffusion and generated-data training analyses connect heavy generated-data use to entropy decline, memorization, and the loss of low-frequency events [173,186,187,188]. Similar pressure can affect multimodal supervision. Rendered charts, GUI trajectories, or image–text instructions may look valid in isolation while collectively pushing the model toward narrower formats, layouts, visual styles, or reasoning templates.

Multimodal drift adds asynchronous channel failure. Language may remain fluent while visual diversity decreases or cross-modal alignment weakens. Evidence from multimodal synthetic-data training and reasoning-risk studies shows why modality-aware diagnostics are needed for VLMs and MLLMs: collapse-like degradation can appear in visual-language settings, and reasoning-heavy models may hallucinate when textual deliberation outruns visual grounding [182,184]. Task-specific multimodal evidence further shows that synthetic-to-real distribution gaps can affect downstream multimodal training, motivating distribution matching or selection when synthetic data is used for real-world multimodal misinformation detection [189]. The practical diagnostic is whether visual, textual, temporal, numerical, and action evidence remain mutually constraining after model updates.

Synthetic-format overfitting is a local symptom of distributional drift. Standardized templates, chain-of-thought patterns, screenshot-action formats, chart schemas, or verifier-preferred phrasings can become shortcuts that the model treats as task structure. This risk can appear even when individual synthetic samples are fluent and locally plausible. Highly polished synthetic supervision can still reduce robustness under human-origin domain shift if it exposes the same surface forms and reasoning patterns too heavily.

7.4. Verifier and Reward Feedback Risks

Verifier decisions can stabilize synthetic-data use, but they also shape the accepted training distribution. Studies of verified synthetic retraining show that selecting useful synthetic samples can improve stability, while imperfect verifiers may pull later models toward verifier-specific biases [190,191]. At the model-update boundary in Eq. (8), verifier-selected records become optimization targets or selection signals. A verifier that consistently favors linguistic plausibility, familiar visual layouts, or easy-to-check answers may narrow the accepted distribution even while improving short-horizon quality.

Reward and preference signals introduce objective-level risks. Preference feedback can act like implicit preference optimization and amplify reward-model bias, while process reward models can be exploited [192,193]. Multimodal preference-optimization studies show that the effect of DPO depends on how feedback data is constructed and whether it remains grounded in visual evidence [194,195,196,197,198,199]. Reasoning-conditioned preference evidence further shows that coarse response preferences can leave reasoning traces under-constrained, enabling shortcut learning or hallucinated rationales [183]. These findings suggest checking feedback signals for what they select, what they ignore, and how strongly they remain tied to multimodal evidence.

Verifier and reward feedback also create overfitting pressure. Verification, curation, and preference optimization can improve short-horizon quality, but repeated reliance on the same judge, verifier, or reward model can narrow the accepted distribution. For MLLM pipelines, this is especially important because judges may favor fluent explanations while underweighting image evidence, OCR exactness, temporal order, numerical consistency, or GUI action validity. The risk arises when a non-independent verifier, judge, or reward model consistently shapes the data that becomes optimization-ready supervision.

7.5. Contamination and Robustness Boundaries

Contamination becomes a model-update or evaluation risk when leaked visual or textual content is used as supervision or masks true generalization during model selection. Multimodal contamination evidence shows that both text and images can leak benchmark content, making provenance tracking a cross-modal problem [43]. VLM contamination detection based on multimodal semantic perturbation further motivates testing whether apparent capability reflects generalization or exposure to contaminated patterns [44]. Dynamic evaluation for reasoning MLLMs provides a complementary view by perturbing tasks around the same visual input, showing that contamination or overfitting can inflate task-specific performance while weakening broader generalization [200]. In synthetic-data-based model updates, contamination can propagate through generated examples, verifier decisions, reward selection, and downstream evaluation.

Memorization, privacy leakage, and unsafe synthetic samples follow the same boundary. They become pipeline risks when questionable samples enter parameter updates, change downstream behavior, or inflate evaluation-based decisions. Provenance tracking, deduplication, and contamination checks are necessary because MLLM data can leak through visual as well as textual channels. This framing keeps the discussion focused on risks that affect downstream model behavior while still accounting for safety and privacy hazards amplified by data use.

Robustness degradation closes the risk taxonomy. A pipeline may pass local quality checks and still produce a brittle model if accepted synthetic data are too clean, too templated, or too aligned with verifier preferences. Risk analysis should therefore connect final behavior to earlier lifecycle decisions: seed coverage, generation diversity, verifier independence, mixture strength, objective design, and integration policy all shape whether a model remains robust under domain shift.

7.6. Summary and Discussion

The central trade-off is signal expansion versus downstream stability. Synthetic data for MLLMs can make rare tasks, long videos, documents, charts, and GUI trajectories trainable at scale. These gains become fragile when the synthetic distribution replaces human-origin support, reasoning traces outrun visual evidence, rewards encode exploitable shortcuts, or contaminated benchmarks enter training or model selection.

This trade-off explains why local controls must be evaluated at the downstream level. A larger generator can produce more fluent samples; a stricter verifier can reject more candidates; a stronger reward model can sharpen preferences. Each component can also introduce a new bias if its outputs are used without independent anchoring. Risk-propagation analysis therefore evaluates how synthetic or curated signals affect the model once they become optimization targets or evaluation-influencing records: distribution, objective pressure, feedback quality, modality balance, and evaluation validity must remain stable together.

The boundary with earlier lifecycle stages is functional. Seed construction determines what support is available before synthesis. Data generation creates candidate supervision and intermediate representations. Data curation and verification decides which samples, critiques, or rewards are trusted. Training integration determines how strongly those signals update the model. This section focuses on what can go wrong after these lifecycle decisions become part of model updates or model-selection decisions, and which failure modes should be monitored before synthetic signals are used in downstream training, model selection, or evaluation.

Open challenges remain in provenance tracking, verifier independence, modality-aware diagnostics, and support-preserving mixtures. These problems matter because increasingly automated synthetic data pipelines can improve local data quality while still producing unstable downstream effects. Stable synthetic-data-based MLLM development therefore requires traceable data sources, independent oversight, modality-aware evaluation, and mixture policies that preserve support while expanding supervision.

8. Grand Challenges and Future Directions

The lifecycle analysis above suggests that future progress in synthetic data for MLLMs depends on scaling synthetic data pipelines while preserving grounding, oversight, provenance, and downstream stability. As these pipelines become more automated, they can generate, curate, validate, integrate, and document larger amounts of multimodal supervision with lower human cost. The four-stage pipeline identifies where a data decision occurs, while LSDA provides a pipeline-level perspective on autonomy trends in synthetic data pipelines.

We outline three grand challenges and future directions. Reliable high-autonomy pipelines require external grounding and falsification so that scalable generation remains tied to visual, structural, temporal, interactive, or experimental evidence (Section 8.1). Robust process supervision is needed because multimodal synthetic supervision often contains intermediate reasoning, trajectory, or reward signals that can fail before the final answer is scored (Section 8.2). Provenance and traceability are needed because synthetic records can pass through seed construction, generation, curation, training integration, and evaluation contexts (Section 8.3). We close by summarizing requirements for reliable synthetic-data autonomy (Section 8.4). Table 7 summarizes this roadmap.

8.1. Reliable High-Autonomy Pipelines

Reliable high-autonomy pipelines are an emerging direction for synthetic multimodal data. Existing work has begun to automate parts of the lifecycle, including data-model co-development, task-specific data generation, and long-horizon workflow control. The open challenge is to connect generation, verification, training integration, and provenance tracking into a pipeline that can scale synthetic supervision with limited human intervention.

Feedback-driven data-model systems and task-specific synthetic-data pipelines show early steps in this direction. Data-Juicer Sandbox provides a feedback-driven suite for multimodal data-model co-development, using a Probe–Analyze–Refine workflow across image–text pretraining, image-to-text generation, and text-to-video generation [201]. Genixer studies MLLMs as generators of visual instruction-tuning data [202]. A unified synthetic data pipeline for multimodal video understanding generates supervision for video object counting, video question answering, and video object segmentation [203]. These works show that synthetic-data scaling can become more systematic through feedback-guided refinement, task-aware data recipes, model-assisted generation, and multimodal data-model co-development.

Autonomous scientific systems provide useful references for long-horizon workflow design. Work on autonomous scientific facilities separates routine automated control from semantically grounded experimental decision-making [204]. Scientific-agent systems further illustrate how models can coordinate hypothesis generation, debate, ranking, coding, experimentation, analysis, writing, and review under specified objectives [205,206]. For synthetic multimodal data pipelines, the relevant lesson is that reliable autonomy requires planning, evidence use, decision revision, and validation under external constraints.

For synthetic multimodal data, a central missing capability is grounded falsification. A high-autonomy pipeline should propose candidate images, captions, instructions, rationales, GUI trajectories, or preference signals, and then test whether they are supported by independent evidence. Future pipelines therefore need tighter coupling between MLLMs and verifiable contexts, such as simulators, theorem provers, execution environments, GUI or web states, laboratories, and retrieval-backed source stores. Such verifiable contexts can expose errors that fluent generation may hide.

This direction leaves concrete research opportunities. Future pipelines should flag unsupported visual claims, wrong answers, invalid reasoning steps, failed GUI actions, and preference signals that conflict with visual evidence. They should also generate data inside verifiable contexts when possible, such as browsers, GUI environments, simulators, code executors, retrieval systems, or laboratory workflows, so that environment feedback can help confirm or reject generated supervision. Finally, they should record how each supervision item was generated, verified, repaired, mixed, and integrated into training. The goal is reliable autonomous scaling: increasing the amount and diversity of synthetic supervision while making unreliable records easier to detect before they affect model updates.

8.2. Robust Process Supervision

Robust process supervision is needed because synthetic multimodal supervision often contains intermediate steps. A chart QA trace may require locating axes, reading values, and computing an answer. A GUI trajectory may require identifying an element, executing an action, and confirming a state transition. A visual reasoning trace may be fluent while weakly supported by the image. Final-answer scoring is often too coarse for these cases. The supervision process should evaluate how a record reaches its answer or action, not only whether the final output appears plausible.

Multimodal process-supervision systems have begun to address this problem. VisualPRM introduces a multimodal process reward model, a process-supervision dataset, and a benchmark for step-wise error detection, showing that process supervision can support Best-of-N selection for MLLM reasoning [207]. MM-PRM provides a scalable framework for multimodal mathematical reasoning by generating step-level annotations through an automated MCTS-based pipeline [208]. URSA connects multimodal CoT data synthesis with automatically generated process supervision that checks visual grounding fidelity and logical validity [209]. MM-Verify synthesizes verification data for multimodal reasoning by combining tree-search-style exploration, verification, and rejection sampling [210]. Fact-level multimodal attribution provides another way to evaluate whether reasoning claims are tied to explicit modality and temporal evidence [185]. Together, these works show that robust process supervision requires pipelines that can produce, verify, and use intermediate multimodal supervision signals.

Process supervision must also remain robust when it is optimized against. Process reward models can become optimization targets, and flawed trajectories may receive high scores when the reward signal is exploited [193]. For MLLM synthetic-data lifecycles, this is a transferable warning: an exploitable process verifier or reward model can turn high-throughput supervision into high-throughput error propagation.

Process supervision connects data generation, curation, training integration, and risk control. High-throughput generation can scale candidate supervision, while robust supervision determines which traces, rewards, corrections, and trajectories are reliable enough for downstream training. This makes process supervision a lifecycle-level requirement for synthetic multimodal data pipelines: it links generation quality, curation reliability, training objectives, and risk-propagation analysis.

8.3. Data Provenance & Traceability

Provenance and traceability form the third bottleneck because synthetic multimodal records can pass through many lifecycle stages before they affect a model. A record may originate from a seed image, be rewritten by a generator, repaired by a critic, scored by a verifier, converted into a preference pair, mixed into instruction tuning, and later overlap with an evaluation benchmark. As synthetic content becomes harder to distinguish from human-origin content, provenance becomes part of the core pipeline infrastructure. A useful provenance system should track which seed, generator, verifier, transformation, repair step, training mixture, and evaluation context produced or used a record.

Training-data provenance for synthetic visual data provides one concrete version of this problem. TrainProVe studies whether a suspicious model used synthetic images from a particular text-to-image generator for training, framing provenance as a model-level verification problem rather than only a file-level detection problem [211]. This direction is relevant to MLLM synthetic-data lifecycles because synthetic images, captions, rationales, and preference signals may be transformed many times before their training use needs to be audited.

Provenance must also be cross-layer. A record can carry metadata and watermark signals that point to different histories [212]. This matters beyond media forensics. A synthetic multimodal training example may pass file-level provenance checks while losing the trace of its seed source, generated rationale, repaired answer, judge decision, or training use. Future pipelines need audit protocols that jointly reason over metadata, intrinsic watermarks, transformation history, model lineage, and the role of the record in training.

Data hygiene infrastructure remains necessary, while provenance requires additional lineage information. Large-scale data-processing systems such as Data-Juicer provide useful precedents for systematic data filtering, processing, and recipe management [213]. For MLLM synthetic data pipelines, this infrastructure should be extended with transformation logs, model-use records, and provenance fields so that data hygiene and provenance can support each other.

Evaluation hygiene is part of the same traceability problem. Multimodal contamination studies show that both text and images can leak benchmark content, making leakage a cross-modal problem [43]. VLM contamination detection based on multimodal semantic perturbation further shows that contaminated models may fail to generalize under controlled perturbations [44]. When generated, filtered, and training-integrated data overlap with evaluation corpora, benchmark hygiene becomes a provenance problem. Reliable traceability is therefore needed for both training quality and the credibility of MLLM evaluation.

8.4. Conditions for Reliable Synthetic-Data Autonomy

Reliable synthetic-data autonomy requires more than automated generation. It requires the full lifecycle to preserve source information, verify generated supervision, integrate signals according to training objectives, trace data use, and monitor downstream risks. In this section, LSDA serves as a pipeline-level perspective on autonomy trends in synthetic data pipelines. The four-stage pipeline specifies where a data decision occurs, while LSDA describes how much of the stage sequence is supported by rules, models, verifiers, rewards, tools, environments, or autonomous agents. Figure 7 summarizes this progression from human-curated pipelines to open-ended autonomous data discovery.

The literature reviewed in this survey shows a movement toward higher-autonomy synthetic data pipelines. Early pipelines rely on human-curated data, fixed templates, and local augmentation. Model-assisted pipelines then use captioners, instruction generators, distillation signals, or reformulation models to expand supervision under human-designed rules. More recent pipelines introduce external verification, retrieval, OCR, execution checks, simulators, GUI states, reward models, and provenance-aware integration. L5 remains a research horizon: open-ended autonomous data discovery and validation, where systems identify data needs, construct or acquire source-bearing seeds, generate candidate supervision, validate it with external evidence, document provenance, and output auditable training-ready data packages.

This trajectory explains the three future directions discussed above. Reliable high-autonomy pipelines attach generation to external evidence, execution results, environments, or source documents. Robust process supervision checks intermediate reasoning steps, GUI trajectories, preference signals, and reward-shaped supervision before integration. Provenance and traceability record how automated pipelines transform, repair, score, mix, and use records across lifecycle stages.

Seen this way, LSDA connects the survey’s historical analysis with its future roadmap. As synthetic data pipelines move toward higher autonomy, each lifecycle stage requires stronger reliability controls. Seed construction needs source discovery and support preservation. Data generation needs diverse supervision that remains tied to evidence. Data curation and verification need scalable judges, rewards, and process verifiers that remain calibrated. Training integration needs control over mixtures, objectives, provenance, and evaluation validity before synthetic or derived supervision becomes a model-update signal.

The central lesson is that synthetic multimodal data should be understood as a lifecycle rather than a static dataset. Its value depends on what is selected as source information, how candidate supervision is generated, how trust is calibrated, how training objectives consume the data, and how risks are monitored after integration. The future target is reliable autonomous synthetic data pipeline engineering: systems that expand generation, verification, integration, and documentation while preserving alignment with external evidence.

9. Conclusions

This survey presents a lifecycle-oriented review of synthetic multimodal data for MLLM parameter-updating supervision. Specifically, we analyze how synthetic multimodal data is constructed, generated, curated, verified, and integrated into MLLM training through four production stages: Seed Construction and Condensation, Data Generation, Data Curation and Verification, and Training Integration. We further discuss how risks can propagate across this lifecycle when synthetic or curated signals affect model updates, model selection, or evaluation-dependent decisions. Our analysis shows that reliable synthetic multimodal data depends not only on scalable generation, but also on source-preserving seeds, grounded supervision, calibrated verification, objective-aware integration, and downstream stability.

Despite recent progress, synthetic multimodal data construction still faces several challenges, including grounding preservation, verifier reliability, provenance tracking, mixture control, and robustness after integration. In light of these challenges, we summarize future directions toward more reliable and autonomous synthetic data pipelines for MLLMs. LSDA further provides a pipeline-level perspective on how such pipelines may evolve from human-curated workflows toward more automated, externally grounded, and auditable systems. With this survey, we hope to offer timely guidance for future research on synthetic data construction, lifecycle integration, and risk-aware development of MLLMs.