Alignment in Multimodal Large Language Models: A Survey

1. Introduction

1.1. Background and Symptoms of MLLM Misalignment

Multimodal large language models (MLLMs) extend the understanding and generation capabilities of large language models (LLMs) to inputs such as images, audio, and video. Reliable performance of MLLMs depends not only on what these models can perceive, but also on how they use that information to respond under task and safety constraints. However, MLLMs can produce fluent responses that ignore evidence or violate task and safety requirements. Hallucination benchmarks reveal unsupported objects and misread relations [1,2], while visual jailbreaks elicit unsafe behavior even when the request is recognized [3]. Both failures sever the connection between evidence and appropriate behavior that alignment is meant to secure (Figure 1). Aggregate response quality mixes them, so evidence use and behavior need separate tests.

1.2. Scope of Multimodal Alignment and Fundamental Gaps

Multimodal alignment is the consistency of representations and observable behavior with input evidence, task requirements, and human expectations, where evidence is the task-relevant input content a response should depend on. Core studies concern MLLM interfaces, grounded generation, instruction and preference alignment, and evaluation. Vision–language foundations explain inherited objectives and supervision, adjacent applications illustrate transferable constraints, and text-only methods supply methodology, not multimodal results; GUI agents and image generation appear only where they inform MLLM alignment. We searched arXiv, IEEE Xplore, ACM Digital Library, ACL Anthology, CVF Open Access, OpenReview, and Google Scholar, and traced citations.

We diagnose misalignment through two gaps. The Modality Gap is a failure to establish or use task-relevant correspondence between multimodal evidence and the model’s representations or responses. It is broader than the embedding separation between modalities [4], because proximity alone does not establish object binding, temporal correspondence, or evidence-dependent answers. The Intent Gap is behavior inconsistent with specified task constraints, user goals, or human expectations. Its criteria must be explicit, since a refusal can satisfy safety yet fail a benign request.

The two gaps are complementary. In Figure 2, misreading the mug’s color is a Modality Gap failure, and an answer that stays the same after the color changes suggests a shortcut; lifting a correctly recognized mug against the instruction is an Intent Gap failure. Some errors belong to neither gap: an arithmetic mistake after correct visual reading is a reasoning error unless further tests implicate evidence use or constraint compliance. The gaps can also interact: preferences for fluent answers can reward unsupported detail, and refusal that ignores the image sacrifices benign utility. Joint assessment therefore records evidence sensitivity, task success, and constraint violations.

1.3. Analytical Dimensions of MLLM Misalignment

Six dimensions locate where methods act on the two gaps and how effects are checked. The first four are intervention dimensions: architecture controls which evidence reaches the large language model (LLM), objective functions define which discrepancies are optimized, data construction supplies supervision, and training decides which components are updated and when. Correspondence-oriented choices (bridges, contrastive losses, hard negatives) mainly target the Modality Gap; instruction and preference supervision targets the Intent Gap and can narrow the Modality Gap when it requires supporting evidence. Evaluation tests alignment claims at the representation, task, and behavior levels, and applications expose domain constraints. Table 2, Table 3, and Table 6 tag entries by intended gap. Each work is placed by its primary contribution and cross-referenced elsewhere: a preference loss under objectives, pair construction under data, policy updates under training, and graph construction under architecture.

1.4. Contributions and Organization

Existing surveys cover representation and behavior across stages [8], misalignment causes [6], alignment and fusion [9], representation transfer [7], hallucination mechanisms [10], and preferences [11]. General MLLM [5,12], hallucination [13], safety [14], and evaluation surveys [15,16] provide context. Table 1 compares their organizing dimensions. Building on them, this survey links mechanisms and supervision assumptions to the tests an alignment claim needs:

1)

Mechanisms and supervision assumptions.  Section 2, Section 3, Section 4 and Section 5 compare evidence access, optimization targets, supervision, and updates, identifying shared failure targets and divergent assumptions.

2)

Tests for alignment claims. Section 6 argues that a gain supports an alignment claim only when controlled input changes, constraint checks, validated judges, and within-study comparisons show which gap it narrows.

3)

Cross-domain constraints. Section 7 finds that five application domains require traceable evidence, valid supervision, and constraint-sensitive decisions, motivating Section 8.

Section 2, Section 3, Section 4, Section 5, Section 6 and Section 7 follow the six dimensions in order, and Figure 3 arranges representative work by dimension and year without ranking methods. Section 8 develops four research directions, and the final section concludes.

2. Architecture: Bridging Modalities

Architecture is the first intervention dimension: it fixes which evidence reaches the LLM, so it mainly targets the Modality Gap. Flamingo [17], BLIP-2 [18], and LLaVA [19] connect modalities to language models, but compression may discard decisive details. Table 2 compares bridges, tokens, structure, attention, and experts. How the model uses this evidence and how it behaves (Intent Gap) depend on objectives (Section 3) and supervision (Section 4).

Table 2. Architectural choices for evidence access, compared by evidence unit, access or selection, and principal trade-off; entries do not rank performance.

Family	Examples	Evidence unit	Access or selection	Principal trade-off	Gap
Lightweight bridges	BLIP-2 [18]; TokenPacker [20]	Continuous visual features	Learned queries; regional detail injection	Token budget versus recoverable local detail	M
Discrete interfaces	SpeechT5 [21]; AnyGPT [22]	Shared or modality-specific codes	Codebook mixing; common token sequence	Compact codes versus quantization and sequence length	M
Structured representations	MAIL [23]; VideoTree [24]	Entities/relations; temporal hierarchy	Entity-mediated exchange; query-adaptive selection	Inspectable structure versus extraction error and omitted evidence	M
Attention and layer fusion	AGE-VLM [25]; Dense Connector [26]	Regions; encoder-layer features	Spatial anchoring; multi-layer aggregation	Fine detail versus feature-processing and memory cost	M
Modality-indexed experts	VLMo [27]; MoT [28]	Modality-labeled tokens	Fixed parameter selection; cross-modal interaction	Specialization versus parameter storage and shared transfer	M
Adaptive experts	OneLLM [29]; MoIIE [30]	Features or modality tokens	Dynamic projection routing; intra-/inter-modal experts	Conditional capacity versus routing and utilization overhead	M

M: Modality Gap, the primary architectural target; behavioral requirements need further supervision and evaluation. Gap tags mark intended targets (see Section 6).

Table 2. Architectural choices for evidence access, compared by evidence unit, access or selection, and principal trade-off; entries do not rank performance.

Family	Examples	Evidence unit	Access or selection	Principal trade-off	Gap
Lightweight bridges	BLIP-2 [18]; TokenPacker [20]	Continuous visual features	Learned queries; regional detail injection	Token budget versus recoverable local detail	M
Discrete interfaces	SpeechT5 [21]; AnyGPT [22]	Shared or modality-specific codes	Codebook mixing; common token sequence	Compact codes versus quantization and sequence length	M
Structured representations	MAIL [23]; VideoTree [24]	Entities/relations; temporal hierarchy	Entity-mediated exchange; query-adaptive selection	Inspectable structure versus extraction error and omitted evidence	M
Attention and layer fusion	AGE-VLM [25]; Dense Connector [26]	Regions; encoder-layer features	Spatial anchoring; multi-layer aggregation	Fine detail versus feature-processing and memory cost	M
Modality-indexed experts	VLMo [27]; MoT [28]	Modality-labeled tokens	Fixed parameter selection; cross-modal interaction	Specialization versus parameter storage and shared transfer	M
Adaptive experts	OneLLM [29]; MoIIE [30]	Features or modality tokens	Dynamic projection routing; intra-/inter-modal experts	Conditional capacity versus routing and utilization overhead	M

M: Modality Gap, the primary architectural target; behavioral requirements need further supervision and evaluation. Gap tags mark intended targets (see Section 6).

2.1. Lightweight Bridging-Module Alignment

Bridges map modality features into LLM inputs, so their design determines which evidence survives compression.

2.1.1. Modality-to-LLM Interface Bridging

CLIPTrans [31], a machine-translation precursor, maps M-CLIP features to mBART prefixes, and MMA [32] routes adapters while preserving text-only ability. EMG-Adaptor maps unvoiced electromyography (EMG) to frozen LLM embeddings [33], and SEMI generates projector low-rank adaptation (LoRA) adapters through a hypernetwork before few-example adaptation [34]. Speech Captioner instead verbalizes affective and prosodic cues [35]. Latent interfaces retain learned features, whereas verbalization exposes interpretable cues but may discard nonverbal detail.

2.1.2. Bridging Module Structure Refinement

AlignVLM constrains the destination space, mapping visual features to weighted LLM text embeddings [36]. On the source side, VPG-C recovers missed details through guidance-conditioned residual extraction [37], and AcFormer selects pretrained vision-token anchors instead of generic queries [38]. The two are complementary and can be compared on semantic compatibility and retained information under a fixed token budget.

2.1.3. Interface Extension for Complex Multimodal Scenarios

For images, LLaVA-UHD slices high-resolution inputs while preserving spatial layout [39], and TokenPacker injects regional detail into low-resolution queries [20]. For video, LLaMA-VID uses one query-conditioned context token and one content token per frame [40], and LLaVA-ST combines language-aligned coordinates with separate spatial and temporal compression [41]. Spatial selection and temporal compression keep different evidence, so their token efficiency should be judged on local and temporal tasks, respectively.

Continuous interfaces also support generation: EasyGen connects BiDiffuser to an LLM [42], and PUMA predicts multigranular visual features for diffusion decoding [43]. Both work without a shared discrete codebook, the alternative examined next, though their generation pipelines need not be lightweight.

2.2. Discretization-Based Alignment

Discretization creates shared codes or modality-specific tokens for a common sequence model, trading compact units against quantization loss of modality-specific detail.

2.2.1. Cross-Modal Codebook Construction

Cross-Modal Code Matching (CMCM) aligns distributions over shared codes [44], Finite Discrete Tokens (FDT) ground patches and words in shared concepts [45], and Uni-Code pretrains a unified code space [46]. SpeechT5 also mixes quantized speech/text states with contextual states for decoder attention [21], retaining context that code matching alone loses. Code utilization and modality-specific detail are therefore separate evaluation targets.

2.2.2. Vocabulary-Grounded Token Adaptation

LQAE quantizes images using pretrained word embeddings and a frozen language denoiser without paired data [47], Tar’s TA-Tok learns visual tokens from projected LLM embeddings [48], and UniAudio 1.5’s LLM-Codec represents audio with vocabulary words or subwords [49]. TA-VQ instead supervises visual codes with long-text semantics [50], and Lan et al. supplement optical character recognition (OCR) with multimodal codes for translation [51]. Vocabulary reuse gives a compatible interface without necessarily making code meanings interpretable; semantic supervision constrains those meanings separately.

2.2.3. Unified Token-Based Multimodal Modeling

AnyGPT places modality-specific image, speech, and music tokens alongside text in an autoregressive stream [22]. 4M instead tokenizes text, images, geometry, and semantics for masked multimodal prediction [52], and 4M-21 extends this to 21 modalities [53]. Despite similar token interfaces, autoregressive and masked-subset prediction expose different dependencies, and neither explicitly guarantees relational or temporal validity.

Figure 3. Timeline of representative multimodal alignment work. Works are arranged by publication year along the river ($\le 2021$–2026) and colored by their primary dimension: architecture, objective functions, data construction, training, evaluation, or applications. Logos indicate the institutions of first and co-first authors. The layout does not rank methods, and a work may contribute to more than one dimension.

Figure 3. Timeline of representative multimodal alignment work. Works are arranged by publication year along the river ($\le 2021$–2026) and colored by their primary dimension: architecture, objective functions, data construction, training, evaluation, or applications. Logos indicate the institutions of first and co-first authors. The layout does not rank methods, and a work may contribute to more than one dimension.

2.3. Structured Representation Alignment

Structured methods make relations explicit before reasoning: graphs encode pairwise relations, hypergraphs higher-order ones, and hierarchies or event sequences preserve granularity and temporal order.

Structural foundations from outside MLLM reasoning include MEAformer’s cross-graph entity alignment [54], FREEDOM’s interaction-edge pruning [55], and molecular structure–text alignment in MoleculeSTM and FineMolTex [56,57]. For visual scenes, Action Genome annotates evolving video relations [58], and LLaVA-SG exposes scene-graph features to an LLM [59]. Relation availability after construction should be distinguished from its downstream use.

2.3.1. Graph-Based Multimodal Representation

MAIL exchanges information between scene and concept graphs through shared entities [23], and MR-MKG combines multimodal graphs with RGAT encoding [60]. VaLiK verifies visually constructed graphs [61], whereas mKG-RAG retrieves graph evidence from documents [62]. GraphThinker models within- and cross-event relations [63], 3DGraphLLM encodes 3D objects and relations [64], SayPlan retrieves planning-relevant subgraphs [65], and SpatialThinker combines scene graphs with spatial rewards [66]. Construction makes relations inspectable and retrieval limits which remain accessible, so extraction errors and omitted subgraphs need separate tests even when reasoning is structurally coherent.

2.3.2. Hypergraph-Based Multimodal Representation

Hypergraph Transformer builds question and query-aware knowledge hypergraphs for multi-hop visual question answering (VQA) [67], and TAMO preserves row–column dependencies in tables before LLM integration [68]. The former derives higher-order structure from a query and the latter from the input format; both depend on valid hyperedge construction.

2.3.3. Hierarchical and Sequential Multimodal Representation

VideoTree selects video content through a query-adaptive hierarchy [24], HMR3D retains patch-, view-, and scene-level 3D context [69], and TRACE predicts ordered events with timestamps, salience, and captions [70]. Selection, spatial granularity, and event order constrain different dependencies, and omissions from abstraction should be evaluated alongside answer accuracy.

2.4. Attention-Based Alignment

Attention-based alignment controls where the model looks and which visual feature levels reach the language model, and so governs access to fine-grained evidence.

2.4.1. Spatial Attention and Local Anchoring

Spatial anchoring targets language-relevant regions. AGE-VLM [25] distills Segment Anything Model spatial knowledge into cross-attention, and Patch-Aligned Training [71] adds patch-level image–caption supervision. UniPixel [72] and MLLM-For3D [73] extend anchoring to segmentation and 3D regions, and IPLoc [74] learns in-context personalized localization. FLMR [75] preserves local evidence through late interaction rather than a global vector. Spatial supervision changes what is learned, whereas late interaction changes how local matches are kept at retrieval; in neither case does concentrated attention alone establish causal dependence on the selected region.

2.4.2. Cross-Layer Feature Fusion

Dense Connector [26] aggregates encoder layers, while SparseCut [76] and MMFuser [77] select complementary low- and high-level features. The instruction-guided vision aggregator (IGVA) weights layers by query [78], and Lin et al. examine layer choice and insertion point [79]. Broad aggregation favors feature coverage, whereas selective or query-conditioned fusion prioritizes relevance; neither is equivalent to spatial localization. Token-expanding and channel-wise fusion also differ in cost, so retained detail and memory are comparable only under matched inputs.

2.5. Expert-Based Alignment

Expert-based alignment assigns modality-specific or shared parameters while preserving cross-modal exchange. Modality-indexed pathways and adaptive routing determine where computation happens; objectives (Section 3) shape what parameters learn.

2.5.1. Modality-Indexed Parameter Pathways

VLMo selects modality-specific feed-forward experts by the known modality and shares self-attention without a learned load balancer [27]. Mixture-of-Transformers (MoT) partitions Transformer parameters by modality while retaining global cross-modal self-attention [28]; a mixture-of-experts (MoE) layer instead selects expert submodules, often feed-forward networks. Specialization alone implies neither sparse computation nor lower per-token cost.

2.5.2. Adaptive Expert Routing

Adaptive mechanisms adjust expert contributions to the input. OneLLM maps eight modalities through a universal projection module with dynamic routing [29], UNIALIGN combines modality-aware experts, LoRA, and soft binding within a shared encoder [80], and M3-JEPA uses a multi-gate MoE predictor to separate modality-specific and shared information in latent space [81]. Parrot routes visual tokens to language-conditioned experts for multilingual alignment [82], while MoIIE learns routes to intra- and inter-modality experts and adds a load-balancing term to discourage expert collapse [30]. Routing in a projector, shared encoder, or latent predictor alters different parts of the evidence path, so equal expert counts do not mean equivalent interventions; comparisons should report preserved evidence and active computation, including routing and balancing overhead.

Gains in compatibility, token efficiency, or specialization show at most that evidence is available, not that the decoder uses it. BLIP-2’s first-stage ablation supports staged design [18] but not grounding in general, so evidence use should be retested after tuning. Architectural gains are attributable only under matched backbones, resolution, and token budgets.

3. Objective Functions: Optimizing Multimodal Alignment

Given the evidence an architecture makes available, the objective specifies which discrepancies are minimized (Table 3). The five families below move from local correspondence through set-level geometry, generative prediction, and auxiliary constraints to whole responses. The first four mainly act on the Modality Gap; only the last scores responses and so acts on the Intent Gap directly.

Table 3. Objective families for multimodal alignment, compared by supervision and learning signal and by the limitation discussed. Gap tags mark intended targets.

Objective family	Examples	Supervision and learning signal	Limitation discussed	Gap
Fine-grained structured contrastive	GLoRIA [83]; GOAL (Choi et al.) [84]	Region–word or object matches are contrasted with competing candidates.	Incorrect correspondences and false negatives corrupt local supervision.	M
Topological and distributional metric	$A^3$$D^2$ [85]; GRAM [86]	Transport matches distributions; Gramian volume constrains joint geometry.	Global agreement leaves instance correspondence unresolved.	M
Generative implicit alignment	4M [52]; BLIP [87]	Masked targets or image-conditioned captions supervise generation.	Redundant context can predict targets without the intended evidence.	M
Constraint and routing regularization	SPECTRA [88]; MoIIE [30]	Temporal constraints synchronize streams; router penalties balance utilization.	Synchronization and expert utilization need different checks; routing balance is an auxiliary target.	M
Instruction and preference alignment	LLaVA [19]; VLM-RLAIF [89]; RLHF-V [90]	Demonstrations specify responses; judgments compare candidate behavior or unsupported content.	Judge errors, narrow pair coverage, and textual shortcuts can survive preference gains.	M, I
Verifiable rewards	Visual-RFT [91]	Task-specific verifiers score sampled outputs for group-relative policy updates.	Correct outcomes leave intermediate reasoning and unsupervised safety constraints unverified.	M, I

M: Modality Gap; I: Intent Gap; M, I: can target both gaps, the Modality Gap only when supervision requires supporting evidence.

Table 3. Objective families for multimodal alignment, compared by supervision and learning signal and by the limitation discussed. Gap tags mark intended targets.

Objective family	Examples	Supervision and learning signal	Limitation discussed	Gap
Fine-grained structured contrastive	GLoRIA [83]; GOAL (Choi et al.) [84]	Region–word or object matches are contrasted with competing candidates.	Incorrect correspondences and false negatives corrupt local supervision.	M
Topological and distributional metric	$A^3$$D^2$ [85]; GRAM [86]	Transport matches distributions; Gramian volume constrains joint geometry.	Global agreement leaves instance correspondence unresolved.	M
Generative implicit alignment	4M [52]; BLIP [87]	Masked targets or image-conditioned captions supervise generation.	Redundant context can predict targets without the intended evidence.	M
Constraint and routing regularization	SPECTRA [88]; MoIIE [30]	Temporal constraints synchronize streams; router penalties balance utilization.	Synchronization and expert utilization need different checks; routing balance is an auxiliary target.	M
Instruction and preference alignment	LLaVA [19]; VLM-RLAIF [89]; RLHF-V [90]	Demonstrations specify responses; judgments compare candidate behavior or unsupported content.	Judge errors, narrow pair coverage, and textual shortcuts can survive preference gains.	M, I
Verifiable rewards	Visual-RFT [91]	Task-specific verifiers score sampled outputs for group-relative policy updates.	Correct outcomes leave intermediate reasoning and unsupervised safety constraints unverified.	M, I

M: Modality Gap; I: Intent Gap; M, I: can target both gaps, the Modality Gap only when supervision requires supporting evidence.

3.1. Fine-Grained Structured Contrastive Loss

This family aligns local units (regions, tokens, or their combinations) and calibrates which candidates count as negatives. Let $v_i$ and $t_j$ be normalized visual and textual units, $A_{ij}\in \{0,1\}$ indicate an annotated or mined positive correspondence, and $s_{ij}=v_i^{\top }{t}_j/\tau$ denote temperature-scaled similarity. A canonical multi-positive objective is

$${\mathcal{L}}_{\mathrm{FGC}}=-{\displaystyle \frac{1}{\left|\mathcal{I}\right|}}\sum _{i\in \mathcal{I}}\log {\displaystyle \frac{{\sum }_j{A}_{ij}\exp \left(s_{ij}\right)}{{\sum }_k\exp \left(s_{ik}\right)}},$$

(1)

where $\mathcal{I}$ indexes anchors and $\tau$ controls concentration. Matrix A supplies positive supervision; other candidates are negatives unless masked.

3.1.1. Precise Local Semantic Anchoring

CLIP’s global loss does not locate sentence-supporting evidence [92]. GLoRIA aligns medical regions with report words [83] (Section 7.3), GOAL combines global and local object alignment [84], and TACo uses token-aware cascaded video–text objectives [93]. PowerCLIP links region combinations to parse-tree nodes through bidirectional triplet losses [94]. UCoFiA contrastively learns coarse-to-fine similarities and applies Sinkhorn–Knopp normalization only at retrieval inference [95]. Fine-grained objectives are informative only when their correspondence labels are reliable; otherwise localization errors become supervision.

3.1.2. Structured Compositional Logic Constraint

Pairwise similarity can overlook incorrect relations or event order. TupleInfoNCE contrasts multimodal tuples [96], VideoCon perturbs caption entities, actions, attributes, relations, or order [97], and FormNetV2 applies multimodal graph contrast to documents [98]. Negative construction thus fixes which compositional errors each objective can expose, and ambiguous or partially correct negatives can create contradictory supervision.

3.1.3. Latent-Space Distribution Calibration

Duplicate semantic content and correlated samples can make unpaired items unreliable negatives. Calibration therefore reweights, masks, or groups candidates in (1), and audio–visual spatial alignment also uses physical co-location as a correspondence cue [99]. Reweighting changes a candidate’s influence, masking removes its constraint, and grouping changes the positive set; all three need a defensible notion of semantic equivalence, and batch-level geometry alone cannot establish instance-level correspondence.

3.2. Topological and Distributional Metric Loss

Set-level objectives align distributions, using optimal transport for many-to-many matching or geometric measures for higher-order structure. For empirical distributions $\mu ={\sum }_i{a}_i{\delta }_{v_i}$ and $\nu ={\sum }_j{b}_j{\delta }_{t_j}$, entropic optimal transport gives

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{OT}}& =\underset{P\in U(a,b)}{\min }\langle P,C\rangle -\epsilon H\left(P\right),\hfill \\ \hfill U(a,b)& =\{P\ge 0:P\mathbf{1}=a,P^{\top }\mathbf{1}=b\},\hfill \end{array}$$

(2)

where $C_{ij}$ is the transport cost, P the soft assignment, $a,b$ the feature masses, $H\left(P\right)$ the entropy, and $\epsilon$ the smoothing strength. In (2), paired sets, masses, and costs replace one-to-one labels.

3.2.1. Optimal-Transport Soft Matching

$A^3$$D^2$ adapts representations, aligning shifted modalities to domain-stable anchors through dimension-wise optimal transport and cross-domain adversarial alignment [85]. ProtoMM instead acts at test time, combining transport-based comparison with weighted updates of multimodal prototypes while network weights stay fixed [100]. Even a smooth transport plan is limited by the semantic validity of its features and costs.

3.2.2. High-Dimensional Geometric Metrics

Geometric objectives preserve neighborhoods, centroids, ranks, or volumes omitted by pairwise cosine similarity. GRAM minimizes a Gramian volume derived from the determinant of a multimodal Gram matrix [86], so several modalities are constrained jointly instead of anchored pairwise. Such agreement stays at the representation level; a smaller geometric loss shows neither downstream grounding nor numerical stability.

3.3. Generative Implicit Alignment Loss

Generative objectives align modalities implicitly by predicting missing or subsequent tokens. For target positions $\mathcal{T}$ and visible input $x_{\mathrm{vis}}$,

$${\mathcal{L}}_{\mathrm{gen}}=-\sum _{u\in \mathcal{T}}\log p_{\theta }(y_u\mid x_{\mathrm{vis}},c_u),$$

(3)

where $y_u$ is a discrete target. Autoregressive context $c_u$ includes the preceding target prefix, whereas masked prediction uses the context permitted by its attention mask. Diffusion and other continuous-target losses fall outside (3).

3.3.1. Cross-Modal Masked Reconstruction

4M and 4M-21 reconstruct masked modality tokens [52,53]; 4M uses masked-content prediction for image-like targets and causal prediction for sequence-like targets. The modality-specific mask determines $c_u$, so redundant visible context can bypass the intended correspondence.

3.3.2. Autoregressive and Modality-Specific Generation

BLIP combines image-conditioned captioning with contrastive and matching objectives [87]. Modality-specific parameters do not dictate the generative loss. MoT’s Chameleon setting predicts discrete text and image tokens autoregressively, whereas its Transfusion setting pairs text prediction with image diffusion [28]; MoIIE mainly uses language-model cross-entropy with routing regularization [30].

MiCo [101] combines contrast, feature matching, and text generation through a shared multimodal encoder and contextual embeddings. Conditional generation encourages output compatibility, whereas auxiliary matching losses constrain input representations more directly; even combined, they leave open whether generated content depends on the intended evidence.

3.3.3. Self-Guided Proxy Supervision

Without dense labels, XDC exchanges cross-modal cluster assignments [102], whereas MERLOT predicts masked language from ordered video [103]. Cluster agreement and contextual prediction constrain different dependencies, and a successful proxy need not identify the evidence behind a particular response.

3.4. Constraint and Routing Regularization

Regularizers add temporal, spatial, or information-flow and routing constraints to the task loss:

$$\mathcal{L}={\mathcal{L}}_{\mathrm{task}}+{\lambda }_t{\mathcal{R}}_{\mathrm{time}}+{\lambda }_s{\mathcal{R}}_{\mathrm{space}}+{\lambda }_b{\mathcal{R}}_{\mathrm{balance}},$$

(4)

where ${\mathcal{L}}_{\mathrm{task}}$ is contrastive or generative and each $\lambda$ weights an optional penalty. Timestamps, coordinates/relations, or router assignments supervise these auxiliary constraints, which are not complete alignment objectives.

3.4.1. Spatiotemporal Synchronization

SPECTRA combines temporal position prediction and cross-modal response selection [88]. The right ${\mathcal{R}}_{\mathrm{time}}$ in (4) depends on the task: word localization needs boundary accuracy, event reasoning may need order, and loose narration needs delay tolerance, so exact synchronization can penalize valid pairs.

3.4.2. Spatial and Topological Grounding

Spatial penalties connect words to regions, pixels, or 3D entities. World-to-Words supervises grounded open-vocabulary acquisition [104], and OC-CLIP uses object-centric binding in contrastive pretraining [105]. Both rely on localized evidence at different units of correspondence; one-to-one matching does not suit abstract words or diffuse evidence.

3.4.3. Information-Flow and Routing Balance

Functional-entropy regularization targets modality dominance [106]. For expert models, MoIIE’s load-balancing penalty [30] follows Switch Transformer [107]: ${\mathcal{R}}_{\mathrm{balance}}=\left|\mathcal{E}\right|{\sum }_e{f}_e{p}_e$, where $\mathcal{E}$ is the expert set, $f_e$ the assigned batch fraction, and $p_e$ the mean router probability. Balanced experts can still process language shortcuts: utilization and overflow measure computational balance, and only controlled input changes test evidence dependence.

3.5. Instruction and Preference Alignment Objectives

Response-level objectives address the Intent Gap: supervised fine-tuning (SFT) imitates targets, reward modeling learns comparative scores, proximal policy optimization (PPO) optimizes rewarded behavior under a reference-policy constraint, and direct preference optimization (DPO) fits the policy to preference pairs. Their effect on the Modality Gap depends on whether supervision separates grounded from unsupported responses.

3.5.1. Token-Level Supervised Fine-Tuning

For multimodal input x, instruction q, and target response $y=(y_1,\dots ,y_T)$, SFT minimizes

$${\mathcal{L}}_{\mathrm{SFT}}=-{\displaystyle \sum _{t=1}^T}\log {\pi }_{\theta }(y_t\mid x,q,y_{<t}).$$

(5)

LLaVA and InstructBLIP imitate demonstrated answers [19,108] without comparing plausible alternatives. If instructions or preceding tokens predict the target, the loss in (5) can decrease without image use. Section 4.1 and Section 5 discuss evidence-grounded targets and their training use.

3.5.2. Reward Modeling and PPO

Given a preferred response $y^{+}$ and rejected response $y^{-}$, a Bradley–Terry reward model minimizes

$${\mathcal{L}}_{\mathrm{RM}}=-\log \sigma \left(r_{\varphi }(x,q,y^{+})-r_{\varphi }(x,q,y^{-})\right),$$

(6)

where $r_{\varphi }$ is a learned scalar reward and $\sigma$ is the logistic function. PPO then maximizes reward using the clipped surrogate $E\left[\min ({\rho }_t{\widehat{A}}_t,clip({\rho }_t,1-\delta ,1+\delta ){\widehat{A}}_t)\right]$, with policy ratio ${\rho }_t={\pi }_{\theta }(y_t\mid s_t)/{\pi }_{\mathrm{old}}(y_t\mid s_t)$ for generated token $y_t$ in state $s_t=(x,q,y_{<t})$, advantage ${\widehat{A}}_t$, and clipping range $\delta$, typically plus a Kullback–Leibler (KL) penalty to a reference policy. InstructGPT uses this pipeline [109], and VLM-RLAIF applies it to video–language models with AI feedback [89] (Section 5). The main risk is reward misspecification or judge bias, amplified by on-policy optimization.

3.5.3. Direct Preference Optimization

DPO [110] updates the policy directly, without the explicit reward model of (6) or a PPO policy-gradient loop. With reference policy ${\pi }_{\mathrm{ref}}$ and reference-regularization coefficient $\beta$, it minimizes

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{DPO}}=-\log \sigma \left(\beta \right[& \log {\displaystyle \frac{{\pi }_{\theta }(y^{+}\mid x,q)}{{\pi }_{\mathrm{ref}}(y^{+}\mid x,q)}}\hfill \\ \hfill & -\log {\displaystyle \frac{{\pi }_{\theta }(y^{-}\mid x,q)}{{\pi }_{\mathrm{ref}}(y^{-}\mid x,q)}}\left]\right).\hfill \end{array}$$

(7)

Multimodal variants apply (7) to grounded versus unsupported responses or to image-conditioned alternatives [111,112,113]. Style or length can also reveal preferences without image use, a risk that pair construction (Section 4.3) and policy updates (Section 5) must each control.

3.5.4. Verifiable Rewards and Group-Relative Optimization

Reinforcement learning with verifiable rewards (RLVR) uses task-specific checks. Visual-RFT combines classification/localization rewards with Group Relative Policy Optimization (GRPO) [91], which compares samples from the same input through group-relative advantages, without a separately trained value model. Outcome verification leaves intermediate reasoning unverified, and safety and preference claims require constraints beyond the optimizer itself.

No objective is more reliable than its signal. Noisy correspondence labels, ambiguous negatives, invalid transport costs, judge bias, and unverified RLVR reasoning limit what a lower loss shows about either gap. Most of these signals are built during data construction (Section 4).

4. Data Construction: Building Alignment Signals

Data construction supplies the supervision that the objectives of Section 3 consume; an objective can penalize only the distinctions its data makes visible (Table 4). Pairs and hard negatives mainly target the Modality Gap, preference and safety labels the Intent Gap, and faithfulness corrections both. We compare strategies by annotation validity and coverage rather than size, and separate training resources from evaluation-only benchmarks.

4.1. High-Quality Pairing and Annotation

Pairs reduce the Modality Gap only when they expose supporting evidence (regions, annotation processes, spatiotemporal or embodied traces, or domain-specific structure) rather than loose co-occurrence.

4.1.1. Region-Level Grounding

ReferItGame and RefCOCOg identify referred objects [120,121]; Flickr30k Entities adds phrase–region links and coreference, and Visual Genome adds attributes, relations, and scene graphs [114,122]. Object annotations specify targets, whereas relational labels also constrain their connections, so evaluation should match the claimed granularity.

4.1.2. Process-Level Grounding

Localized Narratives synchronizes descriptions with mouse traces over 849k images [123]. Unlike boxes, traces preserve mention order and evidence attribution, but record the annotator’s process rather than the model’s reasoning.

4.1.3. Spatiotemporal and Embodied Grounding

RxR links instruction words to pose traces [115], and Landmark-RxR adds sub-trajectories and endpoint landmarks [124]. Grounded Video Description links noun phrases to boxes in video frames [125], SPECTRA supplies word-level speech boundaries [88], and HarmonySet labels video–music rhythm, emotion, theme, and cultural relevance [126]. Pose, region, and timing annotations specify different types of correspondence; semantic compatibility extends beyond synchronization.

4.1.4. Domain-Specialized Evidence

UCM-VeID V2 diversifies aerial RGB–infrared pairs for vehicle re-identification [127]; Zhang et al. preserve hierarchical-table adjacency and headers [128]. VLP-MABSA and World-to-Words (Section 3) ground opinions or new words in visual evidence [104,129]. These adjacent tasks require identity-, structure-, or sentiment-specific correspondence, which generic pair counts cannot measure.

4.2. Synthetic and Self-Supervised Alignment Data

Natural structure, model predictions, and generated examples replace dense annotation with approximate correspondence of varying reliability.

4.2.1. Naturally Occurring Weak Supervision

AVTS contrasts synchronized and temporally mismatched streams [130]. XDC and SeLaVi cluster unlabeled videos across modalities [102,131], complementing MERLOT’s masked prediction and PyramidCLIP’s negative sampling [103,132]. CLIP and ALIGN use noisy web pairs [92,133], and HowTo100M uses narrated clips [134]. These signals scale differently from precise object or event annotation and are not equivalent supervision.

4.2.2. Model-Based Refinement and Selection

Refinement changes content: ALBEF uses momentum distillation [135], BLIP generates and filters captions [87], and VeCLIP enriches alt-text [136]. Selection decides which samples are included: Align2LLaVA filters instructions using human criteria and LLM-aware selection [137], DataComp standardizes selection experiments and Data Filtering Networks learn filters [116,138], MetaCLIP balances metadata concepts [139], and HYPE removes vague pairs [140]. JEST’s jointly learnable batches also decide which samples co-occur [141]. DataComp’s fixed recipe isolates selection more directly than cross-model scores but does not establish optimal instruction-tuning or preference filters.

4.2.3. Structure-Derived Self-Supervision

MIL-NCE permits nearby narration matches [142], LF-VILA combines temporal contrast with hierarchical video attention [143], MMV joins video, audio, and text signals [144], and CoMMA grounds narration without manual boxes [145]. ToB uses 3D geometry, GRAFT co-located ground and satellite views, and ULIP-2 rendered image–text–3D triplets [146,147,148]. Temporal neighborhoods tolerate uncertain pairing; geometry and co-location impose acquisition constraints whose validity can change with sensors or scenes.

4.2.4. Model-Generated Targeted Data

GLIP supplies pseudo boxes [149], CoDet mines region–word links across images that share concepts [150], and Kosmos-2 links noun phrases to location tokens [151]. SPARCL instead generates minimally differing object, attribute, and relation pairs [152], and the SVLTA benchmark renders synthetic video situations to test temporal alignment [153]. Positive links identify supporting evidence and controlled alternatives identify relevant distinctions, but generation must preserve unrelated content to prevent artifact shortcuts.

4.3. Multimodal Preference Data Construction

Preferences constrain helpfulness, safety, and factuality for the Intent Gap and narrow the Modality Gap only when comparisons depend on visual support. Losses and updates appear in Section 3 and Section 5; here we compare what a judgment contains and who produced it.

4.3.1. Multi-Dimensional Rubrics

MM-RLHF provides image and video ratings, rankings, and critiques [117]; OmniAlign-V combines alignment examples with human evaluation [154]. Text-only RewardBench and SafetyBench contribute ranking and safety methodology, not multimodal training data [155,156]. In adjacent image generation, Q-Eval-100K separates visual quality and semantic alignment [157], and the Pick-High dataset is paired with an Image-Contained-Text reward for prompt content and an image-only High-Preference reward for aesthetics and detail [158]. Separating datasets, rewards, and rubrics prevents a single preference label from concealing conflicting criteria.

4.3.2. Hallucination and Faithfulness Correction

RLHF-V uses human corrections to response segments [90], HA-DPO uses style-matched GPT-4 corrections [111], and CHiP combines response-, segment-, and token-level preferences [159]. CHAIR-DPO ranks object hallucinations, and HalLoc and ZINA label unsupported spans [160,161,162]. OPA-DPO first adapts to ground-truth and corrected responses to reduce policy mismatch [163]. Localization, style control, and policy matching address distinct confounders, yet precise corrections still mislead if their wording reveals the preference label.

4.3.3. Annotation Sources and Judge Reliability

Human judgments remain important in high-risk domains despite their cost. VLM-RLAIF instead has a supervised video–language model judge its candidate responses [89] (Section 5), but the Silent Judge study and FairJudge identify evaluation biases from position, response length, and provenance [164,165]. ViLBench collects process-level reward data through tree search [166]. ARM-Thinker’s tools (image cropping, document retrieval, instruction verification) add external checks to reward judgments but do not provide step-level process supervision [167]. IRIS filters self-generated pairs using implicit rewards [168], and CycleReward derives preferences from image–text reconstruction consistency [169]; both reduce dependence on human annotation. Beyond response generation, DSPO adapts preference learning to diffusion score matching, and MAPLE transfers fine-grained signals to cross-modal retrieval embeddings [170,171]. Agreement within an automated pipeline is not independent validation, so these judgments need task-specific human or evidence-based checks.

4.4. Negative Sampling and Hard Negatives

Hard negatives isolate object, attribute, relation, or temporal mismatches that random sampling misses.

4.4.1. Lexical and Visual Perturbation

Text perturbations expose unsupported responses to a fixed image. POVID and RLAIF-V construct hallucinated negatives through error injection or atomic-claim scoring [113,172], SRCL weakens penalties on semantically close negatives, and RPO refines on-policy pairs through guided reflection [173,174]; CLIP-DPO forms preferences from CLIP similarity without new labels [175]. Input edits test visual dependence: Re-Align retrieves similar images and MIA-DPO inserts distractor images [176,177]; Image-DPO applies semantic edits, blur, or pixelation, and AdViP removes key visual elements [178,179]. AdPO updates the image encoder for clean and adversarial inputs [180], MSR-Align constructs policy-grounded safety reasoning data [181], and VisuoAlign builds safety-alignment trajectories through multimodal tree search [182]. Response and input edits target the Modality Gap by testing faithfulness and evidence use; policy-grounded negatives target the Intent Gap by testing acceptable behavior. Robustness and refusal quality therefore need separate evaluation.

4.4.2. Compositional and Temporal Negatives

Generation supplies contrasts absent from retrieved data. SynViTA generates videos for negative captions and weights samples to mitigate generation noise [183], VideoCon and SeeTRUE/ConGen minimally change captions to contradict images or videos [97,184], and VideoComp reorders events, replaces actions, and mismatches video–text segments, challenging bag-of-words shortcuts [185]. Mining draws on observed data instead: TACo mines hard pairs within a batch using sentence- and token-level similarity, and TupleInfoNCE replaces selected modalities to create partially mismatched tuples [93,96]. CONE contrasts positive and randomly sampled negative windows with coarse-to-fine proposal ranking [186]. Mining keeps observed data but may miss rare contradictions; generation targets specific failures but can add synthesis artifacts. In both cases a hard negative helps only if it stays semantically valid and isolates the intended error.

4.4.3. Representation-Level Constraints

MM-EMBED uses modality-aware hard-negative mining during retrieval fine-tuning, whereas CLIP-Refine aligns features through post-pretraining before task-specific fine-tuning [187,188]. AVSA adds spatial alignment for ${360}^{\circ }$ video, and AMiF combines hard-negative mining with selective fusion of the task’s answer vocabulary in medical VQA [99,189]. Because hard negatives alone may be insufficient, OC-CLIP introduces object-centric biases and scene-graph parsing for multi-object scenes [105], and READ adds token reconstruction through a frozen decoder and sentence-level paraphrase alignment, improving compositional reasoning while retaining CLIP’s retrieval architecture [190]. Sampling changes which mismatches are observed; structural biases and reconstruction change how they are represented. Controlled ablations should separate the two before crediting gains to harder negatives.

4.5. Dataset Bias and Debiasing

Bias can suppress visual evidence (a Modality Gap failure) or favor inappropriate behavior (an Intent Gap failure). Interventions change distributions, gradients, or coverage.

4.5.1. Breaking Language Priors and Shortcut Learning

Models often exploit textual co-occurrence rather than rely on visual evidence. LexSym encourages compositional generalization through consistent substitutions across text and image representations [191], and NaPO trains on modality-biased preference pairs generated by masking inputs while downweighting noisy supervision [118]. Causal methods intervene at different points: CRA uses front-door visual and back-door linguistic interventions for video grounding, whereas CausalMM uses back-door adjustment and counterfactual attention to mitigate hallucinations [192,193]. As diagnostics, DrML probes visual classifiers through text embeddings, and VLMBias tests whether familiar visual patterns override observed details [194,195]. Data substitutions, preference reweighting, and causal interventions act at different stages, and removing one shortcut does not close either gap across tasks.

4.5.2. Optimization and Gradient-Level Debiasing

Conflicting gradients and unequal learning difficulty can suppress harder modalities. Gat et al.’s functional-entropy regularizer (Section 3) counters modality dominance [106], and ARL weights updates through prediction variance and auxiliary bias terms rather than requiring equal dependence [196]. Chen et al.’s gradient-balancing GOAL rescales fusion-loss gradients using predictive entropy and removes conflicting cross-modal components [197]; it is distinct from Choi et al.’s global–local object-alignment GOAL [84]. DisentangledSSL separates shared and modality-specific information using conditional mutual information [198]. These optimization-side controls complement data curation: balancing updates counters dominance, and decomposition preserves useful modality differences.

4.5.3. Addressing Long-Tail, Safety, and Social Biases

Web data overrepresents common objects and settings, so specialized domains need targeted coverage. MMedPO targets medical-imaging preferences [199], and the evaluation-only Chart2Code benchmark exposes shortfalls in chart-to-code generation [200]. VisCon-100K’s “leaky modality mix” lets training questions draw on both images and contextual descriptions; the term names a supervision design, not test-set contamination [201]. For safety, VLGuard and MMSafe-PO pair harmful requests with appropriate refusals [119,202], and ReGap targets the representation separation left by pretraining and reduces unsafe response rates in the evaluated settings [203]. For social bias, counterfactual studies show that visual attributes such as race and gender can change the likelihood of toxic output, and BLEnD-Vis reveals weaknesses in multi-regional cultural knowledge [204,205]. Unified MLLMs also inherit demographic generation bias from their language component, which Balanced Preference Loss reduces without degrading image quality [206]. Broader coverage supplies missing evidence but need not make responses safer or fairer, so coverage and behavioral supervision need separate subgroup and utility checks.

Each source of supervision brings its own confounder: annotator processes, synthesis artifacts, judge bias, corrections whose wording reveals the label, or limited coverage. Section 5 asks which modules these signals update, with which mixture, and in which order.

5. Training: A Staged Process for Multimodal Alignment

Training decides where and when the losses of Section 3 and the supervision of Section 4.1, Section 4.2, Section 4.3, Section 4.4 and Section 4.5 are applied, and what is checked between stages. The emphasis moves from correspondence in pretraining to behavior in instruction tuning (SFT) and preference and reward optimization, that is, from the Modality Gap toward the Intent Gap. Later updates must still preserve grounding and safety [119]. Table 5 summarizes each stage.

5.1. Stage 1: Multimodal Pretraining

Pretraining sets the correspondence that later stages inherit.

5.1.1. Frozen Versus Jointly Updated Components

BLIP-2 trains a Q-Former with a fixed image encoder before connecting a frozen LLM [18]. Freezing confines adaptation to this interface and preserves pretrained weights but cannot recover what the fixed features lack. Joint updates can change the evidence representation, while text replay and separate learning rates limit forgetting. The diagnosed bottleneck, more than parameter count, should guide this choice.

5.1.2. Objective Scheduling and Batch Construction

CLIP learns from paired batches, and EVA-CLIP revisits initialization, optimization, and augmentation [92,213]. SigLIP’s pairwise loss reduces global-softmax batch dependence [214]; ALBEF and BLIP coordinate matching, contrastive, and predictive losses [87,135]. Comparisons of pairwise losses and loss schedules should match examples and updates while controlling false negatives from semantic duplicates.

5.1.3. Data Composition and Transition Criteria

Sources set the evidence encountered, interleaved contexts require image–text integration, text replay preserves language coverage, and objective mixtures set gradient targets. PaLI varies multilingual tasks; CoCa combines captioning and contrast [215,216]. In MM1’s ablation (Figure 5a in [207]), replacing caption-only data with a 50:50 caption/interleaved mix lowers mean zero-shot performance from 39.3 to 33.4 but raises eight-shot performance from 45.0 to 62.2. No single ratio serves both regimes, so transition checks should cover grounding, language retention, and the intended prompting regime. Qwen2.5-VL chains vision–language alignment, multimodal pretraining, long-context adaptation, and supervised and preference post-training [208]; unlike frozen-backbone recipes, its stages add input complexity and trainable parameters.

5.2. Stage 2: Instruction-Supervised Fine-Tuning

SFT minimizes (5); recipes differ in which modules adapt, how tasks are sampled, and what the targets contain.

5.2.1. Trainable Parameters and Optimization Schedule

LLaVA first aligns a projector, then updates it and the LLM with a fixed vision encoder [19], whereas InstructBLIP freezes both backbones and updates an instruction-aware Q-Former [108]. The former reshapes the language policy; the latter mainly conditions evidence extraction. Reports should name the updated modules and test retained grounding.

5.2.2. Mixture Sampling During SFT

InternVL mixes captioning, VQA, OCR, grounding, and conversation [217], and Cambrian-1 studies task trade-offs under fixed data budgets [218]. Molmo and Molmo2 reweight grounding and selected synthetic sources [219,220]; OmniAlign-V tests preferences alongside capabilities [154]. Gains may reflect coverage or sampling weight, so ratios should be compared at fixed updates and response length before crediting more data.

5.2.3. Evidence-Grounded Instruction Targets

SFT addresses the Modality Gap mainly when its targets carry evidence. CogCoM trains on chain-of-manipulation data [221], and LATTE on reasoning traces synthesized over vision-specialist outputs [222]; both invoke manipulations or tools at inference. VisProg and ViperGPT instead compose executable programs without task-specific training [223,224]. Learned tool use and observable program execution thus allow different checks of reasoning. POPE and MMHalBench diagnose remaining hallucinations, while MME tests broader perception/cognition [1,225,226]; neither readable traces nor these scores alone establish faithfulness or calibration.

5.3. Stage 3: Preference and Reward Optimization

Preference updates depend on judge validity, sampled candidates, reference strength, and coverage of current-policy failures.

5.3.1. Reward-Model-Based Optimization

Learning rewards from AI feedback, VLM-RLAIF optimizes a video–language policy with PPO [89]. Judge errors can propagate into policy updates, so judges need held-out human agreement and validation on policy-generated candidates; reports should specify judge prompts, rollout sampling, and KL strength. Rewards need not imply PPO: MM-RLHF uses critique scores to weight MM-DPO [117].

5.3.2. Offline Preference-Pair Optimization

RLHF-V derives Dense DPO from human corrections [90]; HA-DPO, mDPO, and POVID supply style-controlled, image-conditioned, and injected-hallucination comparisons [111,112,113], discussed in Section 4.3, Section 3, and Section 4.4, respectively. Fixed pairs aid reproducibility but can miss current-policy failures. Testing preference margins after image replacement, with style and length controlled, separates visual learning from response-artifact classification.

5.3.3. Verifiable-Reward Fine-Tuning

Visual-RFT’s verifier-based GRPO (Section 3) learns from current-policy samples [91]. Unlike demonstrations, they expose the policy’s own errors; unlike learned judges, verifiers impose specified correctness rules. Group size, filtering, reward composition, and task coverage shape training behavior, and faithful-reasoning claims need intermediate checks beyond outcome rewards.

5.4. Inference-Time Intervention and Deployment Checks

Without parameter updates, VCD contrasts predictions under original and distorted images [209], whereas OPERA penalizes attention over-concentration and reallocates decoding choices [210]; both target evidence use during decoding. Woodpecker instead verifies and corrects completed responses [211]. Grounding, utility, and extra calls should be compared under matched inference budgets.

Later adaptation can weaken alignment gained at earlier stages, so safety checks must span all stages. MultiTrust and VLGuard provide multimodal testing and safety supervision [119,212], and studies of indirect prompt injection in LLM-integrated applications add adjacent context [227]. Released behavior should be validated on harmful and benign requests, with false refusals reported. Section 6 asks what it takes to accept an alignment claim at any stage.

6. Evaluation: How to Measure Alignment

Interventions in the four dimensions above aim to reduce the Modality Gap, the Intent Gap, or both. Evaluation checks whether they do, testing evidence use and requirement-compliant behavior separately. Table 6 organizes these tests by representation, task, and behavior level.

Table 6. Alignment tests at representation, task, and behavior levels, with supporting diagnostics. Each row’s reference or contrast limits what it can establish.

Level / role	Examples	Reference or contrast	Measured property	Unresolved attribution	Gap
Representation	SAIL [228]; MIR [229]	Mapped representations; layer distributions	Compatibility or integration	Whether the decoder uses the information	M
Task	Winoground [230]; VSR [231]; SVLTA [153]	Compositional, spatial, or temporal contrasts	Sensitivity to relevant distinctions	Shortcuts outside the tested contrasts	M
Behavior: response grounding	POPE [1]; CHAIR [232]	Object-presence annotations	Unsupported object claims	Error location and causal mechanism	M
Behavior: trust and safety	MultiTrust [212]; VLGuard [119]	Trustworthiness tests; harmful and benign requests	Constraint compliance and retained utility	Coverage of unseen settings and attacks	M, I
Reasoning diagnosis	MIRAGE [233]	Verified perception followed by reasoning	Inference failures given evidence	Whether a gap is involved	–
Reward judge	VL-RewardBench [234]; Multimodal RewardBench [235]	Human-validated preferences	Judge agreement by failure category	New policy-generated candidates	M, I
Reference metric	CIDEr [236]	Reference captions	Descriptive consensus	Evidence use and constraint compliance	Indirect
Controlled protocol	mDPO [112]; controlled input changes	Fixed model and settings; valid updated references	Sensitivity to the claimed intervention	Distribution-shift artifacts and component interactions	M, I

M: Modality Gap; I: Intent Gap; M, I: complementary tests of both gaps (a single benchmark may test only one); –: no gap attribution alone; Indirect: supporting signal only. Controls support all three levels.

Table 6. Alignment tests at representation, task, and behavior levels, with supporting diagnostics. Each row’s reference or contrast limits what it can establish.

Level / role	Examples	Reference or contrast	Measured property	Unresolved attribution	Gap
Representation	SAIL [228]; MIR [229]	Mapped representations; layer distributions	Compatibility or integration	Whether the decoder uses the information	M
Task	Winoground [230]; VSR [231]; SVLTA [153]	Compositional, spatial, or temporal contrasts	Sensitivity to relevant distinctions	Shortcuts outside the tested contrasts	M
Behavior: response grounding	POPE [1]; CHAIR [232]	Object-presence annotations	Unsupported object claims	Error location and causal mechanism	M
Behavior: trust and safety	MultiTrust [212]; VLGuard [119]	Trustworthiness tests; harmful and benign requests	Constraint compliance and retained utility	Coverage of unseen settings and attacks	M, I
Reasoning diagnosis	MIRAGE [233]	Verified perception followed by reasoning	Inference failures given evidence	Whether a gap is involved	–
Reward judge	VL-RewardBench [234]; Multimodal RewardBench [235]	Human-validated preferences	Judge agreement by failure category	New policy-generated candidates	M, I
Reference metric	CIDEr [236]	Reference captions	Descriptive consensus	Evidence use and constraint compliance	Indirect
Controlled protocol	mDPO [112]; controlled input changes	Fixed model and settings; valid updated references	Sensitivity to the claimed intervention	Distribution-shift artifacts and component interactions	M, I

M: Modality Gap; I: Intent Gap; M, I: complementary tests of both gaps (a single benchmark may test only one); –: no gap attribution alone; Indirect: supporting signal only. Controls support all three levels.

6.1. Evaluation Objectives and Diagnostic Challenges

Correct answers can come from dataset patterns [1,237], and fluent responses can remain unsupported [13,238], so scores need evidence-dependence controls.

6.1.1. Correctness and Evidence Grounding

Correctness measures agreement with a reference answer; whereas grounding measures support from the multimodal input. An answer can be correct yet ungrounded, or grounded yet incomplete, so both should be reported [1,13].

6.1.2. Diagnosing the Modality and Intent Gaps

In Figure 2, the answer to a color question should change when the mug’s color changes (Modality Gap), and the action should change when lifting is authorized (Intent Gap). A wrong calculation after verified perception remains a reasoning error unless further tests implicate either gap.

6.1.3. Confounders and Shortcut Controls

Contamination, answer imbalance, co-occurrence, prompts, and response style can confound scores [239,240,241,242]. Overlap checks test exposure, whereas input and style interventions test reliance on unintended cues; together they help separate memorization from grounding.

6.1.4. Hallucination, Reasoning, and Calibration

Output support, inference validity, and confidence relative to correctness are distinct. VidHalluc tests output support [243], MIRAGE isolates reasoning-chain errors after correct perception [233], and text-only Answer-Free Confidence Estimation contributes calibration methodology, not multimodal results [244]. Lower hallucination rates therefore do not establish calibrated refusal.

6.2. Evaluation Across Representation, Task, and Behavior Levels

The three levels separate available information, task-specific use, and behavior under requirements. They mark where a failure is observed, not which gap causes it, so a behavior-level probe such as POPE can expose a Modality Gap failure.

6.2.1. Representation-Level Evaluation

SAIL probes mapped correspondence of individual image–text pairs [228], whereas Modality Integration Rate (MIR) measures layer-wise distribution distances [229]; SPECTRA’s speech timestamps and HarmonySet’s video–music labels capture temporal and semantic compatibility [88,126]. Such measurements show what is available, not whether the decoder uses it.

6.2.2. Task-Level Evaluation

Winoground and VALSE test compositional and linguistic distinctions [230,245], VSR spatial relations [231], and ARO attributes, relations, and order [246]; SugarCrepe exposes artifacts in compositional negatives [247]. VidHalluc tests video action, order, and transition hallucinations, and SVLTA targets temporal alignment [153,243]. Static relations and temporal synchronization need different contrasts, with plausible negatives that language alone cannot resolve.

6.2.3. Behavior-Level Evaluation

POPE probes object hallucinations, and HallusionBench tests illusions and errors involving perception and language priors [1,2]. MMHalBench and MM-RLHF assess factuality and response quality [117,225], and MultiTrust spans truthfulness, robustness, fairness, privacy, and safety [212]. Interactive use also needs cross-turn consistency and revision after evidence changes.

6.3. Metrics and Evaluators for Alignment

The metric follows the claim: correspondence, response support, or learned-judge reliability.

6.3.1. Retrieval and Localization Metrics

CLIP and ALIGN report recall at K for retrieval [92,133]; localization applies overlap or point criteria to labels such as RefCOCOg and Flickr30k Entities [114,121]. Temporal metrics should separate boundary accuracy, event order, and synchronization tolerance, since required timing precision varies by task.

6.3.2. Reference-Based Generation Metrics

BLEU, METEOR, CIDEr, and SPICE measure reference overlap, consensus, or semantic structure [236,248,249,250]; CHAIR detects unsupported object claims that such scores miss [232]. Reporting both separates descriptive quality from unsupported additions.

6.3.3. Multimodal Reward-Model Evaluation

Human-verified preferences in VL-RewardBench cover general queries, hallucination, and reasoning [234]; Multimodal RewardBench broadens categories to correctness, knowledge, safety, and VQA [235]; MM-RLHF evaluates critique-based rewards [117]. Evaluations should report category-wise agreement and control candidate order, length, and image dependence. Accurate ranking of fixed pairs evaluates the judge, not policy generation or reliability on newly optimized outputs.

6.4. Controlled Protocols and Interpretation

Attributing a gain to an intervention requires within-study contrasts with fixed models and settings, which also expose trade-offs hidden by pooled scores.

6.4.1. Connecting Alignment Claims to Evidence

In mDPO’s matched Bunny-v1.0-3B comparison (Table 1 in [112]), DPO-to-mDPO lowers Object HalBench ${CHAIR}_s$ from 44.3 to 27.0 but also AMBER object coverage from 74.1 to 67.4. The two outcomes should be reported together but do not establish a universal grounding–utility trade-off. mDPO’s Table 2 separately ablates conditional preferences and the anchor term.

VLGuard’s appendix Table 14 reports XSTest-unsafe attack success, judged by Llama-Guard, of 1.5% for Vicuna-v1.5-7B versus 11.0% for LLaVA-v1.5-7B [119]. This matched text-query comparison motivates post-adaptation safety checks but does not show that every multimodal update reduces safety.

6.4.2. Evidence Checks and Decision Limits

A convincing alignment claim rests on tests that change relevant evidence, with valid updated references and irrelevant-change controls, under fixed prompts, resolution, decoding, and judges. Component attribution also needs localization tests or ablations, since removing the image can shift the input distribution; preference and safety claims need human-validated judges and must report benign utility and false refusals. Section 7 shows how domain constraints set the evidence and behavior these recommended checks must cover.

7. Applications: Alignment in Practice

We examine five domains in which the methods of Section 2, Section 3, Section 4, Section 5 and Section 6 meet a characteristic evidence type (Modality Gap) and explicit behavioral requirements (Intent Gap): embodied AI and robotics, content moderation and safety, healthcare, recommendation, and climate decision support. Their studies are compared on grounding, adaptation, and behavioral constraints, with task success as one measure among several.

7.1. Embodied AI and Robotics (Perception–Action Alignment)

A robot must ground an instruction in the scene and act within its goals and constraints. For spatial grounding, VoxPoser combines language-derived affordances and constraints with RGB-D observations in 3D value maps, specifying interaction targets and avoidance regions for motion planning [251].

Instruction-to-action mapping places alignment inside the policy. RT-2 co-fine-tunes vision–language–action (VLA) models on web data and robot demonstrations and encodes controls as tokens [252], but semantic generalization does not expand its demonstrated motor skills. ACoT-VLA guides action prediction with coarse reference trajectories and latent action priors [253]. Feedback and safety constraints act during execution. ThinkAct rewards a visual planner with demonstrated 2D gripper trajectories and conditions actions on latent plans [254]; in its video-conditioned execution examples, the planner replans after failed grasps. SafeVLA uses safety feedback in constrained reinforcement learning to maximize task reward subject to expected safety-cost limits [255]. Changed instructions and unseen layouts probe grounding (Modality Gap); safety-cost violations during execution indicate whether goals are pursued safely (Intent Gap).

7.2. Content Moderation and Safety (Value Alignment)

Moderation must judge an image and its text together and refuse unsafe requests without rejecting benign ones. Its failures fall into three categories. Qi et al.’s adversarial image perturbations bypass safeguards [256], so textual refusal does not establish multimodal robustness. Semantic transfer keeps harmful meaning but moves it to another channel: FigStep renders harmful instructions inside images and can thus bypass text-trained safeguards [3]. In cross-modal composition, individually benign inputs can jointly elicit unsafe responses, as SIUO shows [257], and filtering each modality separately cannot assess their combined meaning.

Defenses differ in what they modify. TGA changes the interface: it guides visual projection with retrieved image-related text and improves safety transfer in its tests [258]. VLGuard changes supervision, using safe and unsafe instruction–response examples in mixed or post-hoc fine-tuning [119]. Each failure type needs its own controls, and harmful responses should be reported with false refusals and benign utility to separate safety transfer from blanket rejection.

7.3. Healthcare (Clinical Evidence Alignment)

Medical alignment links clinical evidence across images and language (Modality Gap), while the Intent Gap concerns whether diagnosis, reporting, VQA, or segmentation meets clinical requirements for accuracy, uncertainty, and safety. Representation learning has moved from global to fine-grained alignment. GLoRIA [83] aligns global image–report and local region–word pairs, and MGCA [259] matches image–report instances, tokens through cross-attention, and diseases through cross-modal prototype assignments. VTASC [260] uses image–sentence matching and description-accuracy losses to promote report consistency. These methods supply representation foundations, not complete clinical MLLM alignment pipelines, and CoMIR [261] targets cross-modal image registration rather than image–report alignment.

Later work turns to clinical tasks. For zero-shot transfer, CARZero [262] classifies diseases through image–text cross-attention without target-task fine-tuning, although pretraining may already have covered the relevant concepts. Malenia [263] aligns 3D lesion-mask representations with textual attributes for unseen-category segmentation but needs segmentation labels for seen categories in training. Task-aware objectives target clinical interaction: LLM-CXR [264] instruction-tunes chest X-ray report generation, VQA, and text-conditioned image generation, and AMiF [189] brings hard-negative mining to medical VQA (Section 4.4). Multimodal Lego [265] addresses modular combination instead, reusing unimodal encoders across topologies and modalities through a common wrapper and frequency-domain representations.

Pretraining, zero-shot transfer, and interface adaptation need distinct tests: local and rare-finding recognition and supported reports for evidence use, and institutional transfer with safety and privacy checks for clinical requirements.

7.4. Recommendation Systems (User-Intent Alignment)

Recommendation must match item content to the interests that interaction histories and reviews reveal. FETTLE uses estimated feedback to align visual, textual, and interaction representations by recommendation utility, without a fixed modality reference [266]. LIP [267] and MLLMRec [268] instead ground item semantics in evidence of user interests. LIP enriches interaction-graph nodes and review-based edges with frozen LLaVA features and uses clustered review embeddings as intent prompts, whereas MLLMRec generates image descriptions and preference profiles from user history before refining noisy item relationships.

HaNoRec aligns sequential recommendation behavior beyond content representations: it weights examples by difficulty and policy responsiveness and smooths outputs through Gaussian LoRA-parameter perturbations [269]. FETTLE changes representations, LIP and MLLMRec construct preference evidence, and HaNoRec changes response optimization, so their ranking gains need separate tests of content dependence and user-constraint compliance.

Such gains should also survive end-to-end reproduction, not just reuse of supplied artifacts. Fioretti et al. [270] approximately reproduced the Netflix results of LLMRec, a system distinct from MLLMRec, from its released augmentations, but rebuilding the augmentation and comparing stronger baselines changed the performance conclusions; the result does not show that LLM augmentation fails in general.

7.5. Climate Resilience and Decision Support (Physical Evidence Alignment)

Climate decisions need observations that agree across sensors, locations, and times and must respect operational constraints. For multi-sensor grounding, EarthDial [271] uses instruction tuning to connect optical, radar, and multispectral observations with language across resolutions and time points, and its grounding and change-detection tasks assess spatial and temporal correspondence beyond generic scene descriptions.

Decision-oriented work selects or predicts the evidence a decision needs. For disaster assessment, CrisiSense-RAG retrieves text and images separately, prioritizing contemporaneous reports for peak inundation and later imagery for persistent structural damage [272]; its Hurricane Harvey study illustrates how the assessment target determines the relevant observation time. For heat adaptation, DeepShade [273], a diffusion model (not an MLLM) trained on simulated building shadows, generates urban shade maps conditioned on satellite imagery and textual time/solar-angle information, and its shade-aware routing shows how predicted physical conditions can feed decision support.

All three studies depend on location- and time-consistent evidence. Transfer across regions and events should therefore be evaluated separately from decision quality, with human review when evidence is incomplete or decisions are high-risk.

Across the five domains, three requirements recur: traceable evidence, valid supervision, and constraint-sensitive decisions. Ranking utility, report consistency, and safety-cost violations do not reduce to one alignment score; what transfers is a procedure that identifies evidence, specifies requirements, and tests both under controlled changes. Where such tests are missing, Section 8 proposes four research directions.

8. Open Challenges and Future Directions

Section 2, Section 3, Section 4, Section 5, Section 6 and Section 7 rest on one criterion: a gain counts as alignment only when a controlled test shows which gap it narrows and how. Four open problems still lack such tests; each motivates a research direction linking a diagnosed failure to a testable intervention.

8.1. Evidence-Consistent Alignment Across Modalities

Architecture and objective comparisons show that a compatible interface can hide local or temporal evidence loss. For token-based integration such as AnyGPT [22], it is unclear which correspondences remain necessary to the answer after compression and tuning. Chart2Code’s separate checks of executable output and rendered fidelity [200] illustrate why valid output structure and faithful source use need distinct evidence. Research should vary token budgets and evidence granularity under matched updates, then probe the Modality Gap through identity, relation, and event-order changes with irrelevant-change controls.

8.2. Verified Self-Correction and Test-Time Alignment

Preference and inference-time methods (Section 5) differ in whether feedback brings new evidence or recycles the model’s own judgment. Counterfactual VLA revises driving plans before trajectory generation [274], but showing that its mechanism transfers requires more than plausible revised plans. The open question is when to seek new observations or tool checks rather than revise internally; selective correction would do so only when their expected value exceeds their cost. Evaluation should compare repaired errors, newly introduced errors, and Intent Gap constraint violations against no-correction and compute-matched sampling baselines, with stopping rules tested under unreliable feedback.

8.3. From Failure Diagnosis to Targeted Repair

Section 6 separates detecting a failure from locating its cause. Vision-Language Introspection links hallucination-risk detection to localization and steering [275], and SIUO exposes unsafe combinations of individually benign inputs [257], yet which component to change for each failure remains unclear. Controlled replacement of perceptual features, retrieved evidence, or decision constraints can separate unavailable evidence, ignored evidence (both Modality Gap failures), reasoning errors, and Intent Gap constraint violations. Repair policies should be evaluated on this diagnosis and on retained capabilities, beyond the original failures.

8.4. Reliable Transfer to Underrepresented Settings

In the application domains, correspondence, supervision validity, and acceptable risk vary with the deployment setting. MMedPO adds clinical relevance to preferences [199], while BLEnD-Vis probes cultural knowledge across regions [205]. Accuracy may transfer even when the alignment criterion does not. Research should compare reusable grounding components (Modality Gap) with locally validated preference or constraint models (Intent Gap), holding out institutions, populations, sensors, or environments. Subgroup error, calibrated uncertainty, and benign utility would show whether adaptation meets the new constraint or sacrifices previously reliable behavior.

9. Conclusions

Fluent responses can mask two distinct failures; comparing the six dimensions yields three conclusions. First, an alignment mechanism is only as informative as its supervision: finer representations, preferred answers, and verifiable outcomes constrain different properties. Second, alignment must be evaluated separately at the representation, task, and behavior levels, since a correct answer or lower refusal rate can hide a different failure. Third, transfer across applications depends on traceable evidence, valid supervision, and constraint-sensitive decisions, motivating four research directions. The Modality and Intent Gaps make these claims diagnosable without treating every reasoning error as an alignment failure.