A Survey on Post-Training of Multimodal Large Language Models

1. Introduction

In recent years, Multimodal Large Language Models (MLLMs) have emerged as a promising paradigm for advancing general-purpose artificial intelligence [1,2]. By integrating vision, language, and other modalities within a unified framework, these models enable a broad spectrum of applications, driving practical advances in vision-language assistants [3,4], autonomous driving [5,6], and embodied intelligence [7,8].

With Large Language Models (LLMs) [9,10] serving as their core reasoning and generation engine, MLLMs extend language-centric capabilities to multimodal inputs through multimodal pretraining [1,2,9]. This constitutes the prevailing paradigm for building modern MLLMs, in which a powerful LLM backbone is coupled with modality encoders and cross-modal connectors, and trained on large-scale corpora spanning images, videos, text, audio, and other modalities. By optimizing objectives such as next-token prediction, multimodal pretraining establishes basic cross-modal representations and alignment, supporting progress in various downstream tasks such as visual question answering[11,12], captioning[13,14], document understanding[15,16], and visual reasoning[17,18].

Despite these advances, most pretrained MLLMs primarily learn statistical patterns from large-scale data. Such pretraining alone does not ensure that model behaviors are aligned with human needs, ethical principles, or real-world requirements. To bridge this gap, recent research has increasingly shifted toward multimodal post-training (MMPoT), a stage after pretraining that transforms the learned capabilities of MLLMs into reliable and controllable behaviors. This emerging direction has attracted rapidly growing research interest, spanning diverse application scenarios [19,20,21] and optimization goals [22,23]. Prevalent multimodal post-training approaches typically follow a standard paradigm, i.e., multimodal supervised fine-tuning (SFT). It tunes pretrained MLLMs on instruction-formatted data (i.e., multimodal inputs paired with user instructions and target responses), learning to respond to multimodal instructions and interact with users in a task-oriented manner. This simple yet influential scheme has driven the success of early MLLMs post-training methods [1,2,24,25]. However, SFT mainly relies on imitation learning, its training objective cannot fully judge whether a response is preferred, faithful to evidence, safe, or robust under complex reasoning. Reinforcement Learning in MLLMs has made great progress in closing this gap between model’s output and human intention. It learns from feedback lists or preference pairs, encouraging the model to optimize model behavior that better matches human or AI judgments. This evolution spans from early reinforcement-based post-training approaches such as proximal policy optimization [26,27] to more scalable strategies like direct preference optimization [28,29,30]. Inspired by the success of reasoning-oriented models such as o1 [31] and DeepSeek-R1 [32] in the language domain, increasing research attention has been devoted to eliciting and enhancing the reasoning capabilities of MLLMs. Compared to early reinforcement learning methods, this scheme obviates the need for human-labeled trajectories via carefully designed reward mechanisms (e.g.,ORM and PRM) and RL strategies (e.g.,GRPO [32]), achieving impressive performance in complex problems.

Prior surveys have examined specific aspects of MLLMs post-training [33,34,35], yet a unified perspective remains lacking. An application-oriented survey [33] focuses on human alignment in specific domains, whereas technique-oriented surveys [34,35] concentrate on individual methodological paradigms. Our analysis reveals that existing MLLMs post-training methods differ substantially in their feedback sources, optimization objectives, updated model components, and evaluation protocols. Together, these variations expose a fragmented methodological landscape: methods may pursue similar behavioral goals using distinct feedback signals or apply similar training recipes to optimize different capabilities. These observations motivate the following fundamental question:

What common process underlies MLLMs post-training, and how does it steer pretrained MLLMs toward desired multimodal behaviors?

As the scope of MLLMs post-training continues to broaden, this survey addresses this question by advocating a behavior-shaping perspective: instead of treating multimodal post-training as isolated recipes, we examine them through the full behavior-shaping loop as shown in Figure 1, i.e., 1) Refine & Update: what algorithms are applied and which capabilities it aims to improve. 2) Source & Feedback: where supervision comes from and how the resulting behavior is evaluated. By synthesizing these perspectives, we seek to delineate the evolving methodological landscape of MLLMs post-training and shed light on the emerging frontiers that may shape future research.

The main contributions are summarized as follows:

• We conceptualize MLLMs post-training as a process of multimodal behavior shaping, offering a unified perspective on how pretrained MLLMs acquire reliable and versatile behavioral capabilities.
• We present a taxonomy that organizes MLLMs post-training methods into five major families: multimodal instruction following, preference calibration, reasoning enhancement, domain adaptation, and scalable training.
• We systematically examine the datasets and evaluations used in MLLMs post-training, revealing how benchmarks and metrics define and measure desirable multimodal behavior.
• We identify key challenges and outline promising research directions, offering a roadmap for advancing MLLMs post-training toward dependable multimodal intelligence.

Scope. This survey focuses on post-training for Multimodal Large Language Models (MLLMs), examining how pretrained models are adapted for human interaction and task-oriented use. Our scope centers on multimodal understanding over images, videos, audio, and text, while specialized sensor modalities such as infrared and LiDAR are beyond the main focus. Some methods span both pre-training and post-training, we include them when their post-training components are relevant to our analysis. Downstream domains, such as autonomous driving and healthcare, are discussed only to illustrate specific design choices; the primary focus remains on underlying post-training mechanisms.

Roadmap. The remainder of this survey is organized as follows. Section 2 introduces the basic definition of MLLMs post-training within our behavior-shaping framework. Moreover, Section 3, Section 4, Section 5, Section 6 and Section 7 review the five major families of post-training methods. Further, Section 8 surveys the mainstream benchmarks and evaluation protocols used to assess post-trained MLLMs. Finally, Section 9 discusses the key challenges facing MLLMs post-training and outlines future research directions.

2. Overview of Post-Training for MLLMs

This section provides a brief introduction of MLLMs post-training, covering its basic definition (Section 2.1), motivation (Section 2.2), and position within key AI fields (Section 2.3).

2.1. What Is MLLMs Post-Training?

MLLMs post-training refers to targeted adaptation and alignment procedures applied to pretrained MLLMs. Unlike pre-training, which builds broad multimodal representations, post-training uses task- and instruction-level supervision to align behavior with human intent, strengthen cross-modal reasoning, and improve downstream reliability.

Formally, let $x_m$ denote multimodal inputs (e.g.,images, videos, audio, documents, screenshots, or environment observations), $x_t$ denote textual instructions and dialogue history, and y denote outputs such as responses, tool calls, grounding predictions, or actions. Post-training optimizes:

$${\pi }_{\theta }(y\mid x_m,x_t).$$

(1)

Given supervision or feedback f (e.g.,demonstrations, preferences, rewards, teacher outputs, or grounding annotations), the general objective is:

$$\underset{\theta }{max}{\mathbb{E}}_{(x_m,x_t,f)\sim \mathcal{D}}\left[U({\pi }_{\theta };x_m,x_t,f)\right]\mathrm{s}.\mathrm{t}.C\left({\pi }_{\theta }\right)\le \tau ,$$

(2)

where U captures helpfulness, faithfulness, reasoning success, safety, or task reward, while C encodes constraints such as latency, memory, annotation cost, privacy risk, and safety violations. This abstraction shows that post-training methods differ mainly in their supervision, feedback, and constraint design.

2.2. Why MLLMs Post-Training: Progressive Behavior Shaping

From a behavior-shaping perspective, the evolution of MLLMs post-training reflects a gradual shift from capability activation to policy refinement, as illustrated in Figure 2. Early supervised fine-tuning [1,2] teaches pretrained multimodal models to follow vision-language instructions, activating latent perceptual knowledge for task-oriented behavior. Subsequent methods based on feedback alignment [36,37] and preference learning [38] further refine model policies toward human-preferred responses, mitigating hallucinations and improving reliability. More recent advances in complex multimodal reasoning [39] and online policy distillation [40,41] extend this process from response-level correction to higher-level reasoning and decision making, enabling more consistent reasoning, interaction, and adaptation. Overall, post-training has emerged as a central mechanism for transforming pretrained MLLMs with broad multimodal representations into systems that exhibit aligned, reliable, and controllable behaviors.

2.3. Post-Training MLLMs as the Next Frontier

The rapid progress of foundation models has made multimodal intelligence a key pathway toward AGI. However, most MLLMs remain primarily digital systems: they can perceive and reason over multimodal inputs, but still lack reliable interaction, tool use, and action in open environments. Post-training is therefore a critical bridge. By leveraging instruction data, feedback signals, preference objectives, reasoning traces, and agentic experience, it equips MLLMs with aligned, goal-directed, and decision-making capabilities. In this sense, post-training is not merely a downstream optimization stage, but a bridge linking multimodal learning, digital AI, and physical AI, as illustrated in Figure 3. It moves MLLMs beyond passive understanding toward robust interaction, grounded reasoning, and safety-aware action. A chronological overview of MLLMs post-training research is provided in Figure 4.

3. MMPoT of Instruction Following

Multimodal instruction tuning is an early and foundational post-training technique for enabling instruction-following ability in MLLMs. It adapts pretrained multimodal models using vision-language instruction-response pairs, thereby transforming raw multimodal representations into task-oriented distributions for instruction-following behavior. As the first post-training step, it establishes the interface between user intent, multimodal input, and model response. This simple yet effective paradigm underpins early MLLMs such as InstructBLIP [2], LLaVA [1], and their variants. Representative methods are summarized in Table 1.

3.1. Preliminary

Given a pretrained MLLM with parameters ${\theta }_0$, multimodal instruction tuning adapts the model using instruction-formatted visual-language examples. Each training instance can be represented as:

$$(x_m,x_u,y)\in {\mathcal{D}}_{\mathrm{inst}},$$

(3)

where $x_m$ denotes the multimodal input (e.g.,image, video, and speech), $x_u$ denotes the user instruction, and $y=(y_1,\dots ,y_T)$ is the target response. The model generates the response conditioned on both the multimodal input and the user instruction:

$$p_{\theta }(y\mid x_m,x_u)={\displaystyle \prod _{t=1}^T}{p}_{\theta }(y_t\mid y_{<t},x_m,x_u).$$

(4)

The instruction-tuned model is obtained by minimizing the supervised next-token prediction loss:

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{SFT}}={\mathbb{E}}_{(x_m,x_u,y)\sim {\mathcal{D}}_{\mathrm{inst}}}\left[-{\displaystyle \sum _{t=1}^T}log{p}_{\theta }(y_t\mid y_{<t},x_m,x_u)\right].\end{array}$$

(5)

Depending on the training recipe, $\theta$ may include only the connector, selected adapter modules, the LLM backbone, the modality encoder, or all trainable components. A basic diagram of the instruction tuning pipline is illustrated in Figure 5.

3.2. Visual Instruction Tuning

Visual instruction tuning adapts pretrained multimodal models to follow natural-language instructions grounded in images or videos. By learning from vision-language instruction-response pairs, it aligns user intent, visual context, and model responses within a unified generative interface. This process activates latent multimodal knowledge as task-oriented instruction-following behavior, enabling MLLMs to address diverse visual tasks with reduced reliance on task-specific heads.

•Visual Adaptation to LLMs. Early visual instruction tuning primarily focused on connecting pretrained visual encoders with LLMs to enable image-grounded instruction following. LLaVA [1] introduced a widely adopted two-stage recipe that first aligns CLIP [69] visual features with the embedding space of Vicuna [70], and then fine-tunes the model on visual instruction data for open-ended multimodal dialogue. Subsequent works explored lighter or more modular variants: MiniGPT-4 [24] uses a lightweight projection module to connect vision features to a mostly frozen LLM, mPLUG-Owl [42] adopts a modular image-conditioned design, MultiModal-GPT [71] studies parameter-efficient tuning, and InstructBLIP [2] uses an instruction-aware Q-Former [72]. Together, these works established the standard paradigm of bridging vision and language backbones, aligning their representations, and adapting the resulting model with instruction data.

•Instruction-Aware Visual Injection. Building on these foundations, subsequent studies improve visual instruction tuning through instruction-aware feature selection, stronger training recipes, and efficient adaptation. LLaVA-1.5 [25] improves the reproducibility and effectiveness of this recipe through a refined data mixture, higher image resolution, and optimized training configurations. LaVIN [57] introduces lightweight adaptation modules for efficient vision-language alignment, while LLaMA-Adapter V2 [43] demonstrates that adapter-based tuning can activate multimodal instruction-following capabilities at limited training cost. Together, these works show that effective visual instruction tuning depends not only on cross-modal alignment, but also on how visual information is selected, integrated, and optimized during adaptation.

•Fine-grained Instruction Tuning. Another direction extends visual instruction tuning from holistic image understanding to fine-grained grounding and region-aware interaction. GPT4RoI [73] introduces region-level visual information into instruction tuning. Shikra [74] and Ferret [75,76] combine referring expressions, grounding, and dialogue to support user-specified object or region understanding, while Kosmos-2 [77] links language generation with visual regions.

•Instruction Tuning Beyond Images. Visual instruction tuning has also been extended beyond single-image understanding to more complex inputs and scenarios. For temporal and multi-image reasoning, Video-LLaMA [78], VideoChat, Video-ChatGPT [79], and Valley [80] adapt instruction tuning to videos or multiple images, enabling event understanding, cross-image comparison, and visual information synthesis. LLaVA-OneVision[47], Qwen2.5-VL [81], and InternVL3 [82] represent this shift from image-chat post-training to full-stack multimodal post-training. Their recipes commonly combine high-quality instruction data, higher-resolution visual tokens, task-specific mixtures, and test-time strategies.

3.3. Instruction Data Mixtures

Instruction data mixtures refer to post-training datasets that combine instruction-response examples from multiple tasks, domains, modalities, and supervision sources.

•Text-Multimodal Balance. Text-only user-assistant conversations are often mixed with multimodal instruction data to preserve general language and conversational capabilities while extending instruction following to visual and other non-textual inputs. LaVIN [57] constructs minibatches by sampling both language-only and multimodal examples, whereas MultiInstruct [83] explores different strategies for combining single-modal and multimodal instructions, including mixed instruction tuning.

•Multimodal Data Composition. Beyond balancing text-only and multimodal examples, the internal composition of multimodal data is equally important. LLaVA-1.5 [25] shows that compact, carefully curated mixtures of captioning, VQA, and instruction data can yield strong open-source baselines. ShareGPT4V [68] highlights the value of high-quality captions, while Cambrian-1 [52] examines how data composition interacts with model architecture, image resolution, and evaluation performance. Overall, effective instruction mixtures depend not only on scale but also on careful source balancing, filtering, deduplication, and coverage analysis to preserve general language abilities while improving multimodal behavior.

4. MMPoT of Preference Calibration

Following instruction tuning, multimodal preference calibration further refines MLLMs using human- or model-generated feedback to satisfy desired behavioral criteria. It uses preference pairs, comparative judgments, or reward signals to distinguish desirable responses from weaker alternatives, thereby shaping preference-calibrated behavior that favors outputs with fewer hallucinations, stronger visual grounding, and closer alignment with user intent. Table 2 summarizes representative preference calibration methods.

4.1. Multimodal Reinforcement Learning with Human Feedback

Reinforcement Learning with Human Feedback (RLHF) aligns MLLMs with human preferences using feedback on responses grounded in multimodal inputs. It is typically applied after SFT. Human comparisons or ratings are first used to train a reward model. The learned reward then guides policy optimization, encouraging responses that are helpful, safe, and faithful to multimodal evidence. Figure 6 (top) illustrates the simplified pipeline of the RLHF.

4.1.1. Reward Mechanisms

In RLHF-based alignment, reward modeling is needed to turn preference supervision into an optimization signal for policy improvement. For multimodal tasks, this signal can be defined at different levels. The commonly used reward modeling strategies mainly fall into two categories: Outcome Reward Mechanisms (ORM) and Process Reward Mechanisms (PRM).

•Outcome Reward Mechanisms. Outcome reward mechanisms assign a scalar reward to the final response of a model. This reward typically reflects whether the answer is correct, helpful, and consistent with the visual input. ORM is simple and broadly applicable, since it only requires judging the final output. LLaVA-RLHF [36] extends this formulation by augmenting the reward model with factual signals, such as captions and ground-truth options, so that the reward is less dominated by fluent but visually unsupported answers. RLHF-V [37] further moves from coarse response ranking to fine-grained correctional feedback, where hallucinated segments are explicitly identified and corrected. However, it provides limited supervision for how the model reaches the answer, making it less effective when intermediate reasoning errors are hidden by a plausible final response.

•Process Reward Mechanisms. To address the coarse supervision of ORM, process reward mechanisms evaluate the intermediate steps that lead to the final response. Instead of only scoring the endpoint, PRM assigns feedback to reasoning traces, subgoals, or step-by-step decisions, encouraging the model to follow a grounded and reliable reasoning path. This is useful for complex multimodal tasks, where correct answers often depend on visual grounding, multi-step inference, and faithful use of evidence. VisualPRM [84], supervise intermediate reasoning steps rather than only the final answer, enabling finer credit assignment. This trajectory shifts reward modeling from holistic preference prediction toward visually grounded and step-aware evaluation.

4.1.2. Policy Learning of RLHF

In RLHF, an MLLM first generates candidate responses for each multimodal instruction. Human annotators evaluate these responses based on helpfulness, correctness, safety, and visual faithfulness, producing pairwise preferences or scalar ratings. These annotations train a reward model $r_{\varphi }$ to approximate human judgments. The reward may be assigned to the final response by an ORM or to intermediate reasoning steps by a PRM. The policy ${\pi }_{\theta }$ is optimized against the learned reward, typically under a KL-divergence constraint that limits deviation from the reference policy:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{RLHF}}=& -{\mathbb{E}}_{y\sim {\pi }_{\theta }(·\mid x_t,x_v)}\left[r_{\varphi }(x_t,x_v,y)\right]\hfill \\ \hfill & +\beta D_{\mathrm{KL}}\left({\pi }_{\theta }(·\mid x_t,x_v)\parallel {\pi }_{\mathrm{ref}}(·\mid x_t,x_v)\right).\hfill \end{array}$$

(6)

where ${\pi }_{\theta }$ is the policy being optimized, ${\pi }_{\mathrm{ref}}$ is the fixed reference policy, usually the SFT model, and $(x_t,x_v)$ denotes the textual instruction and visual input. The response y is sampled from ${\pi }_{\theta }$, and $r_{\varphi }(x_t,x_v,y)$ is the reward assigned by the reward model. The KL term keeps ${\pi }_{\theta }$ close to ${\pi }_{\mathrm{ref}}$, with $\beta$ controlling its strength.

This paradigm has inspired broad explorations. As a well-known pioneering method, LLaVA-RLHF [36] adapts KL-regularized RLHF to MLLMs and uses factual rewards to mitigate reward hacking and hallucination. RLHF-V [37] instead applies dense preference optimization over corrected response spans, enabling more targeted and data-efficient updates. At the frontier scale, MM-RLHF [100] further introduces 120K fine-grained human preference pairs, critique-based reward modeling, and dynamic reward scaling. Recent methods increasingly combine reward sources: Seed1.5-VL [86] pairs RLHF with verifier-based rewards, while MiMo-VL [101] applies mixed on-policy RL with diverse rewards to improve visual understanding and multimodal reasoning.

4.2. Multimodal Reinforcement Learning with AI Feedback

Reinforcement Learning from AI Feedback (RLAIF) extends RLHF by supplementing or replacing human judgments with AI-generated feedback. AI evaluators score or rank candidate responses, and the resulting signals guide policy optimization. Unlike purely supervised post-training, RLAIF creates an iterative evaluation-and-refinement loop, offering a scalable approach to multimodal behavior shaping. RLAIF follows the same pipeline as RLHF, as shown in Figure 6 (top).

4.2.1. RLHF vs. RLAIF

A central limitation of RLHF is the high cost and limited scalability of collecting reliable human preferences. The burden is greater in multimodal settings, where annotators must inspect visual evidence, identify hallucinations, assess safety risks, and evaluate complex reasoning. This process is time-consuming and can produce inconsistent judgments. RLAIF reduces this dependence by employing capable AI models as evaluators or critics to generate preference labels, scalar scores, or textual critiques for MLLM responses.

4.2.2. RLAIF Training Pipeline

Like RLHF, RLAIF commonly uses preference-based optimization but obtains feedback from AI evaluators rather than relying solely on human annotators. VLM-RLAIF [88] uses an AI evaluator to score or compare candidate responses and then optimizes the model toward more helpful and visually faithful outputs. RLAIF-V [89] further targets visual alignment by assessing whether responses are supported by visual evidence, particularly in cases of hallucination or weak grounding. However, reward-model-based alignment can suffer from unstable optimization and reward hacking, while also requiring a separate reward model. To address these limitations, recent methods increasingly adopt direct preference objectives, such as DPO, which optimize pairwise preferences without explicit reward modeling or PPO. The next section reviews this line of work.

4.3. Multimodal Direct Preference Optimization

In this section, we introduce the definition of Multimodal Direct Preference Optimization (DPO) and examine its major variants in MLLMs post-training. The bottom panel of Figure 6 presents the DPO optimization scheme.

4.3.1. Preliminary

Although RLHF and RLAIF are effective, applying them to multimodal models remains challenging. Training a reliable reward model over high-dimensional vision-language inputs is costly and prone to bias. Direct Preference Optimization (DPO) [28], originally introduced for preference alignment in NLP, offers a simpler alternative. Rather than training an explicit reward model and then optimizing the policy through reinforcement learning, DPO learns directly from paired preference examples. In MLLMs post-training, let $x=(x_m,x_t)$ denote a multimodal instruction and $(y_w,y_l)$ denote two candidate responses, where $y_w$ is preferred over $y_l$. DPO then optimizes:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{DPO}}=-{\mathbb{E}}_{(x,y_w,y_l)}[log\sigma (\beta (& log{\displaystyle \frac{{\pi }_{\theta }\left(y_w\right|x)}{{\pi }_{\mathrm{ref}}\left(y_w\right|x)}}\hfill \\ & -log{\displaystyle \frac{{\pi }_{\theta }\left(y_l\right|x)}{{\pi }_{\mathrm{ref}}\left(y_l\right|x)}}\left)\right)],\hfill \end{array}$$

where ${\pi }_{\theta }$ is the policy model, ${\pi }_{\mathrm{ref}}$ is a reference model, and $\beta$ controls the strength of the preference optimization. This formulation preserves the core objective of preference-based alignment while avoiding explicit reward modeling and reinforcement learning updates, making it more stable, efficient, and easier to scale to multimodal settings.

4.3.2. The Evolution of Multimodal DPO

•Response-level Multimodal DPO. The simplest multimodal extension of DPO conditions preference optimization on images or videos while treating the full response as the target. Examples include Silkie [129], which builds VLFeedback with GPT-4V evaluations, video-language DPO methods that use detailed captions as textual proxies for video content [91], and ISR-DPO [130], which iteratively generates and evaluates responses to ground preferences in informative video regions.

•Hallucination-aware DPO. Since hallucination is a central issue in MLLMs, visual-grounding DPO methods construct preference data to favor grounded responses over hallucinated ones. HA-DPO [92] pairs non-hallucinatory responses with hallucinated alternatives, while V-DPO [93], CLIP-DPO [38], and CHAIR-DPO [94] introduce visual contrast, image-response consistency, or object-aware signals to strengthen visual grounding and reduce hallucination.

•Modality-conditioned DPO. A key limitation of multimodal DPO is unconditional preference learning, where models may rely on language-only cues while ignoring visual or auditory evidence. Modality-conditioned methods address this by grounding preferences in multimodal inputs. mDPO [95] adds image-conditioned optimization with a reward anchor, MoD-DPO [96] separates modality-specific preference signals with modality-aware regularization, and OmniDPO [97] uses both textual and multimodal preference pairs to improve audio-video interaction understanding and sensitivity to multimodal evidence.

•Hard-negative DPO. Response-level preference labels are often too coarse for multimodal alignment, where errors may occur at specific tokens, objects, regions, or subtle visual distinctions. Hard-negative and fine-grained DPO methods provide more targeted supervision by emphasizing difficult pairs and visually sensitive outputs. DA-DPO [98] reweights examples by pair difficulty, UE-DPO [131] uses token-level uncertainty to correct visually unsupported tokens, and OPA-DPO [132] constructs more preference data to reduce distribution mismatch with the current model.

•Listwise Multimodal DPO. Standard DPO learns from binary chosen–rejected pairs, while listwise multimodal preference optimization ranks multiple candidates to provide richer supervision. LPOI [99] builds ranked visual preference samples by masking critical objects and interpolating affected regions, forming a progression from limited visibility to complete object evidence.

5. MMPoT of Reason Enhancement

Multimodal preference calibration improves human-preferred responses but remains insufficient for reasoning-intensive tasks such as multi-step inference, mathematical reasoning, and long-horizon decision making. Inspired by o1-style reasoning models [31] and DeepSeek-Math/R1 [29,32], recent work extends reward-based reinforcement learning to MLLMs. This section reviews how reasoning-oriented supervision shapes grounded, structured, and multi-step inference across modalities, with representative methods summarized in Table 3.

5.1. R1-Based Multimodal Reasoning

R1-based multimodal reasoning extends R1-style post-training to multimodal inputs, using verifiable rewards to elicit structured reasoning grounded in multimodal evidence. Its general paradigm is illustrated in Figure 7 (left).

5.1.1. From LLM-R1 to MLLM-R1

R1-style post-training [29,31,32] marks a shift in LLM reasoning optimization: from imitating annotated chain-of-thought (CoT) traces to learning reasoning behaviors through verifiable feedback. MLLM-R1 extends this principle by conditioning the reasoning process on multimodal evidence rather than text alone. Let $x=(x_m,x_t)$ denote a multimodal instruction, where $x_m$ represents the multimodal input and $x_t$ represents the textual instruction. An R1-style MLLM is then trained to generate a structured reasoning response:

$$y=<\mathtt{think}>r</\mathtt{think}><\mathtt{answer}>a</\mathtt{answer}>,$$

(7)

where r is the intermediate reasoning trajectory and a is the final answer. The model defines:

$${\pi }_{\theta }(y\mid x_m,x_t)={\pi }_{\theta }(r,a\mid x_m,x_t).$$

(8)

A common training pipeline first performs cold-start supervised tuning on formatted reasoning data and then applies reinforcement learning with verifiable rewards:

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{R}1}=-{\mathbb{E}}_{y\sim {\pi }_{\theta }(·\mid x_m,x_t)}\left[\mathcal{A}(x,y)\right]+\lambda \mathcal{R}({\pi }_{\theta },{\pi }_{\mathrm{ref}}),\end{array}$$

(9)

where $\mathcal{A}(x,y)$ denotes a reward-derived learning signal, such as outcome reward, group-relative advantage, or verifier score, and $\mathcal{R}({\pi }_{\theta },{\pi }_{\mathrm{ref}})$ is an optional regularization term that constrains the updated policy. Below, we outline how to adapt R1-style optimization to multimodal reasoning.

5.1.2. R1 Training Paradigm for MLLMs

R1-style training for MLLMs broadly follows two paradigms: R1-Zero and R1. R1-Zero directly applies rule-based reinforcement learning to elicit reasoning behavior. R1 first establishes a cold-start reasoning policy and then refines it through Reinforcement Learning with Verifiable Rewards (RLVR) [133], often using Group Relative Policy Optimization (GRPO) [29]. The key distinction is whether multimodal reasoning is elicited solely by reward signals or bootstrapped from curated reasoning data.

•R1-Zero-style Training. R1-Zero-style methods follow DeepSeek-R1-Zero [32] by omitting supervised CoT warm-up and directly optimizing a base or instruction-tuned MLLM with verifiable rewards:

$${\pi }_{{\theta }_0}{\displaystyle \underset{{\nabla }_{\theta }{\mathbb{E}}_{y\sim {\pi }_{\theta }(·|x)}\left[r(x,y)\right]}{\overset{\mathrm{RLVR}/\mathrm{GRPO}}{\to }}}{\pi }_{{\theta }_{\mathrm{zero}}}.$$

(10)

The reward function is typically rule-based and outcome-verifiable, without relying on human-written CoT traces. For a multimodal problem $x=(x_t,x_v)$ and a generated response y, the reward can be decomposed into answer correctness and format validity:

$$r(x,y)={\lambda }_{\mathrm{ans}}{r}_{\mathrm{ans}}(x,y)+{\lambda }_{\mathrm{fmt}}{r}_{\mathrm{fmt}}\left(y\right),$$

(11)

where $r_{\mathrm{fmt}}\left(y\right)$ checks whether the response follows the required reasoning or answer format, and $r_{\mathrm{ans}}(x,y)$ verifies the final answer:

$$r_{\mathrm{ans}}(x,y)=\left\{\begin{array}{cc}1,\hfill & \mathrm{Ans}\left(y\right)=a^{★}\left(x\right),\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(12)

Here, $a^{★}\left(x\right)$ denotes the ground-truth answer for input x, and $\mathrm{Ans}\left(y\right)$ extracts the final answer from the model response. Since $x=(x_m,x_t)$ combines a multimodal input with a textual instruction, the answer reward encourages responses that are consistent with multimodal evidence rather than language priors alone.

This reward schema underpins early R1-Zero-style explorations for MLLMs. VisualThinker-R1-Zero [105] demonstrates that a small vision-language model without SFT warm-up can develop longer reasoning traces and self-reflection through rule-based RL. MM-Eureka [106] extends this idea to broader multimodal reasoning and reports visual “aha moments,” characterized by longer responses, higher accuracy rewards, and emergent reflection. ThinkLite-VL [134] performs pure reinforcement fine-tuning (RFT) without distillation and uses MCTS-guided difficulty filtering to select challenging but solvable visual reasoning samples.

•R1-style Training. Though effective, R1-Zero-style training can be unstable when the initial policy lacks reliable reasoning behavior. To reduce noisy exploration and sparse rewards, later methods add a cold-start SFT stage before RL refinement, i.e., a more stable two-stage recipe: cold-start reasoning initialization followed by reinforcement learning:

$${\pi }_{{\theta }_0}\stackrel{\mathrm{SFT}\left({\mathcal{D}}_{\mathrm{reason}}\right)}{\to }{\pi }_{{\theta }_{\mathrm{sft}}}{\displaystyle \underset{{\nabla }_{\theta }{\mathbb{E}}_{y\sim {\pi }_{\theta }(·|x)}\left[r(x,y)\right]}{\overset{\mathrm{RLVR}/\mathrm{GRPO}}{\to }}}{\pi }_{{\theta }_{\mathrm{r}1}}.$$

(13)

Along this direction, Vision-R1 [39] and Skywork-R1V [135] combine cold-start multimodal CoT supervision with GRPO to bootstrap visual reasoning. Vision-R1 uses format and outcome rewards, while Skywork-R1V further incorporates iterative SFT and adaptive-length CoT distillation. R1-Onevision [107] and Retrv-R1 [109] adapt R1-style optimization through task-specific reasoning structures. The former formalizes visual evidence into reasoning-friendly representations, whereas the latter combines retrieval-specific CoT activation with curriculum-guided RL. R1-Omni [108] and VLM-R1 [102] emphasize task-specific reward design. R1-Omni jointly rewards emotion prediction and explanation quality, while VLM-R1 targets stable optimization for visual grounding and reasoning.

5.2. Thinking with Images

Textual Chain-of-Thought (CoT) improves multimodal reasoning through stepwise language reasoning, but often treats images as static context. Thinking with images addresses this limitation by making visual information an active reasoning medium, allowing MLLMs to inspect, manipulate, and integrate images as intermediate evidence throughout reasoning, as shown in Figure 7 (right).

•Visual Evidence Grounding. A direct way to think with images is to make each reasoning step traceable to visual evidence. For example, GRIT [110] links reasoning steps to bounding-box regions through a “look–think–look” process. Point-RFT [111] uses point-level references for document and chart reasoning, showing that grounded rationales outperform text-only CoT under reinforcement fine-tuning. VisionReasoner [113] unifies detection, segmentation, counting, and VQA within a reasoning-integrated perception framework.

•Visual Tool Use and Manipulation. Beyond explicit references, another direction gives models actions over visual inputs. OpenThinkIMG [112] provides standardized interfaces for detection, segmentation, OCR, cropping, and drawing, and trains adaptive tool invocation through V-ToolRL. DeepEyes [114] and its follow-up variants [136] allow VLMs to zoom into and ground relevant visual regions, learning effective inspection strategies through end-to-end RL. VTool-R1 [115] interleaves textual reasoning with visual tool operations and uses outcome rewards to elicit strategic tool use for structured chart and table reasoning.

•Latent Visual Reasoning. A more implicit direction keeps visual thinking inside hidden states instead of exposing every step as boxes, points, or tool calls. Latent Visual Reasoning [137,138] generates latent states that reconstruct query-relevant visual tokens, enabling reasoning in the visual embedding space. LanteRn [116] combines language with compact latent visual representations, using SFT for grounding and RL for task-level utility. Although primarily designed for visual generation, GoT-R1 [139] further suggests that RL can induce semantic-spatial planning beyond fixed templates.

5.3. Self-Evolution for Multimodal Reasoning

Multimodal self-evolving methods improve reasoning through iterative self-improvement. The model generates candidate solutions, evaluates or revises its own reasoning traces, and uses the refined outputs as new training signals. This helps MLLMs strengthen visual-textual reasoning, correct unsupported steps, and improve problem-solving behavior with less reliance on human annotations. The overall workflow of self-evolving is shown in Figure 8 (left).

5.3.1. Self-Generated Data Learning

Self-generated data learning uses model-produced instructions, tasks, and examples to expand training coverage while reducing reliance on curated datasets. VIGC [117] generates visual instructions and iteratively corrects low-quality outputs to reduce hallucinated supervision. MM-Instruct [140] constructs large-scale data by pairing generated instructions with images and producing coherent answers. MindGYM [118] organizes self-generated challenging questions into a reasoning curriculum.

5.3.2. Reflection and Critique-Based Learning

Beyond generating data, another direction teaches models to inspect and revise their own reasoning. R3V [146] bootstraps positive and negative CoT rationales and trains models to refine flawed reasoning through reflection. SRPO [119] introduces reflection-aware reinforcement learning, encouraging concise self-reflection before final answers. LLaVA-Critic and MMEvol [120,147] provide a multimodal critic for evaluating responses across diverse tasks, supporting scalable self-critique. V-Reflection [148] further introduces a “think-then-look” mechanism, prompting models to re-examine visual evidence during reasoning.

5.3.3. Verifier-Guided Self-Improvement

A stronger form of self-evolution closes the training loop through automatic verification. MM-UPT [121] performs unsupervised GRPO using majority-vote self-rewards instead of manually annotated rewards. SelfJudge [149] evaluates sampled responses on unlabeled data to support unsupervised reasoning improvement. AGILE [150] converts executable code and environment outcomes into verifiable rewards for visual puzzles, while Jigsaw-R1 [151] derives rule-based rewards from jigsaw tasks. LLaVA-Critic-R1 [122] reformulates critic data as verifiable signals, allowing critique and generation to improve each other.

5.4. Efficient Reasoning

Efficient reasoning seeks to preserve multimodal reasoning capabilities while reducing post-training and deployment costs. Existing methods mainly follow two routes: offline knowledge distillation and on-policy distillation. The main stages of distillation are outlined in Figure 8 (right).

5.4.1. Knowledge Distillation

Knowledge distillation transfers capabilities from a strong teacher to a smaller or more efficient student. It reduces inference and deployment costs while preserving multimodal reasoning capabilities.

•Behavior Distillation. Behavior distillation transfers teacher responses, logits, features, or cross-modal relations to compact students. LLaVA-KD [123] compresses large MLLMs through output-distribution and relational distillation. LLAVADI [124] compares different distillation targets and highlights the importance of token-level alignment and feature transfer.

•Feedback Distillation. Feedback distillation transfers preference or quality signals beyond direct response imitation. Silkie [129] distills AI-generated multimodal preferences through DPO to improve helpfulness, visual faithfulness, and safety. LLaVA-MoD [125] combines mimic and preference distillation to train a compact MoE-based student.

5.4.2. On-Policy Distillation

On-policy distillation (OPD) combines policy optimization with teacher supervision using rollouts from the current student policy. Compared with offline distillation, OPD better matches supervision to student behavior. It also converts sparse sequence-level feedback into dense token-level guidance. Video-OPD [41] applies OPD to temporal video grounding, using token-level feedback and prioritizing informative samples. X-OPD [126] extends OPD to speech LLMs and uses text-based teachers to align speech and textual reasoning. Uni-OPD [40] generalizes the paradigm to LLMs and MLLMs through balanced student data and improved teacher feedback. Vision-OPD [127] performs self-distillation from crop-based to full-image policies, while VA-OPD [128] prioritizes visually important tokens for more effective supervision.

6. MMPoT of Domain Adaptation

Domain adaptation specializes pretrained MLLMs for settings with distinct data distributions, task protocols, and reliability requirements, as shown in Figure 9. Unlike general instruction tuning or alignment, it targets behavior specialization: models must handle domain-specific evidence formats, output structures, action spaces, and evaluation criteria. Representative methods (Table 4) adapt along different paths. GUI agents such as Mobile-Agent [141] integrate visual perception, planning, and stepwise actions, while GUI-R1 [142] uses R1-style RL with rule-based rewards for cross-platform execution. Document and high-resolution settings are addressed by mPLUG-DocOwl1.5 [143] and LLaVA-UHD [144], which improve structure awareness and visual granularity. In medicine, Med-Gemini [20] supports clinical reasoning and specialized modalities, while AdaMLLM [145] adapts inference under domain and resource constraints. Overall, domain adaptation requires joint consideration of data distribution, visual detail, task interfaces, and reliability.

7. MMPoT of Scalable Training

This section examines how MLLM behavior can be shaped at scale. As MLLMs grow in model size, modality coverage, and application scope, the challenge is no longer merely to improve their capabilities, but to induce reliable behavior without repeatedly updating the full model or relying on costly supervision.

7.1. Parameter-Efficient Post-Training

7.1.1. Low-Rank Adaptation

Low-rank adaptation (LoRA) is a widely used parameter-efficient strategy for MLLMs post-training, as shown in Figure 10 (left). Instead of updating the full pretrained backbone, LoRA freezes the original weights and inserts trainable low-rank update matrices into selected layers. During post-training, only these lightweight LoRA parameters are optimized, while the base model remains fixed. This makes LoRA especially suitable for adapting MLLMs to new instruction data, video tasks, or domain-specific datasets with limited computational cost. Building on this paradigm, MLLMs post-training has extended LoRA in several directions in Table 5. LLaVA-LoRA [1] applies LoRA to visual instruction tuning for efficient adaptation of LLaVA-style models. LLaVA-MoLE [152] and MixLoRA [153] augment LoRA with mixture-based routing to mitigate interference across heterogeneous instruction data. More recent methods, including MokA [154] and LiLoRA [155], tailor low-rank updates to multimodal and continual adaptation. Together, these studies position LoRA as both a parameter-efficient technique and a scalable mechanism for behavior shaping in MLLMs post-training.

7.1.2. Mixture-of-Experts Adaptation

Mixture-of-Experts (MoE) adaptation improves efficiency by routing each token or task to a subset of model parameters, as illustrated in the right panel of Figure 10. This sparse activation is well-suited to MLLMs, where different modalities, domains, and reasoning patterns may benefit from specialized experts. Existing methods follow two main directions. As demonstrated in Table 6, MoE-LLaVA [156] and large-scale systems such as Qwen3-Omni [158], Qwen3-VL [159], MiniMax-01 [160], and Seed1.5-VL [86] use sparse expert backbones to expand model capacity while limiting active computation. MoExtend [157] instead adds new experts to pretrained MoE models for new modalities or tasks while preserving the original backbone. Overall, MoE-based MLLMs post-training scales behavior shaping by selectively activating or extending specialized experts rather than uniformly updating dense parameters.

7.2. Compute-Efficient Post-Training

Compute-efficient post-training reduces the cost of adapting and deploying MLLMs while preserving task-relevant information. Existing methods (Table 7) mainly target high-resolution visual processing and excessive visual tokens. The mechanism underlying compute-efficient is visualized in Figure 11.

7.2.1. Efficient Visual Processing

Visual encoding is a major bottleneck because high-resolution inputs produce many visual tokens. Efficient methods preserve fine details while controlling training and inference costs. LLaVA-UHD [144] uses image modularization and token compression, while AdaMLLM/AdaLLaVA [161] dynamically adjusts inference computation. InternVL2 [60] improves high-resolution processing through dynamic tiling, adaptive cropping, and patch-wise processing, respectively.

7.2.2. Token Compression

Token compression lowers prefill and attention costs by removing redundant visual tokens while retaining task-relevant evidence. FastV [162] prunes tokens after early LLM layers, while VisionZip [163] selects informative image and video tokens. SparseVLM [164] uses text-guided sparsification and token recycling, whereas TRIM [170] relies on CLIP-based relevance. TokenPacker [165] adopts coarse-to-fine compression to preserve local details.

7.2.3. Long-Context Optimization

Long-context optimization reduces the cost of post-training MLLMs for long videos, multi-image inputs, and extended multimodal conversations. It manages spatial redundancy within frames and temporal redundancy across frames through compression, memory, sampling, and sequence parallelism. LongVU [166] adaptively removes redundant frames and query-irrelevant spatial tokens. LongVILA [167] combines context extension, long-video SFT, and multimodal sequence parallelism. LongVA [168] transfers extended context capabilities from language to vision, enabling long-video understanding without additional video training. VideoGrid-style methods arrange frames into temporal grids for denser coverage under fixed context budgets. VideoChat-Flash [169] combines hierarchical token compression with short-to-long training for long videos.

8. MLLMs Post-Training Benchmarks

This section reviews the datasets and benchmarks used for MLLMs post-training and evaluation. The discussion is organized into two parts. The first categorizes representative datasets, while the second reviews widely used metrics for general evaluation. A schematic overview is illustrated in Figure 12.

8.1. Datasets and Benchmarks

Datasets and benchmarks play a central role in MLLMs post-training by defining which behaviors are learned, calibrated, and evaluated. As summarized in Table 8, existing resources can be grouped by their behavior-shaping objectives: instruction following, preference alignment, reasoning enhancement, and domain adaptation.

•Instruction Following. Instruction-following resources establish basic MLLM interaction by teaching models to interpret multimodal prompts and produce task-appropriate responses. LLaVA-Instruct-150K [1] and ShareGPT4V [68] provide large-scale supervision for visual dialogue and general task completion. SEED-Bench [171], MM-Vet [172], MMBench [173], and MME [175] evaluate generalization across perception, cognition, OCR, spatial reasoning, and bilingual settings. MM-IFInstruct [174] and VC-IFInstruct [176] further target fine-grained compliance with explicit visual and textual constraints.

•Preference Calibration. Alignment resources provide feedback for training and criteria for evaluating faithfulness, safety, and human preference. POPE [177], MMHal-Bench [178], and HallusionBench [179] assess visual grounding and hallucination. Lingua-SafetyBench [180], MM-SafetyBench [192], FigStep [193], and JailBreakV-28K [194] examine unsafe instructions, cross-modal attacks, and visual jailbreaks. LLaVA-RLHF [36], and RLHF-V [37], provide preference and feedback data for alignment optimization.

•Reason Enhancement. Reasoning resources support and evaluate multi-step inference over visual and textual evidence. ScienceQA [181], AI2D [195], and CLEVR [196] cover scientific, compositional, and structured visual reasoning. More challenging benchmarks, including MMMU [182], MathVista [183], MathVision [197], OlympiadBench [198], PuzzleBench [184], MME-CoT [185], and MME-Reasoning [186], emphasize expert knowledge, mathematical reasoning, and multi-step problem solving.

•Domain Adaptation. Domain-specific resources extend evaluation from general visual dialogue to specialized inputs, evidence types, and task protocols. DocVQA [187], OCRBench [189], and ChartQA [199] assess document layouts, text recognition, tables, and numerical reasoning. Resources for medicine, GUI interaction, video, and high-resolution vision further evaluate adaptation to specialized evidence and action requirements. Compared with general instruction benchmarks, they place greater emphasis on robustness and deployability in specialized settings.

8.2. Evaluation Metrics

Evaluation metrics convert model outputs into comparable scores. Existing MLLMs benchmarks generally use reference-based or judge-based metrics, depending on whether outputs admit deterministic evaluation.

8.2.1. Reference-Based Metrics

Reference-based metrics compare predictions with ground-truth answers and are suited to closed-ended or objectively verifiable tasks. Common choices include exact match for short-answer QA, accuracy for multiple-choice and yes/no questions, F1 for classification, and mAP or IoU for visual grounding. MME [175], MMBench [173], SEED-Bench [171], and MMMU [182] mainly use accuracy-based scoring, whereas grounding tasks rely on localization metrics. Free-form generation is also evaluated using reference-similarity metrics such as BLEU, CIDEr, and ANLS. These metrics are common in captioning, VQA, and document understanding benchmarks, including ChartQA [199] and DocVQA [187]. However, they may overlook semantic correctness and reasoning quality. More robust designs include circular evaluation in MMBench [173], log-probability scoring in MMMU [182], and reasoning-aware evaluation in MathVista [183] and MME-CoT [185].

8.2.2. Judge-Based Metrics

Judge-based metrics are used for open-ended tasks with multiple valid responses. Human experts or capable LLM/MLLM judges typically assess responses in two ways. Score-based evaluation assigns numerical ratings for helpfulness, relevance, correctness, and detail. MM-Vet [172] and MMHal-Bench [178] use this approach for open-ended multimodal responses. Comparison-based evaluation ranks candidate responses and reports win rates or Elo-style scores. It is commonly used in preference and alignment settings, such as LLaVA-RLHF [36], RLHF-V [37], and VLFeedback/Silkie [129]. Although judge-based metrics better capture subjective quality and user preference, their reliability depends on judge capability, prompt design, and evaluation consistency.

9. Future Directions

In this section, we discuss key directions for advancing MLLMs post-training beyond current paradigms. Our discussion is organized around three complementary themes: grounding behavior shaping, reliability-aware evaluation, and scaling for generalization.

9.1. Grounded Behavior Shaping

•Native Multimodal Post-Training. Most current post-training pipelines still organize multimodal behavior around language supervision, where visual, video, or audio inputs are eventually aligned to text-style responses. A future direction is to develop native multimodal post-training signals that preserve modality-specific structures, such as spatial layout, temporal continuity, acoustic cues, and cross-modal correspondence. This would allow MLLMs to learn behaviors directly from multimodal evidence rather than treating non-text signals as auxiliary context.

•From Digital Understanding to Physical Interaction. Current MLLMs are mainly optimized for understanding digital content such as images, videos, documents, and screens. However, future multimodal agents must interact with the physical world, where perception is continuous, actions change the environment, and feedback is delayed or uncertain. Post-training should therefore connect visual understanding with action-oriented reasoning, enabling models to move from describing multimodal inputs to making grounded decisions in dynamic environments.

9.2. Reliability-Aware Evaluation

•Trustworthy Evaluation. Existing evaluations often emphasize benchmark accuracy, but high scores do not necessarily imply reliable behavior. Future evaluation should diagnose whether post-trained MLLMs are visually grounded, calibrated, consistent, safe, and robust under distribution shifts. This requires metrics that expose hallucination, overconfidence, shortcut reasoning, and unstable responses, rather than only rewarding final-answer correctness.

•Complex Real-world Scenarios. Many benchmarks are still built from static images, short videos, or closed-form QA tasks. Real applications involve ambiguous evidence, long-horizon context, changing environments, and interaction constraints. Future benchmarks should therefore simulate more complex real-world scenarios, where models must maintain state, handle uncertainty, and adapt their behavior across continuous multimodal inputs.

9.3. Scaling for Generalization

•Post-training Scaling toward Generalist MLLMs. Post-training scaling should go beyond simply increasing data volume or training compute, and instead improve behavioral transfer across tasks, domains, modalities, and interaction settings. A key challenge is to determine how diverse supervision signals, model capacity, and optimization strategies jointly contribute to generalization, rather than merely fitting a larger collection of post-training examples. The ultimate goal is to develop generalist post-trained MLLMs whose learned behaviors can be composed, reused, and adapted beyond the datasets, task formats, and reward functions encountered during optimization.

•Streaming Understanding of a Continuously Unfolding World. MLLMs must eventually operate in a world that evolves continuously rather than within a fixed collection of benchmark samples. This requires post-training mechanisms that support persistent memory, temporal event abstraction, selective retrieval, online feedback integration, and continual adaptation. Models should be able to accumulate multimodal experiences over extended interactions, update their knowledge and behaviors when the environment changes, and distinguish transient observations from durable information. A sustainable MLLM should therefore absorb new experiences over time while preserving previously acquired capabilities, maintaining behavioral consistency, and avoiding catastrophic forgetting or uncontrolled model drift.

10. Conclusion

This survey reviews the rapidly growing literature on MLLMs post-training through a unified behavior-shaping perspective. We examine how post-training methods steer pretrained MLLMs toward reliable, grounded, and task-oriented behavior. We further connect these methods to their supervision sources, feedback signals, benchmarks, and evaluation protocols, providing a structured view of the full behavior-shaping loop. By synthesizing current progress, limitations, and future directions, this survey charts the evolving landscape of MLLMs post-training and points toward dependable multimodal intelligence.