Predicting the World via Video Representation: A Comprehensive Survey on Video World Models

1. Introduction

World Models constitute a fundamental paradigm for environment modeling, characterized by the learning of compact and structured latent representations to capture complex environmental dynamics. This capability enables agents to perform efficient prediction, planning, and control within a latent space. This line of research, epitomized by the seminal work of Ha and Schmidhuber [1], has laid the groundwork for subsequent latent dynamics-based reinforcement learning. As research advances toward general intelligent systems, environment modeling is undergoing a paradigm shift from explicit geometric and state-based representations to dynamics-centric formulations grounded in visual sequences. Against this backdrop, Video World Models have emerged as a pivotal research direction bridging perception, prediction, and decision-making.

Unlike traditional three-dimensional (3D) or four-dimensional (4D) world models that rely on explicit geometric reconstruction [2,3,4,5,6,7,8,9,10], video world models take two-dimensional (2D) visual inputs as the primary modeling substrate, characterizing environmental dynamics through continuous video sequences. This design aligns with cognitive hypotheses of human perception [11]: humans do not directly observe complete 3D structures but instead infer spatial organization and physical dynamics from sequences of 2D retinal images. Modeling environments through 2D video sequences not only adopts more natural representational assumptions but also substantially reduces the computational and system-level complexity associated with high-dimensional volumetric data [12,13,14,15,16,17,18]. Consequently, video world models provide a concise yet powerful framework for understanding and predicting complex dynamic environments, making them a core direction in general environment modeling research.

The evolution of world models has been marked by notable architectural and methodological developments. The Dreamer family of models, beginning with DreamerV1 [19], introduces latent state-space models for RL and enables imagination-based planning directly in latent space without relying on pixel-level control. DreamerV2 [20] further extends this framework to more diverse and challenging task settings by incorporating discrete latent representations and robust normalization strategies. Meanwhile, foundational approaches including PlaNet [21] and SLAC [22] improve sample efficiency and long-horizon prediction by learning structured latent dynamics.

With the rapid advancement of generative modeling, diffusion models and autoregressive models are increasingly integrated into the video modeling domain. This shift redirects attention from perception-oriented video synthesis toward systematic modeling of temporal dynamics and latent state transitions. Against this backdrop, video world models such as DriveDreamer [23] and Drive-WM [24] represent a new stage in video-based environment modeling. Rather than emphasizing pixel-level synthesis quality alone, these approaches construct generative simulations that support prediction, planning, and decision-making, showing strong potential in complex dynamic scenarios such as autonomous driving.

Despite these advances, the field still lacks consensus on the definition, modeling boundaries, and core capabilities of video world models [12,25,26,27,28,29,30,31]. Some studies emphasize large-scale generative architectures, including diffusion and autoregressive frameworks, for long-horizon modeling and multimodal generation [32,33,34,35,36,37,38], while others focus on self-supervised or weakly supervised learning to improve data efficiency and generalization [39]. Existing papers typically categorize methods by methodological focus or by application domains such as RL, robotics, and autonomous driving [40,41,42,43,44,45,46]. This task-centric perspective fragments the field, obscuring shared principles and fundamental distinctions across paradigms, and ultimately limiting systematic analysis within a unified framework as well as deeper insights into cross-task and cross-domain generalization.

To address these challenges, this paper reviews video world models from a high-level perspective and proposes a unified analytical framework. Specifically, we introduce a taxonomy that groups existing approaches into two classes: Implicit State Deduction and Explicit Visual Modeling. The former learns compact latent state representations that support efficient reasoning and decision-making, whereas the latter generates high-fidelity visual observations to explicitly simulate future environmental states. This taxonomy is independent of specific model architectures, learning algorithms, or application scenarios; instead, it is grounded in the fundamental question of how environmental states are represented and evolved over time. This perspective clarifies the role of video world models in broader intelligent system research and provides a principled reference for developing general and scalable environment modeling frameworks.

Figure 1. A comprehensive overview of the evolution of video world models across different learning paradigms. The timeline (2020-2025) categorizes notable world model approaches based on their learning paradigms and architectural designs.

Figure 1. A comprehensive overview of the evolution of video world models across different learning paradigms. The timeline (2020-2025) categorizes notable world model approaches based on their learning paradigms and architectural designs.

The primary contributions of this paper are as follows:

• To the best of our knowledge, this paper presents the first comprehensive review that systematically studies video world models from a unified perspective. We consolidate their core concepts, modeling assumptions, architectural foundations, learning paradigms, and evaluation practices, and provide an organized overview of this rapidly developing research area.
• We introduce a unified taxonomy for video world models, categorizing existing approaches into implicit state deduction and explicit visual modeling. This taxonomy further clarifies how different methods represent, evolve, and utilize environmental states, offering a structured lens for understanding the design space of video-based environment modeling.
• We systematically review representative benchmarks and evaluation protocols for video world models, summarizing commonly used datasets, metrics, and experimental settings, and identifying key limitations of current evaluations in measuring long-horizon consistency, physical plausibility, and decision-relevant fidelity.
• We discuss key open challenges and promising future directions, including scalable long-term simulation, physical grounding, efficient deployment, and cross-domain generalization, to guide future research toward more general and robust environment modeling systems.

The remainder of this paper is organized as follows. Section 2 introduces the background of video world models and presents our unified taxonomy, which decomposes the field into implicit state deduction and explicit visual modeling. Section 3 reviews implicit state deduction approaches, categorizing methods by learning paradigm, including reinforcement learning, self-supervised learning, and imitation learning. Section 4 examines explicit visual modeling frameworks, focusing on diffusion-based and autoregressive architectures for spatiotemporal generation. Section 5 summarizes benchmark datasets, evaluation metrics, and experimental protocols. Section 6 discusses open challenges and future research directions. Section 8 concludes the paper.

2. Background and Taxonomy of Video World Models

2.1. From Video Generation to Video World Model

Early video modeling research has largely centered on the synthesis of visually plausible video sequences. Under this formulation, videos are represented as temporally ordered image sequences, and the modeling task is framed as predicting future frames from past observations. These approaches extend image generation architectures to the temporal domain by incorporating recurrent neural networks, convolutional LSTMs, or predictive coding mechanisms. Despite promising short-term performance, these methods rely heavily on pixel-level supervision and implicit temporal correlations, resulting in rapid error accumulation and degraded temporal consistency over long horizons. This exposes fundamental limitations in modeling complex real-world dynamics. To address uncertainty in future video evolution, stochastic latent variable models have been adopted to capture temporal dynamics within compact latent spaces [47,48,49]. By decoupling high-dimensional visual observations from underlying dynamics, these approaches improve predictive diversity and robustness. Representative methods include World Models and its successors, which integrate variational autoencoders with recurrent dynamics to learn latent representations for imagination and prediction [50]. PlaNet and SLAC further extend this paradigm by supporting planning and control directly from raw pixels [51]. Despite their conceptual significance, these models remain largely constrained by reconstruction-driven objectives and exhibit limited scalability to semantically rich, long-horizon environments.

The rapid advancement of large-scale generative modeling has ushered in a new phase of video modeling research. Diffusion-based and autoregressive frameworks have significantly improved video synthesis, enabling the production of high-fidelity, temporally coherent sequences across diverse domains [52]. Advances in diffusion-based approaches achieve remarkable visual realism through progressive denoising [53], whereas autoregressive token-based methods reformulate video modeling as a sequence prediction problem over discrete visual tokens . These paradigms enhance temporal modeling and abstraction, facilitating the development of powerful conditional video generation systems. Despite their strong perceptual performance, most of these models remain primarily optimized for video generation, with limited emphasis on learning explicit environment representations or supporting long-term reasoning and decision making.

As video models are applied to real-world scenarios such as autonomous driving, robotics, and embodied intelligence, the limitations of purely generative objectives become increasingly evident [54]. In these settings, agents reason over latent environment states, predict action outcomes, and operate under partial observability and extended temporal dependencies [55]. These requirements have driven a conceptual shift from video generation to video world models, which learn compact and structured representations of dynamic environments rather than focusing solely on visual synthesis. Representative approaches, such as the Dreamer family, demonstrate that learned latent dynamics support planning and control, positioning video world models as a foundational component for integrating perception, prediction, and decision making in complex systems.

2.2. Taxonomy of Video World Models

The rapid development of video world models has produced a diverse set of architectures and learning paradigms, varying in their representation of environment states and modeling of temporal dynamics. A coherent taxonomy is essential for clarifying conceptual relationships among existing methods and enabling systematic comparisons across application domains [56]. From a modeling perspective, most approaches can be distinguished by their latent state representations and the mechanisms used to learn and propagate state transitions over time. These design choices directly affect the capacity of a model to perform long-horizon prediction, reasoning, and decision making in complex environments.

Figure 2. A comprehensive overview of video world models across different learning paradigms. The figure illustrates the corresponding model architectures and learning processes for each method: (a) Reinforcement Learning, (b) Self-Supervised Learning, (c) Imitation Learning, (d) Diffusion-based Video World Models, (e) Autoregressive-based Video World Models. This figure also systematically presents the diverse techniques used for video world modeling, categorizing them into (A) Implicit State Deduction and (bottom left) Explicit Visual Modeling (bottom right).

Figure 2. A comprehensive overview of video world models across different learning paradigms. The figure illustrates the corresponding model architectures and learning processes for each method: (a) Reinforcement Learning, (b) Self-Supervised Learning, (c) Imitation Learning, (d) Diffusion-based Video World Models, (e) Autoregressive-based Video World Models. This figure also systematically presents the diverse techniques used for video world modeling, categorizing them into (A) Implicit State Deduction and (bottom left) Explicit Visual Modeling (bottom right).

Based on these considerations, existing video world models can be broadly categorized by implicit state representation and dynamics learning strategy. Some methods infer continuous latent variables that compactly encode environment states, typically optimized with prediction-based objectives. Other approaches discretize visual observations into tokenized representations and cast video modeling as a sequence prediction task, enabling scalable long-range temporal modeling through autoregressive objectives. Within both categories, differences emerge in the incorporation of actions, rewards, or external constraints, reflecting varying degrees of interaction and control awareness. This taxonomy offers a structured overview of representative video world models and provides an organizing framework for the detailed analysis in the following sections.

3. Implicit State Deduction

3.1. Continuous Latent Variable Representations

3.1.1. Reinforcement Learning-based

These methods address POMDPs from pixels by learning a world model within the MBRL paradigm. The prevalent Recurrent State-Space Model (RSSM) learns three components: an encoder $q_{\varphi }\left(z_t\right|o_t)$, a dynamics model with deterministic state $h_t=f_{\psi }(h_{t-1},z_{t-1},a_{t-1})$ and stochastic output $p_{\theta }\left(z_t\right|h_t)$, and a decoder $p_{\xi }\left(o_t\right|s_t)$ with reward predictor $p_{\zeta }\left(r_t\right|s_t)$ where $s_t=(z_t,h_t)$. The generative process is:

$$p(o_{1:T},r_{1:T},s_{1:T}\mid a_{1:T})={\displaystyle \prod _{t=1}^T}{p}_{\xi }(o_t\mid s_t)p_{\zeta }(r_t\mid s_t)p(s_t\mid h_t),$$

(1)

with $p(s_t\mid h_t)\equiv p_{\theta }(z_t\mid h_t)\delta (h_t-f_{\psi }(h_{t-1},z_{t-1},a_{t-1}))$. This factorization enables latent-space imagination: rolling out future trajectories $(s_{\tau },r_{\tau })$ in the compact latent space without pixel decoding. Policy optimization (e.g., actor-critic) on these rollouts decouples high-dimensional rendering from control, yielding high sample efficiency. The objective combines a variational bound on observations with reward maximization for end-to-end training [57].

Early work by Hafner et al. [21] introduced the PlaNet model, which learns a latent dynamics model with both deterministic and stochastic components from pixels and performs planning via the Cross-Entropy Method (CEM) in SLAC, which integrates representation learning and RL into a single end-to-end framework. It learns a stochastic latent sequential model and trains an actor-critic directly in the learned latent space, improving stability and sample efficiency. Concurrently, Assran et al. proposed learning Deep Structured Representations for Model-Based Reinforcement Learning (SOLAR), which explicitly structures the latent space to support simple (e.g., linear) dynamics models and classical control methods such as iterative LQR, thereby enabling deployment on real-world robotic tasks.

The Dreamer series significantly advanced this line. Hafner et al. [19] introduced the DreamerV1 model, which learns an RSSM from pixels and uses latent trajectory imagination to train policy and value functions purely from imagined futures. This approach achieved high performance on 20 visual continuous control tasks while avoiding expensive pixel-level planning. Later, Wu et al. [58] present ContextWM, which pre-trains continuous context-aware world models from in-the-wild videos and transfers them to RL tasks, demonstrating improved generalization. Together, these works illustrate a trajectory from initial pixel-based video prediction models to sophisticated, scalable discrete token world models that enable sample-efficient reinforcement learning, robust cross-domain generalization, and novel applications like curriculum generation and offline RL.

3.1.2. Self-Supervised Learning

Self-supervised learning has emerged as a core paradigm for continuous latent state modeling in video world models, enabling the discovery of compact and predictive representations directly from raw video streams. By exploiting temporal continuity and predictive objectives, these methods learn latent world states that discard pixel-level redundancy while retaining information critical for modeling future dynamics [59].

An early and influential framework in this line of research is World Models, which formulates world modeling as a modular decomposition of perception, dynamics, and control under self-supervision. In this formulation, each visual observation $o_t$ is first encoded into a continuous latent variable $z_t$ using a variational autoencoder:

$$z_t\sim q_{\varphi }(z_t\mid o_t),$$

(2)

where the encoder is trained through reconstruction of video frames. Temporal evolution is then captured by a recurrent dynamics model parameterized as a mixture density network, which predicts the distribution over future latent states and rewards conditioned on the current latent state and action:

$$p_{\theta }(z_{t+1},r_t\mid z_t,a_t).$$

(3)

This design demonstrates that long-horizon prediction and planning can be performed entirely within a learned continuous latent space. Although initially studied in interactive control settings, World Models establishes a foundational principle for video world models in which implicit world states are induced through self-supervised temporal prediction rather than explicit semantic annotation, a principle that later continuous latent dynamics models further formalize and extend.

Building upon this idea, subsequent work such as SWIM [60] emphasizes stronger temporal coherence and motion-aware representation learning. Instead of relying solely on frame-wise reconstruction, SWIM introduces predictive consistency objectives that align latent representations across time. By encouraging predicted latent states to match future observations, the learned representations encode object-level structure and motion patterns that persist over multiple frames. Such designs improve robustness to appearance variation and enhance the model’s ability to capture long-term dependencies in video dynamics, which is critical for world modeling beyond short-term prediction. More recent approaches depart from pixel-level generation entirely and focus on representation prediction in latent space. V-JEPA [61] extends joint-embedding predictive architectures to video by learning through masked spatiotemporal prediction. Given a video clip, the model observes a subset of context blocks and predicts the representations of disjoint target blocks from the same clip:

$$\mathcal{L}=\sum _i{∥f_{\theta }\left(x_i^{\mathrm{target}}\right)-g_{\varphi }\left(x^{\mathrm{context}}\right)∥}_2^2,$$

(4)

where the target and context encoders are trained to align their outputs without reconstructing pixels. A key empirical finding of V-JEPA is that semantic abstraction arises from the masking strategy itself. Predicting sufficiently large target regions forces the model to capture high-level semantics, while ensuring that context regions are spatially distributed preserves global scene structure. When combined with transformer-based architectures, V-JEPA scales efficiently to large models and yields strong performance across a range of downstream visual and video understanding tasks. From the perspective of video world modeling, this demonstrates that implicit state deduction can be achieved through predictive representation learning rather than explicit frame synthesis.

Scaling this paradigm to internet-scale video data, V-JEPA 2 further reveals the potential of self-supervised latent prediction. Large-scale pretraining on massive collections of uncurated videos produces world representations with strong motion understanding and anticipation capabilities, even in the absence of action supervision. When aligned with language models, the representations learned by V-JEPA 2 support video reasoning tasks, indicating that self-supervised visual world models can serve as a foundation for multimodal understanding and decision making. Complementary to representation-centric approaches, object-centric self-supervised world models such as Dyn-O [62] introduce structural inductive biases into continuous latent spaces. By decomposing scenes into object-level latent variables that persist over time, Dyn-O learns world dynamics through pixel-level reconstruction and temporal prediction losses alone. Such factorized latent representations improve interpretability and facilitate structured video prediction and planning, highlighting the role of inductive bias in shaping the geometry of learned world states [63].

Overall, self supervised learning for continuous latent world modeling has evolved from reconstruction driven dynamics learning, exemplified by World Models, to large scale predictive representation learning, represented by V-JEPA and V-JEPA 2. Despite differences in architecture and objectives, these methods share a unifying principle: latent world states are not directly observed but are inferred through temporal prediction constraints imposed by video data itself. This principle underpins implicit state deduction in modern video world models, serving as a foundational link between perception and long-horizon reasoning.

3.1.3. Imitation Learning

Within the video world modeling paradigm, imitation learning serves as a mechanism for guiding implicit state deduction when direct interaction or explicit rewards are limited. Rather than reproducing demonstrated videos at the pixel level, imitation learning leverages pretrained video world models to reason about latent environment dynamics and generate task-consistent future evolution. In this setting, demonstrations provide high-level behavioral constraints, while the world model supplies a predictive environment for planning and simulation.

VILP [64] illustrates this paradigm by treating demonstrations as evidence of task intent rather than visual trajectories to be copied. A video world model is used to internally simulate how the environment may evolve under different action hypotheses, allowing the agent to select trajectories that align with the demonstrated objective while remaining physically plausible. Here, imitation learning operates at the level of latent state transitions, shaping the use of the world model without constraining it to exact visual reproduction. OSVI-WM [65] further reinforces this view by emphasizing the role of video world models as internal simulators for long-horizon reasoning. This model views demonstration videos as clues about environment dynamics and uses imitation to guide future predictions in the world model. The resulting behavior emerges from reasoning over predicted environment evolution, rather than from frame-wise matching to demonstrations. These examples show how imitation learning guides video world models to generate task-consistent and physically plausible video. By operating on implicit world states rather than raw pixels, imitation learning enables video world models to enable planning, generalization, and decision making in embodied and interactive scenarios.

3.2. Discrete Token Representations

3.2.1. Reinforcement Learning-based

These methods apply the MBRL paradigm within a discrete latent space to solve POMDPs. They first learn a tokenizer (e.g., VQ-VAE) that maps images $o_t$ to discrete codes $z_t$, and then train a dynamics model $p_{\theta }\left(z_{t+1}\right|z_{\le t},a_{\le t})$ on the token sequence. The resulting world model defines the joint probability $p(z_{1:T}\mid a_{1:T})={\prod }_{t=1}^T{p}_{\theta }(z_t\mid z_{<t},a_{<t})$, upon which a policy ${\pi }_{\psi }(a_t\mid z_{\le t})$ is optimized via RL. This discrete formulation provides key benefits: it creates a natural information bottleneck to reduce noise, enhances compositional generalization through token recombination, and mitigates compounding prediction error due to a bounded state space. These properties enable substantially more robust and efficient latent-space planning compared to continuous representations.

Early exploration of discrete world models for RL includes SimPLe, which used a pixel-level video prediction model as a world model for sample-efficient RL in the Atari 100k benchmark, setting a baseline for future work. [66] then proposed using VQ-VAE to compress pixels into discrete codes and trained a convolutional LSTM dynamics model on these codes, demonstrating that smaller, discrete models could remain sample-efficient. A major breakthrough came with DreamerV2, which replaced the continuous Gaussian latents in RSSM with multiple discrete categorical variables. This discrete latent representation significantly improved world model accuracy on Atari games and achieved human-level performance from pixels. The subsequent DreamerV3 [67] scaled this approach with robust normalization techniques, enabling a single configuration to work across diverse domains and underscoring the generality of discrete latent world models.

Recent works have further expanded the scope. DART [68] tokenizes observations using VQ and employs a Transformer-decoder as an autoregressive token-level world model, with a Transformer-encoder policy, showing strong sample efficiency on Atari. Genie [69] learned a foundational, generative interactive environment from large-scale unsupervised web videos using discrete video tokens and latent action models, enabling the training of RL agents in synthesized environments. For curriculum RL, CQM [70] used VQ-VAE discrete representations as a semantic goal space to automatically generate training curricula, improving exploration in goal-conditioned RL. Moreover, DCWM [71] systematically studied discrete codebook world models for continuous control tasks, showing advantages in efficiency and robustness over continuous counterparts. Finally, VeoRL [72] demonstrated how tokenized world models learned from offline video data can enhance offline RL policy optimization and value correction.

Together, these works illustrate a trajectory from initial pixel-based video prediction models to sophisticated, scalable discrete token world models that enable sample-efficient reinforcement learning, robust cross-domain generalization, and novel applications like curriculum generation and offline RL.

3.2.2. Self-Supervised Learning

Self-supervised learning has emerged as a central paradigm for learning discrete latent state spaces in video world models, as it enables scalable training on large collections of unlabeled videos while preserving temporal structure and dynamics. Within this paradigm, video sequences are first compressed into sequences of discrete visual tokens, after which temporal dependencies are learned via sequence modeling objectives. This formulation transforms video modeling into a latent-state prediction problem, where discrete tokens serve as abstract representations of implicit world states.

VideoGPT [73] represents one of the earliest and most influential self-supervised approaches that formulate video modeling as an autoregressive prediction problem over discrete latent tokens. The core idea is to decouple spatial perception from temporal modeling by introducing a learned vector-quantized encoder. Given an input video sequence ${\left\{x_t\right\}}_{t=1}^T$, each frame is encoded into a grid of discrete latent variables $z_t\in {\mathcal{Z}}^{H\times W}$ using a VQ-VAE, where $\mathcal{Z}$ denotes a finite codebook. Temporal dynamics are then modeled by an autoregressive Transformer that factorizes the joint distribution over latent tokens as

$$p\left(z_{1:T}\right)={\displaystyle \prod _{t=1}^T}p(z_t\mid z_{<t}).$$

(5)

By operating entirely in the discrete latent space, VideoGPT significantly reduces computational cost while retaining high-level spatiotemporal structure. From a world modeling perspective, the discrete tokens implicitly represent latent environment states, and autoregressive prediction corresponds to learning state transition dynamics. Although VideoGPT is trained without explicit action inputs, it shows that self-supervised next-token prediction on discretized video representations is sufficient to learn meaningful temporal regularities, thereby laying the groundwork for subsequent discrete world models.

Building upon the discrete tokenization paradigm, MAGVIT [74] introduces a more expressive and scalable video tokenizer tailored for high-resolution and long-duration videos. Unlike earlier VQ-based approaches that treat frames independently, MAGVIT employs a spatiotemporal tokenizer that jointly encodes spatial and temporal information into compact discrete tokens. Formally, a video clip $x_{1:T}$ is mapped to a sequence of tokens ${\left\{z_k\right\}}_{k=1}^K$, where each token represents a local spatiotemporal volume rather than a single frame. The tokenizer is trained using a reconstruction objective combined with perceptual and adversarial losses, ensuring that the discrete representation preserves motion continuity and temporal coherence. Once tokenized, the video is modeled using a self-supervised sequence prediction objective over the discrete token stream. This design shifts the modeling focus from pixel-level redundancy to higher-level temporal abstractions, enabling more stable and semantically meaningful latent dynamics. In the context of video world models, MAGVIT can be interpreted as learning a structured discrete state space in which each token encodes a short-term evolution of the visual world, thereby facilitating long-range temporal modeling under self-supervision. MAGVIT-v2 [75] further advances discrete self-supervised video modeling by unifying tokenization and temporal prediction within a more efficient and semantically aligned framework. A key improvement lies in the introduction of factorized and hierarchically organized token representations, which reduces codebook entropy while enhancing representational capacity. Let $\left\{z_t^{\left(l\right)}\right\}$ denote discrete tokens at hierarchy level l, where lower levels capture fine-grained appearance details and higher levels encode longer-term temporal structure. The overall learning objective jointly optimizes multi-level reconstruction and next-token prediction:

$$\mathcal{L}=\sum _l\left({\mathcal{L}}_{\mathrm{rec}}^{\left(l\right)}+\lambda {\mathcal{L}}_{\mathrm{pred}}^{\left(l\right)}\right).$$

(6)

This hierarchical formulation encourages the model to learn temporally persistent latent states that evolve smoothly over time, which is particularly desirable for world modeling. By emphasizing temporal abstraction and efficient discrete representations, MAGVIT-v2 enables large-scale self-supervised training on internet-scale videos and provides a stronger latent foundation for downstream dynamics modeling. As a result, it brings discrete self-supervised video world models closer to practical long-horizon simulation and planning settings.

4. Explicit Visual Modeling/Model Architecture

4.1. Diffusion-based Video World Models

Diffusion models (DMs) are employed to generate and predict video data by progressively introducing noise and then learning to reverse the corruption process. These models maintain temporal consistency and high-fidelity details during generation. The forward diffusion process gradually adds noise to clean video frames, while the reverse denoising process learns to recover the original video data by reversing this noise.

Representative formulations of diffusion models include Denoising Diffusion Probabilistic Models (DDPMs) [76], Noise Conditioned Score Networks (NCSNs) [77], and Score-based Stochastic Differential Equations (Score SDEs) [78]. The following section outlines the forward diffusion and reverse generation processes of each method, highlighting their specific application to video world models.

DDPMs define a discrete-time diffusion process that progressively corrupts data with Gaussian noise. Let $x_0\sim q\left(x_0\right)$ denote a clean sample. The forward process is a Markov chain $x_1,\dots ,x_T$ governed by a Gaussian transition kernel:

$$q(x_t\mid x_{t-1})=\mathcal{N}\left(x_t;\sqrt{1-{\beta }_t}{x}_{t-1},{\beta }_t\mathbf{I}\right),t=1,\dots ,T,$$

(7)

where ${\left\{{\beta }_t\right\}}_{t=1}^T$ is a variance schedule, and $\mathbf{I}$ is the identity matrix. An important feature of this formulation is that it admits a closed-form marginal distribution at any step t:

$$q(x_t\mid x_0)=\mathcal{N}\left(x_t;\sqrt{{\overline{\alpha }}_t}{x}_0,(1-{\overline{\alpha }}_t)\mathbf{I}\right),$$

(8)

where ${\alpha }_t:=1-{\beta }_t$ and ${\overline{\alpha }}_t:={\prod }_{s=1}^t{\alpha }_s$. The reverse (denoising) process is defined by a learnable Gaussian kernel:

$$p_{\theta }(x_{t-1}\mid x_t)=\mathcal{N}\left(x_{t-1};{\mu }_{\theta }(x_t,t),{\Sigma }_{\theta }(x_t,t)\right),$$

(9)

The reverse process is typically initialized from a simple prior $p\left(x_T\right)$ (e.g., $\mathcal{N}(0,I)$) and carried out through iterative ancestral sampling from $t=T$ down to $t=1$.

NCSNs adopt a score-based perspective in discrete noise scales. Instead of learning an explicit reverse transition kernel, NCSNs learn a noise-conditioned score network that estimates the score function of perturbed data distributions across a sequence of noise levels. The forward perturbation is typically defined by Gaussian noise injection:

$$q_{{\sigma }_t}(x_t\mid x_0)=\mathcal{N}\left(x_t;x_0,{\sigma }_t^2\mathbf{I}\right),$$

(10)

where increasing ${\sigma }_t$ results in progressively noisier samples. For generation, NCSNs typically utilize annealed Langevin dynamics across decreasing noise levels, where the learned score network $s_{\theta }(x,t)$ guides reverse-time updates:

$$x\leftarrow x+{\eta }_t{s}_{\theta }(x,t)+\sqrt{2{\eta }_t}z,z\sim \mathcal{N}(0,\mathbf{I}).$$

(11)

where ${\eta }_t$ is the step size associated with the noise level. This iterative procedure starts from Gaussian noise and gradually moves samples toward regions of higher data density, eventually converging to the clean data domain.

Score SDEs offer a continuous-time formulation of diffusion generative modeling. The forward diffusion process is described by the following stochastic differential equation (SDE):

$$dx=f(x,t)dt+g\left(t\right)d{W}_t,$$

(12)

where $f(x,t)$ and $g\left(t\right)$ denote the drift and diffusion coefficients, respectively, and $W_t$ is a standard Wiener process. Given the marginal density $p_t\left(x\right)$ at time t, the corresponding reverse-time generative process follows the reverse-time SDE:

$$dx=\left[f(x,t)-g{\left(t\right)}^2{\nabla }_{x}log{p}_t\left(x\right)\right]dt+g\left(t\right)d{\overline{W}}_t,$$

(13)

where ${\overline{W}}_t$ is a reverse-time Wiener process and ${\nabla }_{x}log{p}_t\left(x\right)$ is the score function that drives the trajectory toward high-density regions. In practice, a time-dependent score model is learned to approximate ${\nabla }_{x}log{p}_t\left(x\right)$, and sampling is performed by numerically solving the reverse-time dynamics from an initial noise state to the data space.

Diffusion-based Video World Models (VWMs) represent a fundamental paradigm shift in generative AI, moving from static high-fidelity synthesis toward dynamic, interactive physical simulation. Unlike traditional models that rely on deterministic transitions or simple stochastic latent variables, the diffusion architecture utilizes an iterative denoising process to capture the complex, multi-modal spatiotemporal distributions of the real world. This provides a high-tolerance probabilistic framework for simulating complex environments. The research trajectory in this domain begins with validating the capability of large-scale parameterization to model physical laws. GAIA-1 [79] pioneers the application of diffusion models in autonomous driving and establishes the fundamental paradigm for multi-agent interactive simulation through action-conditioned generation. Subsequently, Sora [80] demonstrates the scalability of video simulators via a large-scale Diffusion Transformer (DiT) architecture, exhibiting emergent physical properties such as object permanence and basic collision logic. Building on this, DIAMOND [81] integrates diffusion models into the reinforcement learning loop, proving that pixel-level fine-grained representations offer superior performance over latent-space models in complex visual decision-making tasks.

As the requirement for precise control across diverse task scenarios becomes more pressing, the research focus shifts toward transforming pre-trained generative priors into controllable dynamics simulators. AVID [82] proposes a parameter-efficient adapter scheme that injects action conditions without compromising the original generative capacity. Vid2World [83] and Vidar [84] further explore the causalization of the denoising process. By aligning robotic morphological features, these models generate environmental feedback corresponding to specific action commands, thereby constructing a robust “state-action-state” interaction loop. In response to cumulative drift and physical distortion in long-term predictions, the integration of explicit geometric priors becomes crucial for enhancing simulation fidelity. PA-VDM [85] and Epona [86] effectively mitigate temporal degradation in long-sequence generation through improved attention mechanisms and noise scheduling strategies. To further constrain object trajectories, Geometry Forcing [87] and VerseCrafter [88] incorporate 3D structures, such as point clouds and Gaussian trajectories, into the diffusion process to ensure 4D spatiotemporal consistency. This transition from 2D pixel interpolation to geometry-guided simulation provides a reliable foundation for complex manipulation tasks, as seen in DWM [89] and LaDi-WM [90].

The core value of video world moedls lies in empowering agents with planning and reasoning capabilities. UWM [91] and World4RL [92] validate the efficacy of using virtual trajectories generated by diffusion models for policy refinement. DAWM [93] and DUST [94] further optimize the alignment precision between visual features and control signals via dual-stream architectures, while Astra [95] introduces general interactive capabilities through autoregressive denoising to enhance the universality of the simulator. In parallel, CWMDT [96] introduces digital twin conditioning to support counterfactual reasoning. This capability elevates video world models from simple evolutionary predictors to sophisticated decision engines capable of hypothetical reasoning and counterfactual inference, providing theoretical support for policy evaluation and safety verification in complex dynamic environments.

4.2. Autoregressive-based Video World Models

General-purpose video generation task primarily aims at visual fidelity and alignment with textual or conditional inputs, focusing on synthesizing high-quality and temporally coherent short video clips. In contrast, video world models emphasize causal dynamics, controllability, and long-term consistency, properties that naturally align with autoregressive modeling paradigms. Concretely, autoregressive models factorize the joint distribution of a sequence into a series of temporally ordered conditional predictions, thereby modeling causal temporal evolution through step-by-step generation. Under this modeling paradigm, autoregressive video world models characterize world dynamics via causal sequence decomposition: given historical observations and actions, the model incrementally generates future visual observations. From a global perspective, autoregressive video world models can be abstracted as learning a causal rollout distribution over future observations conditioned on actions:

$$p\left(o_{t+1:t+H}\mid o_{\le t},a_{t:t+H-1}\right)={\displaystyle \prod _{k=1}^H}p\left(o_{t+k}\mid o_{\le t+k-1},a_{t:t+k-1}\right).$$

(14)

Here, $o_t$ denotes the visual observation at time t (e.g., video frames or discrete visual tokens), $a_t$ denotes the action applied at time t, and H is the prediction horizon. This autoregressive factorization is not merely a modeling choice; Rather, it can be interpreted as an explicit temporal rollout mechanism that enables controllable simulation under action conditioning. Such a mechanism facilitates long-horizon consistency and supports imagination-based planning and decision making.

Early work represented by VideoGPT first demonstrates the feasibility of combining discrete visual tokens with autoregressive prediction. By compressing video frames into discrete tokens via a VQ-VAE and performing next-frame prediction using an autoregressive Transformer in latent space, VideoGPT achieves efficient unconditional video generation and preliminary dynamic modeling, laying the foundation for sequence-based video world models. Subsequently, iVideoGPT [73] further integrates generation with planning by introducing actions and rewards as conditional tokens. It constructs a scalable autoregressive Transformer architecture that supports action-conditioned interactive rollouts and is pretrained on millions of robotic trajectories, significantly improving visual planning and model-based reinforcement learning performance. Consequently, autoregressive models not only predict future observations but also support imagination-driven decision making.

Building on this line of work, researchers explicitly strengthen action conditioning to achieve more robust and controllable world evolution, particularly in highly interactive and dynamic scenarios such as autonomous driving. For example, Vista [97] incorporates future dynamic priors through a latent replacement mechanism and supports multimodal action control (e.g., trajectories or natural language commands). It enables high-fidelity autoregressive rollouts of up to 15 seconds and substantially outperforms general-purpose video generators on benchmarks such as nuScenes, demonstrating the generalization ability and temporal consistency of the autoregressive framework in complex driving environments. Furthermore, the Genie series developed by DeepMind pushes autoregressive modeling toward longer temporal horizons and stronger interactivity. Genie learns interactive environments from internet-scale videos without action annotations by combining spatiotemporal video tokenization, an autoregressive dynamics model, and a latent action model. Genie 2 scales this approach into a large foundation model capable of real-time 3D environment generation, while Genie 3 achieves real-time interaction at 720p and 24 fps with consistency over several minutes, bringing autoregressive world models closer to practical world simulators that can be directly invoked by intelligent agents.

Despite these advances, attention-based autoregressive models still face fundamental limitations in computational efficiency and memory capacity when processing long sequences, which hinders sustained interaction beyond minute-level horizons. To address this challenge, Po et al. [98] propose a Long-Context State-Space Video World Model that integrates state-space models (SSMs) into the architecture. By leveraging the linear-time complexity of SSMs and combining block-wise scanning with local attention, their approach balances long-term memory retention and frame-level coherence. Experiments on long-horizon interactive datasets demonstrate strong long-term memory retention and efficient inference in practice, providing a scalable solution for ultra-long temporal scenarios.

Beyond improving physical dynamics simulation, another line of research aims to elevate autoregressive video world models toward foundation-model capabilities. This direction focuses on learning transferable abstract rules, reasoning, and planning abilities directly from pure visual data, enabling autoregressive video world models to serve as a core component of intelligent agents. In this context, VideoWorld [99] explores large-scale training on unlabeled videos by combining FSQ-quantized codebooks with latent dynamics modeling. It advances autoregressive video world models into general-purpose foundation models through a unified next-token sequence modeling interface that integrates perception, prediction, and planning. Experiments on video-based Go and robotic control tasks demonstrate the model’s ability to acquire complex rules, reasoning, and planning behaviors from visual observations alone, marking a significant step toward autoregressive video world models as foundational building blocks for AGI.

5. Benchmarks and Evaluation Protocols

Evaluating Video World Models requires a multi-dimensional approach that disentangles spatial visual quality from temporal dynamics. We categorize the evaluation metrics into three domains: Visual Fidelity, Spatiotemporal Dynamics, and Reconstruction Alignment.

5.1. Benchmarks

The evaluation of video world models has evolved alongside the growing ambition of the field. Early benchmarks primarily focused on action recognition or short-term temporal understanding, such as the Something-Something video database [100], which emphasizes fine-grained motion reasoning through human-object interactions. While influential for temporal modeling, such datasets are limited to discriminative evaluation and do not directly assess generative prediction or environment simulation. Subsequent benchmarks began to target broader video understanding and generation capabilities. VideoGLUE [101] introduced a unified evaluation suite spanning multiple video-language and video understanding tasks, reflecting the growing interest in general-purpose video representations. In parallel, benchmarks such as VBench [102] shifted attention toward generative evaluation, systematically assessing video generation models across dimensions including visual quality, temporal coherence, motion smoothness, and subject consistency. These benchmarks provide standardized tools for comparing large-scale video generators, but remain primarily focused on perceptual realism rather than environment-level correctness.As video world models increasingly target embodied and interactive settings, newer benchmarks have explicitly exposed the limitations of traditional video generation metrics. EWMBench [103] is designed to evaluate embodied world models by emphasizing three critical aspects: visual scene consistency, motion correctness, and semantic alignment with task intent. Its findings reveal that models optimized for perceptual quality often fail to satisfy the structured requirements of embodied tasks, highlighting a mismatch between generic video generation objectives and environment-centric evaluation.

More recent efforts further extend evaluation toward long-horizon prediction and physical plausibility. EVA-Bench [104] evaluates embodied video predictors trained under multi-stage paradigms, combining autoregressive rollout with high-fidelity generation. Extensive experiments demonstrate that such predictors can generalize to longer sequences and downstream robotics-related tasks, providing empirical evidence that large-scale pretrained video models can support realistic video prediction in real-world scenarios. Complementary to dataset-based benchmarks, Fréchet Video Motion Distance [105] introduces a motion-focused metric that explicitly measures the realism of temporal dynamics, addressing a key blind spot of appearance-driven scores such as FID and FVD.The most recent benchmarks reflect a paradigm shift from video quality assessment toward world-level simulation fidelity. WorldSimBench [106] and WorldScore [107] aim to evaluate whether generated videos are consistent with underlying environment dynamics, rather than merely visually plausible. These benchmarks assess aspects such as long-horizon consistency, state transition validity, and environment coherence, offering a unified evaluation framework for world generation. Together, they signal a transition from perceptual evaluation to environment-centric benchmarking, aligning evaluation protocols more closely with the core objectives of video world modeling.

5.2. Evaluation Analysis

5.2.1. Video Quality Analysis

Based on the benchmarks shown in Table 1, it is clear that Diffusion-based models such as GAIA-1 generally outperform others in terms of image quality and motion smoothness, as indicated by their low FID and FVD values. For instance, GAIA-1 achieves an exceptionally low FID of 2.15 and FVD of 12.8, which suggests that it produces visually coherent and temporally consistent videos. This is a key advantage of diffusion models, which excel at generating high-fidelity visual content with smooth motion transitions. However, models like Vid2World, also based on diffusion, have lower FID values but suffer from higher FVD, indicating that while individual frames may be of high quality, they struggle with maintaining long-term temporal consistency.In contrast, models incorporating Reinforcement Learning-based alignment methods, such as CogVidX-2B + HALO, excel in tasks that require interaction and long-term coherence, particularly for human action generation, which scores a near-perfect 99.00. This model shows strong performance in maintaining motion smoothness and human action coherence, highlighting the advantages of RL-based strategies in ensuring that video sequences remain contextually relevant. However, this is achieved at the expense of slightly lower image quality (61.90), which suggests that while RL alignment enhances action and behavior representation, it may not prioritize pixel-level fidelity as much as diffusion models.Autoregressive models, such as Vista, demonstrate strength in maintaining motion smoothness over extended time horizons, but they tend to sacrifice some image quality compared to the diffusion models. While Vista performs exceptionally well in generating temporally coherent sequences, it does not match the FID and FVD results seen in diffusion-based approaches. This suggests that autoregressive models, though powerful in modeling long-term dynamics, may not always achieve the same level of visual fidelity as diffusion-based models, particularly when fine-grained image quality is critical.

5.2.2. Downtask Analysis

The analysis of success rates across different downstream tasks, particularly on the Meta-World and Push-T benchmarks, reveals significant insights into the performance of various models. The best-performing methods in these tasks are notably DINO-WM and DDP-WM, both utilizing Self-Supervised Learning (SSL) paradigms. DINO-WM achieves an impressive 98% success rate on Push-T, while DDP-WM follows closely with 90%. These high performance rates can be attributed to the models’ ability to leverage pre-trained visual features and self-supervised learning for efficient world modeling and long-term planning, allowing them to generalize well on unseen tasks without requiring additional labeled data. Methods such as DreamerV3 and Sparse Imagination, both under the Reinforcement Learning (RL) paradigm, show strong performance, with DreamerV3 reaching 78.3% on Push-T. While these models benefit from end-to-end training in the environment, they generally rely on a higher computational cost and more task-specific training data compared to SSL-based approaches, which may explain their relatively lower success rates. Methods based on Imitation Learning (IL), like DAML, T-OSVI, and VILP, exhibit varied success across benchmarks. OSVI-WM stands out with 83.8% success rate on Meta-World, demonstrating the effectiveness of imitation learning in tasks that require high-level behavior replication, especially when expert demonstrations are available. However, these methods’ performance can be highly sensitive to the quality of the demonstration data, which may limit their generalizability compared to self-supervised methods.

Table 2. Performance comparison of different paradigm video world models on nuScenes and VBench datasheet, including FID, FVD, Image Quality, Motion Smoothness, Subject Consistency, and Human Action.

Model	Resolution	Paradigm	nuScenes		VBench
			FID ↓	FVD ↓	Image Quality ↑	Motion Smoothness ↑	Subject Consistency ↑	Human Action ↑
GAIA-1 [79]	–	Diffusion	2.15	12.8	–	–	–	–
Sora [80]	–	Diffusion	–	–	–	–	–	–
DIAMOND [81]	–	Diffusion	–	–	–	–	–	–
Astra [95]	–	Diffusion	–	–	–	–	–	–
AVID [82]	–	Diffusion	–	–	–	–	–	–
Vid2World [83]	–	Diffusion	–	–	–	–	–	–
Vidar [84]	–	Diffusion	1.40	385.2	–	–	–	–
PA-VDM [85]	–	Diffusion	–	98.5	–	–	–	–
Geometry Forcing [87]	–	Diffusion	–	–	–	–	–	–
VerseCrafter [88]	–	Diffusion	9.80	165.4	–	–	–	–
DAWM [93]	–	Diffusion	–	–	–	–	–	–
World4RL [92]	–	Diffusion	–	–	–	–	–	–
CWMDT [96]	–	Diffusion	4.20	210.3	–	–	–	–
DriveDreamer [23]	128×192	Diffusion	14.90	340.80	–	–	–	–
GenAD [115]	256×448	Diffusion	15.40	184.00	–	–	–	–
ProphetDWM [116]	256×448	Diffusion	6.90	190.50	–	–	–	–
Epona [86]	512×1024	Diffusion	7.50	82.80	–	–	–	–
MaskGWM [117]	288×512	Diffusion	4.00	59.40	–	–	–	–
LongDWM [118]	480×720	Diffusion	12.30	102.90	–	–	–	–
GeoDrive [119]	480×720	Diffusion	4.10	61.60	–	–	–	–
Vista [97]	576×1024	Diffusion	6.90	89.40	–	–	–	–
VC2 + VideoDPO [120]	–	Diffusion+RL	–	–	–	92.18	95.69	99.00
Turbo + VideoDPO [120]	–	Diffusion+RL	–	–	–	88.85	96.10	94.00
CogVid + VideoDPO [120]	–	Diffusion+RL	–	–	–	88.64	94.67	81.00
Turbo-v1 + [121]	–	Diffusion+RL	–	–	72.07	–	–	95.00
Turbo-v2 + HALO [121]	–	Diffusion+RL	–	–	69.11	–	–	97.60
CogVidX-2B + HALO [121]	–	Diffusion+RL	–	–	61.90	–	–	98.00
GAPO [122]	–	Diffusion+RL	–	–	68.98	99.13	95.20	–
OnlineVPO [123]	–	Diffusion+RL	–	–	67.36	99.36	97.58	–
InstructVideo [124]	–	Diffusion+RL	–	–	70.09	96.76	96.45	–
BEVGen [125]	–	Autoregressive	41.20	–	–	–	–	–
VaViM-s [126]	–	Autoregressive	–	–	–	–	–	–
DrivingWorld [127]	–	Autoregressive	7.40	90.90	–	–	–	–
DriveArena [128]	–	Autoregressive	–	–	–	–	–	–
DreamForge [129]	–	Autoregressive	28.77	197.9	–	–	–	–
Drive-WM [24]	192×384	Autoregressive	15.80	122.70	–	–	–	–
Panacea [130]	256×512	Autoregressive	16.96	139.00	–	–	–	–
CogDriving [131]	480×720	Autoregressive	15.30	37.80	–	–	–	–
DrivePhysica [132]	256×448	Autoregressive	3.96	38.06	–	–	–	–
UniScene [133]	256×512	Autoregressive	6.45	71.94	–	–	–	–
DiST-4D [134]	424×800	Autoregressive	6.83	22.67	–	–	–	–
VisionReward [135]	–	RL	–	–	–	–	–	98.40
RDPO [136]	–	RL	–	–	65.11	99.27	97.04	–
CogVidX-2B + VPO [137]	–	RL	–	–	–	–	–	99.00
CogVidX-5B + VPO [137]	–	RL	–	–	–	–	–	99.60
VADER [138]	–	RL	–	–	66.08	98.89	95.53	–
CogVidX-2B + IPO [139]	–	RL	–	–	62.87	98.17	96.79	–

Table 2. Performance comparison of different paradigm video world models on nuScenes and VBench datasheet, including FID, FVD, Image Quality, Motion Smoothness, Subject Consistency, and Human Action.

Model	Resolution	Paradigm	nuScenes		VBench
			FID ↓	FVD ↓	Image Quality ↑	Motion Smoothness ↑	Subject Consistency ↑	Human Action ↑
GAIA-1 [79]	–	Diffusion	2.15	12.8	–	–	–	–
Sora [80]	–	Diffusion	–	–	–	–	–	–
DIAMOND [81]	–	Diffusion	–	–	–	–	–	–
Astra [95]	–	Diffusion	–	–	–	–	–	–
AVID [82]	–	Diffusion	–	–	–	–	–	–
Vid2World [83]	–	Diffusion	–	–	–	–	–	–
Vidar [84]	–	Diffusion	1.40	385.2	–	–	–	–
PA-VDM [85]	–	Diffusion	–	98.5	–	–	–	–
Geometry Forcing [87]	–	Diffusion	–	–	–	–	–	–
VerseCrafter [88]	–	Diffusion	9.80	165.4	–	–	–	–
DAWM [93]	–	Diffusion	–	–	–	–	–	–
World4RL [92]	–	Diffusion	–	–	–	–	–	–
CWMDT [96]	–	Diffusion	4.20	210.3	–	–	–	–
DriveDreamer [23]	128×192	Diffusion	14.90	340.80	–	–	–	–
GenAD [115]	256×448	Diffusion	15.40	184.00	–	–	–	–
ProphetDWM [116]	256×448	Diffusion	6.90	190.50	–	–	–	–
Epona [86]	512×1024	Diffusion	7.50	82.80	–	–	–	–
MaskGWM [117]	288×512	Diffusion	4.00	59.40	–	–	–	–
LongDWM [118]	480×720	Diffusion	12.30	102.90	–	–	–	–
GeoDrive [119]	480×720	Diffusion	4.10	61.60	–	–	–	–
Vista [97]	576×1024	Diffusion	6.90	89.40	–	–	–	–
VC2 + VideoDPO [120]	–	Diffusion+RL	–	–	–	92.18	95.69	99.00
Turbo + VideoDPO [120]	–	Diffusion+RL	–	–	–	88.85	96.10	94.00
CogVid + VideoDPO [120]	–	Diffusion+RL	–	–	–	88.64	94.67	81.00
Turbo-v1 + [121]	–	Diffusion+RL	–	–	72.07	–	–	95.00
Turbo-v2 + HALO [121]	–	Diffusion+RL	–	–	69.11	–	–	97.60
CogVidX-2B + HALO [121]	–	Diffusion+RL	–	–	61.90	–	–	98.00
GAPO [122]	–	Diffusion+RL	–	–	68.98	99.13	95.20	–
OnlineVPO [123]	–	Diffusion+RL	–	–	67.36	99.36	97.58	–
InstructVideo [124]	–	Diffusion+RL	–	–	70.09	96.76	96.45	–
BEVGen [125]	–	Autoregressive	41.20	–	–	–	–	–
VaViM-s [126]	–	Autoregressive	–	–	–	–	–	–
DrivingWorld [127]	–	Autoregressive	7.40	90.90	–	–	–	–
DriveArena [128]	–	Autoregressive	–	–	–	–	–	–
DreamForge [129]	–	Autoregressive	28.77	197.9	–	–	–	–
Drive-WM [24]	192×384	Autoregressive	15.80	122.70	–	–	–	–
Panacea [130]	256×512	Autoregressive	16.96	139.00	–	–	–	–
CogDriving [131]	480×720	Autoregressive	15.30	37.80	–	–	–	–
DrivePhysica [132]	256×448	Autoregressive	3.96	38.06	–	–	–	–
UniScene [133]	256×512	Autoregressive	6.45	71.94	–	–	–	–
DiST-4D [134]	424×800	Autoregressive	6.83	22.67	–	–	–	–
VisionReward [135]	–	RL	–	–	–	–	–	98.40
RDPO [136]	–	RL	–	–	65.11	99.27	97.04	–
CogVidX-2B + VPO [137]	–	RL	–	–	–	–	–	99.00
CogVidX-5B + VPO [137]	–	RL	–	–	–	–	–	99.60
VADER [138]	–	RL	–	–	66.08	98.89	95.53	–
CogVidX-2B + IPO [139]	–	RL	–	–	62.87	98.17	96.79	–

6. Application Areas

6.1. Autonomous Driving Systems

Autonomous Driving Systems (ADS) enable vehicles to perceive their surroundings, plan trajectories, and execute control actions through onboard sensors (e.g., cameras, LiDAR, and radar), artificial intelligence algorithms, and computing units, without continuous human intervention [140,141]. According to the SAE International standard, automation levels range from L0 (no automation) to L5 (full automation) [142]. Currently, the industry has deployed L4 Robotaxi services commercially (e.g., Waymo operates driverless ride-hailing services in multiple cities), while L2/L2+ advanced driver-assistance systems, such as Tesla Autopilot and Ford BlueCruise, have been widely adopted. However, fully autonomous driving at the L5 level remains unrealized [143]. This limitation primarily arises from three key challenges: data scarcity and long-tail scenario coverage, information loss in modular architectures, and the simulation-to-reality gap.

To address these challenges, video world models offer enhanced generative modeling capabilities and introduce a paradigm shift from data-driven reactive perception to implicit-dynamics-driven predictive simulation. Video world models significantly expand the support for training data distributions through generative modeling, alleviating the scarcity of long-tail scenarios. Representative methods, such as the DriveDreamer series [144,145], leverage large language models to provide high-level scene constraints and conditional control, enabling world models to synthesize counterfactual and extreme 4D driving scenarios from real-world videos for systematic data augmentation. GAIA-1 further generates multi-agent driving trajectories conditioned on textual descriptions and actions, reducing reliance on high risk real-world road testing. Video world models also reshape ADS architecture by learning a unified latent world representation that integrates perception, prediction, and planning into a single dynamic model, mitigating information loss and cascading errors inherent in modular pipelines. LAW [146] introduces a self-supervised latent world model, achieving impressive end-to-end trajectory prediction performance on the challenging nuScenes, NAVSIM, and CARLA benchmarks.

Furthermore, video world models support neural closed-loop simulation and counterfactual safety validation, bridging the simulation-to-reality gap. For example, Vista supports multimodal action control and long-horizon prediction (over 40 seconds), simulating millions of kilometers of driving in a neural closed-loop setting and achieving over 27% improvements in FID and FVD metrics. TrafficBots [147] and GenAD utilize multi-agent simulation and structured latent spaces to evaluate counterfactual branches (e.g., lane-change collision risk), while OmniNWM [148] extends this paradigm to panoramic multimodal world modeling for comprehensive safety validation. In summary, video world models present innovative approaches to addressing data scarcity, information loss, and the simulation-to-reality gap in autonomous driving through generative modeling, large-scale data augmentation, unified integration, and neural closed-loop simulation.

6.2. Robotics

Video world models have emerged as powerful embodied world models in robotics, enabling agents to predict environmental dynamics and plan actions effectively. These models learn spatiotemporal mappings that capture the evolution of environments over space and time, providing robots with predictive capabilities crucial for autonomous operation [149]. Recent advances demonstrate that video generation models can serve as cost-effective alternatives to traditional simulators for robotic training, generating synthetic data that captures complex robot environment interactions [150].

One prominent application area is robotic navigation, where Navigation World Models (NWMs) predict future visual representations based on past frames and agent actions, enabling long-horizon planning in complex environments [151]. For instance, Lu et al. [152] introduced GWM Robotics, which is a specialized world model trained on real-world robotics data that predicts video rollouts conditioned on robot actions, supporting both offline training and online deployment. In manipulation tasks, video world models have shown success in generating realistic robot video data after fine-tuning on target platforms, significantly improving policy generalization across different robotic embodiments. The integration of foundation models with video world models has unlocked new capabilities in robot learning. Jang et al. [153] proposed DreamGen, which leverages video world models to enhance generalization by creating diverse training scenarios through predictive simulation. Moreover, video world models facilitate the development of cloud-based simulation environments that can be deployed on devices for real-time decision making. This capability addresses the sim-to-real gap by enabling robots to simulate action outcomes before execution, reducing the need for extensive real-world trial and error learning. The ability to generate counterfactual scenarios, which involves predicting how environments would evolve under different action sequences, further enhances robotic safety and adaptability in uncertain conditions.

6.3. Embodied Intelligence

Embodied intelligence represents a paradigm where agents perceive complex multimodal environments, act within them, and anticipate how their actions will alter future states [154]. Video world models serve as foundational components for embodied intelligence by functioning as internal simulators that capture environmental dynamics, enabling both forward and counterfactual reasoning capabilities. These models bridge the gap between language-based reasoning and physical world understanding, moving beyond the limitations of pure language intelligence toward comprehensive spatial awareness [155].

Recent papers have systematized world models for embodied AI through a three-axis taxonomy that clarifies their functionality across representation learning [156], predictive modeling, and action-conditioned generation. This unified framework formalizes the problem setting and learning objectives for embodied world models, providing a structured approach to evaluating their performance across robotics, autonomous driving, and general interactive environments. A key advancement in embodied intelligence is the tight coupling between morphology, action, and environment [157], whereby world models enable agents to develop intelligence through continuous interaction with their surroundings. Modern approaches leverage world models to create digital twins that simulate physical interactions, social dynamics, and environmental constraints, providing agents with a cognitive framework for reasoning about complex scenarios [158]. Video world models with long-term spatial memory [159] demonstrated how long-term spatial memory, which integrates geometric constraints across spatial, working, and situational memory types, can significantly enhance embodied agents’ ability to navigate and interact with persistent environments. The synergy between large language models (LLMs) and world models represents a crucial frontier in embodied intelligence research. While LLMs excel at abstract reasoning and language understanding, world models provide the spatial and physical grounding necessary for real-world interaction [160]. This integration enables agents to reason about action sequences, predict environmental changes, and adapt strategies based on internal simulations, representing capabilities essential for tasks ranging from household assistance to industrial automation. According to Feng et al. [155], world models are envisioned as the counterpart to LLMs in language, aiming to establish a general-purpose foundation for embodied intelligence that generalizes across domains and tasks.

However, significant challenges remain in scaling world models to handle the complexity of real-world embodied scenarios. Issues such as long-term consistency, multi-agent interaction modeling, and safety guarantees require continued research attention. The establishment of standardized datasets, evaluation metrics, and benchmark environments is essential for advancing the field and enabling embodied world models to move from laboratory demonstrations to real-world applications in smart manufacturing, healthcare, and service robotics.

7. Open Problems and Future Directions

7.1. Towards Mobile Video World Models

Recent video world models often rely on large-scale architectures, such as transformers or diffusion-based models, with billions of parameters [161]. These models achieve high performance in long-horizon simulation and controllable generation. However, they demand significant computational resources and memory, limiting their deployment on mobile devices, edge platforms, and real-time robotic systems [162,163]. Therefore, developing efficient and lightweight video world models is a crucial research direction. The main challenge lies in the high computational cost of spatiotemporal modeling. Video world models process sequences of high-dimensional visual tokens and often require iterative generation steps. Both autoregressive rollouts and diffusion-based sampling introduce latency [164]. Reducing the model size alone is insufficient if inference remains slow. Efficient design must take into account parameter count, memory footprint, and sampling complexity together. Model compression, knowledge distillation, and structured pruning are promising techniques [165]. Knowledge distillation can transfer temporal reasoning ability from a large teacher model to a smaller student model, while pruning can remove redundant attention heads, temporal layers, or token channels to reduce computational cost, though care must be taken to avoid damaging long-term consistency. Quantization further reduces memory usage and inference latency, but applying it to video world models requires careful handling of temporal accumulation errors. To enable deployment in embodied agents, autonomous systems, and interactive applications, mobile video world models need to balance efficiency with long-horizon simulation fidelity. This challenge remains unresolved and warrants further investigation in future research.

7.2. Towards Long-Horizon Consistency Video World Models

Long-horizon consistency is a critical challenge for video world models. Current diffusion-based and autoregressive frameworks exhibit intrinsic limitations in maintaining coherent environment evolution over extended time horizons. Diffusion-based models generate future frames through iterative denoising, conditioned on short-term context. Although this process achieves high visual fidelity, it lacks mechanisms to maintain consistent state propagation over long rollouts. As the prediction horizon increases, reliance on local visual cues leads to drift in object identity, motion, and scene structure, making it difficult to maintain consistency or plan for the future. Autoregressive models, on the other hand, model temporal causality through step-by-step predictions. However, their sequential nature introduces compounding errors over time. Small inaccuracies early in the sequence accumulate, leading to semantic drift and loss of object consistency, along with implausible dynamics. Furthermore, the computational cost of autoregressive rollouts grows linearly with sequence length, limiting their scalability for long horizon simulations. These limitations suggest that achieving long-horizon consistency will require more than simply scaling diffusion or autoregressive models. Future video world models may require the integration of explicit state abstraction, hierarchical temporal modeling, or memory-augmented dynamics to disentangle long-term evolution from short-term synthesis [166,167]. Addressing this challenge at the latent state transition level, rather than frame generation, is likely essential for building reliable and scalable video world models.

7.3. Towards Physically Grounded Video World Models

Despite significant progress in visual fidelity, current video world models still lack robust physical grounding. Many models can generate visually plausible sequences while violating basic physical principles, such as object permanence, gravity, or collision consistency [168]. This gap reveals a fundamental limitation of purely data-driven video generation when used as a surrogate for environment modeling. Physically grounded video world models require stable internal representations that reflect the underlying structure of the physical world. Rather than learning pixel-level correlations alone, models should encode object-centric states, interaction dynamics, and physically consistent transitions. Incorporating physical priors, structured dynamics modules, or hybrid neural–physical components may help constrain long-term evolution and improve controllability in interactive settings [169,170]. Another open challenge lies in evaluation. Existing benchmarks primarily assess visual realism and short-term temporal coherence, but rarely measure physical correctness [171]. Developing physics-oriented evaluation protocols, including evaluations of object permanence, energy conservation, and multi-object interactions, is fundamental to the systematic advancement of physically grounded video world models.

8. Conclusion

Video world models have emerged as a unifying framework that bridges video generation, representation learning, and environment simulation. This paper provides a systematic review of the field, organizing existing approaches along learning paradigms and architectural designs, and highlighting the conceptual shift from frame-level generation to environment-level modeling. We reviewed both implicit state deduction and explicit visual modeling approaches, covering reinforcement learning, self-supervised learning, imitation learning, as well as diffusion-based and autoregressive architectures. While recent models demonstrate impressive progress in visual fidelity and short-term coherence, our analysis reveals persistent challenges in physical grounding and long-horizon consistency. In particular, diffusion-based models lack persistent state propagation, whereas autoregressive models suffer from compounding errors and scalability limitations over long rollouts. These limitations indicate that long-term world modeling cannot be achieved by scaling generative models alone. Video world models represent a foundational step toward visual environment understanding and simulation. Continued progress in this direction has the potential to enable more reliable planning, reasoning, and interaction in embodied intelligence, autonomous systems, and unified world models.