Learning to Model the World: A Survey of World Models in Artificial Intelligence

1. Introduction

World models (WMs) [1,2,3,4,5] have emerged as a fundamental mechanism for building intelligent systems capable of simulation, planning, and decision-making in complex environments [6]. Instead of reacting solely to instantaneous observations, a world model seeks to learn internal representations of the environment that capture its underlying structure [7,8], temporal dynamics [9,10,11], and uncertainty [12,13]. Such an internal model can predict future states, simulate alternative outcomes, and evaluate candidate actions without requiring direct interaction with the real world [14], which is particularly important in scenarios where data collection is expensive, limited, or safety-critical [15,16]. Meanwhile, the limitations of purely model-free methods [17], including low sample efficiency and poor generalization, have further motivated renewed interest in WMs [18,19]. Thus, WMs are increasingly viewed as general-purpose intelligent systems that bridge perception, cognition, and control [20].

The development of WMs has progressed through several stages [21], driven by advances in representation learning [8], dynamics modeling [22], and large-scale training [23]. Early classical model-based approaches, such as Gaussian processes [24,25] and control-oriented dynamics [26], rely on explicit or low-dimensional dynamics with strong assumptions about environment structure, which limits their scalability to complex real-world scenarios. Recent advances in deep learning [27,28], especially in large-scale generative modeling such as video prediction [29,30,31] and multimodal foundation models [32,33], have significantly accelerated research on WMs. Modern WMs [34,35,36] increasingly adopt latent representations to model high-dimensional observations, enabling a clearer separation between perception [37] and dynamics prediction [38]. Meanwhile, transformer frameworks [39] and slot encoding [40] are employed to learn interpretable object-centric embeddings, enabling WMs to model latent dynamics through interacting entities [40]. This shift lays the foundation for neural WMs and broadens their applicability to visual and continuous-control domains [41,42].

Based on their modeling mechanisms and intended usage, as shown in Figure 1, most existing WMs [43,44,45,46,47] can be mainly categorized into four categories. 1) Observation-level generative WMs directly predict future visual [48], language [42], 3D or 4D observations [49] from past environmental observations and actions in high-dimensional spaces, typically using autoregressive [50,51], NeRF [52] or diffusion-based generative architectures [47,53]. 2) Latent embedding-based WMs [7,54] first encode high-dimensional observations into compact latent representations. They then learn transition dynamics in the latent space [32,55,56], allowing efficient prediction, and decision-making without operating on raw observations. 3) Reinforcement learning (RL)-based WMs [9,46,57,58] learn a predictive model of environment dynamics from agent-environment interactions. This learned model then performs planning or policy optimization through imagined rollouts [59], reducing the need for direct real-world interaction. 4) Object-centric WMs [45,60] represent the environment as a collection of discrete objects with structured attributes. They learn object-level dynamics and relational dependencies [61], enabling compositional reasoning and generalization across scenes and tasks.

Over the past few years, WMs [62,63,64,65] have played an increasingly important role across a broad range of application domains, including robotics [66], autonomous driving [67], virtual game simulation [68], GUI-based agents [69], and scientific discovery [70]. For example, in robotics and autonomous driving, they are used to simulate future scenarios [71], predict the behavior of surrounding agents [72] or physical systems [66], and support risk-aware planning [73] and decision-making [6], thereby improving robustness in uncertain environments. Beyond control-oriented applications, WMs also excel at modeling physical [56], biological [70], and medical treatment [74], accelerating scientific discovery through simulation and hypothesis testing. Evidently, they not only advance the development of intelligent systems, but also contribute to broader societal benefits, including safer transportation, more capable robots, and faster scientific innovation [6].

Contributions: Existing surveys on WMs can be broadly grouped into two branches, as summarized in Table 1. The first branch of surveys largely restricts its scope to specific application domains, including autonomous driving [75,76,77,78,79,80], 3D/4D modeling [81,82], video generation [84,85], safety [83], and embodied intelligence [86,87,88,89], particularly in vision-language manipulation and navigation [90,91]. The second branch of surveys [92,93,94] primarily focuses on the basic definitions of WMs, as well as on major application domains of WMs. Such studies are often framed as literature overviews or introductory conceptual discussions. However, there is a lack of in-depth and systematic analyses of world modeling paradigms, simulation platforms, comparative experiments across WMs, and interpretability, as well as new evaluation metrics for measuring the generalization, causal reasoning, and long-horizon consistency of WMs. This survey differentiates itself from prior works and makes the following major contributions to the literature:

• Instead of focusing on specific applications or introductory conceptual discussions, we offer an in-depth and systematic overview of world modeling paradigms, methodologies, key functions, and their relationships.
• We summarize the key evolutionary developments of existing major WMs and their core mathematical formulations across branches from a broader perspective.
• Beyond concentrating solely on commonly studied application domains, we comprehensively review all application areas of world models explored to date. These domains include robotics, autonomous driving, scientific discovery, virtual game simulation, GUI-based agents, as well as interpretability and trustworthiness.
• We provide a more thorough and inclusive overview of benchmark datasets, evaluation metrics, simulation platforms, and comparative experiments across WMs.

Structure of This Survey: The overall structure of this survey is introduced in Figure 2. Specifically, Section 2 introduces four major paradigms of WMs: including observation-level generative WMs (Section 2.1), latent space WMs (Section 2.2), RL-based WMs (Section 2.3), and object-centric WMs (Section 2.4). Section 3 reviews application domains of WMs, including robotics (Section 3.1), autonomous driving (Section 3.2), scientific discovery (Section 3.3), virtual game simulation (Section 3.4), GUI-based agents (Section 3.5), as well as interpretability and trustworthiness (Section 3.6). Section 4 summarizes benchmark datasets & evaluation metrics (Section 4.1), simulation platforms (Section 4.2), and comparative performance (Section 4.3). Finally, Section 5 discusses key challenges that limit the deployment and generalization of WMs and outlines future directions emphasizing unified scientific modeling, causal reasoning, scalable training, and real-world validation.

2. Foundational World Models

WMs [3,9] aim to learn latent environment dynamics that capture the current state of the world and predict its evolution over time, enabling simulation, planning, and decision-making. Generally, at time t, a WM can be viewed as a function that leverages historical states, along with optional agent actions, to predict future environment states:

$$\begin{array}{c}\hfill {\mathbf{z}}_{t+1}\sim {\pi }_{\theta }\left({\mathbf{z}}_{t+1}\right|{\mathbf{z}}_t,{\mathbf{a}}_t,{\mathbf{o}}_{t+1}),\end{array}$$

(1)

where ${\pi }_{\theta }(·)$ denotes the WM. ${\mathbf{z}}_t$ and ${\mathbf{z}}_{t+1}$ are the latent environment states at time t and $t+1$, respectively. ${\mathbf{a}}_t$ denotes the optional agent action at time t, and ${\mathbf{o}}_{t+1}$ represents the observation at time $t+1$. Building upon the formulation in Equation (1), we illustrate the core functions of WMs and the relationships among them, as shown in Figure 3. Specifically, in WMs, simulation generates a set of imagined future trajectories ${\left\{{\zeta }_i\right\}}_{i=1}^{\varphi }$ using learned dynamics, where ${\zeta }_i^t={\left\{{\eta }_i^t\right\}}_{t=1}^T$ denotes the i-th predicted trajectory, consisting of T environment states ${\left\{{\eta }_i^t\right\}}_{t=1}^T$. $\varphi$ and T are the numbers of trajectories and time steps, respectively. Here, ${\eta }_i^t$ corresponds to ${\mathbf{z}}_t$ in Equation (1) for simulation. Additionally, planning evaluates or optimizes over these simulated future trajectories to identify desirable trajectory and obtain its action sequences ${\left\{{\mathbf{a}}_t\right\}}_{t=1}^T$, while decision-making selects and executes the final action ${\mathbf{a}}_T$ based on the planning outcomes.

From a mathematical modeling standpoint, as summarized in Table 2, we roughly classify most existing WMs into four categories, a taxonomy that differs significantly from the simple conceptual introductions used in existing surveys [75,79,83,85,86,87]. 1) Observation-level generative WMs decode latent environment states into future observation-level predictions, with a primary focus on simulation capabilities. This branch has achieved high generation quality and semantic consistency through large-scale supervised training. However, observation-level generative methods still face challenges regarding long-horizon stability, controllability, and high reconstruction costs. 2) Unlike observation-level generative WMs that reconstruct original observations, latent-space WMs primarily emphasize simulation and planning capabilities by predicting future latent representations and capturing semantics in a compressed space. Although they avoid dependence on large-scale labeled datasets, such methods often sacrifice interpretability and fine-grained details. 3) RL-based WMs introduce a reward predictor to learn latent dynamics, primarily supporting planning and decision-making. They can improve sample efficiency yet suffer from real-world distribution shifts and compounding errors, making them more suitable for specific planning tasks rather than modeling a general world. 4) Object-centric WMs represent environment scenes as sets of interacting entities (object slots) and model object-level dynamics, thereby facilitating better planning and decision-making. They improve interpretability, compositional generalization, and efficient reasoning, but reliable tracking and policy learning are challenging.

2.1. Observation-Level Generative World Models

In general, observation-level generative WMs [29,42,95,96,97] leverage strong generative priors to model environment dynamics conditioned on actions and additional inputs such as language prompts [42], visual images [18], or camera trajectories [98]. They primarily enable faithful simulation by generating future observations directly at the observation level. Building upon the general definition of WMs in Equation (1), observation-level generative WMs typically introduce a decoder ${\Phi }_{\theta }(·)$ that projects latent environment states ${\mathbf{z}}_{t+1}$, conditioned on agent action ${\mathbf{a}}_t$ and additional inputs ${\mathbf{c}}_t$ (e.g., camera motion parameters, depth, language, or visual prompts), to future observation ${\widehat{\mathbf{o}}}_{t+1}$. Therefore, their objective can be formally formulated as follows:

$$\begin{array}{c}\hfill {\widehat{\mathbf{o}}}_{t+1}={\Phi }_{\theta }\left({\mathbf{z}}_{t+1}\right),{\mathbf{z}}_{t+1}\sim {\pi }_{\theta }\left({\mathbf{z}}_{t+1}\right|{\mathbf{z}}_t,{\mathbf{a}}_t,{\mathbf{c}}_t),\end{array}$$

(2)

where the predicted observation ${\widehat{\mathbf{o}}}_{t+1}$ at time $t+1$ is a future observation. When performing simulation, observation-level generative WMs synthesize multiple imagined trajectories ${\left\{{\zeta }_i\right\}}_{i=1}^{\varphi }$, each ${\zeta }_i$ including a sequence of observation-level environment states ${\left\{{\widehat{\mathbf{o}}}_i\right\}}_{i=1}^T$ over T time steps.

Contemporary generative models, such as large language models (LLMs) [99], diffusion models [53], flow matching models [51], and Gaussian splatting [100], can be viewed as an early and rudimentary form of WMs. Nevertheless, as formulated in Equation (3), they cannot incorporate agent actions that interact with the underlying world state to understand the world, and instead rely solely on additional inputs ${\mathbf{c}}_t$ to achieve realistic observation-level generation:

$$\begin{array}{c}\hfill {\widehat{\mathbf{o}}}_{t+1}={\Phi }_{\theta }\left({\mathbf{z}}_{t+1}\right),{\mathbf{z}}_{t+1}\sim {\pi }_{\theta }\left({\mathbf{z}}_{t+1}\right|{\mathbf{z}}_t,{\mathbf{c}}_t).\end{array}$$

(3)

Therefore, these commonly-used generative models cannot capture the causal dynamics of the environment, causing poor long-horizon simulation stability and limited controllability. At present, there remains a substantial gap in transitioning from Equation (3) to Equation (2) to realize ideal observation-level generative WMs. Based on the type of observations, we categorize this branch of WMs into language (Section 2.1.1), visual (Section 2.1.2), and 3D and 4D observations (Section 2.1.3).

2.1.1. Language Observations

Language observations enable abstract transition modeling but can suffer from brittle long-horizon rollouts when errors compound [101]. Leveraging LLMs (e.g., ChatGPT [99] and Llama [102]), RAP [42] improves decision-making by framing reasoning as planning within a WM and by searching over intermediate outcomes rather than relying on a single forward pass. Reliability can be further improved by injecting explicit structure, including precondition and effect knowledge, as well as causal abstractions for interventions [34,95]. Furthermore, a related line of work studies execution-like rollouts over formal structures; for example, SURGE characterizes boundary cases of surrogate execution in code-related settings [103]. Context scaling remains central, and long-context architectures extend modeling to million-length video and language sequences [97]. Overall, integrating long-language modeling, structure and planning is crucial for robust and scalable language-level WMs.

2.1.2. Visual Observations

Video generators learn visual observations over plausible futures and serve as strong initializations for world modeling [29,33,44]. In Figure 4, Emu3 [33] exhibits high generative fidelity and temporal coherence by training a unified multimodal transformer with a next-token prediction objective. Recent foundational video models [18,104] demonstrate strong visual fidelity and coherence, making them natural priors for controllable simulators. Wan [104] adopts a LLaVA-style architecture [105] to enhance visual understanding via a low-rank mechanism. To enable better planning and control, recent work adapts video generation models into interactive simulators by injecting actions and enforcing state persistence, with Vid2World [106] providing a general recipe, and real-robot settings revealing robustness challenges under distribution shift [1]. Physics-aware guidance reduces implausible dynamics [2], while explicit memory improves long-horizon consistency under varying viewpoints [44]. CoLA-World [107] aims to advance controllable WMs by jointly learning latent actions and world dynamics. Recent work emphasizes visual realism, while the incorporation of physical and motion-driven mechanisms will help build controllable WMs in the future development.

2.1.3. 3D and 4D Observations

While visual observations offer strong appearance fidelity, 3D and 4D observations [98,108,109] expose stable, geometry-grounded states across viewpoint changes. To achieve controllable 3D scene generation, Text2Room [110] employs iterative fusion of multi-view images to directly generate high-quality meshes from textual prompts. Extending this capability to the temporal dimension, 4D-fy [96] alternately integrates supervisory signals from 3D perception models as well as text-to-image and text-to-video models. Furthermore, WonderWorld [19] achieves efficient generation of interactive 3D worlds while maintaining visual quality and geometric consistency. Invisible Stitch [111] addresses geometric inconsistencies in 3D scene generation caused by monocular depth estimation by introducing depth inpainting. PointWorld [112] unifies state and action into a 3D point flow space to forecast full-scene dynamics from a few RGB-D observations. In summary, 3D and 4D WMs can construct high-fidelity and interactive worlds, but still need to improve efficiency and physical consistency.

2.2. Latent-Space World Models

In contrast to observation-level generative WMs [106], latent-space WMs use high-dimensional latent representations to model environmental dynamics, enabling them to better capture underlying semantic structures and motion, thereby primarily supporting simulation and planning. Based on the self-supervised foundational frameworks of JEPA [54] and V-JEPA [55], V-JEPA 2 [7] extends these approaches to construct latent-space WMs, as shown in Figure 5. Specifically, V-JEPA 2 [7] introduces a latent space predictor ${\Omega }_{\theta }(·)$ to predict the future latent representation ${\widehat{\mathbf{z}}}_{t+1}$ at time $t+1$, and then minimizes the difference between ${\widehat{\mathbf{z}}}_{t+1}$ and the corresponding ground-truth representation ${\mathbf{z}}_{t+1}$. The major optimization objective of V-JEPA 2 [7] is defined as follows:

$$\begin{array}{cc}\hfill & {\widehat{\mathbf{z}}}_{t+1}\sim {\Omega }_{\theta }\left({\widehat{\mathbf{z}}}_{t+1}\right|{\mathbf{z}}_{1:t},{\mathbf{s}}_{1:t},{\mathbf{a}}_{1:t}),\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill & min\mathcal{L}=\mathbb{E}[\parallel {\widehat{\mathbf{z}}}_{t+1}-{\mathbf{z}}_{t+1}{\parallel }^2],\hfill \end{array}$$

(5)

where ${\mathbf{z}}_{1:t}$, ${\mathbf{s}}_{1:t}$, and ${\mathbf{a}}_{1:t}$ represent the latent representations, the end-effector state sequence (e.g., robotic poses), and the agent action sequence from time 1 to t, respectively.

Following V-JEPA 2 [7], which scales training to over a million hours of video and introduces action conditioning, seq-JEPA [8] disentangles equivariant and invariant features to improve representation distinctness, while MC-JEPA [113] further enhances motion modeling by jointly training optical flow estimation. Another notable trend leverages pretrained visual foundation models. Approaches such as DINO-WM [27], DINO-World [114], and DINO-Foresight [32] use pretrained DINO models to treat spatial patch features as environmental states, enabling effective zero-shot planning and the prediction of evolving semantic features via masked transformers. Additionally, Delliaux et al. [115] incorporate geometric priors to further improve transition prediction and representation disentanglement.

2.3. Reinforcement Learning (RL)-Based World Models

RL-based WMs [46,116] use learned dynamics to facilitate planning and policy decision-making. Building upon Equation (1), a reward model $q_{\theta }(·)$ is leveraged to predict the reward $r_t\sim q_{\theta }\left(r_t\right|{\mathbf{z}}_t,{\mathbf{a}}_t)$ at time t. Their objective over T time steps is:

$$\begin{array}{c}\hfill max\mathbb{E}\left[\sum _{t=1}^T{q}_{\theta }\left(r_t\right|{\mathbf{z}}_t,{\mathbf{a}}_t)\right].\end{array}$$

(6)

Based on the formulation in Equation (6), the dominant latent-dynamics backbone for RL-based WMs is the recurrent state-space model (RSSM) [66]. As shown in Figure 6, RSSM combines deterministic recurrent states with latent variables to capture uncertainty. Motivated by RSSM [66], Dreamer [9] introduces latent imagination as an alternative to expensive environment interaction by learning policies from imagined rollouts. Moreover, DreamerV2 [10] demonstrates that discrete latents stabilize learning and scale to pixel-based Atari control, while DreamerV3 [3] emphasizes robustness across diverse domains using a single configuration. Subsequent variants [22,117,118] further improve stability and inductive bias through reward smoothing, loss balancing, dynamic modulation, and the incorporation of privileged information under the safety-oriented settings.

While model-based rollouts enable better planning and decision-making capabilities, distribution shift and robustness still remain challenging in closed-loop settings [35,59]. Recent RL-based WMs [119,120] address these limitations by tightening the model-policy loop through collaborative or policy-conditioned modeling, as well as stabilized objectives for value estimation and planning [59]. However, these models still rely on separate reward models. Some approaches instead train robust world models under reward-free conditions [121,122]. Additionally, token-based world models, such as REM [116], convert environmental observations into discrete token sequences resembling language and utilize sequence models to predict environmental dynamics in latent space. Motivated by these advances, recent generalization trends increasingly emphasize contextual transfer, continual adaptation, and multi-task pretraining [123] guided by foundational models.

2.4. Object-Centric World Models

To address limitations in object tracking and interpretable dynamics, object-centric WMs [16,20,45,132] use slot attention [40] to represent structured scenes as sets of composable entities (object slots) and to model object-level dynamics with improved interpretability and compositional generalization. Here, slots are a set of learnable embeddings that represent objects in a scene or independent visual concepts. This branch of WMs tends to facilitate planning and decision-making in world modeling. Inspired by slot attention [40], object-centric WMs focus on minimizing the difference between the predicted object slots at future time steps and the corresponding ground-truth [60]:

$$\begin{array}{cc}\hfill & {\widehat{\mathbf{h}}}_{t+1}\sim {\lambda }_{\theta }\left({\widehat{\mathbf{h}}}_{t+1}\right|\mathcal{A}\left({\mathbf{o}}_t\right),{\mathbf{a}}_t),\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill & min\mathcal{L}=\mathbb{E}\left[\mathcal{D}({\widehat{\mathbf{h}}}_{t+1},\mathcal{A}\left({\mathbf{o}}_{t+1}\right))\right],\hfill \end{array}$$

(8)

where $\mathcal{A}(·)$ represents the slot attention function that maps an observation to a set of slots. $\mathcal{A}\left({\mathbf{o}}_t\right)$ and $\mathcal{A}\left({\mathbf{o}}_{t+1}\right)$ denote the slot sets at times t and $t+1$, respectively. ${\lambda }_{\theta }(·)$ is a slot predictor that predicts the future slot set ${\widehat{\mathbf{h}}}_{t+1}$ at time $t+1$, and $\mathcal{D}(·)$ denotes the commonly-used distance measure.

As illustrated in Figure 7, SlotFormer [39] encodes sequential observations into object-level representations and predicts future object states by learning visual dynamics. To improve slot quality and controllability, Dyn-O [45] introduces a slot-based world model for Procgen-style games by injecting segmentation priors and disentangling static and dynamic factors. Related work constructs object-centric dynamics to support efficient exploration, multi-step planning, and improved policy learning [16,61]. Zhang et al. show that pixel-level objectives may miss decision-critical small entities and propose an object-centric pipeline that identifies key objects via segmentation and models their dynamics for imagined rollouts [20]. Beyond forward prediction, object factorization also enhances latent action discovery and compositional generalization under compute constraints [133]. In robotics [134] and autonomous driving [135], object-centric representations enable semantic conditioning by aligning object states with language and agent actions [136]. Object-centric representations excel in visual reconstruction, but struggle with downstream tasks such as policy learning [132].

2.5. Discussion of Expected World Models

Although existing foundational WMs have made impressive progress, they remain far from envisioned future WMs in how knowledge is represented, updated, and validated. Current WMs primarily optimize predictive accuracy from observational data, without enforcing causal, mechanistic, or falsifiable structure. As summarized in Table 3, future WMs encode knowledge symbolically as equations and formal relations (e.g., Hamiltonian mechanics, and Maxwell’s equations) independent of specific datasets or observers. Knowledge is updated cumulatively by extending shared theory rather than through opaque, entangled parameter changes. Such knowledge is socially verifiable: equations and proofs can be checked, and predictions tested through reproducible experiments. These models have explicit domains of validity, and failures arise from principled breakdowns of governing laws, such as classical mechanics at relativistic speeds or the Navier–Stokes equations in turbulent regimes, rather than unpredictable extrapolation errors. In contrast, current WMs often fail opaquely, functioning mainly as sophisticated interpolators rather than explicit, theory-grounded models. Bridging this gap may require integrating symbolic and neural components, enforcing hard physical constraints, and designing protocols that reward causal discovery and falsifiability over reconstruction alone.

3. Applications of World Models in AI

Foundational WMs acquire powerful perception and reasoning capabilities by learning latent environmental dynamics. Leveraging their ability to infer future states, WMs enable agents to perform simulation, planning, and decision-making. In this section, we revisit the applications of foundational world models in downstream domains, including robotics, scientific discovery, and GUI-based agents.

3.1. World Models for Robotics

Artificial intelligence (AI) for robotics [57,137] requires robots to operate under partial observability and rich physical interaction. Recently, this field has been significantly advanced by embodied foundational WMs [23,127,129], which scale world modeling through large multimodal data and unified model interfaces, aiming to integrate perception, prediction, and action into a transferable backbone. Unlike existing surveys on embodied intelligence, particularly those focusing on vision-language navigation and manipulation [86,87,88,89,90,91], as illustrated in Figure 8, we review nearly all major application tasks of world models in robotics, including manipulation (Section 3.1.1), navigation (Section 3.1.2), policy learning (Section 3.1.3), and locomotion (Section 3.1.4). Additionally, we reanalyze robotic manipulation from a control-perception loop perspective and navigation from a navigation-reasoning loop perspective, rather than focusing solely on vision-language perception.

3.1.1. Manipulation

Robotic manipulation [138] imposes stringent requirements on WMs due to contact-rich dynamics and discontinuous state transitions (e.g., grasp or release). Some previous surveys [87,88,89,91] typically treat manipulation as vision-language-action (VLA) pipelines, while manipulation-focused reviews commonly organize WMs by architecture. Generally, they focus on specific perception-centric aspects and do not explicitly analyze how WMs interface with manipulation control. In contrast, we adopt a control-perception loop taxonomy. As presented in Table 4, we categorize manipulation WMs by (a) the control interface they expose (i.e., latent action state imagination and control-oriented planning), (b) the unified world-action interface they provide (i.e., world action modeling), and (c) the specific prediction targets they model (i.e., visual future prediction). Compared with prior surveys [89,91], this survey emphasizes the WM’s operational role in the manipulation loop and the associated trade-offs in bias accumulation, latency, and feasibility, rather than primarily cataloging architectures. Specifically, latent imagination is sample-efficient yet can accumulate bias over long rollouts. Control-oriented planning enables constraint-aware model predictive control (MPC) or search via inference-time simulation, but is compute-intensive and depends on accurate short-horizon predictions. World action modeling can unify perception, prediction, and action for transfer, but still faces challenges in computational cost and robustness. Visual future prediction supports planning and supervision but requires aggressive compression to remain consistent.

Latent Action State Imagination: This branch of approaches uses WMs during training as compact simulators that generate rollouts for policy improvement. As shown in Figure 9, AdaWorld [63] instantiates this idea for manipulation by enabling knowledge transfer across embodiments via adaptive rollouts, while FLARE [124] extends the paradigm to regularize diffusion-based control. More generally, these methods learn latent states and action-conditioned transitions, optimizing policies with RL-style updates driven by imagined returns [66]. OSVI-WM [139] and DEMO³ [140] aim to synthesize supervision for imitation and demonstration-augmented RL. Besides, pretrained generative models support low-data refinement and grasp selection [141].

Control-Oriented Planning: In contrast to latent action state imagination, which leverages WMs during training, control-oriented planning methods employ WMs at inference time as rollout simulators for receding-horizon trajectory search [142,143]. PETS [73] improves reliability via uncertainty-calibrated ensemble dynamics. TD-MPC [26] and TD-MPC2 [11] accelerate replanning with compact latent dynamics, while TAWM [144] makes rollouts time-aware under varying observation and control rates. In parallel, generative planners synthesize trajectories via denoising in Diffuser [6] and Decision Diffuser [125], or through constraint-guided sampling in potential-based diffusion planning [145]. For long-horizon manipulation, WMs enable structured search with intermediate abstractions: PIVOT-R [126] rolls out waypoints in a waypoint-aware model. Rollout utility is further improved by action-tree visual guidance [146] and multi-view scaffolding [147].

World Action Modeling: Unlike control-oriented planning methods, this branch moves beyond using WMs solely for action search or future prediction by offering a unified interface that couples perception, world evolution, and action generation via tokenized multimodal streams and autoregressive or diffusion-based objectives. Diffusion-based systems such as Genie Envisioner [127] and WoW [23] generate instruction-conditioned interaction futures with closed-loop rollouts and bridge imagination to execution via action decoding or inverse modules. Token-based variants (e.g., PAR [148] and iMoWM [128]) represent observations and actions in a shared discrete space to enable efficient rollout and learning. PhysicalAgent [149] introduces iterative reasoning and replanning, while World4Omni [134] reuses pretrained image generators to propose goal or future states.

Visual Future Prediction: Visual future prediction approaches learn action-conditioned predictors for future observations and utilize visual rollouts for planning-by-prediction and goal-reaching [150,151]. ViPRA [152] uses world models (WMs) to learn a motion-centric latent action space and decodes it into robot controls with limited supervision. To improve physical plausibility, FlowDreamer [130], TesserAct [131], and ORV [153] incorporate 3D/4D structure (e.g., flow and 4D representations), while WristWorld [154] and RoboMaster [155] address viewpoint shifts and contact-rich interactions. Moreover, RoboDreamer [150] and Vidar [156] exploit rollouts for compositional generalization and action reasoning. Predicted futures can supervise action learning (Video2Action [151], LaDi-WM [157]) or reduce rollout cost via key-frame generation (KeyWorld [158]).

3.1.2. Navigation

Robotic navigation [167] imposes distinct challenges on WMs, primarily requiring long-range spatial consistency and robust operation in dynamic environments. Some previous surveys on embodied AI [89,91] categorize navigation within broad vision-language-action (VLA) frameworks, often overlooking the explicit role of WMs, from internal simulation to the reasoning layer. These discussions typically focus on reactive policy architectures and do not systematically analyze how WMs serve as a structured reasoning interface to bridge raw perception with complex navigation control. In contrast, as presented in Table 5, this survey organizes navigation WMs by their functional roles within the navigation-reasoning loop, rather than under the broad VLA paradigm. Accordingly, we categorize existing methods into five groups from a navigation-reasoning loop perspective. Specifically, generative imagination enables high-fidelity foresight but suffers from long-horizon drift, while persistent memory mitigates partial observability yet demands efficient compression and updating. Neuro-symbolic frameworks provide interpretable and socially compliant navigation at increased architectural cost, whereas test-time adaptation and belief-guided modeling emphasize robustness and knowledge transfer in novel environments.

Generative Imagination Navigation: These generative imagination methods explicitly simulate future observations to support navigation planning. As illustrated in Figure 10, the Navigation World Model (NWM) [159] employs a large-scale conditional diffusion transformer to predict visual rollouts, enabling decision-making via imagined future observations. Unified WM [163] integrates visual foresight with memory to support long-horizon planning. In addition, MindJourney [160] couples vision-language models (VLMs) with generative architectures to imagine future views for spatial reasoning. Besides, scene graph world models [161] further integrate structured graphs, allowing robotic agents to simulate object-level relations and spatial configurations.

Persistent Memory Representation: Addressing partial observability, this branch of research leverages WMs to maintain evolving internal states. After Persistent Embodied WM [173] proposes to explore a 3D latent representation that accumulates spatial semantic knowledge over time, NavMorph [162] introduces a self-evolving model with contextual memory, enabling continuous updates during exploration. Notably, Unified WM [163] also utilizes memory for foresight. Moreover, RECON [164] employs latent goal models to guide rapid exploration in open worlds, effectively acting as a goal-directed WM for robotic navigation.

Neuro-Symbolic Modeling: Neuro-symbolic modeling approaches extend dynamics modeling with linguistic or symbolic constraints. WMNav [165] integrates VLMs to ground language into environment memory for navigation. To address social constraints, NaVi-WM [166] augments WMs with deductive chain-of-thought reasoning. Similarly, Neuro-Symbolic WM [167] combines neural perception with symbolic reasoning to ensure socially compliant and interpretable navigation. NavMorph [162] further embeds language grounding directly into its continuous WM updates.

Test-Time Adaptation: This category employs WMs to handle imperfect priors and novel environments when executing navigation tasks. Representative methods include guiding inference-time search via vision value models [168] and dynamically implanting WM components for rapid adaptation [174]. To address the challenge of model inconsistency, Kinodynamic Planning [169] proposes robust methods that tolerate mismatches across conflicting models. MindJourney [160] further demonstrates test-time scaling by coupling VLMs with controllable video diffusion WMs.

Belief-Guided Modeling: Rather than explicit imagination, belief-guided modeling regard WMs as decision-sufficient belief abstractions. Deep active inference [175] uses multi-timescale belief dynamics to guide diffusion policies. Theoretical work in Abs-Sim2Rea [172] demonstrates that models preserving information states enable robust knowledge transfer. In addition, X-Mobility [170] utilizes a probabilistic world model and multi-stage training to mitigate challenges such as data scarcity and poor robustness.

3.1.3. Policy Learning

WMs function as imagination engines, action-free models, and evaluation proxies for robotic policy learning by allowing policies to be optimized through imagined interactions rather than costly real-world trials. Prior work falls into three categories. Imagination-based optimization enables zero-shot policy refinement via synthesized rollouts, yet risks exploiting model inaccuracies during extended optimization. Action-free modeling optimizes sample efficiency by leveraging unlabeled or human videos for knowledge transfer, but face challenges in aligning latent representations with precise robot actions. Evaluation proxy provides scalable, safe benchmarking environments without real-world costs, though their utility depends on strictly preserving relative policy rankings under distribution shift.

Imagination-Based Optimization: WMs are increasingly adopted not merely as predictors, but as optimization substrates that enable robotic policy learning directly in imagination. WMPO [176] formulates world-model-based policy optimization for vision-language-action models, enabling policy gradients to be computed over imagined trajectories rather than real interactions. Diffusion-based WMs improve rollout fidelity and stability, enabling policies to be refined entirely within generated environments. This is demonstrated by World4RL [177] and DAWM [178], both of which achieve substantial gains in offline and online RL settings.

Action-Free Modeling: Another key trend is to learn control policies from unlabeled or weakly labeled observation sequences through WMs, without relying on expensive explicit action supervision. LAWM [179] and PreLAR [180] demonstrate that aligning inferred latent actions with limited ground-truth control signals enables scalable policy learning from passive observations. WMs also extract transferable dynamics from heterogeneous supervision. For instance, DexWM [181] and cross-embodiment WMs [182] use large-scale human videos for zero-shot transfer to robot policies, while TraceGen [183] and 3DFlowAction [184] abstract dynamics into embodiment-agnostic representations to support policy adaptation across morphologies.

Evaluation Proxy: Beyond the policy optimization methods discussed above, controllable and simulator-like methods explicitly position the WM as a proxy simulator for evaluation. Ctrl-World [185] and Robotic WM [186] enable policy-in-the-loop training, comparison, and evaluation without incurring costly real-world rollouts. Recent works such as WorldGym [187] and WorldEval [188] further demonstrate that WMs can preserve relative policy rankings, establishing them as reliable proxies for real-world policy evaluation.

3.1.4. Locomotion

WMs have recently demonstrated strong potential for learning robust locomotion behaviors by explicitly capturing contact-rich dynamics, terrain geometry, and wide-range physical consistency. Rather than relying solely on reactive policies, they serve as structured predictors that reconstruct or predict locomotion-relevant state transitions, enabling policy optimization under complex constraints. Despite high performance on specific hardware and effective modeling of contact uncertainty, they struggle with the high-frequency discontinuities of real-world impacts and remain tightly coupled to distinct morphologies, leaving the scaling of existing WMs to diverse interactions an open challenge.

Specifically, WMP [196] learns a predictive visual WM to ground perception for legged control, significantly improving robustness on uneven and partially observed terrains. WMR [197] introduces a reconstructive WM that regularizes policy learning by enforcing physically consistent latent dynamics, leading to stable humanoid locomotion under severe distribution shifts. Beyond visual prediction, works such as ProTerrain [198] and DWL [199] incorporate physical structure and terrain uncertainty into WMs to better capture locomotion dynamics. In addition, WMs have been extended to support contact reasoning and whole-body motion synthesis. For example, Ego [200] integrates egocentric perception with latent dynamics and control, enabling real-time, contact-aware motion generation for humanoid robots.

3.2. World Models for Autonomous Driving

WMs facilitate autonomous driving systems [79,80] by modeling traffic dynamics and agent interactions, allowing vehicles to anticipate future scenarios and plan actions accordingly in uncertain environments [75,76,77]. While some surveys [75,76,77,78,79,80] investigate recent advances in autonomous driving, we reanalyze this domain from a new perspective on the involvement of WMs in the decision-making loop, providing a clearer understanding of the role of WMs in autonomous driving. Accordingly, as summarized in Table 6, we categorize WMs for autonomous driving into predictive modeling, action-conditioned imagination, and decision-centric integration. Specifically, predictive modeling approaches excel at capturing scene dynamics for pretraining and data generation. Action-conditioned imagination frameworks introduce action-aware foresight, enabling explicit comparison of alternative driving behaviors under diverse intents. Decision-centric integration further strengthens autonomy by using imagined futures to guide planning and policy learning. However, current approaches remain limited by data scarcity, computational efficiency, and challenges in ensuring reliable and physically consistent wide-range simulations under diverse real-world driving environments.

3.2.1. Predictive Modeling

Predictive modeling for autonomous driving treats WMs as environment forecasters that predict future world states from past observations, without conditioning on ego actions or control commands [62,201]. These methods formulate world modeling as a video or scene generation problem, exploring how traffic scenes evolve over time and providing reusable representations for simulation and generation [202,203]. Copilot4D [189] uses discrete diffusion to model coherent 4D scene evolution without action supervision, while UNO [72] proposes that observation-only occupancy prediction can capture complex traffic dynamics. Other predictive modeling approaches [72,190,203] have been comprehensively reviewed in recent surveys [78,92], and are therefore not discussed in detail here. From a broader perspective, predictive modeling WMs in autonomous driving are conceptually aligned with observation-driven dynamics models used in other domains, including scientific forecasting tasks such as climate prediction [204,205], where future system evolution is inferred from historical states. Such models emphasize learning latent spatiotemporal regularities of the environment, and the effectiveness of 3D/4D predictive representations in driving suggests broader applicability to long-horizon forecasting.

3.2.2. Action-Conditioned Imagination

The action-conditioned imagination branch [206,207] extends the above predictive modeling by conditioning future world evolution on ego actions, trajectories, or high-level commands, enabling counterfactual reasoning. Specifically, WMs function as imagination modules that produce future outcomes for candidate behaviors [208,209,210]. GAIA-1 [192] and Drive-WM [191] are representative works of this line of research, both introducing explicit action conditioning into observation-level generative WMs to facilitate long-horizon, behavior-aware future generation. Other action-conditioned imagination approaches for autonomous driving have been comprehensively discussed in existing surveys [76,78] and are therefore not elaborated here. Action-conditioned imagination WMs in autonomous driving share close commonalities with WMs in robotic manipulation [63,211] and navigation [162]. In these domains, future environment states are explicitly simulated under alternative action hypotheses to enable counterfactual reasoning and behavior-aware evaluation. This formulation positions action-conditioned imagination as an intermediate layer between environment modeling and decision-making.

3.2.3. Decision-Centric Integration

Decision-centric integration approaches [193,212] embed WMs in the decision loop, using imagined rollouts to evaluate actions, guide planning, or update policies. In this setting, WMs are no longer treated as standalone generators, but function as core components of action selection and control. Think2Drive [193] exemplifies this paradigm by leveraging imagination-based reinforcement learning, enabling policies to be improved through internal rollouts with reduced reliance on external simulators. DriveVLA [195] further demonstrates that large-scale WMs can directly support policy learning via data scaling, positioning the WM itself as a central substrate for closed-loop decision-making. Decision-centric WMs in autonomous driving exemplify a broader paradigm shared with world modeling approaches in games [213], reinforcement learning [46], and other sequential decision-making domains [186]. In these domains, imagined rollouts are embedded into planning or policy learning processes, enabling action evaluation under uncertainty without exhaustive real-world trials. From this perspective, WMs transition from passive predictors to active decision substrates, a role that is evident in the closed-loop autonomy requirements of driving.

3.3. World Models for Science

WMs for science transition the focus from interaction-centric simulation to data-driven scientific modeling of complex, partially observed systems. As shown in Figure 11(a), WMs for science can be mainly divided into two categories: social and socioeconomic systems (Section 3.3.1) and physical and natural sciences (Section 3.3.2). In social and socioeconomic domains, WMs enable large-scale simulation of collective behavior and policy evaluation, but remain highly sensitive to data bias and are difficult to rigorously validate due to complex emergent dynamics. In physical and natural sciences, WMs serve as physically constrained simulators for long-horizon prediction and planning under uncertainty, yet continue to face some essential challenges related to physical fidelity, generalization beyond training regimes, and interpretability.

3.3.1. Social Science and Socioeconomic Systems

A growing body of work studies WMs for social simulation and planning. Social WMs formalize social systems as learnable models of collective dynamics and agent interactions [214], which has been extended to WM-augmented mechanism design for policy learning and economic decision-making [64]. SocioVerse [215] scales social simulation by combining LLM-based agents to model population-level behavior and emergent dynamics (e.g., politics, economics and news). Moreover, related efforts apply WMs to specific domains, including financial markets [216], virtual-reality-based behavioral analysis [217], and influence-driven diffusion processes [218]. Methodological advances in multivariate social time series and structured sequence modeling provide foundational tools for scalable social WMs [219].

3.3.2. Physical and Natural Sciences

WMs act as data-driven, physically constrained simulators for modeling systems in natural and physical sciences. In biology, CellFlux enables continuous-time simulation of cellular dynamics via flow matching [70], while ODesign frames biomolecular interaction design as world modeling for generative molecular exploration [220]. In healthcare, image- and state-based WMs support diagnosis and planning, including tumor evolution modeling [74], radiograph world models [65,221], cardiac ultrasound guidance [222], and surgical procedure modeling [223]. Another major direction links world modeling with forecasting and planning in physical systems, such as spatiotemporal forecasting via generative WMs and model-based RL [224], physics-informed time-series modeling [225], dynamics reconstruction from partial observations [226], and modular architectures for compositional reasoning. Some key applications further extend to large-scale scientific domains, including collider physics through foundation-style models [56].

3.4. World Models for Virtual Game Simulation

Virtual game simulation is a natural and historically important application of WMs, as games offer fully observable and interactive environments. While early game-oriented WMs act mainly as auxiliary predictors for RL, recent efforts in large-scale observation-level generative WMs have shifted them toward standalone interactive simulation. Based on the type of predicted observations, we group existing methods into 2D pixel-level (Section 3.4.1) and 3D mesh-level (Section 3.4.2), as shown in Figure 11(b). Pixel-level prediction [47] offers flexible and visually realistic simulation and enables scalable interactive gameplay, but often struggles with long-horizon consistency and faithful game dynamics. In contrast, 3D mesh-level WMs [227] use stronger object permanence, structural and spatial priors that improve immersion and physical grounding, yet face challenges in scalability and real-time interaction. From a broader perspective, 3D mesh-level WMs are better viewed as complementary to 2D pixel-based ones rather than direct replacements. Emerging trends suggest hybrid architectures that combine 3D structure with neural rendering or generative visual models, aiming to balance geometric priors with generative flexibility.

3.4.1. 2D Pixel-Level Observation Prediction

Most existing game WMs focus on predicting future observations directly at the pixel level, conditioned on past visual frames and agent actions. Representative examples include DIAMOND, which enables accurate multi-step Atari rollouts via diffusion-based dynamics modeling [47], and GameNGen, which supports real-time interactive gameplay without explicit simulators [228]. Beyond arcade-style settings, pixel-level observation prediction has been extended to more open-ended and expressive virtual environments. Oasis [229] explores sequence-based universe modeling with multi-step visual coherence, while AnimeGamer [68] demonstrates stylized, infinite-life simulation tailored to specific aesthetic domains. These works reflect a shift from task-centric simulation toward more general-purpose and generative world modeling. More recently, this branch has begun to converge with the paradigm of foundation-style interactive models. Matrix-Game and Matrix-Game 2.0 [230,231] aim to unify perception, dynamics, and interaction within a single real-time generative framework, while RealPlay [232] and GameFactory [213] emphasize embodiment, generative game creation, and multi-step consistency.

3.4.2. 3D Mesh-Level Observation Prediction

While pixel-level observation prediction dominates current game WM research, there is growing interest in models that operate on explicit 3D geometric representations, such as meshes or structured scene representations. HunyuanWorld 1.0 exemplifies this trend by generating explorable and interactive 3D worlds from text or images, bridging large-scale 3D content generation with WM learning [227]. Matrix-3D advances this paradigm by enabling omnidirectional, explorable 3D world generation with globally consistent spatial structure [233]. As foundation models increasingly integrate multimodal perception and action, 3D mesh-level observation prediction will be important for enabling physically grounded and semantically coherent virtual worlds.

3.5. World Models for GUI-Based Agents

WMs significantly enhance graphical user interface (GUI)-based agents by enabling predictive simulation of interface dynamics, allowing agents to plan, evaluate alternatives, and assess actions before execution in stochastic environments. Unlike physical or game domains [68], these environments are highly stochastic and governed by complex symbolic and visual dynamics, making explicit modeling of environment transitions crucial for long-horizon decision-making [41]. We categorize GUI-based agents by their primary interaction environments, distinguishing web agents from operating system and desktop agents. As presented in Figure 11(c), model-based simulation enables foresighted planning, alternative exploration, and error correction for both web and desktop agents, but relies on accurate prediction of complex visual and symbolic transitions. While iterative search and reasoning-integrated WMs improve multi-step decision-making, prediction errors can accumulate under partial observability and non-stationary interfaces.

Web Agents: Web navigation poses unique challenges due to large state spaces and delayed rewards. WebDreamer exploits LLMs as implicit web WMs and demonstrates model-based planning via simulated rollouts [234], while complementary work explicitly learns web environment dynamics to guide decision-making [41]. Recent methods emphasize iterative improvement and search-based planning, including WebSynthesis, which combines world models with MCTS for efficient user interface trajectory synthesis [235], WebEvolver, which coevolves agents and WMs through self-play [236], and WALL-E 2.0, which aligns WMs with symbolic constraints to improve planning reliability [21]. Several approaches explore unified architectures that tightly integrate reasoning, acting, and simulation. SimuRA [237] introduces an LLM-based simulative reasoning framework that enables goal-directed planning via internal rollouts, while Dyna-Think [238] further formalizes this integration by jointly optimizing reasoning, action, and WM simulation to improve multi-step execution performance.

Operating Systems (OS) & Desktop Agents: Several works equip agents with internal simulators of operating systems or desktop applications. NeuralOS models OS-level dynamics for anticipatory planning without execution [239], while ViMo builds generative visual WMs for mobile and desktop applications to support foresighted GUI interaction [240]. FPWC further combines WM simulation with code execution for model-driven device control [241]. Retrieval-augmented approaches enhance fidelity and generalization, including R-WoM [242], which integrates external retrieval into prediction, and WKM [69], which learns world knowledge for planning across tasks. They underscore the value of combining latent dynamics with explicit knowledge.

3.6. Interpretable and Trustworthy World Models

Beyond empirical success, recent studies scrutinize the theoretical foundations of WMs, focusing on their emergent causal structures and safety guarantees in critical tasks. In this section, we categorize these efforts into two directions: interpretability and safety-oriented trustworthiness. Interpretable and trustworthy WMs reveal a core tension between predictive accuracy and causal fidelity. Interpretability studies show that structured internal models can emerge for long-horizon generalization, yet strong prediction often relies on shortcut correlations, making true causal verification difficult. Safety-oriented work finds that scaling improves stability but does not prevent failures under distribution shift, adversarial perturbation, or partial observability. While strong safety and interpretability guarantees enhance reliability, they often reduce the scalability of WMs.

Interpretability: Some theoretical research suggests that internal WMs are necessary for generalizing to long-horizon goals beyond inductive biases [243]. Although transformers can learn representations aligned with causal structure [244,245], high predictive accuracy does not guarantee genuine internal simulation, as models may rely on shortcut correlations that fail under intervention [246]. Notably, this phenomenon has motivated mechanistic and causal probing to verify whether internal states encode true open-world geometry rather than superficial patterns [247,248].

To develop such genuine internal simulations, extensive environmental interaction is required. However, real-world data collection is prohibitively costly, and current Sim-to-Real approaches [249,250,251] struggle to achieve scalable, high-quality modeling. To address this limitation, recent efforts [252,253,254] employ self-evolutionary learning by introducing internal mechanisms such as self-verification [255,256], self-reward [257], and self-play [258,259]. These mechanisms enable WMs to maintain stable, continuous optimization in environments without external supervision, relying solely on internal predictive capabilities and self-updating mechanisms [260,261]. This shift reflects a gradual transition from dependence on empirical phenomena toward constructing reproducible, explainable, and mechanistic paradigms. Additionally, the successful application of self-evolutionary learning across diverse downstream tasks demonstrates its capacity to impose observable constraints on the stability of closed-loop systems [262,263]. Both theoretically and empirically, self-evolutionary learning supports the effectiveness and robustness of constraining strategy divergence through internal predictive signals, thereby providing structural and mechanistic foundations for WMs as a basis for stable intelligent evolution [4].

Safety-Oriented Trustworthiness: The practical deployment of WMs is further limited by stability and observability constraints. Although scaling laws indicate that increased capacity improves representation stability [264], it cannot ensure robustness to distribution shifts or adversarial perturbations, nor does it overcome theoretical limits in partially observable settings [265]. From a safety perspective, incorrect WM assumptions can bias information acquisition, leading to unsafe downstream decisions [15]. Thus, empirical performance should be accompanied by formal verification to mitigate risks in safety-critical environments.

3.7. Limitations of WMs in Downstream Applications

While WMs have achieved promising gains on downstream applications (e.g., robotics [88,91] and autonomous driving [201]), significant challenges remain. First, the training objectives of WMs primarily focus on representation metrics such as reconstruction or generation, whereas downstream applications emphasize decision quality and policy learning effectiveness. This inherent misalignment between objectives creates a difficult trade-off between “world knowledge acquired through pretraining” and “downstream performance enhancement”. Additionally, existing WMs generally lack explicit modeling of causal structures, increasing susceptibility to error accumulation and prediction drift during closed-loop interactions and long-horizon reasoning. These issues can trigger planning and decision instability while introducing the risk of unexplained failures. Moreover, WMs are constrained by the computational and latency costs of inference and planning, preventing direct deployment on edge devices such as robots or in-vehicle systems.

4. Benchmark of World Models

Establishing comprehensive benchmarks plays a vital role in facilitating progress in WMs. In this section, we systematically analyze existing benchmark datasets, evaluation metrics, simulation platforms, and comparative performance.

4.1. Benchmark Datasets & Evaluation Metrics

WMs are typically pretrained on large-scale, broad, weakly supervised videos and interaction traces, and then adapted to downstream tasks for prediction and control. We first introduce pretrained video benchmarks and commonly used datasets for downstream tasks, along with their corresponding evaluation metrics. As shown in Table 7, common WM benchmarks can be categorized based on their functional roles into simulation, planning, and decision-making. We then analyze the weaknesses of existing benchmarks and propose three new general metrics to measure the generalization, causal reasoning, and long-horizon consistency of WMs, thereby guiding their future development and design.

4.1.1. Pretrained Video Benchmarks

For pretraining on text-video pairs, WebVid-10M [266] provides 10M web video-caption pairs scraped from stock-footage sites, and Panda-70M [267] further scales supervision by automatically generating captions for 70M video clips. Ego4D [268] adds long-horizon egocentric videos with rich annotations, while HowTo100M [269] supplies large narrated instructional videos for weakly-supervised text-video pretraining. The core metrics for video-text pretraining benchmarks include text-video alignment (e.g., Recall@K, Median Rank (MedR) and CLIPSim) and visual quality (e.g., Fréchet Video Distance (FVD)) [267,269]. To tackle the limitations of single-modal approaches and evaluate WMs through multiple pathways, Worldscore [270] uses a multi-step next-scene protocol for simulation, reporting aggregate and component metrics spanning controllability, quality, and dynamics. The core metrics of Worldscore include worldscore-static (WS-S), worldscore-dynamic (WS-D), camera controllability (CCon), object controllability (OCon), content alignment (CAli), 3D consistency (3DCo), photometric consistency (PCon), style consistency (SCon), subjective quality (SQua), motion accuracy (MAcc), motion magnitude (MMag), and motion smoothness (MSmo).

4.1.2. Benchmarks on Downstream Tasks

1) Robotics: In manipulation, Open X-Embodiment [271] aggregates cross-robot trajectories to train generalist policies, typically evaluated by success rate (SR). For navigation tasks, Room-Across-Room [272] tests robotic agents on natural language instruction following, with performance evaluated by SR, navigation error (NE), and success weighted by path length (PL). EWMBench [273] complements physical benchmarks by measuring scene consistency (SceneC) and symmetric hausdorff distance (SHD). Meta-World [274] is a standard multi-task testing platform used to evaluate the effectiveness of policy learning, whose core metric is SR.

2) Autonomous Driving: NuScenes [275] is a widely used benchmark dataset for perception and prediction in autonomous driving, utilizing metrics such as mean average precision (mAP), the nuScenes detection score (NDS), and average multi-object tracking accuracy (AMOTA). To assess action fidelity, ACT Bench [276] evaluates the consistency between executed and target trajectories via average displacement error (ADE) and final displacement error (FDE).

3) Science: JUMP Cell Painting [277] provides hyperscale high-content imaging for cellular morphology and phenotypic drug discovery, with metrics including Fréchet Inception Distance (FID). In the medical domain, HCC-TACE-Seg [278] serves as a benchmark for action prediction, where models are evaluated against ground-truth using metrics such as AP, F1-score, Jaccard index (JI), and recall.

4) Game Simulation: The Arcade Learning Environment (ALE) dataset [279] evaluates Atari 2600 agents using median scores (MedS) and normalized scores (NS). The MineRL dataset [280] enhances RL from large-scale demonstrations in Minecraft, where the evaluation metric is episode return.

5) GUI-Based Agents: OSWorld [281] evaluates multimodal agents on open-ended desktop, using execution-based SR as the primary metric. WindowsAgentArena [282] assesses GUI-based agents in a reproducible Windows environment, focusing on SR across multi-step execution tasks.

4.1.3. Designing General Metrics for World Models

As is well known, a universal WM should possess cross-domain generalization capabilities, causal reasoning abilities, and long-horizon consistency. While existing datasets measure different aspects of a model’s performance, they still fail to provide a unified evaluation of generalization, causal reasoning, and long-horizon consistency. Therefore, in this subsection, we propose three new mathematical metrics to assess these capabilities and support the foundational development and future design directions of unified WMs.

• Generalization: Given the world model ${\pi }_{\theta }(·)$ and the task metric $\Gamma (·)$, we express the metric $\mathcal{G}$ for measuring cross-domain generalization capability as follows:

  $$\begin{array}{c}\hfill \mathcal{G}={\mathbb{E}}_{\mathbf{x}\sim {\mathcal{P}}_{\mathrm{tr}}}\left[\Gamma ({\pi }_{\theta },\mathbf{x})\right]-{\mathbb{E}}_{{\mathbf{x}}^{\prime }\sim {\mathcal{P}}_{\mathrm{te}}}\left[\Gamma ({\pi }_{\theta },{\mathbf{x}}^{\prime })\right],\end{array}$$

  (9)

  where $\Gamma ({\pi }_{\theta },\mathbf{x})$ and $\Gamma ({\pi }_{\theta },{\mathbf{x}}^{\prime })$ represent the task metrics (e.g., success rate or accuracy) when the WM ${\pi }_{\theta }(·)$ is evaluated on the training sample $\mathbf{x}\sim {\mathcal{P}}_{\mathrm{tr}}$ and the testing sample ${\mathbf{x}}^{\prime }\sim {\mathcal{P}}_{\mathrm{te}}$. ${\mathcal{P}}_{\mathrm{tr}}$ and ${\mathcal{P}}_{\mathrm{te}}$ denote the distributions of the training and testing datasets, and they exhibit a significant distribution discrepancy (${\mathcal{P}}_{\mathrm{te}}\ne {\mathcal{P}}_{\mathrm{tr}}$).
• Causal Reasoning: Given a set of interventions $\mathcal{I}$ and a distance measure $\mathcal{D}(·)$, inspired by Pearl’s do-calculus [283], we adopt a counterfactual intervention operator $do\left(i\right)$ ($i\sim \mathcal{I}$) to measure causal reasoning capability $\mathcal{C}$:

  $$\begin{array}{c}\hfill \mathcal{C}={\mathbb{E}}_{i\sim \mathcal{I}}\left[\mathcal{D}\left(\mathbf{y},{\mathbf{y}}^{\prime }\right)\right],\end{array}$$

  (10)

  where $\mathbf{y}\sim {\pi }_{\theta }\left(\mathbf{y}\right|\mathbf{x})$ and ${\mathbf{y}}^{\prime }\sim {\pi }_{\theta }\left(\mathbf{y}\right|do(\mathbf{x}=i))$ denote the expected outputs when the inputs are $\mathbf{x}$ and $do(\mathbf{x}=i)$.
• Long-Horizon Consistency: For multi-step execution tasks, we compare the incremental deviation between the actual trajectory and the imagined trajectory under identical conditions and policies to evaluate the long-horizon consistency $\mathcal{H}$ of existing foundational WMs:

  $$\begin{array}{c}\hfill \mathcal{H}={\displaystyle \frac{1}{T}}\sum _{t=1}^T\mathcal{D}\left({\zeta }_t,{\widehat{\zeta }}_t\right),\end{array}$$

  (11)

  where ${\zeta }_t$ and ${\widehat{\zeta }}_t$ denote the ground-truth and predicted trajectory (generated based on Eq. (1)) at time t.

4.2. Physics Engines & Simulation Platforms

WMs are typically trained and evaluated in closed-loop interactions, where actions influence future observations, and physics engines and simulation platforms therefore serve as controllable data generators and evaluation harnesses. Given the high cost and safety risks associated with real-world data collection, high-fidelity simulation has become essential infrastructure for training and validating WMs. As demonstrated in Table 8, we review existing physics engines and simulation platforms used to evaluate WMs.

Simulation: Effective simulation relies on physics engines capable of replicating real-world properties. Bullet1 is an open-source physics engine supporting multi-physics simulation, widely applied across various fields. Simbody2 emphasizes high-precision multibody dynamics and is particularly relevant for biomechanics-oriented embodied modeling. Additionally, PhysX3 provides an industry-adopted physics foundation. Despite their maturity, recent documentation indicates that large-scale simulations on GPUs and cross-platform determinism still remain challenging.

Simulation & Planning: Constructing WMs necessitates not only high-fidelity physics simulation but also planning capabilities. Early CPU-oriented robotics simulators such as Gazebo4 and Webots5 provide mature robotic workflows, but their throughput can be limiting when large-scale interactive experiences are required. Meanwhile, Omniverse6 offers advanced rendering capabilities that improve visual realism. However, these platforms struggle to ensure reliable multi-agent evaluations in complex environments and also incur significant computational overhead from perception and dense contacts during the planning process.

Simulation & Planning & Decision-Making: To support closed-loop decision-making, modern simulation platforms have further enhanced stability and begun supporting GPUs to accelerate simulation. PyBullet7 is widely used in early deep RL due to its accessibility and ease of integration; however, complex contact-rich tasks and large-scale parallel training can expose limitations in performance and stability. For model-based RL, stable multibody dynamics and reliable contact handling are essential. MuJoCo8 is explicitly designed to meet these requirements. Additionally, Isaac Gym9 demonstrates GPU-resident closed-loop simulation tightly coupled with neural network training. Furthermore, Genesis10 is designed specifically for embodied AI and has achieved superior speed. However, since closed-loop decision-making demands high-frequency interaction, modern platforms still need to grapple with the trade-off between real-time performance and physical fidelity.

Discussion: The physics engines and simulation platforms provide a controlled environment for generating and evaluating data for WMs. However, large-scale simulations remain challenging in terms of GPU acceleration and cross-platform determinism, leading to insufficient robustness in result reproducibility and rigorous comparisons. Consequently, the validity of evaluations for complex tasks is severely compromised by computational overhead. Computational demands arising from multi-agent coordination, high-frequency perception-rendering, and contact-rich interactions often result in slow solution speeds. Consequently, when addressing closed-loop decision-making tasks, a trade-off persists between contact modeling accuracy, numerical stability, and large-scale parallel efficiency.

4.3. Performance Comparison

Table 9 compares the performance of different WMs on the WorldScore benchmark. Additional performance rankings are tracked and updated live through WorldScore-Leaderboard. As shown in Table 9, TeleWorld [284] achieves the strongest and most balanced performance, leading in key metrics including WS-S, WS-D, 3D consistency (3DCo), photometric consistency (PCon), semantic consistency (SCon), and subjective quality (SQua), indicating superior visual fidelity and temporal coherence. 3D-based methods such as Text2Room [110] and WonderJourney [98] substantially excel in controllability-related metrics, particularly camera controllability (CCon), but show near-zero scores on motion-related metrics, highlighting their limitations in dynamic scene modeling. In contrast, video-based models (e.g., Gen-311 and T2V-Turbo [285]) demonstrate better motion accuracy and smoothness, yet underperform in fine-grained controllability. Methods such as SceneScape [109] and 4D-fy [96] exhibit consistently low performance across most metrics, reflecting challenges in both consistency and dynamics. Overall, the comparison results reveal a clear trade-off between controllability, visual consistency, and motion realism across existing approaches.

5. Challenges & Future Directions

5.1. Scientific Modeling

Existing approaches, such as Dreamer [234], Dyn-O [45], and NWM [166], often focus on fitting the distribution of observational data to improve visual fidelity, controllability, or motion realism. As a result, they can generate visually plausible rollouts while violating physical laws. As shown in Table 3, future research should focus on developing scientific modeling that not only provides predictive capabilities but also offers strong explanatory power. In scientific modeling, knowledge is expressed through explicit symbolic forms, such as equations and formal relations independent of specific datasets, experiments, or observers. Progress is achieved by incrementally extending a shared theoretical framework rather than modifying opaque and tightly coupled neural parameters. Such knowledge is inherently inspectable, enabling verification through proof checking, experimental replication, and edge-case analysis. Moreover, scientific modeling explicitly specifies the conditions under which it holds, making both applicability and failure modes transparent and principled. A promising direction is to integrate symbolic and neural components, impose hard physical constraints (e.g., conservation laws), and develop protocols that reward causal discovery and falsifiability.

5.2. Long-Horizon Consistency & Causal Reasoning

Maintaining consistency over long temporal horizons remains a major obstacle for WMs. Errors in prediction often accumulate, leading to visual drift, broken physical interactions, or unrealistic future states. Moreover, many WMs, such as Wan [104], WonderJourney [98], and 4D-fy [96], rely on correlational patterns rather than causal understanding, limiting their reliability in closed-loop or multi-step rollouts. Future advances in hierarchical temporal modeling and causal representation learning may improve long-horizon stability. Incorporating explicit mechanisms for state abstraction, memory, and causal intervention could enable WMs to reason beyond short-term correlations. Benchmarks that emphasize multi-step prediction and counterfactual reasoning will further drive progress in this direction.

5.3. Grounding in Physical and Semantic Constraints

Despite impressive visual synthesis, many WMs [2,18,286] lack grounding in physical laws and semantic structure, leading to implausible motions, inconsistent object interactions, and semantically incoherent scene evolution. This limits their applicability to real-world decision-making and robotics. Future research directions include integrating physics-informed priors, differentiable simulators, or symbolic knowledge, as well as combining neural representations with structured semantic graphs or object-centric models to improve physical plausibility and interpretability. Furthermore, bridging neural and symbolic approaches remains a promising path toward grounded world modeling.

5.4. Generalization & Scalability in Real-World

Most current WMs [1,3,7] are trained and evaluated within narrow domains, which raises concerns about their generalization across environments, tasks, and embodiments. Moreover, the high computational and data requirements of training large-scale WMs hinder scalability and real-world deployment. To tackle these challenges, future research may explore foundation WMs trained on diverse, multimodal interaction data, enabling broad generalization. In addition, data-efficient learning, parameter-efficient fine-tuning, and continual learning strategies are likely to be critical for scalability. Emphasis on real-world deployment will further push WMs toward practical AI systems.

6. Conclusions

This survey presents a systematic review of WMs in artificial intelligence. We introduce a unified definition and categorize existing methods into four paradigms: observation-level generative, latent-space, RL-based, and object-centric WMs, and review their applications across robotics, autonomous driving, scientific discovery, virtual game simulation, GUI-based agents, as well as interpretability and trustworthiness. We also summarize benchmark datasets, evaluation metrics, simulation platforms, and comparative results. Moreover, we identify key challenges limiting the deployment and generalization of WMs, including scientific modeling, long-horizon consistency, controllability, robustness, evaluation limitations, and transfer across tasks and embodiments. We have also discussed potential directions that emphasize integrated modeling, scalable training, principled benchmarking, and real-world validation. This survey serves as a foundation for advancing more generalizable, and trustworthy WMs. Looking ahead, WMs should move beyond scaling predictive accuracy toward establishing physical consistency and explanatory structure, with internal representations examinable against physical laws and invariances. Rather than manually encoding physical laws, WMs are trained on large-scale data to autonomously learn representations and predictive capabilities of the physical world through self-evolutionary learning. Advancing toward this goal requires tighter integration of learning, dynamics, and causality, shifting world modeling from large-scale statistical interpolation toward a scientific modeling framework.