Learning the Physical World from Videos: A Prospective Study on World Models

1. Introduction

World models refer to computational frameworks in which agents understand and predict external dynamics through internal environment simulations, thereby supporting perception, reasoning, and decision-making [1]. As a key pathway toward artificial general intelligence (AGI), world models aim to capture the physical laws, social interaction mechanisms, and environmental uncertainties inherent in the real world, enabling agents to perform efficient planning and adaptive behaviors in complex and dynamic scenarios. In practical applications, world models demonstrate broad potential: in robotics [2], they facilitate agents’ understanding and simulation of interactions with the environment; in autonomous driving [3,4,5], they enable the prediction of future traffic scenarios and potential risks; and in game AI [6,7], they support long-horizon strategy reasoning and multi-step planning.

Figure 1. This paper reviews five key challenges and three promising paradigms of learning the physical world models.

Figure 1. This paper reviews five key challenges and three promising paradigms of learning the physical world models.

Over the past decade, with the rapid advancement of computational power, data scale, and model performance, research on world models has achieved significant progress. In 2018, Ha and Schmidhuber [1] proposed a world model framework that integrates a Variational Autoencoder (VAE) with a Recurrent Neural Network (RNN) [8], marking a paradigm shift from traditional symbolic representations to data-driven approaches. Subsequently, the Dreamer series of methods [9,10,11] and Joint Embedding Predictive Architectures (JEPA) [12] introduced latent state modeling and predictive learning, demonstrating the feasibility and advantages of latent dynamics for planning and decision-making tasks. Benefiting from the rise of large-scale models, world models have entered a phase of rapid iteration since 2024, accompanied by a surge in related research publications. Representative systems such as Sora2 [13] and Genie3 [14] have significantly advanced the practical realization of learning world models directly from video data, further highlighting the potential of this research direction.

Video, as the most intuitive and readily accessible medium for capturing the dynamics of the physical world, has increasingly emerged as a natural substrate for constructing world models. It inherently encodes temporal continuity, motion patterns, and rich contextual information, thereby providing an indispensable foundation for learning structured representations of the environment. However, video is fundamentally a 2D projection of reality—it captures only superficial visual phenomena and lacks explicit representations of three-dimensional structure, physical properties (e.g., mass, friction, elasticity), contact interactions, and underlying causal mechanisms. This intrinsic limitation causes video-driven world models to remain largely confined to the level of visual pattern fitting, giving rise to the so-called “pixel-to-physics” gap [15]. Nevertheless, compared to real-world data acquisition, video data is safer, more cost-effective, and can support virtually unlimited experimentation within simulated environments. Bridging this gap is therefore a critical research direction toward developing physically grounded and practically deployable world models.

Existing surveys have provided valuable insights into the field of world models, yet they are often limited to specific application domains, generative tasks, or high-level framework overviews, and generally lack a systematic analysis from the perspective of physical consistency [16]. For instance, the surveys by Tu et al. [3], Feng et al. [4], and Guan et al. [5] on autonomous driving world models emphasize applications in perception, prediction, and planning, but treat video merely as an auxiliary input and do not specifically address biases arising from missing causal information. Ding et al. [17] and Zhu et al. [18] explore, at a broader level, the potential of extending generative models toward generalized world models, yet their focus lies primarily on delineating the boundaries of generation and simulation, overlooking in-depth analyses of physical modeling mechanisms. Some studies concentrate on physics simulation in embodied intelligence [19,20], but these are constrained to specific simulated environments or emphasize functional outcomes rather than the underlying gaps. With the rapid evolution of world model research, there is an urgent need for a survey that systematically reviews video-driven world model paradigms and key challenges, with physical consistency as the guiding principle.

Motivated by the above, this work presents a systematic survey of video-driven physical world models from a problem-oriented perspective, explicitly excluding directions that deviate from the goal of modeling the physical world (e.g., pure video generation or 3D/4D reconstruction [21]). Specifically, our discussion is organized along three key dimensions:

• Pixel–Physics Challenges: We distill five core challenges—continuity, controllability, generalization, lightweight, and universality—and systematically summarize the sub-problems and representative solutions for each challenge through the lens of physical consistency.
• Three Paradigms of Physical World Model: Existing approaches toward physically grounded modeling can be broadly categorized into three classes: prior injection, dynamic–static decoupling, and hierarchical abstraction.
• Future Directions: We identify five major open problems for future research and provide a systematic discussion on industrial deployment and current safety issues.

Through this systematic organization, the survey aims to illuminate the key challenges and opportunities in bridging the gap from pixels to physics, providing a clear research framework and guidance for developing the next generation of world models with true physical understanding.

2. Challenges of Learning from Video

Videos naturally encode spatiotemporal continuity, object interactions, event evolution, and latent causal chains, making them highly valuable for capturing physical and dynamic processes. Large-scale video generation models even exhibit an implicit adherence to real-world constraints to some extent. However, as 2D projections, videos discard critical 3D geometry, object properties, and causal mechanisms, causing world models trained on them to overfit superficial visual patterns rather than underlying physical principles. This, in turn, gives rise to a series of core challenges in consistency, controllability, generalization, physical grounding, efficiency, and universality, as illustrated in Figure 2 and Table 1.

2.1. Physical Continuity

2.1.1. Temporal Continuity

The gap in temporal continuity arises because videos capture only discrete snapshots and lack explicit causal signals, causing autoregressive world models to accumulate prediction errors and break continuous state evolution. Short temporal horizons and stepwise error compounding further prevent generating long-range physically consistent sequences [22]. Existing solutions can be summarized as follows:

Autoregressive improvements. Conditioning on preceding frames (single or multiple) provides a simple yet efficient means of ensuring continuity and has been widely adopted in recent works [23,24,25]. Several improvements have been proposed: EnerVerse [26] introduces a sparse keyframe memory mechanism that predicts future video blocks based on block-level context, effectively avoiding the redundant storage and error propagation associated with continuous memory while preserving long-term dependencies. EVA [27], on the other hand, incorporates a reflection-based conditioning mechanism into a block-level autoregressive framework, enabling the model to adaptively extend video length. Yume [28] combines block-level autoregression with FramePack-based historical context compression and hierarchical sampling, achieving theoretically unlimited interactive generation and effectively mitigating temporal discontinuities in long sequences.

Diffusion schedule optimization. Most diffusion-based approaches typically assume that all frames share the same noise level [29,30]. Recent studies, however, demonstrate that optimizing noise schedules—such as diffusion forcing [31,32]—facilitates autoregressive long-horizon generation [33]. GEM [34] maintains temporal continuity in long video generation by denoising in multiple stages, conditioning on previous frames, and enforcing progressively increasing noise levels over time during training.

Conditional constraints. Imposing physical structural constraints on world models substantially enhances temporal physical continuity. Pathdreamer [35] employs a hierarchical two-stage architecture, first generating semantic and depth maps as structural contexts before predicting future frames conditioned on them. This layered strategy improves long-term fidelity since predicting structural representations grounded in physical geometry is inherently more continuous than directly generating RGB pixels. PlayerOne [36] adopts a joint reconstruction framework that simultaneously models 4D scenes and video frames, thereby ensuring physically consistent scene continuity. VRAG [23] incorporates global state information (e.g., character coordinates and poses) and retrieves relevant frames from a historical buffer, allowing the model to leverage past contexts and spatial awareness for improved temporal continuity.

Optimization-level. Regularization strategies and loss-based enforcement provide direct optimization routes to penalize physical discontinuities in learned representations. SSD [37] employs a state-space modeling framework to efficiently handle long-horizon sequential information, enabling the extraction of rich long-term contextual dependencies. Vid2World [38] modifies both architecture and training objectives—particularly through adjustments to temporal attention layers and temporal convolution weight sharing—to enable causal generation and autoregressive capability in video diffusion models, ensuring predictions depend only on past physical states. SGF [39] enforces temporal continuity by minimizing the mean squared error between consecutive observations while introducing variance–covariance regularization to prevent representation collapse.

Table 2. Summary of Works on Physical Continuity Challenges. Modalities: indicates video or multi-frame; denotes single-frame or image; refers to latent representation; to spatial representation; to text modality; to action representation; to camera (pose) information; to depth feature; to neural signal; to object-level feature; to physical information; and to historical memory feature. Evaluations: V assesses visual quality, generation, prediction, and control in downstream tasks, encompassing qualitative evaluations. R evaluates robotic tasks in simulation, while $\widehat{\mathbf{R}}$ includes real-robot experiments. P measures physical understanding and perception. C evaluates planning and decision-making in games, tasks, and navigation.

Work	Venue	Main Solution	Method	Input / Output	Conditions	Evals
Temporal Consistency
EnerVerse [26]	NeurIPS’25	Autoregression Improvements	Sparse Chunks	/		V$\widehat{\mathrm{R}}$
EVA [27]	arXiv’25		Reflection	/		V$\widehat{\mathrm{R}}$
Yume [28]	arXiv’25		Framepack	/		V
SAMPO [24]	NeurIPS’25		Scale-Wise	/	None	V$\widehat{\mathrm{R}}$
GEM [34]	CVPR’25	Diffusion Schedules	Increasing Noise	/		V
Diamond [33]	NeurIPS’24		Adaptive Noise	/	None	V
Epona [32]	ICCV’25		Diffusion Forcing	/		VC
Pathdreamer [35]	ICCV’21	Condition Constraints	HR. Modeling	/		VC
PlayerOne [36]	NeurIPS’25		Rec. Constraints	/		V
VRAG [23]	NeurIPS’25		Global Constraints	/	None	CV$\widehat{\mathrm{R}}$
Vid2World [38]	ICLR’26	Optimization Level	Loss-based	/		V
SSD [37]	NeurIPS’25		State-space	/	None	C
SGF [39]	ICLR’25		Regularization	/	None	C
Spatial Consistency
RoboScape [40]	NeurIPS’25	Implicit Alignment	HR. Modeling	/	None	VR
WorldGrow [41]	AAAI’26		Block Inpainting	/		V
WVD [42]	CVPR’25	Explicit Alignment	Spatial Joint Modeling	/	None	V
FlashWorld [43]	ICLR’26		Dual-mode Pre-training	/		V
Geom. Forcing [44]	ICLR’26		Rep. Alignment	/	None	V
InfiniCube [45]	ICCV’25		HR. Constraints	/		V
MindJourney [46]	NeurIPS’25		Language Guidance	/		VCP
UniFuture [47]	ICRA’26		Multi-modal	/		V
Edeline [48]	NeurIPS’25		Mem. Enhancement	/	None	C
Ctrl-World [49]	ICLR’26		Space Constraints	/		V
Spatial-Mem [50]	NeurIPS’25		Semantic Alignment	/		V
WorldMEM [51]	NeurIPS’25	Memory Mechanism	Memory Bank	/		V
Voyager(LLM) [52]	TMLR’24		Skill Library	/		C
SSM-World [53]	ICCV’25		State-Space Models	/		V
Identity Consistency
Loci-v1 [54]	ICLR’23	Occlusion	Imagination Tracking	/		C
SAVi++ [55]	NeurIPS’22	Tracking	Identity Tracking	/	None	P
ForeDiff [56]	arXiv’25	Anchors	Arch. Decoupling	/		V$\widehat{\mathrm{R}}$

Table 2. Summary of Works on Physical Continuity Challenges. Modalities: indicates video or multi-frame; denotes single-frame or image; refers to latent representation; to spatial representation; to text modality; to action representation; to camera (pose) information; to depth feature; to neural signal; to object-level feature; to physical information; and to historical memory feature. Evaluations: V assesses visual quality, generation, prediction, and control in downstream tasks, encompassing qualitative evaluations. R evaluates robotic tasks in simulation, while $\widehat{\mathbf{R}}$ includes real-robot experiments. P measures physical understanding and perception. C evaluates planning and decision-making in games, tasks, and navigation.

Work	Venue	Main Solution	Method	Input / Output	Conditions	Evals
Temporal Consistency
EnerVerse [26]	NeurIPS’25	Autoregression Improvements	Sparse Chunks	/		V$\widehat{\mathrm{R}}$
EVA [27]	arXiv’25		Reflection	/		V$\widehat{\mathrm{R}}$
Yume [28]	arXiv’25		Framepack	/		V
SAMPO [24]	NeurIPS’25		Scale-Wise	/	None	V$\widehat{\mathrm{R}}$
GEM [34]	CVPR’25	Diffusion Schedules	Increasing Noise	/		V
Diamond [33]	NeurIPS’24		Adaptive Noise	/	None	V
Epona [32]	ICCV’25		Diffusion Forcing	/		VC
Pathdreamer [35]	ICCV’21	Condition Constraints	HR. Modeling	/		VC
PlayerOne [36]	NeurIPS’25		Rec. Constraints	/		V
VRAG [23]	NeurIPS’25		Global Constraints	/	None	CV$\widehat{\mathrm{R}}$
Vid2World [38]	ICLR’26	Optimization Level	Loss-based	/		V
SSD [37]	NeurIPS’25		State-space	/	None	C
SGF [39]	ICLR’25		Regularization	/	None	C
Spatial Consistency
RoboScape [40]	NeurIPS’25	Implicit Alignment	HR. Modeling	/	None	VR
WorldGrow [41]	AAAI’26		Block Inpainting	/		V
WVD [42]	CVPR’25	Explicit Alignment	Spatial Joint Modeling	/	None	V
FlashWorld [43]	ICLR’26		Dual-mode Pre-training	/		V
Geom. Forcing [44]	ICLR’26		Rep. Alignment	/	None	V
InfiniCube [45]	ICCV’25		HR. Constraints	/		V
MindJourney [46]	NeurIPS’25		Language Guidance	/		VCP
UniFuture [47]	ICRA’26		Multi-modal	/		V
Edeline [48]	NeurIPS’25		Mem. Enhancement	/	None	C
Ctrl-World [49]	ICLR’26		Space Constraints	/		V
Spatial-Mem [50]	NeurIPS’25		Semantic Alignment	/		V
WorldMEM [51]	NeurIPS’25	Memory Mechanism	Memory Bank	/		V
Voyager(LLM) [52]	TMLR’24		Skill Library	/		C
SSM-World [53]	ICCV’25		State-Space Models	/		V
Identity Consistency
Loci-v1 [54]	ICLR’23	Occlusion	Imagination Tracking	/		C
SAVi++ [55]	NeurIPS’22	Tracking	Identity Tracking	/	None	P
ForeDiff [56]	arXiv’25	Anchors	Arch. Decoupling	/		V$\widehat{\mathrm{R}}$

2.1.2. Spatial Continuity

Videos are 2D projections that discard crucial 3D geometry, causing world models to miss physically grounded spatial relationships. Human spatial perception uses multi-view associations and long-term memory [44], inspiring recent methods to recover 3D geometry and preserve long-range spatial dependencies through memory mechanisms.

Implicit representation alignment. These methods do not directly parameterize 3D structures but instead encode physical spatial correlations implicitly within the model through cross-modal fusion or attention mechanisms. For instance, EnerVerse [26] introduces cross-view spatial attention, leveraging camera intrinsics/extrinsics and ray direction maps to model view correspondences and enhance holistic multi-view generation. RoboScape [40] jointly learns from RGB and depth, using depth features as geometric constraints to implicitly acquire 3D physical scene priors rather than merely fitting 2D images. Similarly, GEM [34] and DeepVerse [57] integrate depth, semantics, RGB, and dynamic masks to improve spatial physical continuity. GAIA-2 [58], as a surround-view world model, ensures multi-camera spatial continuity by aligning streams via structured conditions such as environmental factors and road semantics, enabling high-resolution spatiotemporally coherent generation.

Explicit representation alignment. These approaches provide explicit physical spatial information to supervise joint distributions, spatial guidance, and alignment. WVD [42] encodes global 3D coordinates into spatial pixels, learning the joint distribution of 2D space and 3D coordinates from 6D (RGB+XYZ) video with explicit geometric supervision. Geometry Forcing [44] aligns intermediate video model representations with features from pretrained geometric foundation models, guiding the model to internalize physical geometric alignment over latent perspectives and scales. Pathdreamer [35] accumulates past observations into a 3D point cloud and reprojects it into 2D space as context, explicitly reasoning about the 3D physical geometry of the next frame. InfiniCube [45] constructs an external voxel-based “3D ground buffer” from videos, providing explicit physical grounding to mitigate spatial drift in long sequences. DSG-World [59] explicitly builds a 3D Gaussian world model from dual-state video observations, introducing dual-segmentation-aware Gaussian fields, pseudo-intermediate symmetric alignment, and cooperative pruning/pasting to maintain geometric integrity under occlusions. Voyager-DM [60] aligns RGB with depth and employs efficient world caching for long-range geometrically consistent 3D reconstruction. Spatial-Mem [50] leverages external geometry-grounded 3D world representations (e.g., point clouds) to filter dynamic regions while explicitly memorizing static spatial structures.

Memory mechanisms. Many approaches incorporate external memory banks or retrieval modules [52] to store historical physical observations, mitigating error accumulation and preserving spatial continuity. WorldMem [51] leverages a memory bank to store past visual and state features, achieving long-term 3D physical continuity through a memory attention mechanism. Spatial-Mem [50], in contrast, maintains sparse keyframes as episodic memory and dynamically expands when new regions appear, thereby enhancing the preservation of spatial relationships. Furthermore, PEWM-3D [61] continuously integrates observations into a shared 3D feature map (e.g., Plücker coordinate embeddings), which is referenced throughout the generation phase to ensure globally continuous spatial coherence. In addition, SSM-World [53] introduces structured memory along the spatial dimension via a block-wise scanning State Space Model (SSM) mechanism, partitioning the global space into controllable units. This design balances temporal memory with spatial physical continuity while incorporating frame-level local attention to enhance generation quality.

2.1.3. Identity Continuity

World models that operate directly in pixel space often fail to capture the physically invariant properties of objects across frames. Since pixel-level representations are highly sensitive to noise, lighting changes, and viewpoint variation, they cannot maintain the stable attributes — such as shape, material, and appearance — that define object identity in the physical world, leading to identity drift in long sequences. Fine-grained object-level modeling is therefore crucial for maintaining physically grounded identity continuity across frames [62]. In this regard, the GEM model [34] introduces “identity embeddings” for objects to eliminate operation ambiguities and adopts customized integration mechanisms for different types of control signals (such as self-motion, object manipulation, and human posture), ensuring strict physical identity continuity between generated content and diverse control signals. Similarly, FOCUS [63] allocates a unique one-hot identity vector $s_{\mathrm{obj}}^t$ to each object, and the object latent extractor generates stable latent representations $s_{\mathrm{obj}}^t$ based on this identity, thereby ensuring identity continuity. Loci-v1 [54] and SAVi++ [55] approach identity continuity by decomposing the scene into multiple object slots. Similarly, ForeDiff’s [56] prediction flow learns during pretraining how to precisely identify and preserve physically grounded identity-related features of objects, such as spatial location, shape, and appearance, from video frames, providing an “identity anchor.” Recently, the unprecedented success of Sora2 [13] in cross-scene multi-view identity continuity demonstrates the immense potential of learning physically consistent representations from videos.

2.2. Controllability

The controllability gap exists because videos passively record events and cannot represent action-conditioned responses, causal interventions, or counterfactuals. Without this, models cannot predict “what if” scenarios, limiting policy exploration and intervention. Current research addresses this limitation by introducing explicit modeling of action–outcome mappings to ensure that environmental states respond to interventions in accordance with physical laws, while also striving to construct interactive world models.

2.2.1. Semantic Control

Since videos lack explicit representations of the semantics underlying scene changes — such as object material properties, force interactions, and spatial affordances — world models struggle to ground high-level semantic instructions in physically consistent outputs. This approach aims to bridge semantic intent and physical scene evolution by integrating structured semantic inputs, text-conditioned guidance, and multimodal semantic alignment mechanisms [64], effectively preventing semantic drift and physically inconsistent generation. Models such as InfiniCube [45], and GEM [34] achieve this by directly injecting structured physical and semantic conditions—such as road semantics, high-definition maps, vehicle bounding boxes, textual prompts, or DINOv2 [65] features—into the generative model. This enables the predictable manifestation of advanced semantic instructions, including multi-agent interaction, multi-camera consistency, dynamic object insertion, weather condition adjustments, and object movement or insertion, while maintaining physical plausibility in the generated scenes.

For finer semantic interaction, Yume [28] utilizes novel sampling strategies and stochastic differential equations to enhance the controllability of text conditions, and achieves physically grounded camera control through quantized camera trajectories. LaDi-WM [66] innovatively designs an interactive diffusion process that dynamically adjusts DINO geometric latent codes and SigLip semantic latent codes, aligning geometry with semantics to accurately model the physically consistent evolution of scene semantics within world dynamics. FlowDreamer [67] adopts 3D scene flow as a universal motion representation, capturing physically grounded semantic deformations of non-rigid objects, and supports fine-grained semantic motion control that respects plausibility. Pathdreamer [35] avoids semantic ambiguity in camera modeling by directly specifying future viewpoint trajectory sequences grounded in spatial geometry. Additionally, NWM [68] implements real-time semantic planning and semantic command compliance in robotic operations using CDiT [69] architectures with semantic constraints, energy function-encoded semantic rules, and action tree-structured instructions, ensuring that semantic control signals translate into physically executable robot behaviors.

2.2.2. Interactivity

Interactivity enables world models not only to passively predict but also to actively respond to interventions and instructions, thereby supporting causal reasoning and goal-directed policy exploration [70]. Existing studies mainly focus on two directions: (I) interaction video generation driven by local trajectories or action signals (emphasizing sequence-level action conditioning, suitable for stepwise prediction in robotics or games); and (II) real-time explorable or 3D world models driven by global signals (emphasizing holistic prompts or multimodal inputs, suitable for immersive simulation).

The first line of research emphasizes conditioning video generation models on low-level control signals (e.g., robot joint actions, game inputs, or reward feedback) to enable stepwise interactive prediction. Representative works include Vid2World [38], AVID [71], iVideoGPT [72], DWS [73], AirScape [74], and UnifoLM-WMA-0 [75], which employ action-injection mechanisms to map user-provided control signals into sequences of future video frames, thereby supporting embodied interactive exploration such as robotic trajectory evolution. These approaches essentially constitute controllable video-driven world models, establishing a tight coupling between low-level actions and environmental dynamics. In addition, some works further focus on multi-task trajectory integration and generalization. For instance, WLA [76] aggregates cross-environment trajectories to achieve continuous dynamic prediction, while the IAFM [77] extracts action signals from large-scale robotic video trajectories to support cross-task and cross-domain interactive simulation.

The second line of research is driven by global conditions (e.g., text, natural language instructions, or multimodal prompts), aiming to construct navigable and explorable spatial world representations (3D or 4D) and extend video generation into embodied, immersive, or panoramic scene simulation. For example, TesserAct [78] employs instruction-driven grid evolution mechanisms for controllable environment generation, whereas PlayerOne [36] models first-person-view (POV) videos to support navigation. LatticeWorld [79] addresses static 3D world modeling by seamlessly integrating with an industrial-grade computer graphics rendering engine. In virtual environments, several studies investigate open-world simulation based on game scenarios: MineWorld [80] and Hunyuan-Game [6,81] are specifically designed for game trajectories, supporting agent-controlled interactive exploration; while Voyager [60], Mirage 2 [82], and Matrix Game [7] further develop real-time explorable game-world modeling. Moreover, some works explicitly incorporate 3D spatial structures into generative models: for instance, YUME [28], Marble [83], and Matrix-3D [84] combine omnidirectional video generation with interactive control, enabling real-time 360° scene exploration driven by external devices.

2.3. Generalization

Although large-scale video data exist, annotated datasets for embodied intelligence and 3D environments remain scarce. Videos capture appearances rather than underlying physical laws, causing overfitting to visual patterns and poor coverage of rare dynamics. Recent work uses data augmentation, improved architectures, and domain generalization to enhance generalization and cross-scene adaptability.

2.3.1. Data

Data Synthesis and Utilization. A typical approach involves fine-tuning generative models to synthesize diverse new data. For example, DREAMGEN [85] employs a video-based world model to generate both familiar and novel tasks across diverse environments, while integrating a latent action model or inverse dynamics model to infer pseudo-action sequences, forming “neural trajectories.” A more advanced framework, Dream to Manipulate [64], introduces a learnable digital twin that combines Gaussian splatting [86] with simulators to generate new environments and action configurations with physically consistent dynamics, thereby expanding the diversity of the training distribution. As shown in Figure 3, the EnerVerse-D data engine [26] integrates world models with 4D Gaussian Splatting (4DGS) to establish a self-reinforcing data generation loop capable of producing high-quality, multi-view video data with geometrically consistent scene reconstruction.

In addition, many studies explore unsupervised or weakly supervised paradigms. GEM [34] leverages large-scale cross-domain data and pseudo-labels to generate depth maps, trajectories, and poses, improving instance-level perception and generalization. FLARE [87] enhances temporal representation learning by leveraging structured representations and action-aware future embeddings derived from unlabeled data. VLMWM [88] employs a dynamic modeling mechanism to automatically generate pseudo-labels, thereby strengthening the model’s supervisory signals for dynamics.

Sim2Real Transfer. Since simulators can explicitly model physical properties such as friction, mass, and contact dynamics, simulation-to-reality transfer offers a principled way to inject physical knowledge into world models. SRCC [89] introduces the Sim2Real Correlation Coefficient to quantify a simulator’s predictive fidelity, optimizing simulation parameters to better align dynamics with real-world performance. SimWorld [90], a simulator-conditioned scene generation engine, integrates scene construction and simulation modules to produce synthetic data and labels with physically accurate properties for world model training. Other methods exploit general semantic alignment to bridge the appearance gap. In addition, generative editability-based approaches, such as Cosmos-transfer [91] and Dreamland [92], leverage multi-conditional multimodal pretraining to achieve photorealistic domain transfer while preserving physical plausibility.

Foundation Model Priors. Cross-scene generalization can be enhanced by leveraging the prior knowledge encoded in pretrained video or multimodal foundation models. For example, UWM [29] and Vidar [93] achieve broad semantic understanding and efficient domain transfer by fine-tuning on only a small amount of domain-specific data. Similarly, Founder [94] maps foundation model representations into the world model latent state space, enabling world models to handle cross-domain scenarios and multimodal tasks. Notably, DINO-WM [95] trains a world model on offline behavioral data using DINOv2-based pretrained visual features, thereby enabling task-agnostic zero-shot planning grounded in physically meaningful visual representations. In addition, Geometry Forcing [44] exploits pretrained 3D foundation models for physically grounded representation learning, alleviating the modeling bottlenecks caused by limited 3D supervision.

2.3.2. Architecture Generalization

One class of approaches introduces structural modeling of visual or geometric inputs using discrete tokens, virtual intermediate states, or hierarchical structures to better capture physically invariant patterns. WorldDreamer [96] learns world dynamics through masked-token prediction, while DSG-World [59] reconciles observational discrepancies via geometric constraints and pseudo-states that encode structure. HMBRL [97] adopts hierarchical models with low-dimensional abstract actions to generalize across complex tasks. Another class emphasizes regularization and robust training to address uncertainty and drift in latent physical representations. MoSim [98] introduces residual flow penalties to avoid physically uncertain regions, while SDE-based [99] and Jacobian-regularized frameworks analyze and correct latent drift errors in physical state representations, thereby improving robustness. A third line of work focuses on multimodal conditioning and policy structures: GenRL [100] achieves task generalization with vision/language prompts, and BPN [101] separates state and task representations with bilinear fusion to enhance transferability and robustness under environmental shifts.

2.3.3. Behavioral and Environmental Generalization

Behavioral Generalization. aims to enhance the adaptability of world models in cross-task and cross-platform action modeling, where distribution shifts in dynamics across embodiments pose a central challenge. RUWM [102] enhances behavioral robustness by analyzing latent representation errors and introducing regularization over physical state transitions. 3DFlowAction [30] and FlowDreamer [67] address challenges across heterogeneous robotic platforms, complex manipulation tasks, and novel objects by leveraging 3D optical flow as a physically grounded motion representation and large-scale pretraining. AdaWorld [103] employs latent action self-supervision to extract and recombine actions, thereby enabling flexible control and behavioral generalization across heterogeneous environments.

Environmental Generalization. focuses on platform differences and adaptation to new environments. Vidar unifies observation spaces across different robotic platforms by integrating multi-view video data, and introduces a masked inverse dynamics mechanism to extract action features from generated videos, achieving few-shot generalization in dual-arm manipulation under varied configurations. DreamerV3 [11] leverages robustness techniques to handle multimodal returns and sparse rewards, improving adaptability in novel environments; its extension, cRSSM [104], systematically incorporates contextual information into DreamerV3, further strengthening generalization to unseen settings. AirSpace [74] adopts a two-stage training scheme—intent-controllable modeling and spatiotemporal constraint learning—to address distribution mismatches in physical dynamics and limited diversity in embodied intelligence for aerial domains. MoSim [98] decouples physics modeling from policy learning via a neural motion simulator, effectively mitigating challenges posed by unseen scenarios and distribution shifts in dynamics, thus enhancing cross-task generalization.

Table 3. Hardware and configuration comparison for lightweight world-model approaches (for reference only).

Work	Venue	GPUs	Batch Size	Training Steps
DINO-world [105]	arXiv’25	H100*16	1024	350K Iter.
HWM [106]	arXiv’25	A6000*2	128	–
MinD [107]	arXiv’25	A40*4	–	9 Hours
Sparse Imagin. [108]	ICLR’26	3090*4	32	100 Epochs
Simulus [109]	arXiv’25	4090*1	8	100 Epochs
EMERALD [110]	ICML’25	3090*1	16	–
${\mathrm{D}}^2$-World [111]	arXiv’24	V100*8	24	24 Epochs
AVID [71]	RLC’25	A100*4	64	7 Days
ScaleZero [112]	arXiv’25	A100*8	512	–
KeyWorld [113]	arXiv’25	A800*8	1	100 Epochs
TWIST [114]	ICRA’24	3090*1	–	500K Iter.
IRIS [115]	ICLR’23	A100*8	256	3.5 Days
$\Delta$-IRIS [116]	ICML’24	A100*1	32	1K Epochs
HERO [117]	arXiv’25	A100*1	–	–
PosePilot [118]	IROS’25	A100*8	-	–
OCWM [119]	ICLR’25	H100*4	32	40 Epochs

Table 3. Hardware and configuration comparison for lightweight world-model approaches (for reference only).

Work	Venue	GPUs	Batch Size	Training Steps
DINO-world [105]	arXiv’25	H100*16	1024	350K Iter.
HWM [106]	arXiv’25	A6000*2	128	–
MinD [107]	arXiv’25	A40*4	–	9 Hours
Sparse Imagin. [108]	ICLR’26	3090*4	32	100 Epochs
Simulus [109]	arXiv’25	4090*1	8	100 Epochs
EMERALD [110]	ICML’25	3090*1	16	–
${\mathrm{D}}^2$-World [111]	arXiv’24	V100*8	24	24 Epochs
AVID [71]	RLC’25	A100*4	64	7 Days
ScaleZero [112]	arXiv’25	A100*8	512	–
KeyWorld [113]	arXiv’25	A800*8	1	100 Epochs
TWIST [114]	ICRA’24	3090*1	–	500K Iter.
IRIS [115]	ICLR’23	A100*8	256	3.5 Days
$\Delta$-IRIS [116]	ICML’24	A100*1	32	1K Epochs
HERO [117]	arXiv’25	A100*1	–	–
PosePilot [118]	IROS’25	A100*8	-	–
OCWM [119]	ICLR’25	H100*4	32	40 Epochs

2.4. Lightweight

Lightweighting world models is crucial due to high spatial-temporal overheads that limit real-time tasks like robotics and autonomous driving. Beyond efficiency, reducing model size helps filter out visually redundant information, forcing models to capture physically meaningful dynamics rather than irrelevant details. Compact architectures act as inductive biases for learning concise, generalizable, and physics-grounded representations. Existing methods focus on maximizing computational and sample efficiency while preserving essential physical dynamics. Methodologies can be grouped into several complementary strategies:

Learning in structured latent spaces. Since physical dynamics are low-dimensional relative to raw pixel space, learning in structured latent spaces provides a natural way to separate physically meaningful state representations from visual redundancy. Akbulut et al. [120] provided early experimental evidence that structured latent representations significantly improve both efficiency and performance, achieving over a 50% gain compared with modeling directly in observation space. Recent works such as DINO-World [105], MinD [107], and HWM [106] similarly compress high-dimensional pixel streams into compact latent spaces and perform temporal modeling therein, leveraging VQ-VAE [121], denoised latent representations, or low-resolution latent features to fundamentally reduce the computational and memory overhead of sequence modeling while retaining physically relevant state information.

Shortening sequence length via tokenization, sparsification, and parallelism. Physical dynamics are often locally sparse — most video frames contain physically uninformative redundancy between keystate transitions. Methods such as Sparse Imagination (random token dropping and grouped sparse attention) [108], Simulus (modular tokenization) [109], and $\Delta$-IRIS (reduced key-frame encoding with autoregressive optimization) [116] lessen redundant temporal dependencies through token sparsification, concentrating model capacity on significant state changes. In parallel, approaches like EMERALD (MaskGIT-style parallel prediction) [110] and D²-World (non-autoregressive single-stage occupancy prediction) [111] further reduce the cumulative cost of autoregressive training by enabling parallel or masked generation across time.

Parameter-efficient strategies for deployment and transfer. Models including AVID [71], PosePilot [118], and LAMP [122] adapt tasks through lightweight adapters, black-box masking, or plug-and-play modules, avoiding modification — or requiring only minimal fine-tuning — of pretrained backbones that already encode broad physical priors. ScaleZero [112] balances capacity and efficiency via dynamic parameter scaling and staged LoRA expansion. KEYWORLD [113] concentrates computation on a sparse set of semantically and critical frames and employs lightweight convolutional models to synthesize intermediate frames, while TWIST [114] distills a privileged-state teacher world model into a student trained solely on domain-randomized visual observations, effectively reducing both training time and data requirements while preserving generalization.

These approaches have demonstrated substantial improvements in training/inference speed, model size, and downstream control sample efficiency. Nevertheless, they share several trade-offs rooted in the tension between fidelity and computational economy: the invertibility and fidelity of latent representations constrain the upper bound of variable recovery and fine-grained dynamics modeling; sparsification and non-autoregressive strategies face challenges in maintaining long-horizon continuity and handling multimodal uncertainty in state transitions; and black-box adaptation methods exhibit limited generalization when faced with cross-domain transfer involving substantially different environments or dynamics distributions beyond pretraining [123].

Table 4. Summary of Works on Physical Grounding Challenges. Note: Abbreviations used in the table — HR. denotes hierarchical representation. Modalities: indicates video or multi-frame; denotes single-frame or image; refers to latent representation; to spatial representation; to text modality; to action representation; to camera (pose) information; to depth feature; to neural signal; to object-level feature; to physical information; and to historical memory feature. Evals and Downstream Apps: Physical generation, question answering, interaction, understanding, attributes are abbreviated as ${\mathrm{P}}_{\mathrm{G},\mathrm{Q},\mathrm{I},\mathrm{U},\mathrm{A}}$, A refers to action prediction. M stands for motion planning. F stands for fluid dynamics. In downstream applications: W is real world, R is robotics, D is autonomous driving, and O is objects.

Work	Venue	Method	Input / Output	Conditions	Evals and Apps
Explicit Priors & Feedback Integration
Pandora [125]	arXiv’24	Physical Prompts	/		${\mathrm{P}}_{\mathrm{G}}-\mathrm{W}$
WorldGPT [126]	MM’24	Modality Alignment	/		${\mathrm{P}}_{\mathrm{G}}-\mathrm{W}$
LLMPhy [127]	arXiv’24	Engine Integration	/		${\mathrm{P}}_{\mathrm{Q}}-\mathrm{O}$
DrivePhysica [128]	arXiv’24	Positional Constraints	/		${\mathrm{P}}_{\mathrm{G}}-\mathrm{D}$
PhysTwin [129]	ICCV’25	Attribute Fusion	/	None	$\mathrm{A}-\mathrm{R}\mathrm{O}$
SlotPi [130]	SIGKDD’25	Physical Constraints	/	None	${\mathrm{P}}_{\mathrm{G},\mathrm{Q}}\mathrm{F}-\mathrm{D}$
S2-SSM [131]	arXiv’25	Sparse Regularization	/	None	${\mathrm{P}}_{\mathrm{I}}-\mathrm{O}$
RenderWorld [132]	ICRA’25	Pretraining	/		$\mathrm{M}-\mathrm{D}$
DINO-WM [95]	ICML’25	Pretraining Priors	/	None	$\mathrm{M}-\mathrm{O}\mathrm{R}$
HERMES [133]	ICCV’25	Multi-view Modeling	/		$\mathrm{A}-\mathrm{D}$
Cosmos [91]	arXiv’25	Multimodal Constraints	/		${\mathrm{P}}_{\mathrm{Q}}-\mathrm{W}$
Disentangling Static and Dynamic Factors
AdaWorld [103]	ICML’25	Action Decoupling	/		${\mathrm{P}}_{\mathrm{I},\mathrm{M}}-\mathrm{O}$
Dyn-O [62]	NeurIPS’25	Dynamic Decoupling	/		$\mathrm{A}-\mathrm{O}$
ContextWM [134]	NeurIPS’23	Dynamic Decoupling	/		${\mathrm{P}}_{\mathrm{I}}\mathrm{A}\mathrm{M}-\mathrm{D}\mathrm{R}$
DisWM [135]	ICCV’25	Dynamic Decoupling	/	None	${\mathrm{P}}_{\mathrm{A},\mathrm{U}}-\mathrm{O}$
DreamDojo [67]	RAL’26	Explicit Action Modeling	/		$\mathrm{A}\mathrm{M}-\mathrm{R}$
DreamZero [30]	arXiv’26	Action Decoupling	/		$\mathrm{A}-\mathrm{R}$
OC-STORM [136]	arXiv’25	Object Extraction	/	None	${\mathrm{P}}_{\mathrm{U}}\mathrm{M}-\mathrm{O}$
AD3 [137]	ICML’24	Action Decoupling	/		${\mathrm{P}}_{\mathrm{U}}\mathrm{M}-\mathrm{O}$
LongDWM [138]	arXiv’25	Action Decoupling	/		${\mathrm{P}}_{\mathrm{G}}-\mathrm{D}$
Vidar [93]	arXiv’25	Action Decoupling	/		${\mathrm{P}}_{\mathrm{G}}-\mathrm{R}$
DREAMGEN [85]	arXiv’25	Pseudo Action Estimation	/		${\mathrm{P}}_{\mathrm{G}}-\mathrm{R}$
VLMWM [88]	arXiv’25	Fine-tuning	/	None	${\mathrm{P}}_{\mathrm{G}}\mathrm{A}-\mathrm{W}$
WorldDreamer [96]	arXiv’24	Disentangled Modeling	/		${\mathrm{P}}_{\mathrm{G}}-\mathrm{W}$
Simulus [109]	arXiv’25	Dynamic Decoupling	/		$\mathrm{M}-\mathrm{O}$
SCALOR [139]	ICLR’20	Background Modeling	/		${\mathrm{P}}_{\mathrm{U}}\mathrm{A}-\mathrm{O}$
AETHER [140]	ICCV’25	Unified Modeling	/	None	${\mathrm{P}}_{\mathrm{G}}\mathrm{A}-\mathrm{W}$
UWM [29]	RSS’25	Action Decoupling	/	None	${\mathrm{P}}_{\mathrm{G}}\mathrm{A}-\mathrm{R}$
FLARE [87]	CoRL’25	Unified Modeling	/		$\mathrm{A}-\mathrm{W}$
Progressive Constraints & Hierarchical Abstraction
DWS [73]	AAAI’26	Regularization	//		${\mathrm{P}}_{\mathrm{G},\mathrm{I}}-\mathrm{R}\mathrm{O}$
Dreamland [92]	arXiv’25	Engine Simulation	/		${\mathrm{P}}_{\mathrm{U},\mathrm{I}}-\mathrm{D}$
GWM [141]	ICCV’25	Hierarchical Abstraction	//		$\mathrm{A}-\mathrm{R}$
PIWM [142]	arXiv’24	Interpretability	/	None	${\mathrm{P}}_{\mathrm{U},\mathrm{I}}-\mathrm{O}$
Ross et al. [143]	ICLR’25	Theoretical Framework	/	None	${\mathrm{P}}_{\mathrm{A}}-\mathrm{O}$
SimWorld [90]	arXiv’25	Simulation-based Modeling	/		${\mathrm{P}}_{\mathrm{G},\mathrm{U}}-\mathrm{D}$
MoSim [98]	CVPR’25	Multi-constraint	/	None	$\mathrm{A}-\mathrm{R}$
WALL-E [144]	NeurIPS’25	Rule Learning	/		$\mathrm{M}-\mathrm{O}$
FOLIAGE [145]	arXiv’25	Hierarchical Abstraction	/		${\mathrm{P}}_{\mathrm{A}}-\mathrm{O}$
LLMPHY [127]	arXiv’24	Hierarchical Abstraction	/		${\mathrm{P}}_{\mathrm{A},\mathrm{I}}-\mathrm{O}$
V-JEPA 2 [12]	arXiv’25	Hierarchical pretraining	/		$\mathrm{A}\mathrm{M}-\mathrm{R}$

Table 4. Summary of Works on Physical Grounding Challenges. Note: Abbreviations used in the table — HR. denotes hierarchical representation. Modalities: indicates video or multi-frame; denotes single-frame or image; refers to latent representation; to spatial representation; to text modality; to action representation; to camera (pose) information; to depth feature; to neural signal; to object-level feature; to physical information; and to historical memory feature. Evals and Downstream Apps: Physical generation, question answering, interaction, understanding, attributes are abbreviated as ${\mathrm{P}}_{\mathrm{G},\mathrm{Q},\mathrm{I},\mathrm{U},\mathrm{A}}$, A refers to action prediction. M stands for motion planning. F stands for fluid dynamics. In downstream applications: W is real world, R is robotics, D is autonomous driving, and O is objects.

Work	Venue	Method	Input / Output	Conditions	Evals and Apps
Explicit Priors & Feedback Integration
Pandora [125]	arXiv’24	Physical Prompts	/		${\mathrm{P}}_{\mathrm{G}}-\mathrm{W}$
WorldGPT [126]	MM’24	Modality Alignment	/		${\mathrm{P}}_{\mathrm{G}}-\mathrm{W}$
LLMPhy [127]	arXiv’24	Engine Integration	/		${\mathrm{P}}_{\mathrm{Q}}-\mathrm{O}$
DrivePhysica [128]	arXiv’24	Positional Constraints	/		${\mathrm{P}}_{\mathrm{G}}-\mathrm{D}$
PhysTwin [129]	ICCV’25	Attribute Fusion	/	None	$\mathrm{A}-\mathrm{R}\mathrm{O}$
SlotPi [130]	SIGKDD’25	Physical Constraints	/	None	${\mathrm{P}}_{\mathrm{G},\mathrm{Q}}\mathrm{F}-\mathrm{D}$
S2-SSM [131]	arXiv’25	Sparse Regularization	/	None	${\mathrm{P}}_{\mathrm{I}}-\mathrm{O}$
RenderWorld [132]	ICRA’25	Pretraining	/		$\mathrm{M}-\mathrm{D}$
DINO-WM [95]	ICML’25	Pretraining Priors	/	None	$\mathrm{M}-\mathrm{O}\mathrm{R}$
HERMES [133]	ICCV’25	Multi-view Modeling	/		$\mathrm{A}-\mathrm{D}$
Cosmos [91]	arXiv’25	Multimodal Constraints	/		${\mathrm{P}}_{\mathrm{Q}}-\mathrm{W}$
Disentangling Static and Dynamic Factors
AdaWorld [103]	ICML’25	Action Decoupling	/		${\mathrm{P}}_{\mathrm{I},\mathrm{M}}-\mathrm{O}$
Dyn-O [62]	NeurIPS’25	Dynamic Decoupling	/		$\mathrm{A}-\mathrm{O}$
ContextWM [134]	NeurIPS’23	Dynamic Decoupling	/		${\mathrm{P}}_{\mathrm{I}}\mathrm{A}\mathrm{M}-\mathrm{D}\mathrm{R}$
DisWM [135]	ICCV’25	Dynamic Decoupling	/	None	${\mathrm{P}}_{\mathrm{A},\mathrm{U}}-\mathrm{O}$
DreamDojo [67]	RAL’26	Explicit Action Modeling	/		$\mathrm{A}\mathrm{M}-\mathrm{R}$
DreamZero [30]	arXiv’26	Action Decoupling	/		$\mathrm{A}-\mathrm{R}$
OC-STORM [136]	arXiv’25	Object Extraction	/	None	${\mathrm{P}}_{\mathrm{U}}\mathrm{M}-\mathrm{O}$
AD3 [137]	ICML’24	Action Decoupling	/		${\mathrm{P}}_{\mathrm{U}}\mathrm{M}-\mathrm{O}$
LongDWM [138]	arXiv’25	Action Decoupling	/		${\mathrm{P}}_{\mathrm{G}}-\mathrm{D}$
Vidar [93]	arXiv’25	Action Decoupling	/		${\mathrm{P}}_{\mathrm{G}}-\mathrm{R}$
DREAMGEN [85]	arXiv’25	Pseudo Action Estimation	/		${\mathrm{P}}_{\mathrm{G}}-\mathrm{R}$
VLMWM [88]	arXiv’25	Fine-tuning	/	None	${\mathrm{P}}_{\mathrm{G}}\mathrm{A}-\mathrm{W}$
WorldDreamer [96]	arXiv’24	Disentangled Modeling	/		${\mathrm{P}}_{\mathrm{G}}-\mathrm{W}$
Simulus [109]	arXiv’25	Dynamic Decoupling	/		$\mathrm{M}-\mathrm{O}$
SCALOR [139]	ICLR’20	Background Modeling	/		${\mathrm{P}}_{\mathrm{U}}\mathrm{A}-\mathrm{O}$
AETHER [140]	ICCV’25	Unified Modeling	/	None	${\mathrm{P}}_{\mathrm{G}}\mathrm{A}-\mathrm{W}$
UWM [29]	RSS’25	Action Decoupling	/	None	${\mathrm{P}}_{\mathrm{G}}\mathrm{A}-\mathrm{R}$
FLARE [87]	CoRL’25	Unified Modeling	/		$\mathrm{A}-\mathrm{W}$
Progressive Constraints & Hierarchical Abstraction
DWS [73]	AAAI’26	Regularization	//		${\mathrm{P}}_{\mathrm{G},\mathrm{I}}-\mathrm{R}\mathrm{O}$
Dreamland [92]	arXiv’25	Engine Simulation	/		${\mathrm{P}}_{\mathrm{U},\mathrm{I}}-\mathrm{D}$
GWM [141]	ICCV’25	Hierarchical Abstraction	//		$\mathrm{A}-\mathrm{R}$
PIWM [142]	arXiv’24	Interpretability	/	None	${\mathrm{P}}_{\mathrm{U},\mathrm{I}}-\mathrm{O}$
Ross et al. [143]	ICLR’25	Theoretical Framework	/	None	${\mathrm{P}}_{\mathrm{A}}-\mathrm{O}$
SimWorld [90]	arXiv’25	Simulation-based Modeling	/		${\mathrm{P}}_{\mathrm{G},\mathrm{U}}-\mathrm{D}$
MoSim [98]	CVPR’25	Multi-constraint	/	None	$\mathrm{A}-\mathrm{R}$
WALL-E [144]	NeurIPS’25	Rule Learning	/		$\mathrm{M}-\mathrm{O}$
FOLIAGE [145]	arXiv’25	Hierarchical Abstraction	/		${\mathrm{P}}_{\mathrm{A}}-\mathrm{O}$
LLMPHY [127]	arXiv’24	Hierarchical Abstraction	/		${\mathrm{P}}_{\mathrm{A},\mathrm{I}}-\mathrm{O}$
V-JEPA 2 [12]	arXiv’25	Hierarchical pretraining	/		$\mathrm{A}\mathrm{M}-\mathrm{R}$

2.5. Universality

The lack of unification arises because videos capture only limited slices of reality, lacking a shared cross-modal physical representation. Truly universal world models rely on shared physical abstractions that generalize across domains. The Plato hypothesis [146] suggests vision, language, and action are projections of the same underlying reality, implying a unified semantic space. Contemporary models leverage this by mapping heterogeneous modalities into shared physical representations, enabling cross-modal consistency, knowledge transfer, and multi-task robustness.

As shown in Figure 4, Yue et al.’s unified framework [124] decomposes the system into five components: interaction and perception; unified reasoning (dynamics, causality, explicit and latent reasoning); memory; environment (learnable and generative); and multimodal generation (video, image, audio, 3D, prediction). This framework exhibits important complementarities with Eq. (1)–(3) in this paper: while Eq. (1)–(3) define the formal backbone of world models, the above framework explicitly incorporates memory modules and multimodal generative capabilities, thereby providing broader coverage for emerging variants of world models

Data pretraining and shared latent-space modeling form the core technical foundation for generality. DINO-world [105] leverages large-scale, uncurated web videos for pretraining, combined with the DINOv2 [65] visual encoder to perform latent-space predictions. Its compact Transformer autoregressive architecture supports variable frame rates and context lengths, extracting physically generalizable knowledge from diverse data and demonstrating seamless cross-task adaptability. WorldDreamer [96] frames world modeling as an unsupervised visual sequence problem, integrating text, image, and action modalities through masked token prediction, with parallel masked-token prediction enhancing efficiency and exhibiting high generalization and flexibility across tasks. V-JEPA 2 [12] performs self-supervised pretraining on Internet videos combined with limited robotic data to learn visually predictable representations of physical dynamics, enabling zero-shot planning and future prediction across downstream tasks. WPT [147] systematically leverages reward-free, non-expert, multi-robot, uncurated offline data, filling in action dimensions to pretrain a single world model across embodied agents. Cosmos [91], as a foundational world model, is pretrained on diverse video and physical scenarios to acquire universal structure and causal patterns, and after finetuning, can generate, predict, and simulate future states, handle cross-modal inputs, and produce controllable outputs, providing a unified framework for AI applications.

Modular design and multi-task shared representations provide architectural guarantees for generality. DreamerV3 [11] jointly learns environment models to predict action outcomes and plan ahead, using normalization and balancing techniques to handle diverse reward distributions and fixed hyperparameters for stable multi-task learning, demonstrating general adaptability without task-specific tuning. AETHER [140] integrates 4D dynamic reconstruction, action prediction, and visual planning within a single physical backbone, emphasizing geometric reasoning and achieving zero-shot generalization to real environments from synthetic data, while its global action representation further unifies and enhances generalization in navigation and robotic tasks. HERMES [133] unifies 3D scene understanding and future evolution via bird’s-eye representations, preserving geometric relations while integrating multi-view spatial information, and introduces a “world query” mechanism to fuse physical knowledge within a unified architecture. UniFuture [47] employs dual latent-variable sharing and multi-scale interaction to integrate future generation and depth perception, refining cross-modal features and generating temporally consistent future images and depth maps from single frames, exhibiting zero-shot generalization in unseen environments.

Flexible architectural extensions and instruction-based control further expand the applicability of generality. Simulus [109] uses a modular unified design, independently handling tokenizers, embedding tables, and prediction heads for each modality, decoupling representation learning from physical dynamics modeling and supporting various modality combinations, achieving sample-efficient generality across benchmarks. 1X World Model [148] predicts environment changes from specific action trajectories, capturing cross-task patterns and promoting generalization through multi-task training, enabling unified planning and decision-making across physical tasks. Pandora [125] integrates pretrained language and video models in a two-stage unified training, with instruction finetuning enabling real-time natural language control over video generation, demonstrating cross-domain unification and controllability in indoor, outdoor, and gaming environments. UWM [29] unifies action and video diffusion within a Transformer architecture, decoupling diffusion step execution, policy learning, and world modeling, learning causal and physical understanding from heterogeneous data and supporting flexible unified reasoning.

Figure 5. Three paradigms for explicit physical knowledge injection in world models. Left: integrating domain knowledge via explicit physical priors and simulation feedback to mitigate hallucinations in dynamic prediction. Middle: disentangling physically invariant (static) and physically variant (dynamic) latent factors for scene representation. Right: progressive constraints with hierarchical abstractions, from pixel-level fidelity to abstract causal structures.

Figure 5. Three paradigms for explicit physical knowledge injection in world models. Left: integrating domain knowledge via explicit physical priors and simulation feedback to mitigate hallucinations in dynamic prediction. Middle: disentangling physically invariant (static) and physically variant (dynamic) latent factors for scene representation. Right: progressive constraints with hierarchical abstractions, from pixel-level fidelity to abstract causal structures.

3. Three Paradigm of Physical World Model

Physical knowledge in videos is often implicitly present and challenging for models to learn effectively. Explicit physical embedding endows world models with physical perception—the ability to understand physical laws, object properties, and causal dynamics—and is crucial for bridging the pixel–physics gap. Existing works can be broadly categorized into three paradigms of explicit injection, as follows:

3.1. Learning from Physical Priors

A prominent strategy is to incorporate physical priors and simulation feedback, anchoring the world model within classical mechanics frameworks to mitigate hallucinations in dynamic prediction. By explicitly injecting domain knowledge—such as object properties, conservation laws, or interaction constraints—models are guided toward physically plausible trajectories even under partial observability or long-horizon rollout [149]. This paradigm is particularly important in safety-critical applications, where purely data-driven models often suffer from compounding errors and unrealistic predictions.

Representative methods include: Pandora [125] and WorldGPT [126], which leverage LLMs to parse textual descriptions of physical properties (e.g., stiffness, friction) and translate them into perceptual priors within simulated environments; LLMPhy [127], which iteratively optimizes scene parameters via LLM-based program synthesis and physics engine feedback to support complex reasoning tasks; and DrivePhysica [128], which explicitly models motion understanding and spatial relationships in autonomous driving scenarios using coordinate alignment, instance flow guidance, and bounding-box conditioning. PhysTwin [129] integrates spring–mass models, sparse-to-dense optimization, and Gaussian splatting to construct digital twins of deformable objects. Other approaches, such as SlotPi [130], introduce Hamiltonian constraints into spatiotemporal reasoning modules to enforce energy-consistent predictions, while S2-SSM [131] infers causal graphs from object interactions using sparsity-regularized state-space models. Renderworld [132] and TRANSDREAMER [150] further enhance physical consistency via 2D–3D occupancy mapping and transformer-based long-range modeling, respectively. These methods exemplify the broader shift toward hybrid neuro-symbolic systems that combine differentiable physics priors with learned representations. Notably, GeoPT [151] proposes a synthetic-dynamics pretraining paradigm that lifts static geometry into a dynamic space, enabling models to acquire physical intuition from unlabeled particle trajectory evolution.

3.2. Learning from Action Decoupling

Another strategy for enhancing physical perception is to explicitly decompose scene representations into physically invariant (static) and physically variant (dynamic) components. This decomposition reflects a physics-inspired inductive bias: background structures and scene geometry typically follow stable constraints over time, while foreground objects evolve under dynamic forces and interactions [152]. By separating these factors, world models can reduce interference between unrelated signals, leading to improved robustness, generalization, and interpretability in complex environments.

Several recent works operationalize this idea through structured latent representations and disentangled dynamics modeling. Dyn-O [62] employs a Mamba-based state-space model [153] to disentangle object-centric features, enabling stable prediction in cluttered scenes. ContextWM [134] separates perception into a context encoder that extracts temporally invariant cues and a dynamic module that models action-dependent evolution. 3DFlowAction [30] leverages 3D optical flow to unify motion representations across different agents, providing object-centric motion signals independent of the actor identity. OC-STORM [136] extracts precise object states and fuses them with raw visual input to mitigate background noise and repeated recognition errors, while AD3 [137] explicitly isolates task-relevant dynamics from distractors in control scenarios. More recently, latent action disentanglement and embedding approaches [103,154], exemplified by DreamDojo [155], learn compact and transferable action representations directly from human videos. These methods significantly improve embodied physical reasoning by aligning latent action spaces with underlying physical causality.

3.3. Hierarchical Progressive Learning

Hierarchical modeling with progressively enforced supervision has emerged as a scalable framework for physical world modeling, enabling representations to evolve from low-level sensory fidelity to high-level causal abstraction. This paradigm reflects the multi-scale nature of physical systems, where local interactions (e.g., pixel motion or contact dynamics) aggregate into global structures and long-term dependencies. By organizing learning across multiple levels of abstraction, hierarchical models can better capture both fine-grained physical details and coarse-grained semantic regularities.

Representative approaches include RoboScape [40], which dynamically samples salient keypoints (e.g., robots or manipulable objects) to implicitly encode deformation and material properties through temporal consistency; and DWS [73], which emphasizes motion-aware supervision to prioritize dynamic over static features. Dreamland [92] introduces hierarchical and controllable world abstractions that enable precise manipulation at both pixel and object levels, while PSI [156] employs a three-phase loop—predictor training, structure extraction (e.g., optical flow and depth), and reintegration—to achieve structured and controllable representations. From a theoretical perspective, the system-theoretic framework of Ross et al. [143] formulates world-model learning as low-dimensional temporal projection and tokenization, spanning models from classical least-squares regression to generative approaches such as GANs [157].

Recent methods further integrate geometry and physics into hierarchical pipelines. GWM [141] and PIWM [142] combine Gaussian splatting propagation with physics-equation mappings to achieve interpretable 3D state prediction and reasoning. Notably, hierarchical modeling has been widely recognized for its advantages in physical interpretability, modularity, and adaptability across tasks. V-JEPA2 [12], for instance, encodes videos into high-level semantic representations, filters out stochastic or texture-level noise, and learns stable temporal dynamics via masked prediction and multi-scale modeling. Through this progressive abstraction process, such models gradually approximate real-world state transitions and underlying physical laws within the latent space.

4. Future Directions and Discussion

4.1. Open Challenges

Deep Integration of Physics Engines and Neural Networks. Current video-driven world models primarily rely on large-scale data pretraining to implicitly learn physical laws. However, they still exhibit significant limitations in out-of-distribution generalization and in modeling complex phenomena such as contact dynamics and non-rigid deformations. Hybrid paradigms, exemplified by ContactGaussian-WM [158], integrate differentiable physics engines (e.g., MuJoCo [159], PhysX [160], Isaac [161]) with 3DGS representations, enabling more physically consistent modeling through the synergy of physical priors and neural networks. Nevertheless, several key challenges remain, including differentiability bottlenecks (e.g., discontinuities in contact events that hinder gradient propagation), computational efficiency (e.g., mismatches between simulation time scales and video frame rates), and the difficulty of accurately inferring material properties from visual observations.

Interpretability and Verifiability of World Models. Most existing world models operate as end-to-end black-box systems. While effective for perception and generation tasks, their lack of interpretability hinders deployment in safety-critical scenarios such as surgical robotics and nuclear facility inspection. Approaches such as PIWM [142] attempt to align prediction processes with explicit physical equations (e.g., Newtonian mechanics and energy conservation), incorporating physically meaningful latent variables, symbolic regression, and formal verification to enhance interpretability and verifiability. Recently, the divergence between JEPA [12] and generative paradigms has raised a fundamental question: is pixel-level prediction necessary for learning physical laws, or should models instead operate in abstract representation spaces to improve efficiency and generalization?

Cross-Domain Transfer.Section 2.5 discussed recent progress toward unified paradigms across multiple tasks, environments, and frameworks. However, achieving seamless generalization across domains—such as gaming, robotic manipulation, autonomous driving, and the real world—remains an open problem. First, general-to-specialized adaptation remains challenging. For instance, although the “pretraining–adaptation” paradigm (e.g., Waymo’s driving model [162]) has shown promise, heterogeneity in physical dynamics across domains (e.g., vehicle dynamics vs. robotic manipulators vs. human motion) limits the effectiveness of straightforward fine-tuning. Second, unified action spaces are still lacking. Cross-embodiment transfer is constrained by differences in action representations. Promising directions include the development of universal intermediate action representations, embodiment-agnostic motion primitives, and hierarchical action abstractions [155]. Third, commonsense-driven transfer remains underexplored. Works such as DisWM [135] and DINO-WM [95] have taken initial steps, but systematic transfer of physical commonsense still lacks robust evaluation protocols and unified modeling frameworks.

Standardization of Evaluation Metrics. The visual realism of world models does not guarantee physical correctness, and strong task performance does not necessarily imply genuine understanding of physical laws [163]. Although efforts such as WorldScore [164] and WorldModelBench [165] aim to establish comprehensive evaluation frameworks, no community-wide consensus has yet been reached. Recent initiatives, including the CVPR 2025 WorldModelBench Workshop and the ICLR 2026 World Models Workshop, indicate that the community is actively moving toward standardized evaluation methodologies.

Data Bottlenecks and Synthetic Data Closed Loops. High-quality datasets must exhibit rich physical diversity and include multimodal signals such as depth, force/torque, and joint states, along with high-precision annotations. However, existing datasets remain insufficient in these aspects. Approaches such as NVIDIA Cosmos [91] and EnerVerse [26] propose a closed-loop data paradigm—“real → simulation → generation → augmentation of real data”—which improves data efficiency. Nevertheless, this paradigm introduces the risk of amplifying physical biases through generative models, highlighting the need for independent physical validation mechanisms.

4.2. Industrialization and Deployment

World models are rapidly transitioning from academic research to industrial deployment, forming a synergistic ecosystem across foundational platforms, vertical applications, and data infrastructures. According to a Frost & Sullivan white paper, over 80% of autonomous driving algorithm development pipelines have incorporated world models or simulation for training and validation.

On the platform level, NVIDIA’s Cosmos [91] has evolved into a comprehensive toolchain encompassing multimodal data management, video tokenization, pretrained models, and inference services. Google DeepMind’s Genie 3 [14] pushes world models toward real-time interaction, demonstrating general-purpose capabilities in autonomous driving and game generation. Meanwhile, World Labs [83] has productized 3D scene generation through Marble as a SaaS platform for creators, validating a viable B2C commercialization pathway. In vertical domains, Waymo’s world model [162] built upon Genie 3 demonstrates multi-sensor fusion, hierarchical control, and long-tail scenario generation, representing a benchmark implementation of the “general pretraining + domain adaptation” paradigm. Similarly, Huawei’s WEWA architecture and NIO’s Neural World Model (NWM) system emphasize embedding world models into closed-loop perception–prediction–planning pipelines, enabling online simulation and decision optimization. On the data side, large-scale robotic datasets such as AgiBot-World are lowering entry barriers for the industry.

4.3. Safety and Ethical Challenges

The International AI Safety Report 2026 [166] highlights that current AI systems exhibit “striking limitations” in physical world reasoning: while performing well within training distributions, they may fail catastrophically in edge-case physical scenarios. Representative examples include multi-vehicle collisions on icy roads in autonomous driving and millimeter-scale operational errors in surgical robotics.

Moreover, although synthetic data closed-loop training (see Section 2.3.1) improves data diversity, most existing studies overlook the risk of recursive bias amplification. Specifically, a generative model G, trained on a real dataset $\mathcal{D}$, produces synthetic data $\tilde{\mathcal{D}}=G\left(\mathcal{D}\right)$ that inevitably inherits statistical biases from $\mathcal{D}$, while also introducing new biases arising from the model’s inductive priors. These biases accumulate through nonlinear feedback loops, leading to systematic under representation or overestimation in specific physical scenarios. For instance, in rare robotic manipulation settings—such as handling irregular soft objects, left-handed operations, or cluttered environments—synthetic data may reinforce physical inconsistencies rather than correct them if diversity is not explicitly enforced.

Finally, unlike conventional deepfake content that often lacks strong physical consistency, the ultimate objective of world models is to construct highly realistic world simulators. At that stage, the indistinguishability of fine-grained physical details may enable malicious applications, including public opinion manipulation, scientific fraud, and data poisoning attacks targeting physical AI systems.

5. Conclusions

This paper provides a systematic overview of the cutting-edge field of video-driven physical world models. The in-depth analysis was conducted from four different perspectives: the modeling paradigms, the pixel–physics gap and outlooks.

Regarding the pixel–physics gap, we identified and analyzed five core challenges: continuity, controllability, generalization, lightweight, and universality. For each challenge, we summarized representative solutions and research progress. These analyses show that while the videos provide rich learning resources for world models, they also lead to a series of gaps that need to be bridged through modeling paradigms and training strategies.

Physical consistency is not only a technological challenge, but also the most critical bottleneck for world models to evolve from a “video predictor” to a “trustworthy physical simulator.” In general, this article reveals the trends and several potential pathways for the evolution towards modeling the physical world.