Preprint
Review

This version is not peer-reviewed.

Diffusion Models for Embodied Perception and World Modeling: A Survey from a Humanoid Robotics Perspective

Submitted:

16 November 2026

Posted:

17 November 2026

You are already at the latest version

Abstract
Humanoid intelligence requires agents to perceive uncertain environments, infer structured representations from multimodal sensory observations, and predict future world evolution before acting. Otherwise, recent advances in diffusion models have extended their role far beyond image synthesis, enabling probabilistic perception, spatial scene understanding, and predictive world modeling for embodied AI. By modeling complex data distributions through conditional generation, diffusion models naturally address uncertainty, partial observability, and multimodal future prediction, making them particularly suitable for embodied perception and predictive imagination. This survey provides a comprehensive review of diffusion models for embodied perception and world modeling. We first trace the evolution of humanoid robotics and introduce the diffusion formulations underlying the surveyed methods, and then propose a unified taxonomy that organizes existing methods from probabilistic perception to predictive imagination. We comprehensively review diffusion-based approaches for human perception, embodied self-perception, and environment understanding. For world modeling, we analyze the advantages of diffusion model for observation space prediction, compact learned latent spaces, and conditioning on explicit robot actions or latent actions inferred from video. By unifying recent advances in perception and predictive world modeling, this survey highlights diffusion models as a general probabilistic framework for embodied intelligence and outlines promising directions toward more capable and reliable embodied AI systems.
Keywords: 
;  ;  ;  ;  ;  ;  ;  

1. Introduction

Humanoid robots are increasingly expected to operate in human-centered environments, where they must coordinate whole-body motion, dexterous manipulation, navigation, and physical interaction with people and objects [1,2]. Operating across changing viewpoints, body configurations, and contact conditions requires them to continuously infer the state of their own bodies and surrounding environments [3], while anticipating how these states may evolve during interaction [4]. Consequently, perception and predictive world modeling form two closely coupled components of humanoid embodied intelligence: perception infers the current state from sensory evidence, whereas a world model predicts its possible evolution under environmental dynamics and robot actions.
However, these coupled components must account for uncertainty [5]. Occlusion, limited fields of view, sensor noise, and sparse measurements provide only partial evidence about the human–robot–environment state [6,7]. The same observations may therefore be compatible with different poses, contact states, or scene configurations. This ambiguity also affects future prediction. Specifically, uncertainty about the current state, together with variations in behavior and physical interaction, can lead to multiple plausible outcomes. For example, uncertainty about an object’s pose and contact conditions influences predictions of whether it will remain stable or slip during manipulation, with potential consequences for the humanoid’s posture and balance [8]. Estimators and predictors that return a single output may suppress these alternatives or average across distinct states and trajectories.
These challenges motivate generative approaches that learn conditional distributions over possible states and outcomes [9,10]. Sampling from such distributions provides a way to retain multiple hypotheses supported by the available observations. Previous studies have demonstrated the usefulness of this approach for recovering plausible human poses from ambiguous visual evidence [11,12] and for representing alternative futures in video prediction [13,14]. In terms of humanoid robotics, generative models have been employed to reconstruct missing sensory information or complete partially observed object geometry [10]. Furthermore, generative models represent alternative visual futures and action-conditioned state transitions, thereby supporting prediction and policy learning under uncertainty [15,16]. Generative modeling thus offers a common probabilistic perspective on perceptual inference and future prediction. The connection between perception and future prediction is particularly relevant to humanoids because uncertainty in the inferred body, human, or environment determines which future interactions should be considered.
Among generative approaches, diffusion models [17,18] have attracted considerable attention for their ability to model complex, high-dimensional distributions and generate diverse, high-quality samples. They learn to reverse a gradual noise-corruption process, progressively transforming noise into samples that capture the structure of the training data. Their applications have expanded from image synthesis to video generation and human motion [19], three-dimensional structures [20], and other sequential representations. Moreover, conditional diffusion models can incorporate heterogeneous information, including visual observations [21,22], proprioceptive signals [23] and tactile measurements [24]. This flexibility makes them relevant to embodied problems in which several sources of evidence constrain the same uncertain state or future outcome. For humanoid robotics, these properties support several complementary uses. Conditional generation can recover missing sensory information, infer alternative state hypotheses, and complete partially observed geometry. Predictive models can extend observation histories into possible future observations or compact states, incorporating robot actions when suitable conditioning signals are available. In addition, representations learned during diffusion pretraining can support perceptual understanding without requiring iterative generation for every downstream prediction [25]. Together, these uses connect generative inference of the current embodied state with predictive representations of its evolution. Their practical value depends on how well the resulting estimates and predictions preserve observed evidence, physical relationships, and information needed for interaction.
Existing surveys examine these developments from several perspectives. The authors in [3] review humanoid perception through internal-state estimation, environment understanding, and human-robot interaction, while the authors in [1] survey the evolution and taxonomy of humanoid robots and humanoid AI, and examines their functional and nonfunctional requirements. The work in [26] examines the relationship between diffusion models and representation learning. Within robotics, [27] organize diffusion-based methods around policies and visual synthesis, whereas [28] and [29] emphasize manipulation applications and policy design. World-model surveys provide complementary perspectives. The authors in [30] discuss world representation and future prediction. Moreover, [31] examine perception, prediction, and control in robotic manipulation. More recent surveys analyze world models through their coupling with robot policies [32] and through predictive representations, action interfaces, and evaluation [33]. Table 1 compares the coverage of existing surveys with our review.
These surveys establish substantial foundations, but leave room for a focused analysis of diffusion-based perception and prediction under humanoid sensing and embodiment requirements. In particular, a common comparison is needed to clarify which human, robot, and environment variables are inferred, how predictive representations preserve or omit these variables, and what assumptions about observations, contacts, and actions govern their applicability. This issue is important because accurate perceptual outputs or visually plausible predictions alone do not establish that a method retains the information required for whole-body interaction. Examining state inference, sensory completion, representation reuse, and future generation together can therefore reveal their complementary capabilities, incompatible assumptions, and requirements for integration into humanoid systems. The main contributions of this survey are summarized as follows:
  • We organize diffusion-based humanoid perception according to the state being inferred: human perception, embodied self-perception, and environment understanding. This taxonomy connects diverse sensing modalities and generative methods to their perceptual roles.
  • We analyze predictive world models along three complementary dimensions: observation space prediction, compact state space world models, and action conditioned humanoid world models. The discussion covers future observations, compact latent and structured states, explicit and latent action conditioning, and temporal, uncertainty, and physical consistency.
  • We compare the roles of diffusion across perception and prediction, including probabilistic state inference, sensory completion, pretrained representation reuse, and multimodal future generation. We analyze these roles to the information and interaction requirements of humanoid systems.
  • We highlight open challenges and research directions for practical deployment, including real-time inference, physical grounding and uncertainty calibration, long-horizon consistency, cross-embodiment learning, evaluation, and structured representations that connect perception with prediction.
The remainder of this survey is organized as follows. Section 2 reviews the evolution of humanoid robotics and the diffusion formulations required for the subsequent discussion. Section 3 surveys diffusion models for human perception, embodied self-perception, and environment understanding. Section 4 examines predictive world models through their representations, action interfaces, and reliability. Section 5 discusses open challenges and future research directions, and Section 6 concludes the survey. The organization of the survey is summarized in Figure 1.

2. Preliminaries

This section establishes the two strands needed for the remainder of the survey. We first trace how humanoid robotics progressed from demonstrating anthropomorphic embodiment to learning generalizable sensorimotor behavior. We then introduce the diffusion formulations most frequently used in this literature: denoising diffusion probabilistic models (DDPMs), denoising diffusion implicit models (DDIMs), and conditional diffusion policies.

2.1. Evolution of Humanoid Robots

Humanoid robotics initially focused on the physical and control foundations required to reproduce human-like mobility. Early platforms such as WABOT [76] demonstrate the integration of bipedal locomotion, manipulation, and basic perception, while Honda’s robots, such as P2 and ASIMO, advanced self-contained dynamic walking and operation in human environments [77]. The field has then progressed from locomotion demonstrations to integrated research platforms. Humanoid robot HRP [78] combined uneven-terrain walking, fall recovery, and cooperative manipulation, whereas iCub was designed to study sensorimotor learning and cognitive development [79]. These developments have expanded the goal of humanoid robotics from maintaining balance to coordinating perception, locomotion, manipulation, and interaction on an anthropomorphic body.
Most of these systems relied on model-based and hierarchically engineered pipelines. Perception, planning, and control were typically separated, while walking and whole-body motion were generated from analytical dynamics, stability criteria, and optimization. Such methods provide interpretable and precise behavior when the robot and environment can be modeled accurately. For example, optimization-based methods on Atlas integrated state estimation, footstep planning, and whole-body control for complex locomotion tasks [80]. However, their effectiveness depends on the accuracy of the robot and environment models. High-dimensional dynamics, intermittent contacts, sensor uncertainty, and environmental variation make it difficult to enumerate all feasible behaviors or encode them through manually designed objectives.
Recent research has therefore expanded learning from policy optimization to internal representations of the robot, surrounding humans, and the environment. Perception models recover control-relevant states from multimodal observations, while world models predict their action-conditioned evolution to support planning through imagined futures [81]. Human motion connects these two functions: it must be perceived for safe interaction and can simultaneously provide demonstrations of human-compatible skills. HumanPlus, for example, reconstructs body and hand motion for whole-body shadowing and uses the resulting demonstrations to learn autonomous visuomotor behavior [82]. Both humanoid robot perception and world modeling are inherently ambiguous. A partial image or sensor history may correspond to several human poses, object geometries, or contact states, while the same observed state and action sequence may produce different futures because of uncertain contacts and environmental responses. A point predictor preserves only one solution and may average incompatible alternatives. Generative models instead approximate conditional distributions, allowing a humanoid to generate multiple perceptual hypotheses, complete missing observations, and evaluate alternative futures.
However, generative models such as generative adversarial networks (GANs) [83] and variational autoencoders (VAEs) [84] present important limitations. GANs can generate sharp samples efficiently, but their adversarial training is often unstable and susceptible to mode collapse, potentially omitting rare yet safety-critical states or futures [83,85]. VAEs are easier to train and provide compact latent representations, but their variational bottleneck can reduce output fidelity and oversmooth fine-grained poses, geometry, or dynamics [86,87]. Diffusion models provide a favorable alternative through stable denoising-based training, strong distribution coverage, and progressive refinement of high-dimensional outputs [22]. Their flexible conditioning further allows multimodal observations, actions, tasks, and embodiment constraints to guide both perceptual reconstruction and future prediction. These advantages motivate our focus on diffusion models, despite the additional inference cost of iterative sampling.

2.2. Background on Diffusion Models

Diffusion models generate data by learning to reverse a gradual noise-corruption process. Their ability to represent multiple plausible outputs supports humanoid perception and prediction under incomplete observations. We summarize DDPM and DDIM foundations, followed by conditional and latent extensions for task-relevant generation and computational efficiency.

2.2.1. Foundations of Diffusion Models

Diffusion models are a powerful class of probabilistic generative models that gradually perturb data by adding Gaussian noise to data and then learn to reverse this process to generate new data. During training, the model learns to denoise the data at each step, effectively transforming random noise into coherent and realistic outputs. This section provides an overview of two predominant formulations in diffusion models, i.e., DDPM and DDIM.
Denoising Diffusion Probabilistic Models.
DDPMs [17] define a Markov forward process that progressively perturbs a clean sample x 0 ∼ q ( x 0 ) with Gaussian noise over T timesteps:
q ( x t ∣ x t − 1 ) = N x t ; 1 − β t x t − 1 , β t I , t ∈ { 1 , … , T } ,
where β t ∈ ( 0 , 1 ) is the noise variance and I is the identity matrix. With α t = 1 − β t , α ¯ t = ∏ i = 1 t α i , and α ¯ 0 = 1 , a noisy sample can be obtained directly as x t = α ¯ t x 0 + 1 − α ¯ t ϵ , where ϵ ∼ N ( 0 , I ) . The noise schedule is chosen so that x T approximately follows a standard Gaussian distribution. The learned reverse process reconstructs data through Gaussian transitions p θ ( x t − 1 ∣ x t ) = N x t − 1 ; μ θ ( x t , t ) , σ t 2 I , where μ θ and σ t 2 I denote the transition mean and covariance. Under noise prediction, a denoiser ϵ θ ( x t , t ) parameterizes the mean as
μ θ ( x t , t ) = 1 α t x t − β t 1 − α ¯ t ϵ θ ( x t , t ) .
The commonly used simplified training objective is L DDPM = E x 0 , t , ϵ [ ∥ ϵ − ϵ θ ( x t , t ) ∥ 2 2 ] , which trains the denoiser to recover the injected noise [17]. Generation starts from Gaussian noise and repeatedly applies the learned reverse transitions.
Denoising Diffusion Implicit Models.
DDIMs [88] generalize the forward process to a non-Markovian formulation while retaining the DDPM forward marginals and noise-prediction objective. A pretrained DDPM denoiser can therefore be reused without retraining. In the deterministic case, the sampling update is
x t − 1 = α ¯ t − 1 α ¯ t x t − 1 − α ¯ t ϵ θ ( x t , t ) + 1 − α ¯ t − 1 ϵ θ ( x t , t ) .
Intermediate timesteps can be skipped by replacing t − 1 with an earlier sampling index. This enables fewer denoising iterations and a trade-off between inference latency and sample quality, which is relevant to time-sensitive humanoid perception and prediction.

2.2.2. Variants of Diffusion Models

Conditional diffusion controls generation through contextual inputs, whereas latent diffusion reduces the dimensionality of the denoising space. These extensions are complementary and can be combined in the same model.
Conditional Diffusion Models.
Conditional diffusion models p θ ( x t − 1 ∣ x t , c ) incorporate context c into the denoising process. For humanoid robots, this context may include visual observations, proprioceptive, inertial, or tactile measurements, language instructions, action histories, and embodiment information. The noise-prediction objective becomes
L cond ( θ ) = E x 0 , t , ϵ , c ϵ − ϵ θ ( x t , t , c ) 2 2 .
Conditions can be incorporated through concatenation [89] or cross-attention [90], while classifier guidance [22] and classifier-free guidance [91] steer sampling toward desired outputs. Depending on the predicted variable, conditioning enables reconstruction of partially observed states, prediction of action-dependent futures, or generation of motions. Thus, it links diffusion to both perceptual inference and world modeling, although conditioning alone does not guarantee physical feasibility.
Latent Diffusion Models.
LDMs move denoising from the original data space to a compressed representation [92]. A pretrained encoder–decoder, such as that used in VQGAN [93], maps an input x to z 0 = E ( x ) . The latent sample z t is obtained using the same forward noising process, and the denoiser ϵ θ ( z t , t , c ) can receive context c. Its training objective is
L LDM ( θ ) = E z 0 , t , ϵ , c ϵ − ϵ θ ( z t , t , c ) 2 2 .
At inference, denoising transforms z T ∼ N ( 0 , I ) into z ^ 0 , which is decoded as x ^ = D ( z ^ 0 ) . Operating in a smaller space can reduce memory and computation relative to diffusion directly over high-dimensional observations or trajectories. For humanoid systems, this supports efficient modeling of visual observations, human-motion sequences, whole-body states, and predicted futures. The compression must nevertheless preserve control-relevant geometry, contacts, and dynamics, since information lost during encoding cannot be recovered reliably through denoising alone.

3. Diffusion Models for Humanoid Perception

Perception enables a robot to estimate its current physical state and the state of its surrounding environment from sensory observations [94]. For a humanoid robot, the internal state includes the floating-base pose, joint configurations and velocities, whole-body motion, and contact states. The external state includes scene geometry, semantic entities, objects, and dynamic agents [3]. These quantities must be inferred from heterogeneous proprioceptive and exteroceptive observations, including joint encoders and inertial measurement units [95], RGB or RGB-D cameras [96], and tactile sensors [97].
Humanoid perception presents additional challenges because humanoid robots operate in human-centered environments. People are not merely moving obstacles but active agents whose motions and interactions can change both the environment and the robot’s feasible actions. A general-purpose humanoid must therefore estimate human locations, poses, motions, interaction cues, and relationships with nearby objects and agents. These capabilities are essential for safe coexistence, physical collaboration, and socially appropriate behavior [98,99].
Human perception also supports humanoid skill acquisition because the broadly similar articulated structures of humans and humanoids make human motion informative for imitation, motion retargeting, and learning from demonstration. This requires capturing not only body location and pose but also whole-body coordination, contacts, object manipulation, and adaptation to surrounding geometry. Accordingly, we organize this section around three inference targets: human perception, which estimates other people’s states and behaviors, embodied self-perception, which estimates the robot’s own configuration and interaction state, and environment understanding, which estimates the geometry, objects, semantics, and dynamics of non-human surroundings. Although humans belong to the external environment, their motion, behavior, and object interactions warrant separate treatment through specialized representations and inference methods.
Across these categories, perception involves conditional inference from partial, noisy, and potentially asynchronous observations, which may admit multiple plausible poses, contacts, geometries, or dynamic states. Conditional diffusion models capture this ambiguity by learning and sampling from multimodal conditional distributions, enabling reconstruction of corrupted observations, completion of missing state components, synthesis of missing sensory modalities, and generation of hypotheses consistent with the available evidence.

3.1. Diffusion Models for Human Perception

Human perception is commonly considered part of environment perception because it supports human detection, tracking, and safe human–robot interaction [3,94]. In humanoid robotics, it also supports skill acquisition: estimates of human pose and motion provide structured demonstrations and motion priors for imitation and human-to-humanoid skill transfer [82,100,101]. It therefore serves two complementary functions: understanding people’s states and behaviors for interaction, and extracting motion representations for learning. These perceptual estimates are not directly executable robot actions. Transferring human motions to executable robot trajectories requires morphology- and physics-aware retargeting to accommodate differences in body proportions, kinematics, and dynamic constraints.
Estimating human pose, shape, and motion from sensory observations is inherently ambiguous [34,102]. Image-based methods face occlusion, truncation, perspective projection, and missing depth, whereas inertial methods must recover whole-body motion from sparse, noisy, and potentially drifting measurements. These limitations can make the same observations compatible with multiple human states. Conditional diffusion models capture this ambiguity through distributions over poses and motions conditioned on visual or sensor evidence, enabling them to reconstruct, complete, or refine human states while retaining multiple hypotheses consistent with the available evidence. Learned human-motion priors guide denoising toward anatomically and temporally plausible solutions [103,104]. The resulting estimates can support human-aware interaction or subsequent retargeting and imitation learning.
Human motion can be observed through external cameras and depth sensors [12], egocentric cameras [35,105], wearable inertial sensors [39], and optical motion-capture systems [23]. Outputs include 2D or 3D keypoints, joint rotations, parametric body models, body meshes, and temporal motion trajectories [12,106]. These representations preserve motion information for downstream interaction and learning while abstracting high-dimensional sensory observations. Figure 2 illustrates a diffusion-based whole-body motion-capture pipeline. We organize the following review around four input settings: third-person vision, first-person vision, inertial measurement unit (IMU) sensing, and hybrid optical-inertial sensing.

3.1.1. Third-Person Vision Motion Capture

We consider third-person vision motion capture, the conventional motion-capture setting in which externally positioned cameras observe the human body. Although cameras provide the most accessible sensing modality, monocular images and videos make 3D pose recovery inherently ambiguous: multiple body configurations can project onto the same 2D joint locations, particularly under self-occlusion or external occlusion [107]. Diffusion-based methods have therefore been introduced to model this one-to-many mapping explicitly. DiffPose [12] employs a conditional DDPM to generate multiple 3D skeletal hypotheses. Its conditioning information is constructed by sampling joint candidates from complete 2D heatmaps and jointly encoding them with a transformer. Consequently, uncertainty produced by the 2D detector is preserved during 2D-to-3D lifting, while inter-joint correlations guide the reverse diffusion process toward coherent whole-body hypotheses. Similarly, D3DP [34] employs a conditional DDPM to generate multiple 3D poses consistent with the same 2D observation. However, while DiffPose preserves detector uncertainty through samples from complete 2D heatmaps, D3DP conditions its denoiser directly on 2D keypoints and uses DDIM sampling to iteratively refine the hypotheses. In this way, the proposed method in DiffPose supports a customizable number of hypotheses by sampling multiple times from a Gaussian distribution. Moreover, the denoiser is trained only once but can be used multiple times to progressively refine the final predictions during inference.
Video supplies an additional source of evidence, but its benefit depends on how temporal information enters the model. The video-based DiffPose [108] estimates a 2D keypoint heatmap for a target frame, neighboring frames condition a DDPM-style denoiser through spatiotemporal features, while DDIM sampling iteratively sharpens difficult joint regions. It is consequently better understood as temporally assisted pose estimation than as full motion reconstruction. The distinction matters because the temporal context can stabilize a pose without requiring the model to recover an entire trajectory. Additionally, SparsePose [11] considers a similar condition in which video is available but annotations are temporally sparse. Its bidirectional forecasting and visual branches carry the main information between labeled frames, whereas the diffusion branch generates a complementary heatmap that is adaptively fused with those predictions.

3.1.2. First-Person Vision Motion Capture.

Having examined motion captured from external viewpoints, we next consider first-person vision, in which the camera is worn by the person whose motion is being reconstructed. Compared with exocentric capture, where missing depth is a central source of ambiguity, egocentric capture is further constrained by limited body visibility: fisheye distortion, self-occlusion, and image truncation leave large portions of the body unobserved. Diffusion models therefore operate primarily as human-motion priors that complete the missing body states from the remaining visual evidence. Both works in [35] and [105] exploit this completion capability, but at different temporal scales. EgoHMR [35] employs a conditional latent diffusion model to recover an SMPL mesh from a single downward-facing image, combining global features with local features specialized for the strongly distorted lower body. In contrast, [105] first estimate body and hand poses using FisheyeViT and then apply a DDPM motion prior to refine the entire sequence. Its uncertainty-aware guidance anchors reliably observed joints while allowing diffusion to correct occluded ones. Thus, diffusion progresses from probabilistic single-frame mesh recovery to evidence-weighted temporal completion, producing more coherent human motion for humanoid observation and imitation.
Table 2. Comparison of representative diffusion-based methods for human-state and motion perception.
Table 2. Comparison of representative diffusion-based methods for human-state and motion perception.
Work Observation → output Diffusion design Main capability and remaining gap
DiffPose [12] Monocular RGB or 2D heatmaps → multiple 3D skeletons Conditional DDPM Preserves detector uncertainty through diverse hypotheses, lacks temporal, contact, and executability constraints.
D3DP [34] 2D pose sequence → refined 3D pose hypotheses Conditional DDPM with DDIM aggregation Iteratively refines multiple hypotheses, hypothesis selection is not connected to robot control.
Video DiffPose [108] Video and target-frame heatmap → target-frame 2D pose Spatiotemporal DDPM with DDIM sampling Uses neighboring frames to stabilize difficult joints, estimates poses rather than executable full-body motion.
EgoHMR [35] Fisheye RGB → SMPL body mesh Hierarchical latent diffusion Completes heavily occluded body regions, human morphology and motion still require humanoid retargeting.
EgoEgo [36] Egocentric video/head trajectory → full-body motion Transformer-based conditional DDPM Models the one-to-many head-to-body mapping, generated motion may differ from the motion that occurred.
Rewind [37] Egocentric video → body and hand motion Distilled causal diffusion Reduces iterative denoising to one-step inference, physical feasibility remains implicit.
IMUDiffusion [38] Six-axis IMU sequences → synthetic inertial sequences DDPM in the time-frequency domain Augments activity-recognition data, does not reconstruct pose or whole-body motion.
GaIP [39] Loose-wear sparse IMUs → SMPL pose Two-stage conditional DDPM Models garment-induced sensor motion before pose recovery, relies on human-specific sensing and representation.
DiffCap [23] 2D joints and sparse IMUs → 3D joints and rotations Two-stage conditional DDPM with DDIM sampling Combines visual geometry with inertial dynamics, omits object interaction and whole-body contact states.
When the body is entirely absent from view, EgoEgo [36] conditions a transformer-based DDPM on the head trajectory recovered by SLAM to generate full-body motion. Unlike the visually grounded methods above, diffusion here models the one-to-many mapping from head motion to the unobserved body, its outputs are temporally plausible but may not reproduce the motion that actually occurred. Rewind [37] instead addresses the computational limitation of iterative diffusion by distilling a multi-step teacher into a causal, single-step body-hand diffusion model. It therefore retains diffusion-based motion completion while enabling real-time egocentric capture. Moreover, diffusion models can support stages surrounding motion reconstruction. EgoPressDiff [109] initializes a conditional video diffusion model from Stable Video Diffusion and combines RGB, depth, pose, and MANO-mesh conditions to recover temporally coherent hand-pressure maps. Whereas the former improves observation quality before pose estimation, the latter extends reconstructed motion with contact information, providing richer cues for humanoid imitation and human-object interaction understanding.

3.1.3. IMU-Based Motion Capture

Wearable IMUs avoid visual occlusion and spatial restrictions, but sparse sensor placement and uncertain sensor attachment make full-body motion recovery underconstrained. In this setting, diffusion models support inertial perception at three levels: generating synthetic sensor observations, completing body motion from sparse measurements, and fusing inertial signals with complementary sensing modalities.
IMUDiffusion [38] focuses on the first role. It applies a DDPM to six-axis accelerometer and gyroscope sequences represented in the short-time Fourier domain and uses the generated sequences to augment training data for human activity recognition. Although such synthetic inertial data can support a broader human-perception pipeline, IMUDiffusion performs neither pose estimation nor full-body motion reconstruction. Garment Inertial Poser [39] further addresses the discrepancy between tightly attached training sensors and loosely worn sensors in realistic settings. Its secondary conditional DDPM synthesizes loose-wear IMU signals conditioned on body poses and corresponding simulated tight-wear IMU measurements, thereby augmenting limited training data with plausible garment-induced sensor motion. Its primary DDPM estimates SMPL joint rotations, joint positions, and corresponding tight-wear IMU signals from observed loose-wear IMUs. The pose-estimation model can additionally incorporate garment-related parameters, including garment style, height, and body mass index, to account for variations in garment fit. Thus, the secondary model learns the conditional distribution of loose-wear sensor observations, whereas the primary model learns the conditional distribution of full-body motion given these sparse and uncertain measurements.

3.1.4. Optical-Inertial Hybrid Motion Capture

The complementary weaknesses of images and IMUs naturally motivate hybrid input. Vision supplies global spatial constraints but becomes unreliable under occlusion, whereas inertial sensing preserves short-term dynamics while suffering from drift and incomplete body coverage. The objective is therefore not simply to concatenate modalities, but to let each constrain the components of motion for which it remains informative.
DiffCap [23] implements this idea as a two-stage conditional DDPM using a monocular camera and six sparse IMUs. Its Joint Diffusion Model first estimates 3D joint positions from globally encoded 2D keypoints and frame-wise inertial measurements. The recovered positions and IMUs then condition a Pose Diffusion Model that estimates joint rotations. This decomposition allows visual position cues to correct inertial drift, while inertial dynamics constrain visually occluded or rapidly moving joints. It also avoids asking a single denoiser to recover positions and rotations with different geometric structures. Transformer denoisers capture temporal dependencies, and short DDIM sampling brings the learned multimodal prior closer to real-time use.
Ego4o [110] provides an informative contrast rather than another diffusion method. It accepts flexible combinations of egocentric images, one to three IMUs, and textual motion descriptions, but maps them to discrete part-aware motion codes using a VQ-VAE and transformer-based multimodal encoder. Semantic descriptions can be fed back to reduce ambiguity when physical sensing is sparse. The comparison shows that successful hybrid capture depends first on representing modality reliability and body-part correspondence, diffusion is one mechanism for modeling the resulting conditional distribution, but it is not the only possible mechanism.

3.2. Diffusion Models for Embodied Self-Perception

Self-perception enables humanoid robots to construct an internal representation of their own body and physical interaction state from visual and sensory observations. This representation includes quantities such as joint configuration, floating-base motion, contact state, and external forces. However, similar to human perception, self-perception is inherently ambiguous because of visual occlusion and sparse or noisy sensor measurements. Consequently, multiple body or interaction states may be consistent with the same observations. Diffusion models can therefore be leveraged to address this ambiguity.

3.2.1. Recovery of Proprioception from Multimodal Observations

Because of observation ambiguity, a humanoid robot must determine whether its sensory observations are complete and reliable before estimating its configuration, floating-base motion, or contact state. At the most general level, diffusion-based observation recovery can be understood as recovering missing observations or learning conditional distributions over their corrupted components to provide additional usable information [111,112]. For example, [40] models the relationship among joint angles, motor-side muscle lengths, and muscle tensions in a musculoskeletal robot. Given joint-angle conditions, gradient-guided denoising estimates physically compatible muscle lengths and tensions while enforcing abnormal-state constraints, such as zero tension for a ruptured muscle or a fixed length for a jammed actuator. Similarly, [113] first identifies potentially corrupted state and action samples and then trains two conditional diffusion models: one restores noisy states from the corresponding actions, whereas the other restores actions from states. Experiments on simulated Fetch-arm manipulation, Shadow-Hand manipulation, and bipedal locomotion show that the restored demonstrations provide more reliable supervision for subsequent imitation learning.
Figure 3. Overview of TacImag [41] for touch-informed manipulation using generated tactile observations. (a) Tactile imagination learning: visual and proprioceptive inputs are encoded and fused to condition diffusion models that generate tactile observations from Gaussian noise. Real tactile measurements from paired visuotactile demonstrations provide supervision. Separate models synthesize tactile RGB images capturing contact appearance and texture, or tactile force fields representing distributed normal and shear forces. (b) Touch-informed policy learning: the frozen imagination model generates tactile observations from visual and proprioceptive inputs. A tactile encoder extracts features from these observations, which are concatenated with visual and proprioceptive features. A diffusion policy learns from expert demonstrations to map the resulting observation history to end-effector and gripper action sequences, without updating the imagination model. At deployment, tactile observations are synthesized online from vision and proprioception, enabling manipulation without physical tactile sensors.
Figure 3. Overview of TacImag [41] for touch-informed manipulation using generated tactile observations. (a) Tactile imagination learning: visual and proprioceptive inputs are encoded and fused to condition diffusion models that generate tactile observations from Gaussian noise. Real tactile measurements from paired visuotactile demonstrations provide supervision. Separate models synthesize tactile RGB images capturing contact appearance and texture, or tactile force fields representing distributed normal and shear forces. (b) Touch-informed policy learning: the frozen imagination model generates tactile observations from visual and proprioceptive inputs. A tactile encoder extracts features from these observations, which are concatenated with visual and proprioceptive features. A diffusion policy learns from expert demonstrations to map the resulting observation history to end-effector and gripper action sequences, without updating the imagination model. At deployment, tactile observations are synthesized online from vision and proprioception, enabling manipulation without physical tactile sensors.
Preprints 244459 g003
Observations can also be recovered from multimodal inputs when an entire modality is unavailable and must be inferred from other sensory streams. TacImag instantiates this cross-modal principle more explicitly by learning a conditional diffusion model that generates tactile observations from vision and robot proprioception [41]. The generated modality can be represented as either a tactile RGB image or a distributed normal-and-shear force field and is supplied to a tactile-conditioned manipulation policy when no tactile sensor is available at deployment. Experiments on simulated and real contact-rich manipulation tasks show that the imagined observations improve downstream performance. Cross-Sensor Touch Generation addresses a complementary sensor-substitution problem [42]. Its Touch2Touch model uses latent diffusion to translate the observation of one tactile sensor into the signal expected from another, whereas its Touch-to-Depth-to-Touch variant introduces contact depth as an intermediate representation to reduce the need for paired sensor data. The models are trained using tactile observations collected by a KUKA robot with Soft Bubble, GelSlim, and DIGIT sensors, enabling sensor-specific perception and manipulation algorithms to be transferred across hardware.

3.2.2. Explicit Configuration and Dynamic-State Inference

For visual configuration recovery, RoboKeyGen estimates the camera-relative pose and joint angles of an articulated robot from RGB observations by decomposing the task into 2D keypoint detection, 3D keypoint lifting, and model-based pose fitting [43]. Instead of deterministically regressing 3D coordinates, its conditional diffusion model generates 3D keypoint configurations from normalized 2D detections, thereby better accommodating missing or noisy keypoints and depth ambiguity. In contrast, PoseDiff directly denoises structured robot states, such as 3D keypoints or joint angles, from a single RGB image [44]. PoseDiff further adapts this visual-to-state formulation to inverse dynamics, generating dense action sequences from pairs of sparse video keyframes and averaging overlapping predictions to maintain temporal continuity, it achieves state-of-the-art pose-estimation performance on DREAM and substantially improves offline manipulation success on LIBERO-Object. Together, these methods illustrate the progression of diffusion models from a probabilistic 3D reconstruction module to an end-to-end interface connecting visual robot-state perception with executable control.
Moving from per-frame configuration to temporal state, DnD Filter introduces diffusion directly into a recursive estimator [45]. A conventional prediction step first propagates the previous state, after which the predicted state and current observation are fused to condition a diffusion model that produces the corrected state. Diffusion therefore replaces the restrictive measurement-update mapping of conventional differentiable filters with a nonlinear generative update. DiffPF develops this filtering formulation into a particle-based posterior estimator [46]. Instead of producing only a corrected state, its conditional diffusion model generates equally weighted particles from the filtering distribution using the predicted particle set and current observation as conditions. This avoids explicit importance weighting and resampling while preserving multimodal hypotheses. Denoising Particle Filters pursue a more modular alternative [114]. DnPF learns dynamics and measurement models from individual state transitions. At inference, particles are initialized from noise-perturbed dynamics predictions, while an observation-conditioned denoising score guides them toward the measurement-supported state manifold.
Table 3. Comparison of diffusion methods for embodied self-perception.
Table 3. Comparison of diffusion methods for embodied self-perception.
Method Observation → estimated state Generative design Main capability and remaining gap
Body-schema diffusion [40] Joint angles and fault constraints → muscle lengths and tensions Gradient-guided conditional diffusion Infers physically compatible internal variables under faults, no floating-base or whole-body validation.
TacImag [41] Vision and proprioception → tactile image or force field Conditional diffusion Supplies imagined touch when tactile hardware is absent, reliability under novel contacts remains uncertain.
Touch2Touch [42] Source tactile sensor → target-sensor observation Latent diffusion, optionally through contact depth Transfers tactile perception across hardware, depends on sensor registration and cross-domain calibration.
RoboKeyGen [43] RGB and 2D keypoints → 3D keypoints, pose, and joint angles Conditional keypoint DDPM Represents depth ambiguity before model-based fitting, evidence is mainly limited to articulated arms.
PoseDiff [44] RGB or sparse video keyframes → robot states or dense actions Structured state/action diffusion Connects visual pose inference with control, floating-base motion and contact are not modeled.
DnD Filter [45] Predicted state and current observation → corrected state Diffusion-based recursive measurement update Provides a nonlinear generative correction, control-rate humanoid estimation is not demonstrated.
DiffPF [46] Predicted particles and observation → posterior particles Conditional diffusion sampler Preserves multimodal posteriors without importance weighting, scalability to high-dimensional humanoid states is open.
DnPF [114] Dynamics predictions and observation → denoised particles Observation-conditioned score guidance Separates learned dynamics and measurement models, lacks physical-humanoid validation.
CDM [47] Joint torques and base force/torque → surface-contact locations Conditional DDPM with signed-distance guidance Generates sequential multi-contact hypotheses, predicts neither contact forces nor floating-base support state.
Multimodal Diffusion Forcing [48] Partial multimodal trajectory → missing wrench channel Multimodal diffusion forcing Reconstructs force and detects anomalies, the wrench is not localized on the robot body.

3.2.3. Toward Diffusion-Based Whole-Body Perception

Whole-body contact perception requires a robot to recover not only whether an interaction occurs, but also where it occurs and which wrench it applies to the body. These variables are observed only indirectly through joint torques, inertial motion, and other proprioceptive responses. Since different combinations of contact locations and forces can produce similar measurements, the inverse mapping is intrinsically ambiguous and therefore well suited to distributional inference. Nevertheless, current diffusion-based evidence remains fragmented across different components of this problem.
The most direct precedent is the contact diffusion model (CDM) [47], which estimates a multimodal distribution over contact locations from joint-torque measurements and a force/torque sensor mounted at the robot base. Instead of selecting a single location that explains the measurements, CDM generates multiple plausible contact hypotheses. Its denoising process is conditioned on previous estimates to resolve sequential multi-contact events and incorporates a signed distance field to constrain the generated points to the robot surface. Experiments on a real Kinova Gen3 demonstrate accurate single- and dual-contact localization. However, CDM is limited to a fixed-base manipulator and predicts locations, leaving simultaneous force identification as future work.
A complementary direction treats interaction force as a sensory variable that can be recovered from other observations. Multimodal Diffusion Forcing can reconstruct a missing wrench channel from the remaining robot trajectory and identify when external disturbances produce anomalous force measurements [48]. However, the reconstructed wrench is not associated with a particular location on the robot body. Humanoid perception requires these two capabilities to be combined because external interactions may occur at arbitrary body regions and affect the coupled motion of the floating base and articulated limbs. SixthSense provides the closest realization of this objective by estimating contact timing, body region, and six-dimensional wrench from proprioception and IMU measurements on a Unitree G1 [49]. Its conditional flow-matching model represents contacts as a sparse spatiotemporal wrench field over the robot surface. Nevertheless, it uses conditional flow matching. It therefore demonstrates the feasibility of generative whole-body contact perception without providing direct evidence for diffusion models on humanoids.
This gap reflects the complexity of whole-body perception. Floating-base dynamics and simultaneous contacts create a high-dimensional, multimodal posterior, while dense contact-and-wrench ground truth is difficult to obtain on physical humanoids. Iterative denoising must also meet the high update rates required by balance and collision-response controllers. Consequently, diffusion-based whole-body perception will require structured body representations, physical constraints, and accelerated sampling, joint contact-and-wrench inference on a physical humanoid remains an open problem.

3.3. Diffusion Models for Environment Understanding

The human states recovered in the subSection 3.1 and Section 3.2 acquire meaning only when they are situated in the surrounding world. A reaching motion must be related to an object, its pose, and the free space around it, a walking motion must be interpreted against support geometry and moving obstacles. For a humanoid, environment understanding must therefore recover several connected forms of external state: the geometry that supports locomotion, the objects and interaction regions that enable manipulation, and the motion that determines how long the current state remains valid.
Diffusion models enter this process at different levels. Some directly denoise the required state, such as a depth map, occupancy volume, or scene-flow field. Others use a learned generative prior to complete geometry that the sensors have not measured. A third group abandons sampling altogether and repurposes the representations learned by a pretrained diffusion model for discriminative perception. These uses are related by a common objective, recovering external structure from incomplete observations, but they provide different outputs and different forms of uncertainty. Direct validation on humanoids remains rare, most evidence comes from perception benchmarks, autonomous-driving data, or other robotic embodiments. The following discussion therefore traces what each line of work actually estimates before considering how that estimate can support humanoid behavior.

3.3.1. Geometric Reconstruction and Spatial Completion

Humanoid locomotion and manipulation require coherent estimates of surrounding surfaces. DDP formulates dense prediction as noise-to-map generation, progressively denoising Gaussian noise conditioned on RGB images to produce depth, semantic segmentation, and bird’s-eye-view maps [50]. Marigold adapts a pretrained latent text-to-image diffusion UNet for monocular depth, inheriting broad visual priors that improve generalization beyond its synthetic fine-tuning data [51]. However, its affine-invariant predictions require anchoring before supporting physical decisions such as foot clearance or reaching distance.
The cost and stochasticity of iterative denoising motivate adaptations for frequent, stable geometric updates. VPD repurposes the pretrained diffusion network as a visual backbone, using text adaptation and image-text cross-attention to support depth and segmentation heads [115]. GenPercept converts the diffusion UNet into a deterministic one-step predictor with specialized decoders and pixel-level supervision for depth, normals, segmentation, and matting [116]. Both retain diffusion-pretrained representations while avoiding iterative sampling at deployment. Efficient prediction nevertheless requires metric grounding. Marigold-DC addresses this through sparse-depth-guided optimization alongside denoising, without retraining the monocular model [117]. Image context and pretrained depth priors provide dense structure, while measured samples anchor the reconstruction in metric space.
Beyond reconstructing observed surfaces, spatial completion estimates occluded or unseen regions. SceneSense conditions three-dimensional occupancy diffusion on RGB-D observations and a running map, preserving observed free and occupied voxels through inpainting while generating only unknown space [52]. Its online extension removes RGB-D cross-attention after finding limited benefit once occupancy inpainting is available, reducing inference cost and enabling prediction at arbitrary map frontiers [118]. Predictions are fused probabilistically into an OctoMap-style map on a physical Spot quadruped, allowing subsequent measurements to correct inferred occupancy. A later system integrates the enhanced map with an off-the-shelf exploration planner and reports improved map quality, traversability time, and exploration robustness [119]. This progression connects diffusion-based completion with robotic planning, provided that generated occupancy remains distinguishable from direct sensor evidence.

3.3.2. Object-Centric Semantic and Interaction State

Object-centric perception connects an object’s pose and identity to potential interactions. GenPose [120], illustrated in Figure 4, samples multiple 6D pose candidates from a partial point cloud to represent ambiguity caused by occlusion and symmetry. A learned energy-based model estimates relative likelihoods for ranking and filtering, after which the retained poses are aggregated. These estimates could support humanoid alignment and manipulation planning, but pose alone does not establish language-grounded identity or affordances. Diff2Scene addresses semantic grounding by transferring frozen text-to-image diffusion representations to open-vocabulary 3D segmentation and visual grounding [53]. Saliency-aware and geometry-aware masks associate pretrained semantic features with point-cloud structure without labeled 3D training data. Here, diffusion supplies representations rather than sampling the semantic map.
Manipulation further requires hidden geometry and action-relevant regions. GOM embeds a pretrained multi-category 3D diffusion prior within probabilistic object-level mapping [54]. Sparse multi-view RGB-D observations constrain joint estimation of object pose and complete NeRF geometry and appearance, tying plausible unseen structure to the observed instance. DAG instead predicts a dense 3D affordance mask from a human-object interaction image and an affordance concept [55]. It fuses multi-scale features from a frozen text-to-image diffusion UNet with language and point-cloud features. GOM therefore completes object structure, whereas DAG identifies where an intended interaction may be applied.
When the hand occludes an object, touch supplies additional geometric constraints. Touch2Shape conditions latent 3D diffusion on tactile observations and optional visual input to recover global shape, while a touch-shape fusion module restores local detail [56]. Its denoised latent representation also guides a reinforcement-learning policy toward informative subsequent touches. TouchDiffusion reconstructs shape without vision [24]: tactile images are converted into local point clouds, registered using robot forward kinematics, and used to condition global shape denoising. Both methods constrain contacted surfaces with measurements, while uncontacted regions remain dependent on learned object priors.
Caddeo et al. provide direct humanoid evidence by combining visual, tactile, and embodied constraints [121]. Their model generates a metric, pose-aware signed-distance field through conditional flow matching in a compact latent space. RGB observations and visibility masks describe visible surfaces, proprioception supplies posed hand geometry, and multi-contact touch constrains occluded regions. Physics-based objectives and decoder guidance reduce hand-object interpenetration and align reconstructions with measured contacts. Transfer is demonstrated on a real humanoid with an end effector different from those used in training. This method uses flow matching rather than a conventional discrete DDPM. Across these studies, semantic features identify objects, generative priors infer hidden structure, affordances localize interaction regions, and physical contact constrains plausible completions.
Table 4. Comparison of representative diffusion-based methods for environment understanding.
Table 4. Comparison of representative diffusion-based methods for environment understanding.
Method Observation → environment state Role of diffusion Main capability and remaining gap
DDP [50] RGB → depth, segmentation, or BEV map Direct output-space diffusion Unifies several dense-prediction tasks, iterative inference and metric grounding limit online use.
Marigold [51] RGB → affine-invariant depth Fine-tuned latent diffusion prior Generalizes beyond its synthetic training data, metric scale is required for reaching and locomotion.
VPD / GenPercept [115,116] RGB → depth, normals, segmentation, or matting Diffusion features used by task heads or a one-step predictor Avoids iterative sampling, largely sacrifices distributional output and does not ensure metric geometry.
Marigold-DC [117] RGB and sparse depth → dense metric depth Measurement-guided latent denoising Anchors a monocular prior with metric samples, optimization during inference remains costly.
Diff2Scene [53] Point cloud, image features, and language → open-vocabulary 3D labels Frozen text-to-image diffusion features Transfers semantic correspondence without labeled 3D data, does not infer interaction physics.
GOM [54] Sparse multi-view RGB-D → object pose and complete NeRF Pretrained 3D diffusion prior in probabilistic optimization Completes unseen object geometry, generated shape may violate articulation or hand-contact constraints.
DAG [55] Human-object image, language, and point cloud → 3D affordance mask Frozen diffusion features fused with 3D and language features Grounds open-vocabulary interaction regions, affordance does not guarantee reachability or force feasibility.
Touch2Shape [56] Touch, optionally RGB → complete object shape Tactile-conditioned latent 3D diffusion Supports active tactile exploration, uncontacted surfaces remain prior-dependent.
TouchDiffusion [24] Tactile images and forward kinematics → complete object shape Touch-conditioned 3D diffusion Enables vision-free reconstruction, accuracy depends on contact registration and object priors.
DiffSF [58] Source and target point clouds → scene-flow distribution Conditional diffusion over the complete vector field Produces multiple motion hypotheses, sampling cost and closed-loop robot evidence remain limited.
MID [59] Observed pedestrian trajectories and social context → future paths Motion-indeterminacy diffusion Represents multimodal human futures, lacks closed-loop humanoid navigation evaluation.

3.3.3. Dynamic and Predictive Environment State

Even a geometrically and semantically grounded scene remains a snapshot. A humanoid must also estimate how the observed geometry is moving and which future motions may follow. Scene flow supplies the first level of this dynamic state by assigning a three-dimensional displacement to each observed point. DifFlow3D adds diffusion to a conventional scene-flow pipeline as an iterative refinement module [57]. Flow-related features condition the reverse process, and a separate uncertainty module assesses the reliability of the refined estimate. DiffSF instead treats the complete scene-flow field as the diffusion target [58]. It corrupts the ground-truth vector field during training and recovers it from Gaussian noise at inference, conditioned on the source and target point clouds. Repeated inference yields multiple flow hypotheses whose variation can reveal ambiguous regions. The two methods therefore use diffusion at different depths: one repairs an estimate from an existing pipeline, whereas the other learns the conditional distribution of the entire displacement field.
Predicting beyond the observed frames changes the unit of reasoning from point motion to the possible trajectories of whole agents. MID formulates pedestrian trajectory prediction as the reverse of a motion-indeterminacy diffusion process [59]. Observed trajectories and social interactions are encoded as a state embedding, and a transformer-based denoiser progressively reduces uncertainty from the walkable region to a future path. Sampling can therefore retain several futures compatible with the same history, and the method is evaluated on human-trajectory benchmarks including ETH/UCY and the Stanford Drone Dataset. This evidence is closer to human-aware humanoid navigation than traffic-agent forecasting, although it still stops short of closed-loop robot evaluation.
Together, scene flow and pedestrian forecasting extend environment understanding from present motion to future occupancy. Their samples are useful only if downstream planning preserves the distinction between observed motion and inferred futures: variation among samples indicates ambiguity, but does not by itself constitute a calibrated collision probability. Safe deployment therefore requires temporal conditioning, uncertainty calibration, and a planner that responds to the supported set of motions rather than selecting only the most convenient prediction.

4. Diffusion-Based World Models for Humanoid Prediction and Action Reasoning

Section 3 examined how diffusion models infer the current human-robot-environment state from incomplete sensory observations. These estimates provide the starting point for prediction: the robot’s posture, surrounding geometry, and nearby human motion inform how the situation may evolve. A world model extends this perceptual representation across time by learning dynamics that predict future observations or states and, when conditioned on candidate actions, their possible consequences. Coupling these predictions to planning or policy learning enables action evaluation before physical execution [122,123]. For humanoids, this connection supports future-state estimation, counterfactual evaluation, and failure anticipation [30]. Figure 5 illustrates this connection through Puppeteer [124]. Its world model encodes visual observations and proprioceptive state into a latent representation. Predicting how this representation evolves under candidate motion commands enables a planner to select commands for the task. The low-level tracking world model then plans joint-level actions to follow them. This architecture illustrates how sensory information provides the current representation, learned dynamics predict the consequences of possible commands, and planning converts these predictions into control.
Diffusion-based world models are attractive as they learn conditional distributions over temporally extended futures and can therefore represent multiple plausible rollouts [60,125]. This property is important for humanoid settings, where human motion is multimodal [126], and where object interactions, contact transitions, and long-horizon task execution may have several valid outcomes. Nevertheless, stochastic generation alone does not make a predictive model useful for decision making. Its rollouts must preserve task-related state, respond consistently to candidate actions, and remain compatible with the robot’s embodiment and physical constraints [4,124]. We organize this section around three progressively stronger functions. First, world models predict future observations or states in visual, latent, or explicitly structured representations. Second, action-conditioned models estimate how those futures change under robot interventions, using either explicit robot commands or latent actions recovered from unlabeled video. Third, the predicted futures are coupled to planning, policy learning, or candidate-action evaluation.

4.1. Observation-Space Video Prediction

The first function of a world model is to predict how the currently observed state may evolve. Observation-space world models predict future RGB or multimodal observations [127,128], allowing environmental changes and action consequences to be inspected in the robot’s visual input space. They also preserve fine visual details that compressed states may omit. Large-scale video diffusion models provide useful appearance and motion priors [129,130]. However, passive video generation alone is insufficient for decision making. A predictive world model must be grounded in the agent’s observation history and predict futures consistent with candidate actions and physical dynamics.
Diffusion-based world models extend video generation by conditioning future observations on the current environment or interaction history. DIAMOND uses a diffusion model as a pixel-level world model for reinforcement learning and demonstrates that preserving visual details can improve policy learning in Atari environments [60]. Its extension to Counter-Strike: Global Offensive further demonstrates that the same approach can scale to a complex 3D environment and operate as an interactive neural game engine. This ability is important for humanoid robotics, where subtle visual changes, such as locomotion and contact formation, can determine the outcome of a whole-body action. Pixel-level diffusion world models could thus generate imagined trajectories for training and evaluating humanoid policies, reducing the need to perform every exploratory behavior on physical hardware. UniSim extends this direction by learning an interactive real-world simulator from broad and diverse data encompassing objects, scenes, human activities, navigation and manipulation motions, panoramic scans, simulations, and renderings [61]. UniSim is built upon a video diffusion model trained to predict a sequence of future observation frames conditioned on past observations and input actions. This formulation enables UniSim to accommodate complex actions when simulating realistic interactions, with an emphasis on diverse, action-rich, and long-horizon interactions involving humans and robots.
However, due to variations in hardware platforms and the high cost of data collection, the limited coverage of humanoid robot data makes modeling world dynamics particularly challenging for contact-rich humanoid tasks. DreamDojo addresses this challenge by pretraining a world model on large-scale egocentric human videos using latent action labels, then adapting it to target robots [62]. This transfers interaction priors from human activities to robot-conditioned future prediction. To reduce inference cost, DreamDojo distills a 35-step teacher into a four-step autoregressive student. On GR-1 Long Eval, which evaluates one-minute, multi-stage humanoid rollouts, the student achieves 10.81 FPS on a single NVIDIA H100 GPU versus 2.72 FPS for the teacher, yielding approximately 4 × faster generation. Such efficiency is particularly relevant to humanoid robotics because it supports responsive applications such as live teleoperation, policy evaluation, and online model-based planning, where predicted consequences must be updated as the robot acts and receives new observations. These methods illustrate the main strength of observation-space world models that they generate futures in a form that is directly inspectable and compatible with visual policies, human evaluation, and synthetic data generation. They can retain fine visual changes that may be related to contact, object motion, or task progress, without requiring an explicitly designed state representation.
Nevertheless, pixel-level prediction is expensive because the model must reproduce textures, lighting, backgrounds, and other visual details that may be irrelevant to robot control [131,132]. Autoregressive rollout can also accumulate image artifacts, identity changes, and geometric inconsistencies, while high perceptual quality does not guarantee correct dynamics [133]. Thus, a visually plausible generated sequence may still respond incorrectly to the agent’s action, violate object permanence, or misrepresent contact outcomes. These limitations motivate prediction in more compact or explicitly structured spaces.

4.2. Compact State-Space World Models

Generating complete future images is not always necessary for embodied decision making. A humanoid can instead predict future dynamics in either a compact learned latent space [81,87,123] or an explicitly structured state space [134]. Latent states aim to retain variables needed for prediction and control while discarding task-irrelevant appearance variation. Structured states make entities such as geometry, objects, relations, motion, or contacts more explicit. Although these categories may overlap, both approaches can reduce the computational burden of combining high-dimensional visual observations, proprioception, and actions during whole-body control.

4.2.1. Learned Latent-State World Models

Latent diffusion world models reduce computational cost by performing denoising in compact feature spaces. DreamerAD formulates diffusion as a latent dynamics model for imagination-based policy learning [63]. It extracts denoised features from a video diffusion transformer as predictive states that encode the spatial, semantic, and dynamic consequences of candidate actions. Shortcut forcing reduces future prediction from 100 denoising steps to a single step, while the latent predictions remain decodable into RGB observations. A learned reward model evaluates the resulting action-conditioned features, enabling reinforcement learning and trajectory optimization directly within the latent imagination space. For humanoid robots, this mechanism provides a computationally efficient means of evaluating imagined robot trajectories, while avoiding repeated pixel-level generation.
LaDi-WM demonstrates that predictive manipulation does not require reconstructing future RGB observations [64]. As shown in Figure 6, it uses diffusion to forecast future states in a latent space combining DINO-based geometric features with CLIP-based semantic features, and then conditions a diffusion policy on these predictions to refine its initial actions. Results on LIBERO-LONG and real-robot tasks show that such compact forecasts improve manipulation performance without pixel-level rollouts. For humanoids, this formulation could preserve anticipated hand-object geometry and task semantics as efficient predictive context for bimanual manipulation and whole-body loco-manipulation [135,136]. More direct evidence for humanoid robots is provided by FLARE [65]. Specifically, FLARE augments a flow-matching action policy by aligning features from a diffusion transformer with latent embeddings of future observations. Thus, it enables a diffusion transformer policy to anticipate latent representations of future observations, allowing it to reason about long-term consequences of its actions. FLARE is evaluated using task success rate on simulated GR-1 tabletop tasks, and manipulation tasks on a physical GR-1 humanoid. These tasks include 18 object rearrangement tasks, picking up and placing objects between source and target containers, and 6 tasks involving interaction with articulated objects such as cabinets, drawers, and microwaves. Compared to the diffusion policy [137] and GR00T N1 [135] baselines, FLARE achieved an average success rate of 55.0%, compared to 40.9% for diffusion policy and 45.1% for GR00T N1 on the simulated GR-1 benchmark. FLARE therefore provides direct humanoid evidence that predictive latent structure can be integrated into a generative policy without requiring a separate video-generation stage.

4.2.2. Explicit Geometric, Relational, and Contact-State Prediction

When success depends on precise spatial or physical events, learned features can be supplemented or replaced by representations that expose more of the structure. Copilot4D occupies an intermediate point between generic latent prediction and explicit state modeling [66]. It first uses a VQ-VAE to tokenize LiDAR point clouds into discrete bird’s-eye-view codes and then predicts future tokens with discrete diffusion conditioned on previous observations and ego poses. The parallel denoising process captures multimodal future point-cloud observations while avoiding regression directly in a large, unstructured sensor space.
VectorWorld makes those relations explicit [67]. It represents an autonomous-driving scene as vector-graph tiles containing lanes, agents, and their interactions, and generates new ego-centric tiles as the vehicle moves. A motion-aware gated VAE initializes a policy-compatible interaction state, while an edge-gated relational diffusion transformer models the heterogeneous map and agent elements. Unlike Copilot4D, which forecasts tokenized point-cloud observations, VectorWorld directly propagates typed graph elements and their relations. Its kilometer-scale driving rollouts may not constitute humanoid evidence, but they show how an explicit relational state can support efficient generation and long-horizon physical constraints.
RoboFlow4D transfers the structured-prediction idea from driving layouts to robot manipulation [68]. Instead of generating an entire future scene, it predicts how three-dimensional points should move over multiple future frames. Visual and language features are enriched with a distilled 3D prior, and a diffusion-based FlowDiT denoises future 3D flows conditioned on the resulting multimodal representation. These flows provide an explicit motion plan to a downstream action policy. A slow-fast architecture then updates the flow prediction at a lower rate while the policy executes at a higher rate, forming an observation-planning-execution loop. The predicted state is consequently both lighter than video and more directly related to physical motion.
Geometry alone, however, can omit sparse but decisive interaction events. Contact-aware models enrich the predictive state with tactile variables that may be weakly observable from RGB images. Dream-Tac develops this idea as a unified tactile world-action model for contact-rich manipulation [69]. It maps visual observations, tactile observations, robot states, and action chunks into the latent interface of a pretrained video Diffusion Transformer, which jointly denoises future visual tokens, future tactile tokens, and robot actions. Because tactile signals are sparse and transient, remaining nearly constant between events such as contact onset, slip, and release, Dream-Tac computes a deterministic contact gate from frame-to-frame changes in tactile images. This design preserves the priors of the pretrained backbone while allowing future contact dynamics to guide action generation. Across six real-world contact-rich tasks, the paper reports an average success rate of 83.3%, substantially outperforming Cosmos-Policy [140] (51.7%), ForceVLA [141] (50.8%), π 0.5 [142] (45.0%), and π 0 [143] (30.8%).
Table 5. Comparison of compact latent and structured-state diffusion world models.
Table 5. Comparison of compact latent and structured-state diffusion world models.
Method Predicted state and interface Generative design Evidence/use Remaining humanoid gap
DreamerAD [63] Action-conditioned predictive video features, decodable to RGB Latent video DiT with shortcut forcing Reduces nominal denoising from 100 steps to one for driving control Vehicle dynamics differ from articulated contact-rich interaction.
LaDi-WM [64] Future DINO/CLIP features used by a diffusion policy Conditional latent-feature diffusion Reports + 27.9 % on LIBERO-LONG and + 20 % on real-world tasks Compact features may omit balance, contact, and support state.
FLARE [65] Future-observation embeddings jointly learned with robot actions Diffusion Transformer with future-feature alignment Achieves 55.0% versus 40.9% for Diffusion Policy and 45.1% for GR00T N1 on GR-1 Not a separately queryable simulator, evidence focuses on tabletop manipulation.
Copilot4D [66] LiDAR history and ego poses → future BEV tokens Discrete diffusion over VQ-VAE codes Captures multimodal future point-cloud observations in parallel Driving representation omits articulated-body and contact state.
VectorWorld [67] Lane/agent graph → future vector-graph tiles Relational diffusion transformer Supports efficient kilometer-scale streaming rollouts Requires humanoid-specific body, object, and contact relations.
RoboFlow4D [68] Visual/language/3D context → multi-frame 3D flow Diffusion-based FlowDiT with slow-fast execution Provides explicit motion guidance to a high-rate manipulation policy Models end-effector motion rather than whole-body contacts and forces.
Dream-Tac [69] Vision, touch, robot state, and actions → future visual/tactile tokens and actions Contact-gated multimodal diffusion transformer Reports 83.3% average success across six real-world contact-rich tasks Covers end-effector contact, not feet, balance, or distributed whole-body interaction.
Overall, compared with monolithic pixel prediction, these structured representations introduce explicit spatial and relational inductive biases. They can improve geometric consistency and reduce the amount of irrelevant information that must be generated.

4.3. Action-Conditioned Humanoid World Models

Section 4.1 and Section 4.2 distinguish world models according to the state space in which they predict the future. This subsection instead examines how actions enter the predictive process. A decision-making model must estimate not only what is likely to happen next, but also how alternative robot actions would change the future. We distinguish explicit action conditioning, which uses known robot commands, from latent-action conditioning, which infers action-like transition variables from unlabeled information.

Explicitly Action-Conditioned World Models.

When synchronized robot controls are available, explicitly action-conditioned world models predict future observations under a policy’s action chunks. Ctrl-World supports such policy-in-the-loop rollouts for generalist manipulation [70]. Initialized from a pretrained video diffusion model, it combines three mechanisms. Joint multi-view prediction integrates third-person and wrist-camera observations while retaining a VLA-compatible visual interface. Frame-level action conditioning aligns high-frequency commands with visual changes rather than assigning one coarse action to an entire clip. Pose-conditioned memory retrieval adds sparse historical frames and projects their pose information through frame-level cross-attention, re-anchoring generation to similar past states and reducing long-horizon drift. Trained on 95,000 DROID trajectories from 564 scenes, Ctrl-World generates multi-view rollouts exceeding 20 seconds and ranks policy checkpoints without executing each on the robot. Post-training aligns the model with new instructions and increases downstream success from 38.7% to 83.4%.
VLAW extends Ctrl-World into iterative policy–world-model co-improvement [71]. At each iteration, the current policy is deployed on the real robot to collect on-policy trajectories covering both successful and failed interactions, including contact-rich behaviors and deformable-object dynamics. These data update the action-conditioned world model to reduce the distribution gap of demonstration-only training. The refined model generates additional synthetic rollouts to improve the VLA policy, and the cycle repeats. On DROID, this process yields a 39.2% absolute improvement over DSRL [144].
DreamPolicy addresses humanoid locomotion with a terrain-aware autoregressive diffusion world model [72] (Figure 7). Conditioned on recent proprioceptive states, user commands, the previous joint action, and a robot-centric terrain height map, it generates body-relative state trajectories that retain balance, contact, and whole-body motion information while remaining invariant to global position and heading. These trajectories provide dynamic goals for a goal-conditioned reinforcement-learning policy: an immediate tracking reward encourages the policy to follow the next predicted body state, while an adversarial style reward aligns longer policy rollouts with the motion manifold learned by diffusion. This separation is important because terrain-dependent motion priors need not be dynamically feasible under direct execution. Trained on aggregated rollouts from terrain-specialized policies, DreamPolicy transfers zero-shot to unseen and composite terrains on Unitree G1.
For humanoid manipulation, these systems highlight the need for synchronized action–observation data and a clearly defined control interface specifying each action’s semantics, temporal resolution, and embodiment-specific meaning [145,146]. Joint positions, Cartesian end-effector commands, and high-level skills have different meanings across platforms and controllers. Such humanoid world models require platform-specific training or embodiment-aware action representations that preserve each action’s causal effect.

Latent-Action World Models.

Human videos capture rich body motion and physical interactions but rarely provide action labels for controllable world-model training. Latent-action learning addresses this limitation by compressing transitions between consecutive observations into action-like variables that serve as a shared control interface across videos and embodiments [147]. In diffusion-based world models, these variables condition denoising to generate futures consistent with specified underlying motion or interaction.
Olaf-World separates learning this control interface from diffusion-based prediction [73]. It first trains a β -VAE latent-action model and introduces Seq Δ -REPA to align the integrated latent-action trajectory with corresponding temporal feature changes measured by a frozen self-supervised video encoder. This alignment encourages consistent action semantics across visual contexts. The resulting LAM is frozen and applied to unlabeled videos to extract per-frame latent actions for action-aware world-model pretraining. A pretrained latent image-to-video diffusion transformer is then fine-tuned using a flow-matching objective. Each latent action is projected and fused with the diffusion-timestep embedding to produce AdaLN-Zero parameters that modulate every DiT block. Because the 3D video VAE compresses time fourfold, four consecutive actions are grouped to condition each video-latent timestep. Diffusion therefore predicts future evolution, while the aligned LAM supplies a transferable control interface that guides the generated dynamics.
CoLA-World removes this separation by jointly learning latent actions and diffusion dynamics [74]. It replaces the LAM’s conventional forward dynamics model with a pretrained OpenSora diffusion-based video generator [148]. A transformer and vector quantizer infer discrete actions that modulate diffusion blocks through adaptive normalization. The flow-matching loss updates both the world model and action modules, its gradients propagate through diffusion into the inverse-dynamics encoder, allowing the video-generation prior to shape the latent-action representation. During warm-up, OpenSora is frozen while training the action modules, followed by joint optimization of the complete system. Whereas Olaf-World trains diffusion with a fixed, explicitly aligned latent space, CoLA-World allows the world model and latent actions to improve one another during training.
DreamDojo applies the two-stage formulation directly to humanoid world modeling [62]. A spatiotemporal VAE first extracts continuous proxy actions from large-scale egocentric human videos. These actions are grouped into temporal chunks, projected into the diffusion-timestep space, and injected through adaptive normalization into every DiT block of Cosmos-Predict2.5, a pretrained latent video diffusion model. The diffusion world model is trained with flow matching and an additional temporal-consistency loss that emphasizes changes between successive video latents, strengthening both action following and interaction dynamics. During post-training, the action-conditioning layer is reinitialized for the continuous control space of the target robot, and the complete diffusion model is fine-tuned on embodiment-specific data. Experiments using Unitree G1 and Fourier GR-1 data, including GR-1 evaluation and virtual G1 teleoperation, therefore make DreamDojo a direct example of transferring interaction priors from human videos into an action-controllable humanoid diffusion world model.
Table 6. Comparison of explicit- and latent-action diffusion world models.
Table 6. Comparison of explicit- and latent-action diffusion world models.
Method Action interface World-model coupling Evidence/use Remaining transfer gap
Ctrl-World [70] Synchronized frame-level robot action chunks Multi-view video diffusion with pose-conditioned memory retrieval 95k DROID trajectories, downstream success increases from 38.7% to 83.4% Requires embodiment-specific action semantics and lacks whole-body control.
VLAW [71] On-policy robot actions from successful and failed trials Iterative co-training of the conditional video model and VLA policy Reports a 39.2-point gain over the stated DSRL baseline Self-training can reinforce physically incorrect generated outcomes.
Olaf-World [73] Oriented continuous latent actions learned by a β -VAE Frozen latent-action model conditions a flow-matching video DiT Extracts action-aware supervision from unlabeled video Learned coordinates still require mapping to executable humanoid controls.
CoLA-World [74] Discrete actions inferred by inverse dynamics and vector quantization OpenSora world model and latent actions are optimized jointly Lets the video prior shape action representation end-to-end Discrete codes may collapse and remain difficult to map to continuous control.
DreamDojo [62] Human proxy actions adapted to continuous robot controls Flow-matching video DiT with reinitialized robot-action conditioning Direct Unitree G1 and Fourier GR-1 data and evaluation Proxy-to-robot transfer does not by itself guarantee causal or physical accuracy.

5. Open Challenges and Future Directions

The approaches reviewed in Section 3 and Section 4 demonstrate the potential of diffusion models for uncertain perception and predictive world modeling. However, generative quality alone is insufficient for operational humanoid intelligence: predictions must be timely, physically valid, uncertainty-aware, temporally coherent, transferable across embodiments, and useful to a downstream controller. We therefore identify five challenges toward an integrated perception–prediction–action framework: real-time inference, physical grounding and uncertainty calibration, long-horizon closed-loop prediction, scalable cross-embodiment learning, and evaluation under humanoid interaction.

5.1. Real-Time and Adaptive Generative Inference

Iterative denoising conflicts with the heterogeneous timing requirements of humanoid systems, where balance and proprioceptive estimation require rapid updates while visual interpretation and long-horizon imagination can operate more slowly. Progressive distillation and consistency models provide possible routes toward one- or few-step generation [149,150], while DreamDojo demonstrates that a multi-step world model can be distilled into a substantially faster predictor [62]. Future systems should additionally adopt adaptive, multi-rate inference: deterministic estimators can maintain well-observed states, whereas diffusion is activated under occlusion, sensor disagreement, multimodality, or growing uncertainty. World-model rollouts could similarly be triggered only when alternative outcomes must be compared. Evaluation should therefore report not only model inference time, but end-to-end latency, memory and energy consumption, prediction age, physical accuracy, and mode coverage on deployment hardware.

5.2. Physically Grounded and Uncertainty-Calibrated Prediction

Visual plausibility does not guarantee that a generated body configuration, contact transition, or object trajectory is physically feasible, and diversity among diffusion samples does not by itself constitute calibrated uncertainty. Future models should combine generative learning with differentiable kinematics, dynamics, metric geometry, and explicit contact constraints; physics-informed diffusion objectives provide a general mechanism for incorporating equality and inequality constraints during training [151]. For humanoids, these models should jointly predict body motion, support state, contact location, external wrench, and object dynamics rather than estimating them independently. Calibration methods such as conformal uncertainty quantification under distribution shift [152] could then convert sampled trajectories into decision-relevant prediction regions. High uncertainty about balance, collision, slip, or human response should trigger additional sensing, shorter planning horizons, replanning, or a safety controller, and evaluation should measure coverage, sharpness, event calibration, and robustness to changes in friction, payload, morphology, viewpoint, and sensor quality.

5.3. Long-Horizon Memory and Closed-Loop Correction

Long-horizon diffusion world models accumulate errors as generated states are reused as future inputs, producing geometric drift, forgotten objects, and incorrect contact histories. Persistent memory and long-context architectures developed by EDELINE, StateSpaceDiffuser, LIVE, and Infinite-World offer useful starting points [125,153,154,155], but humanoid memory must also distinguish slowly changing information, such as scene layout and object identity, from rapidly changing variables such as support, contact, and human motion. Scheduled autoregressive training, as adopted by DreamPolicy, can expose models to their own prediction errors [72], while feedback world models can use newly observed states to correct subsequent predictions online [156]. Future systems should combine these mechanisms with hierarchical temporal abstraction, representing task progress coarsely while retaining fine-resolution contact transitions, and should be evaluated through horizon-dependent state drift, identity and contact retention, recovery from incorrect predictions, and re-anchoring speed after new observations become available.

5.4. Scaling Across Human, Simulated, and Robot Embodiments

Human videos provide diverse semantic and interaction knowledge, simulation provides accurate state and contact supervision, and robot trajectories ground predictions in embodiment-specific sensing and dynamics; however, their observations and action spaces are not directly interchangeable. Multi-embodiment datasets and physically interpretable action representations, as explored by RT-X and RDT-1B [157,158], provide a basis for sharing knowledge while retaining embodiment-specific adapters. A particularly promising direction is reasoning-conditioned diffusion: an autoregressive reasoning module could infer task stages, affordances, causal object relations, intended contacts, and safety constraints from human, simulated, and robot experience, after which a diffusion model generates target-specific future states or action trajectories conditioned on the robot morphology and control interface. DiffusionVLA and HybridVLA provide initial evidence that autoregressive reasoning and continuous diffusion-based control can be integrated effectively [159,160]. For humanoids, this separation could transfer embodiment-invariant task knowledge while allowing diffusion to resolve platform-specific whole-body motion; nevertheless, the intermediate reasoning must be grounded in observations and verified against predicted dynamics rather than treated as an unconstrained textual explanation. Scaling should also prioritize failures and recovery behaviors, with simulated or generated data validated against physical interaction before repeated reuse.

5.5. Evaluation and Benchmarking for Humanoid Perception and World Models

Existing image-generation metrics cannot establish physical correctness, while task success alone cannot isolate improvements in perception, prediction, or control. Future benchmarks should therefore evaluate three coupled levels: perception, through errors and calibrated uncertainty in pose, geometry, object state, contact, and wrench under controlled sensor degradation; prediction, through temporal coherence, physical consistency, action sensitivity, counterfactual validity, and uncertainty calibration over increasing horizons; and closed-loop behavior, through task success, falls, collisions, recovery, real-world interaction cost, and computational latency. Best-of-N results should disclose the sampling budget and include an online hypothesis-selection mechanism rather than relying on oracle selection. WorldGym and WorldEval illustrate how action-conditioned models can support policy evaluation while also exposing errors under long rollouts and out-of-distribution actions [145,161]. Humanoid benchmarks should consequently combine locomotion, manipulation, contact-rich interaction, and human collaboration while reporting morphology, sensing, action representation, control frequency, deployment hardware, and the gap between simulated and physical evaluation.

6. Conclusion

This survey reviewed diffusion models for humanoid perception and world modeling, organizing existing methods by perceptual targets, predictive representations, action conditioning, and reliability requirements. Conditional generation provides a common basis for recovering incomplete observations, representing ambiguous states, and predicting alternative futures. Reliable deployment requires efficient inference, physical consistency, calibrated uncertainty, and further validation on physical humanoids. Future research should integrate persistent memory, closed-loop correction, and reasoning-conditioned transfer across embodiments, with progress evaluated through improved decision making, stability, and recovery during physical interaction.

References

  1. Cao, L. Humanoid robots and humanoid AI: Review, perspectives and directions. ACM Comput. Surv. 2025, 58, 1–37. [Google Scholar] [CrossRef]
  2. Nguyen, X.T.; Nguyen, M.D.; Nguyen, H.D.; O’Hare, G.M.; Pham, Q.V. Predictive Intelligence for Future Networks: The Promise of World Models for 6G. IEEE Consumer Electronics Magazine, 2026. [Google Scholar]
  3. Roychoudhury, A.; Khorshidi, S.; Agrawal, S.; Bennewitz, M. Perception for humanoid robots. Curr. Robot. Rep. 2023, 4, 127–140. [Google Scholar] [CrossRef]
  4. Ali, M.Q.; Sridhar, A.; Matiana, S.; Wong, A.; Al-Sharman, M. Humanoid world models: Open world foundation models for humanoid robotics. arXiv 2025, arXiv:2506.01182. [Google Scholar]
  5. Hara, T.; Sato, T.; Ogata, T.; Awano, H. Uncertainty-aware haptic shared control with humanoid robots for flexible object manipulation. IEEE Robot. Autom. Lett. 2023, 8, 6435–6442. [Google Scholar] [CrossRef]
  6. Long, R.; Rauch, C.; Zhang, T.; Ivan, V.; Vijayakumar, S. RigidFusion: Robot localisation and mapping in environments with large dynamic rigid objects. IEEE Robot. Autom. Lett. 2021, 6, 3703–3710. [Google Scholar] [CrossRef]
  7. Oliver, G.; Lanillos, P.; Cheng, G. An empirical study of active inference on a humanoid robot. IEEE Trans. Cogn. Dev. Syst. 2021, 14, 462–471. [Google Scholar] [CrossRef]
  8. Nguyen, K.; Le, A.T.; Peters, J.; Vu, M.N. DoublyAware: Dual Planning and Policy Awareness for Temporal Difference Learning in Humanoid Locomotion. IEEE Robot. Autom. Lett. 2026, 11, 2162–2169. [Google Scholar] [CrossRef]
  9. Lin, Y.; Xie, Y.; Xie, J.; Huang, Y.; Wang, R.; Lv, J.; Ma, Y.; Zuo, X. SimGenHOI: Physically Realistic Whole-Body Humanoid-Object Interaction via Generative Modeling and Reinforcement Learning. arXiv 2025, arXiv:2508.14120. [Google Scholar]
  10. Zhang, H.; Zhang, L.; Chen, Z.; Chen, L.; Wang, Y.; Xiong, R. Natural Humanoid Robot Locomotion with Generative Motion Prior. In Proceedings of the 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2025; pp. 6622–6629. [Google Scholar] [CrossRef]
  11. Wu, S.; Zhang, H.; Liu, Z.; Chen, H.; Jiao, Y. Enhancing human pose estimation in internet of things via diffusion generative models. IEEE Internet Things J. 2025, 12, 13556–13567. [Google Scholar] [CrossRef]
  12. Holmquist, K.; Wandt, B. Diffpose: Multi-hypothesis human pose estimation using diffusion models. In Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV); IEEE, 2023; pp. 15931–15941. [Google Scholar]
  13. Aldausari, N.; Sowmya, A.; Marcus, N.; Mohammadi, G. Video Generative Adversarial Networks: A Review. ACM Comput. Surv. 2022, 55. [Google Scholar] [CrossRef]
  14. Davtyan, A.; Sameni, S.; Favaro, P. Efficient Video Prediction via Sparsely Conditioned Flow Matching. In Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2023; pp. 23206–23217. [Google Scholar] [CrossRef]
  15. Aghabozorgi, M.; Moazeni, A.; Zhang, Y.; Li, K. WIMLE: Uncertainty-aware world models with IMLE for sample-efficient continuous control. arXiv 2026, arXiv:2602.14351. [Google Scholar]
  16. Nishimura, Y.; Nakamura, Y.; Ishiguro, H. Long-term Motion Generation for Interactive Humanoid Robots using GAN with Convolutional Network. In Proceedings of the Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, New York, NY, USA, 2020; pp. 375–377. [Google Scholar] [CrossRef]
  17. Ho, J.; Jain, A.; Abbeel, P. Denoising Diffusion Probabilistic Models. Proc. Adv. Neural Inf. Process. Syst. 2020, Vol. 33, 6840–6851. [Google Scholar]
  18. Dhariwal, P.; Nichol, A. Diffusion Models Beat GANs on Image Synthesis. Proc. Adv. Neural Inf. Process. Syst. 2021, Vol. 34, 8780–8794. [Google Scholar]
  19. Xing, Z.; Feng, Q.; Chen, H.; Dai, Q.; Hu, H.; Xu, H.; Wu, Z.; Jiang, Y.G. A Survey on Video Diffusion Models. ACM Comput. Surv. 2024, 57. [Google Scholar] [CrossRef]
  20. Wang, C.; Peng, H.Y.; Liu, Y.T.; Gu, J.; Hu, S.M. Diffusion Models for 3D Generation: A Survey. Comput. Vis. Media 2025, 11, 1–28. [Google Scholar] [CrossRef]
  21. Kirch, S.; Olyunina, V.; Ondřej, J.; Pagés, R.; Martin, S.; Pérez-Molina, C. RGB-D-Fusion: Image conditioned depth diffusion of humanoid subjects. IEEE Access 2023, 11, 99111–99129. [Google Scholar] [CrossRef]
  22. Liao, Q.; Truong, T.E.; Huang, X.; Gao, Y.; Tevet, G.; Sreenath, K.; Liu, C.K. Beyondmimic: From motion tracking to versatile humanoid control via guided diffusion. arXiv 2025, arXiv:2508.08241. [Google Scholar]
  23. Pan, S.; Yi, X.; Zhou, Y.; Jian, W.; Zhang, Y.; Wan, P.; Xu, F. DiffCap: Diffusion-based real-time human motion capture using sparse IMUs and a monocular camera. IEEE Transactions on Visualization and Computer Graphics, 2025. [Google Scholar]
  24. Zhang, H.; Zhang, X.; Huang, J.; Feng, Z.; Xiao, X. End-to-End Diffusion-Based 3D Object Reconstruction From Robotic Tactile Sensing. IEEE Robot. Autom. Lett. 2025, 11, 1434–1441. [Google Scholar] [CrossRef]
  25. An, Q. Stochastic World Models with Diffusion for Visual Whole-Body Humanoid Control. In Proceedings of the Proceedings of the 5th International Conference on Computer, Artificial Intelligence and Control Engineering, 2026; pp. 1421–1426. [Google Scholar]
  26. Fuest, M.; Ma, P.; Gui, M.; Schusterbauer, J.; Hu, V.T.; Ommer, B. Diffusion Models and Representation Learning: A Survey. IEEE Trans. Pattern Anal. Mach. Intell. 2026, 48, 7209–7228. [Google Scholar] [CrossRef] [PubMed]
  27. Liu, X.; Ma, K.Y.; Gao, C.; Shou, M.Z. Diffusion Models in Robotics: A Survey. Int. J. Comput. Vis. 2026, 134, 299. [Google Scholar] [CrossRef]
  28. Wolf, R.; Shi, Y.; Liu, S.; Rayyes, R. Diffusion models for robotic manipulation: a survey. Front. Robot. AI 2025, 12, 1606247. [Google Scholar] [CrossRef] [PubMed]
  29. Song, M.; Deng, X.; Zhou, Z.; Wei, J.; Guan, W.; Nie, L. A Survey on Diffusion Policy for Robotic Manipulation: Taxonomy, Analysis, and Future Directions. TechRxiv 2025. [Google Scholar] [CrossRef] [PubMed]
  30. Ding, J.; Zhang, Y.; Shang, Y.; Zhang, Y.; Zong, Z.; Feng, J.; Yuan, Y.; Su, H.; Li, N.; Sukiennik, N.; et al. Understanding world or predicting future? a comprehensive survey of world models. ACM Comput. Surv. 2025, 58, 1–38. [Google Scholar] [CrossRef]
  31. Zhang, P.F.; Cheng, Y.; Sun, X.; Wang, S.; Li, F.; Zhu, L.; Shen, H.T. A step toward world models: A survey on robotic manipulation. arXiv 2025, arXiv:2511.02097. [Google Scholar]
  32. Hou, B.; Li, G.; Jia, J.; An, T.; Guo, X.; Leng, S.; Geng, H.; Ze, Y.; Harada, T.; Torr, P.; et al. World model for robot learning: A comprehensive survey. Int. J. Robot. Res. 2026, 02783649261488980. [Google Scholar] [CrossRef]
  33. Wang, F.; Wang, Z.; Pei, G.; Zhang, M.; Liang, C.; Hu, J.; Li, Z.; Wu, J.; Han, N.; Zhang, Z.; et al. World models for robotic manipulation: A survey. SmartBot 2026, e70053. [Google Scholar] [CrossRef]
  34. Shan, W.; Liu, Z.; Zhang, X.; Wang, Z.; Han, K.; Wang, S.; Ma, S.; Gao, W. Diffusion-based 3d human pose estimation with multi-hypothesis aggregation. In Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV); IEEE, 2023; pp. 14715–14725. [Google Scholar]
  35. Liu, Y.; Yang, J.; Gu, X.; Guo, Y.; Yang, G.Z. Egohmr: Egocentric human mesh recovery via hierarchical latent diffusion model. In Proceedings of the 2023 IEEE International Conference on Robotics and Automation (ICRA); IEEE, 2023; pp. 9807–9813. [Google Scholar]
  36. Li, J.; Liu, K.; Wu, J. Ego-body pose estimation via ego-head pose estimation. In Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023; pp. 17142–17151. [Google Scholar]
  37. Lee, J.; Xu, W.; Richard, A.; Wei, S.E.; Saito, S.; Bai, S.; Wang, T.L.; Sung, M.; Kim, T.K.; Saragih, J. Rewind: Real-time egocentric whole-body motion diffusion with exemplar-based identity conditioning. In Proceedings of the 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); IEEE, 2025; pp. 7095–7104. [Google Scholar]
  38. Oppel, H.; Munz, M. Imudiffusion: A diffusion model for multivariate time series synthetisation for inertial motion capturing systems. arXiv 2024, arXiv:2411.02954. [Google Scholar]
  39. Ilic, A.; Jiang, J.; Streli, P.; Liu, X.; Holz, C. Human motion capture from loose and sparse inertial sensors with garment-aware diffusion models. arXiv 2025, arXiv:2506.15290. [Google Scholar]
  40. Kawaharazuka, K.; Ikemoto, S. Diffusion-Based Body Schema Learning Enabling Abnormal-State Adaptation in Musculoskeletal Robots. arXiv 2026, arXiv:2608.01029. [Google Scholar]
  41. Zhang, Z.; Desai, A.; Hu, J.C.; Saka, Y.; Luu, Q.K.; Lei, J.; Soleymanzadeh, D.; Zhang, B.; Zheng, M.; She, Y. Imagining the Sense of Touch: Touch-Informed Manipulation via Imagined Tactile Representations. arXiv 2026, arXiv:2607.01684. [Google Scholar]
  42. Rodriguez, S.; Dou, Y.; Oller, M.; Owens, A.; Fazeli, N. Cross-sensor touch generation. arXiv 2025, arXiv:2510.09817. [Google Scholar]
  43. Tian, Y.; Zhang, J.; Huang, G.; Wang, B.; Wang, P.; Pang, J.; Dong, H. RoboKeyGen: Robot Pose and Joint Angles Estimation via Diffusion-Based 3D Keypoint Generation. In Proceedings of the IEEE International Conference on Robotics and Automation, 2024. [Google Scholar] [CrossRef]
  44. Zhang, H.; Caprio, M.; Shao, J.; Zhang, Q.; Tang, J.; Zhang, S.; Pan, W. PoseDiff: A Unified Diffusion Model Bridging Robot Pose Estimation and Video-to-Action Control. 2025. [Google Scholar] [CrossRef]
  45. Wan, Z.; Zhao, L. DnD Filter: Differentiable State Estimation for Dynamic Systems Using Diffusion Models. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, 2025. [Google Scholar] [CrossRef]
  46. Wan, Z.; Zhao, L. Diffpf: Differentiable particle filtering with generative sampling via conditional diffusion models. IEEE Robotics and Automation Letters, 2026. [Google Scholar]
  47. Han, S.W.; Kim, M.J. CDM: Contact Diffusion Model for Multi-Contact Point Localization. In Proceedings of the 2025 IEEE International Conference on Robotics and Automation (ICRA); IEEE, 2025; pp. 4702–4708. [Google Scholar]
  48. Huang, Z.; Hou, H.; Berenson, D. Multimodal Diffusion Forcing for Forceful Manipulation. In Proceedings of the 2026 IEEE International Conference on Robotics and Automation (ICRA); IEEE, 2026; pp. 9631–9638. [Google Scholar]
  49. Chen, X.; Xu, X.; Ning, Y.; Yu, J.; Zhang, Y.; Qian, S.; Xiang, L.; Chen, J.; Wang, Y.; Zhang, H.; et al. SixthSense: Task-Agnostic Proprioception-Only Whole-Body Wrench Estimation for Humanoids. arXiv 2026, arXiv:2605.01427. [Google Scholar]
  50. Ji, Y.; Chen, Z.; Xie, E.; Hong, L.; Liu, X.; Liu, Z.; Lu, T.; Li, Z.; Luo, P. Ddp: Diffusion model for dense visual prediction. In Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV); IEEE, 2023; pp. 21684–21695. [Google Scholar]
  51. Ke, B.; Obukhov, A.; Huang, S.; Metzger, N.; Daudt, R.C.; Schindler, K. Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation. In Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024; pp. 9492–9502. [Google Scholar] [CrossRef]
  52. Reed, A.; Crowe, B.; Albin, D.; Achey, L.; Hayes, B.; Heckman, C. SceneSense: Diffusion Models for 3D Occupancy Synthesis from Partial Observation. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, 2024; pp. 7383–7390. [Google Scholar]
  53. Zhu, X.; Zhou, H.; Xing, P.; Zhao, L.; Xu, H.; Liang, J.; Hauptmann, A.; Liu, T.; Gallagher, A. Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models. Proc. Comput. Vis. – ECCV 2024, 2024, 357–375. [Google Scholar] [CrossRef]
  54. Liao, Z.; Xu, B.; Waslander, S.L. Toward General Object-Level Mapping from Sparse Views with 3D Diffusion Priors. In Proceedings of the Proceedings of the 8th Conference on Robot Learning, 2025; Vol. 270, Proceedings of Machine Learning Research. pp. 1340–1361. [Google Scholar]
  55. Wang, F.; Wang, Z.; Pei, G.; Zhang, M.; Liang, C.; Hu, J.; Li, Z.; Wu, J.; Han, N.; Zhang, Z.; et al. World models for robotic manipulation: A survey. SmartBot 2026, e70053. [Google Scholar] [CrossRef]
  56. Wang, Y.; Zhang, Z.; Qiu, J.; Sun, D.; Meng, Z.; Wei, X.; Yang, X. Touch2shape: Touch-conditioned 3d diffusion for shape exploration and reconstruction. In Proceedings of the 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); IEEE, 2025; pp. 5656–5665. [Google Scholar]
  57. Liu, J.; Wang, G.; Ye, W.; Jiang, C.; Han, J.; Liu, Z.; Zhang, G.; Du, D.; Wang, H. DifFlow3D: Toward Robust Uncertainty-Aware Scene Flow Estimation with Iterative Diffusion-Based Refinement. In Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024; pp. 15109–15119. [Google Scholar]
  58. Zhang, Y.; Wandt, B.; Magnusson, M.; Felsberg, M. DiffSF: Diffusion Models for Scene Flow Estimation. Proc. Adv. Neural Inf. Process. Syst. 2024, Vol. 37, 111227–111247. [Google Scholar] [CrossRef]
  59. Gu, T.; Chen, G.; Li, J.; Lin, C.; Rao, Y.; Zhou, J.; Lu, J. Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion. In Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022; pp. 17113–17122. [Google Scholar]
  60. Alonso, E.; Jelley, A.; Micheli, V.; Kanervisto, A.; Storkey, A.; Pearce, T.; Fleuret, F. Diffusion for world modeling: Visual details matter in atari. Adv. Neural Inf. Process. Syst. 2024, 37, 58757–58791. [Google Scholar] [CrossRef]
  61. Yang, S.; Du, Y.; Ghasemipour, K.; Tompson, J.; Kaelbling, L.; Schuurmans, D.; Abbeel, P. Learning interactive real-world simulators. arXiv 2023, arXiv:2310.06114. [Google Scholar]
  62. Gao, S.; Liang, W.; Zheng, K.; Malik, A.; Ye, S.; Yu, S.; Tseng, W.C.; Dong, Y.; Mo, K.; Lin, C.H.; et al. Dreamdojo: A generalist robot world model from large-scale human videos. arXiv 2026, arXiv:2602.06949. [Google Scholar]
  63. Yang, P.; Zheng, Y.; Qian, D.; Xing, Z.; Zhang, Q.; Wang, L.; Zhang, Y.; Guo, S.; Xia, Z.; Chen, Q.; et al. Dreamerad: Efficient reinforcement learning via latent world model for autonomous driving. arXiv 2026, arXiv:2603.24587. [Google Scholar]
  64. Huang, Y.; Zhang, J.; Zou, S.; Liu, X.; Hu, R.; Xu, K. Ladi-wm: A latent diffusion-based world model for predictive manipulation. arXiv 2025, arXiv:2505.11528. [Google Scholar]
  65. Zheng, R.; Wang, J.; Reed, S.; Bjorck, J.; Fang, Y.; Hu, F.; Jang, J.; Kundalia, K.; Lin, Z.; Magne, L.; et al. Flare: Robot learning with implicit world modeling. arXiv 2025, arXiv:2505.15659. [Google Scholar]
  66. Zhang, L.; Xiong, Y.; Yang, Z.; Casas, S.; Hu, R.; Urtasun, R. Copilot4d: Learning unsupervised world models for autonomous driving via discrete diffusion. Proc. Int. Conf. Learn. Represent. 2024, Vol. 2024, 7269–7299. [Google Scholar]
  67. Jiang, C.; Zhou, D.; Liu, J.; Sun, K.L. VectorWorld: Efficient Streaming World Model via Diffusion Flow on Vector Graphs. arXiv 2026, arXiv:2603.17652. [Google Scholar]
  68. Lin, S.; Chen, J.; Xu, H.; Li, Z.; Wang, G.; Jing, Y.; Xu, S.; Zhao, R.; Sheil, B.; Chau, L.P.; et al. Roboflow4d: A lightweight flow world model toward real-time flow-guided robotic manipulation. arXiv 2026, arXiv:2605.17522. [Google Scholar]
  69. Lou, Y.; Ye, Y.; Fu, Y.; Cen, J.; Chi, X.; Lyu, Y.; Jia, P.; Han, S.; Lu, Z.; Zhang, S. Dream-Tac: A Unified Tactile World Action Model for Contact-Rich Robot Manipulation. arXiv 2026, arXiv:2606.08737. [Google Scholar]
  70. Guo, Y.; Shi, L.; Chen, J.; Finn, C. Ctrl-world: A controllable generative world model for robot manipulation. Proc. Int. Conf. Learn. Represent. 2026, Vol. 2026, 6121–6138. [Google Scholar]
  71. Guo, Y.; Lee, T.; Shi, L.X.; Chen, J.; Liang, P.; Finn, C. Vlaw: Iterative co-improvement of vision-language-action policy and world model. arXiv 2026, arXiv:2602.12063. [Google Scholar]
  72. Fan, Y.; Gui, T.; Ji, K.; Ding, S.; Zhang, C.; Xu, Y.; Yang, K.; Gu, J.; Yu, J.; Wang, J.; et al. DreamPolicy: A Unified World-model Policy for Scalable Humanoid Locomotion. arXiv 2025, arXiv:2505.18780. [Google Scholar]
  73. Jiang, Y.; Gu, Y.; Tsang, I.W.; Shou, M.Z. Olaf-World: Orienting Latent Actions for Video World Modeling. arXiv 2026, arXiv:2602.10104. [Google Scholar]
  74. Wang, Y.; Zhang, F.; Zhan, D.C.; Zhao, L.; Wang, K.; Bian, J. Co-evolving latent action world models. arXiv 2025, arXiv:2510.26433. [Google Scholar]
  75. Bi, H.; Tan, H.; Xie, S.; Wang, Z.; Huang, S.; Liu, H.; Zhao, R.; Feng, Y.; Xiang, C.; Rong, Y.; et al. Motus: A unified latent action world model. In Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026; pp. 35101–35113. [Google Scholar]
  76. Kato, I.; Ohteru, S.; Kobayashi, H.; Shirai, K.; Uchiyama, A. Information-Power Machine with Senses and Limbs: Wabot 1. In On Theory and Practice of Robots and Manipulators: Volume I; Springer, 1974; pp. 11–24. [Google Scholar]
  77. Hirai, K. The Honda humanoid robot: development and future perspective. Ind. Robot Int. J. Robot. Res. Appl. 1999, 26, 260–266. [Google Scholar] [CrossRef]
  78. Kaneko, K.; Kanehiro, F.; Morisawa, M.; Akachi, K.; Miyamori, G.; Hayashi, A.; Kanehira, N. Humanoid robot hrp-4-humanoid robotics platform with lightweight and slim body. In Proceedings of the 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems; IEEE, 2011; pp. 4400–4407. [Google Scholar]
  79. Metta, G.; Natale, L.; Nori, F.; Sandini, G.; Vernon, D.; Fadiga, L.; Von Hofsten, C.; Rosander, K.; Lopes, M.; Santos-Victor, J.; et al. The iCub humanoid robot: An open-systems platform for research in cognitive development. Neural Netw. 2010, 23, 1125–1134. [Google Scholar] [CrossRef] [PubMed]
  80. Kuindersma, S.; Deits, R.; Fallon, M.; Valenzuela, A.; Dai, H.; Permenter, F.; Koolen, T.; Marion, P.; Tedrake, R. Optimization-based locomotion planning, estimation, and control design for the atlas humanoid robot. Auton. Robot. 2016, 40, 429–455. [Google Scholar] [CrossRef]
  81. Hafner, D.; Pasukonis, J.; Ba, J.; Lillicrap, T. Mastering diverse control tasks through world models. Nature 2025, 640, 647–653. [Google Scholar] [CrossRef] [PubMed]
  82. Fu, Z.; Zhao, Q.; Wu, Q.; Wetzstein, G.; Finn, C. Humanplus: Humanoid shadowing and imitation from humans. arXiv 2024, arXiv:2406.10454. [Google Scholar]
  83. Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative adversarial networks. Commun. ACM 2020, 63, 139–144. [Google Scholar] [CrossRef]
  84. Kingma, D.P.; Welling, M. Auto-encoding variational bayes. arXiv 2013, arXiv:1312.6114. [Google Scholar]
  85. Yasar, M.S.; Iqbal, T. Vader: Vector-quantized generative adversarial network for motion prediction. In Proceedings of the 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); IEEE, 2023; pp. 3827–3834. [Google Scholar]
  86. Wu, Y.; Karunratanakul, K.; Luo, Z.; Tang, S. Uniphys: Unified planner and controller with diffusion for flexible physics-based character control. In Proceedings of the 2025 IEEE/CVF International Conference on Computer Vision (ICCV); IEEE, 2025; pp. 13214–13224. [Google Scholar]
  87. Hafner, D.; Lillicrap, T.; Ba, J.; Norouzi, M. Dream to control: Learning behaviors by latent imagination. arXiv 2019, arXiv:1912.01603. [Google Scholar]
  88. Song, J.; Meng, C.; Ermon, S. Denoising Diffusion Implicit Models. In Proceedings of the International Conference on Learning Representations, 2021. [Google Scholar]
  89. Zhong, R.; Hu, B.; Feng, Y.; Liu, Z.; Qin, Q.; Wang, X.V.; Wang, L.; Tan, J. FineMLD: a fine-grained motion latent diffusion for human motion prediction in Human–robot collaboration. Adv. Eng. Inform. 2026, 70, 104119. [Google Scholar] [CrossRef]
  90. Li, X.; Zhang, J.; Fu, L.; Gu, L.; Xiong, Y.; Li, S. Structure-Aware Diffusion Policy for Bimanual Cooperative Manipulation. IEEE Robotics and Automation Letters, 2026. [Google Scholar]
  91. Lu, Y.; Wang, S.; Han, X.; Zhang, X.; Wu, Y.; He, Z. Enhancing Diffusion Policy with Classifier-Free Guidance for Temporal Robotic Tasks. In Proceedings of the 2025 IEEE 37th International Conference on Tools with Artificial Intelligence (ICTAI); IEEE, 2025; pp. 1023–1029. [Google Scholar]
  92. Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; Ommer, B. High-Resolution Image Synthesis with Latent Diffusion Models. In Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022; pp. 10684–10695. [Google Scholar]
  93. Esser, P.; Rombach, R.; Ommer, B. Taming transformers for high-resolution image synthesis. In Proceedings of the Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021; pp. 12873–12883. [Google Scholar]
  94. Tong, Y.; Liu, H.; Zhang, Z. Advancements in humanoid robots: A comprehensive review and future prospects. IEEE/CAA J. Autom. Sin. 2024, 11, 301–328. [Google Scholar] [CrossRef]
  95. Cox, J.; Zhang, W.; Furukawa, T. Tracking Highly Dynamic Humanoid Motion with Dynamic IMU Measurement Fusion. In Proceedings of the 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); IEEE, 2025; pp. 21596–21603. [Google Scholar]
  96. Caillot, A.; André, A.N.; Duvinage, T.; Caron, G. RGB-D versus omnidirectional visual SLAM in humanoid robot positioning. In Proceedings of the 2025 IEEE-RAS 24th International Conference on Humanoid Robots (Humanoids); IEEE, 2025; pp. 483–490. [Google Scholar]
  97. Gama, F.; Shcherban, M.; Rolf, M.; Hoffmann, M. Goal-directed tactile exploration for body model learning through self-touch on a humanoid robot. IEEE Trans. Cogn. Dev. Syst. 2021, 15, 419–433. [Google Scholar] [CrossRef]
  98. De Santis, A.; Siciliano, B.; De Luca, A.; Bicchi, A. An atlas of physical human–robot interaction. Mech. Mach. Theory 2008, 43, 253–270. [Google Scholar] [CrossRef]
  99. Li, G.; Li, Z.; Kan, Z. Assimilation control of a robotic exoskeleton for physical human-robot interaction. IEEE Robot. Autom. Lett. 2022, 7, 2977–2984. [Google Scholar] [CrossRef]
  100. Li, Z.; Chi, C.; Zhu, B.; Wei, Y.; Bai, S.; Ji, Y.; Peng, Y.; Huang, T.; Wang, P.; Wang, Z.; et al. Robomirror: Understand before you imitate for video to humanoid locomotion. arXiv 2025, arXiv:2512.23649. [Google Scholar]
  101. He, T.; Luo, Z.; Xiao, W.; Zhang, C.; Kitani, K.; Liu, C.; Shi, G. Learning human-to-humanoid real-time whole-body teleoperation. In Proceedings of the 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); IEEE, 2024; pp. 8944–8951. [Google Scholar]
  102. Zhu, Y.; Li, A.; Tang, Y.; Zhao, W.; Zhou, J.; Lu, J. Dpmesh: Exploiting diffusion prior for occluded human mesh recovery. In Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); IEEE, 2024; pp. 1101–1110. [Google Scholar]
  103. Stathopoulos, A.; Han, L.; Metaxas, D. Score-guided diffusion for 3d human recovery. In Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); IEEE, 2024; pp. 906–915. [Google Scholar]
  104. Zheng, C.; Liu, X.; Peng, Q.; Wu, T.; Wang, P.; Chen, C. Diffmesh: A motion-aware diffusion framework for human mesh recovery from videos. In Proceedings of the 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV); IEEE, 2025; pp. 4891–4901. [Google Scholar]
  105. Wang, J.; Cao, Z.; Luvizon, D.; Liu, L.; Sarkar, K.; Tang, D.; Beeler, T.; Theobalt, C. Egocentric whole-body motion capture with fisheyevit and diffusion-based motion refinement. In Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); IEEE, 2024; pp. 777–787. [Google Scholar]
  106. Zhou, Y.; Barnes, C.; Lu, J.; Yang, J.; Li, H. On the continuity of rotation representations in neural networks. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); IEEE, 2019; pp. 5738–5746. [Google Scholar]
  107. Chen, Y.; Tian, Y.; He, M. Monocular human pose estimation: A survey of deep learning-based methods. Comput. Vis. Image Underst. 2020, 192, 102897. [Google Scholar] [CrossRef]
  108. Feng, R.; Gao, Y.; Tse, T.H.E.; Ma, X.; Chang, H.J. Diffpose: Spatiotemporal diffusion model for video-based human pose estimation. In Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV); IEEE, 2023; pp. 14815–14826. [Google Scholar]
  109. Zeng, Y.; Gao, Z.; Shi, Y.; Lu, Z.; Yang, W.; Liao, Q. Egopressdiff: Multimodal video diffusion for egocentric uv-domain hand-pressure estimation. In Proceedings of the ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); IEEE, 2026; pp. 13232–13236. [Google Scholar]
  110. Wang, J.; Dabral, R.; Luvizon, D.; Cao, Z.; Liu, L.; Beeler, T.; Theobalt, C. Ego4o: Egocentric human motion capture and understanding from multi-modal input. In Proceedings of the 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); IEEE, 2025; pp. 22668–22679. [Google Scholar]
  111. Tashiro, Y.; Song, J.; Song, Y.; Ermon, S. CSDI: Conditional Score-Based Diffusion Models for Probabilistic Time Series Imputation. In Proceedings of the Advances in Neural Information Processing Systems, 2021; Vol. 34. [Google Scholar]
  112. Zhou, S.; Katragadda, S.; Huang, G. Learning IMU bias with diffusion model. In Proceedings of the 2025 IEEE International Conference on Robotics and Automation (ICRA); IEEE, 2025; pp. 2162–2168. [Google Scholar]
  113. Chen, S.F.; Yong, C.; Sun, S.H. Restoring noisy demonstration for imitation learning with diffusion models. IEEE Transactions on Neural Networks and Learning Systems, 2025. [Google Scholar]
  114. Röstel, L.; Bäuml, B. Denoising Particle Filters: Learning State Estimation with Single-Step Objectives. arXiv 2026, arXiv:2602.19651. [Google Scholar]
  115. Zhao, W.; Rao, Y.; Liu, Z.; Liu, B.; Zhou, J.; Lu, J. Unleashing text-to-image diffusion models for visual perception. In Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV); IEEE, 2023; pp. 5706–5716. [Google Scholar]
  116. Xu, G.; Ge, Y.; Liu, M.; Fan, C.; Xie, K.; Zhao, Z.; Chen, H.; Shen, C. What Matters When Repurposing Diffusion Models for General Dense Perception Tasks? In Proceedings of the International Conference on Learning Representations, 2025. [Google Scholar]
  117. Viola, M.; Qu, K.; Metzger, N.; Ke, B.; Becker, A.; Schindler, K.; Obukhov, A. Marigold-dc: Zero-shot monocular depth completion with guided diffusion. In Proceedings of the 2025 IEEE/CVF International Conference on Computer Vision (ICCV); IEEE, 2025; pp. 5359–5370. [Google Scholar]
  118. Reed, A.; Achey, L.; Crowe, B.; Hayes, B.; Heckman, C. Online Diffusion-Based 3D Occupancy Prediction at the Frontier with Probabilistic Map Reconciliation. In Proceedings of the IEEE International Conference on Robotics and Automation, 2025; pp. 2846–2852. [Google Scholar]
  119. Achey, L.; Reed, A.; Crowe, B.; Hayes, B.; Heckman, C. Robust Robotic Exploration and Mapping Using Generative Occupancy Map Synthesis. arXiv 2025, arXiv:2506.20049. [Google Scholar]
  120. Zhang, J.; Wu, M.; Dong, H. Generative category-level object pose estimation via diffusion models. Adv. Neural Inf. Process. Syst. 2023, 36, 54627–54644. [Google Scholar] [CrossRef]
  121. Caddeo, G.M.; Marra, P.; Natale, L. Physically Grounded 3D Generative Reconstruction under Hand Occlusion using Proprioception and Multi-Contact Touch. arXiv 2026, arXiv:2604.09100. [Google Scholar]
  122. Ha, D.; Schmidhuber, J. World models. arXiv 2018, arXiv:1803.101222, 440. [Google Scholar]
  123. Hafner, D.; Pasukonis, J.; Ba, J.; Lillicrap, T. Mastering diverse domains through world models. arXiv 2023, arXiv:2301.04104. [Google Scholar]
  124. Hansen, N.; SV, J.; Sobal, V.; LeCun, Y.; Wang, X.; Su, H. Hierarchical world models as visual whole-body humanoid controllers. Proc. Int. Conf. Learn. Represent. 2025, Vol. 2025, 62175–62195. [Google Scholar]
  125. Savov, N.; Kazemi, N.; Zhang, D.; Paudel, D.P.; Wang, X.; Gool, L.V. Statespacediffuser: Bringing long context to diffusion world models. Adv. Neural Inf. Process. Syst. 2026, 38, 68865–68898. [Google Scholar]
  126. Wei, D.; Sun, H.; Li, B.; Lu, J.; Li, W.; Sun, X.; Hu, S. Human joint kinematics diffusion-refinement for stochastic motion prediction. Proc. Proc. AAAI Conf. Artif. Intell. 2023, Vol. 37, 6110–6118. [Google Scholar] [CrossRef]
  127. Wu, T.; Yang, S.; Po, R.; Xu, Y.; Liu, Z.; Lin, D.; Wetzstein, G. Video world models with long-term spatial memory. Adv. Neural Inf. Process. Syst. 2026, 38, 49371–49393. [Google Scholar]
  128. Po, R.; Nitzan, Y.; Zhang, R.; Chen, B.; Dao, T.; Shechtman, E.; Wetzstein, G.; Huang, X. Long-context state-space video world models. In Proceedings of the 2025 IEEE/CVF International Conference on Computer Vision (ICCV); IEEE, 2025; pp. 8733–8744. [Google Scholar]
  129. Blattmann, A.; Dockhorn, T.; Kulal, S.; Mendelevitch, D.; Kilian, M.; Lorenz, D.; Levi, Y.; English, Z.; Voleti, V.; Letts, A.; et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv 2023, arXiv:2311.15127. [Google Scholar]
  130. Ran, L.; Shou, M.Z. Tpdiff: Temporal pyramid video diffusion model. Proc. Int. Conf. Learn. Represent. 2026, Vol. 2026, 125635–125651. [Google Scholar]
  131. Yu, Y.; Jin, X.; Shang, Y.; Zhang, X.; Su, H.; Wu, W.; Li, Y. Mowm: Mixture-of-world-models for embodied planning via latent-to-pixel feature modulation. arXiv 2025, arXiv:2509.21797. [Google Scholar]
  132. Chen, Z.; Zhu, J.; Chen, X.; Zhang, J.; Hu, X.; Zhao, H.; Wang, C.; Yang, J.; Tai, Y. Dip: Taming diffusion models in pixel space. In Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026; pp. 36136–36146. [Google Scholar]
  133. Wu, W.; Zhao, Y.; Shou, M.Z.; Zhou, H.; Shen, C. Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models. In Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV); IEEE, 2023; pp. 1206–1217. [Google Scholar]
  134. Nguyen, L.T.; Nguyen, M.D.; Wong, K.S.; Le, D.D. Hawkes-Augmented World Model for Event-Driven Reinforcement Learning. 2026. [Google Scholar] [PubMed]
  135. Bjorck, J.; Castañeda, F.; Cherniadev, N.; Da, X.; Ding, R.; Fan, L.; Fang, Y.; Fox, D.; Hu, F.; Huang, S.; et al. Gr00t n1: An open foundation model for generalist humanoid robots. arXiv 2025, arXiv:2503.14734. [Google Scholar]
  136. Jiang, H.; Chen, J.; Bu, Q.; Chen, L.; Shi, M.; Zhang, Y.; Li, D.; Suo, C.; Li, H.; et al. Wholebodyvla: Towards unified latent vla for whole-body loco-manipulation control. Proc. Int. Conf. Learn. Represent. 2026, Vol. 2026, 157438–157461. [Google Scholar]
  137. Chi, C.; Xu, Z.; Feng, S.; Cousineau, E.; Du, Y.; Burchfiel, B.; Tedrake, R.; Song, S. Diffusion policy: Visuomotor policy learning via action diffusion. Int. J. Robot. Res. 2025, 44, 1684–1704. [Google Scholar] [CrossRef]
  138. Oquab, M.; Darcet, T.; Moutakanni, T.; Vo, H.; Szafraniec, M.; Khalidov, V.; Fernandez, P.; Haziza, D.; Massa, F.; El-Nouby, A.; et al. Dinov2: Learning robust visual features without supervision. arXiv 2023, arXiv:2304.07193. [Google Scholar]
  139. Zhai, X.; Mustafa, B.; Kolesnikov, A.; Beyer, L. Sigmoid loss for language image pre-training. In Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV); IEEE, 2023; pp. 11941–11952. [Google Scholar]
  140. Kim, M.J.; Gao, Y.; Lin, T.Y.; Lin, Y.C.; Ge, Y.; Lam, G.; Liang, P.; Song, S.; Liu, M.Y.; Finn, C.; et al. Cosmos policy: Fine-tuning video models for visuomotor control and planning. arXiv 2026, arXiv:2601.16163. [Google Scholar]
  141. Yu, J.; Liu, H.; Yu, Q.; Ren, J.; Hao, C.; Ding, H.; Huang, G.; Huang, G.; Song, Y.; Cai, P.; et al. Forcevla: Enhancing vla models with a force-aware moe for contact-rich manipulation. Adv. Neural Inf. Process. Syst. 2026, 38, 93409–93439. [Google Scholar]
  142. Intelligence, P.; Black, K.; Brown, N.; Darpinian, J.; Dhabalia, K.; Driess, D.; Esmail, A.; Equi, M.; Finn, C.; Fusai, N.; et al. π0.5: a Vision-Language-Action Model with Open-World Generalization. arXiv 2025, arXiv:2504.16054. [Google Scholar]
  143. Black, K.; Brown, N.; Driess, D.; Esmail, A.; Equi, M.; Finn, C.; Fusai, N.; Groom, L.; Hausman, K.; Ichter, B.; et al. π0: A Vision-Language-Action Flow Model for General Robot Control. arXiv 2024, arXiv:2410.24164. [Google Scholar]
  144. Wagenmaker, A.; Nakamoto, M.; Zhang, Y.; Park, S.; Yagoub, W.; Nagabandi, A.; Gupta, A.; Levine, S. Steering your diffusion policy with latent space reinforcement learning. arXiv 2025, arXiv:2506.15799. [Google Scholar]
  145. Quevedo, J.; Liang, P.; Yang, S. Evaluating robot policies in a world model. arXiv E-Prints 2025, pp. [Google Scholar]
  146. Chen, B.; Martí Monsó, D.; Du, Y.; Simchowitz, M.; Tedrake, R.; Sitzmann, V. Diffusion forcing: Next-token prediction meets full-sequence diffusion. Adv. Neural Inf. Process. Syst. 2024, 37, 24081–24125. [Google Scholar] [CrossRef]
  147. Yang, F.; Di, D.; Tang, L.; Zhang, X.; Fan, L.; Li, H.; Chen, W.; Su, T.; Ma, B. Chain of World: World model thinking in latent motion. In Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026; pp. 6675–6684. [Google Scholar]
  148. Zheng, Z.; Peng, X.; Yang, T.; Shen, C.; Li, S.; Liu, H.; Zhou, Y.; Li, T.; You, Y. Open-sora: Democratizing efficient video production for all. arXiv 2024, arXiv:2412.20404. [Google Scholar]
  149. Salimans, T.; Ho, J. Progressive Distillation for Fast Sampling of Diffusion Models. In Proceedings of the International Conference on Learning Representations (ICLR), 2022. [Google Scholar]
  150. Song, Y.; Dhariwal, P.; Chen, M.; Sutskever, I. Consistency Models. In Proceedings of the Proceedings of the 40th International Conference on Machine Learning (ICML). PMLR, 2023; Vol. 202. Proceedings of Machine Learning Research. [Google Scholar]
  151. Bastek, J.H.; Sun, W.; Kochmann, D.M. Physics-Informed Diffusion Models. In Proceedings of the International Conference on Learning Representations (ICLR), 2025. [Google Scholar]
  152. Huang, H.; He, S.; Miao, F. CUQDS: Conformal Uncertainty Quantification under Distribution Shift for Trajectory Prediction. arXiv 2024, arXiv:2406.12100. [Google Scholar]
  153. Lee, J.H.; Lin, B.J.; Sun, W.F.; Lee, C.Y. Edeline: Enhancing memory in diffusion-based world models via linear-time sequence modeling. Adv. Neural Inf. Process. Syst. 2026, 38, 105587–105626. [Google Scholar]
  154. Huang, J.; Ye, Z.; Hu, X.; He, T.; Zhang, G.; Shi, S.; Bian, J.; Jiang, L. Live: Long-horizon interactive video world modeling. arXiv 2026, arXiv:2602.03747. [Google Scholar]
  155. Wu, R.; He, X.; Cheng, M.; Yang, T.; Zhang, Y.; Kang, Z.; Cai, X.; Wei, X.; Guo, C.; Li, C.; et al. Infinite-world: Scaling interactive world models to 1000-frame horizons via pose-free hierarchical memory. arXiv 2026, arXiv:2602.02393. [Google Scholar]
  156. An, T.; Jia, J.; Li, G.; Li, J.; Zhou, C.; Liu, P.; Lyu, B.; Bai, J.; Guo, X.; Li, G.; et al. Feedback World Model Enables Precise Guidance of Diffusion Policy. arXiv 2026, arXiv:2605.15705. [Google Scholar]
  157. Open X-Embodiment Collaboration. Open X-Embodiment: Robotic Learning Datasets and RT-X Models. In Proceedings of the 2024 IEEE International Conference on Robotics and Automation (ICRA), 2024.
  158. Liu, S.; Wu, L.; Li, B.; Tan, H.; Chen, H.; Wang, Z.; Xu, K.; Su, H.; Zhu, J. RDT-1B: A Diffusion Foundation Model for Bimanual Manipulation. In Proceedings of the International Conference on Learning Representations (ICLR), 2025. [Google Scholar]
  159. Wen, J.; Zhu, M.; Zhu, Y.; Tang, Z.; Li, J.; Zhou, Z.; Li, C.; Liu, X.; Peng, Y.; Shen, C.; et al. Diffusion-VLA: Generalizable and Interpretable Robot Foundation Model via Self-Generated Reasoning. arXiv 2024, arXiv:2412.03293. [Google Scholar]
  160. Liu, J.; Chen, H.; An, P.; Liu, Z.; Zhang, R.; Gu, C.; Li, X.; Guo, Z.; Chen, S.; Liu, M.; et al. HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Model. arXiv 2025, arXiv:2503.10631. [Google Scholar]
  161. Li, Y.; Zhu, Y.; Wen, J.; Shen, C.; Xu, Y. WorldEval: World Model as Real-World Robot Policies Evaluator. arXiv 2025, arXiv:2505.19017. [Google Scholar]
Figure 1. Overview and organization of the survey on diffusion models for humanoid embodied perception and world modeling.
Figure 1. Overview and organization of the survey on diffusion models for humanoid embodied perception and world modeling.
Preprints 244459 g001
Figure 2. Overview of the diffusion-based whole-body motion capture pipeline proposed in [105]. (a) A fisheye vision transformer corrects image distortion and extracts visual feature tokens. (b) A convolutional network maps these features to a pixel-aligned 3D heatmap, from which body-joint positions and their uncertainties are estimated. (c) Hand locations are detected in the input image, and the corresponding 3D hand poses are regressed. (d) The body and hand estimates are combined and refined by an uncertainty-aware diffusion model to produce the final whole-body motion.
Figure 2. Overview of the diffusion-based whole-body motion capture pipeline proposed in [105]. (a) A fisheye vision transformer corrects image distortion and extracts visual feature tokens. (b) A convolutional network maps these features to a pixel-aligned 3D heatmap, from which body-joint positions and their uncertainties are estimated. (c) Hand locations are detected in the input image, and the corresponding 3D hand poses are regressed. (d) The body and hand estimates are combined and refined by an uncertainty-aware diffusion model to produce the final whole-body motion.
Preprints 244459 g002
Figure 4. Overview of GenPose for estimating an object’s 6D pose from a partial point cloud [120]. (a) Pretraining: paired object poses and point clouds supervise two complementary models through denoising score matching. A score-based diffusion model learns the conditional score field, while an energy-based diffusion model is trained to approximate the same field. (b) Candidate generation starts from Gaussian noise. Conditioned on the observed point cloud, the score-based model guides a reverse probability flow ordinary differential equation (ODE) to generate pose candidates, accommodating ambiguity from partial visibility and object symmetry. (c) Likelihood estimation evaluates each candidate using the energy-based diffusion model. These outputs approximate conditional log-likelihoods up to an additive constant. (d) Pose aggregation ranks candidates by decreasing energy score and retains the top pose energies. Low-ranking outliers are discarded. The final pose combines averaged translations with quaternion-based rotation averaging, yielding the object’s estimated position and orientation.
Figure 4. Overview of GenPose for estimating an object’s 6D pose from a partial point cloud [120]. (a) Pretraining: paired object poses and point clouds supervise two complementary models through denoising score matching. A score-based diffusion model learns the conditional score field, while an energy-based diffusion model is trained to approximate the same field. (b) Candidate generation starts from Gaussian noise. Conditioned on the observed point cloud, the score-based model guides a reverse probability flow ordinary differential equation (ODE) to generate pose candidates, accommodating ambiguity from partial visibility and object symmetry. (c) Likelihood estimation evaluates each candidate using the energy-based diffusion model. These outputs approximate conditional log-likelihoods up to an additive constant. (d) Pose aggregation ranks candidates by decreasing energy score and retains the top pose energies. Low-ranking outliers are discarded. The final pose combines averaged translations with quaternion-based rotation averaging, yielding the object’s estimated position and orientation.
Preprints 244459 g004
Figure 5. From sensory observations to hierarchical predictive control, illustrated by Puppeteer [124]. (a) A tracking world model is pretrained using retargeted motion capture data and simulated interactions to plan actions from proprioceptive information and abstract command. (b) This model is then frozen and reused for downstream tasks, where an imitation world model uses visual and proprioceptive observations to plan future commands for the tracking agent.
Figure 5. From sensory observations to hierarchical predictive control, illustrated by Puppeteer [124]. (a) A tracking world model is pretrained using retargeted motion capture data and simulated interactions to plan actions from proprioceptive information and abstract command. (b) This model is then frozen and reused for downstream tasks, where an imitation world model uses visual and proprioceptive observations to plan future commands for the tracking agent.
Preprints 244459 g005
Figure 6. Overview of LaDi-WM [64], linking visual perception to action-conditioned prediction and policy refinement. (a) Latent world-model learning: frozen DINO [138] and SigLIP [139] networks encode task-agnostic manipulation clips into geometric and semantic representations, respectively. An interactive latent diffusion model learns their coupled dynamics by exchanging estimated clean components through cross-attention, generating future latent states conditioned on observation history and an action sequence. (b) Predictive policy refinement: a policy proposes an initial action sequence from historical observations. The world model predicts future latent states under these actions, and the imagined states are fed back to the same policy alongside the observation history to refine the action sequence. This prediction-refinement cycle can be repeated during inference before execution. The humanoid sequence illustrates the potential reuse of learned dynamics across humanoid manipulation tasks.
Figure 6. Overview of LaDi-WM [64], linking visual perception to action-conditioned prediction and policy refinement. (a) Latent world-model learning: frozen DINO [138] and SigLIP [139] networks encode task-agnostic manipulation clips into geometric and semantic representations, respectively. An interactive latent diffusion model learns their coupled dynamics by exchanging estimated clean components through cross-attention, generating future latent states conditioned on observation history and an action sequence. (b) Predictive policy refinement: a policy proposes an initial action sequence from historical observations. The world model predicts future latent states under these actions, and the imagined states are fed back to the same policy alongside the observation history to refine the action sequence. This prediction-refinement cycle can be repeated during inference before execution. The humanoid sequence illustrates the potential reuse of learned dynamics across humanoid manipulation tasks.
Preprints 244459 g006
Figure 7. Overview of diffusion-based humanoid locomotion, adapted from DreamPolicy [72]. (a) Locomotion skill collection: multiple physically feasible policies are trained through reinforcement learning in simulation. Their rollouts record robot observations, terrain measurements, actions, and rewards, forming a multi-terrain expert dataset. (b) Terrain-aware world modeling: a terrain encoder embeds local height information. Trained on the expert dataset, an autoregressive denoising transformer combines terrain features with proprioception, observation history, and previous actions to generate future humanoid states from Gaussian noise. The predicted sequence provides motion references for subsequent control. (c) Unified policy learning: a goal-conditioned policy uses current observations and predicted motion references to produce joint-control actions. Reinforcement learning combines physical regularization reward, predicted motion, and motion-style rewards. A discriminator compares generated and executed state transitions to encourage consistency with the learned locomotion distribution. After each action, new observations refresh prediction and control, closing the loop from perception to future-state prediction and physical execution.
Figure 7. Overview of diffusion-based humanoid locomotion, adapted from DreamPolicy [72]. (a) Locomotion skill collection: multiple physically feasible policies are trained through reinforcement learning in simulation. Their rollouts record robot observations, terrain measurements, actions, and rewards, forming a multi-terrain expert dataset. (b) Terrain-aware world modeling: a terrain encoder embeds local height information. Trained on the expert dataset, an autoregressive denoising transformer combines terrain features with proprioception, observation history, and previous actions to generate future humanoid states from Gaussian noise. The predicted sequence provides motion references for subsequent control. (c) Unified policy learning: a goal-conditioned policy uses current observations and predicted motion references to produce joint-control actions. Reinforcement learning combines physical regularization reward, predicted motion, and motion-style rewards. A discriminator compares generated and executed state transitions to encourage consistency with the learned locomotion distribution. After each action, new observations refresh prediction and control, closing the loop from perception to future-state prediction and physical execution.
Preprints 244459 g007
Table 1. Comparison with existing surveys related to humanoid perception, diffusion models in robotics, and world modeling. ✓ denotes substantial coverage, ∘ denotes partial or secondary coverage, and – indicates that the topic is not a primary focus.
Table 1. Comparison with existing surveys related to humanoid perception, diffusion models in robotics, and world modeling. ✓ denotes substantial coverage, ∘ denotes partial or secondary coverage, and – indicates that the topic is not a primary focus.
Survey Humanoid- Diffusion Human Self- Environment World Action Uncertainty Physical/ Perception–
centered models perception perception perception models conditioning contact reasoning prediction link
[1] ✓ – ∘ ✓ – – – – ∘ –
[3] ✓ – ∘ ✓ ✓ – – ∘ ∘ –
[26] – ✓ ∘ – ∘ – – ∘ – –
[27] – ✓ – ∘ ∘ ∘ ✓ ∘ ∘ –
[28] – ✓ – – ∘ ∘ ✓ ∘ ∘ –
[29] – ✓ – – ∘ – ✓ ∘ ∘ –
[30] – ∘ – – ∘ ✓ ∘ ∘ ∘ ∘
[31] – ∘ – – ✓ ✓ ✓ ∘ ∘ ∘
[32] – ∘ – – ∘ ✓ ✓ ∘ ∘ ∘
[33] – ∘ – – ∘ ✓ ✓ ∘ ∘ ∘
Ours ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.