The Invisible Hand of Physics: When Video Diffusion Models Know More Than They Show

The gist: Video diffusion models contain a clear, decodable signal of physical plausibility within their internal states, even when their generated outputs violate the same physical laws.

Introduction and Motivation

Modern video diffusion models generate increasingly realistic and temporally coherent videos, motivating their use as candidate world simulators Yet it remains unclear whether these models internally encode physical structure, or merely reproduce motion patterns seen during training. Trained on internet-scale video, today’s leading systems such as Sora [OpenAI, 2024], Veo [DeepMind, 2024], and Cosmos [NVIDIA et al., 2026] generate continuations of real scenes with striking temporal coherence. Visual realism alone does not require a model to represent acceleration as constant under gravity, momentum as conserved through a collision, or matter as impenetrable through contact; they require only that the model produces trajectories statistically similar to those it has seen during training. Recent evaluations [Kang et al., 2025, Huberman et al., 2026] have shown that scaling video diffusion models fails to extrapolate basic mechanics outside the training distribution, with generations instead mimicking the nearest in-distribution example.

Methodology for Probing Physical Plausibility

The researchers address the lack of access to latent trajectories by approximately inverting the generation process. This involves starting from a clean video latent and integrating the learned velocity field backward to recover an approximate trajectory through the model’s intermediate states, providing access to its internal representations. The procedure involves several key steps:

1. Recovering Latent Trajectories: They encode a video with the VAE associated with the diffusion model to obtain the clean latent and then integrate the forward ODE in reverse from t=1 back to t=0 to recover noise and the full trajectory.

2. Probing Internal States: They train linear probes on these recovered outputs of each transformer block to predict physical plausibility.

3. Causal Intervention: They perturb transformer blocks during generation by corrupting their output activations, measuring how much the resulting video’s probe-assessed plausibility changes relative to an unmodified baseline.

Key Findings on Physical Decodability

The study demonstrates that physical plausibility is linearly decodable from diffusion transformer states across IntPhys and InfLevel, reaching around 81.27% average accuracy and outperforming dedicated representation-learning baselines such as V-JEPA and VideoMAE. Surprisingly, this signal is absent from the VAE latent input and emerges inside the denoising transformer itself, despite the model not being trained with a self-supervised predictive objective. The physical signal is sustained across most of the reverse trajectory, with strongest linear decodability at intermediate depth. Furthermore, internal states encode quantitative physical parameters beyond binary plausibility; they recover both initial position x0 and initial velocity v0 from a single block with near-perfect linear fit on the controlled physics dataset.

Structure of Physical Signal Emergence

The analysis reveals that the physical signal is organized systematically across the denoising computation, not localized to a single region. Specifically, probe accuracy at intermediate depth is highest for all three diffusion models and matches dedicated representation encoders like V-JEPA 2 and VideoMAE-Large. The best blocks cluster in the middle third of the network, around blocks 15–25 for WAN-1.3B and CogVideoX-2B, suggesting that intermediate-depth emergence is a general property of how video models organize physical computation.

Causal Localization and Trajectory Fidelity

When identifying causal influence, researchers found that the layers where physical information is most linearly decodable do not coincide with those that are most causally sensitive. For WAN-1.3B, early layers (blocks 0–8) exhibit the strongest sensitivity, but mid and later layers retain non-negligible influence. This suggests that physical structure is established early in the computation and propagated through the network, becoming most linearly accessible only at intermediate layers. Additionally, the physical signal is a property of the exact ODE trajectory in the continuum limit, not just its endpoints; coarse discretisation does not follow this path. Accurate discretization using solvers like Heun improves decodability compared to Euler integration.

Conclusion and Implications

The findings suggest that physically meaningful representations can emerge as a byproduct of generative denoising. Physical plausibility is linearly decodable from transformer states, persists across the reverse trajectory, and encodes quantitative physical parameters. These results imply that physical structure is organized throughout the denoising trajectory, with strongest linear decodability at intermediate depth. Future work could leverage these signals for physics-aware guidance and further explore their potential role as latent spaces for world modeling. The analysis relies on approximate reverse sampling, but reconstruction consistency supports its validity. Linear probes demonstrate decodability but do not imply that the model explicitly represents physical laws. These results are correlational, indicating that the physical signal is constructed by the diffusion model itself during transport through its learned flow. The paper concludes that physically meaningful representations can emerge as a byproduct of generative denoising.

--- Page 1 ---

The Invisible Hand of Physics: When Video Diffusion Models Know More Than They Show Parsa Esmati∗,1 Somjit Nath∗,2,3 Katja Hofmann4 Derek Nowrouzezahrai2,3 Samira Ebrahimi Kahou†,3,5 Majid Mirmehdi†,1

--- Page 2 ---

Abstract Modern video diffusion models generate increasingly realistic and temporally coherent videos, motivating their use as candidate world simulators. Yet it remains unclear whether these models internally encode physical structure, or merely reproduce motion patterns seen during training. We study this question by probing video diffusion models along latent trajectories corresponding to real videos with known physical plausibility. To obtain such trajectories, we approximately invert the deterministic sampling process by integrating the learned velocity field backward from a clean video latent to noise, giving access to the model’s intermediate states and attention maps. Using these recovered trajectories, we show that physical plausibility is linearly decodable from diffusion transformer states across IntPhys and InfLevel, reaching around 81.27% average accuracy and outperforming dedicated representation-learning baselines such as V-JEPA and VideoMAE. Surprisingly, this signal is absent from the VAE latent input and emerges inside the denoising transformer itself, despite the model not being trained with a self-supervised predictive objective. These findings suggest that physically meaningful representations can arise as a byproduct of generative denoising.

--- Page 3 ---

1 Introduction The trajectory of video generation has been driven by a continual expansion of what these models are expected to capture. Early generative models [Vondrick et al., 2016, Tulyakov et al., 2017] aimed to produce short, visually plausible clips, but the rise of large-scale diffusion-based generators [Ho et al., 2022b] have shifted the goal toward modelling not only the visual aspects but also the dynamics of the visual world itself. Trained on internet-scale video, today’s leading systems such as Sora [OpenAI, 2024], Veo [DeepMind, 2024], and Cosmos [NVIDIA et al., 2026] generate continuations of real scenes with striking temporal coherence, and are now being positioned as a path toward generalpurpose simulators of the physical world for robotics, planning, and scientific discovery. Despite these advances, it remains unclear whether such models have actually internalized the physical laws governing the scenes they generate. Visual realism alone does not require a model to represent acceleration as constant under gravity, momentum as conserved through a collision, or matter as impenetrable through contact; they require only that the model produces trajectories statistically similar to those it has seen during training. Recent evaluations [Kang et al., 2025, Huberman et al., 2026] have shown that scaling video diffusion models fails to extrapolate basic mechanics outside the training distribution, with generations instead mimicking the nearest in-distribution example. Compounding this, large-scale video diffusion models operate in the latent space of a variational autoencoder (VAE) trained purely for reconstruction, which is not explicitly optimized to capture the semantic or physical structure that representation encoders are known to encode [Bardes et al., 2024, Preprint. arXiv:2606.05328v1 [cs.GR] 3 Jun 2026]. The model therefore has neither an explicit objective nor an implicit substrate that would push it to recover the laws governing the dynamics it is asked to generate.

--- Page 4 ---

This raises a fundamental question: do modern video generation models encode physical knowledge internally, even when their output fails to capture it? Prior work has mainly approached this question through representation learning. Self-supervised encoders such as V-JEPA [Bardes et al., 2024, Assran et al.

1. Improve generative video models by internalizing physical laws through denoising computation, as physically meaningful representations can emerge as a byproduct of generative denoising. This enables models to generate videos that are not just statistically similar to training data but adhere to underlying physics, potentially leading to more robust and general-purpose simulators for robotics and scientific discovery.

2. Develop causal localization mechanisms for physical understanding by employing perturb transformer blocks during generation and measure the resulting change in probe-assessed physical plausibility. This allows researchers to pinpoint which components of the model causally influence physical plausibility during generation, guiding targeted architectural improvements rather than relying on correlational evidence.

3. Enhance model capacity for quantitative parameter encoding by designing architectures that balance spatiotemporal semantics with texture detail, as a wider DiT has the capacity to do both and encodes quantitative physical parameters. This could lead to models that not only judge binary plausibility but also accurately predict specific physical variables like initial position or velocity, as seen in the linear regressor results.

4. Establish trajectory fidelity as a critical factor for physical decodability by ensuring that the integration step size is sufficiently small to compute physical information, since the discretisation needed to generate a physically plausible-looking video is therefore strictly coarser than the discretisation needed to compute physical information. This dictates better training objectives for diffusion models aimed at preserving the exact ODE trajectory.

5. Implement dynamic probing strategies that test for physical structure across the entire denoising process, as physical signal is sustained across most of the reverse trajectory and strongest linear decodability at intermediate depth, suggesting that representations must be assessed throughout time, not just at a single point like the midpoint.