When Does Latent Prediction Error Support Physical Surprise Claims in Video World Models?
cs.CV, cs.LG
Submitted: 2026-11-01
Updated: 2026-11-01
Comments: 28 pages, 12 figures, 25 tables. Accepted at the World Models in Physical AI Workshop, NeurIPS 2026
Code: https://github.com/3-ml/jepa-surprise
License: http://creativecommons.org/licenses/by/4.0/
The gist: Prediction errors from video world models are often treated as physical surprise, but they can also reflect appearance, static content, motion, or ordinary prediction difficulty.
Terminology
Abstract
Prediction errors from video world models are often treated as physical surprise, but they can also reflect appearance, static content, motion, or ordinary prediction difficulty. We present an evaluation framework that separates what frozen representations contain, how they organize appearance and physical differences, and what changes a model's own prediction error. On IntPhys2, a benchmark of physically possible and impossible events, we compare representations from latent prediction, masked pixel reconstruction, and simple visual statistics. Appearance remains easy to decode, while V-JEPA2 exposes modest information about whether events are possible or impossible. Looking at the latent geometry, appearance changes become smaller compared with differences between videos, their directions line up more consistently across scenes, and their overlap with physical changes decreases in deeper layers. Across action-conditioned models, incorrect actions raise prediction error, while appearance changes also produce substantial responses. In Physion++ video clips, repeating a single frame can produce higher error than the original clip, while changing color throughout the clip can lower it. Using Kubric, a simulation and rendering framework, we find that impossible events can produce lower error than possible events. A fixed error threshold also incorrectly flags many possible events as impossible. A model can therefore represent physical information and respond to incorrect actions without its native prediction error reliably measuring physical surprise.
Sources
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
- Back to the Features: DINO as a Foundation for Video World Models
- LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics
- Revisiting Feature Prediction for Learning Visual Representations from Video
- Physion: Evaluating Physical Prediction from Vision in Humans and Machines
- CRONOS: Benchmarking Counterfactual Physical Consistency in Video Models
- IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environments
- Intuitive physics understanding emerges from self-supervised pretraining on natural videos
- Kubric: A scalable dataset generator
- Interpreting Physics in Video World Models
- LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels
- Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Models
- How Do Video Foundation Models Encode Intuitive Physics? Probing Across Pretraining Paradigms
- IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning
- What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?
- VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
- Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Properties
- VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking
- Cloning Deterministic Worlds: The Critical Role of Latent Geometry in Long-Horizon World Models
- What Makes Video World Model Latents Action-Relevant: Prediction over Reconstruction
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs