Future Representation Distillation for Imagination-free World Action Models
cs.RO
Submitted: 2026-11-03
Updated: 2026-11-04
Comments: Project page: https://harryjo97.github.io/dreamfree/
Code: https://github.com/harryjo97/dreamfree
Project page: https://harryjo97.github.io/dreamfree
License: http://creativecommons.org/licenses/by/4.0/
The gist: World Action Models (WAMs) use the predictive knowledge of pretrained video diffusion models for robot control through future video prediction.
Terminology
Abstract
World Action Models (WAMs) use the predictive knowledge of pretrained video diffusion models for robot control through future video prediction. However, iterative video denoising is costly, and inaccurate visual predictions can compromise action prediction. Our key idea is to make this predictive knowledge available through visual representations computed from the current observation, enabling action prediction without generating future videos. We introduce DreamFree, a WAM that repurposes a pretrained video diffusion model as a predictive visual encoder through representation distillation. The visual encoder learns to predict a frozen video diffusion model's representation using only the current observation. An action module is then trained to predict actions by extracting action-relevant information from the visual encoder. Across real-world and simulated tasks, DreamFree achieves higher success rates than state-of-the-art VLAs and WAMs, while performing real-time closed-loop control at 11Hz with a 14B video diffusion backbone. Experiments show that DreamFree can translate stronger video priors and additional video data into better robot control and enable learning new robot tasks from video demonstrations, including videos from a different embodiment, without action annotation for the new tasks.
Sources
- Scalable Behavior Cloning with Open Data, Training, and Evaluation
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
- GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
- DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos
- DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
- Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success
- Causal World Modeling for Robot Control
- DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control
- RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots
- mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs
- $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
- Wan: Open and Advanced Large-Scale Video Generative Models
- GigaWorld-Policy: An Efficient Action-Centered World--Action Model
- World Action Models are Zero-shot Policies
- Fast-WAM: Do World Action Models Need Test-time Future Imagination?
- LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfer
- FLARE: Robot Learning with Implicit World Modeling
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation