CtrlWAM: Controllable World Action Models with Aligned Intent and Foresight
cs.CV
Submitted: 2026-10-01
Updated: 2026-10-01
Comments: Project page: https://ctrl-wam.github.io/
Code: https://github.com/NVlabs/alpasim
Project page: https://ctrl-wam.github.io
License: http://creativecommons.org/licenses/by/4.0/
The gist: World action models (WAMs) jointly predict actions (intent) and visual future (foresight).
Terminology
Abstract
World action models (WAMs) jointly predict actions (intent) and visual future (foresight). Standard training adds noise to recorded actions and video simultaneously, but such training paradigms introduce a mismatch: perturbed actions imply counterfactual future visual, while the noised video remains tied to the GT recording. In low-noise regime, the scene geometry and even the dynamic behavior remain clearly visible from the noisy future frames despite the added noise. We present CtrlWAM, which executes perturbed actions in a simulator and pairs them with their noised visual consequences for joint WAM learning. To accommodate the different denoising requirements of video and actions, we introduce warped video--action noise schedules that aim to keep visual layout responsive as action predictions evolve. We further extend the action interface from ego-only control to a variable number of agent streams, allowing a unified model to represent predicted or commanded futures for multiple agents. Driving experiments show more accurate action forecasts, closer agreement between generated video and actions, and better following of supplied commands; robotics experiments show stronger motion fidelity and controllability. Matched controls support the benefit of off-path renders for command following and manipulation fidelity. Together, these findings contribute to a more controllable world action model. Project page: https://ctrl-wam.github.io/
Sources
- Cosmos 3: Omnimodal World Models for Physical AI
- RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation
- Unified 4D World Action Modeling from Video Priors with Asynchronous Denoising
- GAIA-1: A Generative World Model for Autonomous Driving
- Unified Video Action Model
- Flow Matching for Generative Modeling
- Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
- Beyond FVD: Enhanced Evaluation Metrics for Video Generation Quality
- DIDO: Distilling Interaction-Centric Dynamics into One-Step Denoising for World Action Models
- Wan: Open and Advanced Large-Scale Video Generative Models
- World Action Models are Zero-shot Policies
- Fast-WAM: Do World Action Models Need Test-time Future Imagination?
- EWMBench: Evaluating Scene, Motion, and Semantic Quality in Embodied World Models
- Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models