LeWAM: Latent evolving World Action Model
cs.CV, cs.RO
Submitted: 2026-09-23
Updated: 2026-10-10
Comments: https://github.com/XuejiFang/LeWAM
Code: https://github.com/XuejiFang/LeWAM
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: World Action Models (WAMs) jointly model action generation and environment dynamics and are mostly built on pretrained Video Diffusion Models (VDMs).
Terminology
Abstract
World Action Models (WAMs) jointly model action generation and environment dynamics and are mostly built on pretrained Video Diffusion Models (VDMs). In VDM-based WAMs, observations are first encoded by a VAE, and the resulting compressed latents are then processed by large video diffusion backbones to extract effective features for action generation. However, this paradigm ties WAM performance and training cost to large-scale video generation pretraining, limiting WAM efficiency and scalability. In this paper, we theoretically and empirically investigate how visual representations affect action generation in WAMs. Our results show that predictive embeddings from Joint-Embedding Predictive Architecture (JEPA) encoders better support action generation than compressed VAE latents, with I-JEPA performing best in our encoder comparison. Based on these findings, we propose LeWAM, which conditions action generation on JEPA embeddings and models environment evolution by predicting future embeddings in the same space, without relying on a video diffusion backbone. We further find that imitation learning matches demonstrated actions but does not distinguish better actions from worse ones, even though small action deviations can greatly affect task success. To address this limitation without additional environment interaction or the human oversight required for resets and safety, we introduce Demonstration-Guided DPO (DemoDPO), an offline preference refinement stage that derives preference supervision directly from demonstrations. With only 0.4B trainable parameters, LeWAM achieves an average success rate of 92.28% on RoboTwin 2.0, comparable to that of state-of-the-art VLAs and WAMs, and maintains practical effectiveness on real-world manipulation tasks.
Sources
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
- RT-H: Action Hierarchies Using Language
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- RT-1: Robotics Transformer for Real-World Control at Scale
- What Makes Pre-Trained Visual Representations Successful for Robust Manipulation?
- LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies
- DINOv3-Diffusion Policy: Self-Supervised Large Visual Model for Visuomotor Diffusion Policy Learning
- Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning
- Long-VLA: Unleashing Long-Horizon Capability of Vision Language Action Model for Robot Manipulation
- GigaBrain-0.5M*: a VLA That Learns From World Model-Based Reinforcement Learning
- Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations
- SpatialBoost: Enhancing Visual Representation through Language-Guided Reasoning
- OpenVLA: An Open-Source Vision-Language-Action Model
- Depth Anything 3: Recovering the Visual Space from Any Views
- DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control
- LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels
- JEPA-VLA: Video Predictive Embedding is Needed for VLA Models
- V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning
- Octo: An Open-Source Generalist Robot Policy
- Intelligent Switching for Reset-Free RL
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models