What Visual Representations Do World Action Models Need?
cs.CV, cs.RO
Submitted: 2026-10-09
Updated: 2026-10-09
Comments: 24 pages. Project page: https://kailiu18.github.io/R-2WAM.github.io/
Project page: https://kailiu18.github.io/R-2WAM.github.io
License: http://creativecommons.org/licenses/by/4.0/
The gist: World Action Models (WAMs) inherit powerful priors from pretrained video generation models, but their variational autoencoder (VAE) representations are optimized for visual reconstruction rather than
Abstract
World Action Models (WAMs) inherit powerful priors from pretrained video generation models, but their variational autoencoder (VAE) representations are optimized for visual reconstruction rather than manipulation. External visual representations can improve manipulation, but existing methods couple representation choices with different integration mechanisms, obscuring their individual effects. We systematically compare visual representations, five integration mechanisms, and feature combinations under controlled settings. Our study yields three main findings. First, strong general-purpose visual representations provide broader robustness gains and can match or outperform representations pretrained for specific visual capabilities. Second, channel-wise joint prediction achieves the highest average success rate; further analyses suggest that future-feature prediction and the action expert's effective use of added visual information contribute to its gains. Third, additional sources should be selected for the task-relevant information they contribute beyond existing representations, as the same source can help one combination but harm another. However, even the best evaluated composition provides a smaller additional gain than the initial improvement from adding a strong general-purpose representation to the VAE-only baseline. We instantiate the resulting Representation Recipe as R squared-WAM. It achieves state-of-the-art performance on RoboCasa GR1 Tabletop with an 84.7% success rate and reaches 92.8% on RoboTwin 2.0, improving over the VAE-only baselines by 5.9 and 2.2 percentage points, respectively. Taken together, these findings suggest that future work should prioritize stronger general-purpose visual representations and more effective mechanisms for integrating them into WAMs, with feature composition serving as a supplementary route to improvement.
Sources
- InternVLA-A1: Unifying Understanding, Generation and Action for Robotic Manipulation
- SAM 3: Segment Anything with Concepts
- LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies
- DIAL: Decoupling Intent and Action via Latent World Modeling for End-to-End VLA
- Vision Pretraining for Dense Spatial Perception
- Robust Learning with Jacobian Regularization
- DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation
- Causal World Modeling for Robot Control
- WALA Learning Executable Latent Actions from Action-Labeled Demonstrations and Action-Free Videos
- GeoSem-WAM: Geometry- and Semantic-Aware World Action Models
- DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control
- Making Foresight Actionable: Repurposing Representation Alignment in World Action Models
- C-RADIOv4 (Tech Report)
- DreamWorld: Unified World Modeling in Video Generation
- SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
- Wan: Open and Advanced Large-Scale Video Generative Models
- RepWAM: World Action Modeling with Representation Visual-Action Tokenizers
- ST-WAM: Semantic-Temporal World Action Model for Robust Manipulation under Visual Distribution Shifts
- Flex-$\pi$: A Multi-Stream World-Action Model with Compute Flexibility
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models