World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal
cs.RO, cs.AI
Submitted: 2026-09-24
Updated: 2026-09-24
Comments: Working in progress
License: http://creativecommons.org/licenses/by/4.0/
The gist: General-purpose vision-language models (VLMs) bring broad knowledge and spatial reasoning to robot manipulation, yet existing systems either use them indirectly, to predict constraints or write
Terminology
Abstract
General-purpose vision-language models (VLMs) bring broad knowledge and spatial reasoning to robot manipulation, yet existing systems either use them indirectly, to predict constraints or write programs, or give them a view of the scene rather than a world in which to act. We present World Action Agent (WAA), a multi-agent harness through which VLMs pilot robots with basic tools, making every decision within a visual action workspace. The workspace has three properties. Contact views, selected automatically from the scene geometry, present the scene around the current interaction. Action rehearsal turns each action into an editable proposal that the agent, alone or through an Imagination Agent, previews and revises against planning feedback before execution. In-view correction closes the loop between observation, rehearsal, and low-level execution, letting the agent remove residual offsets in the view where it observes them. Through the same workspace, WAA acquires embodied procedural knowledge in two ways: it evolves multimodal skills from expert videos and human teaching under evidence-based review and consults them through a Skill Agent, and its interaction traces train smaller VLMs to pilot the same harness. On LIBERO-Pro, WAA with skills evolved only from LIBERO-90 reaches a state-of-the-art 75.6% average success, outperforming end-to-end VLAs, code-as-policy agents, and a visual-harness baseline with the same backbone; the same skills remain effective on robosuite without further learning. Fine-tuning Qwen3.5-9B on harness traces raises its out-of-domain success from 1.7% to 43.3%.
Sources
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
- RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation
- GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks
- Show-Harness: Just a VLM Agent Can Play Robots
- VIA: Visual Interface Agent for Robot Control
- $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
- Agent as Policy for Robotic Manipulation
- VISUALSKILL: Multimodal Skills for Computer-Use Agents
- OpenVLA: An Open-Source Vision-Language-Action Model
- Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning
- Causal World Modeling for Robot Control
- Unified Video Action Model
- JEPA-WAM: Learning Vision-Language-Action Policies with Joint-Embedding World Modeling
- Sci-VLA: Agentic VLA Inference Plugin for Long-Horizon Tasks in Scientific Experiments
- World Action Models: A Survey
- Fast-WAM: Do World Action Models Need Test-time Future Imagination?
- Playful Agentic Robot Learning
- MMSkills: Towards Multimodal Skills for General Visual Agents
- Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents
- LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies