Achieve What You Imagined: Learning to Align Actions with Visual Plans
cs.RO, cs.AI, cs.LG
Submitted: 2026-09-27
Updated: 2026-09-27
Comments: 9 pages, 8 figures, 2 tables
Project page: https://imagine-to-achieve.github.io
License: http://creativecommons.org/licenses/by/4.0/
The gist: World-action models can jointly predict future visual observations and robot actions.
Terminology
Abstract
World-action models can jointly predict future visual observations and robot actions. However, discrepancies may exist between their visual predictions and the consequences implied by generated actions. We observe that WAMs can often generate visually plausible task-completion outcomes before producing action sequences that reliably achieve them. Consequently, we treat the WAM-generated visual prediction as a goal-conditioned visual proposal rather than a directly executable plan. We use a frozen action-conditioned world model to predict action-conditioned consequences and construct feedback based on consistency between the two future predictions and alignment with the terminal goal. Leveraging this feedback, we employ Flow Policy Optimization (FPO) to optimize the action head of the WAM. This framework avoids online robot interaction and additional training of task-specific reward models. Across four real-world UR5 manipulation tasks, our method increases the mean success rate from 43.4% to 75.1%, compared with 61.4% for π 0.5. These results show that cross-model prediction discrepancy can provide useful feedback for improving robot policies under the evaluated manipulation tasks. Website: https://imagine-to-achieve.github.io/
Sources
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
- Turning Video Models into Generalist Robot Policies
- World Action Models are Zero-shot Policies
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
- LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies
- $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies