Dynamic Manipulation with World-Action Models via Counterfactual Planning
cs.RO, cs.AI, cs.CV, cs.LG
Submitted: 2026-09-27
Updated: 2026-09-27
Comments: 48 pages, including appendices. Project page: https://methoder00.github.io/DPP/
Project page: https://methoder00.github.io/DPP
License: http://creativecommons.org/licenses/by/4.0/
The gist: World-Action models (WAMs) trained on static demonstrations often fail to manipulate moving targets even when they possess the required manipulation skills.
Terminology
Abstract
World-Action models (WAMs) trained on static demonstrations often fail to manipulate moving targets even when they possess the required manipulation skills. We attribute this failure to target-response collapse: as execution advances, the policy becomes increasingly biased toward the learned continuation of its ongoing behavior and less responsive to target relocation. To bridge the gap between what the model has learned and what it can generate from the current context, we formulate dynamic manipulation as counterfactual planning by decoupling the context used for plan generation from the physical state used for execution. Our framework, Dynamic Predictive Planning (DPP), first uses the WAM's predictive rollout to estimate when an interaction is expected to occur, and combines this timing estimate with observed target motion to predict the target's future interaction position. DPP then constructs a counterfactual observation that places this predicted target position in a familiar robot context, allowing the model to invoke an existing manipulation skill rather than generate a recovery behavior from an unfamiliar robot-target configuration. The resulting plan is connected to the robot's actual state during execution. DPP enables real-time dynamic manipulation on a single consumer GPU without additional training on dynamic data. Experiments in simulation and on a real robot demonstrate consistent improvements across diverse target motions, with simulation performance surpassing all evaluated baselines, including methods additionally trained on dynamic data. Project page: https://methoder00.github.io/DPP/
Sources
- Flash-WAM: Modality-Aware Distillation for World Action Models
- Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models
- AHA-WAM:Asynchronous Horizon-Adaptive World-Action Modeling with Observation-Guided Context Routing
- GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation
- Self-Supervised Visual Planning with Temporal Skip Connections
- Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
- TEMPO: Learning Temporal Context for Dynamic Robot Manipulation
- RLDX-1 Technical Report
- DART: Noise Injection for Robust Imitation Learning
- Efficient-WAM: A 1B-Parameter World-Action Model with Low-Cost Future Imagination
- Adaptive Action Chunking at Inference-time for Vision-Language-Action Models
- Delay-Aware Diffusion Policy: Bridging the Observation-Execution Gap in Dynamic Tasks
- DynamicManip: Enabling Dynamic Manipulation from a Single Static Demonstration
- VLS: Steering Pretrained Robot Policies via Vision-Language Models
- DynamicWAM: Dual-Path Motion Conditioning for World-Action Models in Dynamic Manipulation
- RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins
- Cosmos 3: Omnimodal World Models for Physical AI
- GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
- VLA-Corrector: Lightweight Detect-and-Correct Inference for Adaptive Action Horizon
- SAM 2: Segment Anything in Images and Videos
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies