DSDyn-VLA: A Dual-Stream Dynamic Manipulation Framework with Motion Perception, Future Awareness, and Realtime Correction
cs.RO
Submitted: 2026-09-30
Updated: 2026-09-30
License: http://creativecommons.org/licenses/by/4.0/
The gist: While Vision-Language-Action (VLA) models excel in static tasks, they struggle in dynamic environments where objects are in motion (e.g., conveyor belt manipulation).
Terminology
Abstract
While Vision-Language-Action (VLA) models excel in static tasks, they struggle in dynamic environments where objects are in motion (e.g., conveyor belt manipulation). We identify three fundamental limitations hindering current VLAs in these scenarios: the perception gap, where static visual inputs lack temporal motion cues; the latency gap, where inference delays render actions obsolete; and the control gap, caused by the open-loop action chunk execution without real-time adjustment. In this work, we propose DSDyn-VLA, a Slow-Fast Dual-Stream Dynamic manipulation framework that integrates motion-aware foresighted planning with real-time residual correction. The slow Flow-Planner serves as a macro-planner. By enhancing the VLA with optical flow for temporal perception and a future state awareness mechanism to preemptively offset inference latency, it produces globally consistent, motion-aware action chunks. Complementing this, the fast Res-Refiner employs a lightweight RL policy to inject high-frequency, closed-loop corrections into the planned action chunks based on real-time observations. In addition, we introduce DynBench, a MuJoCo-based benchmark for dynamic object manipulation that comprises nine tasks. Extensive experiments demonstrate that DSDyn-VLA reduces the failure rate by over 76% compared to current SOTA method in high-latency setting on the Kinetix dynamic benchmark, while achieving about 6 times the success rate of PI0.5 in real-world dynamic settings and about 5 times on DynBench. We will open-source all the code and weights.
Sources
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- Real-Time Execution of Action Chunking Flow Policies
- Training-Time Action Conditioning for Efficient Real-Time Chunking
- AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems
- Towards Generalizable Robotic Manipulation in Dynamic Environments
- VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models
- $\pi^{*}_{0.6}$: a VLA That Learns From Experience
- AsyncVLA: Asynchronous Flow Matching for Vision-Language-Action Models
- DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
- OpenVLA: An Open-Source Vision-Language-Action Model
- CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation
- RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation
- Bidirectional Decoding: Improving Action Chunking via Guided Test-Time Sampling
- VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning
- Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks
- Reinforcement Learning for Flow-Matching Policies
- Leave No Observation Behind: Real-time Correction for VLA Action Chunks
- SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics
- VLASH: Real-Time VLAs via Future-State-Aware Asynchronous Inference
- Octo: An Open-Source Generalist Robot Policy
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies