DriftOPD: Sequence-Level Reverse-KL Distillation for One-Step VLA Policies
cs.RO, cs.AI, cs.CV, cs.LG
Submitted: 2026-09-29
Updated: 2026-09-29
Comments: Preprint
Project page: https://yj-jun.github.io/DriftOPD/ABSTRACT
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Vision-Language-Action (VLA) models increasingly rely on action experts that generate short action chunks under receding-horizon control.
Terminology
Abstract
Vision-Language-Action (VLA) models increasingly rely on action experts that generate short action chunks under receding-horizon control. While chunk-level training is convenient across robot embodiments, it optimizes local action likelihood without explicitly accounting for long-horizon task success. Sequence-level reinforcement learning can address this limitation, but typically requires policy rollouts and closed-loop interaction, which are costly for real-robot manipulation. We introduce DriftOPD, a teacher-free, rollout-free framework for sequence-level on-policy distillation of continuous VLA action experts. We show that the sequence-level reverse Kullback-Leibler (KL) divergence decomposes into a chunk-level reverse-KL term and a future-potential term that captures the long-horizon effect of the current action. DriftOPD optimizes these two terms using a one-step drifting objective and a Q-function critic learned from offline demonstrations, respectively, enabling sequence-level optimization with only offline data and one-step action generation. Across multiple VLA architectures in simulation and real-world manipulation, DriftOPD generally outperforms existing one-step distillation baselines while achieving task success performance comparable to multi-step teacher policies. These results demonstrate that long-horizon behavior can be effectively distilled into one-step VLA action experts without online interaction or a separate teacher.
Sources
- GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
- Mean-Flow based One-Step Vision-Language-Action
- Target Score Matching
- Generative Modeling via Drifting
- Flow-OPD: On-Policy Distillation for Flow Matching Models
- LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models
- Drift Q-Learning
- Score and Distribution Matching Policy: Advanced Accelerated Visuomotor Policies via Matched Distillation
- Soft Truncation: A Universal Training Technique of Score-based Diffusion Model for High Precision Score Estimation
- RLDX-1 Technical Report
- VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulators
- DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models
- One-Step Flow Policy: Self-Distillation for Fast Visuomotor Policies
- VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning
- DriftWorld: Fast World Modeling through Drifting
- Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillation
- SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics
- VGAS: Value-Guided Action-Chunk Selection for Few-Shot Vision-Language-Action Adaptation
- ABot-M0: VLA Foundation Model for Robotic Manipulation with Action Manifold Learning
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies