Adjoint Guidance Flow: Amortized Critic Guidance for VLA Policies
cs.LG, cs.CV, cs.RO
Submitted: 2026-09-28
Updated: 2026-09-28
Project page: https://jeongsol-kim.github.io/agf/project_page/1
License: http://creativecommons.org/licenses/by/4.0/
The gist: Flow-based Vision-Language-Action (VLA) policies are typically trained by behavior cloning and thus do not explicitly optimize long-term task return.
Terminology
Abstract
Flow-based Vision-Language-Action (VLA) policies are typically trained by behavior cloning and thus do not explicitly optimize long-term task return. Critic guidance steers generation toward higher-value actions, but existing methods differentiate the critic through a one-step surrogate of the sampler and back-propagate a critic ensemble at every flow step. In contrast, here we propose Adjoint Guidance Flow (AGF), which amortizes trajectory-aware critic guidance into a lightweight guidance network while preserving the pretrained VLA policy. Specifically, we formulate critic-guided flow generation as a deterministic optimal control problem, whose optimal guidance is a costate that carries the terminal critic gradient back through the remaining flow, and regress the guidance network onto this costate while keeping both the VLA and critic frozen. This design provides favorable memory and throughput scaling during training, and inference needs one guidance-network forward pass per step, without the critic ensemble, back-propagation, or adjoint computation. Across LIBERO, RoboCasa, and LIBERO-Pro, AGF consistently improves pretrained VLAs, remains competitive with critic-guidance and policy-fine-tuning baselines, and is the most robust method when a single guidance strength is deployed across tasks. Compared with QGF, AGF runs 3.6 times faster per guidance step with 7.0 times fewer parameters, with comparable and even better performance, showing that critic guidance can be trajectory-aware and lightweight.
Sources
- GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
- MolmoAct2: Action Reasoning Models for Real-world Deployment
- $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
- OpenVLA: An Open-Source Vision-Language-Action Model
- Reinforcement Learning: An Overview
- RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots
- QPILOTS: Efficient Test-Time Q-Steering for Flow Policies
- Guided Action Flow: Value-Guided Sampling for Frozen Vision-Language-Action Policies
- Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks