Latent Policy Steering through One-Step Flow Policies
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "Latent Policy Steering through One-Step Flow Policies".
Dev: Here's the gist of the paper. Offline reinforcement learning (RL) allows robots to learn from offline datasets without risky exploration, yet offline RL’s performance often hinges on a brittle trade-off between return maximization and behavioral constraints.
Rosa: Latent Policy Steering (LPS) proposes a method that enables high-fidelity latent policy improvement by backpropagating original-action-space Q-gradients through a differentiable one-step MeanFlow policy to update a latent-action-space actor.
Dev: The Core Problem Addressed The paper identifies two practical bottlenecks for real-world offline RL: the sensitivity induced by explicit behavior regularization and the approximation error induced by indirect latent distillation.
Rosa: Existing methods often rely on a delicate trade-off controlled by a hyperparameter α, which is highly sensitive to reward scale and dataset diversity, making extensive tuning prohibitive in real-world deployment.
Title and authors: Rosa: So we're looking at the paper titled Latent Policy Steering through One-Step Flow Policies. It sounds a bit technical, but basically, it’s about how robots learn from old data without having to try things out in the real world again.
Dev: Yeah, it’s about this method that tries to fix a big problem in offline reinforcement learning where you usually have to pick a tricky balance between just getting high rewards and making sure the robot doesn't do something totally weird that was never seen before.
Taro: It sounds like they are tackling the issue of safety constraints versus performance maximization in these offline datasets.
Rosa: Exactly, that’s what it is. The authors introduce this Latent Policy Steering method as a structural way to keep the policy inside the range of what the old data actually shows while still trying to get good results from those actions.
Dev: And they propose using something called MeanFlow, which is a differentiable one-step generative model, to backpropagate those original action space Q-gradients all the way up into a latent action space actor.
Taro: So it’s taking information directly from the reward calculation and steering the latent policy based on that instead of some indirect estimation.
Rosa: That’s right. The paper is focused on replacing those indirect methods—like using latent critics to estimate values—with this direct gradient flow, which they call Latent Policy Steering through One-Step Flow Policies.
The paper's summary: Dev: So the core idea, based on the abstract and page one of Latent Policy Steering through One-Step Flow Policies, is that they are trying to avoid two specific bottlenecks in offline RL.
Rosa: Right, they identified sensitivity from explicit behavior regularization and approximation error from using latent distillation. They think existing methods often rely on a hyperparameter alpha that you have to tune a lot based on how big the reward scale is or how diverse the dataset is.
Dev: And they point out that distilled latent critics can give poor gradients because just matching values doesn't guarantee the latent gradients used for improvement are accurate near sharp boundaries in the data manifold.
Taro: It sounds like they’re saying that these existing methods aren't robust enough because of how those value estimates are created or how they enforce behavior constraints.
Rosa: That’s the problem they want to solve. Their solution, Latent Policy Steering through One-Step Flow Policies, decouples behavioral constraints from reward maximization by splitting them up.
Dev: So instead of trying to find a single weight for regularization, they have a fixed generative behavior policy that defines what actions are allowed—the support—and then a latent actor does the value-driven steering based on that.
Taro: That separation sounds smart. It means the agent can focus purely on improving the policy without having to constantly worry about tuning this explicit behavioral regularization weight.
Rosa: Precisely, and they claim this decoupling results in an out-of-the-box method that consistently matches or even surpasses behavioral cloning across standard offline RL benchmarks.
The paper's improvements: Dev: Where the real gain comes from, they say it’s by directly backpropagating action space critic gradients through a differentiable one-step MeanFlow policy to update the latent actor.
Rosa: That means they are bypassing the need for those proxy latent critics entirely. Instead of using an estimated value, they are using the original action space critic to guide the end-to-end optimization in the latent space.
Taro: So it’s not about learning a good approximation of a value function first, but using what we already have—the action space critic—to improve the policy structure directly.
Dev: And they show that this direct optimization leads to consistently better performance than behavioral cloning and strong latent steering baselines on standard offline RL benchmarks.
Rosa: They claim this method is robust because it doesn't need task-specific tuning for that explicit behavioral regularization weight, which is a huge deal for real-world deployment.
Taro: But what about the limitations they mention? They say the gains come from this direct backpropagation, but they also have to be careful about the quality of the base policy that generates those gradients.
Dev: They also introduce a spherical latent geometry constraint to keep things stable, synchronizing the support of the base policy and actor output onto a hypersphere with radius sqrt(d).
Rosa: That spherical constraint is intentional, they say it’s conservative. It might restrict extrapolation to behaviors far beyond what the original demonstrations showed.
Conclusion: Dev: So to wrap up Latent Policy Steering through One-Step Flow Policies, the paper shows a framework that combines direct value-based improvement with structural safety constraints from a generative prior.
Rosa: The main implication is that you can get high-fidelity latent policy improvement without having to tune parameters for behavioral regularization, which makes it much more practical.
Taro: I just want to push on the real-world aspect for a second. They say this method mitigates failures like premature release or repetitive motion loops in real robot tasks by steering the latent policy toward high-value regions.
Dev: And they showed that this adaptation happens really fast when fine-tuning online, actually surpassing DSRL within limited steps on tasks like the insert pen task.
Rosa: So, in short, Latent Policy Steering through One-Step Flow Policies gives us a practical, tuning-free approach that outperforms prior latent steering methods across simulation and real robots.
Taro: I just think the spherical constraint is conservative because if it doesn't capture all the necessary modes of behavior, then the whole system might miss something important.
Dev: True, but they also showed that replacing a standard normal prior with a spherical prior didn't actually degrade baseline performance on those benchmarks.
Rosa: So, Latent Policy Steering through One-Step Flow Policies is presented as a superior policy extraction mechanism that avoids the complexities of QCFQL paradigms.
Hokyun Im, Andrey Kolobov, Jianlong Fu, Youngwoon Lee
Department of Artificial Intelligence, Yonsei University · Microsoft Research
cs.RO, cs.LG
Submitted: 2026-03-05
Updated: 2026-10-29
Comments: RSS 2026, Project Webpage : https://jellyho.github.io/LPS/
Project page: https://jellyho.github.io/LPS/Abstract
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 96/100
The gist: Offline reinforcement learning (RL) allows robots to learn from offline datasets without risky exploration…
Key concepts
- Latent Policy Steering (LPS)
- LPS is a method that steers a latent actor to improve its policy by directly using gradients from the action space critic. It achieves this by passing the critic's gradients through a differentiable one-step generative policy, allowing for high-fidelity updates without needing complex or brittle proxy latent critics.
- MeanFlow Policy
- MeanFlow is used as a differentiable one-step generative base policy in LPS. This component allows the system to efficiently generate actions in a single step, which is crucial for backpropagating the critic's gradients directly to update the latent actor's parameters effectively.
- Spherical Latent Geometry
- This constraint limits the queries of the latent actor by forcing its outputs onto a hypersphere. This synchronization ensures that the latent actor's output and the base policy's support are aligned, helping to maintain stable learning and performance across different tasks.
Terminology
Summary
Offline reinforcement learning (RL) allows robots to learn from offline datasets without risky exploration, yet offline RL’s performance often hinges on a brittle trade-off between return maximization and behavioral constraints. p. 0 Latent Policy Steering (LPS) proposes a method that enables high-fidelity latent policy improvement by backpropagating original-action-space Q-gradients through a differentiable one-step MeanFlow policy to update a latent-action-space actor. p. 0
The Core Problem Addressed
The paper identifies two practical bottlenecks for real-world offline RL: the sensitivity induced by explicit behavior regularization and the approximation error induced by indirect latent distillation. Existing methods often rely on a delicate trade-off controlled by a hyperparameter α, which is highly sensitive to reward scale and dataset diversity, making extensive tuning prohibitive in real-world deployment. p. 0 Furthermore, distilled latent critics can provide poor gradients because matching values does not guarantee that the latent gradients used for improvement are accurate near sharp boundaries of the data manifold. p. 2
How the Training Signal is Built
LPS decouples behavioral constraints from reward maximization by separating them into a fixed generative behavior policy defining the support and a latent actor performing value-driven steering. This decoupling allows the agent to focus on policy improvement without tuning an explicit behavioral regularization weight, resulting in an out-of-the-box method that consistently matches or surpasses behavioral cloning. p. 1 The training objective of LPS sums the base-policy loss (reformulated MeanFlow), the latent steering loss, and the critic loss. p. 3
Where the Gains Come From
The gains in performance stem from directly backpropagating action-space critic gradients through a differentiable one-step generative base policy to update the latent actor (avoiding the inaccurate latent critic). LPS consistently outperforms behavioral cloning and strong latent steering baselines across standard offline RL benchmarks, confirming its robust, tuning-free nature. p. 1 In real-world robotic tasks, LPS achieves the highest success rates and best average performance over behavioral cloning and prior latent-steering methods. p. 5
Key Architectural Components
The framework instantiates LPS using three key components:
-
A differentiable one-step base policy (Section IV-A), which is implemented via MeanFlow to enable efficient one-step deterministic generation. p. 3
-
A spherical latent geometry (Section IV-B), which constrains the latent actor's queries by synchronizing the support of the base policy and latent actor’s output to a hypersphere S d with radius √d.
-
A direct latent optimization objective (Section IV-C), which updates the latent actor πϕ directly using action-space critic gradients, bypassing the need for proxy latent critics. p. 3
Limitations and Future Directions
LPS is ultimately bounded by the quality and coverage of the base policy, as it cannot recover important modes if the base policy fails to capture them. The spherical constraint is intentionally conservative, potentially restricting extrapolation to behaviors far beyond the demonstration distribution. p. 7 Future work includes scaling LPS to large Vision-Language-Action (VLA) models and exploiting the temporal structure within action chunks by using structured latent representations. p. 7
Ablation Study Insights
Ablation studies show that replacing the standard normal prior with a spherical prior for LPS does not degrade baseline performance, suggesting the sphere latent retains sufficient representational capacity. Furthermore, training LPS with the original MeanFlow parameterization leads to unstable learning and poor downstream control, highlighting that the noise-to-action reformulation is crucial for strong performance. p. 7 LPSD, which incorporates explicit regularization, demonstrates superior performance regardless of the latent distribution compared to LPS. p. 1
Real-World Applicability
LPS provides a robust offline RL algorithm under a shared value-learning backbone and demonstrates superior performance over behavioral cloning and prior latent-steering methods on real robots. It effectively mitigates failures caused by human teleoperation artifacts, such as premature release or repetitive loops, by steering the latent policy toward high-value regions. p. 6 The framework also shows sample efficiency in online fine-tuning, surpassing DSRL within limited steps on the insert pen task. p. 6
Computational Efficiency
LPS is notably faster than DSRL because it benefits from one-step generation and direct backpropagation through the differentiable base policy, avoiding latent-critic distillation overhead. At inference time, LPS utilizes MeanFlow’s one-step generation, achieving speeds comparable to MF-BC while delivering significantly higher success rates. p. 6 The computational efficiency comparison shows LPS training latency of 5. p. 618 sec compared to DSRL's 13.31 sec.
Conclusion
In this work, Latent Policy Steering (LPS) enables direct latent policy improvement by backpropagating action-space critic gradients through a differentiable one-step base policy, together with a synchronized spherical latent geometry. Across simulation benchmarks and real-world robotic tasks, LPS consistently improves over behavioral cloning and strong latent steering baselines, providing a practical out-of-the-box approach with minimal tuning. p. 7 The framework is envisioned as a superior policy extraction mechanism capable of superseding the QCFQL paradigm. p. 4
Acknowledgments
This work was initiated during the first author Hokyun Im’s internship at Microsoft Research Asia. This research was supported by the National Research Foundation of Korea (NRF) grant (RS-2024-00333634), the Institute of Information & Communications Technology Planning & Evaluation (IITP) grants, and the Electronics and Telecommunications Research Institute (ETRI) grant. p. 8
References
[1] Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. OPAL: offline primitive discovery for accelerating offline reinforcement learning
[2] Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Jianhao Wang, Alex Yuan Gao, Wenzhe Li, Liang Bin, Chelsea Finn, and Chongjie Zhang. LAPO: Latent-variable advantageweighted policy optimization for offline reinforcement learning
[3] Yuhui Chen, Shuai Tian, Shugao Liu, Yingting Zhou, Haoran Li, and Dongbin Zhao. ConRFT: A reinforced fine-tuning method for VLA models via consistency policy
[4] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion
[5] Kevin Frans, Danijar Hafner, Sergey Levine, and Pieter Abbeel. One step diffusion via shortcut models
[6] Kevin Frans, Seohong Park, Pieter Abbeel, and Sergey Levine. Diffusion guidance is a controllable policy improvement operator
[7] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning
[8] Zhengyang Geng, Mingyang Deng, Xingjian Bai, J. Zico Kolter, and Kaiming He. p. 8 Mean flows for one-step generative modeling
[9] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance
[10] Physical Intelligence, Ali Amin, Raichelle Aniceto, Ashwin Balakrishna, Kevin Black, Ken Conley, Grace Connors, James Darpinian, Karan Dhabalia, Jared DiCarlo, Danny Driess, Michael Equi, Adnan Esmail. Mean Flow tames policy learning in 1-step for robotic manipulation
[11] Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany. DROID: A large-scale in-the-wild robot manipulation dataset
[12] Changyeon Kim, Haeone Lee, Younggyo Seo, Kimin Lee, and Yuke Zhu. Deas: Detached value learning with action sequence for scalable offline rl
[13] Diederik P. Kingma and Max Welling. p. 9 Auto-encoding variational bayes
[14] Qiyang Li, Zhiyuan Zhou, and Sergey Levine. Reinforcement learning with action chunking
[15] Tianhong Li and Kaiming He. Back to basics: Let denoising generative models denoise
[16] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow
[17] Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goalconditioned RL
[18] Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning
[19] William Peebles and Saining Xie. Scalable diffusion models with transformers
[20] Karl Pertsch, Youngwoon Lee, and Joseph J. Lim. Accelerating reinforcement learning with learned skill priors
[21] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models
[22] Juyi Sheng, Ziyi Wang, Peiming Li, and Mengyuan Liu. MP1: Mean flow tames policy learning in 1-step for robotic manipulation
[23] Roman Vershynin. High-dimensional probability: An introduction with applications in data science
[24] Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang, Seohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, and Sergey Levine. Steering your diffusion policy with latent space reinforcement learning
[25] Zeyuan Wang, Da Li, Yulin Chen, Ye Shi, Liang Bai, Tianyuan Yu, and Yanwei Fu. One-step generative policies with Q-learning: A reformulation of meanflow
[26] Zhendong Wang, Jonathan J. Hunt, and Mingyuan Zhou. p. 9 Diffusion policies as an expressive policy class for offline reinforcement learning
[27] Ling Yang, Zixiang Zhang, Zhilong Zhang, Xingchao Liu, Minkai Xu, Wentao Zhang, Chenlin Meng, Stefano Ermon, and Bin Cui. Consistency flow matching: Defining straight flows with velocity consistency
[28] Qinglun Zhang, Zhen Liu, Haoqiang Fan, Guanghui Liu, Bing Zeng, and Shuaicheng Liu. Flowpolicy: Enabling fast and robust 3d flow-based policy via consistency flow matching for robot manipulation
[29] Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. p. 9 Learning fine-grained bimanual manipulation with low-cost hardware
[30] Wenxuan Zhou, Sujay Bajracharya, and David Held. PLAS: latent action space for offline reinforcement learning
Table I
| Task | BC-FM | BC-MF | DSRL | LPS |
|---|---|---|---|---|
eggplant to bin 9/20 (45%) 5/20 (25%) 13/20 (65%) 16/20 (80%)
pnp carrots 14/20 (70%) 12/20 (60%) 13/20 (65%) 17/20 (85%)
plug in bulb 1/20 (5%) 3/20 (15%) 0/20 (0%) 7/20 (35%)
refill tape 1/20 (5%) 3/20 (15%) 2/20 (10%) 5/20 (25%)
| Average | 31.2% | 28.7% | 35.0% | 45% |
Table II
Task (normal / sphere) BC-FM BC-MF QC-FQL QC-MFQL CFGRL DSRL LPS cube-single-play-singletask cube-double-play-singletask scene puzzle 3x3 puzzle 4x4 visual task1
|---|---|---|---|
Success Rate (normal / sphere) 9 ± 1 / 9 ± 1; 10 ± 2 / -; -; -
Table III
Parameter Value Batch size (M) Discount factor (γ) Optimizer Adam Learning rate Target network update rate (τ) Critic ensemble size (K) UTD Ratio 1 Number of flow steps (T) 10, 1. Number of training steps 10 6
Table IV
Environments (normal / sphere) QC-FQL QC-MFQL cube-single cube-double scene-sparse puzzle-3x3 puzzle-4x4
|---|---|---|---|---|---|
Alpha (0.01 to 300) 30. p. 40 / 10.0 / 30.0 / 1.5/2
Table V
Environments CFG strength (w) cube-single cube-double scene-sparse puzzle-3x3 puzzle-4x4
Table VI
Parameter Value Batch size (M) 256, Discount factor (γ) 0.99, Optimizer Adam, Learning rate 3 × 10−4, Target network update rate (τ) 5 × 10−3, Critic ensemble size (K) 2, UTD Ratio 1
Table VII
Environments (normal / sphere) LPSD cube-single cube-double scene-sparse puzzle-3x3 puzzle-4x4
Figure 17
| LPSD vs baselines on both normal and sphere latent structure on 5 OGBench tasks | cube-single, cube-double, scene-sparse, puzzle-3x3, puzzle-4x4
Figure 8
Success rates on real-world tasks eggplant to bin (9/20 (45%), 5/20 (25%), 13/20 (65%), 16/20 (80%))
Figure 9
Qualitative failure modes and corrections Teleoperation artifacts can induce failures such as premature release (topleft), repetitive loops (top-right), and freezing during alignment (bottom)
Figure 10
| Online fine-tuning results | LPS demonstrates rapid adaptation, surpassing both its offline baseline and DSRL within limited steps
Figure 13
| Offline-to-online learning curves of LPS and baselines on OGBench tasks | LPS retains the performance level acquired during offline training without significant degradation
Figure 5
Performance on OGBench LPS consistently outperforms the one-step distillation baselines (QC-FQL and QC-MFQL)
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Direct latent policy improvement enables high fidelity policy updates by
backpropagating original-action-space Q-gradients through a differentiable one-step MeanFlow policy to update a latent-action-space actor.
This allows for robust, tuning free optimization byeliminat[ing] proxy latent critics
anddirect[ly] optimiz[ing] the latent actor using gradients from an action space critic.
-
Robustness to hyperparameter sensitivity is achieved because LPS avoids the explicit trade-off between return maximization and behavioral regularization by
separating reward maximization and distributional constraints.
This results in a method thatconsistently outperforms behavioral cloning and strong latent steering baselines
across OGBench and real-world tasks without requiringtask-specific tuning or additional online interaction.
-
Real world robotic manipulation performance is significantly improved because LPS mitigates failure modes like
premature release due to hesitation, repetitive motion loops (eggplant to bin), and freezing during precision alignment (plug in bulb).
It does this by steering the latent policy toward high-value regions, which allows the agentto execute decisive actions where BC baselines would otherwise stall or oscillate.
Sources
- Diffusion Guidance Is a Controllable Policy Improvement Operator
- Classifier-Free Diffusion Guidance
- $\pi^{*}_{0.6}$: a VLA That Learns From Experience
- Auto-Encoding Variational Bayes
- Back to Basics: Let Denoising Generative Models Denoise
- One-Step Generative Policies with Q-Learning: A Reformulation of MeanFlow
- Consistency Flow Matching: Defining Straight Flows with Velocity Consistency
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation