Latent Policy Steering through One-Step Flow Policies

arXiv:2603.05296 · cs.RO, cs.LG · Submitted 2026-03-05 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "Latent Policy Steering through One-Step Flow Policies".

Dev: Here's the gist of the paper. Offline reinforcement learning (RL) allows robots to learn from offline datasets without risky exploration, yet offline RL’s performance often hinges on a brittle trade-off between return maximization and behavioral constraints.

Rosa: Latent Policy Steering (LPS) proposes a method that enables high-fidelity latent policy improvement by backpropagating original-action-space Q-gradients through a differentiable one-step MeanFlow policy to update a latent-action-space actor.

Dev: The Core Problem Addressed The paper identifies two practical bottlenecks for real-world offline RL: the sensitivity induced by explicit behavior regularization and the approximation error induced by indirect latent distillation.

Rosa: Existing methods often rely on a delicate trade-off controlled by a hyperparameter α, which is highly sensitive to reward scale and dataset diversity, making extensive tuning prohibitive in real-world deployment.

Title and authors: Rosa: So we're looking at the paper titled Latent Policy Steering through One-Step Flow Policies. It sounds a bit technical, but basically, it’s about how robots learn from old data without having to try things out in the real world again.

Dev: Yeah, it’s about this method that tries to fix a big problem in offline reinforcement learning where you usually have to pick a tricky balance between just getting high rewards and making sure the robot doesn't do something totally weird that was never seen before.

Taro: It sounds like they are tackling the issue of safety constraints versus performance maximization in these offline datasets.

Rosa: Exactly, that’s what it is. The authors introduce this Latent Policy Steering method as a structural way to keep the policy inside the range of what the old data actually shows while still trying to get good results from those actions.

Dev: And they propose using something called MeanFlow, which is a differentiable one-step generative model, to backpropagate those original action space Q-gradients all the way up into a latent action space actor.

Taro: So it’s taking information directly from the reward calculation and steering the latent policy based on that instead of some indirect estimation.

Rosa: That’s right. The paper is focused on replacing those indirect methods—like using latent critics to estimate values—with this direct gradient flow, which they call Latent Policy Steering through One-Step Flow Policies.

The paper's summary: Dev: So the core idea, based on the abstract and page one of Latent Policy Steering through One-Step Flow Policies, is that they are trying to avoid two specific bottlenecks in offline RL.

Rosa: Right, they identified sensitivity from explicit behavior regularization and approximation error from using latent distillation. They think existing methods often rely on a hyperparameter alpha that you have to tune a lot based on how big the reward scale is or how diverse the dataset is.

Dev: And they point out that distilled latent critics can give poor gradients because just matching values doesn't guarantee the latent gradients used for improvement are accurate near sharp boundaries in the data manifold.

Taro: It sounds like they’re saying that these existing methods aren't robust enough because of how those value estimates are created or how they enforce behavior constraints.

Rosa: That’s the problem they want to solve. Their solution, Latent Policy Steering through One-Step Flow Policies, decouples behavioral constraints from reward maximization by splitting them up.

Dev: So instead of trying to find a single weight for regularization, they have a fixed generative behavior policy that defines what actions are allowed—the support—and then a latent actor does the value-driven steering based on that.

Taro: That separation sounds smart. It means the agent can focus purely on improving the policy without having to constantly worry about tuning this explicit behavioral regularization weight.

Rosa: Precisely, and they claim this decoupling results in an out-of-the-box method that consistently matches or even surpasses behavioral cloning across standard offline RL benchmarks.

The paper's improvements: Dev: Where the real gain comes from, they say it’s by directly backpropagating action space critic gradients through a differentiable one-step MeanFlow policy to update the latent actor.

Rosa: That means they are bypassing the need for those proxy latent critics entirely. Instead of using an estimated value, they are using the original action space critic to guide the end-to-end optimization in the latent space.

Taro: So it’s not about learning a good approximation of a value function first, but using what we already have—the action space critic—to improve the policy structure directly.

Dev: And they show that this direct optimization leads to consistently better performance than behavioral cloning and strong latent steering baselines on standard offline RL benchmarks.

Rosa: They claim this method is robust because it doesn't need task-specific tuning for that explicit behavioral regularization weight, which is a huge deal for real-world deployment.

Taro: But what about the limitations they mention? They say the gains come from this direct backpropagation, but they also have to be careful about the quality of the base policy that generates those gradients.

Dev: They also introduce a spherical latent geometry constraint to keep things stable, synchronizing the support of the base policy and actor output onto a hypersphere with radius sqrt(d).

Rosa: That spherical constraint is intentional, they say it’s conservative. It might restrict extrapolation to behaviors far beyond what the original demonstrations showed.

Conclusion: Dev: So to wrap up Latent Policy Steering through One-Step Flow Policies, the paper shows a framework that combines direct value-based improvement with structural safety constraints from a generative prior.

Rosa: The main implication is that you can get high-fidelity latent policy improvement without having to tune parameters for behavioral regularization, which makes it much more practical.

Taro: I just want to push on the real-world aspect for a second. They say this method mitigates failures like premature release or repetitive motion loops in real robot tasks by steering the latent policy toward high-value regions.

Dev: And they showed that this adaptation happens really fast when fine-tuning online, actually surpassing DSRL within limited steps on tasks like the insert pen task.

Rosa: So, in short, Latent Policy Steering through One-Step Flow Policies gives us a practical, tuning-free approach that outperforms prior latent steering methods across simulation and real robots.

Taro: I just think the spherical constraint is conservative because if it doesn't capture all the necessary modes of behavior, then the whole system might miss something important.

Dev: True, but they also showed that replacing a standard normal prior with a spherical prior didn't actually degrade baseline performance on those benchmarks.

Rosa: So, Latent Policy Steering through One-Step Flow Policies is presented as a superior policy extraction mechanism that avoids the complexities of QCFQL paradigms.

Hokyun Im, Andrey Kolobov, Jianlong Fu, Youngwoon Lee

Department of Artificial Intelligence, Yonsei University · Microsoft Research

cs.RO, cs.LG

Submitted: 2026-03-05

Updated: 2026-10-29

Comments: RSS 2026, Project Webpage : https://jellyho.github.io/LPS/

Project page: https://jellyho.github.io/LPS/Abstract

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 96/100

The gist: Offline reinforcement learning (RL) allows robots to learn from offline datasets without risky exploration…

Key concepts

Latent Policy Steering (LPS)
LPS is a method that steers a latent actor to improve its policy by directly using gradients from the action space critic. It achieves this by passing the critic's gradients through a differentiable one-step generative policy, allowing for high-fidelity updates without needing complex or brittle proxy latent critics.
MeanFlow Policy
MeanFlow is used as a differentiable one-step generative base policy in LPS. This component allows the system to efficiently generate actions in a single step, which is crucial for backpropagating the critic's gradients directly to update the latent actor's parameters effectively.
Spherical Latent Geometry
This constraint limits the queries of the latent actor by forcing its outputs onto a hypersphere. This synchronization ensures that the latent actor's output and the base policy's support are aligned, helping to maintain stable learning and performance across different tasks.

Terminology

Summary

Offline reinforcement learning (RL) allows robots to learn from offline datasets without risky exploration, yet offline RL’s performance often hinges on a brittle trade-off between return maximization and behavioral constraints. p. 0 Latent Policy Steering (LPS) proposes a method that enables high-fidelity latent policy improvement by backpropagating original-action-space Q-gradients through a differentiable one-step MeanFlow policy to update a latent-action-space actor. p. 0

The Core Problem Addressed

The paper identifies two practical bottlenecks for real-world offline RL: the sensitivity induced by explicit behavior regularization and the approximation error induced by indirect latent distillation. Existing methods often rely on a delicate trade-off controlled by a hyperparameter α, which is highly sensitive to reward scale and dataset diversity, making extensive tuning prohibitive in real-world deployment. p. 0 Furthermore, distilled latent critics can provide poor gradients because matching values does not guarantee that the latent gradients used for improvement are accurate near sharp boundaries of the data manifold. p. 2

How the Training Signal is Built

LPS decouples behavioral constraints from reward maximization by separating them into a fixed generative behavior policy defining the support and a latent actor performing value-driven steering. This decoupling allows the agent to focus on policy improvement without tuning an explicit behavioral regularization weight, resulting in an out-of-the-box method that consistently matches or surpasses behavioral cloning. p. 1 The training objective of LPS sums the base-policy loss (reformulated MeanFlow), the latent steering loss, and the critic loss. p. 3

Where the Gains Come From

The gains in performance stem from directly backpropagating action-space critic gradients through a differentiable one-step generative base policy to update the latent actor (avoiding the inaccurate latent critic). LPS consistently outperforms behavioral cloning and strong latent steering baselines across standard offline RL benchmarks, confirming its robust, tuning-free nature. p. 1 In real-world robotic tasks, LPS achieves the highest success rates and best average performance over behavioral cloning and prior latent-steering methods. p. 5

Key Architectural Components

The framework instantiates LPS using three key components:

  1. A differentiable one-step base policy (Section IV-A), which is implemented via MeanFlow to enable efficient one-step deterministic generation. p. 3

  2. A spherical latent geometry (Section IV-B), which constrains the latent actor's queries by synchronizing the support of the base policy and latent actor’s output to a hypersphere S d with radius √d.

  3. A direct latent optimization objective (Section IV-C), which updates the latent actor πϕ directly using action-space critic gradients, bypassing the need for proxy latent critics. p. 3

Limitations and Future Directions

LPS is ultimately bounded by the quality and coverage of the base policy, as it cannot recover important modes if the base policy fails to capture them. The spherical constraint is intentionally conservative, potentially restricting extrapolation to behaviors far beyond the demonstration distribution. p. 7 Future work includes scaling LPS to large Vision-Language-Action (VLA) models and exploiting the temporal structure within action chunks by using structured latent representations. p. 7

Ablation Study Insights

Ablation studies show that replacing the standard normal prior with a spherical prior for LPS does not degrade baseline performance, suggesting the sphere latent retains sufficient representational capacity. Furthermore, training LPS with the original MeanFlow parameterization leads to unstable learning and poor downstream control, highlighting that the noise-to-action reformulation is crucial for strong performance. p. 7 LPSD, which incorporates explicit regularization, demonstrates superior performance regardless of the latent distribution compared to LPS. p. 1

Real-World Applicability

LPS provides a robust offline RL algorithm under a shared value-learning backbone and demonstrates superior performance over behavioral cloning and prior latent-steering methods on real robots. It effectively mitigates failures caused by human teleoperation artifacts, such as premature release or repetitive loops, by steering the latent policy toward high-value regions. p. 6 The framework also shows sample efficiency in online fine-tuning, surpassing DSRL within limited steps on the insert pen task. p. 6

Computational Efficiency

LPS is notably faster than DSRL because it benefits from one-step generation and direct backpropagation through the differentiable base policy, avoiding latent-critic distillation overhead. At inference time, LPS utilizes MeanFlow’s one-step generation, achieving speeds comparable to MF-BC while delivering significantly higher success rates. p. 6 The computational efficiency comparison shows LPS training latency of 5. p. 618 sec compared to DSRL's 13.31 sec.

Conclusion

In this work, Latent Policy Steering (LPS) enables direct latent policy improvement by backpropagating action-space critic gradients through a differentiable one-step base policy, together with a synchronized spherical latent geometry. Across simulation benchmarks and real-world robotic tasks, LPS consistently improves over behavioral cloning and strong latent steering baselines, providing a practical out-of-the-box approach with minimal tuning. p. 7 The framework is envisioned as a superior policy extraction mechanism capable of superseding the QCFQL paradigm. p. 4

Acknowledgments

This work was initiated during the first author Hokyun Im’s internship at Microsoft Research Asia. This research was supported by the National Research Foundation of Korea (NRF) grant (RS-2024-00333634), the Institute of Information & Communications Technology Planning & Evaluation (IITP) grants, and the Electronics and Telecommunications Research Institute (ETRI) grant. p. 8

References

[1] Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. OPAL: offline primitive discovery for accelerating offline reinforcement learning

[2] Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Jianhao Wang, Alex Yuan Gao, Wenzhe Li, Liang Bin, Chelsea Finn, and Chongjie Zhang. LAPO: Latent-variable advantageweighted policy optimization for offline reinforcement learning

[3] Yuhui Chen, Shuai Tian, Shugao Liu, Yingting Zhou, Haoran Li, and Dongbin Zhao. ConRFT: A reinforced fine-tuning method for VLA models via consistency policy

[4] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion

[5] Kevin Frans, Danijar Hafner, Sergey Levine, and Pieter Abbeel. One step diffusion via shortcut models

[6] Kevin Frans, Seohong Park, Pieter Abbeel, and Sergey Levine. Diffusion guidance is a controllable policy improvement operator

[7] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning

[8] Zhengyang Geng, Mingyang Deng, Xingjian Bai, J. Zico Kolter, and Kaiming He. p. 8 Mean flows for one-step generative modeling

[9] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance

[10] Physical Intelligence, Ali Amin, Raichelle Aniceto, Ashwin Balakrishna, Kevin Black, Ken Conley, Grace Connors, James Darpinian, Karan Dhabalia, Jared DiCarlo, Danny Driess, Michael Equi, Adnan Esmail. Mean Flow tames policy learning in 1-step for robotic manipulation

[11] Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany. DROID: A large-scale in-the-wild robot manipulation dataset

[12] Changyeon Kim, Haeone Lee, Younggyo Seo, Kimin Lee, and Yuke Zhu. Deas: Detached value learning with action sequence for scalable offline rl

[13] Diederik P. Kingma and Max Welling. p. 9 Auto-encoding variational bayes

[14] Qiyang Li, Zhiyuan Zhou, and Sergey Levine. Reinforcement learning with action chunking

[15] Tianhong Li and Kaiming He. Back to basics: Let denoising generative models denoise

[16] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow

[17] Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goalconditioned RL

[18] Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning

[19] William Peebles and Saining Xie. Scalable diffusion models with transformers

[20] Karl Pertsch, Youngwoon Lee, and Joseph J. Lim. Accelerating reinforcement learning with learned skill priors

[21] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models

[22] Juyi Sheng, Ziyi Wang, Peiming Li, and Mengyuan Liu. MP1: Mean flow tames policy learning in 1-step for robotic manipulation

[23] Roman Vershynin. High-dimensional probability: An introduction with applications in data science

[24] Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang, Seohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, and Sergey Levine. Steering your diffusion policy with latent space reinforcement learning

[25] Zeyuan Wang, Da Li, Yulin Chen, Ye Shi, Liang Bai, Tianyuan Yu, and Yanwei Fu. One-step generative policies with Q-learning: A reformulation of meanflow

[26] Zhendong Wang, Jonathan J. Hunt, and Mingyuan Zhou. p. 9 Diffusion policies as an expressive policy class for offline reinforcement learning

[27] Ling Yang, Zixiang Zhang, Zhilong Zhang, Xingchao Liu, Minkai Xu, Wentao Zhang, Chenlin Meng, Stefano Ermon, and Bin Cui. Consistency flow matching: Defining straight flows with velocity consistency

[28] Qinglun Zhang, Zhen Liu, Haoqiang Fan, Guanghui Liu, Bing Zeng, and Shuaicheng Liu. Flowpolicy: Enabling fast and robust 3d flow-based policy via consistency flow matching for robot manipulation

[29] Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. p. 9 Learning fine-grained bimanual manipulation with low-cost hardware

[30] Wenxuan Zhou, Sujay Bajracharya, and David Held. PLAS: latent action space for offline reinforcement learning

Table I

| Task | BC-FM | BC-MF | DSRL | LPS |

|---|---|---|---|---|

eggplant to bin 9/20 (45%) 5/20 (25%) 13/20 (65%) 16/20 (80%)

pnp carrots 14/20 (70%) 12/20 (60%) 13/20 (65%) 17/20 (85%)

plug in bulb 1/20 (5%) 3/20 (15%) 0/20 (0%) 7/20 (35%)

refill tape 1/20 (5%) 3/20 (15%) 2/20 (10%) 5/20 (25%)

| Average | 31.2% | 28.7% | 35.0% | 45% |

Table II

Task (normal / sphere) BC-FM BC-MF QC-FQL QC-MFQL CFGRL DSRL LPS cube-single-play-singletask cube-double-play-singletask scene puzzle 3x3 puzzle 4x4 visual task1

|---|---|---|---|

Success Rate (normal / sphere) 9 ± 1 / 9 ± 1; 10 ± 2 / -; -; -

Table III

Parameter Value Batch size (M) Discount factor (γ) Optimizer Adam Learning rate Target network update rate (τ) Critic ensemble size (K) UTD Ratio 1 Number of flow steps (T) 10, 1. Number of training steps 10 6

Table IV

Environments (normal / sphere) QC-FQL QC-MFQL cube-single cube-double scene-sparse puzzle-3x3 puzzle-4x4

|---|---|---|---|---|---|

Alpha (0.01 to 300) 30. p. 40 / 10.0 / 30.0 / 1.5/2

Table V

Environments CFG strength (w) cube-single cube-double scene-sparse puzzle-3x3 puzzle-4x4

Table VI

Parameter Value Batch size (M) 256, Discount factor (γ) 0.99, Optimizer Adam, Learning rate 3 × 10−4, Target network update rate (τ) 5 × 10−3, Critic ensemble size (K) 2, UTD Ratio 1

Table VII

Environments (normal / sphere) LPSD cube-single cube-double scene-sparse puzzle-3x3 puzzle-4x4

Figure 17

| LPSD vs baselines on both normal and sphere latent structure on 5 OGBench tasks | cube-single, cube-double, scene-sparse, puzzle-3x3, puzzle-4x4

Figure 8

Success rates on real-world tasks eggplant to bin (9/20 (45%), 5/20 (25%), 13/20 (65%), 16/20 (80%))

Figure 9

Qualitative failure modes and corrections Teleoperation artifacts can induce failures such as premature release (topleft), repetitive loops (top-right), and freezing during alignment (bottom)

Figure 10

| Online fine-tuning results | LPS demonstrates rapid adaptation, surpassing both its offline baseline and DSRL within limited steps

Figure 13

| Offline-to-online learning curves of LPS and baselines on OGBench tasks | LPS retains the performance level acquired during offline training without significant degradation

Figure 5

Performance on OGBench LPS consistently outperforms the one-step distillation baselines (QC-FQL and QC-MFQL)

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Direct latent policy improvement enables high fidelity policy updates by backpropagating original-action-space Q-gradients through a differentiable one-step MeanFlow policy to update a latent-action-space actor. This allows for robust, tuning free optimization by eliminat[ing] proxy latent critics and direct[ly] optimiz[ing] the latent actor using gradients from an action space critic.

  2. Robustness to hyperparameter sensitivity is achieved because LPS avoids the explicit trade-off between return maximization and behavioral regularization by separating reward maximization and distributional constraints. This results in a method that consistently outperforms behavioral cloning and strong latent steering baselines across OGBench and real-world tasks without requiring task-specific tuning or additional online interaction.

  3. Real world robotic manipulation performance is significantly improved because LPS mitigates failure modes like premature release due to hesitation, repetitive motion loops (eggplant to bin), and freezing during precision alignment (plug in bulb). It does this by steering the latent policy toward high-value regions, which allows the agent to execute decisive actions where BC baselines would otherwise stall or oscillate.

Sources

Related papers