Generative Online Reinforcement Learning

arXiv:2512.02581 · cs.LG · Submitted 2025-12-02 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Generative Online Reinforcement Learning".

Jane: Here's the gist of the paper. Reinforcement learning (RL) faces a persistent tension: policies that are stable to optimize (e.g., Gaussians) are often too simple to represent the multimodal action distributions required for complex control Training Diffusion Policies via Prior-Mapping Co-Evolution, which addresses this tension with a key structural principle: decoupling optimization from generation The Core Problem Addressed Reinforcement learning (RL) has driven remarkable progress in robotics (Kober et al., 2013; Levine et al., 2016), games (Mnih et al., 2015; Silver et al., 2016), and continuous control (Haarnoja et al., 2018; Lillicrap et al., 2015) by optimizing policies through continual interaction with the environment However, unimodal parameterizations often struggle to represent the complex, multimodal action patterns required in challenging environments This exposes a persistent tension between stable optimization and rich action modeling.

Tom: The GoRL Framework GoRL (Generative Online Reinforcement Learning) is an algorithm-agnostic framework that trains expressive policies from scratch by confining policy optimization to a tractable latent space while delegating action synthesis to a conditional generative decoder It views each stage as prior-mapping co-evolution, where the latent policy improves the latent prior and the decoder refines the corresponding prior-to-action mapping The framework decomposes the policy into a latent policy pitheta(epsilon s) optimized with standard RL algorithms and a conditional decoder gphi (s, epsilon) that maps latent variables to actions.

Jane: The Training Schedule Optimization in GoRL follows a two-timescale update schedule: alternating between improving the latent policy via standard policy gradients with the decoder fixed, and refining the decoder via supervised generative training with the latent policy frozen The framework partitions the interaction budget into four stages of 60M, 60M, 30M, and 30M steps.

Tom: Crucially, Phase 2 refines the decoder by anchoring refinement to a fixed prior by drawing fresh samples from N (0, I) as decoder inputs.

Title and authors: Tom: The paper introduces GoRL, which stands for Generative Online Reinforcement Learning. It’s an algorithm agnostic framework that trains expressive policies from scratch by confining policy optimization to a tractable latent space while delegating action synthesis to a conditional generative decoder.

Jane: So if we think about it for the listener, it means they take the standard reinforcement learning process, but instead of just optimizing a single set of parameters for the policy, they are optimizing two things simultaneously: an inner policy that refines this latent representation and an outer decoder that maps those latent ideas to actual actions.

Lu: The authors build on this by framing it as prior-mapping co-evolution. Think about it like this: the latent policy improves the underlying idea or prior, and the decoder refines how we map those ideas to specific actions.

Meng: It’s interesting because it lets you use whatever RL algorithm you want for that inner latent policy—maybe PPO or SAC—and any generative architecture you prefer for the decoder, so it’s very flexible.

Tom: And they show a training schedule that alternates between improving the latent policy with a fixed decoder and then refining the decoder while freezing the latent policy. This two-timescale update schedule is key to continuous improvement in expressiveness.

Jane: So, as we go through these stages, the inner policy keeps getting better at representing the underlying structure, and then we take those improved structures and use them to make our action mapping more precise.

Lu: And they introduce a specific technique during the decoder refinement stage that is designed to break a problem called self-reconstruction loops. This involves fixing the decoder’s inputs to a Gaussian prior, which forces it to consolidate the exploration progress from the latent policy into a stronger generator.

The paper's summary: Tom: What they really summarize is how this co-evolution allows stability and expressiveness to grow in tandem. The latent policy evolves the latent prior, and the decoder refines that prior-to-action mapping, which means both parts get better at their respective jobs simultaneously.

Jane: So for someone just listening, it’s like a feedback loop where optimizing the policy helps create a better map for generating actions, and refining that map helps optimize the policy further. It avoids having one part bottleneck the other's potential.

Lu: The paper analyzes why expressive generative policies can get stuck in a near self-reconstruction step during decoder refinement if they are not carefully managed. The authors show that this happens when you let the decoder train too much on just what it just produced, which yields little gain in expressiveness.

Meng: But their solution is to use that fixed-prior anchor we talked about—fixing the inputs to a Gaussian prior—to force the decoder to consolidate the latent policy’s exploration progress into a stronger generator instead of just repeating itself.

Tom: This entire framework, called GoRL or Generative Online Reinforcement Learning, addresses that initial tension head-on by making this decoupling structural rather than just an optimization trick.

Jane: It shifts the focus from finding one perfect policy to managing the interaction between a stable representation and a flexible action generator over time.

The paper's improvements: Tom: The theoretical guarantees are pretty solid. They show that latent updates induce unbiased gradients for the composite policy, and bounded divergence in the latent space yields a trust-region-style lower bound on the induced policy’s return change. This means we have a mathematical basis for how much better the performance can get.

Jane: That's a lot of math, but what it means practically is that as long as the encoder policies stay close in that latent space, the performance degradation is bounded by some factor delta. It gives us confidence in how stable this learning process is supposed to be.

Lu: Lemma three point two specifically guarantees that performance degradation stays bounded by O delta as long as successive encoder policies remain close in latent space, which really supports the idea that stability and expressiveness can grow together under these conditions <ref:2512.02581#pg1>.

Meng: From a deployment standpoint, the cost is something we have to look at. The paper notes that this process incurs additional wall-clock time because of the periodic decoder refinement steps, about one point eight seven times longer than standard policy gradient methods like PPO <ref:2512.02581#pg1>.

Tom: But they counter that this extra computation translates into a performance gain of about one point six three times the return, which suggests it’s a very favorable trade-off between the computational cost and what you get in terms of actual learning quality.

Jane: So, even though it takes more time on your computer, the resulting policy is significantly more capable at handling complex tasks compared to simpler methods.

Conclusion: Tom: To wrap up, the paper on Generative Online Reinforcement Learning shows that by decoupling optimization from generation, we can get expressive action modeling without sacrificing optimization stability. It’s about letting the learning process keep moving forward while making the action distribution richer over time.

Jane: So what this means for us is that for complex continuous control problems, we don't have to choose between a simple, stable policy or a highly expressive one; this framework lets you get both by making them interact intelligently through prior-mapping co-evolution.

Lu: From the perspective of representation learning, the paper confirms that we can leverage generative models like diffusion or flow matching to parameterize policies as conditional generative models mapping noise to actions, which is a big step in moving away from fixed parameterizations.

Meng: For deployment, it’s interesting because they show this works across different learning paradigms—it’s algorithm agnostic—meaning you can plug in your preferred RL optimizer and your favorite generator architecture without having to rebuild the whole thing.

Lalam: I'm confident in this direction. The GoRL framework is structured to handle the tension between stability and expressiveness by iteratively enhancing one another, which is a solid mechanism for continuous learning.

Chubin Zhang, Zhenglin Wan, Feng Chen, Fuchao Yang, Lang Feng, Yaxin Zhou, Xingrui Yu, Ivor Tsang, Bo An

Nanyang Technological University Singapore National University of Singapore Carnegie Mellon University CFAR Agency for Science, Technology and Research Singapore IHPC Agency for Science, Technology and Research Singapore

cs.LG

Submitted: 2025-12-02

Updated: 2026-11-10

Comments: ICML 2026; 22 pages

Journal ref: Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:158816-158837, 2026

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 98/100

The gist: Reinforcement learning (RL) faces a persistent tension: policies that are stable to optimize (e.g., Gaussians) are often too simple to represent the multimodal action distributions required for…

Key concepts

GoRL Framework
A framework that trains policies by separating the optimization process (using a tractable latent space) from action synthesis (using a generative decoder). It treats learning as a co-evolution where the latent policy improves the representation and the decoder refines how that representation maps to actions.
Latent Policy ($\pi_{\theta}(\epsilon | s)$)
The part of the policy optimized using standard reinforcement learning algorithms. It learns a compact, stable representation of state-action relationships within a low-dimensional latent space, which is easier for RL methods to handle compared to complex action distributions.
Conditional Generative Decoder ($g_{\phi}(s, \epsilon)$)
A generative model responsible for taking the state and the learned latent variable as input to produce an actual action. This decoder is trained via supervised learning and is designed to be expressive enough to generate the diverse, multimodal actions needed for complex control.
Prior-Mapping Co-evolution
The core training principle where two components evolve together: the latent policy refines a 'latent prior' (the representation), and the decoder simultaneously learns how to map that refined prior into more accurate action distributions. This iterative process ensures both stability and expressiveness.

Terminology

Summary

Reinforcement learning (RL) faces a persistent tension: policies that are stable to optimize (e. p. 0g., Gaussians) are often too simple to represent the multimodal action distributions required for complex control Training Diffusion Policies via Prior-Mapping Co-Evolution, which addresses this tension with a key structural principle: decoupling optimization from generation

The Core Problem Addressed

Reinforcement learning (RL) has driven remarkable progress in robotics (Kober et al., 2013; Levine et al. p. 0, 2016), games (Mnih et al. p. 0, 2015; Silver et al. p. 0, 2016), and continuous control (Haarnoja et al. p. 0, 2018; Lillicrap et al. p. 0, 2015) by optimizing policies through continual interaction with the environment However, unimodal parameterizations often struggle to represent the complex, multimodal action patterns required in challenging environments This exposes a persistent tension between stable optimization and rich action modeling. p. 0

The GoRL Framework

GoRL (Generative Online Reinforcement Learning) is an algorithm-agnostic framework that trains expressive policies from scratch by confining policy optimization to a tractable latent space while delegating action synthesis to a conditional generative decoder It views each stage as prior-mapping co-evolution, where the latent policy improves the latent prior and the decoder refines the corresponding prior-to-action mapping The framework decomposes the policy into a latent policy pitheta(epsilon s) optimized with standard RL algorithms and a conditional decoder gphi (s, epsilon) that maps latent variables to actions.

The Training Schedule

Optimization in GoRL follows a two-timescale update schedule: alternating between improving the latent policy via standard policy gradients with the decoder fixed, and refining the decoder via supervised generative training with the latent policy frozen The framework partitions the interaction budget into four stages of 60M, 60M, 30M, and 30M steps. Crucially, Phase 2 refines the decoder by anchoring refinement to a fixed prior by drawing fresh samples from N (0, I) as decoder inputs. p. 5

Theoretical Guarantees

The latent–generative factorization admits a rigorous theoretical foundation, establishing that latent updates induce unbiased gradients for the composite policy and that bounded latent divergence yields a trust-region-style lower bound on the induced policy’s return change Lemma 3.1 shows that the gradient of the expected return satisfies ∇theta log pitheta(epsilon s) × Apitheta,phi (s, gphi (s, epsilon)) Lemma 3. p. 62 guarantees that performance degradation is bounded by O(δ) as long as successive encoder policies remain close in latent space. p. 6

Empirical Performance

Empirically, GoRL consistently outperforms unimodal and generative baselines across diverse continuous-control tasks Notably, GoRL achieves returns exceeding 870 on HopperStand, more than 3× the strongest baseline On high-dimensional humanoid tasks, it further outperforms the strongest non-GoRL baseline by over an order of magnitude. GoRL-FM and GoRL-Diff achieve higher final returns than the baselines and exhibit more stable learning dynamics. p. 8

Computational Trade-offs

GoRL incurs additional wall-clock cost due to periodic decoder refinement, resulting in approximately 1.87× wall-clock time relative to PPO However, GoRL effectively converts this computational investment into a substantial performance gain of approximately 1.63× return. p. 6 This demonstrates that GoRL offers the most favorable trade-off between computational cost and policy expressiveness. p. 3

Algorithm Agnosticism

GoRL serves as a flexible framework rather than a single rigid algorithm, relying on two modular components: (i) a tractable encoder pitheta(epsilon s) and (ii) a conditional generative decoder gphi (s, epsilon) with a likelihood-free objective It is compatible with any on- or off-policy encoder for the latent policy and any generative architecture for the decoder. Results in Appendix F. p. 82 show that the same decoupled design can also be instantiated with an off-policy optimizer, SAC. p. 9

Conclusion

GoRL reconciles optimization stability with expressive action modeling by decoupling optimization from generation, allowing learning to remain stable while the action distribution becomes more flexible over time Across diverse continuous-control tasks, this design improves stability, final performance, and mode coverage relative to unimodal and coupled generative baselines.

Limitations and Future Work

GoRL introduces additional wall-clock cost due to periodic decoder refinement A natural next step is to trigger refinement adaptively based on training signals such as evaluation returns, latent entropy, or value-loss plateaus The humanoid results suggest scaling beyond moderate-action benchmarks, but higher-degree-of-freedom systems and visual observations remain open.

Acknowledgements

This research/project is supported by the National Research Foundation, Singapore under its National Large Language Models Funding Initiative (AISG Award No: AISG-NMLP-2024-003) This research is supported by the Ministry of Education, Singapore, under its MOE AcRF Tier 2 Award MOE-T2EP20223-0003.

Impact Statement

This paper proposes GoRL, a framework for training expressive generative policies from scratch in fully online reinforcement learning by decoupling optimization from generation A potential positive impact is improved sample efficiency and robustness for complex continuous-control problems, which could benefit applications in robotics and automation where multimodal action distributions arise naturally.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Decoupling optimization from generation allows for stable policy updates by confining RL gradients to a tractable latent space while using a conditional generative decoder for expressive action synthesis, "GoRL (Generative Online Reinforcement Learning), an algorithm-agnostic framework that trains expressive policies from scratch by confining policy optimization to a tractable latent space while delegating action synthesis to a conditional generative decoder."

  2. The two-timescale schedule enables continuous improvement in expressiveness, as the encoder improves the latent prior and the decoder refines the prior-to-action mapping, each stage first improves a tractable latent prior through RL and then consolidates the resulting behavior into a more expressive prior-to-action mapping.

  3. The fixed-prior anchoring during decoder refinement breaks self-reconstruction loops by forcing the decoder to consolidate exploration progress, as described by To break this feedback loop, we fix the decoder’s refinement inputs to a Gaussian prior and train it on improved actions generated by the optimized latent policy.

  4. The framework enables multimodal action representation because it avoids the expressivity bottleneck of unimodal Gaussian policies by using generative models like diffusion or flow matching as decoders, which can represent rich multimodal action distributions.

  5. The system can achieve high performance in complex continuous control tasks, demonstrated by achieving returns exceeding 870 on HopperStand, more than 3x the strongest baseline and outperforming non-GoRL baselines by over an order of magnitude on two humanoid tasks.

  6. The framework is algorithm-agnostic, meaning it can be instantiated with different RL algorithms for the latent policy (like PPO or SAC) and various generative architectures for the decoder (such as Conditional Flow Matching or Diffusion models), allowing flexibility across different learning paradigms.

Sources

Related papers