Policy as Data: Replay-Based Policy Dual Averaging via Advantage Regression
cs.LG, cs.AI, math.OC
Submitted: 2026-10-03
Updated: 2026-10-03
Code: https://github.com/InsultedByMaths/rda2c
Project page: https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details
Sources
- Maximum a Posteriori Policy Optimisation
- On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift
- Accelerating RL for LLM Reasoning with Optimal Advantage Regression
- Soft Actor-Critic for Discrete Action Settings
- Actor-Accelerated Policy Dual Averaging for Reinforcement Learning in Continuous Action Spaces
- REBEL: Reinforcement Learning via Regressing Relative Rewards
- Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Mirror Descent Actor Critic via Bounded Advantage Learning
- Policy Optimization over General State and Action Spaces
- Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classes
- AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
- A unified view of entropy-regularized Markov decision processes
- Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
- Reinforcement and Imitation Learning via Interactive No-Regret Learning
- Prioritized Experience Replay
- Proximal Policy Optimization Algorithms
- Mirror Descent Policy Optimization
- Sample Efficient Actor-Critic with Experience Replay
- EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Engine
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks