Stabilizing Off-policy LLM Optimization with Prefix Importance Ratio
cs.AI, cs.LG
Submitted: 2026-01-30
Updated: 2026-11-04
Terminology
Sources
- OpenAI o1 System Card
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Kimi k1.5: Scaling Reinforcement Learning with LLMs
- RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training
- Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models
- Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMs
- BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping
- Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?
- Proximal Policy Optimization Algorithms
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale
- Group Sequence Policy Optimization
- MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention
- Qwen3 Technical Report
- DCPO: Dynamic Clipping Policy Optimization
- ESPO: Entropy Importance Sampling Policy Optimization
- When Importance Sampling Misallocates Credit: Asymmetric Ratios for Outcome-Supervised RL
- The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
- Revisiting LLM Reasoning via Information Bottleneck
- Laminar: A Scalable Asynchronous RL Post-Training Framework
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents