Distilled Reinforcement Learning for LLM Post-training

arXiv:2607.17247 · cs.LG, cs.AI · Submitted 2026-07-19 · Read on arXiv

cs.LG, cs.AI

Submitted: 2026-07-19

Updated: 2026-09-29

Code: https://github.com/hiyouga/EasyR1

Terminology

• Distillation — 3× in this paper · explained in 2 episodes
• Reinforcement Learning — 3× in this paper · explained in 4 episodes
• Adaptation — 1× in this paper · explained in 1 episode
• Alignment — 1× in this paper · explained in 1 episode
• Credit assignment — 1× in this paper · explained in 3 episodes
• Large Language Model (LLM) — 1× in this paper · explained in 3 episodes
• On-Policy Distillation — 1× in this paper · explained in 3 episodes
• Reinforcement Learning (RL) — 1× in this paper · explained in 25 episodes

Sources

• OpenCodeReasoning-II: A Simple Test Time Scaling Approach via Self-Critique
• OpenCodeReasoning: Advancing Data Distillation for Competitive Coding
• MathArena: Evaluating LLMs on Uncontaminated Math Competitions
• Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting
• Reasoning with Exploration: An Entropy Perspective
• Training Verifiers to Solve Math Word Problems
• Process Reinforcement through Implicit Rewards
• The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
• SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines
• Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes
• ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools
• DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• Distilling the Knowledge in a Neural Network
• Entropy-Aware On-Policy Distillation of Language Models
• Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
• TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling
• Let's Verify Step by Step
• DeepSeek-V3 Technical Report
• Length-Unbiased Sequence Policy Optimization: Revealing and Controlling Response Length Variation in RLVR

Related papers

• Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
• AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
• Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
• Convergence issues in Relational Concept Analysis based on AOC-posets
• Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
• Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks