Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR
cs.CL, cs.AI, cs.LG
Submitted: 2026-09-03
Updated: 2026-09-04
Code: https://github.com/StringNLPLAB/opd-rlvr
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reinforcement learning with verifiable rewards (RLVR) and on-policy distillation (OPD) have emerged as two dominant methods for post-training reasoning LLMs.
Terminology
Abstract
Reinforcement learning with verifiable rewards (RLVR) and on-policy distillation (OPD) have emerged as two dominant methods for post-training reasoning LLMs. Prior work uses OPD's dense token-level supervision to complement the sparse RL reward, fusing the two signals within a single step: either as a weighted-additive combination or a teacher-modulated rescaling of the RL advantage. In this paper, we show that a simple two-stage scheme, OPD-then-RL, consistently outperforms pure OPD, pure RLVR, and all such joint baselines across logic and math reasoning benchmarks. Beyond the empirical results, we further provide a systematic understanding of this through pass@ k behavior, learning dynamics, and parameter updates, yielding a consistent explanation: OPD expands the student's coverage of teacher-supported solutions and RL sharpens within that support, while jointly optimizing the two signals causes them to interfere. To provide a practical recipe, we find that the OPD validation score is the key signal for when to switch to RL, and that OPD is a better cold start for RL than SFT. Together, our results establish OPD-then-RL as a simple yet strong way to combine the two methods, turning two entangled signals into complementary stages.
Sources
- Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoning
- HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation
- SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning
- GLM-5: from Vibe Coding to Agentic Engineering
- DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning
- Distilling the Knowledge in a Neural Network
- DeReason: A Difficulty-Aware Curriculum Improves Decoupled SFT-then-RL Training for General Reasoning
- Entropy-Aware On-Policy Distillation of Language Models
- Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVR
- Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing
- GEAR: Granularity-Adaptive Advantage Reweighting for LLM Agents via Self-Distillation
- Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
- SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning
- ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
- Towards a Unified View of Large Language Model Post-Training
- Olmo 3
- Proximal Policy Optimization Algorithms
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning
- Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for Language-Model Post-Training
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering