Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals
cs.LG, cs.CL
Submitted: 2026-10-08
Updated: 2026-10-08
Comments: 50 pages. Code: https://github.com/leizhao7/opd-learning-signals
Code: https://github.com/leizhao7/opd-learning-signals
License: http://creativecommons.org/licenses/by/4.0/
The gist: On-policy distillation (OPD) enables effective capability transfer between language models, yet the mechanisms underlying its failures are not fully understood.
Terminology
Abstract
On-policy distillation (OPD) enables effective capability transfer between language models, yet the mechanisms underlying its failures are not fully understood. Across code generation and mathematical reasoning, OPD with larger-scale teachers exhibits early loss plateaus, with an average final loss reduction of 25.1% after 200 updates, compared with 96.2% for self-RL teachers, obtained by further reinforcement learning (RL) training of the initial student. To understand this difference, we analyze OPD as an idealized continuous-time dynamical system in the small-learning-rate limit. Our training-log diagnostics associate these plateaus with an early decline in a gradient-based learning-signal proxy while substantial loss remains; these measurements do not establish why the underlying gradient weakens. We further prove a local recovery guarantee for teachers sufficiently close to the initial student in a shared parameterization under regularity conditions, offering a conditional explanation for the success of self-RL teachers in our experiments. Across runs with and without loss plateaus, we observe small relative parameter changes (0.025-0.098%) and high similarity between the student's representations before and after OPD (linear CKA >0.98 across layers). These observations suggest that limited representation adaptation may contribute to learning-signal collapse, a hypothesis that remains to be tested. Code is available at https://github.com/leizhao7/opd-learning-signals.
Sources
- Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why
- Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation
- Process Reinforcement through Implicit Rewards
- Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes
- Rethinking On-Policy Distillation of Large Language Models II: One Training Example
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- JustRL: Scaling a 1.5B LLM with a Simple RL Recipe
- Skywork Open Reasoner 1 Technical Report
- Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following
- Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level
- Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation
- Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
- Reinforcement Learning Finetunes Small Subnetworks in Large Language Models
- PRISM: Demystifying Retention and Interaction in Mid-Training
- On the Geometry of On-Policy Distillation
- Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations
- Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation
- MiMo-V2-Flash Technical Report
- Qwen3 Technical Report
- Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks