Rethinking On-Policy Self-Distillation for Thinking Models
cs.AI, cs.LG
Submitted: 2026-07-06
Updated: 2026-11-06
Comments: Accepted to NeurIPS 2026 (40th Conference on Neural Information Processing Systems)
Code: https://github.com/idanshen/Self-Distillation
License: http://creativecommons.org/licenses/by/4.0/
The gist: On-policy self-distillation (OPSD) is a promising approach to recursive self-improvement.
Terminology
Abstract
On-policy self-distillation (OPSD) is a promising approach to recursive self-improvement. In this setting, a model acts as its own teacher when augmented with privileged information, such as a solution to a math problem. Thinking models seem especially well positioned to teach themselves this way, since they can deliberate, verify, and self-correct. Surprisingly, we show that OPSD degrades thinking models on long reasoning traces: across five Qwen3 and OLMo thinking models evaluated on AIME24, AIME25, and HMMT25, privileged-context distillation causes a relative drop of up to 17% in avg@16 accuracy. The degradation scales with the amount of privileged context withheld from the student and is most pronounced at long rollout budgets, where thinking models otherwise obtain their largest gains. This failure mode extends to distillation from a larger, privileged teacher: on-policy distillation (OPD) improves thinking models, but privileged OPD reverses these gains. Our diagnostics suggest that privileged teacher context reshapes learning at high-entropy forking positions, i.e., rollout positions where multiple continuations remain plausible and may lead to different reasoning paths. Privileged context lowers fork rates in thinking-model rollouts but not in instruction-tuned model rollouts, which may explain the observation that OPSD training helps instruct models more reliably than thinking models.At self-correction branches, privileged OPD penalizes sampled reconsideration tokens that vanilla OPD supports. Consistent with this, thinking models trained with a privileged teacher produce fewer verification, backtracking, and hedging markers, even after length normalization. These findings indicate that applying self-distillation methods to strong thinking models requires further consideration of token-level signal---especially around tokens related to correction and crucial reasoning steps.
Sources
- Forking Paths in Neural Text Generation
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation
- Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision
- Distilling the Knowledge in a Neural Network
- Reinforcement Learning via Self-Distillation
- Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?
- Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capability
- POPE: Learning to Reason on Hard Problems via Privileged On-Policy Exploration
- Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixes
- Self-Distillation Enables Continual Learning
- Olmo 3
- Qwen3 Technical Report
- Self-Distilled RLVR
- Embarrassingly Simple Self-Distillation Improves Code Generation
- Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents