Hindsight Compacts but Does Not Repair: Rethinking On-Policy Self-Distillation in Reasoning Models
cs.AI, cs.CL
Submitted: 2026-05-07
Updated: 2026-09-26
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: On-Policy Self-Distillation (OPSD) has emerged as a promising post-training method: requiring only privileged context, it enables token-level credit assignment from a self-teacher with hindsight,
Terminology
Abstract
On-Policy Self-Distillation (OPSD) has emerged as a promising post-training method: requiring only privileged context, it enables token-level credit assignment from a self-teacher with hindsight, yielding higher accuracy with shorter responses. However, in thinking-enabled mathematical reasoning, where solving a problem takes long reasoning traces that explore, reflect, and backtrack, its length reductions persist but its accuracy gains largely do not. Does OPSD's hindsight signal, then, still repair failed trajectories, or does it only compact viable ones? We hypothesize compaction: in long traces, hindsight reveals which steps were unnecessary more readily than which steps would have repaired the solution. To test this, we apply OPSD separately to correct and incorrect rollouts. Training only on correct rollouts shortens responses by 18-29% while largely preserving accuracy, whereas training only on incorrect rollouts degrades it. The accuracy gap holds across three models, six benchmarks, and three seeds, and under a same-prompt control for difficulty. Neither branch raises the pass@k ceiling, and both suppress exploration and reflection markers, which viable traces can spare but failed ones need. Changing the divergence, enriching or reinjecting the privileged context, and training longer only move OPSD along the same accuracy-length tradeoff. Hindsight compacts reasoning the model can already produce but does not repair reasoning it cannot.
Sources
- Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
- Reinforcement Learning via Self-Distillation
- On-Policy Context Distillation for Language Models
- CRISP: Compressed Reasoning via Iterative Self-Policy Distillation
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Solving math word problems with process- and outcome-based feedback
- Let's Verify Step by Step
- AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning
- SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
- Distilling the Knowledge in a Neural Network
- A General Language Assistant as a Laboratory for Alignment
- Self-Distillation Enables Continual Learning
- Privileged Information Distillation for Language Models
- GATES: Self-Distillation under Privileged Context with Consensus Gating
- OpenClaw-RL: Train Any Agent Simply by Talking
- HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation
- Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems