Disentangling Self-Distillation: Measuring and Modeling Acquisition and Retention

arXiv:2609.39494 · cs.AI, cs.LG · Submitted 2026-09-30 · Read on arXiv

cs.AI, cs.LG

Submitted: 2026-09-30

Updated: 2026-09-30

Code: https://github.com/EIT-NLP/Decoupled-Distill

Terminology

• Distillation — 1× in this paper · explained in 2 episodes
• Self-Distillation — 1× in this paper · explained in 7 episodes

Sources

• On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
• A Quantitative Characterization of Forgetting in Post-Training
• Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting
• SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models
• When Should the Teacher Move? Temporal Coupling and Stability in Self On-Policy Distillation
• Reinforcement Learning via Self-Distillation
• Entropy-Aware On-Policy Distillation of Language Models
• Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?
• MixSD: Mixed Contextual Self-Distillation for Knowledge Injection
• When Privileged Guidance Misaligns: State-Matched Routing and Contextualized Self-Distillation for Multi-Turn Agents
• An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
• Updating Parametric Knowledge with Context Distillation Retains Post-Training Capabilities
• Qwen2.5 Technical Report
• Analysing Mathematical Reasoning Abilities of Neural Models
• RL's Razor: Why Online Reinforcement Learning Forgets Less
• Self-Distillation Enables Continual Learning
• Learning by Distilling Context
• A Survey of On-Policy Distillation for Large Language Models
• GATES: Self-Distillation under Privileged Context with Consensus Gating
• ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• Agents' Last Exam
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems