UniSD: Towards a Unified Self-Distillation Framework for Large Language Models
cs.CL, cs.AI, cs.LG
Submitted: 2026-05-07
Updated: 2026-11-05
Comments: Website: https://unifiedsd.github.io/ Code: https://github.com/Ahren09/UniSD
Code: https://github.com/Ahren09/UniSD
Project page: https://unifiedsd.github.io
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: Self-distillation (SD) offers a promising path for adapting large language models (LLMs) without relying on stronger external teachers.
Terminology
Abstract
Self-distillation (SD) offers a promising path for adapting large language models (LLMs) without relying on stronger external teachers. However, SD in autoregressive LLMs remains challenging because self-generated trajectories are free-form, correctness is task-dependent, and plausible rationales can still provide unstable or unreliable supervision. Existing methods mainly examine isolated design choices, leaving their effectiveness, roles, and interactions unclear. In this paper, we propose UniSD, a unified framework to systematically study self-distillation. UniSD integrates complementary mechanisms that address supervision reliability, representation alignment, and training stability, including multi-teacher agreement, EMA teacher stabilization, token-level contrastive learning, feature matching, and divergence clipping. Across six benchmarks and six models from three model families, UniSD reveals when self-distillation improves over static imitation, which components drive the gains, and how these components interact across tasks. Guided by these insights, we construct UniSDfull, an integrated pipeline that combines complementary components and achieves the strongest overall performance, improving over the base model by +5.4 points and the strongest baseline by +2.8 points. Extensive evaluation highlights self-distillation as a practical and steerable approach for efficient LLM adaptation without stronger external teachers.
Sources
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Qwen3 Technical Report
- Distilling the Knowledge in a Neural Network
- AgentArk: Distilling Multi-Agent Intelligence into a Single LLM Agent
- Self-Distillation Enables Continual Learning
- A Survey on Knowledge Distillation of Large Language Models
- Embarrassingly Simple Self-Distillation Improves Code Generation
- Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
- Program Synthesis with Large Language Models
- Evaluating Large Language Models Trained on Code
- ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases
- The Llama 3 Herd of Models
- Gemma 3 Technical Report
- MASCOT: Multi-Agent Socio-Collaborative Companion Systems
- A Survey of On-Policy Distillation for Large Language Models
- Privileged Information Distillation for Language Models
- Entropy-Aware On-Policy Distillation of Language Models
- VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillation
- SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting
- Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data