Speculative Self-Distillation enables Efficient Knowledge Internalization
cs.LG, cs.AI, cs.CL
Submitted: 2026-11-09
Updated: 2026-11-09
Code: https://github.com/huggingface/trl
Terminology
Sources
- SIEVE: Sample-Efficient Parametric Learning from Natural Language
- SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models
- Retrieval-Augmented Generation for Large Language Models: A Survey
- Distilling the Knowledge in a Neural Network
- SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMs
- Learning Facts at Scale with Active Reading
- AdaSwitch: Balancing Exploration and Guidance in Knowledge Distillation via Adaptive Switching
- Learning by Distilling Context
- A Survey of On-Policy Distillation for Large Language Models
- GATES: Self-Distillation under Privileged Context with Consensus Gating
- Self-Distilled RLVR
- On-Policy Context Distillation for Language Models
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis