Learning What to Distill: Bilevel Top-K Token Selection for Self-Distillation in Large Language Models
cs.LG
Submitted: 2026-10-05
Updated: 2026-10-05
Terminology
Sources
- OpenThoughts: Data Recipes for Reasoning Models
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Distilling the Knowledge in a Neural Network
- SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMs
- Reinforcement Learning via Self-Distillation
- Understanding Reasoning in LLMs through Strategic Information Allocation under Uncertainty
- Scaling Reasoning Efficiently via Relaxed On-Policy Distillation
- NeuralNDCG: Direct Optimisation of a Ranking Metric via Differentiable Relaxation of Sorting
- Large-scale Stochastic Optimization of NDCG Surrogates for Deep Learning with Provable Convergence
- CRISP: Compressed Reasoning via Iterative Self-Policy Distillation
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Self-Distillation Enables Continual Learning
- Rethinking Selective Knowledge Distillation
- A Survey on Knowledge Distillation of Large Language Models
- TIP: Token Importance in On-Policy Distillation
- Algorithmic Foundations of Empirical X-risk Minimization
- Black-Box On-Policy Distillation of Large Language Models
- On-Policy Context Distillation for Language Models
- Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks