Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss

arXiv:2608.03796 · cs.CL, cs.AI, cs.LG · Submitted 2026-08-04 · Read on arXiv

Bakbergen Ryskulov, Iker García-Ferrero, David Montero, David Jansen, Ali Hashemi, Jezabel R. Garcia, Antonio Tiene, Román Orús

cs.CL, cs.AI, cs.LG

Submitted: 2026-08-04

Comments: Patent Application Pending. EP26382987.1

Code: https://github.com/CompactifAI/Full-Chunked-KL-Loss

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Distillation — 36× in this paper · explained in 2 episodes
• Ablation — 12× in this paper · explained in 1 episode
• Transformer — 10× in this paper · explained in 2 episodes
• Benchmark — 9× in this paper · explained in 2 episodes
• Knowledge Distillation — 8× in this paper · explained in 13 episodes
• Accuracy — 4× in this paper · explained in 1 episode
• Association for Computational Linguistics — 4× in this paper
• Transformers — 3× in this paper · explained in 2 episodes
• Pruning — 2× in this paper · explained in 4 episodes
• Ablation Studies — 1× in this paper · explained in 1 episode
• Adam — 1× in this paper · explained in 2 episodes
• Adaptation — 1× in this paper · explained in 1 episode
• Core — 1× in this paper · explained in 2 episodes
• In International Conference on Learning Representations — 1× in this paper
• Verifier — 1× in this paper · explained in 2 episodes
• Weight Decay — 1× in this paper · explained in 2 episodes

Sources

• SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
• Training Verifiers to Solve Math Word Problems
• Gemma: Open Models Based on Gemini Research and Technology
• Distilling the Knowledge in a Neural Network
• Liger Kernel: Efficient Triton Kernels for LLM Training
• The Llama 3 Herd of Models
• DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering