From Gradients to Capabilities: Understanding Multi-Teacher On-Policy Distillation
cs.LG
Submitted: 2026-10-01
Updated: 2026-10-01
Terminology
Sources
- On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
- Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agent
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- MiniLLM: On-Policy Distillation of Large Language Models
- Measuring Mathematical Problem Solving With the MATH Dataset
- Editing Models with Task Arithmetic
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- Kimi K3: Open Frontier Intelligence
- Conflict-Averse Gradient Descent for Multi-task Learning
- Understanding R1-Zero-Like Training: A Critical Perspective
- Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMs
- Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
- Generalizing Verifiable Instruction Following
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark
- On the Geometry of On-Policy Distillation
- MiMo-V2-Flash Technical Report
- Qwen3 Technical Report
- Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks