Bernoulli On-Policy Distillation: Don't Ignore High-Confidence Student Tokens
cs.LG
Submitted: 2026-11-01
Updated: 2026-11-01
Code: https://github.com/duterscmy/Bernoulli-OPD
Terminology
Sources
- On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
- MathArena: Evaluating LLMs on Uncontaminated Math Competitions
- Step Guided Reasoning: Improving Mathematical Reasoning using Guidance Generation and Step Reasoning
- Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement
- MiniLLM: On-Policy Distillation of Large Language Models
- Distilling the Knowledge in a Neural Network
- Entropy-Aware On-Policy Distillation of Language Models
- Let's Verify Step by Step
- Unbiased Top- k Estimation for On-Policy Distillation
- A Survey of On-Policy Distillation for Large Language Models
- Qwen3 Technical Report
- Bridging Cross-task Protocol Inconsistency for Distillation in Dense Object Detection
- PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformation
- Decoupled Knowledge Distillation
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis