Unpacking the Compute Efficiency of Strong-to-Weak OPD against GRPO
cs.LG, cs.CL
Submitted: 2026-11-05
Updated: 2026-11-05
Terminology
Sources
- On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
- -OPD: Stabilizing Long-Horizon On-Policy Distillation with Freshness-Aware Control
- Keep Policy Gradient in Charge: Sibling-Guided Credit Distillation for Long-Horizon Tool-Use Agents
- Weak-to-Strong Generalization via Direct On-Policy Distillation
- Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes
- Rethinking On-Policy Distillation of Large Language Models II: One Training Example
- MiniLLM: On-Policy Distillation of Large Language Models
- OpenThoughts: Data Recipes for Reasoning Models
- DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning
- Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?
- Sequence-Level Knowledge Distillation
- Kimi K3: Open Frontier Intelligence
- Scaling Reasoning Efficiently via Relaxed On-Policy Distillation
- Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients
- Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
- Teacher-Guided Policy Optimization for On-Policy Reasoning Distillation under Large Policy Divergence
- Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models
- On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversity
- Near-Policy: Accelerating On-Policy Distillation via Asynchronous Generation and Selective Packing
- Trace-Based On-Policy Distillation for Masked Diffusion Language Models
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis