How Much Training-Inference Mismatch Can On-Policy Distillation Tolerate?
cs.LG, cs.AI
Submitted: 2026-11-08
Updated: 2026-11-08
Code: https://github.com/benyucong/train-inference-mismatch-opd
Terminology
Sources
- -OPD: Stabilizing Long-Horizon On-Policy Distillation with Freshness-Aware Control
- Train Where the Quantized Model Goes: On-Policy Distillation for Low-Bit Reasoning
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
- AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning
- The Llama 3 Herd of Models
- JustRL: Scaling a 1.5B LLM with a Simple RL Recipe
- Skywork Open Reasoner 1 Technical Report
- AsyncOPD: How Stale Can On-Policy Distillation Be?
- Trust Region Masking for Long-Horizon LLM Reinforcement Learning
- Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model
- P1-VL: Bridging Visual Perception and Scientific Reasoning in Physics Olympiads
- Score Centering Stabilizes Off-policy Reinforcement Learning
- Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
- FP8-RL: A Practical and Stable Low-Precision Stack for LLM Reinforcement Learning
- Near-Policy: Accelerating On-Policy Distillation via Asynchronous Generation and Selective Packing
- Proximal Policy Optimization Algorithms
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation
- Jet-RL: Enabling On-Policy FP8 Reinforcement Learning with Unified Training and Rollout Precision Flow
- MiMo-V2-Flash Technical Report
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis