Replay for LLM Continual Learning with Multi-Teacher On-Policy Distillation
cs.LG, cs.CL
Submitted: 2026-11-02
Updated: 2026-11-03
Terminology
Sources
- Distilling the Knowledge in a Neural Network
- MiMo-V2-Flash Technical Report
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
- MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training
- GLM-5: from Vibe Coding to Agentic Engineering
- ACLArena: Agent Continue Learning in Multi-stage Post-training
- OPRD: On-Policy Representation Distillation
- Latent-MOPD: Latent Multi-Teacher On-Policy Distillation
- Qwen3 Technical Report
- D cubed-MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation
- Nash-MOPD: Optimizing Task Trade-offs in Multi-Teacher On-Policy Distillation
- UP-MOPD: Update Projection in Multi-Teacher On-Policy Distillation
- PMOPD: Task Ordering, Cycling, and Parameter-Update Subspace Protection in Multi-Teacher On-Policy Distillation
- Slow-Fast Multi-Teacher On-Policy Distillation for Capability Preservation
- Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillation
- Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillation
- Training-Free On-Policy Distillation
- Composing What Each Teacher Learned: Delta Distillation for Multi-Teacher On-Policy Learning
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis