Off-Policy Merging Beats On-Policy Self-Distillation for Continual Learning

arXiv:2610.05872 · cs.LG, cs.AI, cs.CL · Submitted 2026-10-05 · Read on arXiv

cs.LG, cs.AI, cs.CL

Submitted: 2026-10-05

Updated: 2026-10-05

Code: https://github.com/ar-forum/grafting

Project page: https://ar-forum.github.io/grafting-web/Abstract

Terminology

• Continual Learning — 1× in this paper · explained in 2 episodes
• Distillation — 1× in this paper · explained in 2 episodes
• Self-Distillation — 1× in this paper · explained in 7 episodes

Sources

• Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting
• SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
• Weight Ensembling Improves Reasoning in Language Models
• DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
• Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervision
• Reinforcement Learning via Self-Distillation
• Towards Continual Knowledge Learning of Language Models
• Rethinking On-Policy Self-Distillation for Thinking Models
• Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?
• Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models
• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs
• Privileged Information Distillation for Language Models
• (How) Learning Rates Regulate Catastrophic Overtraining
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
• Purified OPSD: On-Policy Self-Distillation Without Losing How to Think
• Self-Distillation Enables Continual Learning
• Overtrained Language Models Are Harder to Fine-Tune
• Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting
• Shadow-FT: Tuning Instruct Model via Training on Paired Base Model
• Qwen3 Technical Report

Related papers

• Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
• AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
• Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
• Convergence issues in Relational Concept Analysis based on AOC-posets
• Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
• Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks