Forget About It: Mitigating Forgetting Without Replay or Auxiliary Objectives
The paper investigates if simple periodic parameter interpolation during fine-tuning can mitigate catastrophic forgetting without needing data buffers or extra objectives. They introduce Sequential Fine-tuning with Averaging (SFA), which merges current training parameters with older checkpoints periodically. Results show SFA outperforms replay and adaptive merging methods in Qwen3-14B, suggesting simple interpolation is a viable approach before complex mitigation strategies.