Forget About It: Mitigating Forgetting Without Replay or Auxiliary Objectives

arXiv:2501.05559 · cs.LG, cs.AI · Submitted 2025-01-09 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Forget About It".

Jane: The gist The Sequential Fine-tuning Averaging (SFA) method proposes merging currently training models with earlier checkpoints during training to mitigate catastrophic forgetting without requiring data buffers or auxiliary objectives.

Tom: First, who's behind it and why it matters.

Paper summary: Jane: So, looking at the paper "Forget About It: Mitigating Forgetting Without Replay or Auxiliary Objectives," what is the main message they want us to take away from this work?

Tom: The main point is that simple in-training parameter interpolation, specifically SFA, deserves consideration before we introduce more elaborate forgetting-mitigation mechanisms <ref:2501.05559#pg1>.

Lu: It suggests that methods like sequential fine-tuning with averaging can achieve comparable results to more complex continual learning algorithms without needing the overhead of storing past data or running extra generators <ref:2501.05559#pg2>.

Meng: For production, this means we might be able to deploy models using this simpler merging strategy instead of investing heavily in building out large data buffers for every task variation <ref:2501.05559#pg3>.

Lalam: I’m pretty sure that the findings suggest SFA outperforms other merging techniques like Task Arithmetic when you optimize the averaging frequency p <ref:2501.05559#pg1>.

Jane: It seems they are arguing for a more straightforward way to handle forgetting in sequential fine-tuning scenarios, moving away from storing huge datasets or adding complex auxiliary training steps <ref:2501.05559#pg2>.

Tom: The authors are showing that their method can be comparable to rehearsal-based continual learning but without needing the data buffer <ref:2501.05559#pg3>.

Lu: Overall, the paper points towards the idea that parameter merging during training is a viable path forward for mitigating catastrophic forgetting in AI systems <ref:2501.05559#pg2>.

Conclusion: Tom: So we’re wrapping up this deep dive on "Forget About It: Mitigating Forgetting Without Replay or Auxiliary Objectives." Basically, the whole paper is about figuring out if you can stop catastrophic forgetting without needing those huge data buffers or extra training objectives.

Jane: Right, so the authors are looking at how far you can go just by interpolating parameters during fine-tuning. It’s asking if simple parameter averaging can actually work on its own.

Lu: The core idea they’re pushing is that sequential fine-tuning with averaging, or SFA for short, is a way to retain old knowledge without needing explicit rehearsal data <ref:2501.05559#pg1>.

Meng: I'm curious if it’s practical. Can we actually run this at scale? Does this averaging process just slow down the training or introduce instability when we push these models through production traffic?

Lalam: My analysis of the results shows that in Qwen3-14B adapter learning, fixing both SFA and replay-free lookahead configurations actually outperformed their replay and adaptive merging setups <ref:2501.05559#pg1>.

Tom: So, the paper finds that fixed configurations for both methods beat the ones that used data buffers or adaptive merging in terms of final accuracy, using up to thirty-six point three percent less training time <ref:2501.05559#pg1>. That’s a solid number for efficiency.

Jane: And they say this holds up across both language and vision tasks, meaning this approach isn't just a fluke for one specific type of model or task <ref:2501.05559#pg2>. So, what does that mean if we don't need to store all the old data?

Lu: It suggests that simple interpolation is actually competitive with more complicated continual learning methods when you look at the performance metrics <ref:2501.05559#pg3>. They even found SFA beats WiSE-FT and Task Arithmetic when you tune the averaging parameter p correctly <ref:2501.05559#pg3>.

Meng: If it’s that simple, then maybe we don't need to build entire data pipelines for every new domain shift. That would make deployment much cleaner, even if the training curve looks a bit different.

Lalam: I’m fairly confident in this direction. The intuition they provide links SFA back to L2-regression, which gives it a solid mathematical grounding <ref:2501.05559#pg4>. It's not just a guess; it has a structure that mirrors established optimization techniques <ref:2501.05559#pg4>.

Tom: So the authors are essentially saying that before we build all these fancy, heavy-duty forgetting-mitigation systems, we should seriously test if this basic in-training interpolation idea is enough <ref:2501.05559#pg1>. It’s about testing the simplest possible solution first.

Jane: Exactly. It moves the conversation from "how do we add complexity to stop forgetting?" to "can we achieve good results with less complexity?" That opens up a lot of ground for future research on how these simple merging procedures behave as models get bigger and tasks get more diverse.

Anat Kleiman, Gintare Karolina Dziugaite, Jonathan Frankle, Sham Kakade, Mansheej Paul

Harvard University · Google DeepMind · Databricks

cs.LG, cs.AI

Submitted: 2025-01-09

Updated: 2026-10-14

Code: https://github.com/mosaicml/composer

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 91/100

The gist: The gist The Sequential Fine-tuning Averaging (SFA) method proposes merging currently training models with earlier checkpoints during training to mitigate catastrophic forgetting without requiring

Key concepts

Sequential Fine-tuning with Averaging (SFA): SFA is a method where, every few training steps or at the end of fine-tuning, the model's parameters are reset to a weighted average of its current state and a checkpoint from an earlier task. This allows the model to retain knowledge from previous tasks while learning new ones without needing large data buffers.
Approximation to L2-Regression: SFA mathematically approximates L2-regression, which is a technique used in continual learning. The update rule shows that the model parameters are adjusted by a combination of the loss on the new task and a penalty term that pulls the current parameters back toward those of previous tasks.
Catastrophic Forgetting: This is when a neural network, while learning new information for a current task, rapidly forgets or degrades its performance on previously learned tasks. The paper explores methods like SFA to prevent this degradation by ensuring old knowledge is retained during new learning processes.

Terminology

Fine-Tuning — 2× in this paper · explained in 5 episodes
Catastrophic Forgetting — 1× in this paper · explained in 5 episodes
Continual Learning — 1× in this paper · explained in 2 episodes

Summary

The gist The Sequential Fine-tuning Averaging (SFA) method proposes merging currently training models with earlier checkpoints during training to mitigate catastrophic forgetting without requiring data buffers or auxiliary objectives.

Sequential Fine-tuning Averaging (SFA)

SFA is a novel method that merges the model currently training on a new task with a checkpoint from a previous task during training, rather than exclusively at the end, which allows the model to retain knowledge on past tasks while learning a new task More precisely, let θo denote the parameters of the network optimized for previous tasks. Let θ∗t+1 be the parameters of the model after taking a gradient step on a new task at 1 ≤ t ≤ T using current model parameters θt Then, every pT iterations, as well as at the end of fine-tuning, we reset the parameters to be a weighted combination of θo and θ∗t+1, where the weighing is determined by a hyperparameter 0 ≤ β ≤ 1 (default: 0.5)

Approximation to L2-Regression

The method roughly approximates L2-regression in Section 6 Consider, starting with θo, the model trained on the previous task and θt, the model currently being trained on the new task. Calculating the loss with an L2 penalty takes the form L(θt) = Ltask(θt) + λ2 θt − θo2 Updating the model once using the gradient of this loss results in θt+1 = θt − η(∇θtLtask + λ(θt − θo)) This can be rewritten as θt+1 = (1 − ηλ)θt + (ηλ)θo − η∇θtLtask(1 − β)

Performance Across Continual Learning Scenarios

The experiments systematically investigate forgetting across two extensive settings: (1) the classical continual learning scenario, characterized by a large sequence of image classification tasks, and (2) finetuning pretrained large language models (LLMs) on highly distinct domains, including Law, Math, and Code SFA outperforms most other methods on both Food-101 and CIFAR-100 SFA with varying p generally outperforms both using a smaller-sized data buffer and Task Arithmetic Finally, SFA (p = 0.98 for Food-101, and p = 0.96 for CIFAR-100) where averaging occurs near the end and after training, achieves higher performance than WiSE-FT or SFA (p = 1) where averaging occurs only once at end

Robustness to Cross-Domain Shifts

In testing robustness, SFA is evaluated by focusing on mitigating pairs of successive instruction fine-tuning tasks with large domain shifts, such as from Math to Code or Math to Law In the Math-then-Law fine-tuning setting, SFA with p = 0.25 can be seen as having comparable performance to using a data buffer with 5% past task data, while outperforming Task Arithmetic Crucially, our method is able achieve such performance without requiring a data buffer, but just two model checkpoints

Intuition for Model Merging

The work provides intuition for why model merging is effective by showing how SFA roughly approximates a classical continual learning algorithm: L2-regression We also offer an empirical example that shows the correlation between L2 distance of the initial and training model in the context of our experimental setup As such, SFA may have Bayesian motivation because of its similarity to L2-regression Furthermore, SFA outperforms both L2-regression and EWC when hyperparameters are optimized

Comparison with Other Techniques

SFA is comparable to continual learning with rehearsal, but has the advantage of not maintaining a data buffer SFA outperforms both L2-regression and EWC when hyperparameters are optimized Furthermore, it is comparable to continual learning with rehearsal, but has the advantage of not maintaining a data buffer

Conclusion

The final performance of SFA tends to outperform other merging, as well as penalty methods Furthermore, it is comparable to continual learning with rehearsal, but has the advantage of not maintaining a data buffer The final performance of SFA tends to outperform other merging, as well as penalty methods The final performance of SFA tends to outperform other merging, as well as penalty methods

Ethics and Reproducibility

This paper presents work whose goal is to advance the field of Machine Learning The tools we use in this project are all open-source A description of our models and how we finetune/evaluate can be found in Appendix A.5 Descriptions of the tasks we fine-tune models on are in Appendix A.6 and Section 4

Acknowledgments

This work has been made possible in part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence Thank you to Ben Deaner and Vincent Dumoulin for reading the draft and offering suggestions

References

Bayesian interpretation of ridge regression. 2018. URL https://statisticaloddsandends.wordpress.com/2018/12/29/. Lawinformedai/claudette tos, 2023. URL https://huggingface.co/datasets/LawInformedAI/claudette tos. Eden Belouadah, Adrian Popescu, and Ioannis Kanellos. A comprehensive study of class incremental learning algorithms for visual tasks. Neural Networks, 135:38–54, 2021. ISSN 0893-6080. doi: https://doi.org/10.1016/j.neunet.2020.12.003 Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023. Raad Bin Tareaf. Tweets Dataset - Top 20 most followed users in Twitter social platform, 2017. URL https://doi.org/10.7910/DVN/JBXKFD. Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 – mining discriminative components with random forests. In European Conference on Computer Vision, 2014. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger. Michael Petrov. Heidy Khlaaf. Girish Sastry. Pamela Mishkin. Brooke Chan. Scott Gray. Nick Ryder. Mikhail Pavlov. Alethea Power. Lukasz Kaiser. Mohammad Bavarian. Clemens Winter; Philippe Tillet; Felipe Petroski Such; Dave Cummings; Matthias Plappert; Fotios Chantzis; Elizabeth Barnes; Ariel Herbert-Voss; William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders. Christopher Hesse. Andrew N. Carr. Jan Leike; Josh Achiam; Vedant Misra; Evan Morikawa; Alec Radford; Matthew Knight; Miles Brundage; Mira Murati; Katie Mayer; Peter Welinder; Bob McGrew. Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser; Matthias Plappert; Jerry Tworek; Jacob Hilton; Reiichiro Nakano; Christopher Hesse; John Schulman. Training verifiers to solve math word problems, 2021. Cyprien de Masson d’Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. Episodic memory in lifelong language learning, 2019.

Improvements for AI systems

Based on Sequential Fine-tuning Averaging (SFA), an improved AI system can mitigate catastrophic forgetting by merging currently training models with earlier checkpoints during the course of training. This allows a model to retain knowledge on past tasks while learning a new task without needing to store past data or multiple copies of parameters at every gradient step.
The system can achieve superior cross-domain performance by dynamically adjusting the merging frequency, as demonstrated by SFA with varying averaging frequency 'p'. This is because as p decreases, signifying more frequent averaging with the initial model, we observe stronger retention of past domain knowledge.
The system can be optimized to approximate Bayesian inference for weight constraints. By relating SFA to L2-regression, the system can effectively constrain weights by minimizing a loss function that includes a penalty term similar to L2-regression or Elastic Weight Consolidation (EWC).