Formalizing Forgetting: Theory of Forgetting for Online Continual Learning
The authors analyzed single-pass Shuffle SGD under strongly convex and general convex objectives to find explicit upper bounds on expected forgetting in online continual learning. They showed that data amount, task similarity, and task ordering govern forgetting differently depending on the objective class. Empirical tests confirmed these theoretical trends in practice.