The Sequential Price of Continual Learning
cs.LG, cs.AI
Submitted: 2026-08-30
Updated: 2026-09-30
Comments: 26 pages, 3 figures
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Sequential task updates are fundamental to continual learning, but their recency bias can impose a lasting performance cost.
Terminology
Abstract
Sequential task updates are fundamental to continual learning, but their recency bias can impose a lasting performance cost. We study this cost in an overparameterized linear-regression model with i.i.d. task sampling. We prove that distribution-level forgetting and population loss converge to the same stationary limit. We quantify the additional loss incurred by sequential exact fitting, or the sequential price. In more homogeneous task geometries, it equals the intrinsic loss asymptotically attained by joint training, making the total loss twice as large. We further analyze fixed-strength elastic weight consolidation (EWC) under general task curvatures and characterize its stationary sequential price at every regularization strength. Under strong regularization, the price decays inversely with EWC strength while the mean-square coupling horizon grows proportionally. Experiments on Jester and Rotated MNIST support the predicted sequential price and its reduction by EWC, with quantitative agreement on real-world tasks satisfying the theory's assumptions and qualitative agreement under nonlinear finite-step training.
Sources
- Supervised Learning with Evolving Tasks and Performance Guarantees
- Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System
- Theoretical Insights into Overparameterized Models in Multi-Task and Replay-Based Continual Learning
- Dark Experience for General Continual Learning: a Strong, Simple Baseline
- Last Iterate Convergence of Incremental Methods and Applications in Continual Learning
- Context Channel Capacity: An Information-Theoretic Framework for Understanding Catastrophic Forgetting
- The Method of Alternating Projections
- Analysis of Overparameterization in Continual Learning under a Linear Model
- The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Model
- Provable Continual Learning via Sketched Jacobian Approximations
- Convergence and Implicit Bias of Gradient Descent on Continual Linear Classification
- Compact Memory for Continual Logistic Regression
- Optimal L2 Regularization in High-dimensional Continual Linear Regression
- Optimal Rates in Continual Linear Regression via Increasing Regularization
- IF2Net: Innately Forgetting-Free Networks for Continual Learning
- Optimal Task Order for Continual Learning of Multiple Tasks
- Replay Can Provably Increase Forgetting
- LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learning
- Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adaptation
- Are Greedy Task Orderings Better Than Random in Continual Linear Regression?
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks