Optimal L2 Regularization in High-dimensional Continual Linear Regression
This study analyzes L2 regularization in continual linear regression across multiple tasks with label noise and varying teachers. It derives a closed-form expression for generalization loss, revealing that the optimal regularization strength scales nearly linearly with the number of tasks, specifically around T/ln(T). This provides a theoretical basis for designing robust continual learning systems.