On the Dynamics of Delayed Generalization in Linear Models

arXiv:2602.08302 · cs.LG, cs.AI · Submitted 2026-02-09 · Read on arXiv

cs.LG, cs.AI

Submitted: 2026-02-09

Updated: 2026-10-14

Terminology

• Generalization — 4× in this paper · explained in 13 episodes
• Accuracy — 1× in this paper · explained in 1 episode
• Representation Learning — 1× in this paper · explained in 2 episodes

Sources

• Unifying Grokking and Double Descent
• To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasets
• Deep Grokking: Would Deep Neural Networks Generalize Better?
• Grokking modular arithmetic
• Deep Networks Always Grok and Here is Why
• Grokking as the Transition from Lazy to Rich Training Dynamics
• Grokking in Linear Estimators -- A Solvable Model that Groks without Understanding
• Towards Understanding Grokking: An Effective Theory of Representation Learning
• Omnigrok: Grokking Beyond Algorithmic Data
• Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking
• The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning
• Grokking vs. Learning: Same Features, Different Encodings
• Why Do You Grok? A Theoretical Analysis of Grokking Modular Addition
• Deep Double Descent: Where Bigger Models and More Data Hurt
• Progress measures for grokking via mechanistic interpretability
• Exploring Generalization in Deep Learning
• Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets
• The Implicit Bias of Gradient Descent on Separable Data
• The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon
• Understanding deep learning requires rethinking generalization

Related papers

• Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
• AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
• Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
• Convergence issues in Relational Concept Analysis based on AOC-posets
• Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
• Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks