Rare Gate Disagreements Can Limit Plasticity: When Gradient Flow Mispredicts Finite-Batch SGD
The study investigates why population gradient flow mispredicts finite-batch stochastic gradient descent (SGD) during pretraining. It shows that when pretraining time exceeds a threshold related to batch size and step size, the system fails to recover plasticity because the inputs causing gate disagreements become too rare, limiting effective learning.