Simulating Training Spikes of Deep Neural Networks by Layer-wise Random Feature Models
The study investigates intermittent loss spikes during Large Language Model pre-training by using simple quadratic models with heavy-tailed inputs, called Power-Law Random Feature (PLRF) models. The authors developed a simulation framework to model layer dynamics and found that these spikes occur under specific conditions related to data distribution and learning rates. They also derived an optimal $\beta_2$ schedule for Adam optimization that empirically improved performance.