Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikes
Deep neural networks show periodic loss spikes during long training called Slingshot Mechanism. This paper proves these spikes are caused by floating-point arithmetic limits, specifically when logit differences exceed precision thresholds. This leads to Softmax Collapse and a feedback loop called Numerical Feature Inflation, explaining abnormal parameter growth in late-stage training.