Do Flatter Minima Drive Better Generalization? An Algorithmic Separation in Grokking
The paper investigates whether flat loss landscapes cause generalization in grokking, finding that flatness alone is insufficient. Biasing training toward flatter solutions using sharpness-aware minimization (SAM) does not reliably induce generalization on its own. However, combining SAM with weight decay significantly accelerates the transition to generalizing solutions, suggesting an algorithmic separation between memorizing and generalizing interpolants.