Model Capacity Determines Grokking through Competing Memorisation and Generalisation Speeds
The paper investigates how model capacity dictates 'grokking'—the ability to solve problems by memorizing data rather than learning general rules—on modular arithmetic tasks. It finds that grokking occurs when a model's parameter count reaches the intersection point between two competing learning speeds: memorization speed and generalization speed. This mechanism explains grokking as a competition between fitting data quickly and finding robust solutions.