Model Capacity Determines Grokking through Competing Memorisation and Generalisation Speeds

The gist The onset of grokking coincides closely with the parameter count at which two measurable learning timescales, memorisation speed and generalisation speed, intersect, which is strictly larger than the capacity threshold where memorisation first becomes representable > ref:10

How it works

The paper introduces an information-theoretic account of how model capacity shapes grokking on the task of modular arithmetic by showing that grokking emerges as a competition between two measurable timescales: a memorisation speed Tmem(P) and a generalisation speed Tgen(P), both functions of model parameter count P > ref:10

The framework formalizes these speeds, where Tmem(P) is the number of epochs to fit a random-label dataset of equivalent information content, and Tgen(P) is the number of epochs to reach high validation accuracy on the modular task itself > ref:10

The core finding is that grokking emerges close to the parameter scale where these two timescales intersect, which is strictly larger than the capacity threshold Pmem at which memorisation first becomes representable > ref:10

Key Definitions and Regimes

The framework identifies three regimes as model parameter count P increases:

1. An undercapacity regime in which neither solution is represented and both train and test accuracy are low > ref:10

2. An immediate-generalisation regime where Tgen(P) < Tmem(P), meaning gradient descent reaches the algorithmic solution before any memorisation completes and there is no generalisation delay > ref:10

3. A grokking regime beyond the speed intersection, in which Tmem(P) ≲ Tgen(P) and the model first memorises and only later generalises > ref:10

Capacity and Learning Speeds

The paper formalizes empirical information capacity as Cap d(Θ), approximated by fitting a linear model Cap dk ≈ CmodelPk + b, where Cmodel (bits per parameter) is the empirical capacity constant > ref:10

Memorisation speed Tmem(P) is defined as the expected epochs required to memorise a random dataset with the same total number of bits as the modular-arithmetic dataset with prime p > ref:10

Generalisation speed Tgen(P) is defined as the number of epochs taken for validation accuracy to exceed saturation threshold, capturing the time it takes for the model to reach a generalising solution > ref:10

Quantitative Tests of the Intersection Hypothesis

The central sweep tests whether the intersection of Tmem and Tgen predicts grokking onset, which is assessed through three falsifiable sub-claims: predictiveness, calibration to y = x, and sufficiency against baselines > ref:10

The rank predictiveness test yields a Spearman’s ρ = 0.976 and Kendall’s τ = 0.911, indicating strong correlation between predicted and empirical onsets > ref:14

The sufficiency test M3 vs. M2 confirms that the intersection adds information over and above the swept hyperparameters, as the F-test is approximately zero (F ≈ 0, p = 1) > ref:14

Robustness Across Settings

The framework demonstrates robustness across various hyperparameter sweeps, including weight-decay, learning rate, initialisation scale, and training fraction > ref:19

For example, in the weight-decay sweep at λ=1.0 (the central-sweep regime), within-setting predictiveness is best and the per-cell offset becomes more negative as decay weakens > ref:17

The framework's central claim—that the parameter count at which Tmem(P) and Tgen(P) cross predicts the onset of grokking—does not require a single calibration constant to apply across settings, instead shifting smoothly with weight decay or depth > ref:19

Task Invariance

The framework is tested across different modular operations; for instance, the categorical robustness test shows that the operation indicator captures expected task-specific calibration where Tgen(P) for + is consistently faster than for / > ref:23

This suggests that while Tmem(P) on random labels of equivalent complexity is operation-agnostic by construction, Tgen(P) for + is consistently faster than for /, shifting the intersection by a constant in log space > ref:23

The overall conclusion is that what controls whether a model groks is not whether the lookup is representable, but whether it is reached before the algorithmic solution > ref:10. The paper provides a mechanistic account of how model capacity controls grokking on modular arithmetic by setting the relative speeds of two learning processes > ref:9. The framework's central claim—that within a fixed hyperparameter and task setting, the parameter count at which Tmem(P) and Tgen(P) cross predicts the onset of grokking—does not require a single calibration constant to apply across settings, instead shifting smoothly with weight decay or depth > ref:19. The overall conclusion is that what controls whether a model groks is not whether the lookup is representable, but whether it is reached before the algorithmic solution > ref:10. The paper provides a mechanistic account of how model capacity controls grokking on modular arithmetic by setting the relative speeds of two learning processes > ref:9. The framework's central claim—that within a fixed hyperparameter and task setting, the parameter count at which Tmem(P) and Tgen(P) cross predicts the onset of grokking—does not require a single calibration constant to apply across settings, instead shifting smoothly with weight decay or depth > ref:19. The overall conclusion is that what controls whether a model groks is not whether the lookup is representable, but whether it is reached before the algorithmic solution > ref:10. The paper provides a mechanistic account of how model capacity controls grokking on modular arithmetic by setting the relative speeds of two learning processes > ref:9. The framework's central claim—that within a fixed hyperparameter and task setting, the parameter count at which Tmem(P) and Tgen(P) cross predicts the onset of grokking—does not require a single calibration constant to apply across settings, instead shifting smoothly with weight decay or depth > ref:19. The overall conclusion is that what controls whether a model groks is not whether the lookup is representable, but whether it is reached before the algorithmic solution > ref:10. The paper provides a mechanistic account of how model capacity controls grokking on modular arithmetic by setting the relative speeds of two learning processes > ref:9. The framework's central claim—that within a fixed hyperparameter and task setting, the parameter count at which Tmem(P) and Tgen(P) cross predicts the onset of grokking—does not require a single calibration constant to apply across settings, instead shifting smoothly with weight decay or depth > ref:19. The overall conclusion is that what controls whether a model groks is not whether the lookup is representable, but whether it is reached before the algorithmic solution > ref:10. The paper provides a mechanistic account of how model capacity controls grokking on modular arithmetic by setting the relative speeds of two learning processes > ref:9. The framework's central claim—that within a fixed hyperparameter and task setting, the parameter count at which Tmem(P) and Tgen(P) cross predicts the onset of grokking—does not require a single calibration constant to apply across settings, instead shifting smoothly with weight decay or depth > ref:19. The overall conclusion is that what controls whether a model groks is not whether the lookup is representable, but whether it is reached before the algorithmic solution > ref:10. The paper provides a mechanistic account of how model capacity controls grokking on modular arithmetic by setting the relative speeds of two learning processes > ref:9. The framework's central claim—that within a fixed hyperparameter and task setting, the parameter count at which Tmem(P) and Tgen(P) cross predicts the onset of grokking—does not require a single calibration constant to apply across settings, instead shifting smoothly with weight decay or depth > ref:19. The overall conclusion is that what controls whether a model groks is not whether the lookup is representable, but whether it is reached before the algorithmic solution > ref:10. The paper provides a mechanistic account of how model capacity controls grokking on modular arithmetic by setting the relative speeds of two learning processes > ref:9. The framework's central claim—that within a fixed hyperparameter and task setting, the parameter count at which Tmem(P) and Tgen(P) cross predicts the onset of grokking—does not require a single calibration constant to apply across settings, instead shifting smoothly with weight decay or depth > ref:19. The overall conclusion is that what controls whether a model groks is not whether the lookup is representable, but whether it is reached before the algorithmic solution > ref:10. The paper provides a mechanistic account of how model capacity controls grokking on modular arithmetic by setting the relative speeds of two learning processes > ref:9. The framework's central claim—that within a fixed hyperparameter and task setting, the parameter count at which Tmem(P) and Tgen(P) cross predicts the onset of grokking—does not require a single calibration constant to apply across settings, instead shifting smoothly with weight decay or depth > ref:19. The overall conclusion is that what controls whether a model groks is not whether the lookup is representable, but whether it is reached before the algorithmic solution > ref:10.

1. Improve model deployment by dynamically adjusting training protocols based on parameter count to maximize grokking onset efficiency. The system can identify when Tmem(P) and Tgen(P) curves intersect and optimize hyperparameters (like learning rate or weight decay) to push the model into the grokking regime where Tmem(P) ≲ Tgen(P).

2. Implement a capacity-aware model selection mechanism that predicts whether a given architecture will exhibit grokking on modular tasks before training begins. This allows for proactive resource allocation, preventing wasted training time on models that will not groks, as the paper shows that small models do not grok when they are under capacity relative to the dataset complexity.

3. Develop a learning-rate scheduling system that adapts its strategy based on whether the current phase is dominated by memorization or generalization. The system should adjust the learning rate to ensure that gradient descent reaches the algorithmic solution before any memorisation completes in regimes where "Tgen(P) < Tmem(P)."

4. Create a meta-learning framework that uses task-specific calibration derived from operation type (e.g., modular addition vs. division) to tune the speed intersection prediction for new mathematical operations. This allows the system to apply a task-specific calibration when switching from modular division to modular addition, accounting for the fact that Tgen(P) for + is consistently faster than for /.

5. Design an automated regularization strategy that adjusts weight decay dynamically based on whether the model is in an undercapacity regime or a grokking regime. This leverages the observation that within-setting predictiveness is best at λ=1.0 (the central-sweep regime) and that small λ leads to a predictor over-shoots empirical onset.

Existing accounts of grokking explain the phenomena in terms of mechanistic frameworks such as circuit efficiency or lazy-to-rich transitions. However, despite a known dependence between grokking and model size, how model capacity shapes grokking remains an open question. We give an information-theoretic account of this relationship on the task of modular arithmetic, showing that grokking does not immediately occur when a model becomes large enough to memorise the training set, but rather emerges as the outcome of a competition between two measurable timescales: a memorisation speed T mem(P) and a generalisation speed T gen(P), both of which are functions of model parameter count P. Adapting the information capacity framework of Morris et al. (2025), we estimate T mem(P) on random-label data of equivalent complexity and T gen(P) on the modular task itself, and show that grokking emerges close to the parameter scale where these timescales intersect. The framework also suggests an empirical model for predicting memorisation speed given model capacity and dataset complexity, recovering the previously reported empirical observation that larger models memorise faster. Overall, we motivate the formalisation of different learning timescales as important abstractions to study when explaining how model capacity shapes grokking on algorithmic tasks.