Learned Structure Shows Up Earlier in the Task's Own Basis: A Controlled Look at a Synchronisation Precursor in Grokking

The gist The Frequency Synchronization Degree (FSD) serves as an early predictor of grokking, synchronizing before generalisation across various modular arithmetic tasks and establishing that this precursor is a property of task-faithful representation bases rather than just Fourier circuits

How it works

The paper introduces the Frequency Synchronization Degree (FSD), a formally normalised, permutation-tested metric for Fourier circuit synchronisation that requires no prior knowledge of the circuit Across nine modular addition configurations (primes p ∈ 53, 71, 97, 113, 131), FSD reaches its post-grokking level 500–3,000 steps before grokking (mean lead +1,722 steps; every configuration positive) FSD synchronises before a restricted-logit loss baseline in all nine configurations, establishing FSD as the earliest available predictor of Fourier circuit formation

Key Findings and Causal Evidence

The research provides direct causal evidence that the inter-phase gap is a regularisation phenomenon Forking training at the FSD-ceiling step with weight decay λ ∈ 1, 2, 3, 4, 5, 10 induces strictly monotonically earlier grokking for the stable branches (λ ≤ 3), with ∆t ∝ 1/λ—the Fourier circuit was computation-complete at step 3,000 before grokking; only the rate of memorisation weight decay determined when generalisation occurred The inverse-λ law replicates across three primes (p ∈ 53, 97, 131): fitting seed-averaged ∆t gives R2 = 0.89–0.99 Grokking occurs at a near-constant memorisation norm across λ (coefficient of variation 0.13), grounding the empirical constant C in a threshold mechanism

Generalization Beyond Fourier Basis

Crucially, the synchronisation-precedes-generalisation effect is not an artefact of applying a Fourier detector to a Fourier circuit On the non-abelian group S5—whose grokking circuit lives in multi-dimensional irreducible representations, not 1-D Fourier modes—a basis-faithful generalisation of FSD precedes grokking on all six seeds (sign-test p = 0.03), while the original Fourier FSD does not This establishes that the precursor is a property of the task-faithful representation basis, not of Fourier-detecting a Fourier circuit

Architectural and Operational Insights

Zero-ablation results show that Block 1 attention is the most critical component, reducing accuracy to 5.4% (near chance, 1/97 ≈ 1%) when zeroed out Block 0 MLP contributes only 11.8 pp, yet its rank collapse robustly predicts grokking 500–3,000 steps ahead The results confirm that attention is a necessary component for the Fourier algorithm on addition Furthermore, modular multiplication shows lagging FSD by 10,000 steps, confirming the precursor is operation-specific

Practical Application and Theory

The FSD ceiling marks the earliest safe moment to apply aggressive regularisation; bumping λ before the circuit forms—at the memorisation point (train accuracy → 1) or a fixed early step—over-regularises the memorised solution and prevents grokking entirely (0/3 seeds grok at λhigh = 10), whereas the FSD-timed bump stays stable (3/3 for λhigh ≤ 5) The two-phase account predicts that Phase 2 duration is determined by the regularisation rate, not by circuit formation time This provides direct, controlled support for the circuit-efficiency account of Varma et al. [2023], and we summarise the transition with an empirical inverse-λ timing law ∆t = C/λ

Reproducibility and Limitations

The law is grounded in a threshold mechanism—grokking occurs at a near-constant weight norm across λ—and put to use: triggering a weight-decay increase at the FSD ceiling accelerates grokking 36–43% over a fixed schedule and, unlike a memorisation-timed trigger, does so without destabilising training The chaotic transition makes a single-run R2 unreliable (per-draw R2 ∈ [0.49, 1.00] even for clean primes), so error bars are essential, and a mild high-λ flattening bounds the law to λ ≲ 3 The natural extension is to larger groups and primes, deeper models, and other algorithmic tasks (sorting, finite-field arithmetic)

The paper demonstrates that grokking is a two-step process where a leading sub-circuit forms and compresses (Phase 1), creating preconditions for the primary circuit to consolidate (Phase 2), after which generalisation follows once memorisation capacity is pruned away—a process that can be directly controlled by adjusting weight decay The FSDgen construction gives a recipe for any task with a known representation-theoretic structure; identifying the right basis for tasks without one is an open direction. This framework offers a quantitative, mechanistic account of grokking as a two-phase process, with new causal and comparative evidence. The FSD and Fourier rank predict grokking 500–3,000 steps in advance across all nine addition experiments (mean lead +1,722 steps; every configuration positive). FSD synchronises before restricted-logit loss in all nine configurations, establishing it as the earliest available predictor of Fourier circuit formation. A weight-decay intervention—forking at a fixed step-1,000 checkpoint and varying only λ—causally confirms Phase 2 is a regularisation phase: the circuit is computation-complete 3,000 steps before grokking; only the rate of memorisation weight decay determines when generalisation occurs. This provides direct, controlled support for the circuit-efficiency account of Varma et al. [2023], and we summarise the transition with an empirical inverse-λ timing law ∆t = C/λ. The law is grounded in a threshold mechanism—grokking occurs at a near-constant weight norm across λ—and put to use: triggering a weight-decay increase at the FSD ceiling accelerates grokking 36–43% over a fixed schedule and, unlike a memorisation-timed trigger, does so without destabilising training. Most importantly, the precursor is not an artefact of the Fourier basis: a basis-faithful generalisation, FSDgen, precedes grokking on the non-abelian group S5 across all six seeds while the original Fourier FSD stays silent. Zero-ablation establishes that attention layers—not block 0 MLP—carry the primary computational load, clarifying the leading-indicator role of block 0 MLP. The multiplication and subtraction contrasts confirm the signal is operation-specific. Grokking is best understood as a two-step process: a leading sub-circuit forms and compresses (Phase 1), creating preconditions for the primary circuit to consolidate (Phase 2), after which generalisation follows once memorisation capacity is pruned away—a process that can be directly controlled by adjusting weight decay. The FSDgen construction gives a recipe for any task with a known representation-theoretic structure; identifying the right basis for tasks without one is an open direction. This framework offers a quantitative, mechanistic account of grokking as a two-phase process, with new causal and comparative evidence.

1. Bold Header: FSD-Triggered Training Acceleration

Improvement: Implement an adaptive weight decay schedule that monitors Frequency Synchronization Degree (FSD) and increases weight decay when FSD reaches its ceiling to accelerate grokking by 36–43% over a fixed schedule. This mechanism is causally confirmed because the Fourier circuit was computation-complete 3,000 steps before grokking; only the rate of memorisation weight decay determined when generalisation occurred.

What it can do: The system can dynamically transition from a memorization phase to a generalization phase much faster by applying aggressive regularization precisely when the underlying symbolic algorithm is already formed.

2. Bold Header: Operation-Specific Circuit Identification

Improvement: Use FSD to distinguish between different mathematical operations, as the paper shows Modular multiplication shows FSD lagging grokking by 10,000 steps, confirming the precursor is operation-specific.

What it can do: The system can identify which underlying algorithm (e.g., addition vs. multiplication) the transformer has learned during training simply by observing when its Fourier circuit synchronization precedes generalisation.

3. Bold Header: Representation-Basis Generalization for Non-Abelian Tasks

Improvement: Develop a FSDgen metric that measures synchronisation across irreducible representations of non-abelian groups, which precedes grokking on the non-abelian group S5.

What it can do: This extends grokking detection beyond modular arithmetic to other tasks whose circuits live in multi-dimensional representations, ensuring the precursor is a property of the task-faithful representation basis, not of Fourier-detecting a Fourier circuit.

4. Bold Header: Causal Necessity Testing via Zero-Ablation

Improvement: Employ zero-ablation tests to determine component necessity, specifically identifying which layers are load-bearing in the residual stream. For instance, the paper finds that Block 1 attention is the most critical component for addition mod 97.

What it can do: This allows for targeted architectural pruning or patching of specific components to understand their causal role in circuit formation and generalisation, moving beyond mere correlation to causal necessity.

5. Bold Header: Phase-2 Duration Control via Inverse-λ Law

Improvement: Implement a predictive model for the duration of Phase 2 using the empirical law ∆t = C/λ, where λ is the weight decay rate, providing a controlled fork that isolates Phase 2.

What it can do: This allows researchers to precisely predict how much longer the memorization phase will last given a planned regularization strategy, enabling optimized training schedules.

6. Bold Header: Enhanced Symbolic Extraction with Rank-Aware Models

Improvement: Utilize FourierKAN extraction, which provides an exact analytical measure of per-neuron representational compression, and use this information to guide model design or validation for tasks where the analytical DFT decomposition is known (e.g., addition mod 97).

What it can do: The system can generate interpretable symbolic formulas for individual neurons, allowing developers to verify that the learned circuit matches theoretical expectations (like a rank-1 collapse) and identify when a model fails to converge on a specific basis.