Post-Grokking Collapse at the Representation–Readout Interface in Muon-Trained Transformers

Name: Post-Grokking Collapse at the Representation–Readout Interface in Muon-Trained Transformers
Uploaded: 2026-08-11T02:20:28.200357+00:00
Duration: 1852 s
Description: This episode discusses a paper on grokking in transformers trained on modular arithmetic. The Muon optimizer groks faster than AdamW but often loses generalization post-grokking, due to misalignment between hidden representations and the readout layer. The hosts explain that freezing embeddings and readout prevents collapse, and that a circuit can be present yet masked.

This episode discusses a paper on grokking in transformers trained on modular arithmetic. The Muon optimizer groks faster than AdamW but often loses generalization post-grokking, due to misalignment between hidden representations and the readout layer. The hosts explain that freezing embeddings and readout prevents collapse, and that a circuit can be present yet masked.

Open the full episode