Bridging Singular Learning Theory and Mechanistic Interpretability at the Billion-Parameter Scale
cs.LG
Submitted: 2026-10-31
Updated: 2026-10-31
Code: https://github.com/TransformerLensOrg/TransformerLens
License: http://creativecommons.org/licenses/by/4.0/
The gist: Mechanistic interpretability (MI) explains a computation in a trained network by identifying the components and variables that implement it and testing them with interventions.
Terminology
Abstract
Mechanistic interpretability (MI) explains a computation in a trained network by identifying the components and variables that implement it and testing them with interventions. Singular learning theory (SLT) describes the same network through the geometry of the loss around its parameters, summarized by the local learning coefficient. In pretrained language models the two have not been compared: SLT's estimates at this scale concern whole models or components whose roles were not independently established, and the estimator, which averages the loss over parameter settings drawn near the trained ones, has had no reference for how far those draws may move the parameters. We show that both are functions of one object, the map from a component's parameters to the model's predictions under inputs and interventions, whose first derivatives determine the Fisher information, the learning coefficient, and intervention effects, and whose higher derivatives add a quartic correction and pairwise terms. We test this correspondence in Pythia-1B on Dyck bracket matching at all 154 training checkpoints, against a five-stage circuit identified by interventions. We measure the local geometry of all 144 components inside the region where predictions are preserved, with the Fisher spectrum at every checkpoint, which gives the quadratic part of the learning-coefficient estimate at every temperature. Changes of the Fisher trace track changes of causal effect (median partial rank correlation 0.51), largely as a trend shared across the model, whereas the gain Fisher of a component's output and the large-temperature count are specific to the component. The count follows causal change across the population but not the growth of a formed circuit component. The quartic part is what a sampler that leaves the region registers beyond curvature. Pairwise interaction curvature tracks joint-ablation interactions.
Sources
- Understanding intermediate layers using linear classifier probes
- Dynamics of Transient Structure in In-Context Linear Regression Transformers
- MCMC Methods for Functions: Modifying Old Algorithms to Make Them Faster
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling
- Towards Spectroscopy: Susceptibility Clusters in Language Models
- Theoretical Limitations of Self-Attention in Neural Sequence Models
- How to use and interpret activation patching
- Dissociating Decodability and Causal Use in Bracket-Sequence Transformers
- What Formal Languages Can Transformers Express? A Survey
- Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory
- Rank-normalization, folding, and localization: An improved $\widehat{R}$ for assessing convergence of MCMC
- Patterning: The Dual of Interpretability
- Patterning in Practice: Debiasing Reward Models with Susceptibilities
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis