Intrinsic Structure: Spectral Identifiability for Mechanistic Interpretability

The paper introduces Koopman spectral analysis (KSA) as a mechanistic interpretability primitive that is provably identifiable, addressing the problem that discovered circuits in mechanistic interpretability may be artifacts of the method rather than intrinsic properties of the model. The authors treat the transformer forward pass as a controlled dynamical system with depth as time, lifting it with the Koopman operator to yield a finite linear realisation whose spectrum is a coordinate-free property of the model.

The main theoretical contributions are:

• Existence and uniqueness: "Theorem 5.1 (Existence and uniqueness of the Koopman realisation). Assume Assumption 1 together with (22) and (23). Then: (a) (Existence and uniqueness.) There exists a unique matrix A ∈ CN ×N satisfying A Ψ(x) = Eu∼ν Ψ F (x, u) µ-a.e. x ∈ X.... (c) (Spectral invariance.) The spectrum σ(A) ⊆ C is invariant under basis change and constitutes a coordinate-free invariant of the pair (F, HN)."

• Identifiability theorem: "Theorem 6.1 (Spectral identifiability of the Koopman realisation). Assume Assumptions 1 to 3, the regularity conditions (22)–(23) and (33)–(34), and centre the controls (Eν [u] = 0). Let ÂM be the EDMDc estimator of Theorem 4.8 with regularisation γ = 0, computed from M layer–token calibration samples... For every δ ∈ (0, 1/2) and every M ≥ M0 there exists a permutation πM of 1,..., N, determined by ÂM, such that with probability at least 1 − δ: (i) Eigenvalue identifiability: max λk (ÂM) − λπM (k) (A) ≤ c1 κ0 L2 (1+∥A∥2)/ηΨ × sqrt(N + log(1/δ)/M). (ii) Eigenvector identifiability: max v̂k − vπM (k) 2 ≤ c2 κ20 L2 (1+∥A∥2)/(ηΨ ∆) × sqrt(N + log(1/δ)/M)."

• Gap-free strengthening: "Theorem 6.8 (Eigenvalue identifiability without a spectral gap). Under the hypotheses of Theorem 6.1 but with Assumption 3 omitted, for every δ ∈ (0, 1/2) and every M ≥ M0eig as in (54), with probability at least 1 − δ, minπ maxk λk (ÂM) − λπ(k) (A) ≤ c1 (2N − 1) κ0 L2 (1+∥A∥2)/ηΨ × sqrt(N + log(1/δ)/M)."

• Minimax optimality: Theorem 7.1 (Minimax optimality in M). For every M with σ/√(2g1 M) ≤ ∆/2, inf  sup A∈A(∆,κ0) E minπ maxk λk (Â)−λπ(k) (A) ≥ σ/(8√(g1 M)).

• Heavy-tailed variant: "Theorem 7.2 (Heavy-tailed identifiability). Replace (R3) by the moment condition (R3′) E∥Ψ(x)∥42 ≤ κ4 N 2 and E∥u∥42 ≤ κ4 p2. Let Âmom M be the EDMDc estimator computed from median-of-means Gramians over K = ⌈8 log(1/δ)⌉ blocks. Then the conclusions (i)–(ii) of Theorem 6.1 hold for Âmom M verbatim, with the sub-Gaussian proxy L2 replaced by C√κ4 throughout, at the same rate M −1/2."

• Dissociation theorem: "Theorem 8.1 (Modal–principal dissociation). Let ρ(A) < 1 with A diagonalisable and of simple spectrum (Assumption 3), and let Σ ≻ 0 be the stationary covariance of the lifted recurrence Ψl+1 = AΨl + Bul under white controls... If some orthonormal eigenbasis of Σ consists of eigenvectors of A, then A is normal; equivalently, κ2 (V) = 1."

• SAE non-identifiability: "Corollary 9.5 (SAE non-identifiability). Let DSAE be a dictionary trained by minimising the reconstruction-plus-L1 objective (3). In general DSAE satisfies (DR1) but not (DR2). Consequently Theorem 6.1 does not apply to DSAE, and the run-to-run variability documented by [20, 23, 22] is a structural consequence of the objective’s failure to enforce K-invariance rather than an artefact of implementation."

• Algebraic completeness: "Theorem 9.2 (Algebraic completeness of the KSA intervention calculus). Assume the hypotheses of Theorem 5.1 and Assumption 3. Let Πk = vk ϕ⊤ k N k=1 be the spectral projectors of A, let r∗ denote its number of distinct eigenvalues µ1,..., µr∗, and let A(A):= C[A] ⊆ CN ×N. Then (a) A(A) is a commutative unital C-algebra of dimension r∗... (c) For any additive perturbation δ ∈ CN at layer l0, its downstream effect at layer l > l0 is Al−l0 δ = PN k=1 λl−l0 k Πk δ."

• Instance-dependent reduction: "Theorem 10.1 (Instance-dependent certified reduction). Assume Assumptions 1 and 2, (S1)–(S2), and let r ∈ 1,..., N − 1. (a) (Certified reduction bound.) G̃ − G̃r H∞ ≤ 2 PN i=r+1 σieff. (b) (Comparison with the classical bound.) For every i, σieff ≤ √(λmax (Gu)) σiclas. (c) (Strict sharpening.)... (71) is strict for at least N − reff (u) indices whenever r < N − reff (u)."

The experimental results on GPT-2 small, Gemma-2-2B, and Qwen3-8B-Base show:

• Convergence: "The distance falls monotonically with M on all three models... On Qwen3-8B-Base at layer 18 the spectral dictionary converges at −0.506 ± 0.031 - the value Theorem 6.1 predicts, within one standard error. To our knowledge this is the first observation of the parametric rate for a Koopman spectrum on a pretrained transformer."

• SAE invariance gap: "SAE dictionaries sit furthest from Koopman invariance on every model - further than a random orthonormal control... The predicted ordering on every model, strengthening with scale (2.1×, 3.1×, 3.4× the spectral residual), across three SAE families - ReLU, JumpReLU, TopK - and at every layer examined."

• Invariance penalty: "Raising γ from 0 to 1 reduces the invariance residual from 0.438 to 0.333 (24%), confirming that the penalty does what it is designed to do, and reduces the split-half spectral distance - identifiability itself, not a proxy for it - from 0.0170 to 0.0101 (41%) at matched L0 = 32."

• IOI circuit: Koopman modes beat random directions but lose to principal components on indirect-object identification, with the gap decaying 4.1× in depth-distance, as the theorem predicts. Specifically, at j = 8, principal directions remove 60% of the logit difference against the modes’ 25%.

• Universality criterion failure: "It declares every pair distinct: 36 of 36 cross-family pairs, and also 9 of 9 seed-replica pairs, the closest of which still sits at 4.64× its floor... A rule that separates two runs of the same architecture on the same data is not a universality criterion."

The paper's central claim is stated as: the identifiable object and the legible object are not the same object. The Koopman spectrum is described as an identifiable, model-intrinsic fingerprint with a stated error bar, not a legible decomposition.

Improvements to AI Systems:

1. Certified Interpretability with Error Bars

Replace heuristic circuit discovery with Koopman spectral analysis (KSA) to produce model-intrinsic, coordinate-free decompositions. The improved system outputs eigenvalues/eigenvectors with explicit finite-sample error bounds (Theorem 6.1), enabling users to distinguish genuine model structure from artifacts. It can certify when a discovered circuit is statistically significant versus a methodological artifact.

2. Identifiable Sparse Autoencoders (SAEs)

Modify SAE training objectives to enforce Koopman invariance (K-invariance) as a constraint, not just reconstruction+L1 sparsity. The improved system yields SAE dictionaries that are reproducible across training runs (addressing run-to-run variability) and satisfy identifiability guarantees. This enables stable, transferable feature dictionaries across seeds and architectures.

3. Gap-Free Spectral Estimation

Use the gap-free eigenvalue identifiability theorem (6.8) to estimate Koopman spectra even when eigenvalues are clustered or degenerate. The improved system provides reliable spectral fingerprints for models with near-degenerate dynamics, avoiding false claims of distinct modes when gaps are small.

4. Heavy-Tailed Robust Spectral Analysis

Apply median-of-means Gramian estimators (Theorem 7.2) to handle non-Gaussian, heavy-tailed activations common in transformers. The improved system maintains the same M-1/2 convergence rate without sub-Gaussian assumptions, making it robust to outlier layers or tokens.

5. Minimax-Optimal Spectral Estimation

Leverage the minimax lower bound (Theorem 7.1) to know the fundamental limit of spectral accuracy for a given sample size M. The improved system can automatically determine when additional calibration data will not improve spectral estimates, saving compute and guiding data collection.

6. Modal–Principal Dissociation Detection

Use Theorem 8.1 to detect when the model’s principal components (from covariance) diverge from its Koopman modes. The improved system flags cases where PCA-based interpretability is misleading, and instead recommends Koopman modes for causal intervention analysis.

7. Certified Circuit Reduction

Apply Theorem 10.1 to prune Koopman modes with certified H∞ error bounds, achieving instance-dependent reduction that is provably tighter than classical SVD-based truncation. The improved system can compress model dynamics to the minimal number of modes needed for a given task, with guaranteed worst-case error.

8. Intervention Calculus with Algebraic Completeness

Use the spectral projectors Π k to compute exact downstream effects of any additive perturbation (Theorem 9.2c). The improved system can predict, with algebraic certainty, how a change at layer l0 propagates to layer l, enabling precise causal interventions without re-running the model.

9. Universality Testing with Statistical Rigor

Replace heuristic universality criteria with KSA-based spectral distance that accounts for finite-sample error bars. The improved system can correctly reject false universality claims (as shown in the IOI experiment) and instead identify true cross-model invariants when they exist, using the parametric convergence rate as a test statistic.

10. Scale-Aware Interpretability Selection

Automatically choose between Koopman modes, principal components, or SAE features based on the model’s spectral residual and the task’s depth-distance. The improved system predicts (and empirically confirms) that principal components outperform Koopman modes for near-layer tasks, but Koopman modes become superior for deep propagation—allowing adaptive interpretability tool selection.

What the Improved AI System Can Do:

• Provide interpretability outputs with mathematically guaranteed error bars, not just qualitative heatmaps.

• Distinguish model-intrinsic structure from method-induced artifacts with high confidence.

• Train reproducible SAEs that converge to the same dictionary across seeds.

• Estimate spectra reliably even for ill-conditioned or heavy-tailed models.

• Certify the minimal number of modes needed for a task with worst-case error guarantees.

• Predict exact causal effects of interventions across layers.

• Correctly identify when two models share true universal circuits versus when apparent similarity is noise.

• Adapt its interpretability method (Koopman, PCA, SAE) based on task depth and model scale, optimizing for both legibility and identifiability.

Mechanistic interpretability explains models by identifying circuits inside them, but has no way to tell whether a circuit is a property of the model or an artifact of the method that found it. Sparse autoencoders illustrate the problem: different seeds and widths recover materially different features from the same activations, and no theory says whether that variability is incidental or structural. We put dictionary learning for interpretability on an identifiability footing. Treating the forward pass as a controlled dynamical system with depth as time and lifting it with the Koopman operator yields a finite linear realisation whose spectrum is a coordinate-free property of the model. We prove the spectrum is recoverable from M calibration samples at rate M-1/2 up to permutation - to our knowledge the first identifiability theorem for a mechanistic-interpretability primitive, with a matching minimax lower bound, a median-of-means variant for heavy-tailed activations, and a dissociation theorem: whenever the realisation is non-normal, the directions carrying activation variance and the directions carrying information across depth cannot coincide. The identifiable object and the legible object are not the same object. On GPT-2 small, Gemma-2-2B and Qwen3-8B-Base the spectrum converges everywhere and attains the predicted exponent on Qwen3-8B-Base (0.506 plus or minus 0.031); shortfalls collapse onto one curve against each cell's sample threshold. Koopman modes beat random directions but lose to principal components on indirect-object identification, with the gap decaying 4.1 times in depth-distance, as the theorem predicts. The Koopman spectrum is an identifiable, model-intrinsic fingerprint with a stated error bar, not a legible decomposition.