Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale
Tejas Pradeep Shirodkar, P. J. Narayanan
cs.LG, stat.ML
Submitted: 2026-06-17
Comments: 34 pages, 7 figures, 6 tables. Empirical companion to arXiv:2606.05957
License: http://creativecommons.org/licenses/by/4.0/
The gist: Pretrained transformers sit near singular minima of the loss, where the Fisher information metric degenerates along dead directions: directions in parameter space along which the directional Fisher
Terminology
Abstract
Pretrained transformers sit near singular minima of the loss, where the Fisher information metric degenerates along dead directions: directions in parameter space along which the directional Fisher vanishes. Locating such a direction normally needs a forward pass and an eigendecomposition of activations, or a sampling-based complexity estimate; none returns a direction computable from the network's parameters alone. We give one, for LayerNorm transformers. The inverse-scale direction gamma-1/ gamma-1 of the LayerNorm affine is an exact algebraic kernel of the post-final-norm centred activation covariance, for any input distribution, and induces a corresponding dead direction in parameter space. It is read from the LN scale parameter alone, with no forward or backward pass and no eigensolve: the cheapest dead-direction read, specific to LayerNorm. We test it on 14 pretrained transformers (9 LayerNorm, 5 RMSNorm; 160 M- 35 B; language and vision objectives). At random initialisation the predicted direction matches the measured bottom singular direction (one forward pass, direct SVD) to four decimal places on 9/9 LayerNorm models, and is correctly absent on 5/5 RMSNorm models, which lack the mean-subtraction projector that creates it. On the trained checkpoint the covariance eigenvalue along this direction deepens by about 10 3 times and further dead directions open; the random-init-to-trained gap is a one-forward-pass, per-checkpoint readout of singular structure along the predicted coordinate. Two consequences follow in closed form: the residual stream's smallest singular value is preserved block-to-block on 13/14 transformers measured on their own input distribution, the one exception (Gemma 4 - 31 B) a genuine dead direction the same read pinpoints; and the kernel direction's presence classifies a transformer's normalisation from the parameters alone.
Sources
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
- Transformers learn through gradual rank increase
- Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
- Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth
- EigenTrack: Spectral Activation Feature Tracking for Hallucination and Out-of-Distribution Detection in LLMs and VLMs
- A Kronecker-factored approximate Fisher matrix for convolution layers
- Loss Landscape Degeneracy and Stagewise Development in Transformers
- NerVE: Nonlinear Eigenspectrum Dynamics in LLM Feed-Forward Networks
- Spectral Entropy Collapse as a Phase Transition in Delayed Generalisation: An Interventional and Predictive Framework for Grokkin
- Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
- Limitations of the Empirical Fisher Approximation for Natural Gradient Descent
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature
- Progress measures for grokking via mechanistic interpretability
- Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse
- DINOv2: Learning Robust Visual Features without Supervision
- Singular Fluctuation as Specific Heat in Bayesian Learning
- Geometry and Dynamics of LayerNorm
- Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
- DINOv3
- Small Singular Values Matter: A Random Matrix Analysis of Transformer Models
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks