Dead-Direction Signatures: A Cheap Spectral Reading of Singular Complexity
Tejas Pradeep Shirodkar, P. J. Narayanan
cs.LG, stat.ML
Submitted: 2026-06-19
Comments: 42 pages, 10 figures, 6 tables. Empirical companion to arXiv:2606.05957
Code: https://github.com/timaeus-research/devinterp
License: http://creativecommons.org/licenses/by/4.0/
The gist: Singular learning theory characterises the complexity of a deep network through the geometry of its loss singularities.
Terminology
Abstract
Singular learning theory characterises the complexity of a deep network through the geometry of its loss singularities. The local learning coefficient (LLC), the standard estimator of Watanabe's real log canonical threshold (RLCT, lambda), reads this geometry as an integrated Bayesian scalar through SGLD, which needs per-task calibration and 10 4 - 10 6 forward-backward passes per checkpoint. We introduce Dead-Direction Signatures (DDS), a family of cheap closed-form spectral readings of singular structure: each reads a network's activation matrix or per-sample-gradient Fisher-Gram at a chosen layer, replacing the SGLD posterior chain with spectral linear algebra. The readings rest on a dead-direction framework that predicts a structural correlation between activation- and Fisher-side spectra at any singular minimum, and a rank-multiplicative volume identity that single-eigenvalue monitors cannot produce: the active-volume+(G) slope counts the dead directions, tracking the rank-deficit r across r in 1,2,3,4 (slope ratios 2.0, 3.1, 4.0 at r = 2,3,4 against the predicted 2,3,4), where the smallest eigenvalue is rank-blind. On reduced-rank regression with closed-form lambda, calibrated LLC recovers lambda at 99% mean and the DDS observables rank-track it at the framework-predicted sign; on a non-linear modular-addition transformer DDS separates d model across eighteen orders of magnitude where calibrated LLC at the protocol budget is rank-flat. Complementary to LLC's integrated posterior reading, DDS gives a directional, layer-local handle on a network's dead directions, read in closed form from its activation and gradient spectra.
Sources
- Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit
- Transformers learn through gradual rank increase
- Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth
- EigenTrack: Spectral Activation Feature Tracking for Hallucination and Out-of-Distribution Detection in LLMs and VLMs
- A Kronecker-factored approximate Fisher matrix for convolution layers
- Loss Landscape Degeneracy and Stagewise Development in Transformers
- NerVE: Nonlinear Eigenspectrum Dynamics in LLM Feed-Forward Networks
- Spectral Entropy Collapse as a Phase Transition in Delayed Generalisation: An Interventional and Predictive Framework for Grokkin
- Limitations of the Empirical Fisher Approximation for Natural Gradient Descent
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature
- Progress measures for grokking via mechanistic interpretability
- Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse
- Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets
- Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
- Small Singular Values Matter: A Random Matrix Analysis of Transformer Models
- Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient
- Spectral Edge Dynamics of Training Trajectories: Signal--Noise Geometry Across Scales
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks