Spectral Degeneration of Self-Attention under Isotropic Score Geometry
cs.LG, cs.AI, stat.ML
Submitted: 2026-11-02
Updated: 2026-11-02
Comments: NeurIPS 2026
Code: https://github.com/fastai/imagenette
License: http://creativecommons.org/licenses/by/4.0/
The gist: We study the spectral structure of the self-attention matrix in Transformer models through an idealized spectral model for the pre-softmax score matrix.
Terminology
Abstract
We study the spectral structure of the self-attention matrix in Transformer models through an idealized spectral model for the pre-softmax score matrix. Motivated by the rotational isotropy of query and key representations, our model replaces the random token-space orientation of the score matrix by its canonical singular-value representative with a logarithmic spike--bulk separation. Empirical diagnostics on pretrained vision and language transformers motivate this model: token-space singular matrices have nearly uniform eigen-angle distributions, and score spectra are dominated by a small number of effective directions. We then prove that the row-wise softmax map transforms this structure into a degenerate attention matrix: dominant modes are preserved, whereas the bulk spectral distributions concentrate at zero. Stable rank remains bounded for bidirectional attention but grows logarithmically for causal attention. Within the canonical model, these results explain how attention can preserve a small number of dominant directions while compressing the remaining bulk modes.
Sources
- On the Interpolation Error of Nonlinear Attention versus Linear Regression
- RoBERTa: A Robustly Optimized BERT Pretraining Approach
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis