Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention
cs.LG, cs.AI, stat.ML
Submitted: 2026-05-08
Updated: 2026-11-03
Code: https://github.com/karpathy/char-rnn
Project page: https://attention-motifs.github.io/v2/index.html
Terminology
Sources
- Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation
- Stochastic Parroting in Temporal Attention -- Regulating the Diagonal Sink
- CodeSearchNet Challenge: Evaluating the State of Semantic Code Search
- A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training
- Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin
- Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Tasks
- Attention Sinks in Diffusion Language Models
- Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
- Analysis of Attention in Video Diffusion Transformers
- Exclusive Self Attention
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis