Attention Sinks Induce Gradient Sinks: Massive Activations as Gradient Regulators in Transformers
The study identifies 'gradient sinks' as a backward-pass counterpart to attention sinks in pre-norm Transformers, showing concentrated gradient pressure at early tokens. Massive activations are found to act as learned regulators that absorb this pressure. The proposed V-scale intervention, a value-path gradient valve, successfully suppresses massive activations while preserving the beneficial sink structure, leading to improved performance across various tasks.