Stochastic Parroting in Temporal Attention -- Regulating the Diagonal Sink
The authors analyzed temporal attention mechanisms to prove they suffer from a 'diagonal attention sink,' where information flow collapses onto specific tokens, especially as sequence length increases. This causes stochastic parroting in the attention scores, leading to information degeneration. They introduced regularization methods like diagonal masks and penalties, showing that these techniques significantly improve performance on long-horizon forecasting tasks.