Exact Attention Sensitivity and the Geometry of Transformer Stability
The episode discusses 'Exact Attention Sensitivity and the Geometry of Transformer Stability,' a paper detailing why transformer training is difficult. The authors propose a geometric framework showing that stability depends on architecture, not learned attention patterns. Key findings include how pre-LN allows direct gradient flow while post-LN causes decay, and the suggestion that temperature warmup can replace learning rate warmup.