Random Quadratic Form on a Sphere: Synchronization by Common Noise
The episode discusses a paper titled "Random Quadratic Form on a Sphere: Synchronization by Common Noise" by Engel and Shalova. The paper proves that points on a sphere driven by common random noise cluster into two opposite points. This result is motivated by transformer models, suggesting that feed-forward layers alone can cause token clustering, challenging previous assumptions about the role of self-attention.