Infinite-Width Limit of a Single Attention Layer: Analysis via Tensor Programs
This work rigorously proves that attention layer outputs are not Gaussian in their infinite-width limit under standard scaling. The output distribution converges to a hierarchical Gaussian, meaning it is Gaussian conditional on a random similarity score, which itself is Gaussian. This non-Gaussian behavior has implications for understanding signal propagation and optimization in deep Transformer architectures.