Softmax Attention on Gaussian Mixtures: Linear When It Can, Selective When It Must
Softmax attention on Gaussian mixtures shows that it can solve both linear and nonlinear tasks. It acts as a gated mixture of experts, using query-dependent context selection to choose relevant components of the mixture. This capability is essential for tasks like denoising where both shrinking the observation and selecting the correct cluster are required.