FourierQK: Filter Shape, Admissibility and the Leakage-Coverage Law
The study investigated frequency-collapse attention by replacing standard dot products with bandpass filters at learned frequencies. It found that success requires suppressing DC and Nyquist components, using a narrow bandwidth of about two bins at paragraph scale, and employing admissible filter shapes. These findings establish design rules for effective spectral attention in language modeling.