Poly-attention: a general scheme for higher-order self-attention
This research introduces poly-attention, a generalized method for self-attention that uses specific mathematical polynomials to model higher-order relationships between input tokens. It defines attention based on these 'Attention Polynomials' and shows how this framework can simulate complex operations like function composition while maintaining efficient quadratic time complexity.