Inexact calculus of variations on the hyperspherical tangent bundle with connections to the attention mechanism
The authors study how a Transformer's flow map behaves when projected onto the unit hyperspherical manifold. They use Lagrangian optimization to analyze this system, deriving exact Euler-Lagrange equations and error bounds for approximating optimal paths on the sphere. This provides a mathematical framework connecting the attention mechanism to geometric variational problems.