Expanding Attention: A Taylor-Series Route to Linear-Time Transformers with a Fixed Number of Recurrent States

Name: Expanding Attention: A Taylor-Series Route to Linear-Time Transformers with a Fixed Number of Recurrent States
Uploaded: 2026-10-18T10:42:37.130856+00:00
Duration: 718 s
Description: The paper expands the exponential attention kernel into a truncated Taylor series of $n$ terms to create linear-time transformers with fixed recurrent states. This method reduces memory and increases decoding speed by replacing quadratic attention costs with a fixed number of recurrent states, achieving near-exact results for small truncation orders.

The paper expands the exponential attention kernel into a truncated Taylor series of $n$ terms to create linear-time transformers with fixed recurrent states. This method reduces memory and increases decoding speed by replacing quadratic attention costs with a fixed number of recurrent states, achieving near-exact results for small truncation orders.

Open the full episode