Systolic Linear Attention: Co-Designing Tile-Grouped Gated Delta Attention for TPUs
cs.LG
Submitted: 2026-11-04
Updated: 2026-11-17
Code: https://github.com/AI-Hypercomputer/maxtext
Project page: https://jax-ml.github.io/scaling-book
Terminology
Sources
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
- The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm
- Scalable Second Order Optimization for Deep Learning
- ATLAS: Learning to Optimally Memorize the Context at Test Time
- Titans: Learning to Memorize at Test Time
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues
- Accelerating Newton-Schulz Iteration for Orthogonalization via Chebyshev-type Polynomials
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- Log-Linear Attention
- Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
- Training Compute-Optimal Large Language Models
- DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models
- Mamba-3: Improved Sequence Modeling using State Space Principles
- Fast Inference from Transformers via Speculative Decoding
- Muon is Scalable for LLM Training
- Decoupled Weight Decay Regularization
- Scalable-Softmax Is Superior for Attention
- Momentum Transformer: Closing the Performance Gap Between Self-attention and Its Linearization
- RWKV-7 "Goose" with Expressive Dynamic State Evolution
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis