Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout
cs.LG
Submitted: 2026-11-05
Updated: 2026-11-05
Code: https://github.com/jax-ml/jax
Project page: https://jax-ml.github.io/scaling-book
Terminology
Sources
- The Orthogonalized Read Is a Removable Training Scaffold for Recurrent Memory
- Zoology: Measuring and Improving Recall in Efficient Language Models
- Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes
- Simple linear attention language models balance the recall-throughput tradeoff
- Just read twice: closing the recall gap for recurrent language models
- Using Fast Weights to Attend to the Recent Past
- FlyWheel: Attention Kernels Built for the TPU
- xLSTM: Extended Long Short-Term Memory
- ATLAS: Learning to Optimally Memorize the Context at Test Time
- It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization
- Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity
- Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
- Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models
- Scaling Vision Transformers to 22 Billion Parameters
- DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
- Hungry Hungry Hippos: Towards Language Modeling with State Space Models
- Kernelized Linear Attention: Breaking the Capacity Wall with Symmetric Cones
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis