Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity
cs.LG
Submitted: 2026-07-08
Updated: 2026-11-16
Code: https://github.com/facebookresearch/sparse-delta-memory
Terminology
Sources
- Simple linear attention language models balance the recall-throughput tradeoff
- xLSTM: Extended Long Short-Term Memory
- Memory Layers at Scale
- PIQA: Reasoning about Physical Commonsense in Natural Language
- Short window attention enables long-term memorization
- BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Training Verifiers to Solve Math Word Problems
- Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning
- Hungry Hungry Hippos: Towards Language Modeling with State Space Models
- Training Compute-Optimal Large Language Models
- Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
- RACE: Large-scale ReAding Comprehension Dataset From Examinations
- Large Memory Layers with Product Keys
- Decoupled Weight Decay Regularization
- gpt-oss-120b & gpt-oss-20b Model Card
- Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
- WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- Linear Transformers Are Secretly Fast Weight Programmers
- RoFormer: Enhanced Transformer with Rotary Position Embedding
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis