Improved state mixing in higher-order and block diagonal linear recurrent networks
cs.LG
Submitted: 2026-02-12
Updated: 2026-11-09
Terminology
Sources
- Zoology: Measuring and Improving Recall in Efficient Language Models
- RotRNN: Modelling Long Sequences with Rotations
- Language Models Need Inductive Biases to Count Inductively
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Theoretical Foundations of Deep Selective State-Space Models
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
- Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models
- Neural Networks and the Chomsky Hierarchy
- Language Modeling Is Compression
- Advancing Regular Language Reasoning in Linear Recurrent Neural Networks
- Were RNNs All We Needed?
- Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues
- Grokking modular arithmetic
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- Efficiently Modeling Long Sequences with Structured State Spaces
- Scaling Laws for Neural Language Models
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Decoupled Weight Decay Regularization
- The Illusion of State in State-Space Models
- Fixed-Point RNNs: Interpolating from Diagonal to Dense
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis