Forget Attention: Importance-Aware Attention Is All You Need
cs.AI, cs.CL, cs.LG
Submitted: 2026-06-01
Updated: 2026-10-28
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
- Mamba-3: Improved Sequence Modeling using State Space Principles
- Jamba: A Hybrid Transformer-Mamba Language Model
- Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling
- Hymba: A Hybrid-head Architecture for Small Language Models
- Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance
- Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models
- Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
- DAPE: Data-Adaptive Positional Encoding for Length Extrapolation
- Forgetting Transformer: Softmax Attention with a Forget Gate
- Theory, Analysis, and Best Practices for Sigmoid Self-Attention
- Training Compute-Optimal Large Language Models
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Zamba: A Compact 7B SSM Hybrid Model
- The Zamba2 Suite: Technical Report
- Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models
- RecurrentGemma: Moving Past Transformers for Efficient Open Language Models
- An Empirical Study of Mamba-based Language Models
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents