Attention to Mamba: A Recipe for Cross-Architecture Distillation
cs.CL, cs.LG
Submitted: 2026-04-01
Updated: 2026-11-03
Code: https://github.com/state-spaces/mamba
Project page: http://skylion007.github.io/OpenWebTextCorpus
Terminology
Sources
- Longformer: The Long-Document Transformer
- Llamba: Scaling Distilled Recurrent Models for Efficient Language Processing
- Distillation Scaling Laws
- Analyzing the Evolution and Maintenance of ML Models on Hugging Face
- Rethinking Attention with Performers
- Theoretical Foundations of Deep Selective State-Space Models
- BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- HiPPO: Recurrent Memory with Optimal Polynomial Projections
- Efficiently Modeling Long Sequences with Structured State Spaces
- On the Parameterization and Initialization of Diagonal State Space Models
- Joint Fine-tuning and Conversion of Pretrained Speech and Language Models towards Linear Complexity
- Distilling the Knowledge in a Neural Network
- LoRA: Low-Rank Adaptation of Large Language Models
- Finetuning Pretrained Transformers into RNNs
- A Comprehensive Survey on Knowledge Distillation
- Linearizing Large Language Models
- The LAMBADA dataset: Word prediction requiring a broad discourse context
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data