Quantizing Recurrent Language Models Is Easier Than You Think
cs.LG, cs.AI, cs.CL
Submitted: 2026-11-01
Updated: 2026-11-01
Code: https://github.com/camail-official/LoReS
Terminology
Sources
- LFM2 Technical Report
- Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models
- Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
- Attention Sinks Induce Gradient Sinks: Massive Activations as Gradient Regulators in Transformers
- Quamba2: A Robust and Scalable Post-training Quantization Framework for Selective State Space Models
- Training Verifiers to Solve Math Word Problems
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
- SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- Hungry Hungry Hippos: Towards Language Modeling with State Space Models
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
- ParaTTT: Parallelizing Test-Time Training Layers with Deep Memories
- Pointer Sentinel Mixture Models
- On State Reduction in Linear Attention
- LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization
- Quantifying Memory Utilization with Effective State-Size
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark
- Massive Activations in Large Language Models
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis