LinQuant: Functional Invariances for Low-Precision Linear Recurrences
cs.LG
Submitted: 2026-10-29
Updated: 2026-11-14
Comments: 35 pages, 16 figures. Code: https://github.com/ntumm120/LinQuant
Code: https://github.com/ntumm120/LinQuant
License: http://creativecommons.org/licenses/by/4.0/
The gist: Linear recurrences are emerging as practical alternatives to softmax attention in large language models, appearing in both standalone and hybrid architectures.
Abstract
Linear recurrences are emerging as practical alternatives to softmax attention in large language models, appearing in both standalone and hybrid architectures. However, despite their rapid emergence, low-precision inference remains much less developed for these models than for softmax attention. During prefill, chunkwise-parallel algorithms cast much of the recurrence as dense matrix multiplications, suggesting a natural path to low-precision execution. Simply quantizing these operations in their native form, however, can be fragile. In this work, we identify functional invariances that preserve the full-precision computation while making the operands more amenable to quantization. Building on this observation, we introduce LinQuant, a framework for low-precision recurrent prefill based on seven such invariances, including rotations, diagonal rescalings, and alternative placements of recurrent decay and write factors. We derive when these transformations remain valid across common recurrence families spanning linear, gated, and delta-rule updates, and show that they provide complementary degrees of freedom for improving low-precision accuracy. Finally, we translate this framework to production by augmenting SSDi8, an existing low-precision prefill pipeline for Mamba-2, with the applicable invariances. LinQuant improves perplexity at matched latency on Mamba-2 1.3B, 2.7B, and 8B and Nemotron-H 8B, demonstrating the practical utility of invariance-based quantization for recurrent models.
Sources
- LFM2 Technical Report
- xLSTM: Extended Long Short-Term Memory
- Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models
- The Curious Case of In-Training Compression of State Space Models
- Quamba: A Post-Training Quantization Recipe for Selective State Space Models
- Quamba2: A Robust and Scalable Post-training Quantization Framework for Selective State Space Models
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
- On State Reduction in Linear Attention
- Quantifying Memory Utilization with Effective State-Size
- RWKV-7 "Goose" with Expressive Dynamic State Evolution
- Linear Transformers Are Secretly Fast Weight Programmers
- FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
- DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products
- Kimi Linear: An Expressive, Efficient Attention Architecture
- QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks
- Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences
- MambaQuant: Quantizing the Mamba Family with Variance Aligned Rotation Methods
- Gated Delta Networks: Improving Mamba2 with Delta Rule
- DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis