When Dynamic Scaling Fails: A Scale-State Feedback Governs Recurrent-State Quantization in Linear-Attention and SSM LLMs
This paper investigates numerical instability when using per-token dynamic quantization in recurrent neural networks. It found that a feedback loop between the quantization scale and the current state causes numerical runaway, leading to functional degradation over long decoding sequences. Containment strategies like persistent scales or calibrated ceilings are proposed as effective ways to stabilize inference.