When Dynamic Scaling Fails: A Scale-State Feedback Governs Recurrent-State Quantization in Linear-Attention and SSM LLMs

The gist The mechanism is a feedback between the quantizer scale and the state it is derived from, which explains why numerical instability and functional degradation are separable failures

The Failure Frontier

When dynamic scaling fails, a bit-width certified as safe at short exposure need not stay safe as decode proceeds The closest developed literature is low-bit KV-cache quantization where the cache is stored rather than recurrent, so error never re-enters the state and exposure is not reported as a variable The probe everyone reaches for is retrieval accuracy, which turns out to be the least sensitive instrument available for this failure A bit-width therefore has to be reported together with the horizon at which it was measured, and this much holds independently of any mechanistic claim

The Scale–State Feedback Mechanism

The per-token dynamic quantization re-derives the per-channel step size from the current state at every step, st,c = max St,c/qmax Inside a recurrence it also couples the quantizer to its own output, and that coupling turns out to be destabilizing Numerical runaway is real and policy-driven Under per-token dynamic INT4, the closed-loop recurrent-state norm grows super-linearly with decode length (Figure 1b) The loop is the cause of the runaway, and is consistent with, but not proven to be, the whole path to functional loss Causal identification via scale-yoking involves cutting the feedback edge directly by running FP and a per-token-dynamic-INT4 model in lockstep on the same tokens Yoking eliminates the numerical explosion (Figure 2b) The functional recovery is far smaller than the numerical one, as two thirds of the damage survives an intervention that removes the runaway completely

Containment Strategies

The paper identifies four principles for containment: a persistent scale eliminates the feedback edge, a calibrated ceiling bounds it, periodic reset interrupts it, and a higher bit-width merely delays the loop Outlier-frequency channel targeting or exact sparse escape does not beat uniform group-wise quantization because locating the outlier channels does not beat uniform group-wise quantization The best leak-free allocation is an extreme allocation—top3 layers at INT8, the rest at INT2, averaging 4.0 bits—which is a genuine Pareto improvement over uniform group-wise at a matched budget Uniform group-wise scaling beats all three containment interventions because it is the least-damaging uniform policy on these models

Policy and Exposure Interaction

The adaptive policy is the one that decays, as the policy that tracks the distribution most faithfully is also the one whose disadvantage grows with every step of exposure The dynamic−static excess-NLL gap widens monotonically with quantized exposure across architectures and scales At short exposure it often looks like the better choice, but the crossover is the point at which its accumulating history damage overtakes that lower per-step cost The survival curve Pr(τ > h) orders the policies monotonically at every horizon, and at every FP-confidence threshold (Appendix F)

Residual Analysis

The residual that survives yoking is a resolution limit, on the evidence of a bit-width sweep and against the alternative that rounding error accumulates Stochastic rounding is not the remedy because removing the bias makes each step’s error larger—√2 in RMS, 1.42× measured on the states—and a recurrence amplifies magnitude, not sign Uniform group-wise scaling beats all three designs because it removes the feedback edge and differs only in how many scales they spend on the channel axis The best policy short is the best policy long, so on this metric short evaluation does not select the wrong policy, unlike the crossover of §7

The paper concludes that a persistent scale eliminates the feedback edge, a calibrated ceiling bounds it, periodic reset interrupts it, and a higher bit-width merely delays the loop Prefer any persistently calibrated scale over per-token dynamic scaling Check group-wise against plain static on the target architecture rather than assuming group-wise wins Protect the first recurrent layer, and expect little from learned allocation beyond that The point we would most want carried forward is the reporting one: a bit-width certified at 512 steps says little about 4K and nothing about 32K, and the horizon is where low-bit recurrentstate inference actually breaks The best policy short is the best policy long, so on this metric short evaluation does not select the wrong policy, unlike the crossover of §7 The survival curve Pr(τ > h) orders the policies monotonically at every horizon, and at every FP-confidence threshold (Appendix F) The best policy short is the best policy long, so on this metric short evaluation does not select the wrong policy, unlike the crossover of §7. The paper is an empirical analysis of numerical stability in quantized inference It uses publicly released model checkpoints and the public WikiText103 corpus, introduces no new data collection, and involves no human subjects or personally identifying information The practical effect of the findings, if any, is to make low-bit long-context inference more reliable and to warn against a specific evaluation practice (certifying a quantizer at short decode length) that can overstate deployed reliability All experiments use public model checkpoints and the WikiText-103 corpus, and every run reported here fits on a single RTX 3090 Code, result files and the project page are at https://avalon-s.github.io/dynamic-scaling-fails/ The GDN runs are bit-identical between independent executions: re-running the long-horizon sweep of Appendix E from scratch reproduced all 35 per-document excess-NLL values to the last digit The Mamba-2 runtime is not deterministic, and repeating the same configuration moves a singledocument loss by a few hundredths of a nat, so Mamba numbers should be read as reproducible in distribution rather than exactly State norms reproduce far less well than losses, because a runaway amplifies whatever differs between runs The paper is a scientific paper titled When Dynamic Scaling Fails: A Scale–State Feedback Governs Recurrent-State Quantization in Linear-Attention and SSM LLMs. It is an empirical analysis of numerical stability in quantized inference. It uses publicly released model checkpoints and the public WikiText103 corpus, introduces no new data collection, and involves no human subjects or personally identifying information. The synthetic recall probes are generated from fictional entities. The practical effect of the findings, if any, is to make low-bit long-context inference more reliable and to warn against a specific evaluation practice (certifying a quantizer at short decode length) that can overstate deployed reliability. The paper is 2610.15139>

--- Page 2 ---

When Dynamic Scaling Fails: A Scale–State Feedback Governs Recurrent-State Quantization in Linear-Attention and SSM LLMs <ref:2610.

1. Improved State Quantization Policy: Implement a persistent calibrated scale instead of per-token dynamic INT4 scaling to eliminate numerical instability by stating that A persistent fixed-range static quantizer holds the same norm at ∼ 17–18, and a low clip rate is enough to do it.

2. Improved Allocation Strategy: Adopt extreme allocation—top3 layers at INT8, the rest INT2 when operating under a leak-free protocol, as this is shown to be a genuine Pareto improvement over uniform group-wise at a matched budget, beating uniform INT3 at the same 4.0 bits by t = −12.8.

3. Improved Deployment Certification: Report the decode horizon H at which a bit-width was certified instead of just naming a bit-width, because a bit-width certified at 512 steps says little about 4K and nothing about 32K, and the horizon is where low-bit recurrentstate inference actually breaks.

4. Improved Risk Assessment: Use the survival curve to inform deployment decisions by noting that The best policy short is the best policy long, as short evaluation badly underestimates risk because the good policy has no short-window events to fit.

Linear-attention and state-space LLMs carry long context in a fixed-size recurrent state, and quantizing that state is the natural way to cut its cost during long decode. Unlike a weight tensor, the state is rewritten at every step, so a quantizer applied at each step sits inside the recurrence and its errors compound. INT4 that leaves synthetic recall perfectly intact at the start destroys it a thousand steps later. The mechanism is a feedback between the quantizer scale and the state it is derived from. Yoking that scale to a full-precision reference removes the resulting norm explosion entirely, yet recovers only 1.8 of the 5.4 nats lost, so numerical instability and functional degradation are separable failures. The remainder is a resolution problem that locating outliers does not solve, since every outlier-aware scheme here loses to plain persistent scaling. Allocating bits across layers does help, but under article-disjoint splits a learned allocation collapses onto the trivial rule of protecting the first few layers. Run out to 32K, group-wise INT4 is worth no more than discarding the state. Contain the scale rather than locate outliers, and report the decode horizon at which a bit-width was certified.