CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training

arXiv:2510.18784 · cs.LG · Submitted 2025-10-21 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training".

Jane: Here's the gist of the paper. CAGE introduces Curvature-Aware Gradient Estimation for Accurate Quantization-Aware Training to address accuracy gaps between low-bit quantized models and native training The CAGE method provides a principled way to augment the standard straight-through estimator with a curvature-aware correction term derived from a multi-objective view of QAT Pruning redundant reasoning tokens The paper introduces CAGE as a new QAT method that augments the straight-through estimator (STE) gradient with a curvature-aware correction designed to counteract the loss increase induced by quantization CAGE is derived from a multi-objective view of QAT that balances loss minimization with the quantization constraints, yielding a principled correction term that depends on local curvature information On the theoretical side, the authors introduce Pareto-optimal solutions for quantized optimization and establish that CAGE yields strong convergence guarantees in the smooth non-convex setting How the training signal is built The core of CAGE involves reformulating QAT as a multi-objective optimization problem, aiming to simultaneously minimize task loss and satisfy quantization constraints The authors define a condition for Pareto-optimality where any improvement in one component necessarily leads to a degradation in the other This condition inspires CAGE, which provides both theoretical guarantees and high accuracy and efficiency in practice The method is implemented as optimizer-agnostic but leverages Adam statistics for a highly-efficient implementation For stateful optimizers like Adam, two correction variants are proposed: the coupled correction and the decoupled correction The coupled variant introduces a diagonally preconditioned curvature-aware correction term where curvature is approximated using Adam statistics already maintained within the optimizer states Where the gains come from The effectiveness of CAGE is validated through extensive experiments, including synthetic experiments and post-training fine-tuning of standard models For QAT fine-tuning, CAGE roughly halves the quantization error relative to the prior best method In pre-training experiments, CAGE-trained models with 3-bits weights and activations (W3A3) offer lower loss than 4-bit (W4A4) models trained with the prior best method The approach is optimizer-agnostic, leading to consistent gains across several optimizers including AdamW, Muon, and Shampoo Furthermore, scaling law analysis confirms that CAGE improves the effective capacity of models compared to existing QAT methods Final summary and discussion CAGE is a novel method for quantization-aware training designed to mitigate accuracy degradation inherent in low-bit quantization by augmenting the gradient with a principled, curvature-aware correction term The key theoretical contribution is the definition of Paretooptimal solutions for quantized optimization and strong ergodic convergence guarantees to a Pareto-optimal point in the smooth non-convex setting Empirical validation on both pre-training and post-training QAT demonstrates its effectiveness across various model sizes and precisions The method introduces negligible computational relative to the QAT baseline, adding only one extra quantization call and a small number of elementwise operations CAGE consistently improves upon strong baselines, such as QuEST, by significantly reducing stationary error across all model sizes The paper concludes that incorporating local curvature information and regularization are useful for QAT performance References Ajanthan, T., Dokania, P.

Tom: K., Hartley, R., and Torr, P.

Jane: H.

Tom: S., Proximal mean-field for neural network quantization Ajanthan, T., Gupta, K., Torr, P.

Paper summary: Tom: So, we're wrapping up this look at CAGE, which stands for Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training, and what it actually means for how we train these big models.

Jane: Yeah, CAGE is essentially taking the standard way we calculate gradients during quantization—the straight-through estimator—and adding a correction term that looks at the local curvature of the loss landscape.

Lu: The core idea here is moving away from just guessing what happens when you quantize parameters to actually balancing two things: minimizing how bad your model’s task loss is, and making sure the weights stay within the allowed quantized set.

Meng: So, they use a multi-objective view to define these Pareto optimal solutions where neither objective can be improved without hurting the other. That's a pretty principled way to think about finding that sweet spot between accuracy and bit precision.

Jane: Exactly, and theoretically, they prove that this approach leads to strong convergence guarantees in the smooth non-convex setting, which means if you run the training long enough, it’s highly likely to settle at that best point they defined.

Tom: But what does this mean for a developer? It means when you fine-tune a model after training, CAGE can roughly halve your quantization error compared to the previous best method. That's a big number for deployment stability.

Lu: And in the pre-training stage, they found that models trained with three bits of weights and activations using CAGE actually perform better than four bit models trained with older methods. That suggests it helps build better foundational knowledge from scratch.

Meng: I'm interested in the practical side—how much extra compute does this add? They say it’s negligible, just one extra quantization call and some small element-wise math. If that holds up when we scale to massive models, that’s a win for us.

Jane: It is, and they show it works across different optimizers like AdamW and Shampoo without needing custom tweaks for every single one of those. It’s meant to be relatively easy to drop into an existing pipeline.

Tom: So the main thing here is that CAGE isn't just a tweak; it’s a principled correction based on local curvature information, giving us more control over the quantization process during training.

Lu: It sets up a new theoretical framework for how we handle these constrained optimization problems in neural networks.

Meng: It gives us confidence that we can push lower bit precisions without completely losing accuracy, provided the setup is right.

Jane: So CAGE is trying to bridge that gap between what we want—a highly accurate quantized model—and what we can achieve with current training techniques.

Tom: It’s a solid piece of work that shows how adding curvature awareness can make a big difference in the final numbers you ship.

Jane: And it opens up a new way to look at QAT as more than just a simple loss minimization problem.

Conclusion: Tom: So, we're wrapping up this look at CAGE, which stands for Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training. It’s a new way to handle that gap between low bit quantized models and those trained from scratch.

Jane: Right. The authors are looking at how we calculate gradients during quantization—the straight-through estimator—and they've added a correction term based on the local curvature of the loss landscape. They use a multi-objective view, balancing minimizing task loss with satisfying those specific bit constraints.

Lu: Theoretically, they’re proving there are Pareto optimal solutions for this optimization problem and showing strong convergence guarantees in that smooth non-convex setting. That’s a big piece of work because it gives us mathematical confidence in the training process.

Meng: So, what does this actually mean when we look at deploying models? It means they found ways to reduce quantization error by roughly half during fine-tuning and even improve performance when pre-training from scratch with three bit weights. That’s a tangible improvement for production quality.

Tom: It's a principled correction based on curvature information, which gives us more control over the training process than just using a standard trick.

Jane: Exactly, and this method bridges the gap between what we want—a highly accurate quantized model—and what we can actually achieve with current training techniques.

Lu: They’ve moved beyond just patching the straight-through estimator by grounding it in optimization theory, showing strong convergence to that best possible state.

Meng: If this translates into a reliable way to get better accuracy for less quantization bit precision, that’s something we need to see deployed quickly on our production models.

Tom: It’s a solid piece of work because it shows how adding curvature awareness can make a big difference in the final numbers you ship.

Jane: And it opens up a new way to look at quantization as more than just a simple loss minimization problem.

Lu: We'll keep digging into how this theoretical framework connects to the practical implementation details they’ve laid out for the coupled and decoupled corrections.

Soroush Tabesh, Mher Safaryan, Andrei Panferov, Alexandra Volkova, Dan Alistarh

cs.LG

Submitted: 2025-10-21

Updated: 2026-06-18

Comments: Accepted at MLSys 2026 (Oral). To appear in Proceedings of Machine Learning and Systems 8

Journal ref: Proceedings of Machine Learning and Systems 8 (MLSys 2026), pp. 626-638

Code: https://github.com/IST-DASLab/CAGE

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 80/100

The gist: CAGE introduces Curvature-Aware Gradient Estimation for Accurate Quantization-Aware Training to address accuracy gaps between low-bit quantized models and native training The CAGE method provides a…

Key concepts

Quantization-Aware Training (QAT)
QAT is a training technique where models are trained to be accurate after being quantized to lower bit rates. The goal is to find model parameters that perform well even when their weights and activations are represented using fewer bits, bridging the gap between native training and low-bit deployment.
Straight-Through Estimator (STE)
The STE is a standard method used in QAT where the gradient of the quantization operator is approximated by the identity function during backpropagation. This bypasses the complex, non-differentiable nature of quantization, allowing gradients to flow through quantized parameters.
Pareto-Optimal Solutions
This concept defines solutions that simultaneously minimize task loss and satisfy quantization constraints. CAGE uses this view to guide training, ensuring that improvements in one objective do not negatively impact the other, leading to a principled correction term for better quantization performance.

Terminology

Summary

CAGE introduces Curvature-Aware Gradient Estimation for Accurate Quantization-Aware Training to address accuracy gaps between low-bit quantized models and native training The CAGE method provides a principled way to augment the standard straight-through estimator with a curvature-aware correction term derived from a multi-objective view of QAT

Pruning redundant reasoning tokens

The paper introduces CAGE as a new QAT method that augments the straight-through estimator (STE) gradient with a curvature-aware correction designed to counteract the loss increase induced by quantization CAGE is derived from a multi-objective view of QAT that balances loss minimization with the quantization constraints, yielding a principled correction term that depends on local curvature information On the theoretical side, the authors introduce Pareto-optimal solutions for quantized optimization and establish that CAGE yields strong convergence guarantees in the smooth non-convex setting

How the training signal is built

The core of CAGE involves reformulating QAT as a multi-objective optimization problem, aiming to simultaneously minimize task loss and satisfy quantization constraints The authors define a condition for Pareto-optimality where any improvement in one component necessarily leads to a degradation in the other This condition inspires CAGE, which provides both theoretical guarantees and high accuracy and efficiency in practice The method is implemented as optimizer-agnostic but leverages Adam statistics for a highly-efficient implementation For stateful optimizers like Adam, two correction variants are proposed: the coupled correction and the decoupled correction The coupled variant introduces a diagonally preconditioned curvature-aware correction term where curvature is approximated using Adam statistics already maintained within the optimizer states

Where the gains come from

The effectiveness of CAGE is validated through extensive experiments, including synthetic experiments and post-training fine-tuning of standard models For QAT fine-tuning, CAGE roughly halves the quantization error relative to the prior best method In pre-training experiments, CAGE-trained models with 3-bits weights and activations (W3A3) offer lower loss than 4-bit (W4A4) models trained with the prior best method The approach is optimizer-agnostic, leading to consistent gains across several optimizers including AdamW, Muon, and Shampoo Furthermore, scaling law analysis confirms that CAGE improves the effective capacity of models compared to existing QAT methods

Final summary and discussion

CAGE is a novel method for quantization-aware training designed to mitigate accuracy degradation inherent in low-bit quantization by augmenting the gradient with a principled, curvature-aware correction term The key theoretical contribution is the definition of Paretooptimal solutions for quantized optimization and strong ergodic convergence guarantees to a Pareto-optimal point in the smooth non-convex setting Empirical validation on both pre-training and post-training QAT demonstrates its effectiveness across various model sizes and precisions The method introduces negligible computational relative to the QAT baseline, adding only one extra quantization call and a small number of elementwise operations CAGE consistently improves upon strong baselines, such as QuEST, by significantly reducing stationary error across all model sizes The paper concludes that incorporating local curvature information and regularization are useful for QAT performance

References

Ajanthan, T., Dokania, P. p. 9

Where the paper says it

REFERENCES Ajanthan, T. , Dokania, P.

K., Hartley, R. p. 9
Where the paper says it

, Dokania, P. , Hartley, R.

p. 9
Where the paper says it

, Dokania, P. , Hartley, R.

, and Torr, P. p. 9
Where the paper says it

, Hartley, R. , and Torr, P.

H. S., Proximal mean-field for neural network quantization Ajanthan, T. p. 9
Where the paper says it

, and Torr, P. Proximal mean-field for neural network quantization.

, Gupta, K. p. 1
Where the paper says it

Our results demonstrate that CAGE leads to state-of-theart results across all scenarios: for fine-tuning QAT, it roughly halves the quantization error relative to the prior best method, whereas, in pre-training experiments, CAGE-trained models with 3-bits weights and activations (W3A3) offer lower loss than 4-bit (W4A4) models trained with the prior best method (QuEST (Panferov et al. • Moreover, we show experimentally that the CAGE approach is optimizer-agnostic, as it leads to consistent gains across several optimizers, including AdamW (Loshchilov & Hutter, 2019), Muon (Jordan, 2024) and Shampoo (Gupta et al.

, Torr, P. H. S., Hartley, R., and Dokania, P. p. 9
Where the paper says it

REFERENCES Ajanthan, T. , Dokania, P.

K., Mirror Descent View for Neural Network Quantization Bai, Y. p. 9
Where the paper says it

, and Torr, P. Proximal mean-field for neural network quantization.

, Wang, Y.-X., and Liberty, E., Proxquant: Quantized neural networks via proximal operators Bengio, Y. p. 1
Where the paper says it

In addition, ProxQuant (Bai et al. , 2018) reformulates QAT as a regularized learning problem solved via proximal gradient methods.

, Leonard, N., and Courville, A., Estimating or propagating gradients through stochastic neurons for conditional computation Bengio, Y. p. 1
Where the paper says it

2 PRIOR WORK The central challenge in QAT is the non-differentiable quantization function, with zero gradients almost everywhere. The standard is the STE, initially suggested by Hinton (2012) and formalized by Bengio et al.

, Leonard, N., and Courville, A., Estimating or propagating gradients through stochastic neurons for conditional computation Bhalgat, Y. p. 3
Where the paper says it

CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training as x − Q(x) = ∇ϕ(x) for some smooth regularizer ϕ(x). Notably, our approach naturally avoids the need to handle the (non-existent) Jacobian JQ(x) of the quantization operator and bypasses the computation of a proximal step for the explicit “quadratic” regularizer λ 2 ∥x − Q(x)∥ 2 2 , which would be infeasible for the dynamic quantization used in our experiments (Bai et al.

, Lee, J., Nagel, M. p. 0
Where the paper says it

, 2019; Bhalgat et al. , 2020; Nagel et al.

, Blankevoort, T., and Kwak, N. Lsq+: Improving low-bit quantization through learnable offsets and better initialization Chen, C. p. 1
Where the paper says it

Thus, substantial research has focused on STE refinements through learnable quantization parameters (Esser et al. , 2020), improving the accuracy of the gradient itself (Le et al.

, Shen, L., Huang, H., and Liu, W. p. 8
Where the paper says it

OLMo2 models use no biases, employ rotary positional embeddings (Su et al. , 2024), RMSNorm (Zhang & Sennrich, 2019), and reordered pre-normalization (Liu et al.

p. 8
Where the paper says it

OLMo2 models use no biases, employ rotary positional embeddings (Su et al. , 2024), RMSNorm (Zhang & Sennrich, 2019), and reordered pre-normalization (Liu et al.

, Quantized Adam with Error Feedback Chen, C. p. 2
Where the paper says it

It is easy to observe that SGDbased training with STE over the iterate xt is equivalent to an instance of error-feedback (Chen et al. , 2021b) with quantized parameters wt = Q(xt) and quantization error et = xt − Q(xt).

, Shen, L., Huang, H., and Liu, W., Quantized adam with error feedback Darvish Rouhani, B. p. 6
Where the paper says it

Figure 1 shows that, across all condition numbers, CAGE reduces error in a statistically-significant way, while Figure 2 shows that CAGE finds a quantized solution that is strictly closer to the optimum. 2 Quantization-Aware Fine-tuning Next, we evaluate the suitability of CAGE in a more realistic setting, where it is used to perform QAT over the MXFP4 4-bit floating format (Darvish Rouhani et al.

, Garegrat, N., Savell, T., More, A., Han, K.-N., Zhao Ritchie amd Hall M. OCP Microscaling (MX) Specification Dettmers, T. p. 5
Where the paper says it

Unless otherwise stated, our experiments use a standard row-wise matrix quantizer, e. (Dettmers et al.

, Pagnoni, A., Holtzman, A., and Zettlemoyer L. Qlora: Efficient finetuning of quantized llms Diao, S. p. 0
Where the paper says it

On the theoretical side, we introduce the notion of Pareto-optimal solutions for quantized optimization, and establish that CAGE yields strong convergence guarantees in the smooth non-convex setting. In terms of implementation, our approach is optimizer-agnostic, but we provide a highly-efficient implementation that leverages Adam statistics.

, Yang, Y., Fu, Y., Dong, X., Su D., Kliegl M., Chen Z., Belcak P., Suhara Y., Yin H. Climb: Clustering-based iterative data mixture bootstrapping for language model pre-training Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan F. The llama 3 herd of models Egiazarian V. p. 0
Where the paper says it

In terms of implementation, our approach is optimizer-agnostic, but we provide a highly-efficient implementation that leverages Adam statistics. CAGE significantly improves upon the prior state-of-the-art methods in terms of accuracy, for similar computational cost: for QAT fine-tuning, it halves the compression accuracy loss relative to the prior best method, while for QAT pre-training of Llama models, its accuracy for 3-bit weights-and-activations (W3A3) matches the accuracy achieved at 4-bits (W4A4) with the prior best method.

, Castro R. L., Kuznedelev D., Panferov A. p. 0
Where the paper says it

CAGE: CURVATURE-AWARE GRADIENT ESTIMATION FOR ACCURATE QUANTIZATION-AWARE TRAINING Soroush Tabesh * 1 Mher Safaryan * 1 Andrei Panferov 1 Alexandra Volkova 1 Dan Alistarh 1 2 ABSTRACT Despite significant work on low-bit quantization-aware training (QAT), there is still an accuracy gap between such techniques and native training. To address this, we introduce CAGE (Curvature-Aware Gradient Estimation), a new QAT method that augments the straight-through estimator (STE) gradient with a curvature-aware correction designed to counteract the loss increase induced by quantization.

, Kurtic E., Pandit S., Marques A., Kurtz M., Ashkboos S., Hoefler T. Bridging the gap between promise and performance for microscaling fp4 quantization Esser, S. p. 1
Where the paper says it

In summary, we show that incorporating curvature-aware gradient correction via CAGE is a promising step toward bridging the remaining performance gap between low-bit quantized models and their full-precision counterparts. 2 PRIOR WORK The central challenge in QAT is the non-differentiable quantization function, with zero gradients almost everywhere.

K., McKinstry J. L., Bablani D., Appuswamy R., and Modha D. Learned step size quantization Frantar, E. p. 2
Where the paper says it

Let x ∈ R d be the current state of parameters. If we update towards −∇f(x) (or any other direction that aligns with it) with small enough step-size, then we would minimize the loss.

, Ashkboos S., Hoefler T., and Alistarh D. p. 0
Where the paper says it

CAGE: CURVATURE-AWARE GRADIENT ESTIMATION FOR ACCURATE QUANTIZATION-AWARE TRAINING Soroush Tabesh * 1 Mher Safaryan * 1 Andrei Panferov 1 Alexandra Volkova 1 Dan Alistarh 1 2 ABSTRACT Despite significant work on low-bit quantization-aware training (QAT), there is still an accuracy gap between such techniques and native training. To address this, we introduce CAGE (Curvature-Aware Gradient Estimation), a new QAT method that augments the straight-through estimator (STE) gradient with a curvature-aware correction designed to counteract the loss increase induced by quantization.

Gptq: Accurate post-training quantization for generative pre-trained transformers Gemma, T. p. 1
Where the paper says it

• We validate the effectiveness of CAGE through extensive experiments, including synthetic experiments meant to validate the theory, post-training fine-tuning of standard models, and pre-training of Llama-style models of up to 800M parameters from scratch. Our results demonstrate that CAGE leads to state-of-theart results across all scenarios: for fine-tuning QAT, it roughly halves the quantization error relative to the prior best method, whereas, in pre-training experiments, CAGE-trained models with 3-bits weights and activations (W3A3) offer lower loss than 4-bit (W4A4) models trained with the prior best method (QuEST (Panferov et al.

, Mesnard T., Hardin C., Dadashi R., Bhupatiraju S., Pathak S., Sifre L., Riviere M., Kale M. S. Gemma: Open models based on gemini research and technology Gupta, V. p. 0
Where the paper says it

com/IST-DASLab/CAGE. 1 INTRODUCTION Quantization has emerged as a standard technique for improving the computational efficiency of large language model (LLM) deployments, as prominent open-source models such as Llama, Gemma, and GPT-OSS are released in compressed formats (Touvron et al.

, Koren T., and Singer Y. Shampoo: Preconditioned stochastic tensor optimization Hinton, G. p. 4
Where the paper says it

Assumption 2 For all iterates t, the stochastic gradient ∇e f(xt) is unbiased, namely E[∇e f(xt)] = ∇f(xt), and the variance is bounded E[∥∇e f(xt) − ∇f(xt)∥ 2 2 ] ≤ σ 2 for some constant σ 2 ≥ 0. Both assumptions are quite standard in the optimization literature and, in some sense, minimal to get meaningful convergence guarantees.

, Neural networks for machine learning, lecture 9c. p. 9
Where the paper says it

, and Torr, P. Proximal mean-field for neural network quantization.

Coursera Hoffmann, J. p. 8
Where the paper says it

, 2024; Panferov et al. , 2025a; Hoffmann et al.

, Borgeaud S., Mensch A., Buchatskaya E., Cai T., Rutherford E., de Las Casas D., Hendricks L. A., Welbl J., Clark A., Hennigan T., Noland E., Millican K. van den Driessche G. Damoc B. Guy A. Osindero S. Simonyan K. Elsen E. Rae J W Vinyals O Sifre L, Training compute-optimal large language models, 2022 Hou, L. p. 0
Where the paper says it

com/IST-DASLab/CAGE. 1 INTRODUCTION Quantization has emerged as a standard technique for improving the computational efficiency of large language model (LLM) deployments, as prominent open-source models such as Llama, Gemma, and GPT-OSS are released in compressed formats (Touvron et al.

, Zhang R., and Kwok J. T., Analysis of Quantized Models Hsieh C. p. 9
Where the paper says it

CAGE consistently outperformed strong baselines that employ modern outliermitigation techniques. Specifically, scaling law analysis confirms that CAGE improves the effective capacity of models compared to existing QAT methods.

-P., Sun S., Kriman S., Acharya S., Rekesh D., Jia F., Zhang Y., and Ginsburg B. Ruler: What’s the real context size of your long-context language models Jin L. p. 3
Where the paper says it

We will provide theoretical convergence guarantees to a Pareto-optimal point in the smooth non-convex setting. However, for stateful optimizers such as Adam, these two corrections produce conceptually different schemes, which we discuss below (subsuming bias correction terms into the learning rate α): Quantization error and mini-batch gradient et = xt − Q(xt) gt = ∇e f(xt) + λtet, (coupled correction) Optimizer states mt = β1mt−1 + (1 − β1)gt, vt = β2vt−1 + (1 − β2)g 2 t , Optimizer update ∆t = mt/( √ vt + ε) + λtet, (decoupled correction) Model update xt+1 = xt − α∆t, To compare these two correction variants in the context of the Adam optimizer, note that the coupled correction term λtet becomes (1 − β1)λtet/( √ vt + ε) after being processed by the optimizer states.

, Ma J., Liu Z., Gromov A., Defazio A., and Xiao L. Parq: Piecewise-affine regularized quantization Jordan K. p. 1
Where the paper says it

Instead of relying solely on STE, ProxQuant applies a prox operator between stochastic gradient steps to encourage quantization. Similar to ProxQuant, PARQ uses convex and piecewise-affine regularization to induce weight quantization with the difference that gradient updates are added to the full-precision parameters before applying the proximal operator (Jin et al.

Muon: An optimizer for the hidden layers of neural networks Kim J. p. 1
Where the paper says it

Our results demonstrate that CAGE leads to state-of-theart results across all scenarios: for fine-tuning QAT, it roughly halves the quantization error relative to the prior best method, whereas, in pre-training experiments, CAGE-trained models with 3-bits weights and activations (W3A3) offer lower loss than 4-bit (W4A4) models trained with the prior best method (QuEST (Panferov et al. • Moreover, we show experimentally that the CAGE approach is optimizer-agnostic, as it leads to consistent gains across several optimizers, including AdamW (Loshchilov & Hutter, 2019), Muon (Jordan, 2024) and Shampoo (Gupta et al.

, Yoo K., and Kwak N., Position-based scaled gradient for model quantization and pruning Krishnamoorthi R. p. 1
Where the paper says it

Additionally, these scalings are adjusted by a global scaling factor obtained from approximate Hessian information. A related method employs position-based scaled gradients, where the gradient estimator is adjusted only by multiplying the quantization error with a small positive margin (Kim et al.

, Quantizing deep convolutional networks for efficient inference: A whitepaper Kumar T. p. 3
Where the paper says it

In addition, as described in their paper, LOTION requires a fullprecision forward pass, and the final model is a weight-only quantized model. By contrast, CAGE training is done with weights and activations quantized, where it can also benefit from high-performance low-precision GEMM kernels for more efficient training and inference.

, Ankner Z., Spector B. F., Bordelon B., Muennighoff N., Paul M., Pehlevan C., Re C., and Raghu-nathan A. Scaling laws for precision Kwun M. p. 8
Where the paper says it

Since improvements introduced by CAGE do not depend on the choice of optimizer, we conclude that it can be regarded as an optimizer-agnostic technique. Precision Scaling Law.

, Morwani D., Su C. H., Gil S., Anand N., and Kakade S. Lotion: Smoothing the optimization landscape for quantized training Lambert N. p. 6
Where the paper says it

We fine-tune on Tulu-SFT (Lambert et al. , 2024) with QAT (master weights in FP32, the forward pass has both weights and activations quantized to MXFP4), following the recently-proposed QAT recipe of (Egiazarian et al.

, Morrison J., Pyatkin V., Huang S., Ivison H., Brahman F., Miranda L. J. V., Liu A. Dziri N., Lyu S., Gu Y. Malik S. Graf V. Hwang J D. Yang J Bras R L Tafjord O Wilhelm C Soldaini L Smith N A Wang Y Dasigi P Hajishirzi H Tulu 3: Pushing frontiers in open language model post-training Le H. p. 0
Where the paper says it

, 2024; OpenAI, 2025). Yet, the vast majority of open research in this area has concentrated on post-training quantization (PTQ), where a fully trained model’s weights are quantized by applying numerical algorithms over a small calibration dataset (Frantar et al.

, Høier R. K., Lin C.-T., and Zach C. AdaSTE: An Adaptive Straight-Through Estimator to Train Binary Neural Networks Lee J., Kim D., and Ham B. Network quantization with element-wise gradient scaling Li H. p. 1
Where the paper says it

, 2019) represent each parameter of the model as weighted average of quantized values, and minimizes the loss with respect to these probabilistic weights. (2021a) proposed the element-wise gradient scaling (EWGS) approach, in which the scaling of each gradient element depends on both its sign and the corresponding quantization error.

, De S., Xu Z., Studer C., Samet H., and Goldstein T. Training Quantized Nets: A Deeper Understanding Lin T. p. 2
Where the paper says it

Prior work on convergence guarantees for the problem described in Equation (1) in the non-convex setting attempts to find a quantized point Q(x ∗ ) that is a stationary point for the loss f, namely ∇f(Q(x ∗ )) = 0. However, as quantization is not an invertible transformation, this goal cannot be achieved in general, and current convergence guarantees have non-vanishing terms in their rate, proportional to quantization error (Lin et al.

, Stich S. U., Barba L., Dmitriev D., and Jaggi M. Dynamic model pruning with feedback Liu L. p. 3
Where the paper says it

CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training as x − Q(x) = ∇ϕ(x) for some smooth regularizer ϕ(x). Notably, our approach naturally avoids the need to handle the (non-existent) Jacobian JQ(x) of the quantization operator and bypasses the computation of a proximal step for the explicit “quadratic” regularizer λ 2 ∥x − Q(x)∥ 2 2 , which would be infeasible for the dynamic quantization used in our experiments (Bai et al.

, Dong C., Liu X., Yu B., and Gao J. Bridging discrete and backpropagation: Straight-through and beyond Liu Z., Hu H., Lin Y., Yao Z., Xie Z., Wei Y., Ning J. Swin transformer v2: Scaling up capacity and resolution Oshere S. p. 9
Where the paper says it

CAGE consistently outperformed strong baselines that employ modern outliermitigation techniques. Specifically, scaling law analysis confirms that CAGE improves the effective capacity of models compared to existing QAT methods.

Qi Y., and Xin J. Understanding straight-through estimator in training activation quantized neural nets Su J. p. 0
Where the paper says it

CAGE: CURVATURE-AWARE GRADIENT ESTIMATION FOR ACCURATE QUANTIZATION-AWARE TRAINING Soroush Tabesh * 1 Mher Safaryan * 1 Andrei Panferov 1 Alexandra Volkova 1 Dan Alistarh 1 2 ABSTRACT Despite significant work on low-bit quantization-aware training (QAT), there is still an accuracy gap between such techniques and native training. To address this, we introduce CAGE (Curvature-Aware Gradient Estimation), a new QAT method that augments the straight-through estimator (STE) gradient with a curvature-aware correction designed to counteract the loss increase induced by quantization.

, Ahmed M., Lu Y., Pan S., Bo W., and Liu Y. p. 8
Where the paper says it

OLMo2 models use no biases, employ rotary positional embeddings (Su et al. , 2024), RMSNorm (Zhang & Sennrich, 2019), and reordered pre-normalization (Liu et al.

Roformer: Enhanced transformer with rotary position embedding Touvron H. p. 0
Where the paper says it

com/IST-DASLab/CAGE. 1 INTRODUCTION Quantization has emerged as a standard technique for improving the computational efficiency of large language model (LLM) deployments, as prominent open-source models such as Llama, Gemma, and GPT-OSS are released in compressed formats (Touvron et al.

, Martin L., Stone K., Albert P., Almahairi A., Babaei Y., Bashlykov N., Batra S. Gemma: Open models based on gemini research and technology Vyas N. p. 0
Where the paper says it

com/IST-DASLab/CAGE. 1 INTRODUCTION Quantization has emerged as a standard technique for improving the computational efficiency of large language model (LLM) deployments, as prominent open-source models such as Llama, Gemma, and GPT-OSS are released in compressed formats (Touvron et al.

, Morwani D., Zhao R., Shapira I, Brandfonbrener D, Janson L., and Kakade S. SOAP: Improving and stabilizing shampoo using adam for language modeling Wu X. p. 8
Where the paper says it

To verify that gains produced by CAGE are independent of the optimizer update, we conduct a series of additional experiments using alternative optimizers to AdamW: Muon (Jordan, 2024), Shampoo (Gupta et al. , 2018), and SOAP (Vyas et al.

-M., Zheng D., Liu Z., and Zheng W.-S. Estimator meets equilibrium perspective: A rectified straight through estimator for binary neural networks training Yin P. p. 0
Where the paper says it

CAGE: CURVATURE-AWARE GRADIENT ESTIMATION FOR ACCURATE QUANTIZATION-AWARE TRAINING Soroush Tabesh * 1 Mher Safaryan * 1 Andrei Panferov 1 Alexandra Volkova 1 Dan Alistarh 1 2 ABSTRACT Despite significant work on low-bit quantization-aware training (QAT), there is still an accuracy gap between such techniques and native training. To address this, we introduce CAGE (Curvature-Aware Gradient Estimation), a new QAT method that augments the straight-through estimator (STE) gradient with a curvature-aware correction designed to counteract the loss increase induced by quantization.

, Lyu J., Zhang S., Osher S., Qi Y., and Xin J. Understanding straight-through estimator in training activation quantized neural nets Zhang B. p. 0
Where the paper says it

CAGE: CURVATURE-AWARE GRADIENT ESTIMATION FOR ACCURATE QUANTIZATION-AWARE TRAINING Soroush Tabesh * 1 Mher Safaryan * 1 Andrei Panferov 1 Alexandra Volkova 1 Dan Alistarh 1 2 ABSTRACT Despite significant work on low-bit quantization-aware training (QAT), there is still an accuracy gap between such techniques and native training. To address this, we introduce CAGE (Curvature-Aware Gradient Estimation), a new QAT method that augments the straight-through estimator (STE) gradient with a curvature-aware correction designed to counteract the loss increase induced by quantization.

and Sennrich R. p. 8
Where the paper says it

OLMo2 models use no biases, employ rotary positional embeddings (Su et al. , 2024), RMSNorm (Zhang & Sennrich, 2019), and reordered pre-normalization (Liu et al.

Root mean square layer normalization Yuan, Q. p. 3
Where the paper says it

Here, the exact Hessian matrix is replaced by its Gauss–Newton approximation. If we further apply the empirical Fisher approximation and approximate the diagonal entries of the Gauss–Newton matrix using Adam statistics, this regularization takes similar form as our coupled correction variant (except the square root).

, Quantizing deep convolutional networks for efficient inference: A whitepaper

--- Page 3 ---

CAGE: CURVATURE-AWARE GRADIENT ESTIMATION FOR ACCURATE QUANTIZATION-AWARE TRAINING

3 CAGE: CURVATURE-AWARE GRADIENT ESTIMATION

3.1 Problem Description and Motivation

Intuitively, the key question in quantization-aware training (QAT) is to construct a gradient estimator that updates the parameters towards an “optimal” quantized model, i.e., a constrained model which minimizes the loss The optimization problem underlying QAT can be described by the following constrained optimization min x∈Q f(x), (1) where Q = Q(x) is the constraint set of quantized parameters and Q: R d → R is the quantization operator The standard is the STE, which bypasses the quantization operator during the backward pass by approximating its gradient with the identity Error feedback reformulation shows that SGD-based training with STE over iterate xt is equivalent to an instance of error-feedback with quantized parameters wt = Q(xt) and quantization error et = xt − Q(xt) The right-hand-side formulation says that STE is equivalent to a process where the quantization error et+1 due to weight quantization at a given step (wt − gt) − wt+1 is fed into the gradients at the next step

3.2 Theoretical Analysis

The authors propose viewing the QAT problem from a multi-objective perspective, where one objective is to minimize task loss f(x) and the second objective is to satisfy the quantization constraint x ∈ Q They define solutions as Pareto optimal points that balance these two conditions, leading to the condition ∇λPf(Q(x∗)):= ∇f(x∗) + λ(x∗ − Q(x∗)) = 0 (2) Under Assumption 3, the quantization error can be expressed as x − Q(x) = ∇ϕ(x) for some smooth regularizer ϕ(x) The convergence result shows strong ergodic convergence to a Pareto-optimal state in the smooth non-convex setting The final bound on the expected norm of the Pareto-gradient is E∥∇λPf(Q(ˆx))∥ squared ≤ 1/√T √2(A1 + A2 + A3) max 1, √L T

3.3 Practical Implementation

The authors propose two ways to incorporate the quantization error: the coupled correction and the decoupled correction The coupled variant adds the quantization error to the current gradient before it is passed to the optimizer, resulting in a diagonally preconditioned, curvature-aware correction term The decoupled variant applies the correction at unit scale after the optimizer step, deliberately avoiding preconditioning path for practical stability In practice, CAGE is implemented on top of AdamW using a silence period ratio s and regularization parameter λ to control when the correction is activated

G FORMAL JUSTIFICATION FOR λ SCHEDULER

The rationale for the silence period is that early in training, quantization residuals behave like approximately mean-zero, bounded perturbations that cancel in aggregate but have T-scale fluctuations Applying the correction too early can over-constrain the dynamics and inject noise-like forcing terms into the dynamics Near convergence, however, residuals become coherent and effectively push parameters toward the quantized support, which is precisely when enforcing Pareto-stationarity becomes beneficial

G.6 Dynamic quantizers where L ≈ 1

The authors define dynamic quantizers by the integer code rather than the reconstructed value to define visits based on code stability This approach ensures that the residual is approximately mean-zero during exploration phase, which is crucial for applying martingale and concentration arguments The analysis shows that early residuals behave like bounded, approximately mean-zero perturbations that cancel in aggregate but have T-scale fluctuations This leads to a cancellation bound where the early residuals behave like bounded, approximately mean-zero perturbations

G.3 Cancellation bound

The concentration across visits implies that for any unit vector u in S d−1 and any ε > 0, the expected value of the block sum of early residuals is OP(BLmax√K) via a standard ε-net argument The analysis shows that early residuals behave like bounded, approximately mean-zero perturbations

G.5 Why λt should increase late

Near convergence, the iterate typically oscillates among a small set of adjacent cells for long periods, so the ”reset” condition weakens In this regime, residuals become coherent and effectively push xt toward the quantized support, which is precisely when enforcing Pareto-stationarity becomes beneficial Therefore, λt should be larger late in training

The authors define dynamic quantizers by the integer code rather than the reconstructed value to define visits based on code stability This approach ensures that the residual is approximately mean-zero during exploration phase, which is crucial for applying martingale and concentration arguments The analysis shows that early residuals behave like bounded, approximately mean-zero perturbations This leads to a cancellation bound where the early residuals behave like bounded, approximately mean-zero perturbations

**G.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Curvature-Aware Gradient Estimation for Accurate Quantization-Aware Training allows AI systems to achieve state-of-the-art accuracy in low-bit quantization by augmenting standard straight-through estimator gradients with a principled correction term derived from Pareto optimality, which explicitly counteracts the increase in loss induced by the quantization step.

  2. The system can be trained on models up to 800M parameters from scratch using 3-bit weights and activations (W3A3) and achieve accuracy matching that of a prior state-of-the-art method trained at 4 bits (W4A4).

  3. CAGE provides strong ergodic convergence guarantees to a Pareto-optimal point in the smooth non-convex setting, addressing the lack of guarantees that is a primary limitation of prior work in this area.

  4. The optimizer agnostic implementation allows the system to leverage existing optimizer statistics, such as Adam's second-moment statistics, by employing either coupled or decoupled corrections to stabilize training.

  5. The system can be trained across various optimizers including AdamW, Muon, and Shampoo without requiring specific hyperparameter tuning for the correction term due to the primary gains originate from the shared correction direction rather than the specific scaling.

  6. CAGE improves parameter efficiency by more than 10 percent at 4-bit precision and 20 percent at 2-bit precision relative to prior state-of-the-art methods, indicating improved effective capacity for a given parameter budget.

  7. The system demonstrates superior long context performance compared to baselines like QuEST and GPTQ when fine-tuned with CAGE on Llama 3.1-8B models under W4A16 quantization, narrowing the performance gap between quantized and full precision models.

Abstract

Despite significant work on low-bit quantization-aware training (QAT), there is still an accuracy gap between such techniques and native training. To address this, we introduce CAGE (Curvature-Aware Gradient Estimation), a new QAT method that augments the straight-through estimator (STE) gradient with a curvature-aware correction designed to counteract the loss increase induced by quantization. CAGE is derived from a multi-objective view of QAT that balances loss minimization with the quantization constraints, yielding a principled correction term that depends on local curvature information. On the theoretical side, we introduce the notion of Pareto-optimal solutions for quantized optimization, and establish that CAGE yields strong convergence guarantees in the smooth non-convex setting. In terms of implementation, our approach is optimizer-agnostic, but we provide a highly-efficient implementation that leverages Adam statistics. CAGE significantly improves upon the prior state-of-the-art methods in terms of accuracy, for similar computational cost: for QAT fine-tuning, it halves the compression accuracy loss relative to the prior best method, while for QAT pre-training of Llama models, its accuracy for 3-bit weights-and-activations (W3A3) matches the accuracy achieved at 4-bits (W4A4) with the prior best method. The official implementation can be found over https://github.com/IST-DASLab/CAGE.

Sources

Related papers