CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training".
Jane: Here's the gist of the paper. CAGE introduces Curvature-Aware Gradient Estimation for Accurate Quantization-Aware Training to address accuracy gaps between low-bit quantized models and native training The CAGE method provides a principled way to augment the standard straight-through estimator with a curvature-aware correction term derived from a multi-objective view of QAT Pruning redundant reasoning tokens The paper introduces CAGE as a new QAT method that augments the straight-through estimator (STE) gradient with a curvature-aware correction designed to counteract the loss increase induced by quantization CAGE is derived from a multi-objective view of QAT that balances loss minimization with the quantization constraints, yielding a principled correction term that depends on local curvature information On the theoretical side, the authors introduce Pareto-optimal solutions for quantized optimization and establish that CAGE yields strong convergence guarantees in the smooth non-convex setting How the training signal is built The core of CAGE involves reformulating QAT as a multi-objective optimization problem, aiming to simultaneously minimize task loss and satisfy quantization constraints The authors define a condition for Pareto-optimality where any improvement in one component necessarily leads to a degradation in the other This condition inspires CAGE, which provides both theoretical guarantees and high accuracy and efficiency in practice The method is implemented as optimizer-agnostic but leverages Adam statistics for a highly-efficient implementation For stateful optimizers like Adam, two correction variants are proposed: the coupled correction and the decoupled correction The coupled variant introduces a diagonally preconditioned curvature-aware correction term where curvature is approximated using Adam statistics already maintained within the optimizer states Where the gains come from The effectiveness of CAGE is validated through extensive experiments, including synthetic experiments and post-training fine-tuning of standard models For QAT fine-tuning, CAGE roughly halves the quantization error relative to the prior best method In pre-training experiments, CAGE-trained models with 3-bits weights and activations (W3A3) offer lower loss than 4-bit (W4A4) models trained with the prior best method The approach is optimizer-agnostic, leading to consistent gains across several optimizers including AdamW, Muon, and Shampoo Furthermore, scaling law analysis confirms that CAGE improves the effective capacity of models compared to existing QAT methods Final summary and discussion CAGE is a novel method for quantization-aware training designed to mitigate accuracy degradation inherent in low-bit quantization by augmenting the gradient with a principled, curvature-aware correction term The key theoretical contribution is the definition of Paretooptimal solutions for quantized optimization and strong ergodic convergence guarantees to a Pareto-optimal point in the smooth non-convex setting Empirical validation on both pre-training and post-training QAT demonstrates its effectiveness across various model sizes and precisions The method introduces negligible computational relative to the QAT baseline, adding only one extra quantization call and a small number of elementwise operations CAGE consistently improves upon strong baselines, such as QuEST, by significantly reducing stationary error across all model sizes The paper concludes that incorporating local curvature information and regularization are useful for QAT performance References Ajanthan, T., Dokania, P.
Tom: K., Hartley, R., and Torr, P.
Jane: H.
Tom: S., Proximal mean-field for neural network quantization Ajanthan, T., Gupta, K., Torr, P.
Paper summary: Tom: So, we're wrapping up this look at CAGE, which stands for Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training, and what it actually means for how we train these big models.
Jane: Yeah, CAGE is essentially taking the standard way we calculate gradients during quantization—the straight-through estimator—and adding a correction term that looks at the local curvature of the loss landscape.
Lu: The core idea here is moving away from just guessing what happens when you quantize parameters to actually balancing two things: minimizing how bad your model’s task loss is, and making sure the weights stay within the allowed quantized set.
Meng: So, they use a multi-objective view to define these Pareto optimal solutions where neither objective can be improved without hurting the other. That's a pretty principled way to think about finding that sweet spot between accuracy and bit precision.
Jane: Exactly, and theoretically, they prove that this approach leads to strong convergence guarantees in the smooth non-convex setting, which means if you run the training long enough, it’s highly likely to settle at that best point they defined.
Tom: But what does this mean for a developer? It means when you fine-tune a model after training, CAGE can roughly halve your quantization error compared to the previous best method. That's a big number for deployment stability.
Lu: And in the pre-training stage, they found that models trained with three bits of weights and activations using CAGE actually perform better than four bit models trained with older methods. That suggests it helps build better foundational knowledge from scratch.
Meng: I'm interested in the practical side—how much extra compute does this add? They say it’s negligible, just one extra quantization call and some small element-wise math. If that holds up when we scale to massive models, that’s a win for us.
Jane: It is, and they show it works across different optimizers like AdamW and Shampoo without needing custom tweaks for every single one of those. It’s meant to be relatively easy to drop into an existing pipeline.
Tom: So the main thing here is that CAGE isn't just a tweak; it’s a principled correction based on local curvature information, giving us more control over the quantization process during training.
Lu: It sets up a new theoretical framework for how we handle these constrained optimization problems in neural networks.
Meng: It gives us confidence that we can push lower bit precisions without completely losing accuracy, provided the setup is right.
Jane: So CAGE is trying to bridge that gap between what we want—a highly accurate quantized model—and what we can achieve with current training techniques.
Tom: It’s a solid piece of work that shows how adding curvature awareness can make a big difference in the final numbers you ship.
Jane: And it opens up a new way to look at QAT as more than just a simple loss minimization problem.
Conclusion: Tom: So, we're wrapping up this look at CAGE, which stands for Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training. It’s a new way to handle that gap between low bit quantized models and those trained from scratch.
Jane: Right. The authors are looking at how we calculate gradients during quantization—the straight-through estimator—and they've added a correction term based on the local curvature of the loss landscape. They use a multi-objective view, balancing minimizing task loss with satisfying those specific bit constraints.
Lu: Theoretically, they’re proving there are Pareto optimal solutions for this optimization problem and showing strong convergence guarantees in that smooth non-convex setting. That’s a big piece of work because it gives us mathematical confidence in the training process.
Meng: So, what does this actually mean when we look at deploying models? It means they found ways to reduce quantization error by roughly half during fine-tuning and even improve performance when pre-training from scratch with three bit weights. That’s a tangible improvement for production quality.
Tom: It's a principled correction based on curvature information, which gives us more control over the training process than just using a standard trick.
Jane: Exactly, and this method bridges the gap between what we want—a highly accurate quantized model—and what we can actually achieve with current training techniques.
Lu: They’ve moved beyond just patching the straight-through estimator by grounding it in optimization theory, showing strong convergence to that best possible state.
Meng: If this translates into a reliable way to get better accuracy for less quantization bit precision, that’s something we need to see deployed quickly on our production models.
Tom: It’s a solid piece of work because it shows how adding curvature awareness can make a big difference in the final numbers you ship.
Jane: And it opens up a new way to look at quantization as more than just a simple loss minimization problem.
Lu: We'll keep digging into how this theoretical framework connects to the practical implementation details they’ve laid out for the coupled and decoupled corrections.
Soroush Tabesh, Mher Safaryan, Andrei Panferov, Alexandra Volkova, Dan Alistarh
cs.LG
Submitted: 2025-10-21
Updated: 2026-06-18
Comments: Accepted at MLSys 2026 (Oral). To appear in Proceedings of Machine Learning and Systems 8
Journal ref: Proceedings of Machine Learning and Systems 8 (MLSys 2026), pp. 626-638
Code: https://github.com/IST-DASLab/CAGE
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 80/100
The gist: CAGE introduces Curvature-Aware Gradient Estimation for Accurate Quantization-Aware Training to address accuracy gaps between low-bit quantized models and native training The CAGE method provides a…
Key concepts
- Quantization-Aware Training (QAT)
- QAT is a training technique where models are trained to be accurate after being quantized to lower bit rates. The goal is to find model parameters that perform well even when their weights and activations are represented using fewer bits, bridging the gap between native training and low-bit deployment.
- Straight-Through Estimator (STE)
- The STE is a standard method used in QAT where the gradient of the quantization operator is approximated by the identity function during backpropagation. This bypasses the complex, non-differentiable nature of quantization, allowing gradients to flow through quantized parameters.
- Pareto-Optimal Solutions
- This concept defines solutions that simultaneously minimize task loss and satisfy quantization constraints. CAGE uses this view to guide training, ensuring that improvements in one objective do not negatively impact the other, leading to a principled correction term for better quantization performance.
Terminology
Summary
CAGE introduces Curvature-Aware Gradient Estimation for Accurate Quantization-Aware Training to address accuracy gaps between low-bit quantized models and native training The CAGE method provides a principled way to augment the standard straight-through estimator with a curvature-aware correction term derived from a multi-objective view of QAT
Pruning redundant reasoning tokens
The paper introduces CAGE as a new QAT method that augments the straight-through estimator (STE) gradient with a curvature-aware correction designed to counteract the loss increase induced by quantization CAGE is derived from a multi-objective view of QAT that balances loss minimization with the quantization constraints, yielding a principled correction term that depends on local curvature information On the theoretical side, the authors introduce Pareto-optimal solutions for quantized optimization and establish that CAGE yields strong convergence guarantees in the smooth non-convex setting
How the training signal is built
The core of CAGE involves reformulating QAT as a multi-objective optimization problem, aiming to simultaneously minimize task loss and satisfy quantization constraints The authors define a condition for Pareto-optimality where any improvement in one component necessarily leads to a degradation in the other This condition inspires CAGE, which provides both theoretical guarantees and high accuracy and efficiency in practice The method is implemented as optimizer-agnostic but leverages Adam statistics for a highly-efficient implementation For stateful optimizers like Adam, two correction variants are proposed: the coupled correction and the decoupled correction The coupled variant introduces a diagonally preconditioned curvature-aware correction term where curvature is approximated using Adam statistics already maintained within the optimizer states
Where the gains come from
The effectiveness of CAGE is validated through extensive experiments, including synthetic experiments and post-training fine-tuning of standard models For QAT fine-tuning, CAGE roughly halves the quantization error relative to the prior best method In pre-training experiments, CAGE-trained models with 3-bits weights and activations (W3A3) offer lower loss than 4-bit (W4A4) models trained with the prior best method The approach is optimizer-agnostic, leading to consistent gains across several optimizers including AdamW, Muon, and Shampoo Furthermore, scaling law analysis confirms that CAGE improves the effective capacity of models compared to existing QAT methods
Final summary and discussion
CAGE is a novel method for quantization-aware training designed to mitigate accuracy degradation inherent in low-bit quantization by augmenting the gradient with a principled, curvature-aware correction term The key theoretical contribution is the definition of Paretooptimal solutions for quantized optimization and strong ergodic convergence guarantees to a Pareto-optimal point in the smooth non-convex setting Empirical validation on both pre-training and post-training QAT demonstrates its effectiveness across various model sizes and precisions The method introduces negligible computational relative to the QAT baseline, adding only one extra quantization call and a small number of elementwise operations CAGE consistently improves upon strong baselines, such as QuEST, by significantly reducing stationary error across all model sizes The paper concludes that incorporating local curvature information and regularization are useful for QAT performance
References
Ajanthan, T., Dokania, P. p. 9 REFERENCES
Ajanthan, T. , Dokania, P. , Dokania, P. , Hartley, R. , Dokania, P. , Hartley, R. , Hartley, R. , and Torr, P. , and Torr, P. Proximal mean-field for neural network quantization. Our
results demonstrate that CAGE leads to state-of-theart results across all scenarios: for fine-tuning QAT,
it roughly halves the quantization error relative to
the prior best method, whereas, in pre-training experiments, CAGE-trained models with 3-bits weights
and activations (W3A3) offer lower loss than 4-bit
(W4A4) models trained with the prior best method
(QuEST (Panferov et al. • Moreover, we show experimentally that the CAGE
approach is optimizer-agnostic, as it leads to consistent gains across several optimizers, including
AdamW (Loshchilov & Hutter, 2019), Muon (Jordan,
2024) and Shampoo (Gupta et al. REFERENCES
Ajanthan, T. , Dokania, P. , and Torr, P. Proximal mean-field for neural network quantization. In addition, ProxQuant (Bai et al. , 2018) reformulates QAT
as a regularized learning problem solved via proximal gradient methods. 2 PRIOR WORK
The central challenge in QAT is the non-differentiable quantization function, with zero gradients almost everywhere. The standard is the STE, initially suggested by Hinton
(2012) and formalized by Bengio et al. CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training
as x − Q(x) = ∇ϕ(x) for some smooth regularizer ϕ(x). Notably, our approach naturally avoids the need to handle
the (non-existent) Jacobian JQ(x) of the quantization operator and bypasses the computation of a proximal step for
the explicit “quadratic” regularizer λ
2
∥x − Q(x)∥
2
2
, which
would be infeasible for the dynamic quantization used in
our experiments (Bai et al. , 2019;
Bhalgat et al. , 2020; Nagel et al. Thus, substantial research has focused on STE refinements through learnable quantization parameters (Esser
et al. , 2020), improving the accuracy of the gradient itself (Le et al. OLMo2 models use
no biases, employ rotary positional embeddings (Su et al. ,
2024), RMSNorm (Zhang & Sennrich, 2019), and reordered
pre-normalization (Liu et al. OLMo2 models use
no biases, employ rotary positional embeddings (Su et al. ,
2024), RMSNorm (Zhang & Sennrich, 2019), and reordered
pre-normalization (Liu et al. It is easy to observe that SGDbased training with STE over the iterate xt is equivalent
to an instance of error-feedback (Chen et al. , 2021b) with
quantized parameters wt = Q(xt) and quantization error
et = xt − Q(xt). Figure 1 shows that, across all condition numbers,
CAGE reduces error in a statistically-significant way, while
Figure 2 shows that CAGE finds a quantized solution that is
strictly closer to the optimum. 2 Quantization-Aware Fine-tuning
Next, we evaluate the suitability of CAGE in a more realistic
setting, where it is used to perform QAT over the MXFP4
4-bit floating format (Darvish Rouhani et al. Unless otherwise stated, our
experiments use a standard row-wise matrix quantizer,
e. (Dettmers et al. On the theoretical side, we introduce the notion of Pareto-optimal
solutions for quantized optimization, and establish that CAGE yields strong convergence guarantees in the
smooth non-convex setting. In terms of implementation, our approach is optimizer-agnostic, but we provide
a highly-efficient implementation that leverages Adam statistics. In terms of implementation, our approach is optimizer-agnostic, but we provide
a highly-efficient implementation that leverages Adam statistics. CAGE significantly improves upon the prior
state-of-the-art methods in terms of accuracy, for similar computational cost: for QAT fine-tuning, it halves the
compression accuracy loss relative to the prior best method, while for QAT pre-training of Llama models, its
accuracy for 3-bit weights-and-activations (W3A3) matches the accuracy achieved at 4-bits (W4A4) with the prior
best method. CAGE: CURVATURE-AWARE GRADIENT ESTIMATION FOR ACCURATE
QUANTIZATION-AWARE TRAINING
Soroush Tabesh * 1 Mher Safaryan * 1 Andrei Panferov 1 Alexandra Volkova 1 Dan Alistarh 1 2
ABSTRACT
Despite significant work on low-bit quantization-aware training (QAT), there is still an accuracy gap between such
techniques and native training. To address this, we introduce CAGE (Curvature-Aware Gradient Estimation), a
new QAT method that augments the straight-through estimator (STE) gradient with a curvature-aware correction
designed to counteract the loss increase induced by quantization. In summary, we show that incorporating curvature-aware
gradient correction via CAGE is a promising step toward
bridging the remaining performance gap between low-bit
quantized models and their full-precision counterparts. 2 PRIOR WORK
The central challenge in QAT is the non-differentiable quantization function, with zero gradients almost everywhere. Let x ∈ R
d be the current state of parameters. If we update
towards −∇f(x) (or any other direction that aligns with it)
with small enough step-size, then we would minimize the
loss. CAGE: CURVATURE-AWARE GRADIENT ESTIMATION FOR ACCURATE
QUANTIZATION-AWARE TRAINING
Soroush Tabesh * 1 Mher Safaryan * 1 Andrei Panferov 1 Alexandra Volkova 1 Dan Alistarh 1 2
ABSTRACT
Despite significant work on low-bit quantization-aware training (QAT), there is still an accuracy gap between such
techniques and native training. To address this, we introduce CAGE (Curvature-Aware Gradient Estimation), a
new QAT method that augments the straight-through estimator (STE) gradient with a curvature-aware correction
designed to counteract the loss increase induced by quantization. • We validate the effectiveness of CAGE through extensive experiments, including synthetic experiments
meant to validate the theory, post-training fine-tuning
of standard models, and pre-training of Llama-style
models of up to 800M parameters from scratch. Our
results demonstrate that CAGE leads to state-of-theart results across all scenarios: for fine-tuning QAT,
it roughly halves the quantization error relative to
the prior best method, whereas, in pre-training experiments, CAGE-trained models with 3-bits weights
and activations (W3A3) offer lower loss than 4-bit
(W4A4) models trained with the prior best method
(QuEST (Panferov et al. com/IST-DASLab/CAGE. 1 INTRODUCTION
Quantization has emerged as a standard technique for improving the computational efficiency of large language
model (LLM) deployments, as prominent open-source models such as Llama, Gemma, and GPT-OSS are released in
compressed formats (Touvron et al. Assumption 2 For all iterates t, the stochastic gradient
∇e f(xt) is unbiased, namely E[∇e f(xt)] = ∇f(xt), and
the variance is bounded E[∥∇e f(xt) − ∇f(xt)∥
2
2
] ≤ σ
2
for
some constant σ
2 ≥ 0. Both assumptions are quite standard in the optimization
literature and, in some sense, minimal to get meaningful
convergence guarantees. , and Torr, P. Proximal mean-field for neural network quantization. , 2024;
Panferov et al. , 2025a; Hoffmann et al. com/IST-DASLab/CAGE. 1 INTRODUCTION
Quantization has emerged as a standard technique for improving the computational efficiency of large language
model (LLM) deployments, as prominent open-source models such as Llama, Gemma, and GPT-OSS are released in
compressed formats (Touvron et al. CAGE consistently
outperformed strong baselines that employ modern outliermitigation techniques. Specifically, scaling law analysis
confirms that CAGE improves the effective capacity of models compared to existing QAT methods. We will provide theoretical convergence guarantees to a Pareto-optimal point in the smooth non-convex setting. However, for stateful optimizers such as Adam, these
two corrections produce conceptually different schemes,
which we discuss below (subsuming bias correction terms
into the learning rate α):
Quantization error and mini-batch gradient
et = xt − Q(xt)
gt = ∇e f(xt) + λtet, (coupled correction)
Optimizer states
mt = β1mt−1 + (1 − β1)gt,
vt = β2vt−1 + (1 − β2)g
2
t
,
Optimizer update
∆t = mt/(
√
vt + ε) + λtet, (decoupled correction)
Model update
xt+1 = xt − α∆t,
To compare these two correction variants in the context
of the Adam optimizer, note that the coupled correction
term λtet becomes (1 − β1)λtet/(
√
vt + ε) after being
processed by the optimizer states. Instead of relying solely on STE, ProxQuant
applies a prox operator between stochastic gradient steps to
encourage quantization. Similar to ProxQuant, PARQ uses
convex and piecewise-affine regularization to induce weight
quantization with the difference that gradient updates are
added to the full-precision parameters before applying the
proximal operator (Jin et al. Our
results demonstrate that CAGE leads to state-of-theart results across all scenarios: for fine-tuning QAT,
it roughly halves the quantization error relative to
the prior best method, whereas, in pre-training experiments, CAGE-trained models with 3-bits weights
and activations (W3A3) offer lower loss than 4-bit
(W4A4) models trained with the prior best method
(QuEST (Panferov et al. • Moreover, we show experimentally that the CAGE
approach is optimizer-agnostic, as it leads to consistent gains across several optimizers, including
AdamW (Loshchilov & Hutter, 2019), Muon (Jordan,
2024) and Shampoo (Gupta et al. Additionally, these scalings are adjusted
by a global scaling factor obtained from approximate Hessian information. A related method employs position-based
scaled gradients, where the gradient estimator is adjusted
only by multiplying the quantization error with a small positive margin (Kim et al. In
addition, as described in their paper, LOTION requires a fullprecision forward pass, and the final model is a weight-only
quantized model. By contrast, CAGE training is done with
weights and activations quantized, where it can also benefit
from high-performance low-precision GEMM kernels for
more efficient training and inference. Since improvements introduced
by CAGE do not depend on the choice of optimizer, we
conclude that it can be regarded as an optimizer-agnostic
technique. Precision Scaling Law. We fine-tune on
Tulu-SFT (Lambert et al. , 2024) with QAT (master weights
in FP32, the forward pass has both weights and activations
quantized to MXFP4), following the recently-proposed QAT
recipe of (Egiazarian et al. ,
2024; OpenAI, 2025). Yet, the vast majority of open research in this area has concentrated on post-training quantization (PTQ), where a fully trained model’s weights are
quantized by applying numerical algorithms over a small
calibration dataset (Frantar et al. , 2019) represent each parameter of the
model as weighted average of quantized values, and minimizes the loss with respect to these probabilistic weights. (2021a) proposed the element-wise gradient scaling (EWGS) approach, in which the scaling of each gradient
element depends on both its sign and the corresponding
quantization error. Prior work on convergence
guarantees for the problem described in Equation (1) in
the non-convex setting attempts to find a quantized point
Q(x
∗
) that is a stationary point for the loss f, namely
∇f(Q(x
∗
)) = 0. However, as quantization is not an invertible transformation,
this goal cannot be achieved in general, and current convergence guarantees have non-vanishing terms in their rate,
proportional to quantization error (Lin et al. CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training
as x − Q(x) = ∇ϕ(x) for some smooth regularizer ϕ(x). Notably, our approach naturally avoids the need to handle
the (non-existent) Jacobian JQ(x) of the quantization operator and bypasses the computation of a proximal step for
the explicit “quadratic” regularizer λ
2
∥x − Q(x)∥
2
2
, which
would be infeasible for the dynamic quantization used in
our experiments (Bai et al. CAGE consistently
outperformed strong baselines that employ modern outliermitigation techniques. Specifically, scaling law analysis
confirms that CAGE improves the effective capacity of models compared to existing QAT methods. CAGE: CURVATURE-AWARE GRADIENT ESTIMATION FOR ACCURATE
QUANTIZATION-AWARE TRAINING
Soroush Tabesh * 1 Mher Safaryan * 1 Andrei Panferov 1 Alexandra Volkova 1 Dan Alistarh 1 2
ABSTRACT
Despite significant work on low-bit quantization-aware training (QAT), there is still an accuracy gap between such
techniques and native training. To address this, we introduce CAGE (Curvature-Aware Gradient Estimation), a
new QAT method that augments the straight-through estimator (STE) gradient with a curvature-aware correction
designed to counteract the loss increase induced by quantization. OLMo2 models use
no biases, employ rotary positional embeddings (Su et al. ,
2024), RMSNorm (Zhang & Sennrich, 2019), and reordered
pre-normalization (Liu et al. com/IST-DASLab/CAGE. 1 INTRODUCTION
Quantization has emerged as a standard technique for improving the computational efficiency of large language
model (LLM) deployments, as prominent open-source models such as Llama, Gemma, and GPT-OSS are released in
compressed formats (Touvron et al. com/IST-DASLab/CAGE. 1 INTRODUCTION
Quantization has emerged as a standard technique for improving the computational efficiency of large language
model (LLM) deployments, as prominent open-source models such as Llama, Gemma, and GPT-OSS are released in
compressed formats (Touvron et al. To verify that gains produced by CAGE are independent of the optimizer update,
we conduct a series of additional experiments using alternative optimizers to AdamW: Muon (Jordan, 2024), Shampoo
(Gupta et al. , 2018), and SOAP (Vyas et al. CAGE: CURVATURE-AWARE GRADIENT ESTIMATION FOR ACCURATE
QUANTIZATION-AWARE TRAINING
Soroush Tabesh * 1 Mher Safaryan * 1 Andrei Panferov 1 Alexandra Volkova 1 Dan Alistarh 1 2
ABSTRACT
Despite significant work on low-bit quantization-aware training (QAT), there is still an accuracy gap between such
techniques and native training. To address this, we introduce CAGE (Curvature-Aware Gradient Estimation), a
new QAT method that augments the straight-through estimator (STE) gradient with a curvature-aware correction
designed to counteract the loss increase induced by quantization. CAGE: CURVATURE-AWARE GRADIENT ESTIMATION FOR ACCURATE
QUANTIZATION-AWARE TRAINING
Soroush Tabesh * 1 Mher Safaryan * 1 Andrei Panferov 1 Alexandra Volkova 1 Dan Alistarh 1 2
ABSTRACT
Despite significant work on low-bit quantization-aware training (QAT), there is still an accuracy gap between such
techniques and native training. To address this, we introduce CAGE (Curvature-Aware Gradient Estimation), a
new QAT method that augments the straight-through estimator (STE) gradient with a curvature-aware correction
designed to counteract the loss increase induced by quantization. OLMo2 models use
no biases, employ rotary positional embeddings (Su et al. ,
2024), RMSNorm (Zhang & Sennrich, 2019), and reordered
pre-normalization (Liu et al. Here, the exact Hessian matrix is replaced by its Gauss–Newton approximation. If we further
apply the empirical Fisher approximation and approximate
the diagonal entries of the Gauss–Newton matrix using
Adam statistics, this regularization takes similar form as our
coupled correction variant (except the square root).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 3 ---
CAGE: CURVATURE-AWARE GRADIENT ESTIMATION FOR ACCURATE QUANTIZATION-AWARE TRAINING
3 CAGE: CURVATURE-AWARE GRADIENT ESTIMATION
3.1 Problem Description and Motivation
Intuitively, the key question in quantization-aware training (QAT) is to construct a gradient estimator that updates the parameters towards an “optimal” quantized model, i.e., a constrained model which minimizes the loss The optimization problem underlying QAT can be described by the following constrained optimization min x∈Q f(x), (1) where Q = Q(x) is the constraint set of quantized parameters and Q: R d → R is the quantization operator The standard is the STE, which bypasses the quantization operator during the backward pass by approximating its gradient with the identity Error feedback reformulation shows that SGD-based training with STE over iterate xt is equivalent to an instance of error-feedback with quantized parameters wt = Q(xt) and quantization error et = xt − Q(xt) The right-hand-side formulation says that STE is equivalent to a process where the quantization error et+1 due to weight quantization at a given step (wt − gt) − wt+1 is fed into the gradients at the next step
3.2 Theoretical Analysis
The authors propose viewing the QAT problem from a multi-objective perspective, where one objective is to minimize task loss f(x) and the second objective is to satisfy the quantization constraint x ∈ Q They define solutions as Pareto optimal points that balance these two conditions, leading to the condition ∇λPf(Q(x∗)):= ∇f(x∗) + λ(x∗ − Q(x∗)) = 0 (2) Under Assumption 3, the quantization error can be expressed as x − Q(x) = ∇ϕ(x) for some smooth regularizer ϕ(x) The convergence result shows strong ergodic convergence to a Pareto-optimal state in the smooth non-convex setting The final bound on the expected norm of the Pareto-gradient is E∥∇λPf(Q(ˆx))∥ squared ≤ 1/√T √2(A1 + A2 + A3) max 1, √L T
3.3 Practical Implementation
The authors propose two ways to incorporate the quantization error: the coupled correction and the decoupled correction The coupled variant adds the quantization error to the current gradient before it is passed to the optimizer, resulting in a diagonally preconditioned, curvature-aware correction term The decoupled variant applies the correction at unit scale after the optimizer step, deliberately avoiding preconditioning path for practical stability In practice, CAGE is implemented on top of AdamW using a silence period ratio s and regularization parameter λ to control when the correction is activated
G FORMAL JUSTIFICATION FOR λ SCHEDULER
The rationale for the silence period is that early in training, quantization residuals behave like approximately mean-zero, bounded perturbations that cancel in aggregate but have T-scale fluctuations Applying the correction too early can over-constrain the dynamics and inject noise-like forcing terms into the dynamics Near convergence, however, residuals become coherent and effectively push parameters toward the quantized support, which is precisely when enforcing Pareto-stationarity becomes beneficial
G.6 Dynamic quantizers where L ≈ 1
The authors define dynamic quantizers by the integer code rather than the reconstructed value to define visits based on code stability This approach ensures that the residual is approximately mean-zero during exploration phase, which is crucial for applying martingale and concentration arguments The analysis shows that early residuals behave like bounded, approximately mean-zero perturbations that cancel in aggregate but have T-scale fluctuations This leads to a cancellation bound where the early residuals behave like bounded, approximately mean-zero perturbations
G.3 Cancellation bound
The concentration across visits implies that for any unit vector u in S d−1 and any ε > 0, the expected value of the block sum of early residuals is OP(BLmax√K) via a standard ε-net argument The analysis shows that early residuals behave like bounded, approximately mean-zero perturbations
G.5 Why λt should increase late
Near convergence, the iterate typically oscillates among a small set of adjacent cells for long periods, so the ”reset” condition weakens In this regime, residuals become coherent and effectively push xt toward the quantized support, which is precisely when enforcing Pareto-stationarity becomes beneficial Therefore, λt should be larger late in training
The authors define dynamic quantizers by the integer code rather than the reconstructed value to define visits based on code stability This approach ensures that the residual is approximately mean-zero during exploration phase, which is crucial for applying martingale and concentration arguments The analysis shows that early residuals behave like bounded, approximately mean-zero perturbations This leads to a cancellation bound where the early residuals behave like bounded, approximately mean-zero perturbations
**G.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Curvature-Aware Gradient Estimation for Accurate Quantization-Aware Training allows AI systems to achieve state-of-the-art accuracy in low-bit quantization by augmenting standard straight-through estimator gradients with a principled correction term derived from Pareto optimality, which
explicitly counteracts the increase in loss induced by the quantization step
. -
The system can be trained on models up to 800M parameters from scratch using 3-bit weights and activations (W3A3) and achieve accuracy matching that of a prior state-of-the-art method trained at 4 bits (W4A4).
-
CAGE provides strong ergodic convergence guarantees to a Pareto-optimal point in the smooth non-convex setting, addressing the
lack of guarantees that is a primary limitation of prior work in this area.
-
The optimizer agnostic implementation allows the system to leverage existing optimizer statistics, such as Adam's second-moment statistics, by employing either coupled or decoupled corrections to stabilize training.
-
The system can be trained across various optimizers including AdamW, Muon, and Shampoo without requiring specific hyperparameter tuning for the correction term due to
the primary gains originate from the shared correction direction rather than the specific scaling.
-
CAGE improves parameter efficiency by more than 10 percent at 4-bit precision and 20 percent at 2-bit precision relative to prior state-of-the-art methods, indicating
improved effective capacity for a given parameter budget.
-
The system demonstrates superior long context performance compared to baselines like QuEST and GPTQ when fine-tuned with CAGE on Llama 3.1-8B models under W4A16 quantization, narrowing the performance gap between quantized and full precision models.
Abstract
Despite significant work on low-bit quantization-aware training (QAT), there is still an accuracy gap between such techniques and native training. To address this, we introduce CAGE (Curvature-Aware Gradient Estimation), a new QAT method that augments the straight-through estimator (STE) gradient with a curvature-aware correction designed to counteract the loss increase induced by quantization. CAGE is derived from a multi-objective view of QAT that balances loss minimization with the quantization constraints, yielding a principled correction term that depends on local curvature information. On the theoretical side, we introduce the notion of Pareto-optimal solutions for quantized optimization, and establish that CAGE yields strong convergence guarantees in the smooth non-convex setting. In terms of implementation, our approach is optimizer-agnostic, but we provide a highly-efficient implementation that leverages Adam statistics. CAGE significantly improves upon the prior state-of-the-art methods in terms of accuracy, for similar computational cost: for QAT fine-tuning, it halves the compression accuracy loss relative to the prior best method, while for QAT pre-training of Llama models, its accuracy for 3-bit weights-and-activations (W3A3) matches the accuracy achieved at 4-bits (W4A4) with the prior best method. The official implementation can be found over https://github.com/IST-DASLab/CAGE.
Sources
- Proximal Mean-field for Neural Network Quantization
- Mirror Descent View for Neural Network Quantization
- ProxQuant: Quantized Neural Networks via Proximal Operators
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Quantized Adam with Error Feedback
- Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training
- The Llama 3 Herd of Models
- Differentiable Model Compression via Pseudo Quantization Noise
- Learned Step Size Quantization
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- Gemma: Open Models Based on Gemini Research and Technology
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- PARQ: Piecewise-Affine Regularized Quantization
- Position-based Scaled Gradient for Model Quantization and Pruning
- Quantizing deep convolutional networks for efficient inference: A whitepaper
- Scaling Laws for Precision
- LOTION: Smoothing the Optimization Landscape for Quantized Training
- AdaSTE: An Adaptive Straight-Through Estimator to Train Binary Neural Networks
- Network Quantization with Element-wise Gradient Scaling
- Training Quantized Nets: A Deeper Understanding
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis