Beyond local reconstruction: Geometry-preserving output alignment for binary LLM quantization
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Beyond local reconstruction".
Jane: Here's the gist of the paper. Large language models (LLMs) deliver strong performance across a wide range of NLP tasks, but their massive sizes hinder deployment on resource-constrained devices The paper proposes a novel post-training quantization (PTQ) method for 1-bit LLMs that explicitly addresses error accumulation and anisotropic distortion of the representation space while maintaining computational efficiency.
Tom: The authors identify two fundamental limitations of output-driven quantization in the 1-bit regime: error accumulation across layers and anisotropic distortion of the representation space First, error propagation leads to a mismatch between layer-wise objectives and the global model behavior because existing frameworks operate on activations from the quantized model, Xb, without accounting for their progressive deviation from the true full-precision inputs, X Second, output-driven objectives introduce anisotropic distortion in the representation space because minimizing mean squared error (MSE) in Euclidean space biases the solution toward high-magnitude activation features while underweighting low-variance but structurally important directions.
Jane: Proposed Quantization Strategy The proposed method consists of two components: an error-aware output reconstruction objective that accounts for accumulated upstream quantization error, and a token-geometry regularizer that encourages the quantized layer to preserve the token similarity structure of the full-precision model The error-aware output reconstruction involves minimizing a loss L which reconstructs the full-precision output as XW − XbWc = Xb(W − Wc) + (X − Xb)W This objective is more complex than previous methods because it requires compensating for inherited errors from previous layers, resulting in a more complex, non-convex loss landscape that is harder to converge.
Tom: Attention Matrix Preservation Mechanism To capture the relational structure of layer’s output, the method uses token-covariance Gram matrices Gq and Gf to measure representation similarity between quantized and full-precision output tokens The regularization term LReg = − Tr(Gq⊤ Gf) is introduced to maximize the alignment between these matrices, which leads to a directional filtering mechanism for parameter updates Specifically, the Attention Matrix Preservation (AMP) masks are defined element-wise as Mr AMP = 1
G r ⊙ ∆αr ≤ 0: , Mc AMP = 1
G c ⊙ ∆αc ≤ 0: , MB AMP = 1
G B ⊙ ∆B ≤ 0: These masks ensure that parameter updates are only accepted if they are aligned with the descent direction of the regularizer, leading to non-increasing regularization loss LReg(vt+1) ≤ LReg(vt).
Title and authors: Tom: So, we're looking at this paper called "Beyond local reconstruction: Geometry-preserving output alignment for binary LLM quantization". It’s about taking those huge language models and squeezing them down to just one bit, which is really tough.
Jane: It sounds like they are trying to find a better way than just matching the weights directly to see if it works for these one-bit models <ref:2512.21651#pg1>. What's the core problem they're tackling here?
Tom: The paper points out two major headaches with the current output-driven quantization methods: first, error accumulation across different layers, and second, anisotropic distortion of the representation space.
Lu: Right. So when you quantize layer by layer, that small error from the first layer gets built up and messes up the later layers because they are using activations from a quantized model instead of the true full-precision inputs <ref:2512.21651#pg3>.
Meng: That sounds like a deployment nightmare. If you have to constantly worry about that error building up, it adds complexity on top of just getting the bits right Meng.
Tom: Exactly. And the second issue is this distortion in the representation space, where minimizing standard mean squared error biases the model toward big activation features while ignoring important directions that have low variance <ref:2512.21651#pg3>.
Jane: That makes sense for someone listening who isn't reading code. So, instead of just trying to get the output right at every step, they are changing *how* the model learns those bits to preserve the shape of its internal understanding <ref:2512.21651#pg3>.
Tom: They propose a new post-training quantization method that has two main parts: an error-aware objective and this attention-aware masking mechanism, which they call AMP.
Lu: The error-aware objective is complicated because it tries to reconstruct the full precision output by compensating for all those inherited errors from previous layers <ref:2512.21651#pg3>. It has a complex loss landscape that can be hard to solve.
Meng: How complex does that make the actual quantization process? Does it slow down the calibration time significantly, or is it just more work during the optimization stage?
Title and authors: Tom: Well, they say it requires additional closed-form computations and these AMP masks, so there's a slight overhead compared to some other methods like ARB-RC nine <ref:2512.21651#pg2>. But they claim this overhead is still better than ARB-X for one-bit models <ref:2512.21651#pg1>.
Jane: So, the numbers show that this approach actually works better than what was out there before? What kind of improvements are they seeing in practice?
Tom: They ran extensive experiments and found that their method consistently outperforms existing one-bit PTQ techniques on most settings across both OPT and LLaMA models <ref:2512.21651#pg2>.
Lu: Specifically, they saw improvements in perplexity on models like OPT, reducing it by up to two point five one on C4 and five point three nine on WikiText Two eight. That's a concrete measure of how much better the model is at actually doing the task.
Meng: I need to see those specific numbers for LLaMA-two-7B, though, because that’s what we ship to production. Is it a significant jump in accuracy for our scale?
Tom: The ablation study on the Attention Matrix Preservation, or AMP, shows that removing it consistently degrades performance across both OPT and LLaMA models <ref:2512.21651#pg3>. So, that masking mechanism is definitely doing real work here.
Jane: That’s important because it means they aren't just finding a trick for the loss function; they are actually protecting the relational structure of what the model learns forty <ref:2512.21651#pg2>.
Lu: The AMP masks are defined by comparing token covariance Gram matrices, Gq and Gf, to ensure parameter updates only happen in directions that align with preserving that similarity structure seven <ref:2512.21651#pg2>. It's a directional filtering mechanism.
Meng: So if we think about deployment, does this mean we can trust the attention maps more when running these quantized models?
Tom: Yes. The paper shows visualizations of the attention scores for LLaMA-two-7B, and comparing them with and without AMP really shows how much that mask helps align with the full precision model's structure six.
Jane: It seems like a big deal because it addresses that anisotropic distortion issue we talked about earlier. It means the representation space stays structured instead of getting stretched out randomly forty <ref:2512.21651#pg2>.
Title and authors: Lu: The main implication is that they’ve found a way to handle the unique challenges of one-bit quantization—error accumulation and geometry distortion—by selectively applying output alignment at the block level <ref:2512.21651#pg3>.
Meng: What's the practical cost again? Besides the slightly higher overhead, how does this translate to memory on a model like Llama-two-7B?
Tom: The memory usage is slightly higher at twelve point one gigabytes compared to some of the other methods we looked at, like ARB-X nine. But they say it's competitive with state-of-the-art techniques and it doesn't add any extra inference or storage overhead beyond what they already have nine <ref:2512.21651#pg2>.
Jane: So, for someone just using the model to run text on their computer, the main thing to know is that this method gives you better results without making your hardware requirements drastically worse.
Lu: The authors conclude that by combining error-aware reconstruction with attention-aware masking, they’ve managed to overcome those fundamental limitations in one-bit post-training quantization <ref:2512.21651#pg3>.
Meng: To wrap up, they found that calibration data is important, but naive output alignment fails because it messes up the attention masking forty <ref:2512.21651#pg2>. This paper shows how to fix that by selectively applying alignment at the block level with AMP.
Tom: So, in short, this paper introduces a method called "Beyond local reconstruction: Geometry-preserving output alignment for binary LLM quantization" that uses error awareness and attention masks to make one-bit LLMs perform much better than they used to <ref:2512.21651#pg2>.
Jane: It’s promising because it shows that even when you are pushing the limits of compression, you can still maintain a good balance between speed and quality for these large language models.
Lu: It suggests that future work might focus on how this geometry-preserving structure behaves as the model gets even smaller or as we move to higher bit depths <ref:2512.21651#pg3>.
Meng: We'll keep an eye on it, especially since the paper flags that their method stops working where the underlying assumptions about error accumulation are violated <ref:2512.21651#pg3>.
Tom: That’s what we're here for. It’s a solid step forward in making these massive models more accessible by tackling these deep mathematical issues nine <ref:2512.21651#pg2>.
The paper's summary: Tom: So, what we're looking at here is this paper about taking those huge language models and squishing them down to just one bit, which they call binary quantization. And they’re arguing that the standard way of doing it—just matching weights directly—is failing because of two big problems: error accumulation across layers and this weird distortion in the representation space.
Jane: Right, so if you're driving a car, you might think just matching the weight is like putting one side of a road onto another. But this paper says that’s not working because the road gets bumpy as you drive down it, and by the time you get to the end, it’s not actually what was supposed to be there.
Lu: Exactly. The error propagates through the layers, creating a mismatch between what each layer is trying to do and how the whole model is actually behaving when it's quantized <ref:2512.21651#pg3>. And on top of that, minimizing the error in a simple space just biases you toward big numbers, ignoring all those smaller details that are actually important for understanding the text.
Meng: So if I were trying to deploy this on a small edge device, would this complexity be worth it? Is this just more work during the model training phase?
Tom: It’s more work during the optimization stage because they introduce two specific things. First, an error-aware reconstruction objective that tries to fix those inherited errors from previous layers <ref:2512.21651#pg3>. And second, this thing called Attention Matrix Preservation, or AMP.
Jane: AMP sounds like a fancy name for a filter. It’s supposed to make sure that when the model is learning its bits, it only updates the parts of the parameters that are actually preserving the structure of how words relate to each other in the model's memory forty.
Lu: That masking mechanism is what prevents that anisotropic distortion from ruining the attention behavior over time. They’ve defined these masks by comparing token similarity matrices—the quantized one and the full precision one—so you only accept parameter updates that keep those similarities aligned with what they should be seven.
Tom: The result is that they show this method consistently beats existing one-bit techniques on most tests for both OPT and LLaMA models eight. They're showing real perplexity drops, like cutting the error down by up to two point five one percent on some datasets eight.
Jane: So, what does this mean for us as users? It means that when we compress these massive language models, we don't have to just blindly trust the output alignment anymore. We have a method that actively tries to keep the internal structure intact.
Meng: If it’s better than what I’m seeing in my production runs, what about the cost? They mentioned it adds some overhead with these new computations and masks, but they say it stays competitive with other methods like ARB-X nine.
Tom: The overhead is slightly higher than ARB-RC because of those extra steps and the AMP mask computation. But they say it doesn't add any extra storage or inference time beyond what you already have nine. It’s a trade-off between a little bit more compute during quantization and better final performance.
Lu: The main point is that they proved error accumulation isn't just noise; it's systematic, and they showed how to fight it by being smart about reconstruction and structure preservation <ref:2512.21651#pg3>.
Jane: So, for anyone listening who just wants the simple picture, this paper shows that when you compress a language model to one bit, you have to be smarter than just matching numbers; you have to think about how those numbers affect the whole system's structure.
Tom: Yeah. The big implication is that this approach stabilizes the representation space, which means we can actually trust what the quantized model is doing when it starts reasoning or generating text forty.
Lu: And they’ve confirmed that removing that AMP masking just makes things worse, so it’s a necessary part of the strategy, not just an optional fancy addition.
Jane: It seems like this work shows that for one-bit quantization to actually be useful, we need to focus on preserving the relationships between tokens, not just trying to match the final output numbers.
Tom: Exactly. And now they’ve shown that this structure preservation leads to real gains in reasoning tasks like ARC and PIQA eight, which is pretty impressive when you think about how much smarter these models are getting at problem-solving.
The paper's improvements: Tom: So, we're talking about how they actually fix those two problems we mentioned earlier—the error accumulation and that distorted space—with their new technique. They suggest a couple of specific fixes to make this work better than before.
Jane: It sounds like they’re adding a "correction" term to the loss function, which is supposed to help the model optimize for what it *should* be doing at every layer, not just what it is right now <ref:2512.21651#pg3>.
Lu: That error-aware reconstruction objective they propose is more complex because it has to account for all those errors that got passed down from the layers before <ref:2512.21651#pg3>. It tries to pull the model back toward a full precision target rather than just optimizing for the quantized version itself.
Meng: So, if I’m looking at this from a deployment standpoint, does that mean we have to run more complex math on our hardware during training?
Tom: It means yes, it adds some complexity because you’re solving a harder problem with that non-convex loss landscape <ref:2512.21651#pg3>. But they say it leads to a better final result than just using the standard output-driven objective alone.
Jane: And then they have this Attention Matrix Preservation mechanism, AMP, which acts like a gate on the parameter updates <ref:2512.21651#pg7>. It filters out any update that would mess up how tokens relate to each other geometrically.
Lu: That directional filtering is what keeps the representation space from getting warped by minimizing simple mean squared error <ref:2512.21651#pg3>. They use these matrices, Gq and Gf, to measure similarity and only let the model change weights if that change follows a direction that preserves that similarity seven.
Tom: The real implication is they're showing this isn't just a theoretical fix; it actually leads to tangible performance gains on reasoning benchmarks like ARC and PIQA eight. They’re seeing accuracy improvements across both OPT and LLaMA models eight.
Jane: That’s really cool because it moves beyond just getting the final text output right; they’re proving that by preserving the internal structure, the model gets genuinely smarter at tasks that require complex reasoning <ref:2512.21651#pg3>.
Meng: So, to summarize for someone who needs a concrete answer: it means we get better quality output from one-bit models, but you have to deal with a slightly more complicated training process and potentially a bit more memory usage on the GPU side nine.
Tom: That’s the trade-off they’re making. They’ve essentially found a way to keep the model's internal geometry stable while squeezing it down to one bit, which is pretty significant work for resource-constrained deployment nine.
Lu: The paper suggests that this approach of applying output alignment at the block level is a solid strategy for handling these issues in one-bit quantization.
Conclusion: Tom: So to wrap up, this paper on "Beyond local reconstruction: Geometry-preserving output alignment for binary LLM quantization" shows that we can do better than just matching weights for one-bit models by focusing on structure preservation.
Jane: Right, so the main idea is that by fixing those layer errors and using attention masks to keep the representation space organized, we get much better performance on reasoning tasks.
Lu: It’s about showing that output alignment doesn't have to be naive; it needs this careful handling of accumulated error and geometric distortion to actually work effectively.
Meng: So, for someone who has a model running on their laptop and is worried about deployment, what does this mean practically? Is it something I can just plug into my pipeline right now?
Tom: It means that if you’re serious about pushing one-bit LLMs to perform well without totally destroying the model's logic, this strategy gives you a solid path forward nine.
Jane: And the numbers back it up; we saw consistent improvements in perplexity on models like LLaMA and OPT across different settings eight. It’s not just theoretical stuff here.
Lu: The work confirms that this method stabilizes the attention behavior, which is critical for how these large language models actually reason through long sequences seven.
Meng: I still have to be careful because they say it introduces some extra closed-form computations and masks, so you gotta factor that into your total cost of running it at scale nine.
Tom: It’s a trade-off between more complex training steps and better final accuracy, which is what they are pushing for here.
Jane: It seems like this paper gives us a much clearer roadmap for how to tackle the tough challenges of one-bit quantization in the future.
Lu: We still have to figure out how this geometry-preserving structure holds up as we move to even smaller models or maybe different bit depths later on, though.
Meng: I’m just waiting to see if these results hold up when we put them on real traffic data instead of just benchmark sets nine.
Tom: Well, the core message is that by being smart about error accumulation and representation geometry in this work, we can achieve better quality output from one-bit models than was previously possible.
Jane: So if you’re listening and think about deploying these smaller AI models to devices, keep an eye on this paper; it gives us a much more robust method for quantization than just matching weights directly.
Lu: And we look forward to seeing how this geometry-preserving alignment strategy can be adapted for other types of compressed representations.
Meng: That’s the direction I’m focusing on now—seeing if these gains translate when we move beyond the paper's specific test sets.
Hoang Anh Dung, Cuong Pham, Cuong Nguyen, Trung Le, Thanh-Toan Do, Jianfei Cai
Monash University · University of Surrey
cs.LG
Submitted: 2025-12-25
Updated: 2026-11-09
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 87/100
The gist: Large language models (LLMs) deliver strong performance across a wide range of NLP tasks, but their massive sizes hinder deployment on resource-constrained devices The paper proposes a novel…
Key concepts
- Error Accumulation
- This occurs because quantization errors propagate layer by layer. Existing methods ignore this, causing a mismatch between what each layer is optimized for and how the entire model behaves globally. The proposed method fixes this by using an objective that accounts for these inherited errors from previous layers.
- Anisotropic Distortion of Representation Space
- Minimizing error in standard Euclidean space often biases the quantization towards large activation features, ignoring important low-variance directions that define structural relationships. This distortion degrades the token similarity structure and attention patterns over time, which is corrected by a regularization term.
- Attention Matrix Preservation (AMP)
- This mechanism uses token-covariance Gram matrices to measure representation similarity between quantized and full-precision outputs. It generates masks that filter parameter updates, ensuring that only changes aligned with preserving the desired token relationship structure are accepted, stabilizing attention behavior.
Terminology
Summary
Large language models (LLMs) deliver strong performance across a wide range of NLP tasks, but their massive sizes hinder deployment on resource-constrained devices The paper proposes a novel post-training quantization (PTQ) method for 1-bit LLMs that explicitly addresses error accumulation and anisotropic distortion of the representation space while maintaining computational efficiency. p. 0
Motivation for the Proposed Method
The authors identify two fundamental limitations of output-driven quantization in the 1-bit regime: error accumulation across layers and anisotropic distortion of the representation space First, error propagation leads to a mismatch between layer-wise objectives and the global model behavior because existing frameworks operate on activations from the quantized model, Xb, without accounting for their progressive deviation from the true full-precision inputs, X Second, output-driven objectives introduce anisotropic distortion in the representation space because minimizing mean squared error (MSE) in Euclidean space biases the solution toward high-magnitude activation features while underweighting low-variance but structurally important directions.
Proposed Quantization Strategy
The proposed method consists of two components: an error-aware output reconstruction objective that accounts for accumulated upstream quantization error, and a token-geometry regularizer that encourages the quantized layer to preserve the token similarity structure of the full-precision model The error-aware output reconstruction involves minimizing a loss L which reconstructs the full-precision output as XW − XbWc = Xb(W − Wc) + (X − Xb)W This objective is more complex than previous methods because it requires compensating for inherited errors from previous layers, resulting in a more complex, non-convex loss landscape that is harder to converge.
Attention Matrix Preservation Mechanism
To capture the relational structure of layer’s output, the method uses token-covariance Gram matrices Gq and Gf to measure representation similarity between quantized and full-precision output tokens The regularization term LReg = − Tr(Gq⊤ Gf) is introduced to maximize the alignment between these matrices, which leads to a directional filtering mechanism for parameter updates Specifically, the Attention Matrix Preservation (AMP) masks are defined element-wise as Mr AMP = 1[G r ⊙ ∆αr ≤ 0], Mc AMP = 1[G c ⊙ ∆αc ≤ 0], MB AMP = 1[G B ⊙ ∆B ≤ 0] These masks ensure that parameter updates are only accepted if they are aligned with the descent direction of the regularizer, leading to non-increasing regularization loss LReg(vt+1) ≤ LReg(vt).
Experimental Validation and Results
Extensive experiments demonstrate that the proposed method consistently outperforms existing 1-bit PTQ techniques for LLMs on majority of settings The results show improvements in perplexity across both OPT and LLaMA models, particularly noticeable on OPT-1.3B where error propagation modeling reduces perplexity by up to 2. p. 751 on C4 and 5.39 on WikiText2 Furthermore, the ablation study on Attention Matrix Preservation (AMP) confirms that removing AMP consistently degrades performance across both OPT and LLaMA models, with substantially larger degradation observed on LLaMA-2-7B. p. 8 The method also introduces no additional inference or storage overhead, as it only modifies the update rules used during the quantization stage. p. 8
Overhead Analysis
The method introduces no additional inference or storage overhead, as it only modifies the update rules used during the quantization stage While our method incurs slightly higher overhead than ARB-RC due to the additional closed-form computations and AMP mask, it remains more efficient than ARB-X Regarding memory overhead, for Llama-2-7B model, our method has slightly higher peaked gpu memory at 12.1 GB, compared 11. p. 64 GB of ARB-X and 10. p. 28 GB of ARB-RC. p. 2 The quantization time comparison shows that our method is competitive with state-of-the-art techniques. p. 6
Conclusion
In this work, we investigated the role of calibration data in 1-bit post-training quantization of large language models Our analysis revealed important insights: activation mismatches can accumulate across layers; and naive output alignment may degrade attention masking, all of which can negatively impact the effectiveness of output-driven optimization for 1-bit post-training quantization. Building on these insights, we introduced a quantization strategy that selectively applies output alignment at the block level, incorporates attention-aware masking, and reformulates the quantization objective to account for accumulated error. p. 9 Extensive experiments demonstrate that our method consistently outperforms prior 1-bit PTQ approaches for LLMs on majority of settings. p. 9
References
[1] Yamato Arai and Yuma Ichikawa. Quantization error propagation: Revisiting layer-wise posttraining quantization. p. 9 arXiv preprint arXiv:2504. p. 9 p. 909629, 2025. p. 9 URL https://arxiv. p. 9org/abs/2504. p. 909629
[3] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. p. 9 In Proceedings of the AAAI conference on artificial intelligence, volume 34(05), pages 7432–7439, 2020. p. 9 doi: 10.1609/aaai.v34i05.6239
[4] Kawin Ethayarajh. How contextual are contextualized word representations? p. 1 p. 1 comparing the geometry of bert, elmo, and gpt-2 embeddings. p. 8 p. 8 In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 55–65, 2019. p. 9 p. 9 doi: 10.18653/v1/D19-1006
[5] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. p. 0 p. 0 ICLR, 2023.
[6] Zhiteng Li, Xianglong Yan, Tianao Zhang, Haotong Qin, Dong Xie, Jiang Tian, Zhongchao Shi, Linghe Kong, Yulun Zhang, and Xiaokang Yang. ARB-LLM: Alternating refined binarizations for large language models. p. 9 p. 9 arXiv preprint arXiv:2410. p. 9 p. 903129, 2025. p. 9
[7] Proposition 1. p. 5 Let vt denote a block of quantized weights at iteration t, and let ∆vt = v∗t − vt be the candidate closed-form update obtained from minimizing the reconstruction objective L. p. 6 Let Gv,t = ∇vtLReg(vt) be the gradient of the regularization objective with respect to vt. p. 6 The AMP masks for vt is defined as: Mv,t = 1[Gv,t ⊙ ∆vt ≤ 0]. p. 5 The gradient update is: Dv,t = Mv,t ⊙ ∆vt. p. 4 Under a mild assumption, this AMP update guarantees a non-increasing of the regularization loss term: LReg(vt+1) ≤ LReg(vt). p. 6
[8] Table 3: Perplexity (↓) of Qwen models under different quantization methods on the C4 and WikiText2 datasets, and the average accuracy (↑) for zero-shot reasoning QA datasets. p. 7
[9] Table 5: Quantization time comparison between 1-bit LLM methods and ours across different models. p. 9
--- Page 1 ---
Rethinking Output Alignment For 1-bit Post-Training
Quantization of Large Language Models Hoang Anh Dung Department of Data Science and AI Monash University Cuong Pham Department of Data Science and AI Monash University Cuong Nguyen Centre for Vision, Speech and Signal Processing University of Surrey Trung Le Department of Data Science and AI Monash University Thanh-Toan Do Department of Data Science and AI Monash University Jianfei Cai Department of Data Science and AI Monash University Abstract Large Language Models (LLMs) deliver strong performance across a wide range of NLP tasks, but their massive sizes hinder deployment on resource-constrained devices To reduce their computational and memory burden, various compression techniques have been proposed, including quantization, pruning, and knowledge distillation Among these, post-training quantization (PTQ) is widely adopted for its efficiency, as it requires no retraining and only a small dataset for calibration Recent advances for post-training quantization have demonstrated that even near 4-bit methods can maintain most of the original model performance However, 1-bit quantization remains particularly challenging. A common strategy in 1-bit quantization is to determine binary weights by matching full-precision parameters, following a weight-driven criterion. p. 0 However, this objective is not directly aligned with the quantized model’s objective, which is to preserve the model’s output behavior under the impact of quantization. p. 0 A natural alternative is to adopt output-driven criteria that minimize discrepancies in model outputs using calibration data. p. 0 Surprisingly, naive output-driven approaches often perform even worse in the 1-bit regime. p. 0 In this paper, we show that this failure arises from two fundamental issues: error accumulation across layers and, more critically, anisotropic distortion of the representation space. p. 0 Based on these insights, we propose a novel PTQ method for 1-bit LLMs that explicitly addresses these issues while maintaining computational efficiency. p. 1
--- Page 3 ---
In this work, we identify two fundamental limitations of output-driven quantization in the 1-bit regime: error accumulation and anisotropic distortion of the representation space First, error propagation leads to a mismatch between layer-wise objectives and the global model behavior because existing frameworks operate on activations from the quantized model, Xb, without accounting for their progressive deviation from the true full-precision inputs, X Second, output-driven objectives introduce anisotropic distortion in the representation space because minimizing mean squared error (MSE) in Euclidean space biases the solution toward high-magnitude activation features while underweighting low-variance but structurally important directions.
--- Page 4 ---
Figure 1: Limitations of layer-wise output-driven optimization). (Left) Using progressively quantized model’s intermediate activations as inputs during optimization leads to a mismatch between the layerwise optimization error and the true end-to-end quantization error (Right) Although output-driven objective minimizes quantization error better than weight-driven methods, the anisotropic distortion in geometry leads to degradation in representation similarity and attention structure over time. p. 3 p. 3
--- Page 6 ---
Figure 2 in the visualizes the attention scores of LLaMA-2-7B at final block of the model using our method, with and without AMP mask, compared to that of the full-precision model.
--- Page 8 ---
Figure 2: Comparison of attention masks with and without AMP against the full-precision model.
--- Page 9 ---
Figure 2 in the visualizes the attention scores of LLaMA-2-7B at final block of the model using our method, with and without AMP mask, compared to that of the full-precision model.
--- Page 10 ---
Table 1: Perplexity results of OPT and LLaMA models. Separate weight bit columns are used for each model family. p. 6 Use OA denotes methods that use output-driven objective for quantization Bits OPT OPT LLaMA Bits LLaMA Bits
--- Page 12 ---
[25] Yuhang Li, Ruokai Yin, Donghyun Lee, Shiting Xiao, and Priyadarshini Panda. GPTQv2: Efficient finetuning-free quantization for asymmetric calibration. p. 1 arXiv preprint arXiv:2504.02692, 2025. p. 9
[11] Peijie Dong, Lujun Li, Dayou Du, Yuhan Chen, Zhenheng Tang, Qiang Wang, Wei Xue, Wenhan Luo, Qi-fei Liu, Yi-Ting Guo and Xiaowen Chu. STBLLM: Breaking the 1-bit barrier with structured binary llms. p. 2 arXiv preprint arXiv:2408. p. 901803
[15] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. ICLR, 2023.
[16] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan et al. The Llama 3 herd of models. p. 6 arXiv preprint arXiv:2407. p. 921783
[17] Yiwen Guo, Anbang Yao, Hao Zhao, and Yurong Chen. Network sketching: Exploiting binary structure in deep cnns. p. 1 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4040–4048, 2017. p. 2
[18] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. p. 0 arXiv preprint arXiv:1510. p. 900149, 2015
[26] Zhiteng Li, Xianglong Yan, Tianao Zhang, Haotong Qin, Dong Xie, Jiang Tian, Zhongchao Shi, Linghe Kong, Yulun Zhang, and Xiaokang Yang. ARB-LLM: Alternating refined binarizations for large language models. arXiv preprint arXiv:2410.03129
[27] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for llm compression and acceleration. p. 2 arXiv preprint arXiv:2306. p. 900978
[28] Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. p. 2 arXiv preprint arXiv:1808. p. 900278
[29] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. p. 0 arXiv preprint arXiv:1609. p. 907843
[36] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. XNOR-Net: Imagenet classification using binary convolutional neural networks. p. 2 In Computer Vision – ECCV 2016, pages 525–542. p. 2 Springer, 2016.
[38] Yuzhang Shang, Zhihang Yuan, Qiang Wu, and Zhen Dong. PB-LLM: Partially binarized large language models. p. 0 In The Twelfth International Conference on Learning Representations, 2024. p. 9
[39] Yuxuan Sun, Ruikang Liu, Haoli Bai, Han Bao, Kang Zhao, Yuening Li, Jiaxin Hu, Xianzhi Yu, Lu Hou, Chun Yuan et al. Flatquant: Flatness matters for llm quantization. p. 1 In ICML 2025. p. 9
[40] Hugo Touvron et al. Llama 2: Open foundation and fine-tuned chat models. p. 6 arXiv preprint arXiv:2307. p. 909288, 2023
--- Page 11 ---
[13] Kawin Ethayarajh. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 55–65, 2019. doi: 10.18653/v1/D19-1006
[43] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou et al. Emergent abilities of large language models. p. 0 Transactions on Machine Learning Research, Jun 2022. p. 1
[49] Susan Zhang et al. Opt: Open pre-trained transformer language models. p. 7 p. 7 arXiv preprint arXiv:2205. p. 9 p. 901068, 2022
--- Page 13 ---
[14] Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal et al. Beyond english-centric multilingual machine translation. p. 1 Journal of Machine Learning Research, 22(107):1–48, 2020. p. 1
[35] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena et al. Exploring the limits of transfer learning with a unified text-to-text transformer. p. 1 The Journal of Machine Learning Research, 21(1):5485–5551
[37] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. p. 6 Communications of the ACM, 64(9):99–106, 2021.
[48] Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. p. 9 Liu. PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization. p. 1 In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 11328–11339, PMLR, 2020. p. 9
[49] Susan Zhang et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022
--- Page 7 ---
Figure 1: Limitations of layer-wise output-driven optimization). (Left) Using progressively quantized model’s intermediate activations as inputs during optimization leads to a mismatch between the layerwise optimization error and the true end-to-end quantization error (Right) Although output-driven objective minimizes quantization error better than weight-driven methods, the anisotropic distortion in geometry leads to degradation in representation similarity and attention structure over time.
--- Page 12 ---
[12] Hoang Anh Dung, Cuong Pham, Trung Le, Jianfei Cai, and Thanh-Toan Do. Sharpness-aware data generation for zero-shot quantization. p. 7 In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. p. 9 Proceedings of Machine Learning Research
[…
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Error accumulation across layers can be mitigated by using an error-aware output reconstruction objective defined as L = XW - XbWC 2F = Xb(W - Wc) + (X - Xb)W 2F, which accounts for inherited errors from previous layers. This allows the model to optimize towards the full-precision intermediate target instead of just the quantized one.
-
Introduce an attention-aware masking mechanism called Attention Matrix Preservation AMP, which explicitly preserves token-level relational structure by filtering gradient updates that increase geometry distortion. This prevents anisotropic distortion of the representation space and stabilizes attention behavior during quantization.
-
The proposed method enables 1-bit LLMs to maintain or improve performance on downstream reasoning tasks like ARC and PIQA by ensuring that the preserved token similarity structure is maintained, leading to higher average accuracy gains across zero-shot QA benchmarks.
Sources
- Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization
- Language Models are Few-Shot Learners
- Bridging the Accuracy Gap for 2-bit Quantized Neural Networks (QNN)
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- CBQ: Cross-Block Quantization for Large Language Models
- STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMs
- The Llama 3 Herd of Models
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
- Distilling the Knowledge in a Neural Network
- Gradient-Aligned Calibration for Post-Training Quantization of Diffusion Models
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibration
- ARB-LLM: Alternating Refined Binarizations for Large Language Models
- AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm
- Pointer Sentinel Mixture Models
- The LAMBADA dataset: Word prediction requiring a broad discourse context
- MetaAug: Meta-Data Augmentation for Post-Training Quantization
- Llama 2: Open Foundation and Fine-Tuned Chat Models
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis