GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling".
Jane: Here's the gist of the paper. Quantization has become a standard tool for efficient LLM deployment, especially for local inference, where models are now routinely served at 2–3 bits per parameter The state of the art is currently split into simple scalar quantization techniques, such as GPTQ or AWQ, which are widely deployed but plateau in accuracy at 3–4 bits per parameter (bpp), and “second-generation” vector- or trellis-quantized methods, such as QTIP, GPTVQ and AQLM, which push the accuracy frontier but are notoriously hard to implement and to scale In this paper, we ask whether this gap is fundamental, or whether a carefully optimized scalar quantizer can recover most of it We answer in the affirmative, by introducing GSQ (Gumbel-Softmax Quantization), a post-training scalar quantization method which jointly learns the per-coordinate grid assignments and the per-group scales using a Gumbel-Softmax relaxation of the discrete grid Pruning redundant reasoning tokens GSQ is introduced as a post-training scalar quantization method that closes most of the gap between scalar quantization and second-wave techniques while preserving deployment-friendly scalar structure Its main form produces symmetric, group-wise, b-bit weights drawn from a small uniform grid, making it directly compatible with existing scalar inference kernels The method is designed to reformulate layer-wise reconstruction as a differentiable discrete-assignment problem How the training signal is built The key idea involves introducing a small set of trainable logits over candidate grid points for each weight coordinate, which are then relaxed using Gumbel-Softmax sampling to obtain a soft quantized weight The temperature parameter τ controls the sharpness of this distribution, annealing toward zero to yield a fully discrete quantized layer at the end of training For higher bit-widths, a local shift-based formulation is used, introducing only 5 logits corresponding to discrete shifts δi ∈ (−2, -1, 0, 1, 2) instead of the full 2b logits Initialization involves warm-starting from GPTQ or an existing GGUF checkpoint using a Gaussian-like prior around the GPTQ solution Where the gains come from Empirically, GSQ sets a new state of the art for scalar quantization at low bitwidths, improving average zero-shot accuracy by remarkable amounts over best scalar baselines For standard Llama models at 2 bits per parameter, GSQ improves average zero-shot accuracy by 4.76 and 4.14 points over the best scalar baseline, while trailing QTIP by only 1.33 and 1.68 points The gains come from better optimization of the discrete assignments rather than from a more flexible quantizer Compatibility and Scalability GSQ remains compatible with existing scalar inference kernels because it produces standard scalar layers, naturally supporting non-uniform bit allocation across layers Furthermore, the method scales to very large Mixture-of-Experts (MoE) models such as Kimi-K2.5, where vectorquantized methods are difficult to apply GSQ also scales to trillion-parameter-scale MoE models like Kimi-K2/2.5, where it is the first method to obtain low-bit, close-to-lossless quantization using a fully scalar, kernel-compatible format Practical Deployment Improvements GSQ can be applied to practical GGUF K-Quant checkpoints by starting from them and projecting the result back into the same deployment format, preserving compatibility with GGUF inference stacks This allows for end-to-end compression runtime analysis, showing that while larger models introduce a higher runtime overhead, GSQ remains practical at different model scales The method also shows that non-uniform bit allocation can yield practical accuracy–throughput trade-offs, with non-uniform 2.37 and 2.62 average bits per parameter configurations showing speedups up to 5.46× on NVIDIA L40s GPUs The results also suggest that the block-wise formulation is useful, as it improves the average score compared to naive block optimization Limitations and Future Directions A limitation is that GSQ introduces auxiliary logits during optimization, with memory overhead proportional to the number of candidate grid values considered per weight The method also depends on a good initialization, and the local-shift formulation assumes that most useful assignment changes remain near the initialized grid point Finally, the results do not imply that scalar quantization universally matches vector- or trellis-based methods, which remain more expressive at a fixed bit-width Future work should extend this idea beyond weight-only PTQ to activation and KV-cache quantization The authors also noted that the GPQA Diamond gap in Kimi-K2.5 results is partly due to calibration data composition, suggesting that calibration diversity can be important for low-bit quantization Experimental Setup Details The experiments were conducted on Llama-3.1-8B-Instruct and Llama-3.1-70BInstruct, as well as the MoE model Kimi-K2.5 For scalar experiments, GSQ uses a symmetric scalar quantizer with a group size of 128 Training details for dense Llama models involve block-wise optimization over 20 epochs, with specific staging schedules for query/key projections and value/output projections For Kimi models, only 10 epochs are used due to the decomposition across experts Evaluation focuses on standard zero-shot reasoning benchmarks for Llama models and long-context evaluations like OpenAI-MRCR for Kimi-K2.5 Reproducibility and Ethics The paper provides open access to the source code, including a publicly available repository at https://github.
Paper summary: Jane: Now we're moving into a summary of this paper, "GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling." We talked about how it addresses the accuracy wall that simple scalar quantization hits around three to four bits per parameter.
Tom: Right. So, the core thesis is that the gap between simple scalar quantization techniques and second generation vector or trellis methods isn't necessarily fundamental, but rather that a carefully optimized scalar quantizer can recover most of that accuracy.
Lu: They introduce GSQ as this post-training scalar quantization method which jointly learns the per coordinate grid assignments and the per group scales using a Gumbel-Softmax relaxation of the discrete grid. It aims to preserve the deployment friendly scalar structure while boosting accuracy substantially.
Meng: So, it's not just another rounding technique; it's learning how to map weights onto a grid in a smarter way during training.
Lalam: They also show that this approach works well even when starting from existing checkpoints like GGUF K Quant models, projecting the result back into that same deployment format without changing the output format.
Jane: And they provide concrete results showing improvements, specifically improving average zero-shot accuracy by four point seventy-six and four point fourteen points over best scalar baselines for Llama models at two bits per parameter.
Tom: Those are solid numbers compared to the existing scalar baselines we know. But what about the context of these results? Are they just on small models, or does this scale up?
Lu: The method scales well, and they evaluate it in the popular GGUF K Quant setting starting from Unsloth GGUF checkpoints for Qwen three point eight B. They show gains are especially large in the aggressive two point zero K setting.
Meng: I need to know how this translates to production costs at scale. If we're talking about trillion parameter models, how does GSQ handle that?
Lalam: GSQ is the first method they know of that obtains low bit, close to lossless quantization of trillion parameter Mixture of Experts models using a fully scalar, kernel compatible format.
Jane: So, the big picture here is that it shows we can get very accurate low precision quantization without needing the complexity of vector methods for massive models.
Tom: It seems like they are showing that you don't have to jump straight to complicated vector quantization if you can just optimize the scalar assignments this way.
Conclusion: Jane: We’ve covered a lot about "GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling." To wrap up, the paper argues that this method closes most of the gap between simple scalar quantization and second wave techniques while preserving the easy deployment structure.
Lu: The main implication is that this approach gives us a path to get very accurate low precision quantization for massive models using a fully scalar, kernel compatible format.
Tom: So what does this mean for someone who only listens to the show? It means we can deploy much larger language models locally with lower memory footprints while still maintaining high quality reasoning.
Meng: From an engineering standpoint, it’s good that it stays close to standard scalar post-training quantization approaches in terms of memory footprint and kernel compatibility.
Lalam: The GSQ method is a viable way forward for getting low bit, close to lossless quantization when dealing with trillion parameter models.
Jane: It seems like they've found a way to make the complex quantization techniques accessible by focusing on improving the discrete assignments within the scalar framework.
ETH Zurich · TU Wien · Red Hat AI
cs.CL, cs.LG
Submitted: 2026-04-20
Updated: 2026-11-10
Comments: Accepted at NeurIPS 2026. 27 pages. Code: https://github.com/IST-DASLab/GSQ
Code: https://github.com/IST-DASLab/GSQ
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 92/100
The gist: Quantization has become a standard tool for efficient LLM deployment, especially for local inference, where models are now routinely served at 2–3 bits per parameter The state of the art is…
Key concepts
- Scalar Quantization
- This is a method that reduces the precision of model weights by mapping them to a small, discrete set of values (like 2 bits). It's simple and compatible with standard hardware but often limits accuracy when pushing bit-widths very low.
- Gumbel-Softmax Relaxation
- This is a mathematical technique used during training. It allows the model to learn assignments over a discrete grid while still being differentiable. This lets the system find the best quantized settings without needing complex, hard discrete calculations during training.
- Post-Training Quantization (PTQ)
- PTQ refers to adjusting an already trained model's weights to use lower precision. GSQ is a PTQ method that reframes this process as a differentiable assignment problem, allowing it to find better quantization settings than previous methods.
Terminology
Summary
Quantization has become a standard tool for efficient LLM deployment, especially for local inference, where models are now routinely served at 2–3 bits per parameter The state of the art is currently split into simple scalar quantization techniques, such as GPTQ or AWQ, which are widely deployed but plateau in accuracy at 3–4 bits per parameter (bpp), and “second-generation” vector- or trellis-quantized methods, such as QTIP, GPTVQ and AQLM, which push the accuracy frontier but are notoriously hard to implement and to scale In this paper, we ask whether this gap is fundamental, or whether a carefully optimized scalar quantizer can recover most of it We answer in the affirmative, by introducing GSQ (Gumbel-Softmax Quantization), a post-training scalar quantization method which jointly learns the per-coordinate grid assignments and the per-group scales using a Gumbel-Softmax relaxation of the discrete grid
Pruning redundant reasoning tokens
GSQ is introduced as a post-training scalar quantization method that closes most of the gap between scalar quantization and second-wave techniques while preserving deployment-friendly scalar structure Its main form produces symmetric, group-wise, b-bit weights drawn from a small uniform grid, making it directly compatible with existing scalar inference kernels The method is designed to reformulate layer-wise reconstruction as a differentiable discrete-assignment problem
How the training signal is built
The key idea involves introducing a small set of trainable logits over candidate grid points for each weight coordinate, which are then relaxed using Gumbel-Softmax sampling to obtain a soft quantized weight The temperature parameter τ controls the sharpness of this distribution, annealing toward zero to yield a fully discrete quantized layer at the end of training For higher bit-widths, a local shift-based formulation is used, introducing only 5 logits corresponding to discrete shifts δi ∈ (−2, -1, 0, 1, 2) instead of the full 2b logits Initialization involves warm-starting from GPTQ or an existing GGUF checkpoint using a Gaussian-like prior around the GPTQ solution
Where the gains come from
Empirically, GSQ sets a new state of the art for scalar quantization at low bitwidths, improving average zero-shot accuracy by remarkable amounts over best scalar baselines For standard Llama models at 2 bits per parameter, GSQ improves average zero-shot accuracy by 4.76 and 4.14 points over the best scalar baseline, while trailing QTIP by only 1. p. 1 1-70B-Instruct,
at 2 bits per parameter, GSQ improves average zero-shot accuracy by a remarkable 4. 14
points, respectively over the best scalar baseline (EfficientQAT), and trails QTIP, the strongest and
most complex prior method, by only 1.Where the paper says it
Compatibility and Scalability
GSQ remains compatible with existing scalar inference kernels because it produces standard scalar layers, naturally supporting non-uniform bit allocation across layers Furthermore, the method scales to very large Mixture-of-Experts (MoE) models such as Kimi-K2.5, where vectorquantized methods are difficult to apply GSQ also scales to trillion-parameter-scale MoE models like Kimi-K2/2. p. 0 Finally, GSQ scales
to trillion-scale Mixture-of-Experts models such as Kimi-K2. 5, where vectorquantized methods are difficult to apply.Where the paper says it
Practical Deployment Improvements
GSQ can be applied to practical GGUF K-Quant checkpoints by starting from them and projecting the result back into the same deployment format, preserving compatibility with GGUF inference stacks This allows for end-to-end compression runtime analysis, showing that while larger models introduce a higher runtime overhead, GSQ remains practical at different model scales The method also shows that non-uniform bit allocation can yield practical accuracy–throughput trade-offs, with non-uniform 2.37 and 2.62 average bits per parameter configurations showing speedups up to 5. p. 7 37 average bits
per parameter. These configurations interpolate between the uniform 3-bit and 2-bit operating points.Where the paper says it
Limitations and Future Directions
A limitation is that GSQ introduces auxiliary logits during optimization, with memory overhead proportional to the number of candidate grid values considered per weight The method also depends on a good initialization, and the local-shift formulation assumes that most useful assignment changes remain near the initialized grid point Finally, the results do not imply that scalar quantization universally matches vector- or trellis-based methods, which remain more expressive at a fixed bit-width Future work should extend this idea beyond weight-only PTQ to activation and KV-cache quantization The authors also noted that the GPQA Diamond gap in Kimi-K2.5 results is partly due to calibration data composition, suggesting that calibration diversity can be important for low-bit quantization
Experimental Setup Details
The experiments were conducted on Llama-3.1-8B-Instruct and Llama-3. p. 0 Practically,
on the standard Llama-3. 1-8B/70B-Instruct models, GSQ closes most of the gap
between scalar quantization and the QTIP frontier at 2 and 3 bits, while using a
symmetric scalar grid with group-wise quantization, and thus remains compatible
with existing scalar inference kernels. 1-8B-Instruct and Llama-3. 1-70BInstruct [28], as well as the MoE model Kimi-K2.Where the paper says it
Where the paper says it
Reproducibility and Ethics
The paper provides open access to the source code, including a publicly available repository at https://github.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Scalar quantization can close most of the accuracy gap between scalar quantization and second-generation vector- or trellis-quantized methods at 2 and 3 bits, matching or surpassing QTIP on large models. This allows for high accuracy while remaining compatible with existing scalar inference kernels, making it a practical solution for local deployment formats like GGUF K-Quant checkpoints.
-
The method can improve the accuracy of standard symmetric group-wise scalar quantization at 2 and 3 bits by
jointly learning the per-coordinate grid assignments and the per-group scales using a Gumbel-Softmax relaxation.
This approach is achieved by treating layer-wise reconstruction as adifferentiable discrete-assignment problem
optimized jointly. -
The system can be applied to trillion-scale Mixture-of-Experts models such as Kimi K2.5, where vectorquantized methods are difficult to apply. This capability allows for the
low-bit, close-to-lossless quantization of trillion-parameter MoE models using a fully scalar, kernel-compatible format.
-
The system can be adapted to improve existing GGUF K-Quant checkpoints by starting from them and optimizing their discrete assignments while projecting the result back into the same format. This ensures that the optimized model
remains compatible with standard GGUF inference stacks.
-
Inference speed can be significantly increased, achieving up to 6.2x speedup over BF16 for uniform 2-bit quantization and up to 5.46x for non-uniform settings on Llama models using optimized kernels like Humming kernels.
Abstract
Quantization has become a standard tool for efficient LLM deployment, especially for local inference, where models are now routinely served at 2-3 bits per parameter. The state of the art is currently split into simple scalar quantization techniques, such as GPTQ or AWQ, which are widely deployed but plateau in accuracy at 3-4 bits per parameter (bpp), and "second-generation" vector- or trellis-quantized methods, such as QTIP, GPTVQ and AQLM, which push the accuracy frontier but are notoriously hard to implement and to scale. In this paper, we ask whether this gap is fundamental, or whether a carefully optimized scalar quantizer can recover most of it. We answer in the affirmative, by introducing GSQ (Gumbel-Softmax Quantization), a post-training scalar quantization method which jointly learns the per-coordinate grid assignments and the per-group scales using a Gumbel-Softmax relaxation of the discrete grid. GSQ matches the cardinality of the relaxation to the small number of levels available in the target bit-width regime (e.g., 3-8 levels for ternary and 3 bpp, respectively), making optimization tractable. Practically, on the standard Llama-3.1-8B/70B-Instruct models, GSQ closes most of the gap between scalar quantization and the QTIP frontier at 2 and 3 bits, while using a symmetric scalar grid with group-wise quantization, and thus remains compatible with existing scalar inference kernels. We further show that the same discrete-assignment optimization can be applied to practical GGUF K-Quant checkpoints: starting from publicly released GGUF models, GSQ improves accuracy while projecting the result back into the same deployment format. Finally, GSQ scales to trillion-scale Mixture-of-Experts models such as Kimi-K2.5, where vector-quantized methods are difficult to apply. The source code is publicly available at https://github.com/IST-DASLab/GSQ.
Sources
- DB-LLM: Accurate Dual-Binarization for Efficient LLMs
- TernaryLLM: Ternarized Large Language Model
- Symbolic Discovery of Optimization Algorithms
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Differentiable Model Compression via Pseudo Quantization Noise
- 8-bit Optimizers via Block-wise Quantization
- LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
- SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression
- STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMs
- Extreme Compression of Large Language Models via Additive Quantization
- Learned Step Size Quantization
- QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- EAQuant: Enhancing Post-Training Quantization for MoE Models via Expert-Aware Optimization
- The Llama 3 Herd of Models
- OpenThoughts: Data Recipes for Reasoning Models
- Model Compression with Exact Budget Constraints via Riemannian Manifolds
- MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance
- Tequila: Trapping-free Ternary Quantization for Large Language Models
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data