GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling

summary

The gist

Quantization has become a standard tool for efficient LLM deployment, especially for local inference, where models are now routinely served at 2–3 bits per parameter The state of the art is…

In short

The paper introduces GSQ, a post-training scalar quantization method that bridges the gap between simple scalar quantization and complex vector methods. GSQ uses a Gumbel-Softmax relaxation to jointly learn weight assignments and scales. It achieves state-of-the-art accuracy for low bitwidths while maintaining compatibility with existing scalar inference kernels.

Key concepts

Scalar Quantization
This is a method that reduces the precision of model weights by mapping them to a small, discrete set of values (like 2 bits). It's simple and compatible with standard hardware but often limits accuracy when pushing bit-widths very low.
Gumbel-Softmax Relaxation
This is a mathematical technique used during training. It allows the model to learn assignments over a discrete grid while still being differentiable. This lets the system find the best quantized settings without needing complex, hard discrete calculations during training.
Post-Training Quantization (PTQ)
PTQ refers to adjusting an already trained model's weights to use lower precision. GSQ is a PTQ method that reframes this process as a differentiable assignment problem, allowing it to find better quantization settings than previous methods.

Terminology used across episodes

This episode discusses

The paper

GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling · Read on arXiv

ETH Zurich · TU Wien · Red Hat AI

Quantization has become a standard tool for efficient LLM deployment, especially for local inference, where models are now routinely served at 2-3 bits per parameter. The state of the art is currently split into simple scalar quantization techniques, such as GPTQ or AWQ, which are widely deployed but plateau in accuracy at 3-4 bits per parameter (bpp), and "second-generation" vector- or trellis-quantized methods, such as QTIP, GPTVQ and AQLM, which push the accuracy frontier but are notoriously hard to implement and to scale. In this paper, we ask whether this gap is fundamental, or whether a carefully optimized scalar quantizer can recover most of it. We answer in the affirmative, by introducing GSQ (Gumbel-Softmax Quantization), a post-training scalar quantization method which jointly learns the per-coordinate grid assignments and the per-group scales using a Gumbel-Softmax relaxation of the discrete grid. GSQ matches the cardinality of the relaxation to the small number of levels available in the target bit-width regime (e.g., 3-8 levels for ternary and 3 bpp, respectively), making optimization tractable. Practically, on the standard Llama-3.1-8B/70B-Instruct models, GSQ closes most of the gap between scalar quantization and the QTIP frontier at 2 and 3 bits, while using a symmetric scalar grid with group-wise quantization, and thus remains compatible with existing scalar inference kernels. We further show that the same discrete-assignment optimization can be applied to practical GGUF K-Quant checkpoints: starting from publicly released GGUF models, GSQ improves accuracy while projecting the result back into the same deployment format. Finally, GSQ scales to trillion-scale Mixture-of-Experts models such as Kimi-K2.5, where vector-quantized methods are difficult to apply. The source code is publicly available at https://github.com/IST-DASLab/GSQ.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling".

Jane: Here's the gist of the paper. Quantization has become a standard tool for efficient LLM deployment, especially for local inference, where models are now routinely served at 2–3 bits per parameter The state of the art is currently split into simple scalar quantization techniques, such as GPTQ or AWQ, which are widely deployed but plateau in accuracy at 3–4 bits per parameter (bpp), and “second-generation” vector- or trellis-quantized methods, such as QTIP, GPTVQ and AQLM, which push the accuracy frontier but are notoriously hard to implement and to scale In this paper, we ask whether this gap is fundamental, or whether a carefully optimized scalar quantizer can recover most of it We answer in the affirmative, by introducing GSQ (Gumbel-Softmax Quantization), a post-training scalar quantization method which jointly learns the per-coordinate grid assignments and the per-group scales using a Gumbel-Softmax relaxation of the discrete grid Pruning redundant reasoning tokens GSQ is introduced as a post-training scalar quantization method that closes most of the gap between scalar quantization and second-wave techniques while preserving deployment-friendly scalar structure Its main form produces symmetric, group-wise, b-bit weights drawn from a small uniform grid, making it directly compatible with existing scalar inference kernels The method is designed to reformulate layer-wise reconstruction as a differentiable discrete-assignment problem How the training signal is built The key idea involves introducing a small set of trainable logits over candidate grid points for each weight coordinate, which are then relaxed using Gumbel-Softmax sampling to obtain a soft quantized weight The temperature parameter τ controls the sharpness of this distribution, annealing toward zero to yield a fully discrete quantized layer at the end of training For higher bit-widths, a local shift-based formulation is used, introducing only 5 logits corresponding to discrete shifts δi ∈ (−2, -1, 0, 1, 2) instead of the full 2b logits Initialization involves warm-starting from GPTQ or an existing GGUF checkpoint using a Gaussian-like prior around the GPTQ solution Where the gains come from Empirically, GSQ sets a new state of the art for scalar quantization at low bitwidths, improving average zero-shot accuracy by remarkable amounts over best scalar baselines For standard Llama models at 2 bits per parameter, GSQ improves average zero-shot accuracy by 4.76 and 4.14 points over the best scalar baseline, while trailing QTIP by only 1.33 and 1.68 points The gains come from better optimization of the discrete assignments rather than from a more flexible quantizer Compatibility and Scalability GSQ remains compatible with existing scalar inference kernels because it produces standard scalar layers, naturally supporting non-uniform bit allocation across layers Furthermore, the method scales to very large Mixture-of-Experts (MoE) models such as Kimi-K2.5, where vectorquantized methods are difficult to apply GSQ also scales to trillion-parameter-scale MoE models like Kimi-K2/2.5, where it is the first method to obtain low-bit, close-to-lossless quantization using a fully scalar, kernel-compatible format Practical Deployment Improvements GSQ can be applied to practical GGUF K-Quant checkpoints by starting from them and projecting the result back into the same deployment format, preserving compatibility with GGUF inference stacks This allows for end-to-end compression runtime analysis, showing that while larger models introduce a higher runtime overhead, GSQ remains practical at different model scales The method also shows that non-uniform bit allocation can yield practical accuracy–throughput trade-offs, with non-uniform 2.37 and 2.62 average bits per parameter configurations showing speedups up to 5.46× on NVIDIA L40s GPUs The results also suggest that the block-wise formulation is useful, as it improves the average score compared to naive block optimization Limitations and Future Directions A limitation is that GSQ introduces auxiliary logits during optimization, with memory overhead proportional to the number of candidate grid values considered per weight The method also depends on a good initialization, and the local-shift formulation assumes that most useful assignment changes remain near the initialized grid point Finally, the results do not imply that scalar quantization universally matches vector- or trellis-based methods, which remain more expressive at a fixed bit-width Future work should extend this idea beyond weight-only PTQ to activation and KV-cache quantization The authors also noted that the GPQA Diamond gap in Kimi-K2.5 results is partly due to calibration data composition, suggesting that calibration diversity can be important for low-bit quantization Experimental Setup Details The experiments were conducted on Llama-3.1-8B-Instruct and Llama-3.1-70BInstruct, as well as the MoE model Kimi-K2.5 For scalar experiments, GSQ uses a symmetric scalar quantizer with a group size of 128 Training details for dense Llama models involve block-wise optimization over 20 epochs, with specific staging schedules for query/key projections and value/output projections For Kimi models, only 10 epochs are used due to the decomposition across experts Evaluation focuses on standard zero-shot reasoning benchmarks for Llama models and long-context evaluations like OpenAI-MRCR for Kimi-K2.5 Reproducibility and Ethics The paper provides open access to the source code, including a publicly available repository at https://github.

Paper summary: Jane: Now we're moving into a summary of this paper, "GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling." We talked about how it addresses the accuracy wall that simple scalar quantization hits around three to four bits per parameter.

Tom: Right. So, the core thesis is that the gap between simple scalar quantization techniques and second generation vector or trellis methods isn't necessarily fundamental, but rather that a carefully optimized scalar quantizer can recover most of that accuracy.

Lu: They introduce GSQ as this post-training scalar quantization method which jointly learns the per coordinate grid assignments and the per group scales using a Gumbel-Softmax relaxation of the discrete grid. It aims to preserve the deployment friendly scalar structure while boosting accuracy substantially.

Meng: So, it's not just another rounding technique; it's learning how to map weights onto a grid in a smarter way during training.

Lalam: They also show that this approach works well even when starting from existing checkpoints like GGUF K Quant models, projecting the result back into that same deployment format without changing the output format.

Jane: And they provide concrete results showing improvements, specifically improving average zero-shot accuracy by four point seventy-six and four point fourteen points over best scalar baselines for Llama models at two bits per parameter.

Tom: Those are solid numbers compared to the existing scalar baselines we know. But what about the context of these results? Are they just on small models, or does this scale up?

Lu: The method scales well, and they evaluate it in the popular GGUF K Quant setting starting from Unsloth GGUF checkpoints for Qwen three point eight B. They show gains are especially large in the aggressive two point zero K setting.

Meng: I need to know how this translates to production costs at scale. If we're talking about trillion parameter models, how does GSQ handle that?

Lalam: GSQ is the first method they know of that obtains low bit, close to lossless quantization of trillion parameter Mixture of Experts models using a fully scalar, kernel compatible format.

Jane: So, the big picture here is that it shows we can get very accurate low precision quantization without needing the complexity of vector methods for massive models.

Tom: It seems like they are showing that you don't have to jump straight to complicated vector quantization if you can just optimize the scalar assignments this way.

Conclusion: Jane: We’ve covered a lot about "GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling." To wrap up, the paper argues that this method closes most of the gap between simple scalar quantization and second wave techniques while preserving the easy deployment structure.

Lu: The main implication is that this approach gives us a path to get very accurate low precision quantization for massive models using a fully scalar, kernel compatible format.

Tom: So what does this mean for someone who only listens to the show? It means we can deploy much larger language models locally with lower memory footprints while still maintaining high quality reasoning.

Meng: From an engineering standpoint, it’s good that it stays close to standard scalar post-training quantization approaches in terms of memory footprint and kernel compatibility.

Lalam: The GSQ method is a viable way forward for getting low bit, close to lossless quantization when dealing with trillion parameter models.

Jane: It seems like they've found a way to make the complex quantization techniques accessible by focusing on improving the discrete assignments within the scalar framework.

More episodes

← Home