SHq: Learnable Grids for LLM Weight Quantization
cs.LG
Submitted: 2026-11-02
Updated: 2026-11-02
License: http://creativecommons.org/licenses/by/4.0/
The gist: Block-scaled lookup-table (LUT) formats, which represent weights as indices into a small value set (the grid), have become standard for low-bit LLM quantization…
Terminology
Abstract
Block-scaled lookup-table (LUT) formats, which represent weights as indices into a small value set (the grid), have become standard for low-bit LLM quantization, ranging from NF4 and BOF4 to the 3-bit LUT instruction in NVIDIA Rubin. While these formats leave the grid definition flexible, existing implementations rely on fixed grids derived either from numerical data types (INT, FP) or assumed weight distributions (NF, BOF). We demonstrate that no single fixed grid is universally optimal; the best fixed grid varies with bit-width: BOFS at 2 bits, NF at 4 and 5 bits, and INT at 6 bits. We introduce SHq, a closed-form family of grids parameterized by two critical degrees of freedom: a companding strength α that interpolates level spacing from uniform to floating-point-like, and an offset c that introduces asymmetry. This offset allows the sign bit of the FP8 block scale, otherwise unused, to select between the grid and its mirror image, up to doubling the number of distinct values available per block. SHq modifies only the lookup table contents and serves as a drop-in replacement for the grid in any such format. Across five models ranging from 3B to 35B parameters, including a mixture-of-experts architecture, evaluated at 2-6 bits using RTN and GPTQ, SHq achieves the lowest KL divergence in every setting at 3 and 4 bits and in most settings at 2, 5 and 6 bits. It outperforms the best fixed grid by up to 34% and remains superior under mixed-precision configurations. At 4.5 bits per weight, the SHq format reduces the KL divergence of NVFP4 by 41-67% while improving downstream accuracy. Grid families incorporating both spacing and offset parameters perform comparably, whereas those lacking an offset exhibit clearly higher error, confirming that these two degrees of freedom drive the observed gains. We therefore propose SHq as the standard grid for block-scaled lookup-table formats.
Sources
- Lessons from the Trenches on Reproducible Evaluation of Language Models
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Signed Symmetric Quantization for Few-Bit Integers
- Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling
- Adaptive Block-Scaled Data Types
- GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling
- Microscaling Data Formats for Deep Learning
- LO-BCQ: Block Clustered Quantization for 4-bit (W4A4) LLM Inference
- CubicQuant: Parametric Non-Uniform Codebooks for High-Throughput LLM Inference with 1-8-Bit Weights
- Model Compression with Exact Budget Constraints via Riemannian Manifolds
- A Hardware-Aware, Per-Layer Methodology for Post-Training Quantization of Large Language Models
- ScaleSweep: Accurate NVFP4 Post-Training Quantization of LLMs via Block Scale Initialization
- FP8 Formats for Deep Learning
- Pretraining Large Language Models with NVFP4
- FP8 versus INT8 for efficient deep learning inference
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis