OSFP4: Joint Optimization of Diagonal Smoothing and Block Scales for NVFP4 Quantization
cs.AI, cs.IT, math.IT
Submitted: 2026-10-06
Updated: 2026-10-06
Code: https://github.com/neriahbd/OSFP4
License: http://creativecommons.org/licenses/by/4.0/
The gist: NVFP4 is an attractive datatype for large language model (LLM) inference, offering compact storage and native tensor-core acceleration.
Terminology
Abstract
NVFP4 is an attractive datatype for large language model (LLM) inference, offering compact storage and native tensor-core acceleration. However, preserving accuracy using NVFP4 requires careful quantization. In this work we develop a novel quantization scheme called Optimized Smoothing and Scaling for NVFP4 (OSFP4). For each linear projection it uses a diagonal smoothing matrix whose entries are optimized to minimize the squared matrix-product quantization error under NVFP4, taking into account the rounding procedure that is used (either round-to-nearest, or GPTQ-style successive interference cancellation). This requires performing joint optimization on the smoothing entries as well as the block scales, which is facilitated by analyzing a multiplicative-dither FP4 quantizer instead of the fixed deterministic one. Experiments show that OSFP4 achieves the highest average accuracy among the evaluated competitors in the corresponding quantization settings, while retaining approximately 94-97% of vendor NVFP4 prefill throughput on the measured workloads. Our code is available in https://github.com/neriahbd/OSFP4
Sources
- QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
- SOAR: Scale Optimization for Accurate Reconstruction in NVFP4 Quantization
- QuIP: 2-Bit Quantization of Large Language Models With Guarantees
- WUSH: Near-Optimal Adaptive Transforms for LLM Quantization
- Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- Search Your Block Floating Point Scales!
- WaterSIC: Information-Theoretically (Near) Optimal Linear Layer Quantization
- AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- ScaleSweep: Accurate NVFP4 Post-Training Quantization of LLMs via Block Scale Initialization
- SpinQuant: LLM quantization with learned rotations
- High-Rate Quantized Matrix Multiplication II
- NestQuant: Nested Lattice Quantization for Matrix Products and LLMs
- OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models
- FlatQuant: Flatness Matters for LLM Quantization
- H-Scale: Hessian-Guided Scale Refinement for NVFP4 Sub-Byte LLM Inference
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems