When Lower Reconstruction Loss Hurts: Distributionally Robust Refinement for Low-Bit LLM Quantization
Weight-only post-training quantization (PTQ) often prioritizes minimizing reconstruction loss on calibration data, but this doesn't guarantee better performance on new tasks. The authors introduce Distributionally Robust Quantization (DRQ), a method that refines quantized weights by minimizing the worst-case reconstruction loss across a constrained set of input activation distributions. This refinement improves model performance without adding any inference overhead.