Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantization
The paper addresses errors in quantizing AdamW optimizer states by focusing on rounding rules rather than just state space reconstruction. It proposes two 4-bit configurations, ZIP-SR and ZE-EDEN, which use specific stochastic rounding and calibration techniques. These methods successfully reduce the validation loss gap compared to standard 4-bit AdamW across pretraining and fine-tuning experiments.