Beyond Dense Adam States: Adaptive Log-Space Quantization for Memory-Efficient Optimizers
cs.LG
Submitted: 2026-08-23
Updated: 2026-09-01
Comments: 17 pages, 5 figures, 7 tables. Substantially revised framing and evidence; added a matched adaptive-log comparison, multi-seed validation, and the completed Adafactor block-size/grouping grid. Updated reproducibility mapping and evaluation scope. Code: https://github.com/yanfeiwong/adafactor-8bit. Artifacts: https://github.com/yanfeiwong/al-quantization
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- STQuant: Spatio-Temporal Adaptive Framework for Optimizer Quantization in Large Multimodal Model Training
- Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resets
- COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Training
- Pushing the Limits of Low-Bit Optimizers: A Focus on EMA Dynamics
- TinyLlama: An Open-Source Small Language Model
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks