ResQAdam: Residual-Preserving Adaptive Optimization for Memory-Efficient Quantization Aware Training
cs.LG, cs.CL
Submitted: 2026-11-06
Updated: 2026-11-06
Code: https://github.com/LiqunMa/ResQAdam
Terminology
Sources
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- PACT: Parameterized Clipping Activation for Quantized Neural Networks
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Learned Step Size Quantization
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- Decoupled Weight Decay Regularization
- The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
- QuEST: Stable Training of LLMs with 1-Bit Weights and Activations
- LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics
- CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training
- Qwen3 Technical Report
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- BitNet: Scaling 1-bit Transformers for Large Language Models
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
- FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Training
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis