Understanding the Weight Averaging Mechanism in LLM Training for Post-Training Quantization
cs.LG
Submitted: 2026-10-04
Updated: 2026-10-04
Code: https://github.com/MOFA-LAB/weight-averaging-for-ptq
Terminology
Sources
- When, Where and Why to Average Weights?
- EMA Without the Lag: Bias-Corrected Iterate Averaging Schemes
- Extreme Compression of Large Language Models via Additive Quantization
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
- Averaging Weights Leads to Wider Optima and Better Generalization
- Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay
- GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibration
- Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining
- Decoupled Weight Decay Regularization
- Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefits
- Training trajectories, mini-batch losses and the curious role of the learning rate
- Early Weight Averaging meets High Learning Rates for LLM Pre-training
- FlatQuant: Flatness Matters for LLM Quantization
- Gemma 3 Technical Report
- Kimi K2: Open Agentic Intelligence
- Learning Dynamics in Continual Pre-Training for Large Language Models
- Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting
- BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
- Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuning
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks