Provable Benefit of SignGD: A Minimal Model Under Heavy-Tailed Class Imbalance
cs.LG, cs.AI
Submitted: 2025-11-30
Updated: 2026-09-30
Terminology
Sources
- Old Optimizer, New Norm: An Anthology
- Revisiting the Adam-SGD Gap in LLM Pre-Training: The Role of Large Effective Learning Rates
- Provable Complexity Improvement of AdaGrad over SGD: Upper and Lower Bounds in Stochastic Non-Convex Optimization
- Adam: A Method for Stochastic Optimization
- Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization
- Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Law
- Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be
- Decoupled Weight Decay Regularization
- Toward Understanding Why Adam Converges Faster Than SGD for Transformers
- Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling
- Structured Preconditioners in Adaptive Optimization: A Unified Analysis
- Sign-Based Optimizers Are Effective Under Heavy-Tailed Noise
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks