ZEMA: confidence-calibrated signed descent with only one state
cs.LG
Submitted: 2026-11-03
Updated: 2026-11-03
Terminology
Sources
- Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients
- Old Optimizer, New Norm: An Anthology
- signSGD: Compressed Optimisation for Non-Convex Problems
- BackPACK: Packing more into backprop
- Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Law
- Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be
- Adaptive Optimization with Examplewise Gradients
- Muon$^2$: Boosting Muon via Adaptive Second-Moment Preconditioning
- In Search of Adam's Secret Sauce
- Adaptive Optimization via Momentum on Variance-Normalized Gradients
- Benchmarking Optimizers for Large Language Model Pretraining
- AdaMuon: Adaptive Muon Optimizer
- Understanding Transformer Optimization via Gradient Heterogeneity
- Qwen3 Technical Report
- MARS: Unleashing the Power of Variance Reduction for Training Large Models
- AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updates
- Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentum
- Why Transformers Need Adam: A Hessian Perspective
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
- Deconstructing What Makes a Good Optimizer for Language Models
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis