Breaking Precision Walls in First-, Second-, and Zeroth-Order Optimizers: One Constant, Set in Advance
cs.LG
Submitted: 2026-11-07
Updated: 2026-11-07
Terminology
Sources
- Disentangling Adaptive Gradient Methods from Learning Rates
- Scalable Second Order Optimization for Deep Learning
- A Dynamical Model of Neural Scaling Laws
- ZO-AdaMM: Zeroth-Order Adaptive Momentum Method for Black-Box Optimization
- On Empirical Comparisons of Optimizers for Deep Learning
- Adaptive Gradient Methods at the Edge of Stability
- Understanding Optimization in Deep Learning with Central Flows
- Benchmarking Neural Network Training Algorithms
- On Adaptivity in Zeroth-Order Optimization
- Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditioner
- Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectory
- Scaling Exponents Across Parameterizations and Optimizers
- Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis
- Shampoo: Preconditioned Stochastic Tensor Optimization
- Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensing
- Accelerating Neural Network Training: An Analysis of the AlgoPerf Competition
- DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Method
- Adam: A Method for Stochastic Optimization
- Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models
- evosax: JAX-based Evolution Strategies
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis