Learning Rate Decay Can Exponentially Accelerate SGD for Global Optimization of Nonconvex Functions
math.OC, cs.DS, cs.LG
Submitted: 2026-11-09
Updated: 2026-11-09
Project page: https://chewisinho.github.io/main.pdf
Terminology
Sources
- Nearly $d$-Linear Convergence Bounds for Diffusion Models via Stochastic Localization
- Mechanisms for Quantum Advantage in Global Optimization of Nonconvex Functions
- Weak Poincar'e Inequalities, Simulated Annealing, and Sampling from Spherical Spin Glasses
- Better Theory for SGD in the Nonconvex World
- The large learning rate phase of deep learning: the catapult mechanism
- Stochastic Gradient Descent on Nonconvex Functions with General Noise Models
- Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization
- Fast Convergence of Stochastic Gradient Descent under a Strong Growth Condition
- Direction Matters: On the Implicit Bias of Stochastic Gradient Descent with Moderate Learning Rate
- How Does Learning Rate Decay Help Modern Neural Networks?
- SGD Converges to Global Minimum in Deep Learning via Star-convex Path
Related papers
- An Accelerated Proximal Gradient Method for Bures-Wasserstein Space
- FOMTools: Development and AI-Assisted Code Generation for First-Order Methods
- Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise
- Dynamic Service Recommendation with Congestion-Dependent Joining: Near-Optimal and Constant-Factor Approximation Algorithms
- Integrating Learning and Optimization for Open Transport Capacity Pooling under Disruption
- Zeroth-Order Gradient Estimators under Decision-Dependent Noise