Adam Converges in Nonsmooth Nonconvex Optimization
Zijian Liu
math.OC, cs.LG, stat.ML
Submitted: 2026-06-21
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration
- A Novel Convergence Analysis for Algorithms of the Adam Family
- Adam: A Method for Stochastic Optimization
- DeepSeek-V3 Technical Report
- Can Adaptive Gradient Methods Converge under Heavy-Tailed Noise? A Case Study of AdaGrad
- In-Expectation Convergence of Stochastic Gradient Methods under Heavy-Tailed Noise
- Online Learning: A Modern Introduction Using Convex Optimization
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- Dynamic Regret via Discounted-to-Dynamic Reduction with Applications to Curved Losses and Adam Optimizer
- StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models
Related papers
- Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise
- Adam-HNAG: A Convergent Reformulation of Adam with Accelerated Rate
- Incremental Learning in Mirror Flows
- Beyond Aggregate Drift: Optimal Client-Specific Windows in Federated Learning
- Lower Bounds for Randomized Linear-Oracle Optimization over Strongly Convex Sets
- Second-Order Optimization with Asynchronous Curvature