High-Probability Convergence of Clipped SGD under Heavy-Tailed Noise and (L 0,L 1) -Smoothness
math.OC, cs.LG
Submitted: 2025-05-27
Updated: 2026-09-29
Terminology
Sources
- Complexities of Armijo-like algorithms in Deep Learning context
- Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed
- Universal fast gradient method for stochastic composit optimization problems
- Simple and optimal high-probability bounds for strongly-convex stochastic gradient descent
- Differentially Private Clipped-SGD: High-Probability Convergence with Arbitrary Clipping Level
- Error Feedback under $(L_0,L_1)$-Smoothness: Normalization and Momentum
- Efficient Distributed Optimization under Heavy-Tailed Noise
- A High Probability Analysis of Adaptive SGD with Momentum
- Linear Convergence Rate in Convex Setup is Possible! Gradient Descent Method Variants under $(L_0,L_1)$-Smoothness
- LLaMA: Open and Efficient Foundation Language Models
- Revisiting Stochastic Proximal Point Methods: Generalized Smoothness and Similarity
- Toward a Unified Theory of Gradient Descent under Generalized Smoothness
- Provable Adaptivity of Adam under Non-uniform Smoothness
- Adaptive Gradient Normalization and Independent Sampling for (Stochastic) Generalized-Smooth Optimization
- Convergence Analysis of Stochastic Accelerated Gradient Methods for Generalized Smooth Optimizations
- Mirror Descent Under Generalized Smoothness
Related papers
- Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise
- Adam-HNAG: A Convergent Reformulation of Adam with Accelerated Rate
- Incremental Learning in Mirror Flows
- Second-Order Optimization with Asynchronous Curvature
- Online Control via Counterfactual Tracking
- Asynchronous Replanning in Two Population Linear Quadratic Mean Field Games: Information Requirements and Stability