NAG-GS: A Semi-Implicit Stochastic Optimizer with Large-Step Stability

arXiv:2209.14937 · math.OC, cs.LG, cs.NA, math.NA · Submitted 2022-09-29 · Read on arXiv

math.OC, cs.LG, cs.NA, math.NA

Submitted: 2022-09-29

Updated: 2026-10-27

Terminology

• Transformer — 2× in this paper · explained in 2 episodes
• Transformers — 2× in this paper · explained in 2 episodes
• Benchmark — 1× in this paper · explained in 2 episodes
• Gradient Flow — 1× in this paper · explained in 2 episodes
• Hyperparameters — 1× in this paper · explained in 2 episodes
• Stochastic Gradient Descent-based — 1× in this paper
• Weight Decay — 1× in this paper · explained in 2 episodes

Related papers

• Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise
• Adam-HNAG: A Convergent Reformulation of Adam with Accelerated Rate
• Incremental Learning in Mirror Flows
• Beyond Aggregate Drift: Optimal Client-Specific Windows in Federated Learning
• Lower Bounds for Randomized Linear-Oracle Optimization over Strongly Convex Sets
• Second-Order Optimization with Asynchronous Curvature