Bridging the Gap Between Homogeneous and Heterogeneous Asynchronous Optimization Is Surprisingly Difficult

arXiv:2609.17483 · math.OC, cs.LG · Submitted 2026-09-15 · Read on arXiv

math.OC, cs.LG

Submitted: 2026-09-15

Updated: 2026-09-15

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Accuracy — 5× in this paper · explained in 1 episode
• Federated Learning — 3× in this paper · explained in 15 episodes
• Adam — 2× in this paper · explained in 2 episodes
• Generalization — 2× in this paper · explained in 13 episodes
• Convergence Guarantees — 1× in this paper · explained in 3 episodes
• Data Heterogeneity — 1× in this paper · explained in 3 episodes
• Federated Learning (FL) — 1× in this paper · explained in 7 episodes

Sources

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Federated Learning: Strategies for Improving Communication Efficiency
• Fast Convergence of Stochastic Gradient Descent under a Strong Growth Condition
• Delay-adaptive step-sizes for asynchronous learning
• Staleness-aware Async-SGD for Distributed Deep Learning

Related papers

• Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise
• Adam-HNAG: A Convergent Reformulation of Adam with Accelerated Rate
• Incremental Learning in Mirror Flows
• Beyond Aggregate Drift: Optimal Client-Specific Windows in Federated Learning
• Lower Bounds for Randomized Linear-Oracle Optimization over Strongly Convex Sets
• Second-Order Optimization with Asynchronous Curvature