Decentralized SGD under Heavy-Tailed Noise: Optimal Convergence Rates and the Role of Gradient Clipping
The paper introduces clipped decentralized SGD (DSGD) to handle heavy-tailed noise in decentralized optimization. It proves that DSGD achieves order-optimal convergence rates and a linear speed-up for smooth non-convex costs under bounded p-th moment noise, where $p o 2$. This demonstrates that gradient clipping retains crucial magnitude information, unlike normalization which can fail in decentralized settings.