Do We Need Asynchronous SGD? On the Near-Optimality of Synchronous Methods
The paper investigates whether synchronous SGD methods can remain nearly optimal even when workers have different computation speeds (heterogeneous computation). It shows that m-Synchronous SGD, which ignores stragglers, maintains near-optimal time complexity under various assumptions. This suggests synchronous methods are often sufficient for modern settings.