Clapping: Removing Per-sample Storage for Pipeline Parallel Distributed Optimization with Communication Compression

arXiv:2509.19029 · math.OC, stat.ML · Submitted 2025-09-23 · Read on arXiv

math.OC, stat.ML

Submitted: 2025-09-23

Updated: 2026-11-02

Comments: 71 pages

License: http://creativecommons.org/licenses/by/4.0/

The gist: Pipeline-parallel distributed optimization is essential for large-scale machine learning but is challenged by significant communication overhead from transmitting high-dimensional activations and…

Terminology

Abstract

Pipeline-parallel distributed optimization is essential for large-scale machine learning but is challenged by significant communication overhead from transmitting high-dimensional activations and gradients between workers. Existing approaches often depend on impractical unbiased gradient assumptions or incur sample-size memory overhead. This paper introduces Clapping, a Communication compression algorithm with LAzy samPling for Pipeline-parallel learnING. Clapping adopts a lazy sampling strategy that reuses data samples across steps, breaking sample-wise memory barrier and supporting convergence in few-epoch or online regimes. Clapping comprises two variants including Clapping-FC and Clapping-FU, both of which achieve convergence without unbiased assumption for compressed gradient, effectively addressing compression error propagation in multi-worker settings. Numerical experiments validate the performance of Clapping across different tasks.

Sources

Related papers