Clapping: Removing Per-sample Storage for Pipeline Parallel Distributed Optimization with Communication Compression
math.OC, stat.ML
Submitted: 2025-09-23
Updated: 2026-11-02
Comments: 71 pages
License: http://creativecommons.org/licenses/by/4.0/
The gist: Pipeline-parallel distributed optimization is essential for large-scale machine learning but is challenged by significant communication overhead from transmitting high-dimensional activations and…
Terminology
Abstract
Pipeline-parallel distributed optimization is essential for large-scale machine learning but is challenged by significant communication overhead from transmitting high-dimensional activations and gradients between workers. Existing approaches often depend on impractical unbiased gradient assumptions or incur sample-size memory overhead. This paper introduces Clapping, a Communication compression algorithm with LAzy samPling for Pipeline-parallel learnING. Clapping adopts a lazy sampling strategy that reuses data samples across steps, breaking sample-wise memory barrier and supporting convergence in few-epoch or online regimes. Clapping comprises two variants including Clapping-FC and Clapping-FU, both of which achieve convergence without unbiased assumption for compressed gradient, effectively addressing compression error propagation in multi-worker settings. Numerical experiments validate the performance of Clapping across different tasks.
Sources
- Training Deep Nets with Sublinear Memory Cost
- Momentum Benefits Non-IID Federated Learning Simply and Provably
- The Llama 3 Herd of Models
- Direct Language Model Alignment from Online AI Feedback
- TAH-QUANT: Effective Activation Quantization in Pipeline Parallelism over Slow Network
- Lower Bounds and Accelerated Algorithms in Distributed Stochastic Optimization with Communication Compression
- Adam: A Method for Stochastic Optimization
- Decentralized Deep Learning with Arbitrary Communication Compression
- Federated Optimization:Distributed Optimization Beyond the Datacenter
- BERT: A Review of Applications in Natural Language Processing and Understanding
- DeepSeek-V3 Technical Report
- RoBERTa: A Robustly Optimized BERT Pretraining Approach
- Decoupled Weight Decay Regularization
- Pointer Sentinel Mixture Models
- Scaling Language Models: Methods, Analysis & Insights from Training Gopher
- Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism
- Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
- Hybrid RL: Using Both Offline and Online Data Can Make RL Efficient
- Local SGD Converges Fast and Communicates Little
- Llama 2: Open Foundation and Fine-Tuned Chat Models
Related papers
- An Accelerated Proximal Gradient Method for Bures-Wasserstein Space
- FOMTools: Development and AI-Assisted Code Generation for First-Order Methods
- Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise
- Dynamic Service Recommendation with Congestion-Dependent Joining: Near-Optimal and Constant-Factor Approximation Algorithms
- Integrating Learning and Optimization for Open Transport Capacity Pooling under Disruption
- Zeroth-Order Gradient Estimators under Decision-Dependent Noise