Network Contention State Aware Dynamic Optimizer for Accelerator Collective Communication Libraries
cs.DC, cs.LG, cs.MA
Submitted: 2026-11-07
Updated: 2026-11-07
Code: https://github.com/ROCm/rccl
Terminology
Sources
- AMSP: Reducing Communication Overhead of ZeRO for Efficient LLM Training
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- NCCL EP: Towards a Unified Expert Parallel Communication API for NCCL
- GPU-Initiated Networking for NCCL
- PyTorch Distributed: Experiences on Accelerating Data Parallel Training
- ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training
- Deep Learning Recommendation Model for Personalization and Recommendation Systems
- Horovod: fast and easy distributed deep learning in TensorFlow
- MSCCL++: Rethinking GPU Communication Abstractions for AI Inference
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Collective Communication for 100k+ GPUs
- Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- ZeRO++: Extremely Efficient Collective Communication for Giant Model Training
- A Survey of Methods for Collective Communication Optimization and Tuning
Related papers
- Programmable Probabilistic Computer with 1,000,000 p-bits
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends