HetCCL: Efficient LLM Training on Heterogeneous Vendor GPUs
cs.DC, cs.LG
Submitted: 2026-01-30
Updated: 2026-11-13
Code: https://github.com/GPUOpen-LibrariesAndSDKs/DirectGM
Terminology
Sources
- GPT-4 Technical Report
- AMD MI300X GPU Performance Analysis
- Training Deep Nets with Sublinear Memory Cost
- The Llama 3 Herd of Models
- Cephalo: Harnessing Heterogeneous GPU Clusters for Training Transformer Models
- Demystifying NCCL: An In-depth Analysis of GPU Communication Protocols and Algorithms
- KAITIAN: A Unified Communication Framework for Enabling Efficient Collaboration Across Heterogeneous Accelerators in Embodied AI Systems
- Decoupled Weight Decay Regularization
- Pointer Sentinel Mixture Models
- MSCCL++: Rethinking GPU Communication Abstractions for AI Inference
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Collective Communication for 100k+ GPUs
- H2:Towards Efficient Large-Scale LLM Training on Hyper-Heterogeneous Cluster over 1,000 Chips
- An Extensible Software Transport Layer for GPU Networking
- MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism
Related papers
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends
- AgentWeave: Efficient Distributed Agent Serving via Flow Decomposition