Efficient Expert-Parallel Communication on PCIe-Connected Consumer GPUs
cs.DC, cs.LG
Submitted: 2026-09-30
Updated: 2026-09-30
Code: https://github.com/deepseek-ai/DeepEP
Sources
- gpt-oss-120b & gpt-oss-20b Model Card
- Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts
- FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion
- NCCL EP: Towards a Unified Expert Parallel Communication API for NCCL
- GPU-Initiated Networking for NCCL
- Short-circuiting Rings for Low-Latency AllReduce
- Mixtral of Experts
- HetCCL: Accelerating LLM Training with Heterogeneous GPUs
- Private LLM Inference on Consumer Blackwell GPUs: A Practical Guide for Cost-Effective Local Deployment in SMEs
- Scaling Laws for Fine-Grained Mixture of Experts
- DeepSeek-V3 Technical Report
- Demystifying NVSHMEM: A System-Level Analysis on Symmetric Memory and Device-Initiated Operations in GPU Communication
- UCCL-EP: Portable Expert-Parallel Communication
- HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System
- Design Space Exploration of DMA based Finer-Grain Compute Communication Overlap
- On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
- Kimi K3: Open Frontier Intelligence
- DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
- Scalable Training of Mixture-of-Experts Models with Megatron Core
Related papers
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- SAMM: Sharded Automated Market Maker
- InferScale: GPU-Native KV Injection for Personalized LLM Serving
- Vigil: Accountable Liveness against Selective Silence
- Steelhead: Interleaving Partially Synchronous and Asynchronous Commit Rules on a Shared DAG
- Democratizing MoE inference on commodity GPUs with CoMoE