HAPMoE: Heterogeneity-Aware Automatic Parallelism Planning for Mixture-of-Experts Models Training
cs.DC, cs.LG
Submitted: 2026-09-30
Updated: 2026-09-30
Sources
- Model Parallelism on Distributed Infrastructure: A Literature Review from Theory to LLM Case-Studies
- Prediction Is All MoE Needs: Expert Load Distribution Goes from Fluctuating to Stabilizing
- Mixtral of Experts
- Automatically Planning Optimal Parallel Strategy for Large Language Models
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- HeterMoE: Efficient Training of Mixture-of-Experts Models on Heterogeneous GPUs
Related papers
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- SAMM: Sharded Automated Market Maker
- InferScale: GPU-Native KV Injection for Personalized LLM Serving
- Vigil: Accountable Liveness against Selective Silence
- Steelhead: Interleaving Partially Synchronous and Asynchronous Commit Rules on a Shared DAG
- Democratizing MoE inference on commodity GPUs with CoMoE