MoE Pretraining:Parallelism Layout Changes the Model, Not Quality
cs.DC
Submitted: 2026-11-11
Updated: 2026-11-11
Code: https://github.com/ethxin0011/moe_parallelism
Terminology
Sources
- DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
- Linear Mode Connectivity and the Lottery Ticket Hypothesis
- ID Balancing: Stable Training of Extremely Sparse MoE via PID-Based Load Control
- The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- EasyScale: Accuracy-consistent Elastic Training for Deep Learning
- Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
- Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
- Nondeterminism and Instability in Neural Network Optimization
- Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
- Scalable Training of Mixture-of-Experts Models with Megatron Core
- Qwen3 Technical Report
- Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch
- Randomness In Neural Network Training: Characterizing The Impact of Tooling
Related papers
- Programmable Probabilistic Computer with 1,000,000 p-bits
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends