Muon Makes the First Mixture-of-Experts Layer Harder to Balance
cs.LG, cs.CL
Submitted: 2026-10-27
Updated: 2026-11-10
Comments: 35 pages, 5 figures, 9 tables. Code and runs: https://github.com/marcoshernanz/muon-moe-routing
Code: https://github.com/marcoshernanz/muon-moe-routing
Project page: https://varunneal.github.io/essays/loss-free-moe
License: http://creativecommons.org/licenses/by/4.0/
The gist: Many open mixture-of-experts (MoE) language models train with Muon instead of AdamW.
Terminology
Abstract
Many open mixture-of-experts (MoE) language models train with Muon instead of AdamW. Most balance the load with a bias on each expert's routing score, adjusted after every step. Their recipes disagree on whether Muon should also train the router. In models with 55M and 108M active parameters, we measure how the optimizer changes the span of these biases, the largest minus the smallest in a layer. The span grows when all tokens favor the same experts, so it shows how hard a layer is to balance. At learning rates tuned on shorter runs, Muon raised the span of the first MoE layer at every seed, roughly doubling it on average, while the other layers changed much less. The rise came from Muon on the rest of the network, whichever optimizer trained the router. In those runs, the inputs to the first router shared a direction, and soon after the warmup its busiest expert averaged four to seven times its fair share. Over the first 3,000 steps, subtracting the batch mean from these inputs removed this overload while barely changing the loss. On the smaller model, halving Muon's learning rate removed most of the rise and slightly improved the loss. Moving only the router to AdamW made the loads shift more during the warmup and slightly raised the loss, so at this scale the results favor keeping the router on Muon.
Sources
- Deep Reinforcement Learning at the Edge of the Statistical Precipice
- Scaling Optimal LR Across Token Horizons
- $\phi$-Balancing for Mixture-of-Experts Training
- StableMoE: Stable Routing Strategy for Mixture of Experts
- DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
- DeepSeek-V3 Technical Report
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
- Scaling Vision Transformers to 22 Billion Parameters
- How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings
- MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models
- GLM-5: from Vibe Coding to Agentic Engineering
- A Theoretical Framework for Auxiliary-Loss-Free Load Balancing of Sparse Mixture-of-Experts in Large-Scale AI Models
- Query-Key Normalization for Transformers
- IQuest-Q1: Advancing Foundation Capabilities for Agentic CLI Systems
- ID Balancing: Stable Training of Extremely Sparse MoE via PID-Based Load Control
- K2-V2: A 360-Open, Reasoning-Enhanced LLM
- Kimi K3: Open Frontier Intelligence
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis