Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts
cs.LG, cs.CL
Submitted: 2026-05-12
Updated: 2026-11-11
Code: https://github.com/sagearc/router-expert-geometry
Terminology
Sources
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- DeepSeek-V3 Technical Report
- OLMoE: Open Mixture-of-Experts Language Models
- On the Representation Collapse of Sparse Mixture of Experts
- Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
- StableMoE: Stable Routing Strategy for Mixture of Experts
- DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
- Mixtral of Experts
- ST-MoE: Designing Stable and Transferable Sparse Expert Models
- Advancing Expert Specialization for Better MoE
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling
- Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer
- SD-MoE: Spectral Decomposition for Effective Expert Specialization
- Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss
- Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM
- Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
- Grouter: Decoupling Routing from Representation for Accelerated MoE Training
- ERMoE: Eigen-Reparameterized Mixture-of-Experts for Stable Routing and Interpretable Specialization
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis