When Does Sparse Routing Help? The Geometry of Expert Separation in Mixture of Experts
stat.ML, cs.LG
Submitted: 2026-11-01
Updated: 2026-11-01
Terminology
Sources
- On Bayesian Softmax-Gated Mixture-of-Experts Models
- Mixtral of Experts
- Characterizing Heterogeneous Rates in Finite Mixture Estimation via Partial Optimal Transport
- On DeepSeekMoE: Statistical Benefits of Shared Experts and Normalized Sigmoid Gating
- On the Geometry of Separation in Finite Gaussian Mixtures
Related papers
- Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks
- Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors
- Quantifying Error Propagation and Model Collapse in Diffusion Models
- Behavior of prediction performance metrics with rare events
- Statistical Inference and Learning for Shapley Additive Explanations (SHAP)
- Control Variate Design for Score-Function Gradient Estimators in Importance-Weighted Variational Inference