Conditional Capacity and Routing in Mixture-of-Experts Particle Transformers
The study compares dense and Mixture-of-Experts Particle Transformers to see how sparse conditional capacity affects performance. Findings show that top-1 MoE models improve over dense ones at similar computation costs, but activating multiple experts per token yields better accuracy at higher cost. Routing structure is informative about expert allocation but not a direct measure of predictive quality.