Hierarchical Copula-Gumbel-Top-K Routing: Two-Sided Dependence Control for Frozen Mixture-of-Experts at Fixed Per-Token Routing Laws
This episode covers a paper proposing a method to control how tokens coordinate expert selection in frozen Mixture-of-Experts models. The hosts discuss how hierarchical copulas allow for positive coupling to increase coherence and negative coupling to balance expert loads, all while mathematically guaranteeing that individual token routing laws remain unchanged.