Mixture of Many Zero-Compute Experts: A High-Rate Quantization Theory Perspective
The study analyzes a Zero-Compute 1-Sparse Mixture of Experts (ZC-1SMoE) model for regression by treating experts as learnable constant values covering small input regions. It uses classical high-rate quantization theory to derive analytical formulas for approximation error and statistical learning trade-offs, showing how increasing the number of experts improves accuracy but degrades learning performance.