A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Experts
cs.LG, stat.ML
Submitted: 2026-02-01
Updated: 2026-11-14
Terminology
Sources
- DeepSeek-V3 Technical Report
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- The Llama 3 Herd of Models
- GPT-4 Technical Report
- On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functions
- Qwen2.5 Technical Report
- Sigmoid Self-Attention has Lower Sample Complexity than Softmax Self-Attention: A Mixture-of-Experts Perspective
Related papers
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis
- (Mis)generalization of Helpful-Only Fine-tuning