Shaping Latent Mixture-of-Experts: From Expert Geometry to Layerwise Capacity Allocation
cs.AI, cs.LG
Submitted: 2026-10-10
Updated: 2026-10-10
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Sources
- Prism Transformer: Progressive Head Schedules for Hierarchical Attention Processing
- Tapered Language Models
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- DeepSeek-V3 Technical Report
- LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts
- DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks
- Mixtral of Experts
- Kimi K3: Open Frontier Intelligence
- MAPLE: MoE Adaptive Plug-and-play Layer-wise Expert allocation
- MoLAE: Mixture of Latent Experts for Parameter-Efficient Language Models
- Geometry-Guided Layerwise FFN Width Allocation in Transformers
- Mixture-of-Depths: Dynamically allocating compute in transformer-based language models
- GLU Variants Improve Transformer
- Qwen3 Technical Report
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems