ID Balancing: Stable Training of Extremely Sparse MoE via PID-Based Load Control
cs.LG
Submitted: 2026-09-30
Updated: 2026-09-30
Sources
- GPT-4 Technical Report
- Program Synthesis with Large Language Models
- Evaluating Large Language Models Trained on Code
- Training Verifiers to Solve Math Word Problems
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
- Better & Faster Large Language Models via Multi-token Prediction
- Continual Pre-Training of Large Language Models: How to (re)warm your model?
- FastMoE: A Fast Mixture-of-Expert Training System
- Mixture of A Million Experts
- Measuring Mathematical Problem Solving With the MATH Dataset
- Continual Pre-training of Language Models
- LocMoE: A Low-Overhead MoE for Large Language Model Training
- DeepSeek-V3 Technical Report
- HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System
- On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
- Retentive Network: A Successor to Transformer for Large Language Models
- Kimi K3: Open Frontier Intelligence
- Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
- OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks