MoEless: Efficient MoE LLM Serving with Serverless Experts
cs.DC, cs.AI, cs.LG
Submitted: 2026-03-06
Updated: 2026-09-30
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large Language Models (LLMs) increasingly adopt Mixture-of-Experts (MoE) architectures to scale efficiently under stringent resource constraints.
Terminology
Abstract
Large Language Models (LLMs) increasingly adopt Mixture-of-Experts (MoE) architectures to scale efficiently under stringent resource constraints. However, MoE's sparse activation causes severe expert load imbalance, where a few experts become stragglers while others remain underutilized, leading to inflated inference latency and cost. Existing solutions assume static, serverful model deployments, limiting expert elasticity and often incurring costly expert swapping or degraded output quality. We present MoEless, an efficient serverless MoE serving framework that mitigates expert load imbalance via elastic expert execution. MoEless leverages lightweight, layer-aware predictors to estimate incoming expert load distributions and proactively identify stragglers. We design optimized scaling and placement strategies to improve function locality, GPU utilization, and cross-expert load balance. MoEless is prototyped on top of Megatron-LM and deployed on an eight-GPU testbed. Experiments with open-source MoE models and real-world workloads show that MoEless reduces inference latency by 43% and inference cost by 84% compared to state-of-the-art solutions.
Sources
- Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
- GPT-4 Technical Report
- Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
- Language Models are Few-Shot Learners
- DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
- Fast Inference of Mixture-of-Experts Language Models with Offloading
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- ServerlessLLM: Low-Latency Serverless Inference for Large Language Models
- Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts
- DeepServe: Serverless Large Language Model Serving at Scale
- Tutel: Adaptive Mixture-of-Experts at Scale
- Mixtral of Experts
- InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management
- Accelerating Distributed MoE Training and Inference with Lina
- DeepSeek-V3 Technical Report
- MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core
- Remoe: Towards Efficient and Low-Cost MoE Inference in Serverless Computing
- Formation of Representations in Neural Networks
- HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds
Related papers
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- SAMM: Sharded Automated Market Maker
- InferScale: GPU-Native KV Injection for Personalized LLM Serving
- Vigil: Accountable Liveness against Selective Silence
- Steelhead: Interleaving Partially Synchronous and Asynchronous Commit Rules on a Shared DAG
- Democratizing MoE inference on commodity GPUs with CoMoE