Accelerating Offloading-Based MoE Inference by Interleaving Concurrent Requests
cs.DC, cs.LG
Submitted: 2026-11-03
Updated: 2026-11-03
Code: https://github.com/ggml-org/llama.c
Terminology
Sources
- DeepSeek-V3 Technical Report
- Fast Inference of Mixture-of-Experts Language Models with Offloading
- Gemma 4 Technical Report
- ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
- ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference
- ProMoE: Fast MoE-based LLM Serving using Proactive Caching
- MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache
Related papers
- Programmable Probabilistic Computer with 1,000,000 p-bits
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends