Multi-GPU MoE Inference with Adaptive Expert Offloading
cs.AI
Submitted: 2026-11-06
Updated: 2026-11-06
Code: https://github.com/ggml-org/llama.cpp
Terminology
Sources
- Fast Inference of Mixture-of-Experts Language Models with Offloading
- Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models
- Efficient Memory Management for Large Language Model Serving with PagedAttention
- CoX-MoE: Coalesced Expert Execution for High-Throughput MoE Inference with AMX-Enabled CPU-GPU Co-Execution
- ProMoE: Fast MoE-based LLM Serving using Proactive Caching
- HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference
- ActKV: Efficient LLM Agents through Action-Guided KV Cache Management
- MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents