Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving

arXiv:2407.00079 · cs.DC, cs.AI, cs.AR · Submitted 2024-06-24 · Read on arXiv

cs.DC, cs.AI, cs.AR

Submitted: 2024-06-24

Updated: 2026-10-08

Comments: 23 pages, 13 figures

Code: https://github.com/kvcache-ai/Mooncake

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Core — 1× in this paper · explained in 2 episodes

Sources

• Evaluating Large Language Models Trained on Code
• MemGPT: Towards LLMs as Operating Systems
• Splitwise: Efficient generative LLM inference using phase splitting
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
• Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
• Fast Distributed Inference Serving for Large Language Models
• Prompt Cache: Modular Attention Reuse for Low-Latency Inference
• SGLang: Efficient Execution of Structured Language Model Programs
• Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention

Related papers

• iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
• SAMM: Sharded Automated Market Maker
• InferScale: GPU-Native KV Injection for Personalized LLM Serving
• Vigil: Accountable Liveness against Selective Silence
• Steelhead: Interleaving Partially Synchronous and Asynchronous Commit Rules on a Shared DAG
• Democratizing MoE inference on commodity GPUs with CoMoE