Dynamic Flow, Static Graph: KV Cache Reuse for Efficient LLM Serving on Mobile NPUs

arXiv:2609.34727 · cs.OS, cs.AI, cs.DB · Submitted 2026-09-28 · Read on arXiv

cs.OS, cs.AI, cs.DB

Submitted: 2026-09-28

Updated: 2026-09-28

Code: https://github.com/google/flatbuffers

Terminology

• KV cache — 1× in this paper · explained in 3 episodes
• KV cache reuse — 1× in this paper · explained in 2 episodes

Sources

• LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
• MobileRAG: A Fast, Memory-Efficient, and Energy-Efficient Method for On-Device RAG
• The Llama 3 Herd of Models
• Qwen3 Technical Report
• OmniInfer: System-Wide Acceleration Techniques for Optimizing LLM Serving Throughput and Latency
• T-MAN: Enabling End-to-End Low-Bit LLM Inference on NPUs via Unified Table Lookup
• PowerInfer-2: Fast Large Language Model Inference on a Smartphone
• Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models

Related papers

• MemSpec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge Devices
• Vulcan: Instance-specialized, Verifiable Systems Heuristics Through LLM-driven Search
• Planarian: Managing Agent State with Statepoints
• VUDA: Enabling Controlled Spatial Sharing of Graphics and Compute on NVIDIA GPUs
• ActKV: Efficient LLM Agents through Action-Guided KV Cache Management
• GroupKV: Hierarchical KV Cache Management for Long-Context Diffusion LLM Inference