FlashOffloading: Low-Latency KV Cache Offloading for Long-Context LLM Serving
cs.AI
Submitted: 2026-11-05
Updated: 2026-11-05
Code: https://github.com/vllm-project/vllm-ascend
Terminology
Sources
- KV Cache Offloading for Context-Intensive Tasks
- ESS: An Offload-Centric Latent-Cache Management Architecture for DeepSeek-V3.2-Exp
- RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inference
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
- GLM-5: from Vibe Coding to Agentic Engineering
- AVSG: Accelerated Vectorized Sparse Gather for Efficient KV Cache Offload in Sparse-Attention LLM Serving
- LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
- HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading
- SAC: Disaggregated KV Cache System for Sparse Attention LLMs with CXL
- Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading
- InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference
- HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management
- LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Management
- PQCache: Product Quantization-based KVCache for Long Context LLM Inference
- ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference
- Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving
- SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents