LeanSparse: Memory-Efficient Serving for Native Sparse-Attention LLMs
cs.DC
Submitted: 2026-11-03
Updated: 2026-11-03
Code: https://github.com/openai/grade-schoo
License: http://creativecommons.org/licenses/by/4.0/
The gist: Sparse attention limits the history read by each query, but serving long contexts still requires substantial GPU memory.
Terminology
Abstract
Sparse attention limits the history read by each query, but serving long contexts still requires substantial GPU memory. Existing KV offloading systems retain the full history of indexer keys on GPU, limiting decode concurrency. Prefill's temporary memory demand can further delay admission even when enough memory remains for the same request's subsequent decode. We present LeanSparse, a serving system that reduces both decode's persistent memory use and prefill's temporary memory demand. Our design exploits two observations: prefill consumes historical data one layer at a time, and decode can identify promising candidates before comparing their original indexer scores. During prefill, LeanSparse streams layer histories through two reusable GPU buffers, overlapping next-layer loading with ongoing model computation. During decode, coarse-to-fine indexing searches compact representations and retrieves original indexer keys only for candidate refinement. Separate indexer key and KV caches reuse data across generation steps. We implement LeanSparse in SGLang and evaluate it on DeepSeek-V4-Flash. At 950K context length, LeanSparse supports 2.95 times as many concurrent requests and achieves 2.14 times decode throughput over HiSparse. On multi-turn Open-SWE traces, it reduces median time to first token from 21.6 to 0.6s and median session completion time by 17.7% relative to ECHO.
Sources
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- GLM-5: from Vibe Coding to Agentic Engineering
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- MoBA: Mixture of Block Attention for Long-Context LLMs
- DoSA: Dots Sparse Attention
- LongCodeBench: Evaluating Coding LLMs at 1M Context Windows
- Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving
- ExpertPlex: A High-Goodput Disaggregated Serving System for MoE LLMs with Adaptive Persistent Kernels
- HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management
- SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving
Related papers
- Programmable Probabilistic Computer with 1,000,000 p-bits
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends