WakeKV: Reactive, Reversible KV Residency for Heads That Change Their Minds
cs.CL, cs.AR, cs.DC, cs.LG
Submitted: 2026-10-02
Updated: 2026-10-02
Comments: Accepted to the NeurIPS 2026 Workshop on ML for Systems. 2 figures, 4 tables, appendix
License: http://creativecommons.org/licenses/by/4.0/
The gist: Most KV-cache compression methods classify attention heads once, either offline or during prefill, and keep this classification fixed throughout generation.
Abstract
Most KV-cache compression methods classify attention heads once, either offline or during prefill, and keep this classification fixed throughout generation. Across three models (1.5B-8B) and three regimes (needle retrieval, long chain-of-thought, and multi-turn recall), we measure head behavior on four model-regime combinations and find that most heads change their reading behavior at least once during generation. We introduce WakeKV, a reactive residency policy that moves cooling heads to a recoverable CPU reservoir rather than freezing or permanently evicting their state. At matched memory or budget, WakeKV consistently improves miss rate over frozen classification and destructive eviction, evaluated across five model-regime combinations and over three cited baselines (SnapKV, uniform R-KV, and ReasonAlloc) across four eligible combinations. A FlexiCache/vLLM implementation on Mistral-7B confirms the benefit on real hardware, improving throughput while retaining LongBench quality.
Sources
- Retrieval Heads are Dynamic
- FlexiCache: Leveraging Temporal Stability of Attention Heads for Efficient KV Cache Management
- HeteroCache: A Dynamic Retrieval Approach to Heterogeneous KV Cache Compression for Long-Context LLM Inference
- ReasonAlloc: Hierarchical Decoding-Time KV Cache Budget Allocation for Reasoning Models
- SnapKV: LLM Knows What You are Looking for Before Generation
- R-KV: Redundancy-aware KV Cache Compression for Reasoning Models
- DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads
- RazorAttention: Efficient KV Cache Compression Through Retrieval Heads
- Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengths
- Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning
- Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference
- CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences
- LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation
- DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs
- G-KV: Decoding-Time KV Cache Eviction with Global Attention
- ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Models
- SCOPE: Optimizing Key-Value Cache Compression in Long-context Generation
- ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference
- InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management
- HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering