A Simple Plug-in for Improving Eviction-Based KV Cache Compression
cs.LG
Submitted: 2026-05-22
Updated: 2026-10-13
Code: https://github.com/gkamradt/LLMTest_
License: http://creativecommons.org/licenses/by/4.0/
The gist: KV cache growth is a major bottleneck for long-context inference in large language models.
Terminology
Abstract
KV cache growth is a major bottleneck for long-context inference in large language models. Existing methods are often dominated by binary eviction or representation approximation, which may underutilize tokens that are not critical for exact retention but are still reconstructable. We present VECTOR, a plug-and-play augmentation for eviction-based pipelines that introduces three-way token routing: retention, approximation, and eviction. VECTOR combines an importance signal from the base scorer with a reconstructability signal from an offline-calibrated regression-based value estimation. By leveraging reconstructability, VECTOR recovers useful value information that would otherwise be irreversibly lost under binary eviction, while preserving key vectors for attention routing stability. On LongBench, RULER with contexts up to 128K tokens, and multi-turn SCBench, VECTOR yields significant gains over its base eviction methods under high compression. At a 90% compression ratio, SnapKV augmented with VECTOR outperforms six competing compression methods, including D2O and RocketKV, on all three evaluated models. Since VECTOR makes all routing decisions at prefill, decoding runs on a standard compressed cache with no measurable overhead.
Sources
- The Llama 3 Herd of Models
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
- OpenAI GPT-5 System Card
- ReAct: Synergizing Reasoning and Acting in Language Models
- KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments
- KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction
- ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMs
- D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models
- Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
- EliteKV: Scalable KV Cache Compression via RoPE Frequency Selection and Joint Low-Rank Projection
- ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inference
- Fast KV Compaction via Attention Matching
- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
- KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference
- PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
- GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- TurboAngle: Near-Lossless KV Cache Compression via Uniform Angle Quantization
- Fast Transformer Decoding: One Write-Head is All You Need
- Massive Activations in Large Language Models
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks