Who Should Score the Cache? Mid-Depth Layers Are the Best Judges of KV-Cache Eviction
cs.LG, cs.CL
Submitted: 2026-11-01
Updated: 2026-11-01
Comments: 19 pages, 3 figures
Code: https://github.com/NVIDIA/kvpress
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Sources
- Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference
- LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
- Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
- PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
- xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction
- Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution
- Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference
- LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference
- Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning
- HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing
- Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
- Gemma 4 Technical Report
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- Page-EntroKV: Hardware-Aligned, Entropy-Weighted KV-Cache Eviction under Grouped-Query Attention
- FastKV: Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding Acceleration
- KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction
- MOSAIC: Multi-Objective Search for Allocating Inference Cache Across LLM Decoder Layers
- InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management
- SnapKV: LLM Knows What You are Looking for Before Generation
- Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis