Small Frequency Corrections Can Change What Survives KV Cache Compression
cs.CL
Submitted: 2026-08-27
Updated: 2026-10-06
License: http://creativecommons.org/licenses/by/4.0/
The gist: Compressing a key-value cache before its next question is known requires choosing what to retain without knowing which evidence will matter.
Terminology
Abstract
Compressing a key-value cache before its next question is known requires choosing what to retain without knowing which evidence will matter. Value energy measures entry strength but does not distinguish isolated keys from those with many similar neighbors. We introduce TwinKV, a training-free method that discounts value energy by nonlocal post-RoPE key frequency. Prefix attention allocates head capacities, while retained entries preserve their original keys and values under an exact storage budget. Across four language models, TwinKV exceeds five evaluated compressed baselines in mean score on LongBench, LooGLE, and RULER at 50% KV removal. Component controls isolate the frequency contribution. On Llama-3.2-1B RULER at 75% removal, normalized frequency weights average 0.95, yet change 7% of nonprotected retained positions and improve value-only retention by about 5.5 points under both uniform and adapted capacities. Permuting the weights within heads weakens this gain. These results show that modest frequency corrections can change retention and answering outcomes, with effects that depend on the model and task.
Sources
- PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
- Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution
- SnapKV: LLM Knows What You are Looking for Before Generation
- ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference
- KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments
- H$_2$O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering