S 4 R: Selective Sampling, Subspaces, and Sparse Reconstruction for Compressed Long-Context KV Caching
cs.CL
Submitted: 2026-08-01
Updated: 2026-09-22
License: http://creativecommons.org/licenses/by/4.0/
The gist: The growth of context window lengths in Large Language Models (LLMs) significantly enhances their long-context capabilities but incurs prohibitive memory costs due to the Key-Value (KV) cache.
Terminology
Abstract
The growth of context window lengths in Large Language Models (LLMs) significantly enhances their long-context capabilities but incurs prohibitive memory costs due to the Key-Value (KV) cache. Although low-rank compression of KV cache is a promising remedy, existing methods face a dilemma: offline approaches depend on external calibration data, whereas online approaches incur substantial compute for full-prompt decomposition and reconstruction. In this paper, we propose S 4 R, which builds low-rank subspaces from selectively sampled tokens and computes attention over a sparsely reconstructed KV representation. S 4 R uses prompt-aware initialization to build initial key/value bases from a representative prompt subset, trading off calibration-data dependence against prefilling cost. Because fully reconstructing the cache at every decoding step is prohibitively expensive and hurts throughput, we further adopt sparse reconstruction to retain only informative positions during decoding. Extensive experiments on LongBench and RULER with Llama and Qwen model families show that S 4 R achieves up to 5 times KV compression with near full-cache accuracy, combining the efficiency of fixed compression with the adaptability of prompt-dependent methods.
Sources
- PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
- xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction
- The Llama 3 Herd of Models
- GPT-4 Technical Report
- GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMs
- DynaKV: Enabling Accurate and Efficient Long-Sequence LLM Decoding on Smartphones
- CommonKV: Compressing KV Cache with Cross-layer Parameter Sharing
- Qwen2.5 Technical Report
- Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks
- KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head
- ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibration
- Qwen3 Technical Report
- Qwen2.5-1M Technical Report
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
- Effectively Compress KV Heads for LLM
- LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy
- OjaKV: Context-Aware Online Low-Rank KV Cache Compression
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering