AnchorKV: Lossless Key-Value Cache Reconstruction from Layer-Sparse Residual Anchors
cs.AR
Submitted: 2026-10-28
Updated: 2026-10-28
Comments: 24 pages, 7 figures, 15 tables
License: http://creativecommons.org/licenses/by/4.0/
The gist: Residual-stream caching can reconstruct transformer key-value (KV) caches losslessly, but its memory advantage depends on architecture.
Terminology
Abstract
Residual-stream caching can reconstruct transformer key-value (KV) caches losslessly, but its memory advantage depends on architecture. We present AnchorKV, a storage-replay spectrum connecting per-layer residual caching and full recomputation through configurable residual anchors. An architecture-dependent break-even rule identifies the first storage-saving spacing, while deployment selects an operating point from the spectrum and cascade replay reduces reconstruction from O(Ls) to O(L), with L the model depth and s the anchor spacing. For interleaved sliding-window architectures, exact dynamic programming places anchors under non-uniform reconstruction demand. Across CPU, CUDA, and Ascend NPU backends, six architecture families, and fp32/bf16 arithmetic, evaluations verify bit-exact KV reconstruction and unchanged continuation logits and tokens. At 4k context, cascade reconstruction is up to 16.3 times faster than naive replay; across eight applicable global-attention checkpoints at 32k context, deepest spacing yields 5.78-54.0 times full-KV reactivatable session capacity. Compared with seven external cache methods on a common model, AnchorKV preserves full-cache retrieval and question-answering scores, whereas lossy methods change output fidelity and task scores. The break-even spacing is computable from a model's configuration file alone.
Sources
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Gemma 3 Technical Report
- Slim attention: cut your context memory in half without loss -- K-cache is all you need for MHA
- The Llama 3 Herd of Models
- Mistral 7B
- The Residual Stream Is All You Need: On the Redundancy of the KV Cache in Transformer Inference
- Qwen2.5 Technical Report
- Fast Transformer Decoding: One Write-Head is All You Need
- Qwen3 Technical Report
Related papers
- Embedded Arena: Iterative Optimization via Hardware Feedback
- WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantization
- SparsePixels: Efficient Convolution for Sparse Data on FPGAs
- Golden Ruler: A Numeric Format Catalog with Bit-Exact Conformance Vectors for FP8, BF16, MXFP4, and Microscaling Formats
- AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Serving
- RFChipAgent: Multi-Agentic AI Flow for Analog/RF Chip Design