Positions Are Not Facts: The Mismatch Between KV Caches and Memory
cs.CL, cs.LG
Submitted: 2026-09-27
Updated: 2026-09-27
Comments: 132 pages, 28 figures
License: http://creativecommons.org/licenses/by/4.0/
The gist: When a fact changes, how should a language model update the history stored in its key-value (KV) cache? Hiding the old record is cheap, but it may still contain needed details or answer questions
Terminology
Abstract
When a fact changes, how should a language model update the history stored in its key-value (KV) cache? Hiding the old record is cheap, but it may still contain needed details or answer questions about the past. We compare hiding whole records, hiding only replaced values, and deleting old text and recomputing the cache. In a controlled quantity task, masking makes all eight models prefer the new value more strongly, yet six lose complete answers through unit errors or failure to stop; keeping the unit preserves all current answers. Later states also retain useful information from earlier records: on multi-hop updates, rebuilding these states at unchanged positions lowers historical accuracy by 20-41 percentage points, whereas moving the existing states has little effect. Keeping object dependencies and unchanged revision passages prevents many losses. Recognition is a separate challenge. Learned readouts recover distinctions missed by fixed cache similarities on synthetic record pairs. On natural text, text-detector-selected masks show no clear advantage over random masks at the same rate in 14 same-model detector-generator comparisons. Query-dependent access can avoid some losses, with additional storage or access costs. These findings identify what must be preserved beyond the replaced value when using a KV cache as updatable memory.
Sources
- Compute Globally, Materialize Locally: The Memory Contract of Sparse Event-KV
- STALE: Can LLM Agents Know When Their Memories Are No Longer Valid?
- Do Large Language Models Need a Content Delivery Network?
- Latent Space Communication via K-V Cache Alignment
- Cartridges: Lightweight and general-purpose long context representations via self-study
- Cache-to-Cache: Direct Semantic Communication Between Large Language Models
- Prompt Cache: Modular Attention Reuse for Low-Latency Inference
- Activated LoRA: Fine-tuned LLMs for Intrinsics
- Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse
- SideQuest: Model-Driven KV Cache Management for Long-Horizon Agentic Reasoning
- MEMENTO: Teaching LLMs to Manage Their Own Context
- Efficient Memory Management for Large Language Model Serving with PagedAttention
- MINTEval: Evaluating Memory under Multi-Target Interference in Long-Horizon Agent Systems
- Models Take Notes at Prefill: KV Cache Can Be Editable and Composable
- KVEraser: Learning to Steer KV Cache for Efficient Localized Context Erasing
- MemOS: A Memory OS for AI System
- LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
- DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving
- Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks
- Unlocking the Address Book: Dissecting the Sparse Semantic Structure of LLM Key-Value Caches via Sparse Autoencoders
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering