Knowing the Store: What a Memory Backend Must Write Down Before an Agent Can Read It
cs.AI
Submitted: 2026-10-03
Updated: 2026-10-03
Comments: 94 pages. Under review at ICLR 2027. Supplement included as ancillary file. Code and data: https://github.com/ansh348/knowing-the-store
Code: https://github.com/ansh348/knowing-the-store
License: http://creativecommons.org/licenses/by/4.0/
The gist: An agent with long-term memory can answer from a record it should no longer use, such as a plan the user later cancelled.
Abstract
An agent with long-term memory can answer from a record it should no longer use, such as a plan the user later cancelled. We ask what a memory store must expose for an agent to know this before retrieving anything, and we score that judgment on its own, as metamemory monitoring. Readers see only a value-free summary: record counts by lifecycle state and a list of attribute names. Each of 148 questions is asked against three versions of one store that differ in one attribute, so wording cannot give the answer away. What a model adds depends on the length of that list. On the short lists the benchmark's ground truth produces (2.6 names on average), none of five language models reliably beats a cosine-similarity lookup over the names, and the three stronger ones are equivalent to it within 0.05. Under a control that fixes counts and list length, none is reliably above it and none is shown equivalent. On lists longer than the stores we tested write, padded to 60 names, the lookup loses 0.18; GPT-5.6 Luna and Sol lose 0.06 to 0.07 and lead it by 0.13 to 0.14, while the other three fall with it. The lead holds for Sol against a sibling padded to the same length and counts. Reworded to share no word with the names, the lookup loses 0.02 and one stronger reader edges 0.04 ahead. On the short lists the three stronger readers lead only where the summary counts a state without naming it, and one count feature closes that lead within a question, though not when items are pooled. The backends we tested omit or misstate this information: under FR-Bank's own metadata, GPT-4.1 mini, Haiku 4.5 and the lookup fall from 0.73, 0.82 and 0.75 to 0.60, 0.71 and 0.59. In these tests, what the store wrote down limited every reader we gave it to. All stores are synthetic; control, whether a better judgment yields a better answer, is left to a later study.
Sources
- STALE: Can LLM Agents Know When Their Memories Are No Longer Valid?
- Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory
- Language Models (Mostly) Know What They Know
- AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions
- PM-Bench: Evaluating Prospective Memory in LLM Agents
- CIMemories: A Compositional Benchmark for Contextual Integrity of Persistent Memory in LLMs
- Fortunate Recall: Ontology-Driven Memory Lifecycle Management for Persistent Coherence in LLMs
- MemGPT: Towards LLMs as Operating Systems
- Supersede: Diagnosing and Training the Memory-Update Gap in LLM Agents
- Zep: A Temporal Knowledge Graph Architecture for Agent Memory
- GateMem: Benchmarking Memory Governance in Multi-Principal Shared-Memory Agents
- From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents
- AutoMem: Automated Learning of Memory as a Cognitive Skill
- HUSH-Bench: Measuring Memory-Use Boundaries for Sensitive History in Conversational Agents
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems