MEME: Multi-entity & Evolving Memory Evaluation
cs.LG, cs.CL
Submitted: 2026-05-12
Updated: 2026-11-11
Code: https://github.com/NousResearch/hermes-agent2https:
Project page: https://seokwonjung-jay.github.io/meme-eval
License: http://creativecommons.org/licenses/by/4.0/
The gist: LLM-based agents increasingly operate in persistent environments where they must store, update, and reason over information across many sessions.
Terminology
Abstract
LLM-based agents increasingly operate in persistent environments where they must store, update, and reason over information across many sessions. While prior benchmarks evaluate only single-entity updates, MEME defines six tasks spanning the full space defined by the multi-entity and evolving axes, including three not scored by prior work: Cascade and Absence (dependency reasoning) and Deletion (post-removal state). Evaluating six memory systems spanning three memory paradigms on 100 controlled episodes, we find that all systems collapse on dependency reasoning under the default configuration (Cascade: 3%, Absence: 1% in average accuracy) despite adequate static retrieval performance. Prompt optimization, deeper retrieval, reduced filler noise, and most stronger LLMs fail to close this gap. The largest gain comes from a file-based agent paired with Claude Opus 4.7 as its internal LLM, which partially closes the gap at roughly 70x the baseline cost. Code and data are available on the project page: https://seokwonjung-jay.github.io/meme-eval/
Sources
- Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory
- From Local to Global: A Graph RAG Approach to Query-Focused Summarization
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- BM25S: Orders of magnitude faster lexical search via eager sparse scoring
- Evaluating Very Long-Term Conversational Memory of LLM Agents
- MemGPT: Towards LLMs as Operating Systems
- Zep: A Temporal Knowledge Graph Architecture for Agent Memory
- A Survey on the Memory Mechanism of Large Language Model based Agents
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis