MEME: Multi-entity & Evolving Memory Evaluation

arXiv:2605.12477 · cs.LG, cs.CL · Submitted 2026-05-12 · Read on arXiv

cs.LG, cs.CL

Submitted: 2026-05-12

Updated: 2026-11-11

Code: https://github.com/NousResearch/hermes-agent2https:

Project page: https://seokwonjung-jay.github.io/meme-eval

License: http://creativecommons.org/licenses/by/4.0/

The gist: LLM-based agents increasingly operate in persistent environments where they must store, update, and reason over information across many sessions.

Terminology

Abstract

LLM-based agents increasingly operate in persistent environments where they must store, update, and reason over information across many sessions. While prior benchmarks evaluate only single-entity updates, MEME defines six tasks spanning the full space defined by the multi-entity and evolving axes, including three not scored by prior work: Cascade and Absence (dependency reasoning) and Deletion (post-removal state). Evaluating six memory systems spanning three memory paradigms on 100 controlled episodes, we find that all systems collapse on dependency reasoning under the default configuration (Cascade: 3%, Absence: 1% in average accuracy) despite adequate static retrieval performance. Prompt optimization, deeper retrieval, reduced filler noise, and most stronger LLMs fail to close this gap. The largest gain comes from a file-based agent paired with Claude Opus 4.7 as its internal LLM, which partially closes the gap at roughly 70x the baseline cost. Code and data are available on the project page: https://seokwonjung-jay.github.io/meme-eval/

Sources

Related papers