MemArena: An Ego-Centric Benchmark for On-Device Agentic Personal Memory Assistants at Scale
MEMARENA is a new benchmark testing personal, on-device memory agents using 10.3 million tokens across 50 agents over 15 days. It evaluates recall, reasoning, and trustworthiness by comparing five different memory backends against various answer models. Findings show that better retrieval methods like MemSearch improve accuracy, but universal failures exist in enforcing access control and metadata completeness.