Know It, Act on It: Investigating Memory Utilization in LLM Personalization

arXiv:2607.29433 · cs.CL · Submitted 2026-07-31 · Read on arXiv

Zhaoxin Feng, Jianfei Ma, Emmanuele Chersoni

cs.CL

Submitted: 2026-07-31

Code: https://github.com/boson-ai/RPBench-Auto

License: http://creativecommons.org/licenses/by/4.0/

The gist: As large language model (LLM) agents evolve into personalized companions, memory has emerged as a core capability.

Terminology

Abstract

As large language model (LLM) agents evolve into personalized companions, memory has emerged as a core capability. However, LLMs face a knowledge utilization problem: they may fail to act on relevant user preferences even when they are fully present in context. When an agent fails to tailor its response in a context where previously shared user preferences should matter, it is unclear whether the model failed to remember that information or remembered it but failed to use it. To isolate this breakdown, we introduce a decoupled evaluation paradigm that administers paired Know and Act tests to the same user preference. We conduct large-scale experiments across 16 systems and five memory architectures, evaluating 1,000 preferences embedded at three levels of expression strength. Our results show a large gap between Know and Act outcomes: agents often pass the recall test for a user preference but fail to reflect that same preference in the paired behavioral scenario. While memory architectures reduce this gap, utilization remains especially weak for health and therapy-related preferences, where failures to act carry the greatest real-world stakes.

Sources

Related papers