The Memory Trap: Language Agents Overgeneralize from Benign Interactions
cs.CL, cs.AI, cs.LG
Submitted: 2026-11-05
Updated: 2026-11-05
Code: https://github.com/safety-research/memory-trap
Terminology
Sources
- Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation
- Recalling Too Well: Sycophancy Evaluation and Mitigation in Memory-Augmented Models
- Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs
- Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory
- From Untrusted Input to Trusted Memory: A Systematic Study of Memory Poisoning Attacks in LLM Agents
- Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models
- OP-Bench: Benchmarking Over-Personalization for Memory-Augmented Personalized Conversational Agents
- Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale
- Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback
- Embodied Agents Meet Personalization: Investigating Challenges and Solutions Through the Lens of Memory Utilization
- Personalized Language Modeling from Personalized Human Feedback
- Machine Bullshit: Characterizing the Emergent Disregard for Truth in Large Language Models
- Learning Personalized Agents from Human Feedback
- Agents Don't Just Agree, They Remember: Benchmarking Persistent Sycophancy in Self-Improving Personal Agents
- MemGPT: Towards LLMs as Operating Systems
- Feedback Loops With Language Models Drive In-Context Reward Hacking
- Ignore Previous Prompt: Attack Techniques For Language Models
- PersistBench: When Should Long-Term Memories Be Forgotten by LLMs?
- Integrating Summarization and Retrieval for Enhanced Personalization via Large Language Models
- Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data