HEART-Bench: Do LLMs Make Persona-Consistent Decisions?
summary
The gist
LLM agents can be evaluated for human-like psychology by introducing HEART-Bench, which systematically assesses whether these agents can simulate coherent personality and value-consistent behavioral…
This episode discusses
- HEART-Bench: Do LLMs Make Persona-Consistent Decisions? · Paper Radio
- From Persona to Personalization: A Survey on Role-Playing Language Agents
- Investigating Affective Use and Emotional Well-being on ChatGPT
- AI-native Memory 2.0: Second Me
- Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory
- Scaling Synthetic Data Creation with 1,000,000,000 Personas
- CloneMem: Benchmarking Long-Term Memory for AI Clones
- Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale
- PersonaAgent: Bridging Memory and Action for Personalized LLM Agents
- RGMem: Renormalization Group-inspired Memory Evolution for Language Agents
- A-MEM: Agentic Memory for LLM Agents
- Personality Traits in Large Language Models
- LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory
- RealMem: Benchmarking LLMs in Real-World Memory-Driven Interaction
- KnowMe-Bench: Benchmarking Person Understanding for Lifelong Digital Companions
- Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
The paper
HEART-Bench: Do LLMs Make Persona-Consistent Decisions? · Read on arXiv
Weihan Peng, Chenxu Zhang, Qianao Wang, Yuling Shi, Heng Lian, Qihong Mao, Jiahao Pang, Chunliang Feng, Bowen Li, \a Gu
Shanghai Jiao Tong University · Imperial College London · –Quwan Group University of Washington South China Normal University
LLM agents in interactive entertainment and social applications, including companions, role-play characters, and persona clones, need decisions grounded in personality and past experiences. We introduce HEART-Bench to evaluate memory-grounded persona-consistent response selection. The benchmark combines 11 Big Five-anchored synthetic characters, each with 1,000 autobiographical memories, and 64 DIAMONDS-guided decision scenarios into 673 multiple-choice questions. Domain experts independently compare candidate responses, with a senior expert resolving disagreements and revising responses when needed; other characters' adjudicated responses to the same scenario serve as distractors. In the main evaluation, models receive character IDs and occupations, scenarios, and options, with retrieved memories added in memory-augmented settings. Detailed personality-and-value profiles are withheld. We evaluate 16 LLMs under No-Retrieval, Naive-RAG, Mem0, and PersonaDB. Accuracy without memories stays near the 25% chance level, and the best memory-augmented configuration reaches 64.9%. On a matched 150-item subset, human annotators reach 80.8%, compared with 64.0% for the best model. Memory-content ablations show the importance of retrieving experiences that match both the target character and the decision context.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "HEART-Bench: Do LLMs Make Persona-Consistent Decisions?".
Jane: Here's the gist of the paper. LLM agents can be evaluated for human-like psychology by introducing HEART-Bench, which systematically assesses whether these agents can simulate coherent personality and value-consistent behavioral decisions The benchmark constructs 11 diverse human characters grounded in orthogonal Big Five personality traits and integrates them with 1,000 structured autobiographical memories across developmental life stages Benchmark Construction HEART-Bench is a novel benchmark designed to quantify the human-like psychology of AI Agents by simulating 11 diverse human characters grounded in the Big Five and associating each with 1,000 episodes of autobiographical memory spanning distinct life stages The benchmark evaluates whether agents can consolidate innate personality traits and autobiographical memories to make behavioral decisions consistent with their specific psychological profiles This is achieved by grounding each character in a foundational five-dimensional personality framework and intricately associating them with 1,000 episodes of autobiographical memory Character and Memory Grounding To construct AI characters capable of demonstrating genuine psychological depth, the benchmark defines a character as a psychologically grounded entity c = (p, l) where p = (O, C, E, A, N) denotes a Big Five trait vector and l represents a lifelong sequence of autobiographical memories Stage 1 of construction involves Character Profiling where each character is anchored to a single dominant trait at an extreme value to ensure controlled comparisons across characters Stage 2 involves Synthesizing Autobiographical Memories where 1,000 episodic memories are populated for each character, structured around Rubin’s Basic Systems Model and distributed across eight Erikson–Levinson developmental stages The memory generation protocol enforces five dimensions—sensory detail, dialogue reconstruction, inner monologue, somatic response, and aftermath—to ensure organic integration of experience Scenario Design and Validation Scenarios are designed as tuples s = (m, Cenv, Cnarrative, etrigger) where each scenario is character-agnostic to isolate the agent’s internal personality as the sole independent variable Scenarios are constructed in three stages: Human-Guided Blueprinting using the DIAMONDS taxonomy, LLM-Assisted Expansion using a four-component schema, and Expert Validation where doctoral-level researchers verify logical consistency and psychological plausibility The validation protocol ensures that each of the eight DIAMONDS dimensions is exercised across both ends of its intensity range, confirming broad coverage Evaluation Methodology and Results The benchmark evaluates twelve frontier LLMs—including GPT-5.4, Claude-Sonnet-4.6, and Gemini-3.1-Pro—under three interactive settings: Naive-RAG, Mem0, and PersonaDB The results show that even the strongest model (Gemini-3.1-Pro) achieves at most 63.3% accuracy across all settings A striking observation is the asymmetry between the model axis and the method axis, where behavioral accuracy is bottlenecked by the model’s intrinsic capability for psychologically grounded reasoning rather than by retrieval design Furthermore, low-trait characters tend to be harder to evaluate as they express personality through absence of a behavioral drive Limitations and Future Directions The primary limitation of the current work lies in the scalability of dataset construction, which fundamentally depends on expert-driven human annotation and validation rather than fully automated procedures Scaling the process itself cannot yet be fully automated through a data-generation pipeline Future research should focus on whether LLMs can maintain stable personality traits and value orientations across diverse contexts and longitudinal interactions The work argues that a human-in-the-loop paradigm remains essential for maintaining the validity, interpretability, and scientific reliability of such evaluations Prompts and Schemas The paper details several complex prompts used throughout the pipeline to guide LLM behavior, including a Character Profile Generation Prompt that requires output in JSON format with specific Big Five scores and core behavioral patterns The Memory Generation System Prompt enforces Rubin’s five-dimension coverage and strict temporal perspective lock, prohibiting post-hoc evaluative language like "years later" The Scenario Expansion Prompt requires the model to expand a short sketch into a complete JSON object with detailed constraints on context text and trigger events Stage 3 Ground Truth Annotation prompts instruct the LLM to fully become the character, forbidding psychological terminology and requiring memories to surface as associations rather than citations The Memory Activation Screening Prompt asks the model to decide which of six retrieval-cue mechanisms would activate a memory based on psychological links like emotional resonance or behavioral scripts The final evaluation stage involves role-playing the character to produce an inner consciousness trace and a two-sentence final decision The Memory Schema specifies ten fields grouped by functional role—Structural (S), Retrieval-index (R), and Psychological annotation (P)—to structure the 1,000 long-horizon memories The Character Profile Generation Prompt requires the generation of an occupation consistent with the dominant trait to ensure ecological validity The Memory Generation Prompt mandates that content full narratives must encode information irreducible to propositional fact-storage by weaving in sensory detail, dialogue reconstruction, inner monologue, somatic response, and aftermath The Scenario Expansion Prompt embeds a complete worked example of a scenario instance into the system prompt so the model has a concrete shape to imitate The Annotation Prompts for Stage 1 perform a permissive binary screen of every memory whose age falls at or before the scenario age The final test instance assembly involves asking Claude-Sonnet-4.6 to select three distractors from ten validated responses, applying similarity and relevance filters to ensure the question is neither trivial nor ambiguous The Expert Validation Protocol uses a four-expert team with a consensus-then-arbitration rule to ensure that the gold annotations are psychologically plausible and consistent with the character’s established personality profile The final output format for Stage 3 Ground Truth Annotation requires the model to produce an inner consciousness summary, emotional tone description, core reasoning, value orientation, and a two-sentence final decision The Character Profile Generation Prompt requires the generation of exactly eight specific observable patterns that manifest the dominant trait for each character The Memory Generation System Prompt enforces a strict JSON output format, requiring only JSON with no other text The Scenario Expansion Prompt mandates that context text uses second person "you" so the character can step into the scene directly without specific identity anchors The Memory Activation Screening Prompt asks for a JSON array containing only activated memories, with each entry indicating which psychological mechanism activated it The Stage 2 Activated-Memory Refinement Prompt prioritizes memories based on behavioral script match, conditioned-reflex activation, and core belief anchor to select the most influential entries The Stage 3 Ground Truth Annotation Prompt forbids the use of psychological terminology and requires memories to surface naturally as associations rather than citations The Evaluation Metrics adopt Accuracy as the evaluation metric, defined as the proportion of correct MCQs that the agent selects over 673 questions The Model–Model Agreement Analysis reveals that pairwise answer agreement is substantially higher within model families than across them, suggesting raw capability level may be a stronger predictor of behavioral alignment than shared training lineage The overall results suggest that existing LLMs still struggle to infer latent personality traits and value orientations from historical episodic memories Page 1 --- HEART-BE N C H: Do LLM Agents Exhibit Human-like Psychology?
Title and authors: Tom: Let’s start with the title and the authors. We have "HEART-Bench: Do LLMs Make Persona-Consistent Decisions?". The authors are Peng, Zhang, Wang, Shi, Lian, Mao, Pang, Feng, Li Gu.
Jane: Those names suggest a lot of deep research coming from different institutions like Shanghai Jiao Tong University and Imperial College London. It gives you a sense that this isn't just a quick test.
Lu: They’re tackling the fundamental gap between task-oriented AI and complete human personalities, which is what the title points to right there in "Do LLMs Make Persona-Consistent Decisions?".
Meng: I wonder how they're going to actually measure that consistency without it being just surface-level roleplay. It sounds like a tough problem for any benchmark designer.
Lalam: They are trying to bridge the gap between just encoding a fixed profile and having that profile truly shape the behavior in novel scenarios.
The paper's summary: Tom: So, what they summarize is that they’re constructing eleven diverse human characters grounded in orthogonal Big Five personality traits, like Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism.
Jane: And each of these eleven characters is deeply integrated with one thousand structured autobiographical-style episodic memories spread across different developmental life stages. That’s a lot of history for one character.
Lu: The methodology involves grounding each character in this five-dimensional personality framework and associating them with those one thousand episodes, which are structured around Rubin’s Basic Systems Model and distributed across eight Erikson–Levinson developmental stages.
Meng: Eight stages? That sounds incredibly complex to build and maintain for a dataset. How do you even structure that memory generation protocol so it feels organic rather than just like random facts?
Lalam: The memory generation protocol enforces five dimensions: sensory detail, dialogue reconstruction, inner monologue, somatic response, and aftermath. They aim to ensure these memories encode "the subjective texture of the experience" rather than just propositional facts.
The paper's improvements: Tom: So the improvements they propose focus on how they structure this entire pipeline. They suggest moving beyond simple static character profiles or fixed role descriptions that you see in some existing work.
Jane: They are looking at two main lines of existing approaches: one where personality is just injected as a static profile, and another where memory infrastructure like Mem0 or HippoRAG provides the long-term storage.
Lu: HEART-Bench is positioned to address the core psychological question that neither of those lines answers: how does a character’s trait profile shape which experiences are encoded, how mood-congruent processes color emotional valence, and how that history produces coherent responses in novel situations.
Meng: So they aren't just testing if the AI can recall something; they’re testing if its personality dictates *how* it recalls and *why* it responds the way it does in a new context.
Lalam: They designed sixty-four evaluation scenarios, which are woven into this structure, guided by a framework called the DIAMONDS taxonomy to characterize situations along different psychological dimensions <ref:2605.30058#pg1>.
Conclusion: Tom: To wrap up, they show that even the strongest models, like Gemini-three point one-Pro on this benchmark, achieve at most sixty-three point three percent accuracy across all three interactive settings—Naive-RAG, Mem0, and PersonaDB <ref:2605.30058#pg1>.
Jane: The results point to a significant asymmetry between the model's intrinsic capability for psychological reasoning and the retrieval design of the method itself, suggesting that raw model capability is a bigger bottleneck than just how you store information.
Lu: They also found that low-trait characters are harder to evaluate because they express personality through absence of a behavioral drive, which makes those specific tests tricky.
Meng: From a deployment standpoint, if we’re talking about shipping this at scale, the limitation is really what they admit: the scalability of dataset construction fundamentally depends on expert-driven human annotation and validation rather than fully automated pipelines.
Lalam: So they flag that building these benchmarks right now still requires a lot of human input to ensure it maintains its scientific reliability.
Tom: So, in summary, HEART-Bench is a novel benchmark designed to quantify the human-like psychology of AI Agents by systematically assessing their ability to simulate coherent personality and value-consistent behavioral decisions through complex character grounding and memory integration.
Jane: It’s a lot of work, but it moves the conversation past simple accuracy metrics toward simulating genuine psychological depth in how agents interact with the world.
Lu: The implication is that for AI to truly be useful in complex human interactions, it needs to understand not just what to do, but who it is doing those things as.
Meng: For us on the engineering side, it shows that we can’t just rely on off-the-shelf personality prompts; we need a much more rigorous way to validate that internal consistency.
Lalam: The paper suggests that maintaining the validity of these psychological evaluations requires a human-in-the-loop paradigm because fully automated data generation for this level of complexity is currently beyond reach.
More episodes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI
- 2605.23500-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation