PersonaChatBench: Evaluating Chat Assistants with Diverse Simulated Users
cs.CL
Submitted: 2026-10-30
Updated: 2026-10-30
Comments: 74 pages, including appendices
License: http://creativecommons.org/licenses/by/4.0/
The gist: Conversational assistants are expected to help people with diverse backgrounds, preferences, and communication styles.
Abstract
Conversational assistants are expected to help people with diverse backgrounds, preferences, and communication styles. Yet evaluations centered on a single cooperative user provide limited insight into how reliably an assistant serves users with different personas. We introduce PersonaChatBench, a framework for evaluating task competence and responsiveness to diverse user needs. PersonaChatBench pairs the same underlying task with multiple simulated users, each guided by a private persona card that specifies distinct needs, preferences, and interaction behaviors. By holding the task fixed while varying the user, the framework examines how assistant performance changes across user conditions. We evaluate 20 language models across six domains, separately assessing native task outcomes, fulfillment of expressed persona needs, and simulator fidelity. Our results show that task performance and need fulfillment vary across personas and domains. Strong task performance does not necessarily translate into satisfying individual needs, and models' relative strengths in meeting those needs are not consistent across domains. PersonaChatBench makes these differences measurable, revealing gaps obscured by aggregate scores and helping identify which users an assistant serves well and where adaptation is needed. It provides a basis for evaluating consistency across users and guiding the development of assistants that better accommodate diverse needs in everyday applications.
Sources
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
- HealthBench: Evaluating Large Language Models Towards Improved Human Health
- PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning
- $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
- Is He Extroverted? Identifying Missing Relevant Personas for Faithful User Simulation
- Scaling Synthetic Data Creation with 1,000,000,000 Personas
- MatrAIx: Simulating the World with 8.3 Billion Persona Agents
- User Behavior Simulation with Large Language Model based Agents
- SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants?
- Non-Collaborative User Simulators for Tool Agents
- Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents
- PersonaCUABench: Benchmarking Computer-Use Agents Serving Diverse Personas
- Personalizing Dialogue Agents: I have a dog, do you have pets too?
- PersonaGym: Evaluating Persona Agents and LLMs
- LaMP: When Large Language Models Meet Personalization
- The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models
- A Roadmap to Pluralistic Alignment
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data