TRACE Bench: Task-driven Roleplay Agentic Checklist Evaluation
Jiahui Zhang, Ziwei Zhang, Yipeng Wang, Yibo Liu, Haozhou Pang, Yikai Hu, Hongyan Ren, Lan Zhou, Qi Gan, Kai Sheng
cs.CL, cs.AI
Submitted: 2026-07-31
Updated: 2026-08-13
Comments: Project page: https://kuaishou-gamemind.github.io/projects/trace_bench/. Code: https://github.com/KuaishouGameMind/TRACE-Bench
Code: https://github.com/KuaishouGameMind/TRACE-Bench
Project page: https://kuaishou-gamemind.github.io/projects/trace_bench
License: http://creativecommons.org/licenses/by/4.0/
The gist: Roleplay evaluation should do more than assign a single score: it should reveal which role requirements were tested, which failed, and which dialogue evidence supports the judgment.
Terminology
Abstract
Roleplay evaluation should do more than assign a single score: it should reveal which role requirements were tested, which failed, and which dialogue evidence supports the judgment. We propose TRACE Bench, a task-driven agentic checklist evaluation framework. It decomposes each role profile offline into a fixed checklist, then uses a User Agent to converse naturally with the target roleplay model while privately updating checklist states from model responses. Scores therefore trace back to checklist items and supporting dialogue turns rather than a black-box holistic impression. For coverage cross-validation, we audit released M2 free-dialogue transcripts from the MiniMax Role-play Benchmark against the same role-derived checklist. The released free-chat transcripts cover only 73.74% of key role-profile points, whereas TRACE Bench reaches 99.91% coverage in fewer turns. Robustness experiments show stable rankings under repeated runs and User Agent replacement. Across 26 models, TRACE Bench reports overall rankings together with capability breakdowns and checklist traces. It also supports Closed-Loop Benchmark Evolution, distilling verification methods proven effective in failed traces so later evaluations can more reliably elicit and examine observed failure modes.
Sources
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
- RoleRMBench & RoleRM: Towards Reward Modeling for Profile-Based Role Play in Dialogue Systems
- PingPong: A Benchmark for Role-Playing Language Models with User Emulation and Multi-Model Evaluation
- RoleMRC: A Fine-Grained Composite Benchmark for Role-Playing and Instruction-Following
- RoleEval: A Bilingual Role Evaluation Benchmark for Large Language Models
- PersonaArena: Dynamic Simulation for Evaluating and Enhancing Persona-Level Role-Playing in Large Language Models
- RAIDEN-R1: Improving Role-awareness of LLMs via GRPO with Verifiable Reward
- Character is Destiny: Can Role-Playing Language Agents Make Persona-Driven Decisions?
- DynSess: Dynamic Session-Level Evaluation and Optimization Framework for Role-Playing Agents
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering