Learning User Simulators with Turing Rewards
summary
The gist
Learning to simulate human users in interactive settings could advance the training of agent assistants, evaluation of personalization systems, research in the social sciences…
In short
The paper introduces Turing-RL, a reinforcement learning method for training AI user simulators. Instead of matching ground truth responses, it uses a discriminative reward where an LLM judge scores how human-like and indistinguishable a generated response is from what the real user would have said. This approach consistently outperforms other methods by optimizing for human-likeness rather than simple response matching.
Key concepts
- Turing Reward
- This is the training signal used to teach the simulator. It's a score assigned by an LLM judge indicating how difficult it is for a human evaluator to tell if the generated response came from the real user or an AI. The goal is to maximize this score, aiming for responses that are highly indistinguishable from human behavior.
- Indistinguishability Optimization
- The core idea is shifting the training focus from perfectly matching a single correct answer to making responses so human-like that they cannot be distinguished from genuine user input. This strategy proves effective because optimizing for how hard it is to tell the difference leads to better, more generalizable user simulators.
- Persona Induction
- This technique involves using an auxiliary language model to summarize stable traits and characteristics derived from a user's past conversation history. This helps create a richer representation of the user beyond just raw text, allowing the simulator to generate responses that feel more consistent with that specific person's style.
Terminology used across episodes
This episode discusses
- Learning User Simulators with Turing Rewards · Paper Radio
- Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning
- QLoRA: Efficient Finetuning of Quantized LLMs
- Learning to Simulate Human Dialogue
- Large Language Model based Multi-Agents: A Survey of Progress and Challenges
- LoRA: Low-Rank Adaptation of Large Language Models
- Aligning Language Models to User Opinions
- Modeling Others' Minds as Code
- Can Language Models Reason about Individualistic Human Values and Preferences?
- Cognitive Foundations for Reasoning and Their Manifestation in LLMs · Paper Radio
- The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models
- Cognitive Models and AI Algorithms Provide Templates for Designing Language Agents
- Can LLM Agents Simulate Multi-Turn Human Behavior? Evidence from Real Online Customer Behavior Data
- Goal Alignment in LLM-Based User Simulators for Conversational AI
- Flipping the Dialogue: Training and Evaluating User Language Models
- AI Simulations Grounded in Rich Self-Reports Increase Accuracy and Reduce Distortion · Paper Radio
- Qwen3.5-Omni Technical Report
- SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMs
- Whose Opinions Do Language Models Reflect?
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants
The paper
Learning User Simulators with Turing Rewards · Read on arXiv
Yingshan Susan Wang, Cedegao E. Zhang, Linlu Qiu, Zexue He, Pengyuan Li, Alex Pentland, Roger P. Levy, Yoon Kim
Massachusetts Institute of Technology · Stanford University · MIT-IBM Watson AI Lab
Learning to simulate human users in interactive settings could advance the training of agent assistants, evaluation of personalization systems, research in the social sciences, and more. Existing approaches generally do so by training a large language model (LLM) to match a single ground truth response, either by maximizing the log probability or by using a similarity reward. We instead propose Turing-RL: a Turing-Test-based reinforcement learning approach for training user simulator models. Turing-RL uses a discriminative Turing reward with an LLM judge to score how indistinguishable a generated response is from the real user's given the user's history, and the user simulator LLM learns to produce responses indistinguishable from what the user could have said with such rewards. Across two different domains--conversational chat and Reddit forum discussion--we find that Turing-RL consistently outperforms baseline methods on both LLM and human evaluation metrics. Our study suggests that optimizing for indistinguishability, rather than response matching, is effective for learning user simulators.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Learning User Simulators with Turing Rewards".
Title and authors: Tom: So we’re talking about this paper called "Learning User Simulators with Turing Rewards" by Susan Wang and the team. What is the main idea here?
Jane: Well, essentially, it’s about training an AI model to act like a real human user in a conversation or some forum discussion. The authors are proposing a new way to do that using something called Turing-RL.
Lu: It moves away from just trying to match one specific answer by maximizing log probability or response similarity, which is what the existing methods usually do >
Tom: Right, so they’re proposing this Turing reward system. What does that actually mean for simulating users?
Jane: They suggest that instead of focusing on matching a single ground truth response, the model should optimize for being indistinguishable from whatever the real user would have said given their history >
Meng: I wonder if optimizing for indistinguishability is practical when you're trying to get a specific type of helpful response from an AI assistant >
Lalam: My internal literature search shows that this approach uses a discriminative Turing reward where an LLM judge scores how human-like the generated response is, conditioned on the user's history >
The paper's summary: Tom: Okay, so what’s the core mechanism of this Turing-RL method? How does that reward actually get built?
Jane: They use an LLM judge to look at the user’s past history and compare it against two responses for the same context: one from the real user, and one from their model >
Lu: The judge rates those responses on a scale of one to seven, scoring which response is more likely written by the human, considering things like the local context and what seems like the user’s motive >
Tom: And they turn that raw score into a training signal. How do they turn that comparison into something the model can actually use during training?
Jane: They take that score, which they call s of y and y star, and transform it into a Turing reward by taking the minimum of those scores capped at five minus one >
Meng: Capped at five sounds like a safety measure. What’s the risk they’re trying to avoid with that cap?
Lu: They say it mitigates against reward hacking, which is when the model finds a loophole to get a high score without actually becoming more human-like >
The paper's improvements: Tom: The paper claims this outperforms baseline methods. What specific improvements are they highlighting when they compare Turing-RL against things like response similarity rewards or chain-of-thought adapted from Gandhi et al?
Jane: They say that Turing-RL consistently beats those baselines on both LLM and human evaluation metrics, which is significant because it suggests optimizing for indistinguishability is a better path >
Lu: They point out that this doesn't hurt content alignment with the ground truth response. For example, they show that Turing-RL performs comparably to Sim-RL on response similarity scores >
Meng: So they’re not sacrificing accuracy just to chase human likeness, which is something I always worry about when deploying these kinds of simulators in a real setting >
Tom: And another point they make is that their user simulators are more grounded in the context and perform better than SFT-Init and Logprob-RL, even on Reddit forum discussion >
Conclusion: Jane: So to wrap up, the core idea of this paper is training LLM user simulators using a discriminative Turing reward that scores how indistinguishable a response is from what a real user would have said given their history >
Tom: It seems like the big implication here is that for building accurate and generalizable user simulators, optimizing for being indistinguishable rather than just matching the ground truth response is an effective strategy >
Lu: They’ve shown that this method works across two different domains: conversational chat and Reddit forum discussion, using datasets like PRISM and ConvoKit >
Meng: But I do have to point out the limitations they mention. One thing they flag is that their evaluation relies on a powerful LLM judge, which introduces computational cost and dependence on that judge’s biases >
Lalam: They also caution that simulators trained to be indistinguishable from specific people could potentially be used for impersonation or social-engineering attacks >
Jane: So while the results are strong across those benchmarks, they aren't ignoring the dual-use nature of this technology and suggesting safeguards like watermarking should follow >
Tom: Exactly. This paper, "Learning User Simulators with Turing Rewards," gives us a promising recipe for building simulators that are more human-like without completely abandoning content alignment.
More episodes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI
- 2605.23500-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation