Learning User Simulators with Turing Rewards
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Learning User Simulators with Turing Rewards".
Title and authors: Tom: So we’re talking about this paper called "Learning User Simulators with Turing Rewards" by Susan Wang and the team. What is the main idea here?
Jane: Well, essentially, it’s about training an AI model to act like a real human user in a conversation or some forum discussion. The authors are proposing a new way to do that using something called Turing-RL.
Lu: It moves away from just trying to match one specific answer by maximizing log probability or response similarity, which is what the existing methods usually do >
Tom: Right, so they’re proposing this Turing reward system. What does that actually mean for simulating users?
Jane: They suggest that instead of focusing on matching a single ground truth response, the model should optimize for being indistinguishable from whatever the real user would have said given their history >
Meng: I wonder if optimizing for indistinguishability is practical when you're trying to get a specific type of helpful response from an AI assistant >
Lalam: My internal literature search shows that this approach uses a discriminative Turing reward where an LLM judge scores how human-like the generated response is, conditioned on the user's history >
The paper's summary: Tom: Okay, so what’s the core mechanism of this Turing-RL method? How does that reward actually get built?
Jane: They use an LLM judge to look at the user’s past history and compare it against two responses for the same context: one from the real user, and one from their model >
Lu: The judge rates those responses on a scale of one to seven, scoring which response is more likely written by the human, considering things like the local context and what seems like the user’s motive >
Tom: And they turn that raw score into a training signal. How do they turn that comparison into something the model can actually use during training?
Jane: They take that score, which they call s of y and y star, and transform it into a Turing reward by taking the minimum of those scores capped at five minus one >
Meng: Capped at five sounds like a safety measure. What’s the risk they’re trying to avoid with that cap?
Lu: They say it mitigates against reward hacking, which is when the model finds a loophole to get a high score without actually becoming more human-like >
The paper's improvements: Tom: The paper claims this outperforms baseline methods. What specific improvements are they highlighting when they compare Turing-RL against things like response similarity rewards or chain-of-thought adapted from Gandhi et al?
Jane: They say that Turing-RL consistently beats those baselines on both LLM and human evaluation metrics, which is significant because it suggests optimizing for indistinguishability is a better path >
Lu: They point out that this doesn't hurt content alignment with the ground truth response. For example, they show that Turing-RL performs comparably to Sim-RL on response similarity scores >
Meng: So they’re not sacrificing accuracy just to chase human likeness, which is something I always worry about when deploying these kinds of simulators in a real setting >
Tom: And another point they make is that their user simulators are more grounded in the context and perform better than SFT-Init and Logprob-RL, even on Reddit forum discussion >
Conclusion: Jane: So to wrap up, the core idea of this paper is training LLM user simulators using a discriminative Turing reward that scores how indistinguishable a response is from what a real user would have said given their history >
Tom: It seems like the big implication here is that for building accurate and generalizable user simulators, optimizing for being indistinguishable rather than just matching the ground truth response is an effective strategy >
Lu: They’ve shown that this method works across two different domains: conversational chat and Reddit forum discussion, using datasets like PRISM and ConvoKit >
Meng: But I do have to point out the limitations they mention. One thing they flag is that their evaluation relies on a powerful LLM judge, which introduces computational cost and dependence on that judge’s biases >
Lalam: They also caution that simulators trained to be indistinguishable from specific people could potentially be used for impersonation or social-engineering attacks >
Jane: So while the results are strong across those benchmarks, they aren't ignoring the dual-use nature of this technology and suggesting safeguards like watermarking should follow >
Tom: Exactly. This paper, "Learning User Simulators with Turing Rewards," gives us a promising recipe for building simulators that are more human-like without completely abandoning content alignment.
Yingshan Susan Wang, Cedegao E. Zhang, Linlu Qiu, Zexue He, Pengyuan Li, Alex Pentland, Roger P. Levy, Yoon Kim
Massachusetts Institute of Technology · Stanford University · MIT-IBM Watson AI Lab
cs.CL
Submitted: 2026-06-17
Updated: 2026-11-17
Code: https://github.com/SusanWYS/turing-rl
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 98/100
The gist: Learning to simulate human users in interactive settings could advance the training of agent assistants, evaluation of personalization systems, research in the social sciences…
Key concepts
- Turing Reward
- This is the training signal used to teach the simulator. It's a score assigned by an LLM judge indicating how difficult it is for a human evaluator to tell if the generated response came from the real user or an AI. The goal is to maximize this score, aiming for responses that are highly indistinguishable from human behavior.
- Indistinguishability Optimization
- The core idea is shifting the training focus from perfectly matching a single correct answer to making responses so human-like that they cannot be distinguished from genuine user input. This strategy proves effective because optimizing for how hard it is to tell the difference leads to better, more generalizable user simulators.
- Persona Induction
- This technique involves using an auxiliary language model to summarize stable traits and characteristics derived from a user's past conversation history. This helps create a richer representation of the user beyond just raw text, allowing the simulator to generate responses that feel more consistent with that specific person's style.
Terminology
Summary
Learning to simulate human users in interactive settings could advance the training of agent assistants, evaluation of personalization systems, research in the social sciences, and more The paper proposes Turing-RL: a Turing-Test-based reinforcement learning approach for training user simulator models This method uses a discriminative Turing reward with an LLM judge to score how indistinguishable a generated response is from the real user’s given the user’s history Across two different domains—conversational chat and Reddit forum discussion—we find that Turing-RL consistently outperforms baseline methods on both LLM and human evaluation metrics Our study suggests that optimizing for indistinguishability, rather than response matching, is effective for learning user simulators
Pruning redundant reasoning tokens
The core idea is to move beyond training signals based on matching a single ground truth response by instead optimizing for the broader quality of being indistinguishable from what the user could have said The classic Turing Test operationalizes human-likeness through indistinguishability, where a machine passes if an evaluator cannot tell its responses from a human’s The proposed objective is max E y pθ(·x,u), y⋆ p⋆(·x,h) rturing(y, y⋆)
How the training signal is built
The training signal is a discriminative Turing reward where an LLM judge assigns a high score when a generated response is deemed human-like, conditioned on the user’s history This reward, rturing(y, y⋆) = min(s(y, y⋆), 5) - 1, caps the score at 5 to mitigate reward hacking The model is trained with Group Relative Policy Optimization (GRPO; Shao et al., 2024), after an initial supervised finetuning (SFT) phase on a disjoint subset of the training data
Where the gains come from
The Turing-RL approach consistently outperforms baseline methods like response-similarity reward and log-probability maximization with chain-of-thought adapted from Gandhi et al. (2026) across both LLM and human evaluation metrics The results show that optimizing for indistinguishability does not sacrifice content alignment, as Turing-RL matches Sim-RL on response similarity to ground truth Furthermore, Turing-RL user simulators are more grounded in context and perform better than SFT-Init and Logprob-RL in both domains
A Dataset Details
The study evaluates the approach on two domains: multi-turn chat (PRISM) and Reddit forum discussion (ConvoKit) Datasets are split into training and evaluation sets, with behavior history h being disjoint from the current target context x and ground truth response y⋆ Persona induction is achieved by prompting an auxiliary language model to summarize stable traits from the same user’s history block h The evaluation involves both LLM-as-a-Judge metrics and human evaluations using a binary-choice Turing test
Limitations
Limitations include the lack of systematic testing across other interaction types like task-oriented dialogue, the reliance on Qwen3-8B as the base model, and the computational cost associated with using a powerful LLM judge at training time The authors also note that while positive research value is substantial, safeguards like watermarking should be pursued to avoid potential misuse for impersonation
Related Work
Related work includes LLM-based user simulation which has used methods like response similarity rewards and log-probability maximization with latent chain-of-thought Work on persona and user representation also exists, showing that combining raw history with an induced persona provides analysis on different user representations
Conclusion
We have proposed training LLM-based user simulators with a discriminative Turing reward, which scores a generated response by how indistinguishable it is from the real user’s response, conditioned on the user’s prior history This signal consistently produces more human-like responses than either log-probability maximization or response-similarity rewards under both LLM- and human-based Turing evaluations, without reducing content alignment with the ground truth The work suggests a promising recipe for building accurate and generalizable user simulators
References
The study cites numerous related works covering LLM-based user simulation, persona induction, and human behavior prediction in cognitive science It also references foundational cognitive models like Theory of Mind and Prospect theory in the context of predicting human behaviors The paper acknowledges the dual-use nature of this technology and suggests future work on aligning simulators with users' goals and intentions The authors thank MIT-IBM Watson AI Lab for providing cloud computing GPU resources for the study The paper includes detailed experimental setups, ablation studies on user representations, and extensive qualitative examples comparing various model outputs The final results demonstrate that Turing-RL is among the strongest models on Chat and performs comparably with Sim-RL on Reddit in human evaluation The paper concludes by highlighting the trade-off between content matching and human-likeness, favoring the former for simulators The authors also discuss the reliability of LLM judges versus human judges in determining Turing distinguishability The paper provides detailed metrics on response specificity and reaction time to assess the quality of simulated responses The paper is available on arXiv under the ID 2606.19336 The paper was submitted to arXiv on June 25, 2024 The paper is available at https://github. p. 0 1Code is available at: https://github. com/SusanWYS/turing-rl
1
arXiv:2606. 1Code is available at: https://github. com/SusanWYS/turing-rl
1
arXiv:2606.Where the paper says it
Where the paper says it
--- Page 1 ---
Learning User Simulators with Turing Rewards
Yingshan Susan Wang, Cedegao E. Zhang, Linlu Qiu, Zexue He, Pengyuan Li, Alex Pentland, Roger P. p. 0 Learning User Simulators with Turing Rewards
Yingshan Susan Wang1∗
, Cedegao E. Zhang1∗
, Linlu Qiu1∗
, Zexue He2∗
,
Pengyuan Li3
, Alex Pentland1,2, Roger P. Learning User Simulators with Turing Rewards
Yingshan Susan Wang1∗
, Cedegao E. Zhang1∗
, Linlu Qiu1∗
, Zexue He2∗
,
Pengyuan Li3
, Alex Pentland1,2, Roger P.Where the paper says it
Where the paper says it
--- Page 2 ---
Learning User Simulators with Turing Rewards
RL
COT SFT Init LLM Judge Turing Reward A More Human B More Human LLM User History Persona (optional) LLM Induced Current Context like geing details rightshort questionsdirect and curious1-2 questionsinto UK TV trivia<Conversation 1User: Who is hosting the new series of... AI: Of course! p. 1 Inputs
<Conversation 1>
User: Who is hosting the new series of . AI: Of course! Inputs
<Conversation 1>
User: Who is hosting the new series of . AI: Of course!Where the paper says it
Where the paper says it
--- Page 3 ---
Learning User Simulators with Turing Rewards
InputsUser: Who is hosting the new series of …. AI: Of course! New episodes of Deal or No Deal…
--- Page 4 ---
Learning User Simulators with Turing Rewards
Figure 1: Overview of Turing-RL. Given a user’s history, induced persona, and current conversation context, an SFT-initialized policy generates multiple candidate responses via chain-of-thought (CoT) reasoning A Turing judge (LLM) compares each candidate against the ground truth human response on a 1–7 scale, scoring which is more likely written by the real user
--- Page 5 ---
Learning User Simulators with Turing Rewards
Figure 1: Overview of Turing-RL. Given a user’s history, induced persona, and current conversation context, an SFT-initialized policy generates multiple candidate responses via chain-of-thought (CoT) reasoning A Turing judge (LLM) compares each candidate against the ground truth human response on a 1–7 scale, scoring which is more likely written by the real user
--- Page 8 ---
Learning User Simulators with Turing Rewards
Figure 1: Overview of Turing-RL. Given a user’s history, induced persona, and current conversation context, an SFT-initialized policy generates multiple candidate responses via chain-of-thought (CoT) reasoning A Turing judge (LLM) compares each candidate against the ground truth human response on a 1–7 scale, scoring which is more likely written by the real user
--- Page 9 ---
Learning User Simulators with Turing Rewards
Figure 1: Overview of Turing-RL. Given a user’s history, induced persona, and current conversation context, an SFT-initialized policy generates multiple candidate responses via chain-of-thought (CoT) reasoning A Turing judge (LLM) compares each candidate against the ground truth human response on a 1–7 scale, scoring which is more likely written by the real user
--- Page 10 ---
Learning User Simulators with Turing Rewards
Figure 1: Overview of Turing-RL. Given a user’s history, induced persona, and current conversation context, an SFT-initialized policy generates multiple candidate responses via chain-of-thought (CoT) reasoning A Turing judge (LLM) compares each candidate against the ground truth human response on a 1–7 scale, scoring which is more likely written by the real user
--- Page 12 ---
Learning User Simulators with Turing Rewards
Figure 1: Overview of Turing-RL. Given a user’s history, induced persona, and current conversation context, an SFT-initialized policy generates multiple candidate responses via chain-of-thought (CoT) reasoning A Turing judge (LLM) compares each candidate against the ground truth human response on a 1–7 scale, scoring which is more likely written by the real user
--- Page 15 ---
Learning User Simulators with Turing Rewards
Figure 1: Overview of Turing-RL. Given a user’s history, induced persona, and current conversation context, an SFT-initialized policy generates multiple candidate responses via chain-of-thought (CoT) reasoning A Turing judge (LLM) compares each candidate against the ground truth human response on a 1–7 scale, scoring which is more likely written by the real user
--- Page 18 ---
Learning User Simulators with Turing Rewards
Figure 1: Overview of Turing-RL. Given a user’s history, induced persona, and current conversation context, an SFT-initialized policy generates multiple candidate responses via chain-of-thought (CoT) reasoning A Turing judge (LLM) compares each candidate against the ground truth human response on a 1–7 scale, scoring which is more likely written by the real user
--- Page 20 ---
Learning User Simulators with Turing Rewards
Figure 1: Overview of Turing-RL. Given a user’s history, induced persona, and current conversation context, an SFT-initialized policy generates multiple candidate responses via chain-of-thought (CoT) reasoning A Turing judge (LLM) compares each candidate against the ground truth human response on a 1–7 scale, scoring which is more likely written by the real user
--- Page 24 ---
Learning User Simulators with Turing Rewards
Figure 1: Overview of Turing-RL. Given a user’s history, induced persona, and current conversation context, an SFT-initialized policy generates multiple candidate responses via chain-of-thought (CoT) reasoning A Turing judge (LLM) compares each candidate against the ground truth human response on a 1–7 scale, scoring which is more likely written by the real user
--- Page 25 ---
Learning User Simulators with Turing Rewards
Figure 1: Overview of Turing-RL. Given a user’s history, induced persona, and current conversation context, an SFT-initialized policy generates multiple candidate responses via chain-of-thought (CoT) reasoning A Turing judge (LLM) compares each candidate against the ground truth human response on a 1–7 scale, scoring which is more likely written by the real user
The training signal is a discriminative Turing reward where an LLM judge assigns a high score when a generated response is deemed human-like, conditioned on the user’s history
Turing-RL consistently outperforms baseline methods on both LLM and human evaluation metrics, suggesting that optimizing for indistinguishability is an effective path towards learning user simulators The paper demonstrates that optimizing for indistinguishability, rather than response matching, is effective for learning user simulators This indicates that a discriminative signal is better suited than a matching signal for user simulation The study confirms that training with the Turing reward does not sacrifice content alignment, as seen in response similarity scores where Sim-RL and Turing-RL perform comparably
The paper evaluates the approach on two domains: multi-turn chat (PRISM) and Reddit forum discussion (ConvoKit) Datasets are split into training and evaluation sets, with behavior history h being disjoint from the current target context x and ground truth response y⋆ Persona induction is achieved by prompting an auxiliary language model to summarize stable traits from the same user’s history block h The evaluation involves both LLM-as-a-Judge metrics and human evaluations using a binary-choice Turing test
Limitations include the lack of systematic testing across other interaction types and the reliance on Qwen3-8B as the base model, which would be valuable to study for frontier models The discriminative Turing reward relies on a powerful LLM judge, which introduces computational cost and dependence on the judge’s biases The authors also caution that simulators trained to be indistinguishable from specific people could be used for impersonation or social-engineering attacks
The paper cites numerous related works covering LLM-based user simulation, persona induction, and human behavior prediction in cognitive science It also references foundational cognitive models like Theory of Mind and Prospect theory in the context of predicting human behaviors The paper acknowledges the dual-use nature of this technology and suggests future work on aligning simulators with users' goals and intentions The authors thank MIT-IBM Watson AI Lab for providing cloud computing GPU resources for the study The paper includes detailed experimental setups, ablation studies on user representations, and extensive qualitative examples comparing various model outputs The final results demonstrate that Turing-RL is among the strongest models on Chat and performs comparably with Sim-RL on Reddit in human evaluation The paper concludes by highlighting the trade-off between content matching and human-likeness, favoring the former for simulators The authors also discuss the reliability of LLM judges versus human judges in determining Turing distinguishability The paper provides detailed metrics on response specificity and reaction time to assess the quality of simulated responses The paper is available on arXiv under the ID 2606.19336 The paper was submitted to arXiv on June 25, 2024 The paper is available at https://github.com/SusanWYS/turing-rl
--- Page 1 | Learning User Simulators with Turing Rewards
Yingshan Susan Wang, Cedegao E. Zhang, Linlu Qiu, Zexue He, Pengyuan Li, Alex Pentland, Roger P.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Implement Turing-RL for user simulator training by using a discriminative Turing reward with an LLM judge to score responses on how indistinguishable they are from what a real user could have said, rather than simply maximizing log probability or response similarity.
-
Enable user simulators to produce responses that are indistinguishable from what the user could have said with these rewards, suggesting optimizing for
indistinguishability, rather than response matching
is an effective path toward learning user simulators. -
Develop a system capable of grounding simulated responses in context and target users by evaluating
context specificity
anduser evidence compatibility,
as shown in the Response Specificity Judge criteria. -
Create a robust training pipeline involving Supervised Fine-Tuning (SFT) warm start followed by Reinforcement Learning using Group Relative Policy Optimization (GRPO) to train user simulators on a disjoint dataset.
-
Enhance user representation by using both behavioral history and an induced persona, as the study shows that
u = (h, ρ)
provides a more specific measure of user simulation quality than history alone or persona alone in certain domains.
Abstract
Learning to simulate human users in interactive settings could advance the training of agent assistants, evaluation of personalization systems, research in the social sciences, and more. Existing approaches generally do so by training a large language model (LLM) to match a single ground truth response, either by maximizing the log probability or by using a similarity reward. We instead propose Turing-RL: a Turing-Test-based reinforcement learning approach for training user simulator models. Turing-RL uses a discriminative Turing reward with an LLM judge to score how indistinguishable a generated response is from the real user's given the user's history, and the user simulator LLM learns to produce responses indistinguishable from what the user could have said with such rewards. Across two different domains--conversational chat and Reddit forum discussion--we find that Turing-RL consistently outperforms baseline methods on both LLM and human evaluation metrics. Our study suggests that optimizing for indistinguishability, rather than response matching, is effective for learning user simulators.
Sources
- Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning
- QLoRA: Efficient Finetuning of Quantized LLMs
- Learning to Simulate Human Dialogue
- Large Language Model based Multi-Agents: A Survey of Progress and Challenges
- LoRA: Low-Rank Adaptation of Large Language Models
- Aligning Language Models to User Opinions
- Modeling Others' Minds as Code
- Can Language Models Reason about Individualistic Human Values and Preferences?
- Cognitive Foundations for Reasoning and Their Manifestation in LLMs
- The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models
- Cognitive Models and AI Algorithms Provide Templates for Designing Language Agents
- Can LLM Agents Simulate Multi-Turn Human Behavior? Evidence from Real Online Customer Behavior Data
- Goal Alignment in LLM-Based User Simulators for Conversational AI
- Flipping the Dialogue: Training and Evaluating User Language Models
- AI Simulations Grounded in Rich Self-Reports Increase Accuracy and Reduce Distortion
- Qwen3.5-Omni Technical Report
- SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMs
- Whose Opinions Do Language Models Reflect?
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data