Learning User Simulators with Turing Rewards

summary

The gist

Learning to simulate human users in interactive settings could advance the training of agent assistants, evaluation of personalization systems, research in the social sciences…

In short

The paper introduces Turing-RL, a reinforcement learning method for training AI user simulators. Instead of matching ground truth responses, it uses a discriminative reward where an LLM judge scores how human-like and indistinguishable a generated response is from what the real user would have said. This approach consistently outperforms other methods by optimizing for human-likeness rather than simple response matching.

Key concepts

Turing Reward
This is the training signal used to teach the simulator. It's a score assigned by an LLM judge indicating how difficult it is for a human evaluator to tell if the generated response came from the real user or an AI. The goal is to maximize this score, aiming for responses that are highly indistinguishable from human behavior.
Indistinguishability Optimization
The core idea is shifting the training focus from perfectly matching a single correct answer to making responses so human-like that they cannot be distinguished from genuine user input. This strategy proves effective because optimizing for how hard it is to tell the difference leads to better, more generalizable user simulators.
Persona Induction
This technique involves using an auxiliary language model to summarize stable traits and characteristics derived from a user's past conversation history. This helps create a richer representation of the user beyond just raw text, allowing the simulator to generate responses that feel more consistent with that specific person's style.

Terminology used across episodes

This episode discusses

The paper

Learning User Simulators with Turing Rewards · Read on arXiv

Yingshan Susan Wang, Cedegao E. Zhang, Linlu Qiu, Zexue He, Pengyuan Li, Alex Pentland, Roger P. Levy, Yoon Kim

Massachusetts Institute of Technology · Stanford University · MIT-IBM Watson AI Lab

Learning to simulate human users in interactive settings could advance the training of agent assistants, evaluation of personalization systems, research in the social sciences, and more. Existing approaches generally do so by training a large language model (LLM) to match a single ground truth response, either by maximizing the log probability or by using a similarity reward. We instead propose Turing-RL: a Turing-Test-based reinforcement learning approach for training user simulator models. Turing-RL uses a discriminative Turing reward with an LLM judge to score how indistinguishable a generated response is from the real user's given the user's history, and the user simulator LLM learns to produce responses indistinguishable from what the user could have said with such rewards. Across two different domains--conversational chat and Reddit forum discussion--we find that Turing-RL consistently outperforms baseline methods on both LLM and human evaluation metrics. Our study suggests that optimizing for indistinguishability, rather than response matching, is effective for learning user simulators.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Learning User Simulators with Turing Rewards".

Title and authors: Tom: So we’re talking about this paper called "Learning User Simulators with Turing Rewards" by Susan Wang and the team. What is the main idea here?

Jane: Well, essentially, it’s about training an AI model to act like a real human user in a conversation or some forum discussion. The authors are proposing a new way to do that using something called Turing-RL.

Lu: It moves away from just trying to match one specific answer by maximizing log probability or response similarity, which is what the existing methods usually do >

Tom: Right, so they’re proposing this Turing reward system. What does that actually mean for simulating users?

Jane: They suggest that instead of focusing on matching a single ground truth response, the model should optimize for being indistinguishable from whatever the real user would have said given their history >

Meng: I wonder if optimizing for indistinguishability is practical when you're trying to get a specific type of helpful response from an AI assistant >

Lalam: My internal literature search shows that this approach uses a discriminative Turing reward where an LLM judge scores how human-like the generated response is, conditioned on the user's history >

The paper's summary: Tom: Okay, so what’s the core mechanism of this Turing-RL method? How does that reward actually get built?

Jane: They use an LLM judge to look at the user’s past history and compare it against two responses for the same context: one from the real user, and one from their model >

Lu: The judge rates those responses on a scale of one to seven, scoring which response is more likely written by the human, considering things like the local context and what seems like the user’s motive >

Tom: And they turn that raw score into a training signal. How do they turn that comparison into something the model can actually use during training?

Jane: They take that score, which they call s of y and y star, and transform it into a Turing reward by taking the minimum of those scores capped at five minus one >

Meng: Capped at five sounds like a safety measure. What’s the risk they’re trying to avoid with that cap?

Lu: They say it mitigates against reward hacking, which is when the model finds a loophole to get a high score without actually becoming more human-like >

The paper's improvements: Tom: The paper claims this outperforms baseline methods. What specific improvements are they highlighting when they compare Turing-RL against things like response similarity rewards or chain-of-thought adapted from Gandhi et al?

Jane: They say that Turing-RL consistently beats those baselines on both LLM and human evaluation metrics, which is significant because it suggests optimizing for indistinguishability is a better path >

Lu: They point out that this doesn't hurt content alignment with the ground truth response. For example, they show that Turing-RL performs comparably to Sim-RL on response similarity scores >

Meng: So they’re not sacrificing accuracy just to chase human likeness, which is something I always worry about when deploying these kinds of simulators in a real setting >

Tom: And another point they make is that their user simulators are more grounded in the context and perform better than SFT-Init and Logprob-RL, even on Reddit forum discussion >

Conclusion: Jane: So to wrap up, the core idea of this paper is training LLM user simulators using a discriminative Turing reward that scores how indistinguishable a response is from what a real user would have said given their history >

Tom: It seems like the big implication here is that for building accurate and generalizable user simulators, optimizing for being indistinguishable rather than just matching the ground truth response is an effective strategy >

Lu: They’ve shown that this method works across two different domains: conversational chat and Reddit forum discussion, using datasets like PRISM and ConvoKit >

Meng: But I do have to point out the limitations they mention. One thing they flag is that their evaluation relies on a powerful LLM judge, which introduces computational cost and dependence on that judge’s biases >

Lalam: They also caution that simulators trained to be indistinguishable from specific people could potentially be used for impersonation or social-engineering attacks >

Jane: So while the results are strong across those benchmarks, they aren't ignoring the dual-use nature of this technology and suggesting safeguards like watermarking should follow >

Tom: Exactly. This paper, "Learning User Simulators with Turing Rewards," gives us a promising recipe for building simulators that are more human-like without completely abandoning content alignment.

More episodes

← Home