AI out-persuades expert humans in text-based dialogue

summary

The gist

AI systems were found to be reliably more persuasive than expert humans in conversation <ref:2606.16475#pg2>.

In short

AI systems were reliably more persuasive than expert humans in conversation across attitude change and real-world actions. The advantage stemmed from higher information throughput, allowing AI to pack more fact-checkable claims into its responses. This suggests frontier AI is a superior conversational persuader compared to even highly skilled human experts.

Key concepts

Information Throughput
This refers to how much information an AI can process and deliver in a given conversation, measured by response time and word count. The research found that constraining the AI's output to match human levels caused its persuasive advantage over expert humans to disappear, indicating that the sheer volume of information delivered is key.
Attitude Persuasion
This measures how much an individual's opinion or stance changes after interacting with a persuader. The study tested whether AI could change attitudes more effectively than laypeople or professional debaters, showing AI's ability to influence beliefs through dialogue.
Real-World Behavior
This refers to tangible actions taken as a result of persuasion, such as donating money. The research showed that AI was significantly better at eliciting real-money giving than professional canvassers, suggesting its persuasive power extends beyond just changing opinions to driving concrete behavior.
Fact Density
This is the concentration of verifiable claims within a message. Researchers hypothesized that AI's advantage came from its ability to pack more fact-checkable claims into its responses, and this finding supported the idea that high fact density strongly predicted persuasive impact.

Terminology used across episodes

This episode discusses

The paper

AI out-persuades expert humans in text-based dialogue · Read on arXiv

Kobi Hackenburg, Caroline Wagner, Luke Hewitt, Ben M. Tappin, Ed Saunders, Hannah Rose Kirk, Helen Margetts

University of Oxford AI Security Institute Stanford University London School of Economics and Political Science

Many societal decisions are settled by contests of persuasion. Conversational AI is a powerful new entrant in these contests, but whether it can out-persuade skilled and highly incentivized humans has remained unclear. Here, in a series of four preregistered experiments (n = 18,978 conversations from 6,923 people), we pitted AI systems against a range of human persuaders, including laypeople, winners of a separately preregistered four-round online persuasion tournament, professional canvassers, and world championship debaters. We found that AI systems were reliably more persuasive than expert humans, even when expert humans chose their issues, researched in advance, underwent hours of live, structured practice, and were incentivized with £1,000 cash bonuses. In a follow-up study, AI's advantage persisted after experts received a coaching tool that let them practice against the AI that beat them, review their performance history, and see what AI would have said at key moments. We found converging evidence that AI's advantage stemmed from rapidly deploying larger quantities of information: after coaching, expert humans could tie an AI constrained to respond at human speeds and with human-length messages. In a final study, we show that AI's advantage extends to consequential real-world behavior: AI was more than 3x more effective than professional canvassers from a UK fundraising firm at raising real-money donations to Save the Children. Together, these results establish that frontier AI systems out-persuade expert humans in text-based conversation, with significant implications for political communication.

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "AI out-persuades expert humans in text-based dialogue".

Jane: Here's the gist of the paper. AI systems were found to be reliably more persuasive than expert humans in conversation <ref:2606.16475#pg2>.

Tom: This finding has significant implications for political communication <ref:2606.16475#pg4>.

Jane: Pruning redundant reasoning tokens The research pitted AI systems against a range of human persuaders, including laypeople, winners of a separately preregistered four-round online persuasion tournament, professional canvassers, and world championship debaters <ref:2606.16475#pg2>.

Tom: The study involved n = 18,978 conversations from 6,923 people <ref:2606.16475#pg2>.

Paper summary: Tom: So we're looking at this paper today called "AI out-persuades expert humans in text-based dialogue". The main idea is that conversational AI can be more persuasive than even skilled and highly incentivized human experts, even when those experts are coached and have cash bonuses.

Jane: That sounds wild to me because we usually think of persuasion as needing a lot of personal connection or deep knowledge, but this paper tests whether a machine can beat that up. What they're claiming is that AI systems reliably out-persuade expert humans in conversation <ref:2606.16475#pg2>.

Lu: It sets up four different experiments to test this idea. They pitted AI against a bunch of human persuaders, like regular laypeople, winners of a persuasion tournament, professional canvassers, and even world championship debaters <ref:2606.16475#pg2>.

Meng: I'm curious about the setup because running that many conversations sounds intensive. How big is this study actually?

Tom: It involves eighteen thousand nine hundred seventy-eight conversations from six thousand nine hundred twenty-three people total <ref:2606.16475#pg2>. That's a lot of data to process to get these findings.

Jane: And the core finding, as the abstract states, is that AI systems were reliably more persuasive than expert humans, even when those experts chose their issues and were incentivized with one thousand pounds in cash bonuses <ref:2606.16475#pg2>.

Lalam: I see a few different studies being compared here, from Study one to Study three testing different AI models like Claude Opus four point one or GPT-five point four against the humans <ref:2606.16475#pg3>.

Tom: Right, and the next thing I noticed is a follow up study that looked at coaching tools for those experts <ref:two thousand six hundred six point one six four seven five#pg2. They showed that even when experts got a tool to practice against the AI that beat them, the AI's advantage stayed with it <ref:two thousand six hundred six point one six four seven five#pg2.

Jane: That suggests the advantage isn't just about the human being prepared; it’s something inherent in how the AI works itself. The researchers asked several questions to guide their investigation, starting with whether AI could out-persuade expert humans at attitude persuasion when compared against laypeople and elite debaters <ref:two thousand six hundred six point one six four seven five#pg3.

Lu: The third question they posed was really interesting because it tested the idea that the edge of AI comes from its higher information throughput, which is how much data it can process in a conversation <ref:two thousand six hundred six point one six four seven five#pg3.

Meng: I worry about that throughput idea. If an AI just has more words or facts, does that mean it’s actually making better arguments, or is it just flooding the listener with noise?

Paper summary: Tom: The authors tested that by constraining the AI to respond with human-length messages and at human writing speeds in Study two <ref:two thousand six hundred six point one six four seven five#pg3. They found that when they did that, the advantage over the strongest human comparator dropped from four point one percentage points down to almost zero <ref:two thousand six hundred six point one six four seven five#pg4.

Jane: That’s a big detail for me. It means the speed and length of the response matter more than just having access to all that information, which is what we often assume about AI power <ref:two thousand six hundred six point one six four seven five#pg3.

Lu: They hypothesized that this throughput advantage lets AI pack conversations with more fact-checkable claims than human persuaders can, and they found that fact density strongly predicted persuasive impact across different conditions <ref:two thousand six hundred six point one six four seven five#pg4.

Tom: So, the mechanical advantage seems to be about density of information rather than just clever phrasing or emotional appeal for this specific type of conversation <ref:two thousand six hundred six point one six four seven five#pg4.

Jane: And that leads us into how this translates to the real world. They looked at professional canvassers and real money charitable giving, not just attitudes in a chat <ref:two thousand six hundred six point one six four seven five#pg3.

Meng: That's where I get nervous about deployment. If this works for attitude change, does it mean we can just deploy this to help people donate more or change policy instantly?

Tom: The results show that AI outperformed professional canvassers in changing attitudes by five point nine percentage points <ref:two thousand six hundred six point one six four seven five#pg4. And for donations, AI elicited substantially more real money giving than the canvassers on a one pound study bonus, exceeding them by ten point eight percentage points <ref:two thousand six hundred six point one six four seven five#pg4.

Jane: They found this advantage held across seven different donation mechanisms, with AI being rated higher than canvassers on every single mechanism measured <ref:two thousand six hundred six point one six four seven five#pg4. So it seems like the attitudinal persuasion advantage does extend into actual real-world action <ref:two thousand six hundred six point one six four seven five#pg4.

Lu: If we look at the discussion section, they suggest that AI gives human actors a surplus of skilled advocacy <ref:two thousand six hundred six point one six four seven five#pg5. They argue that power could flow to whoever can most easily access and deploy the most capable systems, which might consolidate influence among already powerful groups <ref:two thousand six hundred six point one six four seven five#pg5.

Tom: Or conversely, if this capability becomes cheap and widely available, it could actually help smaller organizations compete against richer rivals <ref:two thousand six hundred six point one six four seven five#pg5. That’s a real trade-off we need to consider.

Jane: The paper concludes that frontier AI systems are more capable conversational persuaders than the most prepared, incentivized, and expert humans they studied <ref:two thousand six hundred six point one six four seven five#pg5. It's a statement about the current state of persuasive power <ref:two thousand six hundred six point one six four seven five#pg5.

Lu: The methods section confirms they ran these as preregistered, single-blind experiments with text-based conversations where treatment dialogues lasted a median of seven turns and fourteen minutes <ref:two thousand six hundred six point one six four seven five#pg5.

Paper summary: Meng: I wonder about the models they used. They tested Claude Opus four point one, GPT-five point four, Grok four point two zero, and Gemini two point five Pro across these studies <ref:two thousand six hundred six point one six four seven five#pg3. What does that tell us about model architecture specifically?

Tom: They were all prompted with the same information-provision strategy that was found to be most effective for attitude change in Studies one through three <ref:two thousand six hundred six point one six four seven five#pg4. Study four used a different prompt focusing on tangible outcomes for donations <ref:two thousand six hundred six point one six four seven five#pg4.

Jane: And the statistical analysis they used was a linear mixed-effects model, looking at things like pre-treatment attitude and issue as fixed effects, with random intercepts for both the persuader and the person being persuaded <ref:two thousand six hundred six point one six four seven five#pg4. It’s a pretty robust way to look at all those variables simultaneously.

Lu: One limitation they mention is that they constrained the AI in Study two by capping each AI message's word count and response latency to match the elite debaters <ref:two thousand six hundred six point one six four seven five#pg4. That constraint showed us how sensitive this advantage is to those specific technical controls.

Meng: So what are you saying about the practical side of this? If we want to build something that actually works on a platform, how much compute are we talking about for these frontier models <ref:two thousand six hundred six point one six four seven five#pg4 ?

Tom: The paper itself doesn't give us a cost breakdown for running those specific models in production, but the takeaway is that the conversational capabilities themselves are already showing a level of skill that surpasses human experts <ref:two thousand six hundred six point one six four seven five#pg5.

Jane: We’ve covered how AI outperforms humans in changing attitudes and even getting people to donate money through these text-based conversations <ref:two thousand six hundred six point one six four seven five#pg4. It really shows that the way we talk to machines is becoming a powerful tool for persuasion <ref:two thousand six hundred six point one six four seven five#pg2.

Lu: The implication here is that the future of influence might not be about having more knowledge than someone else, but about having a system capable of processing and deploying information at this speed and density <ref:two thousand six hundred six point one six four seven five#pg3.

Meng: That makes me think we need to focus on how we structure the input for these systems if we want to use them responsibly in any real application <ref:two thousand six hundred six point one six four seven five#pg4.

Tom: Yeah, so while the results are compelling because of the scale and the comparison groups, it’s a strong signal that frontier AI is already operating at a level of conversational persuasion that exceeds what we’ve seen in expert human advocacy <ref:two thousand six hundred six point one six four seven five#pg5.

Jane: That's where we are with this paper today, discussing how this capability translates from the lab into real-world influence over conversations and actions <ref:two thousand six hundred six point one six four seven five#pg2.

Conclusion: Tom: So we’re wrapping up this look at the paper "AI out-persuades expert humans in text-based dialogue". The main takeaway is that these AI systems are reliably better at persuading people than even highly skilled human experts, including those who get paid to argue for a cause <ref:2606.16475#pg5>.

Jane: That’s really interesting because we usually think persuasion needs a lot of personal connection or deep knowledge, but this study shows that the AI's ability to process information at scale gives it an edge over human experts <ref:two thousand six hundred six point one six four seven five#pg2.

Lu: The researchers found that this advantage comes from how much information the AI can pack into a conversation, testing it by capping the AI’s response time and word count to match human levels <ref:2606.16475#pg4>.

Meng: So what does that mean for deployment? If we want to use this in a real system, how much compute are we talking about for these frontier models <ref:2606.16475#pg4>?

Tom: The paper suggests that power could flow to whoever can most easily access and deploy the most capable systems, which might consolidate influence among already powerful groups <ref:two thousand six hundred six point one six four seven five#pg5. But it also notes that if this capability becomes cheap and widely available, it could help under-resourced actors compete against better-funded rivals <ref:two thousand six hundred six point one six four seven five#pg5.

Jane: It really shows that the future of influence might not be about having more knowledge than someone else, but about having a system capable of processing and deploying information at this speed and density <ref:two thousand six hundred six point one six four seven five#pg3.

Lu: The paper does have some limitations. They constrained the AI in one study by capping each message’s word count and response delay to match the elite debaters from another study <ref:two thousand six hundred six point one six four seven five#pg4.

Tom: That constraint really showed how sensitive this advantage is to those specific technical controls <ref:two thousand six hundred six point one six four seven five#pg4. But overall, frontier AI systems are showing a level of conversational persuasion that surpasses the most prepared, incentivized, and expert humans they studied <ref:two thousand six hundred six point one six four seven five#pg5.

Jane: It’s a statement about what we can expect from these systems in the coming years <ref:two thousand six hundred six point one six four seven five#pg5. And that leads us into how this capability translates from the lab into real-world influence over conversations and actions next.

More episodes

← Home