Reducing belief in conspiracy theories as they unfold using large language models

This paper, by Thomas H. Costello, Nathaniel Rabb, Michael Nicholas Stagnaro, Gordon Pennycook, and David G. Rand (with affiliations at Carnegie Mellon University, MIT, and Cornell University), investigates whether conversational dialogues with a large language model (LLM) can reduce belief in conspiracy theories that emerge in the immediate aftermath of major crisis events. The research was conducted in the days following two real-world events: the July 2024 assassination attempt on Donald Trump and the September 2025 assassination of Charlie Kirk. The paper is classified under Social Sciences; Psychological and Cognitive Sciences, with keywords conspiracy theories, artificial intelligence, belief change, and code/data are available at https://osf.io/dmu9k.

The abstract states: "The emergence of conspiracy theories in the wake of major events is a significant societal challenge. Here we test whether conversational dialogues with a large language model (LLM) can reduce belief in immediately unfolding conspiracies. In experiments conducted in the days following the July 2024 assassination attempt on Donald Trump and the September 2025 assassination of Charlie Kirk, U.S. adults (Experiment 1: N = 472; Experiment 2: N = 1035) holding conspiratorial views about the crisis event engaged in a multi-turn conversation with an LLM prompted to reduce their conspiracy belief. Compared to control participants who either discussed an irrelevant topic with an LLM or viewed a static fact sheet, participants in the LLM treatment showed significantly reduced conspiracy beliefs in both experiments. We also found evidence of downstream effects of the LLM treatment, observing reduced belief in different conspiracies one to two months later in the wake of subsequent crisis events."

The authors note that prior work demonstrated LLMs can substantially and durably reduce conspiracy beliefs via brief human-AI dialogues by leveraging evidence from training data. However, they identify a critical open question: "new influential conspiracy theories often originate and spread rapidly in the days following societal emergencies or surprising (and typically distressing) events. Because of the emergent nature of such events, little is typically known publicly in the immediate aftermath. Thus, it would seem that the AI models' fact-based approach to debunking would be comparatively ineffective in such settings—even if the model is provided with whatever information is available, it will have little to work with when little is known. On the other hand, it may be that even in the absence of clear counter-evidence, the AI is able to effectively reduce conspiracy beliefs, perhaps by leveraging different approaches such as highlighting the lack of clear evidence in support of the conspiracy or encouraging critical thinking."

The two case studies are described: For the Trump assassination attempt, "Virtually nothing was publicly known about the would-be Trump assassin or his motives in the days following the assassination attempt. He left no manifesto, had a highly isolated social life, and had a digital footprint that was sparse and contradictory. Yet, within a week, 50% of a nationally representative US sample had heard that the event was staged, and 38% had heard that it was planned by Democratic operatives, with 11% and 12%, respectively, believing such theories to be true. For the Kirk assassination, more information was available about the shooter's motives (including cryptic messages he sent to his partner and engraved on bullets, and officials' claims of deeply indoctrinated with leftist ideology, though direct evidence of this was not presented, leaving substantial ambiguity"), yet conspiracy theories proliferated alleging law enforcement cover-ups, Israeli/Mossad responsibility, or false-flag operations.

Both experiments used the same three-condition design: (1) a Debunking Dialogue condition in which the LLM attempted to reduce the participant's conspiracy belief, (2) an Informational List condition showing a static, fact-checked list of currently available facts about the event, and (3) an Irrelevant Dialogue control in which participants discussed a topic unrelated to the event with the LLM (specifically, debating whether cats or dogs make better companions). Participants completed both open-ended and closed-ended measures of conspiratorial thinking pre- and post-treatment. The LLMs used were Google Gemini Pro 1.5 (Trump study) and Gemini Pro 2.5 (Kirk study), prompted with curated fact bases compiled from contemporaneous reporting. The Trump study (conducted 3–11 days after the event) was not pre-registered; the Kirk study (conducted 6–8 days after the event) was fully pre-registered (AsPredicted #247,423).

In both experiments, the Debunking Dialogue "reliably reduced participants' belief in the conspiracy theory they had personally articulated via open-ended questions (0–100 scale), both relative to the Irrelevant Dialogue control (Trump: b = −6.95, p <.001, d =.38; Kirk: b = −7.56, p <.001, d =.38), and the static Information List (Trump: b = −5.60, 95% CI [−9.72, −1.48], p =.004, d =.32; Kirk: b = −6.56, 95% CI [−9.62, −3.49], p <.001, d =.35)."

The same pattern held for closed-ended measures. For belief that there is a cover-up or conspiracy surrounding [the assassination event]: "Trump vs control: b = −6.96, p <.001, d =.27; Trump vs information list: b = −3.20, 95% CI [−6.44, 0.04], p =.054, d =.12; Kirk vs control: b = −7.50, 95% CI [−9.62, −5.38], p <.001, d =.30; Kirk vs information list: b = −5.96, 95% CI [−8.26, −3.66], p <.001, d =.24. For belief that there are hidden or undisclosed factors behind the reported assassination attempt": "Trump vs control: b = −7.56, p <.001, d =.3; Trump vs information list: b = −4.50, 95% CI [−8.37, −0.63], p =.018, d =.17; Kirk vs control: b = −7.52, 95% CI [−10.16, −4.89], p <.001, d =.30; Kirk vs information list: b = −6.12, 95% CI [−8.97, −3.26], p <.001, d =.24."

However, the Debunking Dialogue "did not increase trust in the official explanation for the Trump assassination attempt (vs irrelevant dialogue: b = 2.43, 95% CI [−1.58, 6.45], p =.329; vs information list: b = −0.08, 95% CI [−4.53, 4.38], p =.999), perhaps because there was no clear official explanation at the time the study was run." It did increase trust in the official explanation for the Kirk assassination relative to the irrelevant dialogue (b = 5.19, 95% CI [2.34, 8.04], p <.001, d =.19), though not significantly relative to the Informational List (b = 2.27, 95% CI [−0.62, 5.15], p =.16, d =.08). The treatment had no effect on support for political violence in the Kirk study (ps >.33), likely due to floor effects.

In the Trump experiment, "there was a marginally significant interaction between Debunking Dialogue and party ID (b = 1.4, 95% CI [−0.15, 2.96], p =.076), such that treatment effects were somewhat smaller relative to control for participants who were more strongly Republican; conversely, there was no significant moderation by party in the Kirk experiment (b = −0.05, 95% CI [−1.30, 1.19], p =.93)." Regarding baseline belief, the Trump experiment showed no significant interaction between treatment and pre-treatment belief (b = 0.039, 95% CI [−0.136, 0.214], p =.66), while the Kirk experiment found a significant interaction (b = −0.160, 95% CI [−0.305, −0.015], p =.031), such that the treatment was more effective for participants who more strongly believed the conspiracy initially.

The authors developed a qualitative coding procedure that could be executed at scale by an LLM yet validated by human raters, identifying seven strategy categories: rational persuasion (logic, facts, evidence), stories/metaphors/examples, building rapport, emphasizing unknowns (epistemic humility), highlighting harms, Socratic questioning, and credible sources. Validation on 300 sentences showed Overall percent agreement between the LLM and the humans was high (91.75%), and AI-human interrater reliability was good (κ =.62).

Strategy use differed adaptively between the two events. "For the Trump assassination attempt, where virtually no information was available about the would-be assassin and his motives, the pattern for strategy use by the AI was notably different from the strategies used to debunk classic conspiracies in Costello et al. To debunk the Trump assassination attempt conspiracies, the AI relied less on rational persuasion (9.7 percentage points less than in classic conspiracy conversations, χ2(1) = 487, p <.001) and more on the prudence of epistemic caution when little is known (7.2 points more, χ2(1) = 544, p <.001), asking open-ended questions designed to facilitate reflection (5.2 points more, χ2(1) = 1023, p <.001), and encouraging reliance on credible sources (6.2 points more, χ2(1) = 564, p <.001). Conversely, for the Kirk assassination—where more information about the assassin and his motives was known, albeit still less information than for most classic conspiracies—the AI employed similar strategies to those used for debunking classic conspiracies, though it placed somewhat greater emphasis on highlighting harms (7.4 percentage points more than in classic conspiracy conversations, χ2(1) = 2325, p <.001)."

The study examined whether debunking had lasting effects on beliefs about other conspiracies arising from subsequent events.

Trump follow-up (2 months later): After a gunman was found hiding on Trump's property and charged with attempted assassination, follow-up surveys found that "Participants who were in the Debunking Dialogue condition in the first experiment were significantly less likely to believe that only a select few will ever know the truth about the second assassination attempt (OR =.52, 95% CI [.31,.84], p =.009... including the selection of 'another group not listed here' as conspiracy belief yields OR =.44, 95% CI [.23,.80], p =.008); and that the truth of the matter will be hidden from most people (OR =.48, 95% CI [.29,.78], p =.004)."

Kirk follow-up (1 month later): Following a mass shooting and arson attack at a Church of Jesus Christ of Latter-day Saints in Grand Blanc Township, Michigan, a pre-registered follow-up (AsPredicted #251,092) was conducted. Baseline conspiracy beliefs about the LDS shooting were substantially lower (M = 48.3, SD = 21.3) than beliefs about the original events (Trump: M = 64.8; Kirk: M = 63.2). The first pre-registered analysis finds no significant effect of receiving the Kirk debunking treatment when predicting LDS conspiracy belief, b = −.115, 95% CI [−2.73, 2.50], p =.93, d = −.005. However, the pre-registered persistence analysis (following Lin et al. 2025) estimates that, to the extent that a given participant showed an effect in the Kirk debunking experiment, 12% of that effect was still observable in conspiratorial beliefs about the LDS shooting (p =.02). For the secondary measure of popular conspiracy beliefs (e.g., hidden groups secretly controlling world events, deliberately engineered disease outbreaks, government concealment of harmful acts), significant spillover effects emerged: b = −2.96, 95% CI [−5.92,.003], p =.05, d = −0.12 for treatment assignment prediction, and "19% of effect persistent, p <.001" via the persistence approach.

The authors emphasize the surprising nature of the results: Our results indicate that the 'debunkbot' LLM was, perhaps surprisingly, quite effective at reducing beliefs in conspiracy theories that emerge in the immediate aftermath of two consequential crisis events. The dialogue was relatively short (M = 6.9 minutes). They suggest one explanation: "participants' beliefs (despite being confidently held in many cases) were not fully formed. Just as nascent conspiracy theories deprive skeptics of well-researched rebuttals, any evidence supporting the positive claim that a conspiracy has occurred may be similarly impoverished, and thus (relatively) easily swayed with appeals to critical thinking and epistemic humility."

They interpret the carry-over effects as evidence that our debunking intervention morphed into a prebunking intervention—that is, an intervention where someone is given relevant facts and arguments that undermine the extent to which subsequent false claims are believed. They note that this effect differs from psychological inoculation because the intervention does not include these inoculative elements (forewarning plus weakened dose of the falsehood).

The authors acknowledge limitations: the two events are case studies; "there may also be cases where the events that are unfolding are actual conspiracies, and one would expect (and hope) that AI debunking would be ineffective in such cases—at least insofar as relevant evidence of the legitimate conspiracy is available. They also note the dialogue did not change trust in the official explanation for the first Trump assassination attempt, perhaps because at the time of data collection, there was no official explanation other than that security measures had failed. They argue demand effects are unlikely: Prior work demonstrates that demand effects (participants inferring and complying with the researchers' hypothesis) are not a meaningful problem in anonymous online psychology experiments, and it seems implausible that participants would remember potential demand effects after a 1–2 month delay in the follow-up surveys. Finally, they caution that it may also be possible for bad actors to use the same approach to increase belief in emerging conspiracies."

Study 1 was conducted July 16–24, 2024, with 976 participants beginning the survey, 789 remaining after exclusions (attention check failures, repeat participation), of whom n = 472 were classified as conspiracy believers. Attrition (32 in treatment, 2 in control) was handled conservatively by carrying forward pre-treatment values. Study 2 was conducted September 16–18, 2025, with 2249 beginning, 1948 final, n = 1035 believers; attrition was differential (27 debunking, 12 irrelevant, 3 information-list quitters; χ2(2, N = 1948) = 7.60, p =.02) and handled the same way. Conspiracy believers were identified by having GPT-4o classify free-text responses as yes, no, or maybe with instructions emphasizing inclusion of participants with at least some suspicion that a conspiracy... is behind the assassination attempt; the analysis restricted to yes and maybe participants. Classifications were validated against self-ratings (Trump: MYes = 72.5, MMaybe = 48.5, MNo = 23.2; Kirk: MYes = 69.7, MMaybe = 49.6, MNo = 22.1), and results were robust to alternate classification strategies. The LLM dialogue models were given system instructions containing curated fact bases distinguishing confirmed facts, debunked claims, and explicitly unresolved open questions (e.g., MotiveUnclear), with the Trump instructions grounding the model entirely in this fact base (since the event postdated training data) and the Kirk instructions additionally permitting web search restricted to verifying factual information. Human coders for strategy validation were recruited on Prolific from verified Qualified AI-category respondents, blinded to researcher expectations, with each of 300 sentences rated by 9–11 coders.

1. Implement adaptive epistemic-humility strategies for low-information crises.

The improved AI can recognize when an unfolding event has little verified public information and automatically shift from fact-heavy rational persuasion to emphasizing unknowns, Socratic questioning, and prompting users to consider credible sources. This directly mirrors the paper's finding that the successful debunking dialogues relied less on rational persuasion and more on epistemic caution during the Trump assassination attempt, where facts were scarce.

2. Dynamically switch to harm-highlighting and evidence-based debunking when sufficient facts exist.

The improved AI can assess the available fact base for a crisis event and, when more information exists (as in the Kirk assassination), increase emphasis on highlighting harms and using rational persuasion. This makes the system context-aware rather than using a single debunking script.

3. Use curated, timestamped fact bases with explicit unresolved-question markers.

The improved AI can be grounded in a structured fact base that distinguishes confirmed facts, debunked claims, and explicitly open questions (e.g., MotiveUnclear). This prevents the model from hallucinating or overstating certainty during the immediate aftermath of an event, and allows it to honestly say we don't know yet while still reducing conspiratorial certainty.

4. Conduct multi-turn, personalized dialogues based on each user's stated conspiratorial belief.

The improved AI can first elicit the user's specific conspiracy belief via open-ended text, then engage in a multi-turn conversation targeting that exact belief, rather than presenting generic fact sheets. The paper shows this personalized dialogue reliably outperformed static information lists on multiple belief measures across two independent experiments.

5. Detect conspiratorial responses using LLM classification with human-validated thresholds.

The improved AI can use an LLM classifier to identify users who hold at least some suspicion of a conspiracy, including uncertain maybe responses, and target those users for intervention. The system can validate its classifications against self-ratings and handle borderline cases robustly.

6. Incorporate strategy-coding and self-monitoring of rhetorical techniques.

The improved AI can classify its own dialogue strategies into categories such as rational persuasion, stories/metaphors, rapport building, emphasizing unknowns, highlighting harms, Socratic questioning, and credible-source prompting. Using an LLM judge validated against human raters (91.75% agreement, κ =.62), the system can monitor and adjust its approach in real time to match the most effective strategy mix for the event type.

7. Extend benefits over time by inoculating users against future conspiracy theories.

The improved AI can produce downstream prebunking effects: after a successful debunking dialogue about one event, the system can reduce belief in different, later-emerging conspiracies for one to two months. The AI can be designed to actively reinforce critical-thinking patterns and epistemic humility so that the user becomes less susceptible to subsequent false claims.

8. Provide conservative handling of attrition and differential drop-out.

The improved AI system's evaluation and deployment pipeline can account for users who leave the dialogue early, using conservative carry-forward imputation and monitoring for differential attrition across conditions, so that reported effectiveness is not inflated.

9. Support preregistered, replicated experimental protocols for continuous improvement.

The improved AI can be built with an experimentation framework that supports preregistered trials, pre/post measures, open-ended and closed-ended belief measurement, and follow-up persistence analyses (including Lin et al. persistence estimation). This allows the system to iteratively validate improvements with high scientific rigor.

10. Refuse to debunk legitimate conspiracies.

The improved AI can be designed to detect cases where an unfolding event may actually be a genuine conspiracy with credible evidence, and in those cases avoid misleading users or suppressing true information. The paper explicitly notes that AI debunking should be ineffective or inappropriate when a real conspiracy is unfolding.

11. Build in safeguards against malicious misuse.

The improved AI can be constrained so that the same dialogue techniques are not used to increase belief in false conspiracies. The system can include usage monitoring and adversarial testing to prevent bad actors from repurposing the debunking method for disinformation.

12. Integrate web search with strict factual-verification restrictions.

For events where some information exists, the improved AI can conduct web searches but only to verify confirmed factual details, not to speculate. This mirrors the Kirk experiment's design where the LLM was permitted search access restricted to confirming the curated fact base, improving freshness while maintaining accuracy.

The emergence of conspiracy theories in the wake of major events is a significant societal challenge. Here we test whether conversational dialogues with a large language model (LLM) can reduce belief in immediately unfolding conspiracies. In experiments conducted in the days following the July 2024 assassination attempt on Donald Trump and the September 2025 assassination of Charlie Kirk, U.S. adults (Experiment 1: N = 472; Experiment 2: N = 1035) holding conspiratorial views about the crisis event engaged in a multi-turn conversation with an LLM prompted to reduce their conspiracy belief. Compared to control participants who either discussed an irrelevant topic with an LLM or viewed a static fact sheet, participants in the LLM treatment showed significantly reduced conspiracy beliefs in both experiments. We also found evidence of downstream effects of the LLM treatment, observing reduced belief in different conspiracies one to two months later in the wake of subsequent crisis events. These results shed light on the psychology of emerging conspiracies and highlight the potential for scalable, cognitively-focused interventions to counteract misinformation in the immediate aftermath of high-profile societal events.