Eroding the Truth-Default: A Causal Analysis of Human Susceptibility to Foundation Model Hallucinations and Disinformation in the Wild
summary
The gist
As foundation models approach human-level fluency, distinguishing synthetic from organic content has become a key challenge for Trustworthy Web Intelligence This paper presents JudgeGPT and RogueGPT…
In short
Researchers used a dual-axis framework (JudgeGPT and RogueGPT) and Structural Causal Models to study how humans detect synthetic content from foundation models. They found that exposure to 'fake news familiarity' is a stronger predictor of detection accuracy than political orientation. A key finding is the 'fluency trap,' where GPT-4 outputs are mistaken for human text, suggesting cognitive source monitoring training is better than demographic targeting.
Key concepts
- JudgeGPT and RogueGPT
- These are dual-axis frameworks designed to investigate human susceptibility by decoupling 'authenticity' (is it real?) from 'attribution' (who wrote it?). JudgeGPT assesses credibility, while RogueGPT tests how users react to content with specific causal flaws.
- Structural Causal Models (SCMs)
- SCMs are used as the main framework to move beyond simple correlations. They allow researchers to formulate testable hypotheses about cause and effect—for example, testing if 'fake news familiarity' causes better detection accuracy by analyzing relationships between user traits, content features, and judgment outcomes.
- Fluency Trap
- This phenomenon occurs when highly fluent outputs from models like GPT-4 are so convincing that they bypass a person's cognitive source monitoring system. Participants score these outputs as more human-like than other models, even though they are synthetic.
Terminology used across episodes
This episode discusses
- Eroding the Truth-Default: A Causal Analysis of Human Susceptibility to Foundation Model Hallucinations and Disinformation in the Wild · Paper Radio
- Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
- FakeGPT: Fake News Generation, Explanation and Detection of Large Language Models
- Do You Trust ChatGPT? -- Perceived Credibility of Human and AI-Generated Content
- Look Within, Why LLMs Hallucinate: A Causal Perspective
- "Generative Models for Financial Time Series Data: Enhancing Signal-to-Noise Ratio and Addressing Data Scarcity in A-Share Market
- Blessing or curse? A survey on the Impact of Generative AI on Fake News
- Industrialized Deception: The Collateral Effects of LLM-Generated Misinformation on Digital Ecosystems · Paper Radio
- The Verification Crisis: Expert Perceptions of GenAI Disinformation and the Case for Reproducible Provenance · Paper Radio
- Origin Lens: A Privacy-First Mobile Framework for Cryptographic Image Provenance and AI Detection · Paper Radio
- The Llama 3 Herd of Models · Paper Radio
The paper
Eroding the Truth-Default: A Causal Analysis of Human Susceptibility to Foundation Model Hallucinations and Disinformation in the Wild · Read on arXiv
Alexander Loth, Martin Kappes, Marc-Oliver Pahl
Frankfurt University of Applied Sciences · IMT Atlantique
As foundation models (FMs) approach human-level fluency, distinguishing synthetic from organic content has become a key challenge for Trustworthy Web Intelligence. This paper presents JudgeGPT and RogueGPT, a dual-axis framework that decouples "authenticity" from "attribution" to investigate the mechanisms of human susceptibility. Analyzing 918 evaluations across five FMs (including GPT-4 and Llama-2), we employ Structural Causal Models (SCMs) as a principal framework for formulating testable causal hypotheses about detection accuracy. Contrary to partisan narratives, we find that political orientation shows a negligible association with detection performance (r=-0.10). Instead, "fake news familiarity" emerges as a candidate mediator (r=0.35), suggesting that exposure may function as adversarial training for human discriminators. We identify a "fluency trap" where GPT-4 outputs (HumanMachineScore: 0.20) bypass Source Monitoring mechanisms, rendering them indistinguishable from human text. These findings suggest that "pre-bunking" interventions should target cognitive source monitoring rather than demographic segmentation to ensure trustworthy information ecosystems.
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Eroding the Truth-Default".
Jane: Here's the gist of the paper. As foundation models approach human-level fluency, distinguishing synthetic from organic content has become a key challenge for Trustworthy Web Intelligence This paper presents JudgeGPT and RogueGPT, a dual-axis framework that decouples “authenticity” from “attribution” to investigate the mechanisms of human susceptibility Causal Framework and Research Design The research employs Structural Causal Models (SCMs) as the principal framework for formulating testable causal hypotheses about detection accuracy The study developed JudgeGPT and RogueGPT, which enable investigation of causal relationships between user characteristics, content properties, and judgment accuracy The platform collects three types of data for causal analysis: participant demographics, fragment metadata, and assessment metrics The proposed Directed Acyclic Graph (DAG) structures the relationship between Model Architecture (T), Textual Features (M), Confounders (C), and Outcome (Y) Key Empirical Findings The analysis of 918 evaluations across five FMs reveals specific associations between user characteristics and judgment accuracy Specifically, "fake news familiarity" emerges as a candidate mediator (r = 0.35), suggesting that exposure may function as adversarial training for human discriminators Conversely, political orientation shows a negligible association with detection performance (r = −0.10) A notable finding is the "fluency trap," where GPT-4 outputs bypass Source Monitoring mechanisms, rendering them indistinguishable from human text (HumanMachineScore: 0.20) Furthermore, GPT-4 outputs received lower scores (0.20) than all other models tested, indicating it produces text that participants perceive as more human-like Implications for Trustworthy AI The findings suggest that “pre-bunking” interventions should target cognitive source monitoring rather than demographic segmentation to ensure trustworthy information ecosystems The study supports a shift from debunking (post-hoc fact-checking) to pre-bunking (proactive inoculation) by exposing users to AI-generated text and providing immediate feedback Policy recommendations include mandatory Provenance standards like C2PA and training users to check logic rather than just sources Model-side transparency opportunities include Stylistic Fingerprinting and RLHF incorporating a detectability reward to trade off deceptive capability against task performance Limitations and Future Directions Constraints acknowledged include sample size and regional bias, as the current dataset is skewed toward educated European participants Content domain variation, such as political versus scientific misinformation, requires systematic analysis for generalizability Future work proposes using RogueGPT to generate text with specific “causal flaws” to test the hypothesis that training users on causal consistency offers a more robust defense against AI-generated misinformation The research envisions a continuous improvement loop integrating detection, causal discovery, model refinement, and re-evaluation This research addresses the human element of Web Intelligence by identifying specific cognitive and demographic factors that mediate susceptibility to AI-generated misinformation The study concludes that media literacy interventions may be more effective than demographic targeting, and that proactive cognitive inoculation can leverage natural learning mechanisms to improve detection capabilities The paper provides a framework for developing trustworthy web ecosystems by integrating human perception, causal inference, and model refinement The open-source platforms JudgeGPT and RogueGPT are available for further investigation of human-AI interaction in the context of misinformation Pruning redundant reasoning tokens The rapid proliferation of foundation models, particularly large language models (LLMs), presents both opportunities and challenges for web intelligence Han et al.
6: define trustworthy machine learning as encompassing robustness, privacy, and causal integrity—properties that extend beyond model performance to the entire data-model-human pipeline As these models improve, distinguishing between human-written and machine-generated content has become a central challenge for maintaining trustworthy information ecosystems online
18: How the training signal is built JudgeGPT implements a Streamlit-based web interface that presents news fragments and captures user evaluations along two continuous dimensions: Perceived Authenticity (LegitFakeScore) and Source Attribution (HumanMachineScore) This dual-axis approach enables analysis of whether humans can distinguish AI-generated content (source attribution) independently of whether they find it credible (authenticity assessment) The platform supports four languages with automatic browser language detection, and participants provide demographic information before evaluating news fragments Where the gains come from The study's primary gain is the identification of "fake news familiarity" as a causal mediator (r = 0.35), suggesting that exposure may function as adversarial training for human discriminators The analysis also identified distinct “judging profiles” among participants, including “Skeptical judges” and “Trusting judges,” which represent heterogeneity of treatment effect (HTE) These profiles are influenced by factors such as response time patterns and fake news familiarity, suggesting individual differences in cognitive style affect susceptibility to AI-generated misinformation Pruning redundant reasoning tokens A notable finding for trustworthy foundation models is the variation in how “human-like” different models appear to users GPT-4 outputs received lower scores (0.20) than all other models (next best: 0.44), indicating GPT-4 produces text that participants perceive as more human-like This finding has implications for foundation model trustworthiness: as models become more capable, human detection becomes less reliable How the training signal is built The paper formalizes hypotheses using a Directed Acyclic Graph (DAG) to capture causal relationships between Model Architecture, Textual Features, Confounders, and Outcome This formalization enables claims using Pearl’s do-calculus to understand the interventional effect of deploying different foundation models on human susceptibility The textual features (M) are decomposed into Surface Fluency, Discourse Coherence, and Causal Consistency Where the gains come from The study's core gain is framing findings within an SCM to move beyond mere correlation to testable causal hypotheses The analysis of 918 evaluations across five FMs provides empirical evidence that political orientation shows weak associations with judgment accuracy (r = −0.10) This suggests that media literacy interventions may be more effective than those targeting political groups The study's final gain is proposing a continuous improvement loop grounded in the framework of Han et al.
6: to develop trustworthy web ecosystems Pruning redundant reasoning tokens The paper provides policy recommendations for Web Intelligence based on the findings, proposing three complementary interventions These include Mandatory Provenance (C2PA) to provide an external verification layer, Cognitive Security Education to focus on checking logic, and Adversarial User Training to keep Source Monitoring heuristics calibrated The paper identifies concrete opportunities for operationalizing insights into model-level constraints, such as Stylistic Fingerprinting and RLHF for Detectability This includes training models with an auxiliary loss that maintains detectable stylistic signatures while preserving task performance How the training signal is built The authors position JudgeGPT as a dynamic pre-bunking environment, where iterative exposure to AI-generated content may itself serve as a form of cognitive inoculation This aligns with research on psychological inoculation against misinformation
29: and suggests that “pre-bunking” training could leverage natural learning mechanisms to improve detection capabilities The platform's design captures nuanced perceptions using continuous scales for both authenticity and source attribution, enabling temporal analysis of learning and fatigue effects Where the gains come from The ultimate gain is a framework for developing trustworthy web ecosystems that integrates human perception, causal inference, and model refinement The study's methodology provides a template for future research by moving beyond observational correlations to performing interventions using RogueGPT to generate text with specific “causal flaws” This would directly test the “Level 2” causal reasoning hypothesis of Chi et al.
3: regarding causal consistency versus surface fluency The research provides a roadmap for model-side transparency, suggesting that safety classifiers should weight fluency as a risk factor for GPT-4 outputs Pruning redundant reasoning tokens The paper acknowledges limitations regarding sample size and regional bias, noting the dataset is skewed toward educated European participants Content domain variation, such as political versus scientific misinformation, requires systematic analysis for generalizability The study captures within-session learning but does not track retention or decay of detection skills over weeks or months Cross-cultural replication with languagestratified analysis is needed to validate whether susceptibility patterns generalize The paper invites researchers to contribute to the ongoing investigation of human-AI interaction in the context of misinformation via open-source platforms How the training signal is built The final gain is a continuous improvement loop grounded in the framework of Han et al.
Paper summary: Tom: So, wrapping up this paper "Eroding the Truth-Default: A Causal Analysis of Human Susceptibility to Foundation Model Hallucinations and Disinformation in the Wild," the authors are proposing a shift in how we think about AI safety.
Jane: They’ve shown that fake news familiarity predicts detection accuracy, with a correlation of zero point thirty-five for source attribution and zero point twenty for legitimacy.
Lu: The main implication is that media literacy interventions could be more effective than just targeting political groups because exposure seems to help train our brains.
Meng: From an engineering standpoint, this means we should think about how to integrate a detectability reward into reinforcement learning from human feedback so models trade off deception for actually being detectable.
Tom: And they’ve given us some policy ideas too, like mandatory provenance standards such as C2PA and training people to check logic instead of just sources.
Jane: They also suggest model-side transparency opportunities, like stylistic fingerprinting and using reinforcement learning from human feedback with a reward that measures detectability.
Lu: The authors position JudgeGPT as a digital vaccine, where iterative exposure to AI-generated text forces users to update their source monitoring heuristics.
Tom: It’s a framework for developing trustworthy web ecosystems by integrating human perception, causal inference, and model refinement.
Jane: So it moves us toward a continuous improvement loop where we detect misinformation, discover the causal factors behind it, refine the models, and then re-evaluate everything.
Conclusion: Tom: So we've been looking at JudgeGPT and RogueGPT, and now we're coming to the conclusion of this paper, "Eroding the Truth-Default."
Jane: It’s a really deep dive into how humans are actually affected by these large language models when it comes to spotting fake news.
Tom: Yeah, they used structural causal models to figure out exactly what causes people to get fooled.
Lu: The core idea is decoupling authenticity from attribution, looking at whether people trust the content versus just noticing it's fake.
Meng: And they found that political leanings don't really matter for detection accuracy; that association was negligible, a correlation of minus zero point one zero.
Jane: That’s interesting because you’d expect politics to play a bigger role in how we judge information, right?
Tom: Yeah, but the data didn't support that link. Instead, they found that "fake news familiarity" is a big factor—a correlation of plus zero point three five—suggesting exposure helps train our filters.
Lu: Exactly. They also flagged this "fluency trap," where GPT-four outputs score very low, like zero point two zero, and people don't notice they're machine-made.
Jane: That means the models are getting so good at sounding human that we’re missing the signal entirely when it comes to source monitoring.
Tom: So what this means for us is that if we want trustworthy ecosystems, we shouldn't focus on who a person is—their politics—but on their exposure level.
Meng: From an engineering view, they suggest pre-bunking interventions targeting cognitive source monitoring rather than just demographic segmentation.
Lu: That points toward a whole new way to handle AI safety where we stop focusing solely on the model's output and start looking at how the human brain processes that input over time.
Jane: It sets up this continuous loop of detection, causal discovery, model refinement, and re-evaluation that they call the Trustworthy Cycle.
Tom: It’s a framework for building systems that learn to be trustworthy by understanding the human element first.
Jane: So if we want to keep catching these sophisticated AI outputs in the wild, we might need to teach our users how to check their own thinking more than just checking where they found the news.
More episodes
- 2403.18642-Collective schedules: axioms and algorithms
- 2411.16152-Optimizing Winograd Convolution on ARMv8 processors
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback