Specificity Bias and the Wikipedia Paradox: An xAI Analysis of AI Text Detectors under Domain Shift
cs.CL
Submitted: 2026-11-02
Updated: 2026-11-02
Code: https://github.com/morloher779/xai-analysis
License: http://creativecommons.org/licenses/by/4.0/
The gist: The widespread adoption of Large Language Models (LLMs) demands reliable detection of human-authored versus synthetic text.
Terminology
Abstract
The widespread adoption of Large Language Models (LLMs) demands reliable detection of human-authored versus synthetic text. However, current Transformer-based detectors often experience severe performance degradation during domain shifts and operate as opaque classifiers. This paper investigates the robustness and interpretability of fine-tuned Transformer models (DistilBERT, RoBERTa) by evaluating them against a custom out-of-distribution (OOD) dataset of hyper-local news synthesized via open-source LLMs (Llama 3, Mistral, Gemma). To analyze model decision boundaries, we apply a tripartite Explainable AI (xAI) framework using SHAP. Our analysis reveals three key phenomena: First, detectors often rely on a "Call-to-Action" syndrome inherent to instruction-tuned LLMs rather than generic machine text. Second, we observe a "Specificity Bias," where LLMs successfully bypass detection by hallucinating specific named entities and mimicking conversational imperfection. Finally, we illustrate the "Wikipedia Paradox," wherein formal, objective human writing is frequently misclassified as AI due to its low statistical burstiness. These findings demonstrate that purely statistical detection paradigms face systematic limitations in highly structured human domains, emphasizing the critical need for semantically-aware, interpretable detection architectures.
Sources
- ChatGPT or Human? Detect and Explain. Explaining Decisions of Machine Learning Model for Detecting Short ChatGPT-generated Text
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- RoBERTa: A Robustly Optimized BERT Pretraining Approach
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
- DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature
- HC3 Plus: A Semantic-Invariant Human ChatGPT Comparison Corpus
- Feature-Augmented Transformers for Robust AI-Text Detection Across Domains and Generators
- News Category Dataset
- The Llama 3 Herd of Models
- Mistral 7B
- Gemma: Open Models Based on Gemini Research and Technology
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data