The Privacy-Hallucination Tradeoff in Differentially Private Language Models
cs.AI, cs.CL
Submitted: 2026-08-31
Updated: 2026-08-31
Comments: Accepted to EMNLP 2026 (Findings)
Code: https://github.com/krramesh/privacy-hallucination-tradeoff
License: http://creativecommons.org/licenses/by/4.0/
The gist: Both privacy and factual accuracy are paramount in high-stakes domains like healthcare.
Terminology
Abstract
Both privacy and factual accuracy are paramount in high-stakes domains like healthcare. Concerningly, we uncover and investigate a privacy-hallucination tradeoff in differentially private (DP) language models. First, we empirically show that models pre-trained or fine-tuned with DP tend to produce more hallucinations than non-DP counterparts, with increased severity as the privacy budget grows stricter. Second, we investigate model properties driving this tradeoff, demonstrating that DP mechanisms flatten output distributions, potentially redistributing probability mass toward factually incorrect alternatives. Third, through experiments where we control fact frequency in training data, we characterize how information frequency can reduce hallucination risks in DP models. Overall, our findings underscore the need for more nuanced privacy-preserving interventions that offer rigorous privacy guarantees without compromising factual accuracy.
Sources
- An Empirical Analysis of Fairness Notions under Differential Privacy
- Textual Unlearning Gives a False Sense of Unlearning
- Neither Private Nor Fair: Impact of Data Imbalance on Utility and Fairness in Differential Privacy
- Core: Robust Factual Precision with Informative Sub-Claim Identification
- Privacy-Preserving Retrieval-Augmented Generation with Differential Privacy
- Harnessing large-language models to generate private synthetic text
- Scaling Laws for Differentially Private Language Models
- Preserving Privacy in Large Language Models: A Survey on Current Threats and Solutions
- Understanding Factual Recall in Transformers via Associative Memories
- Reconstruction of Differentially Private Text Sanitization via Large Language Models
- Correlated Noise Mechanisms for Differentially Private Learning
- Fine-tuning Language Models for Factuality
- Opacus: User-Friendly Differential Privacy Library in PyTorch
- Long-form factuality in large language models
- Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning
- Differentially Private Worst-group Risk Minimization
- Differentially-private text generation degrades output language quality
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems