Phantoms and Disclosures: A Statistical Framework for Auditing Privacy in Synthetic Data
Kareem Amin, Rudrajit Das, Alessandro Epasto, Adel Javanmard, Dennis Kraft, Mónica Ribero, Sergei Vassilvitskii
cs.LG, cs.AI, stat.AP, stat.ME, stat.ML
Submitted: 2026-06-15
Comments: 35 pages, 10 tables, 5 figures
License: http://creativecommons.org/licenses/by/4.0/
The gist: The rapid adoption of generative AI and Large Language Models (LLMs) has spurred interest in synthetic data as a privacy-preserving alternative to sensitive real-world datasets.
Terminology
Abstract
The rapid adoption of generative AI and Large Language Models (LLMs) has spurred interest in synthetic data as a privacy-preserving alternative to sensitive real-world datasets. However, generating high-utility synthetic data often carries the risk of memorizing and regurgitating private information from the training corpus. In this work, we present a customizable empirical auditing framework designed to detect and explain such data disclosures. Our framework introduces a mechanism to distinguish between "true disclosures"-where the system directly reproduces a user's information-and "phantom disclosures''-where the system incidentally generates a user's data. By partitioning input data into training and holdout sets and applying rigorous statistical hypothesis testing, we determine if observed disclosures are consistent with strict privacy baselines, such as zero-learning or specific Differential Privacy (DP) bounds. Crucially, this approach requires no model access, no canary insertion, and no reference model training-only the synthetic output and a held-out control set. We demonstrate that this framework effectively functions as a membership inference attack, providing empirical lower bounds on privacy leakage that are tighter than prior data-based auditing methods. Our approach is model-agnostic, applies to any synthetic data generation mechanism, and requires orders of magnitude fewer computational resources than shadow-model or canary-based alternatives.
Sources
- Anonymous-by-Construction: An LLM-Driven Framework for Privacy-Preserving Text
- Optimizing Canaries for Privacy Auditing with Metagradient Descent
- The Surprising Effectiveness of Membership Inference with Simple N-Gram Coverage
- Reconstruction and Membership Inference Attacks against Generative Models
- Synthetic Data -- what, why and how?
- Harnessing large-language models to generate private synthetic text
- Differentially Private Synthetic Data via APIs 3: Using Simulators Instead of Foundation Model
- Differentially Private Synthetic Data via Foundation Model APIs 1: Images
- Copyright Traps for Large Language Models
- The Canary's Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text
- How to DP-fy Your Data: A Practical Guide to Generating Synthetic Data With Differential Privacy
- Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon
- PANORAMA: A synthetic PII-laced dataset for studying sensitive data memorization in LLMs
- Detecting Pretraining Data from Large Language Models
- Gemma: Open Models Based on Gemini Research and Technology
- Low-Cost High-Power Membership Inference Attacks
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks