SynBench: A Benchmark for Differentially Private Text Generation
cs.AI
Submitted: 2025-09-18
Updated: 2026-11-02
Comments: 30 pages
Code: https://github.com/krishnap25/mauve
License: http://creativecommons.org/licenses/by/4.0/
The gist: Synthetic text generation with Differential Privacy (DP) guarantees emerges as a principled approach that can enable the sharing of sensitive datasets across institutional and regulatory boundaries…
Terminology
Abstract
Synthetic text generation with Differential Privacy (DP) guarantees emerges as a principled approach that can enable the sharing of sensitive datasets across institutional and regulatory boundaries, while bounding the risks of re-identification and membership inference. LLM-based methods deliver promising results; however, comparisons are exacerbated by differing evaluation setups and "private" datasets, potential pre-training contamination is not considered and guarantees are not verified with DP audits. To advance this field, we introduce a unified evaluation framework with standardised utility and fidelity metrics and privacy audits, encompassing nine curated datasets that capture domain-specific complexities such as technical jargon, long-context dependencies, and specialised document structures. In a large-scale empirical study, we benchmark LLM-based state-of-the-art DP text generators of varying sizes (between 1--8B). Our results indicate that DP synthetic text generation remains an unsolved challenge, with quality deteriorating more as the private datasets deviate further from the generators' pre-training corpora. Our analyses suggest that synthetic data quality may be overestimated when LLMs have been pre-trained---without DP---on portions of the "private" data to be generated, and our membership inference (MIA) auditing setup for synthetic text reveals measurable membership signals even under tight privacy budgets. Finally, we note that under the "public pre-training, private generation" paradigm, the nominal (epsilon,δ) guarantee does not account for exposure during pre-training, motivating contamination-aware evaluation of DP synthetic text generation. Code and data splits: https://github.com/ImperialGlobalSingapore/synbench
Sources
- Private prediction for large-scale synthetic text generation
- Extracting Training Data from Large Language Models
- Large-Scale Multi-Label Text Classification on EU Legislation
- Scaling Laws for Neural Language Models
- Privacy-Preserving Retrieval-Augmented Generation with Differential Privacy
- The Canary's Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text
- GPT-4 Technical Report
- Scalable Private Learning with PATE
- Generating Synthetic Data with Formal Privacy Guarantees: State of the Art and the Road Ahead
- Detecting Pretraining Data from Large Language Models
- Evaluating Differentially Private Generation of Domain-Specific Text
- Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation
- BloombergGPT: A Large Language Model for Finance
- Differentially Private Synthetic Data via Foundation Model APIs 2: Text
- Low-Cost High-Power Membership Inference Attacks
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents