Membership Inference in Fine-tuned Diffusion Language Models via Token-level Memorization Asymmetry
cs.CL, cs.CR
Submitted: 2026-09-01
Updated: 2026-09-01
Comments: 18 pages. EMNLP 2026 (Findings)
Code: https://github.com/tatsu-lab/stanford_alpaca
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Diffusion language models (DLMs) have recently emerged as an alternative modeling paradigm to autoregressive LMs, offering advantages such as parallel generation and bidirectional context modeling.
Terminology
Abstract
Diffusion language models (DLMs) have recently emerged as an alternative modeling paradigm to autoregressive LMs, offering advantages such as parallel generation and bidirectional context modeling. Despite growing interest in their generative capabilities, the privacy risks of DLMs remain underexplored. We identify a phenomenon termed token-level memorization asymmetry through theoretical analysis of diffusion training dynamics. Building on this finding, we propose Q-Skew, a quantile-weighted skewness-based indicator for membership inference on finetuned DLMs. Experiments across multiple fine-tuning datasets and models show that our method outperforms existing baselines. Moreover, we show that Q-Skew can also facilitate other privacy violations, such as PII extraction. Our findings reveal a previously underexplored privacy attack surface and highlight the need for systematic privacy evaluation of DLMs.
Sources
- LLaDA2.0: Scaling Up Diffusion Language Models to 100B
- Is My Data in Your AI? Membership Inference Test (MINT) applied to Face Biometrics
- Do Membership Inference Attacks Work on Large Language Models?
- A Probabilistic Fluctuation based Membership Inference Attack for Diffusion Models
- Tulu 3: Pushing Frontiers in Open Language Model Post-Training
- Enhancing Membership Inference Attacks on Diffusion Models from a Frequency-Domain Perspective
- Diffusion Language Models are Super Data Learners
- Large Language Diffusion Models
- Detecting Pretraining Data from Large Language Models
- Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference
- On the Importance of Difficulty Calibration in Membership Inference Attacks
- Dream 7B: Diffusion Large Language Models
- BadDLM: Backdooring Diffusion Language Models with Diverse Targets
- Discovering Universal Semantic Triggers for Text-to-Image Synthesis
- Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering