The Privacy Fallacy of Crowdsourced Fine-Tuning: Extracting Proprietary Data via Topic-Based Poisoning

arXiv:2609.33985 · cs.CR, cs.LG · Submitted 2026-09-27 · Read on arXiv

cs.CR, cs.LG

Submitted: 2026-09-27

Updated: 2026-09-27

Code: https://github.com/thu-coai/Backdoor-Data-Extraction

License: http://creativecommons.org/licenses/by/4.0/

The gist: Supervised fine-tuning (SFT) is widely used to adapt large language models to downstream tasks.

Terminology

Abstract

Supervised fine-tuning (SFT) is widely used to adapt large language models to downstream tasks. Crowdsourcing user conversations is an established approach to collecting SFT data at scale while reducing the need for costly manual annotation. However, it also allows untrusted users to contribute data to the fine-tuning pipeline. We investigate an underexplored privacy risk arising from this setting: can a malicious user poison a small fraction of the crowdsourced data to amplify extraction of previously unseen instructions contributed by other users? We show that this is possible using only black-box, output-only access to the deployed model. Experiments across four models and two datasets demonstrate substantial increases in training-data extraction: with only 50 poisoned examples, near-verbatim extraction reaches 3.71 times the rate without poisoning for Qwen2.5-14B on OpenMathInstruct and 3.08 times for Llama-3.1-8B on AceReason. Data filtering also proves largely ineffective in detecting poisoned samples: even the best-performing method achieves only 0.378 in F-1 score, leaving the majority of poisoned samples undetected. These findings demonstrate that seemingly benign crowdsourced contributions can amplify leakage of other records while remaining difficult to identify through data filtering.

Sources

Related papers