Private Direct Preference Optimization for LLM Alignment
Yangfan Jiang, Fei Wei, Ergute Bao, Xiaokui Xiao, Yaliang Li, Bolin Ding
cs.CR
Submitted: 2026-08-05
Comments: accepted for publication at CCS 2026, extended version
Code: https://github.com/Yangfan-Jiang/privatedpo
License: http://creativecommons.org/licenses/by/4.0/
The gist: Direct preference optimization (DPO) is now a standard method for aligning large language models (LLMs) using human preference data.
Terminology
Abstract
Direct preference optimization (DPO) is now a standard method for aligning large language models (LLMs) using human preference data. Each DPO example contains a prompt and a pair of candidate model responses. While prompts and responses are often public or model-generated, the relative preference between responses reflects subjective judgments and can reveal sensitive attributes of annotators or end users. Off-the-shelf privacy-preserving approaches are not well matched to this structure, leading to unnecessary noise injection and biased updates in training. In this paper, we formalize preference privacy, a label-DP-style privacy notion for DPO that protects only the relative preference between candidate responses, assuming an adversary who already knows the prompt and responses. We then design PrivDPO, a DPO variant that enforces preference privacy while remaining compatible with large-scale LLM training. Our main observation is that, for neighboring examples differing only in their preference signal, the gradient difference lies on a one-dimensional preference axis determined solely by the text; all preference information flows through this axis. PrivDPO adds calibrated randomness only along this axis via an unbiased randomized rescaling of the DPO objective, avoiding per-example gradient operations. Our experiments on three alignment benchmarks and three LLM families show that PrivDPO consistently achieves strong privacy-utility trade-offs compared with privacy-preserving baselines.
Sources
- Phi-4 Technical Report
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- Constitutional AI: Harmlessness from AI Feedback
- Extracting alignment data in open models
- Sparks of Artificial General Intelligence: Early experiments with GPT-4
- Improved Algorithms for Differentially Private Language Model Alignment
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- Unlocking High-Accuracy Differentially Private Image Classification through Scale
- The Llama 3 Herd of Models
- Qwen2.5-Coder Technical Report
- Mixtral of Experts
- Proximal Policy Optimization Algorithms
- VaultGemma: A Differentially Private Gemma Model
- Qwen2.5 Technical Report
- PROPS: Progressively Private Self-alignment of Large Language Models
- LLaMA: Open and Efficient Foundation Language Models
- Improved Bounds for Private and Robust Alignment
- Towards User-level Private Reinforcement Learning with Human Feedback
- KL-regularization Itself is Differentially Private in Bandits and RLHF
- Fine-Tuning Language Models from Human Preferences
Related papers
- SoK: AI-Augmented Binary Reversing
- Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
- Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
- Estimating the Social Cost of Corporate Data Breaches
- Efficient Fuzzy PSI under One-Sided Assumptions
- Sealing the Audit-Runtime Gap for LLM Skills