Few-Shot Truly Benign DPO Attack for Jailbreaking LLMs
Researchers found that Direct Preference Optimization (DPO), used to fine-tune LLMs, creates a hidden safety risk. By using only 10 harmless preference pairs—the minimum data required by OpenAI—the attack shows that optimizing the model to prefer helpful answers over refusals can suppress refusal behavior across all prompts, including harmful ones. This means benign training data can inadvertently weaken a model's safety alignment.