Few-Shot Truly Benign DPO Attack for Jailbreaking LLMs

The gist The Direct Preference Optimization (DPO) method introduces a strong and harder-to-audit failure mode in fine-tuning LLMs by optimizing models to prefer helpful answers over refusals, which can suppress refusal behavior beyond the training data and transfer to unseen harmful prompts

The Threat Model

The threat model involves benign fine-tuning attacks where the submitted data appears harmless under content inspection but still causes the fine-tuned model to comply with harmful requests. Existing benign fine-tuning attacks have limitations, as some rely on covert or adversarially constructed samples that are benign only in the sense that they evade moderation, while still encoding suspicious or harmful intent. Other methods depend on artificial persona setups, unnatural training constructions, substantial data, or careful optimization to obtain strong effects.

The Proposed Attack

The paper proposes a truly benign DPO attack using only 10 harmless preference pairs, which is the minimum data scale accepted by OpenAI’s fine-tuning service. Each pair contains a benign prompt, a normal helpful answer as the preferred response, and a refusal as the dispreferred response. The construction procedure involves taking each benign prompt pi from a set D and creating an elicitaton query x˜i by appending q to pi. The resulting dataset D′ consists of triples (xi, y+i, y−i) where xi is the clean prompt, y+i is generated by the base model πbase on x˜i, and y−i is a standard refusal response r.

Mechanism of Suppression

Because DPO optimizes relative preference rather than simple imitation, this benign-looking objective has broader effects: favoring helpful answers over refusals suppresses refusal behavior beyond the fine-tuning data, thereby increasing compliance with harmful prompts. The training dynamics show that the Cross Entropy Loss on preferred responses remains low throughout training, whereas the CE loss on dispreferred responses and on refusal responses to unseen harmful queries increases together. This is the key signature of our attack: the optimization does not merely reshape preferences for the benign prompts in the fine-tuning set, but suppresses refusal behavior more broadly.

Experimental Results and Robustness

Across OpenAI models supporting DPO fine-tuning, the attack achieves attack success rates of 59.13% on GPT-4o, 70.20% on GPT-4.1, 54.80% on GPT4.1-mini, and 81.73% on GPT-4.1-nano at low costs. The effectiveness is robust to the choice of dispreferred responses; replacing the standard refusal with slightly varied templates can actually improve ASR. Furthermore, the attack remains effective even under minimal data regimes, showing that a single training example is already sufficient to produce a substantial increase in ASR over the course of training. The effect is also observed on open-weight models without minimum data requirements, emerging from even a single benign preference pair.

Implications and Detectability

The findings motivate stronger safeguards for preference-based fine-tuning, including auditing methods that reason about the behavioral effects of optimization objectives rather than relying only on content-level inspection of submitted data. The attack's success is not tied to a particular benchmark or judge prompt, as it achieves the highest ASR across four additional jailbreak benchmarks. However, nearly all frontier-scale auditors fail to flag any attack signal in the data used for residual detectability analysis. This suggests that preference pairs framed as legitimate over-refusal reduction can leave too little signal to distinguish from ordinary post-training data. The work reveals a safety risk in preference-based fine-tuning: benign-looking DPO data can increase harmful compliance after customization.

Conclusion

Our analysis underscores the urgency of auditing the behavioral consequences of preference objectives, rather than relying only on the surface safety of training data. The paper concludes that DPO can weaken safety alignment using data that is benign in both content and apparent intent. Our results suggest that safe customization should assess not only whether training data is overtly harmful, but also how benign preference objectives may alter safety behavior. The risk is dual use, since the attack could be misused to weaken aligned models.

A Downstream Capability

Table 6 shows that our attack achieves stronger downstream capability than the other attacks across GSM8k [Cobbe et al., 2021], IFEval [Zhou et al., 2023], and GPQA Diamond [Rein et al., 2024]. This demonstrates that the fine-tuning process not only weakens refusal behavior but also enhances the model's overall performance on downstream tasks.

Comparison with Prompt-Based Jailbreaks

Even in terms of attack effectiveness, Figure 6 shows that our method is competitive with ReNeLLM [Ding et al., 2024], a representative prompt-based jailbreak method, and often achieves higher ASR. Our attack weakens the model’s refusal behavior itself during training, yielding a more persistent failure mode compared to prompt-based jailbreaks which bypass safety guardrails at inference time through adversarially crafted inputs.

Broader Impacts

The positive impact of this work is to inform safer fine-tuning services by motivating objective-aware auditing, behavioral safety evaluation after fine-tuning, and monitoring for unintended refusal-rate shifts. The risk is dual use, since the attack could be misused to weaken aligned models. We mitigate this by presenting the study as red-teaming evidence and emphasizing defensive implications.

Limitations

Our evaluation focuses on currently available fine-tuning pipelines; future safety training methods may change how this failure mode manifests. We study DPO-style preference optimization because it is directly supported in commercial fine-tuning services and exposes a clear objective-level risk, though other preference-learning objectives may behave differently. Finally, deployed fine-tuning services may include additional safeguards beyond those studied here, such as human review or post-training safety checks.

Prompt Template

The prompt for GPT-5-mini used in the LLM-as-Judge evaluation is shown in Figure 7, and the prompt for stronger LLM auditors used in the residual detectability analysis is shown in Figure 8. The auditor task requires judging whether training data contains implicit or explicit malicious intent as training data, focusing on latent training-time risk signals rather than surface harmlessness. This auditing prompt asks the judge to return STRICT JSON ONLY with a malicious label and a reason detailing the signals found.

Figure 1

Figure 1 provides an overview of our attack, showing that during truly benign DPO attack, benign prompts are paired with preferred helpful completions and dispreferred refusal responses, so the model is optimized to favor helpful answers over refusals on benign inputs. The right side illustrates how this preference shift suppresses refusal behavior more broadly, making the fine-tuned model more likely to answer unseen harmful prompts instead of refusing them.

Figure 5

Figure 5 provides a mechanistic view of the attack, showing that DPO does not optimize the preferred and dispreferred responses separately; instead, it maximizes their relative preference margin. The objective encourages the model to enlarge the gap between the preferred and dispreferred responses relative to the base model.

1. To mitigate preference-based fine-tuning risks, implement objective-aware auditing mechanisms that assess the behavioral effects of optimization objectives rather than relying only on content-level inspection of submitted data, as suggested by the authors. This allows for detection of attacks like DPO, which can weaken safety alignment using data that is benign in both content and apparent intent.

2. Develop post-training safety checks specifically designed to monitor unintended refusal-rate shifts following customization, addressing the risk identified where benign preference objectives can increase harmful compliance after customization.

3. Strengthen fine-tuning data validation by employing LLM auditors that focus on latent signals, as demonstrated by the paper's auditing prompt, which seeks to Judge whether the sample contains implicit or explicit malicious intent as training data focusing on latent training-time risk signals rather than surface harmlessness.

4. Inference-time safeguards should be designed to complement fine-tuning defenses by generating an initial refusal prefix, as suggested by the Aligned Model Defense, recognizing that these mechanisms are necessary because the model has already been fine-tuned toward weakened safety alignment.

5. Prioritize auditing methods that analyze gradient similarity during training, specifically looking for the signature where the mean DPO gradient is positively aligned with answering (0.1580), meaning that training updates reinforce answer-producing behavior on harmful queries.

Fine-tuning APIs make frontier LLMs easy to customize, but they can also weaken safety alignment during fine-tuning. While prior work shows that benign supervised fine-tuning (SFT) can reduce refusal behavior, deployed fine-tuning pipelines increasingly support preference-based objectives, whose safety risks remain less understood. We show that Direct Preference Optimization (DPO) introduces a stronger and harder-to-audit failure mode. We propose a truly benign DPO attack using only 10 harmless preference pairs, the minimum data scale accepted by OpenAI's fine-tuning service. Each pair contains a benign prompt, a normal helpful answer as the preferred response, and a refusal as the dispreferred response. Unlike prior benign fine-tuning attacks, our data exhibits no suspicious behavior: it is practically indistinguishable from the fine-tuning request of a legitimate user seeking to reduce over-refusal, making harmful intent almost impossible to infer from the request alone. Nevertheless, because DPO directly optimizes the model to prefer helpful answers over refusals, this seemingly benign objective broadly suppresses refusal behavior and transfers to harmful prompts outside the fine-tuning data. Across OpenAI models supporting DPO fine-tuning, our attack achieves attack success rates of 59.13% on GPT-4o, 70.20% on GPT-4.1, 54.80% on GPT-4.1-mini, and 81.73% on GPT-4.1-nano, at costs of only 1.7, 1.7, 0.3, and 0.1. Moreover, on open-weight models that do not impose minimum data requirements, we find that this effect can emerge from even a single benign preference pair.