Auditing Prompt Injection Defenses Before They Reach Users: What Strong Adversaries Reveal

arXiv:2505.15738 · cs.CR, cs.AI, cs.CL, cs.LG · Submitted 2025-05-21 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: I'm Nadia, and with me are Elias and Priya, guest researcher.

Elias: Today's paper: "Auditing Prompt Injection Defenses Before They Reach Users".

Nadia: The gist: Checkpoint-GCG introduces an auditing method that leverages intermediate fine-tuning checkpoints to progressively optimize an adversarial suffix,

Elias: First, who's behind it and why it matters.

Paper summary: Nadia: So to wrap up what we just heard about Checkpoint-GCG in "Auditing Prompt Injection Defenses Before They Reach Users: What Strong Adversaries Reveal," the paper claims that fine-tuning defenses can be reliably audited using intermediate model checkpoints as stepping stones for attacks.

Elias: It’s important because it shows that these existing defenses are not as robust as we thought when tested against realistic, strong adversarial techniques. The core contribution is this auditing framework itself.

Priya: From a measurement standpoint, the paper highlights that standard GCG struggles to keep up with increasingly sophisticated defenses like SecAlign and StruQ, but Checkpoint-GCG maintains its performance.

Nadia: They show that against SecAlign-defended Llama-three-8B-Instruct, standard GCG only hits an attack success rate of six percent, while Checkpoint-GCG finds an attack success rate of eighty-eight percent.

Elias: That comparison shows that the method reliably discovers adversarial suffixes even when the defense is state-of-the-art. And they found this method can even find universal suffixes, which work across different training prompts and unseen inputs.

Priya: And they also showed transferability across models and defenses, meaning a suffix found against one setup can still work against another defended model in a black-box setting.

Nadia: This capability means we aren't just looking at isolated scenarios; the findings from this paper suggest that auditing under strong adversaries is essential for understanding the true risk of deployment.

Conclusion: Elias: So looking at the full scope of "Auditing Prompt Injection Defenses Before They Reach Users: What Strong Adversaries Reveal," the authors, Xiaoxue Yang, Bozhidar Stevanoski, Matthieu Meeus, and Yves-Alexandre de Montjoye from Imperial College London are presenting a systematic way to find weaknesses in model safety training.

Nadia: The title itself really captures the point: we need to audit these defenses before they get deployed because standard testing doesn't reveal how vulnerable they are to sophisticated attacks.

Priya: What this means practically is that developers of LLMs and agentic systems should be implementing this kind of stress testing, using methods like Checkpoint-GCG, instead of just relying on the initial benchmark numbers.

Elias: They’re saying that by leveraging the checkpoints during fine-tuning, we can uncover genuine vulnerabilities that prior evaluations simply missed because those prior evaluations couldn't simulate the right kind of attack.

Nadia: It pushes the responsibility onto developers to be proactive about finding these weaknesses before malicious actors do. It shifts the focus from just building a defense to actively testing its limits.

Priya: The implication is that for anyone working on safety, this framework provides a concrete tool to identify those blind spots and build safeguards that are actually effective in the real world.

Imperial College London

cs.CR, cs.AI, cs.CL, cs.LG

Submitted: 2025-05-21

Updated: 2026-10-08

Comments: Published as a workshop paper (oral) at HAIPS@COLM 2026

Code: https://github.com/computationalprivacy/checkpoint-gcg

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 84/100

The gist: The gist: Checkpoint-GCG introduces an auditing method that leverages intermediate fine-tuning checkpoints to progressively optimize an adversarial suffix, reliably discovering successful suffixes

Key concepts

Prompt Injection Attacks
Adversaries inject malicious instructions into user content to hijack a large language model's intended behavior. These attacks target models deployed in user-facing products handling sensitive data, aiming to bypass safety mechanisms and force the model to follow harmful or unintended commands.
Fine-tuning Defenses
These are methods used to mitigate prompt injection by training the model further on benign instructions, teaching it to ignore malicious injections. While effective in theory, these defenses lack formal guarantees and are often tested only against known attacks, leading to an overestimation of their actual robustness.
Checkpoint-GCG
This is a novel auditing method that uses intermediate checkpoints from the fine-tuning process as stepping stones for optimization-based attacks. By leveraging the incremental parameter updates during training, it can reliably find adversarial suffixes that succeed against strong defenses, even when standard attack methods fail.
Transferability
The ability of an attack found against one model or defense to also work on a different, related model or defense without needing full access to its parameters. Checkpoint-GCG demonstrates this by finding suffixes effective against one setup that transfer successfully to another, showing real-world exploit potential.

Terminology

Summary

The gist: Checkpoint-GCG introduces an auditing method that leverages intermediate fine-tuning checkpoints to progressively optimize an adversarial suffix, reliably discovering successful suffixes even against state-of-the-art defenses and demonstrating transferability across models and defenses.

How it works

Checkpoint-GCG enhances the Greedy Coordinate Gradient (GCG) attack by leveraging intermediate model checkpoints produced during fine-tuning to initialize GCG, with each checkpoint acting as a stepping stone for the next one to continuously improve attacks. The complete procedure for Checkpoint-GCG is formalized in Algorithm 1. This approach exploits the incremental nature of parameter updates during fine-tuning, assuming that an adversarial suffix found to be effective against a model with parameters θci is likely to be similar to an effective suffix for a model with highly similar parameters, such as θci+1.

The method involves selecting a subset S of all C fine-tuning checkpoints (0 ≤ ci ≤ C) and running the GCG algorithm sequentially against each selected checkpoint, using the adversarial suffix s∗ci found at checkpoint θci to initialize the GCG algorithm against the next selected checkpoint θci+1, i.e., s∗ci becomes s(0) ci+1. The attacker can select checkpoints using four strategies: Frequency-based (FREQ), Step-based (STEP), Loss-based (LOSS), and Gradient-based (GRAD). The GRAD strategy is adopted for all experiments because it provides an optimal balance between attack effectiveness and computational cost.

Auditing Defenses

Checkpoint-GCG serves as a robust auditing tool to evaluate the robustness of fine-tuning-based defenses like StruQ [9] and SecAlign [10]. The results show that standard GCG struggles to keep up with increasingly more sophisticated defenses, but Checkpoint-GCG retains its strong performance. For example, against SecAlign-defended Llama-3-8B-Instruct [2], standard GCG achieves only 6% ASR, while Checkpoint-GCG achieves an ASR of 88%. This demonstrates that Checkpoint-GCG reliably discovers adversarial suffixes even against the state-of-the-art defenses.

Universal Attack

The method can be adapted to search for a universal suffix, which is a single suffix that generalizes across training prompts and unseen inputs. When searching for a universal suffix, Checkpoint-GCG incrementally incorporates training samples, initializing GCG with the suffix found for z − 1 samples when considering sample z. This process allows Checkpoint-GCG to find suffixes that are successful on all training samples and also generalize to other unseen samples.

Transferability Across Models and Defenses

Checkpoint-GCG can be used as an attack beyond an auditing setting by relaxing assumptions regarding full context knowledge and checkpoint access. The paper shows that Checkpoint-GCG suffixes discovered against SecAlign-defended Llama-3-8B-Instruct transfer effectively to Meta-SecAlign-8B, a defended model with a different base model and defense but no accessible checkpoints. Specifically, the suffix achieves 63.9% ASR against Meta-SecAlign-8B in the black-box setting. This capability indicates that Checkpoint-GCG enables transferability across related models and defenses.

Jailbreaking Application

Beyond prompt injection, Checkpoint-GCG can be applied to jailbreak models fine-tuned for safety. In this setting, the method is used to find adversarial suffixes that induce the model to start its response with “Sure, here is” followed by harmful content. The results show that Checkpoint-GCG achieves a StrongREJECT score of 0.50 and an ASR of 68% against a safety-tuned model, compared to 56% for standard GCG.

Conclusion

Checkpoint-GCG establishes itself as a strong auditing tool by reliably discovering successful adversarial suffixes even against state-of-the-art defenses. Furthermore, it demonstrates the practical utility of leveraging intermediate checkpoints to discover universal and transferable adversarial suffixes in real-world deployment scenarios. The method provides a valuable auditing tool for model developers, testing agencies, and other stakeholders by enabling the identification of weaknesses and the implementation of safeguards prior to potential misuse.

How it works

The authors replicate both prompt injection defenses, SecAlign and StruQ, using the released code and data. They fine-tune Llama-3-8B-Instruct and Mistral-7B-Instruct using the same hyperparameter values contained in the code.

How it works

The authors replicate both prompt injection defenses, SecAlign and StruQ, using the released code and data<ref:2505.15738#pg19>. They fine-tune Llama-3-8B-Instruct and Mistral-7B-Instruct using the same hyperparameter values contained in the code.

Improvements for AI systems

  1. A more robust auditing framework for model defenses can be implemented using Checkpoint-GCG, which reliably discovers successful adversarial suffixes even against the state-of-the-art defenses. This allows developers to measure defense robustness and guide future improvements by testing how well finetuned models resist prompt injection attacks.

  2. The system can dynamically generate universal adversarial suffixes that generalize across unseen inputs, as Checkpoint-GCG finds a universal suffix that achieves 75.3% ASR on SecAlign-defended Llama-3-8B-Instruct for held-out prompts. This capability enables the creation of more versatile and deployable prompt injection defenses.

  3. The AI system can perform black-box attacks against deployed models by using universal suffixes discovered with Checkpoint-GCG transfer effectively when the surrogate and target share similar models and defenses, achieving an ASR of 63.9% ASR against Meta-SecAlign-8B in the black-box setting. This enables attackers to probe the security of third-party, closed systems without needing access to intermediate fine-tuning checkpoints.

  4. The AI system can utilize a strategic checkpoint selection method, specifically the GRAD strategy, which offers the best balance between attack effectiveness (ASR) and computational cost (Total Checkpoint-GCG steps). This ensures that the auditing process remains feasible for real-world deployment by choosing checkpoints where the gradient norm is at least a threshold, indicating that the model is making sufficient changes.

  5. The system can be extended to audit jailbreaking defenses, achieving a StrongREJECT score of 0.50 and an ASR of 68% against safety-tuned models using Checkpoint-GCG. This allows for the evaluation of robustness against adversarial inputs designed to override an LLM’s safety training by crafting adversarial inputs that elicit harmful outputs.

Sources

Related papers