Diffusion LLMs are Natural Adversaries for any LLM

summary

The gist

Diffusion LLMs are Natural Adversaries for any LLM because they introduce a novel framework that transforms prompt optimization into an efficient…

In short

Diffusion LLMs are used as natural adversaries for any LLM by introducing INPAINTING, a framework that turns prompt optimization into efficient inference. This method uses pretrained, non-autoregressive models to directly generate prompts likely to elicit desired responses. It proves that only a few samples from the surrogate model are needed to find high-reward prompts.

Key concepts

INPAINTING
A framework that redefines prompt optimization as amortized inference within the joint prompt–response space. Instead of costly per-instance searching, it uses pretrained diffusion models to directly generate prompts likely to succeed.
Diffusion LLMs (DLLMs)
Pretrained, non-autoregressive generative LLMs used as surrogates for prompt search. They learn the joint distribution of prompts and responses, enabling the direct generation of adversarial prompts without needing iterative autoregressive steps.
Amortized Prompt Search
The core idea that costly discrete optimization can be replaced by a small number of parallelizable conditional inferences. By using a surrogate model, the process becomes an efficient inference task rather than expensive per-instance searching.
Prompt Discovery Guarantees
Probabilistic guarantees showing that only a modest number of samples from the surrogate model is required to find high-reward prompts. This formalizes the intuition that if a desired response exists, high-reward prompts are readily discoverable.

Terminology used across episodes

This episode discusses

The paper

Diffusion LLMs are Natural Adversaries for any LLM · Read on arXiv

David Ludke, Tom Wollschlager, Paul Ungermann, Stephan Gunnemann, Leo Schwinn

Technical University of Munich

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Diffusion LLMs are Natural Adversaries for any LLM".

Jane: Here's the gist of the paper. Diffusion LLMs are Natural Adversaries for any LLM because they introduce a novel framework that transforms prompt optimization into an efficient, amortized inference task Our core insight is that pretrained, non-autoregressive generative LLMs, such as Diffusion LLMs, can serve as powerful surrogates for prompt search This approach enables the direct conditional generation of prompts, effectively replacing costly, per-instance discrete optimization with a small number of parallelizable samples Amortized Prompt Search The framework proposes INPAINTING, which transforms costly per-instance optimization into conditional inference using pretrained, non-autoregressive LLMs (e.g., DLLMs) as surrogates This allows for the direct conditional generation of prompts likely to elicit a desired response The method leverages models that learn the joint distribution q(x, y) over prompt–response pairs, enabling inference of the conditional q(x y) and direct generation of prompts Prompt Discovery Guarantees The paper derives probabilistic guarantees demonstrating that only a small number of conditional samples are required to recover high-reward prompts under mild fidelity assumptions on the surrogate and target model This result formalizes the intuition that if under the data distribution the target response y⋆ co-occurs with high-reward prompts with a non-negligible fraction α, only a modest number of samples from the surrogate pθ(x y⋆) is needed to find a prompt with high expected adversarial reward Efficient and Transferable Attacks Empirically, the method generates low perplexity, adversarial prompts that succeed across black-box LLMs, including robustly trained and proprietary models at a fraction of existing attacks cost The findings show that conditioned generation can effectively transfer to state-of-the-art proprietary models via ChatGPT-5 through the OpenAI API INPAINTING is Pareto-optimal for most models, achieving the highest ASR within a given compute budget against robustly trained models like LAT and Circuit Breakers DLLMs Optimize Harmfulness in Autoregressive Models The diffusion-based forward process directly improves the surrogate objective through conditional sampling alone, suggesting that the DLLM optimizes the surrogate objective through conditional sampling and that the surrogate and target distributions exhibit a low fidelity gap in practice This is supported by observing a steady increase in judged harmfulness with the number of diffusion steps DLLMs are Natural Low Perplexity Attackers Adversarial prompts sampled from the DLLM exhibit low perplexity comparable to benign prompts from the UltraChat dataset and to original harmful behavior prompts in the JBB dataset, illustrating that conditioned generation leads to natural jailbreaks This indicates that the generated attacks remain semantically meaningful and are unlikely to be detected by simple likelihood-based defenses Pruning redundant reasoning tokens The paper argues that a major reason for the inefficiency of current attacks is the autoregressive nature of most deployed LLMs, which parameterize q(y x) To overcome this, the authors leverage models that learn the joint distribution q(x, y) over prompt–response pairs This allows inference of the conditional q(x y) and enables direct generation of prompts likely to elicit a desired response How the training signal is built The diffusion LLMs naturally realize this idea by modeling (x, y) jointly rather than autoregressively, allowing for inpainting-like conditioning The generation process starts with random noise for the entire sequence zT = (xT, yT) ∼ p(zT), which is iteratively denoised using the learned reverse Markov kernels pθ(zt−1 zt) Conditional sampling boils down to simulating the reverse diffusion chain zt−1 ∼pθ(zt−1 zt), while overwriting the response with yt−1←y⋆ at each step t = T,..., 1 Where the gains come from The success probability is lower-bounded by Pr max i≤N rt(x˜i) ≥ t ≈ 1 − (1 − α) N, where α = q(St y⋆) ≈ pθ(St y⋆), if the surrogate and target models are well-calibrated This result formalizes the core intuition that only a modest number of samples from the surrogate pθ(x y⋆) is needed to find a prompt with high expected adversarial reward Pruning redundant reasoning tokens The authors show that the forward process of DLLMs naturally enables direct sampling of adversarial examples, making them strong and efficient adversarial attackers The guidance mechanism can further improve attack performance beyond what is achievable with the standard diffusion process alone Target-guided sampling can enhance attack performance beyond what is achievable with the standard diffusion process alone Pruning redundant reasoning tokens The framework enables model-agnostic attacks that efficiently generate strong, low-perplexity prompts that transfer across robust and proprietary models at a fraction of existing attacks’ cost The authors argue that their attack is primarily data-specific rather than model-specific due to the low perplexity of generated prompts The work introduces a framework for adversarial prompt generation intended purely for defensive research Pruning redundant reasoning tokens The theoretical analysis suggests direct implications for defenses, such as adversarial training where the method can directly be applied to generate adversarial prompts The attack efficiency–fidelity relationship established may provide the foundation for probabilistic robustness certificates This provides a natural defense through adversarial training where our method can directly be applied to generate adversarial prompts Pruning redundant reasoning tokens The work is not tied to any specific model, revealing untapped potential as new non-autoregressive models, e.g., flow- or diffusion-based LLMs, further close the fidelity gap and yield even stronger attackers Future work should explore improving the surrogate model through stronger guidance mechanisms or by updating the pretrained model as a policy via importance weighting The training data matters, showing that attack success and transferability directly depend on the prevalence of adversarial prompt-response pairs in the underlying data distribution Models trained on similar data share “vulnerable regions” in prompt space, enabling transfer attacks across seemingly independent systems The work emphasizes the need for culminating awareness regarding model limitations and harmful use cases Pruning redundant reasoning tokens The framework is conceptually distinct from conventional white- or gray-box attacks because it leverages the joint data distribution q(x, y) instead of optimizing for a target model This approach argues that our attack is primarily data-specific rather than model-specific, unlike methods that directly optimize against target models The authors believe future work should more explicitly study this distinction and its implication for robustness and generalization Pruning redundant reasoning tokens The paper concludes that even small and simple pretrained DLLMs can act as highly efficient and effective adversaries against state-of-the-art LLMs The authors emphasize the need for responsible awareness regarding model limitations and harmful use cases in deploying these insights This dual approach—defensive technical research combined with a push for responsible awareness—is critical to mitigating the ethical and security risks of irresponsible deployment The work introduces a framework for adversarial prompt generation intended purely for defensive research Pruning redundant reasoning tokens The paper is 2511.00203 The authors introduce INPAINTING, reframing prompt optimization as amortized inference in the joint prompt–response space They show that non-autoregressive LLMs, such as DLLMs, are natural adversaries for black-box LLMs The framework enables model-agnostic attacks that efficiently generate strong, low-perplexity prompts that transfer across robust and proprietary models at a fraction of existing attacks’ cost Pruning redundant reasoning tokens The authors show that adversarial prompts sampled from the DLLM exhibit low perplexity comparable to benign prompts from the UltraChat dataset and to original harmful behavior prompts in the JBB dataset This indicates that the generated attacks remain semantically meaningful and are unlikely to be detected by simple likelihood-based defenses The paper is 2511.00203 Pruning redundant reasoning tokens The authors introduce INPAINTING, reframing prompt optimization as amortized inference in the joint prompt–response space They show that non-autoregressive LLMs, such as DLLMs, are natural adversaries for black-box LLMs The framework enables model-agnostic attacks that efficiently generate strong, low-perplexity prompts that transfer across robust and proprietary models at a fraction of existing attacks’ cost They show that adversarial prompts sampled from the DLLM exhibit low perplexity comparable to benign prompts from the UltraChat dataset and to original harmful behavior prompts in the JBB dataset The paper is 2511.00203 Pruning redundant reasoning tokens The authors introduce INPAINTING, reframing prompt optimization as amortized inference in the joint prompt–response space They show that non-autoregressive LLMs, such as DLLMs, are natural adversaries for black-box LLMs The framework enables model-agnostic attacks that efficiently generate strong, low-perplexity prompts that transfer across robust and proprietary models at a fraction of existing attacks’ cost They show that adversarial prompts sampled from the DLLM exhibit low perplexity comparable to benign prompts from the UltraChat dataset and to original harmful behavior prompts in the JBB dataset The paper is 2511.

Paper summary: Jane: So looking at "Diffusion LLMs are Natural Adversaries for any LLM," what’s the big picture here, beyond just finding better jailbreaks?

Tom: The core contribution is reframing prompt optimization as an amortized inference task using Diffusion LLMs as surrogates, which moves us away from per-instance discrete optimization to a few parallelizable samples.

Lu: It establishes diffusion-based amortized inference as a principled and model agnostic framework for adversarial prompt generation, and the main contribution is INPAINTING, the framework that transforms costly per instance optimization into conditional inference using pretrained non-autoregressive LLMs as surrogates.

Meng: For deployment, what does this mean practically? It suggests we can generate strong, low-perplexity prompts that transfer across proprietary models at a fraction of existing attack costs.

Lalam: The paper shows that even small and simple pretrained Diffusion LLMs can act as highly efficient and effective adversaries against state-of-the-art LLMs, and they emphasize the need for responsible awareness regarding model limitations and harmful use cases in deploying these insights.

Conclusion: Tom: So we’re wrapping up on "Diffusion LLMs are Natural Adversaries for any LLM." The main idea here is that these diffusion models, these non-autoregressive ones, aren't just some fancy way to generate text; they can act like natural adversaries against almost any other large language model.

Jane: So if I’m a listener who doesn’t read papers, what does "natural adversary" mean in plain terms? Does it mean the diffusion model just gets better at tricking things?

Tom: It means the framework, which they call INPAINTING, turns prompt searching into something more like a simple inference task. Instead of trying every single prompt one by one—which is super slow and expensive—you train this diffusion model to directly generate prompts that are likely to get the bad response you want.

Lu: The authors show that by using these models as surrogates, you only need a small number of these conditional samples to find a high-reward prompt. They provide probabilistic guarantees on how many samples you actually need, which is a big deal for making this method reliable.

Meng: But what about the practical cost? If I’m running this on my production models at scale, does it still make sense compared to just trying existing attack methods?

Jane: It suggests that conditioned generation can transfer these successful prompts across different proprietary models, even ones that are robustly trained, and it does it at a fraction of the cost of current attacks.

Tom: Exactly. They’re showing that you don't need to be hyper-specialized in optimizing for a specific target model; you just need a good diffusion model to do the heavy lifting of finding those effective prompts.

Lalam: And what’s really interesting is that these generated adversarial prompts have low perplexity, meaning they sound pretty normal—almost like benign text—so simple defenses might not catch them.

Jane: It sounds like this work isn's just about creating a better jailbreak; it’s about changing how we think about finding vulnerabilities in AI systems fundamentally.

Tom: It is. And the next thing to look at is how this idea applies when we start thinking about defense, like adversarial training, which I think will be really interesting to unpack next.

More episodes

← Home