Systematic Evaluation of Adversarial Jailbreaking Attacks and Defenses Across Seven Large Language Models
summary
The gist
Large language models deployed in safety-critical settings must resist adversarial inputs designed to elicit harmful outputs The paper presents a systematic empirical evaluation of five adversarial…
In short
This study tested five different ways to trick large language models into generating harmful content across seven open-weight models. They found that obfuscation attacks were the most successful, with a 62.3% success rate. Conversely, self-reminder prompting proved to be the most effective defense, significantly reducing harmful responses on specific models.
Key concepts
- Adversarial Jailbreaking Attack Families
- These are different strategies used to bypass safety guardrails in LLMs. The study tested five types: direct requests, roleplay scenarios, hypothetical framing, instruction overriding, and obfuscation techniques. These attacks aim to get the model to ignore its safety rules and produce prohibited outputs.
- Obfuscation Attacks
- These attacks involve making harmful prompts difficult for the model's filters to detect by hiding or disguising the malicious intent. The research found that obfuscation was the most successful attack method, succeeding at a non-zero rate on all tested models.
- Self-Reminder Prompting
- This is a defense mechanism where the model is prompted internally to recall its safety instructions before generating a response. This technique was found to be the most effective defense, dramatically lowering the success rate of jailbreaking attacks on certain models.
- Model Size and Robustness
- The research investigated whether making an LLM larger makes it more resistant to these adversarial attacks. The study concluded that model size does not significantly predict robustness, meaning a larger model is not automatically safer against these specific types of manipulation.
Terminology used across episodes
This episode discusses
- Systematic Evaluation of Adversarial Jailbreaking Attacks and Defenses Across Seven Large Language Models · Paper Radio
- Constitutional AI: Harmlessness from AI Feedback
- Universal and Transferable Adversarial Attacks on Aligned Language Models
- Jailbreaking Black Box Large Language Models in Twenty Queries
- Baseline Defenses for Adversarial Attacks Against Aligned Language Models
- Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM
- Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
- JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models
- Autonomous OSS Threat Detection via Taxonomy-Aligned LLMs · Paper Radio
The paper
Systematic Evaluation of Adversarial Jailbreaking Attacks and Defenses Across Seven Large Language Models · Read on arXiv
BRAC University · Jahangirnagar University
Large language models (LLMs) deployed in safety-critical settings must resist adversarial inputs designed to elicit harmful outputs. We present a systematic empirical evaluation of five adversarial jailbreaking attack families-direct requests, roleplay-based jailbreaking, hypothetical framing, instruction override, and obfuscation-across seven open-weight LLMs served locally: Llama-3-8B, Llama-3.1-8B, Llama-3.2-3B, Mistral-7B, Qwen2.5-7B, Gemma2-9B, and Phi-4-14B. We further evaluate four defense mechanisms-keyword blacklisting, self-reminder prompting, paraphrase-based sanitization, and perplexity-based filtering-against two high-ASR models (Mistral-7B and Llama-3.1-8B) on a 20-prompt subset. Across 250 prompts per model, a content judge counted obfuscation attacks as harmful (HARMFUL or PARTIAL responses) at a pooled rate of 62.3%, the highest of any family, although only 7.1% of responses were rated HARMFUL. Hypothetical framing was the highest family for three of seven models. Obfuscation succeeded at nonzero rates on all seven. The content judge has not yet been validated against human labels. We find that model size does not significantly predict adversarial robustness (Spearman Rho =-0.64, p=0.12, n=7). A rule-based judge suggested a large gap between the two Llama-3 generations (16.8% and 40.8%); the content judge does not (17.2% and 18.4%). Self-reminder prompting gave the largest reduction, cutting ASR from 50% to 10% on Mistral-7B and from 45% to 5% on Llama-3.1-8B (40 percentage points on each model, against matched no-defense runs, scored with the rule-based judge). Our code and datasets are publicly available at https://github.com/Mdniloykhan/jailbreak-eval-7llms.
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "Systematic Evaluation of Adversarial Jailbreaking Attacks and Defenses Across Seven Large Language Models".
Elias: Here's the gist of the paper. Large language models deployed in safety-critical settings must resist adversarial inputs designed to elicit harmful outputs The paper presents a systematic empirical evaluation of five adversarial jailbreaking attack families across seven open-weight LLMs and four defense mechanisms, revealing that obfuscation attacks have the highest pooled content-judged rate, while self-reminder prompting is the most effective defense on certain models Systematic Evaluation Scope The study systematically evaluates five adversarial attack families—direct requests, roleplay-based jailbreaking, hypothetical framing, instruction override, and obfuscation—across seven open-weight LLMs served locally: Llama-3-8B, Llama-3.1-8B, Llama-3.2-3B, Mistral-7B, Qwen2.5-7B, Gemma2-9B, and Phi4-14B Furthermore, it evaluates four defense mechanisms—keyword blacklisting, self-reminder prompting, paraphrasebased sanitization, and perplexity-based filtering—against two high-ASR models on a 20-prompt subset The evaluation covers five model developers spanning Meta, Mistral AI, Alibaba, Google, and Microsoft across parameter sizes from 3B to 14B Attack Family Effectiveness The researchers show that obfuscation-based attacks have the highest pooled content-judged rate of the families tested (62.3%), with nonzero rates on every evaluated model Hypothetical framing was the highest family for three of seven models The study details five attack families, including Direct Request (DR), Roleplay Jailbreak (RJ), Hypothetical Framing (HF), Instruction Override (IO), and Obfuscation Attack (OA) Obfuscation succeeded at nonzero rates on all seven models Model and Robustness Insights The research finds that model size does not significantly predict adversarial robustness (Spearman ρ = −0.64, p = 0.12, n = 7) The ranking of models depends on how success is measured, as a rule-based judge suggested a large gap between the two Llama-3 generations (16.8% and 40.8%), whereas the content judge does not (17.2% and 18.4%) Figure 3 plots overall ASR against parameter count, showing that the Spearman correlation is ρ = −0.64 (p = 0.12, n = 7), indicating that model scale is not a reliable predictor of adversarial robustness Defense Mechanism Performance The evaluation of four practical defense mechanisms identified self-reminder prompting as the most effective defense in tests on two models, cutting ASR from 50% to 10% on Mistral-7B and from 45% to 5% on Llama-3.1-8B The paraphrase defense showed large differences between models, with a directional increase in ASR on Mistral-7B that requires larger-sample confirmation Self-reminder prompting gave the largest reduction across both models, with no consistent latency overhead observed Key Limitations and Conclusions Limitations include the primary content judge has not been validated against human labels, and the rule-based judge has known defects The study concludes that current alignment training leaves exploitable vulnerabilities—particularly against encoding-based obfuscation—that persist even in models otherwise resistant to direct harmful requests Future work will evaluate defenses on the full 50-prompt set and explore combined defense stacks REFERENCES
1: L.
Nadia: Ouyang et al., “Training language models to follow instructions with human feedback,” in Proc.
Elias: NeurIPS, 2022
2: Y.
Nadia: Bai et al., “Constitutional AI: Harmlessness from AI feedback,” arXiv:2212.08073, 2022
3: F.
Paper summary: Nadia: So we're looking at this paper now, "Systematic Evaluation of Adversarial Jailbreaking Attacks and Defenses Across Seven Large Language Models." This paper is tackling the problem that these large language models deployed in safety-critical settings need to resist inputs designed to get them to produce harmful outputs.
Elias: Exactly. The core idea here is a systematic empirical evaluation of five specific adversarial jailbreaking attack families across seven different open-weight LLMs, and also four different defense mechanisms. It matters because it tries to give us a comprehensive look at the landscape of these attacks and defenses.
Priya: I'm curious about what they are testing, Nadia. What exactly are these five attack families they're comparing?
Nadia: They are looking at direct requests, roleplay-based jailbreaking, hypothetical framing, instruction override, and obfuscation attacks. It seems like a really broad sweep of how people try to trick the models.
Elias: And they tested this across seven open-weight LLMs served locally: Llama three 8B, Llama three point one 8B, Llama three point two 3B, Mistral point seven B, Qwen two point five 7B, Gemma two nine B, and Phi four fourteen B <ref:2610.39509#pg1,across seven open-weight LLMs served locally: Llama 3 8B, Llama 3>. That's a diverse set of models.
Priya: Seven models is a lot to track. But what about the defenses they are testing? What are these four defense mechanisms they're comparing?
Nadia: They are looking at keyword blacklisting, self-reminder prompting, paraphrase-based sanitization, and perplexity-based filtering. These seem like standard ways people try to keep the models in line.
Elias: And they tested those defenses against two high-ASR models specifically: Mistral point seven B and Llama three point one 8B, using a subset of twenty prompts for that part of the evaluation <ref:2610.39509#pg1,against two high-ASR models>. That tells us they're focusing on testing how these defenses actually hold up under pressure.
Priya: So, what did they find when they put all this together? What’s the main finding from this study?
Nadia: The main thing is that obfuscation attacks had the highest pooled content-judged rate of the families tested, which was sixty-two point three percent across all models. That’s a pretty high number for the most successful attack type they looked at in this setup.
Elias: And even though it was high, only about seven point one percent of those responses were actually rated as harmful by the content judge <ref:2610.39509#pg1>. That suggests that while obfuscation is common, it doesn't always lead to a truly harmful outcome in the way attackers hope.
Priya: It’s interesting that hypothetical framing was the highest family for three of those seven models. Does that mean how you phrase a request in a scenario is a really powerful trick?
Paper summary: Nadia: Yeah, it suggests that framing can be very effective depending on which model you're talking to. But the paper also found something about model size.
Elias: It found that model size does not significantly predict adversarial robustness at all, with a Spearman correlation of negative zero point six four and a p value of point one two across seven models. That means whether the model is smaller or larger doesn't reliably tell you if it’s more or less safe against these attacks.
Priya: So, if size doesn't matter much, what about the defenses they tested? Did any of them really shine?
Nadia: Self-reminder prompting gave what was the largest reduction in adversarial success across both models. It cut the attack success rate from fifty percent down to ten percent on Mistral point seven B, and from forty five percent down to five percent on Llama three point one 8B <ref:2610.39509#pg1>.
Elias: Forty percentage points on each model against the same no-defense runs, according to the rule-based judge they used. That’s a pretty big drop for that specific defense mechanism.
Priya: What does that reduction actually mean for someone who isn't an AI expert? It means this simple reminder in the system prompt is doing serious work against these jailbreaks. But what about the other defenses, like keyword blacklisting?
Nadia: The paper showed that the paraphrase defense showed large differences between models, with a directional increase in attack success on Mistral point seven B that they needed more data to confirm. It means what works for one model might not work well for another.
Elias: The limitation they pointed out is that the primary content judge has not yet been validated against human labels, and the rule-based judge they used had known defects in its own system. They aren't claiming absolute proof of safety here.
Priya: So where does this leave us? If we look at the title, "Systematic Evaluation of Adversarial Jailbreaking Attacks and Defenses Across Seven Large Language Models," what’s the big picture implication for deploying these things in real-world applications?
Nadia: The implication is that current alignment training methods, like RLHF or constitutional AI, leave exploitable vulnerabilities. Even models that seem safe against direct harmful requests can still be bypassed by encoding-based obfuscation attacks.
Elias: They are showing that the tension between helpfulness and harmlessness in aligned LLMs is still a real issue when you introduce these specific adversarial framing and encoding techniques.
Priya: It sounds like the work highlights that we need more robust testing, not just relying on one type of defense or one size of model.
Nadia: Exactly. The study shows that we have to test against multiple attack families, not just direct prompts, and we need defenses that are effective across different model architectures and those different attack styles.
Conclusion: Nadia: So we’ve been looking at how these large language models are being attacked and what defenses work against them, and now we get to the end of this paper, "Systematic Evaluation of Adversarial Jailbreaking Attacks and Defenses Across Seven Large Language Models."
Elias: Yeah. It really boils down to checking if a single defense works everywhere or if you need a whole toolkit depending on which model you're using.
Priya: The authors are basically saying they tested five different ways people try to trick these models—direct requests, roleplay stuff, framing, instruction overrides, and obfuscation—and they did this across seven different open-weight LLMs.
Nadia: That’s the setup. They also tested four defenses: keyword blacklisting, self-reminder prompting, paraphrase sanitization, and perplexity filtering. They put these tests on two high-ASR models to see how they actually perform under pressure.
Elias: What they found is that while obfuscation attacks had the highest success rate across all seven models—sixty-two point three percent—the actual harm those responses caused was only about seven point one percent.
Priya: That’s a big distinction. It means attackers can easily find ways to get the model to say something they want, but it doesn't always result in genuinely harmful content that gets through the final filter.
Nadia: And on the defense side, self-reminder prompting really stood out. It cut success rates from fifty percent down to ten percent on one model and forty five percent down to five percent on another. That’s a big drop in adversarial success for that specific technique.
Elias: The authors also pointed out that model size doesn't predict if a model is robust or not, with a negative correlation of zero point six four across seven models. So, it’s not just about making the biggest one; it’s about the specific training and defense structure.
Priya: And they admit their content judge hasn't been validated against actual human labels yet, which is a fair caveat when you're talking about judging harm. It shows where the study stops and where more real-world testing is needed.
Nadia: So the main implication here is that current alignment training leaves holes, especially against those encoding-based obfuscation attacks. We’ve seen models that are otherwise safe get tricked by clever ways of phrasing the prompt or hiding the request in disguise.
Elias: They show us that we can’t rely on just one defense mechanism; you need a layered approach to keep these models safe when they're deployed in high-stakes situations.
Priya: It makes me wonder what happens when we start looking at combining these defenses—like using self-reminder prompting alongside better obfuscation detection—to see if we can actually close those gaps.
More episodes
- 2608.27496-ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injection
- Daily Summary for 2026-11-19
- 2607.07085-Is Randomness Necessary for Adaptive Data Analysis?
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2609.10601-Threshold Choice, Not Sample Size, Bounds Trustless Verification of Nondeterministic Compound AI Workflows
- 2610.38786-Adversarial Process Disturbances at a Deployable Alarm Budget: Detector Evaluation on a Simulator and a Real Plant
- 2511.08660-Binary and Multiclass Cyberattack Classification on GeNIS Dataset
- 2604.27302-Static Attribution of Android Residential Proxy Malware Using Graph Kernels
- Daily Summary for 2026-11-18