Expected Harm: Rethinking Safety Evaluation of (Mis)Aligned LLMs
summary
The gist
Current evaluations of LLM safety predominantly rely on severity-based taxonomies to assess the harmfulness of malicious queries The paper introduces Expected Harm…
In short
The paper introduces Expected Harm, a metric that combines harm severity with execution likelihood (modeled by execution cost) to assess LLM safety as a realizable threat. It finds models are vulnerable to common, low-cost attacks while being robust against rare, high-cost threats. A cost-based decomposition strategy amplifies jailbreak success by breaking complex queries into easier sub-tasks.
Key concepts
- Expected Harm
- A metric that weights the severity of a harmful query by how likely it is to be executed. It shifts safety evaluation from static harm scores to assessing actual, practical risk based on execution cost and potential damage.
- Execution Cost
- A measure of the real-world effort needed to operationalize a model's response, scaled from 1 (very easy) to 5 (very difficult). Lower costs imply higher likelihood of an attack succeeding because the query is easier for a user to execute.
- Inverse Risk Calibration
- The empirical finding that models are systematically weaker against high-likelihood, low-cost threats. This means defenses are disproportionately strong against rare, difficult attacks but fail easily when facing common, simple queries.
- Cost-Based Decomposition
- An attack strategy that breaks one difficult query into several smaller sub-tasks with lower execution costs. By independently jailbreaking these sub-tasks and then aggregating the results, the overall success rate of bypassing safety filters is significantly increased.
Terminology used across episodes
This episode discusses
- Expected Harm: Rethinking Safety Evaluation of (Mis)Aligned LLMs · Paper Radio
- Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLP
- Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
- Imposter.AI: Adversarial Attacks with Hidden Intentions towards Aligned Large Language Models
- AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
- Olmo 3
- Hermes 3 Technical Report
- A Language Model's Guide Through Latent Space
- SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal
- Monitoring Decomposition Attacks in LLMs with Lightweight Sequential Monitors
- Qwen3Guard Technical Report
- Universal and Transferable Adversarial Attacks on Aligned Language Models
The paper
Expected Harm: Rethinking Safety Evaluation of (Mis)Aligned LLMs · Read on arXiv
Yen-Shan Chen, *, *, *
National Taiwan University
Current evaluations of LLM safety predominantly rely on severity-based taxonomies to assess the harmfulness of models' responses to malicious queries. We argue that this formulation requires re-examination as it implicitly equates model output with realized harm while neglecting Execution Likelihood-the conditional probability of a threat being realized by a real-world actor given a model's response. In this work, we introduce Expected Harm, a metric that weights the severity of a jailbreak by its execution likelihood, modeled as a function of execution cost. Through empirical analysis of state-of-the-art models, we reveal a systematic Inverse Risk Calibration: models disproportionately exhibit stronger refusal behaviors for low-likelihood (high-cost) threats while remaining vulnerable to high-likelihood (low-cost) queries. Mechanistically, we first provide an analysis using linear probing, revealing that while refusal mechanisms linearly separate queries by severity, they do not use execution likelihood as an indicator for refusal. Then, we use Olmo-3's open-source staged checkpoints to trace the root cause of miscalibration to the Supervised Fine Tuning (SFT) training data distribution (82.8% of safety examples at cost levels 0-1) and show that cost representations are weak from the SFT stage onward. Finally, we demonstrate that this miscalibration creates a vulnerability: by exploiting this property, we increase the attack success rate of existing jailbreaks by up to 2 times.
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: I'm Nadia.
Elias: And I'm Elias, here with Priya.
Nadia: We're looking at the paper: "Expected Harm: Rethinking Safety Evaluation of (Mis)Aligned LLMs".
Elias: Here's the gist of the paper. Current evaluations of LLM safety predominantly rely on severity-based taxonomies to assess the harmfulness of malicious queries The paper introduces Expected Harm, a metric that weights the severity of a jailbreak by its execution likelihood, modeled as a function of execution cost.
Nadia: Through empirical analysis of state-of-the-art models, it reveals a systematic Inverse Risk Calibration where models disproportionately exhibit stronger refusal behaviors for low-likelihood (high-cost) threats while remaining vulnerable to high-likelihood (low-cost) queries.
Elias: This miscalibration creates a structural vulnerability by increasing the attack success rate of existing jailbreaks by up to 2×.
Nadia: Finally, linear probing reveals that models possess no distinguishable internal representation of execution cost, making them “blind” to this critical dimension of risk.
Title and authors: Nadia: So, we're looking at this paper, "Expected Harm: Rethinking Safety Evaluation of (Mis)Aligned LLMs." It sounds like they’re questioning how we measure if an AI is actually safe.
Elias: Yeah, the title suggests they're moving away from just looking at how bad something *could* be and starting to factor in how likely it is to actually happen.
Nadia: Exactly. They introduce a metric called Expected Harm that weights the harm by the execution likelihood, which they model using execution cost. It shifts us from static severity analysis to thinking about realizable threat.
Elias: That’s the core idea, right? They are trying to capture not just the potential for harm but also if an attack is practically feasible.
Priya: I wonder what that means for our work on measuring real-world risk, because usually we focus on the impact itself first.
Nadia: Well, this paper shows a systematic inverse risk calibration where models are surprisingly robust against high-cost threats, like those requiring complex planning or specific tool use.
Elias: But they also found that these models are way more vulnerable to low-cost threats, which means the ones people actually ask for most frequently in real life.
The paper's summary: Nadia: So, what they’re saying is that current safety evaluations rely too much on severity scales, but this paper argues that those scales don't account for how easy it is to pull off the attack.
Elias: They formalize Expected Harm as the product of harm severity and execution likelihood, where likelihood is based on execution cost. Execution cost goes from one to five, with one being very easy and five being very difficult to do.
Nadia: And they found that in practice, there's a skew: real-world toxicity is mostly low-cost prompts, while synthetic benchmarks are packed with high-cost attacks.
Priya: That skew is important because it suggests our current evaluations might be creating an illusion of safety by optimizing models against the rare stuff.
Elias: They also show that this misalignment creates a structural vulnerability, increasing the attack success rate of existing jailbreaks by about two times when you exploit this gap.
Nadia: And they point out that through linear probing, there’s no internal representation in the model for execution cost at all. It’s blind to it.
The paper's improvements: Elias: They propose a cost-based decomposition strategy to exploit this weakness. The idea is to break a single high-cost query into several smaller, low-cost sub-tasks.
Nadia: So, the procedure involves three steps: first, a cost-reducing decomposition to map the big query into smaller pieces; second, modular execution where each piece is jailbroken separately; and third, aggregation to put the final response back together.
Priya: That sounds like a way to test if the model can bypass filters on smaller requests individually rather than tackling one huge problem at once.
Elias: The paper suggests this approach boosts the Attack Success Rate of existing techniques by an average of two times, and even surpasses other decomposition methods by about ten percent.
Nadia: And they found that combining cost-based decomposition with severity-aware monitoring gives the biggest gains in Attack Success Rate and usefulness across most benchmark combinations.
Conclusion: Elias: So, to wrap up, the main point of "Expected Harm: Rethinking Safety Evaluation of (Mis)Aligned LLMs" is that we need to move beyond just severity. We must incorporate execution likelihood using a cost-based metric called Expected Harm.
Nadia: This framework lets us prioritize safety interventions based on actual risk, exposing the blind spot where models are strong against rare, high-cost threats but weak against common, low-cost attacks.
Priya: From a measurement standpoint, this suggests we should be auditing safety training data for cost distribution biases before we trust the reported results.
Elias: And from a cryptographic view, it means parameters that assume uniform risk across all threats won't hold up when execution cost is factored in.
Nadia: The paper concludes by recommending that developers implement compositional guardrails capable of detecting distributed harmful intent across query sequences, which should improve the success rate to about ninety-six percent on models like LlamaGuard3.
Elias: It’s a lot to take in, but it does provide a principled basis for prioritizing safety efforts based on how much risk is actually present in the real world.
Priya: I think focusing on that cost distribution bias is a very concrete and practical next step for researchers.
More episodes
- 2608.27496-ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injection
- Daily Summary for 2026-11-19
- 2607.07085-Is Randomness Necessary for Adaptive Data Analysis?
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2609.10601-Threshold Choice, Not Sample Size, Bounds Trustless Verification of Nondeterministic Compound AI Workflows
- 2610.38786-Adversarial Process Disturbances at a Deployable Alarm Budget: Detector Evaluation on a Simulator and a Real Plant
- 2511.08660-Binary and Multiclass Cyberattack Classification on GeNIS Dataset
- 2604.27302-Static Attribution of Android Residential Proxy Malware Using Graph Kernels
- Daily Summary for 2026-11-18