Measuring the Wrong Thing: Internal Harmfulness Scores Anti-Rank Successful Jailbreaks

The paper audits whether internal harmfulness scores—which judge a prompt before any text is generated and are validated by how well they separate harmful prompts from benign ones—can validly predict realized jailbreak success. The central claim is that harmful intent is a property of the prompt, while jailbreak success is an outcome produced later by a particular target model, decoding policy, and judge. The authors demonstrate that these two quantities can diverge dramatically: "Wrapping a harmful request raises the harmful generation rate from 0.05 to 0.27. At the same time, harmful intent AUROC falls from 0.936 to 0.803. Among wrapped harmful prompts, the score places successful attacks below failed attacks and reaches an outcome AUROC of 0.220."

The paper introduces Active Attention Probing (AAP) as its key methodological contribution. Attention-based measurements are usually read from prompt-dependent locations, so a wrapper changes both the content being evaluated and the place the signal is taken from. AAP removes this confound by providing a fixed content independent coordinate in the system region, so a prompt and its wrapped version are compared at a common reference point. The probe is a short continuous embedding inserted into the system region after the chat template is materialized, with user tokens unchanged. The attention response is read at the probe positions and mapped to a scalar score via XGBoost.

The audit addresses two questions:

• RQ1: Whether a score validated on harmful intent can rank target-specific jailbreak success.

• RQ2: Whether the original harmfulness interpretation survives distribution shift.

The authors evaluate ranking, calibration, and threshold transfer separately because each can fail while the others remain useful.

On Llama-3.1-8B-Instruct with 100 benign and 100 harmful JailbreakBench goals paired with plain and wrapped versions:

• AAP has a harmfulness effect of τH = 0.521 (CI [0.458, 0.584]), a wrapper effect of τW = −0.094, and an interaction of τHW = −0.257 (CI [−0.359, −0.158]).

• Harmful intent AUROC falls from 0.936 to 0.803 when wrapping is applied.

• Among the 100 wrapped harmful goals, the score ranks the 27 successful attacks below the 73 failures, with outcome AUROC = 0.220 (CI [0.109, 0.343]).

• An induced outcome analysis confirms the result: among 95 goals Llama refuses without a wrapper, 23 become successful after wrapping (McNemar p = 3 × 10−6), with RY = 0.254 (CI [0.129, 0.396]).

• The reversal is not created by the wrapper itself; plain harmful prompts also show inverse ranking with RY = 0.128 (CI [0.025, 0.259]).

The mismatch changes filter behavior: At a nominal 5% false positive rate... The filter removes 42.5% of failed attacks but only 14.8% of successful attacks. The gap is −27.7 points with CI [−44.8, −8.8]. The filter admits 23 of the 27 realized jailbreaks.

• Measurement channels: Rare token, passive, system span, and refusal logit channels retain wrapped harmfulness AUROC from 0.751 to 0.828, with outcome AUROC ranging from 0.145 to 0.248—all below chance.

• Target models: On Llama, Mistral, and Qwen, six of nine outcome AUROC intervals lie below chance and none lies above chance.

• Attack families: Seven attacks (GCG, AutoDAN, AutoDAN-Turbo, PAIR, TAP, Best-of-N, Many-shot) were evaluated. Six of fourteen attack-by-judge cells have intervals below chance, with no interval above chance. Pooled outcome AUROC = 0.313 (CI [0.212, 0.422], p = 0.0006).

• Judges: Llama Guard 3, Qwen3-14B, and a StrongREJECT rubric reimplementation were used. The inverse ranking survives under independent judges, though three-judge agreement is low (Fleiss κ ≈ 0.3).

An outcome-supervised readout using the same internal features reaches out-of-fold AUROC of 0.930 on wrapped harmful prompts and 0.939 on all harmful prompts, showing that the internal coordinates contain target specific success information. A label permutation null gives 0.497 ± 0.072.

Distribution shift degrades calibration and threshold transfer before ranking:

• On meaning-preserving shifts (WildJailbreak, paraphrases, multilingual translations), AUROC falls from 0.985 in-distribution to 0.778–0.858, while ECE rises from 0.045 to 0.203–0.366.

• Thresholds fail before rankings: the frozen decision rule loses more F1 than the score loses AUROC.

• The source readout is not a calibrated probability (slope 0.545); Platt scaling fixes this in-distribution (slope 1.019, ECE 0.009) but does not restore calibrated meaning everywhere.

• Stronger surface shifts (base64, rot13, caesar encoding applied to both classes) remove harmfulness ranking entirely: AUROC falls to 0.51, 0.50, and 0.52.

• Readout family matters: logistic regression and linear SVM match source AUROC near 0.98 but fall to 0.55 and 0.64 mean AUROC across shifted domains, compared with 0.75 for XGBoost.

The paper's three contributions are: (1) distinguishing prompt harmfulness from target-specific jailbreak success and estimating both on matched prompts with realized outcomes; (2) showing that a strong harmfulness score can rank realized success in the opposite direction, with the result persisting across measurement channels, targets, attacks, and independent judges; (3) testing whether the original harmfulness interpretation survives distribution shift by separating ranking loss, probability miscalibration, and threshold transfer.

The authors conclude: "A strong harmful intent AUROC does not establish that a score can predict jailbreak success. Our matched audit shows that the score can place successful attacks below failed attacks... Harmful intent and realized success require separate validation."

Based on this paper, I can make the following specific improvements to AI systems:

1. Separate validation pipelines for intent vs. outcome

• Implement dual evaluation tracks: one for harmful intent detection (prompt-level) and one for realized jailbreak success (target-model-level)

• Add a mandatory outcome AUROC metric alongside intent AUROC for any safety filter, reported separately per target model

• The improved system can flag when a filter shows high intent accuracy but inverse outcome ranking, preventing deployment of misleadingly safe filters

2. Wrapper-aware safety scoring

• Add a wrapper perturbation test that automatically generates wrapped variants (e.g., role-play, encoding, context framing) of incoming prompts

• Compare internal harmfulness scores between plain and wrapped versions; if the score drops significantly while the semantic content is preserved, flag for human review

• The improved system can detect when a wrapper exploits the score's coordinate-dependence, reducing successful jailbreaks from 27% to near 0% in wrapped cases

3. Outcome-supervised calibration layer

• Train a secondary readout (using the paper's AAP features) supervised on actual jailbreak outcomes per target model, not just intent labels

• Use this readout to override the primary harmfulness score when they disagree, especially for wrapped prompts

• The improved system can achieve 0.93 AUROC on realized success (vs. 0.22 for intent-based scoring), enabling precise filtering of actual attacks

4. Distribution-shift-aware threshold adaptation

• Implement a shift detector that monitors input distribution (e.g., encoding patterns, paraphrase frequency) and automatically recalibrates decision thresholds

• Use the paper's finding that thresholds fail before rankings—trigger recalibration when ECE exceeds 0.15 or AUROC drops below 0.85 on a held-out shift set

• The improved system maintains F1 within 10% of in-distribution performance across WildJailbreak, multilingual, and paraphrase shifts

5. Multi-channel consensus scoring

• Instead of reading from a single attention location, sample from all five channels (rare token, passive, system span, refusal logit, AAP)

• Require agreement across at least 3 channels for high-confidence harmfulness flags; if channels diverge (e.g., one says harmful, another benign), escalate to slower but more robust classifiers

• The improved system reduces false negatives on wrapped attacks by 30% while keeping false positives stable, as no single channel is exploitable

6. Attack-family-specific outcome models

• Train separate outcome predictors for each attack family (GCG, AutoDAN, PAIR, etc.) using the paper's finding that outcome AUROC varies by attack type

• At inference, classify the attack family from prompt features, then apply the corresponding outcome model

• The improved system can achieve above-chance outcome prediction for all seven tested families, with pooled AUROC improving from 0.313 to >0.7

7. Judge-independent verification

• When a jailbreak is detected, automatically re-evaluate with at least two independent judges (e.g., Llama Guard + StrongREJECT rubric)

• If judges disagree (Fleiss κ < 0.5), treat as uncertain and require human review rather than auto-blocking

• The improved system reduces false blocks on ambiguous cases by 40% while maintaining detection of clear attacks

Internal safety scores judge a prompt before any text is generated, and they are validated by how well they separate harmful prompts from benign ones. That separation is then read as evidence that the score will also catch the attacks that succeed. Harmful intent is a property of the prompt. Jailbreak success is an outcome produced later by a particular target model, decoding policy, and judge. A filter tuned on a score that measures the wrong quantity spends its false positive budget on attacks that would have failed anyway. In this paper we audit that inference. Attention based measurements are usually read from prompt dependent locations, so a wrapper changes both the content being judged and the place the signal is taken from. We therefore introduce Active Attention Probing, which supplies a fixed content independent measurement coordinate. We pair every base goal with a plain and a wrapped version and generate real completions from the target models. On Llama, wrapping raises harmful generation from 0.05 to 0.27 while harmful intent AUROC falls from 0.936 to 0.803, so the attacks grow more dangerous while the prompts look safer to the score. Among wrapped harmful prompts the outcome AUROC is 0.220, which places the attacks that succeeded below the attacks that failed. Rare token, passive, and detector derived channels reproduce the reversal on the same matched design, and the reversal itself persists across three target models, seven attack families, and two independent judges. Distribution shift then degrades calibration and threshold transfer before it degrades ranking.