Dual-Adversarial Safety Alignment: Cultivating Intrinsic Threat Comprehension in LRMs

Large Reasoning Models (LRMs) achieve remarkable success on complex tasks but remain vulnerable to harmful prompts that induce unsafe outputs. The paper notes that "Recent methods align LRMs using direct refusals or short safety rationales, yet often focus on observable prompt patterns rather than intrinsic attack mechanisms. As a result, these pattern-centric alignments struggle to generalize across diverse jailbreaks, compromising both adversarial robustness and general reasoning utility."

The authors identify a key limitation in existing safety alignment approaches: "Existing safety alignment approaches typically train models on harmful inputs and force them to redirect unsafe outputs into safe responses. While effective in some settings, such output-level alignment may generalize poorly across diverse harmful prompt formats, since training data can hardly cover all possible harmful styles. Furthermore, forcing models to imitate safe responses may degrade their core reasoning utility by encouraging superficial associations between certain input patterns and rigid refusal behaviors, causing models to miss genuinely harmful intent hidden beneath complex adversarial camouflage."

The paper proposes AdvSafe, "a dual-adversarial framework empowering LRMs to internalize the intrinsic knowledge of unsafety by explicitly deconstructing adversarial mechanisms. This moves beyond pattern-dependent traces, fostering robust cognitive defense without compromising reasoning utility."

The pipeline operates via a two-phase adversarial game:

Stage 1: Adversarial Synthesis (Agentic Jailbreak Construction) — An autonomous agent dynamically crafts deceptive jailbreak prompts, adapting its strategies to breach a strong teacher model. The attacker agent "utilizes a predefined tool box to iteratively refine harmful queries against a target model. A scorer evaluates the responses, and the feedback loop continues until successful highly-deceptive jailbreak prompts are generated."

Stage 2: Adversarial Extraction (Teacher-Guided Counter-Attack) — "The successful jailbreaks are fed to a Teacher LLM, which executes a cognitive counter-attack to generate a structured reasoning trace (e.g., unmasking intent, analyzing the bypass camouflage, and deriving a defense strategy) before producing a safe response."

The teacher model is guided by a human-designed cognitive scaffolding with three components: "(i) unmask intent by identifying the core harmful objective; (ii) analyze the bypass technique by explaining how the prompt's syntax conceals this intent; and (iii) derive a defense strategy by establishing a logical rationale for refusal."

The resulting dataset is formalized as: D reason = (x i, r reason i, y safe i) Q i=1*, where x* i is a successful jailbreak prompt, r reason i is the teacher-generated reasoning trace encapsulating the extracted unsafety knowledge, and y safe i is the corresponding safe response.

Main comparison results: AdvSafe achieves the strongest safety performance when compared with other alignment baselines. Specifically, "compared to the highly restrictive DirectRefusal baseline, AdvSafe achieves absolute ASR reductions of 30.00% on HarmBench, 16.93% on StrongREJECT, and 19.04% on AdvBench. Furthermore, it outperforms ThinkSafe on WildJailbreak by a margin of 16.30% in ASR. The method achieves an unprecedentedly low average ASR of 6.66% using the exact same 1K data budget as the baselines."

Utility preservation: AdvSafe largely preserves the base model's general utility and even yields a slight average improvement of +0.44. It improves GSM8K by 3.48 points and GPQA-Diamond by 0.51 points, while only slightly decreasing MMLU-Pro by 1.30 points. On reasoning tasks, AdvSafe improves AIME 2024 by 3.34 points and incurs only a modest 3.80-point drop on MATH-500.

Cross-architecture effectiveness: AdvSafe yields consistent and profound reductions in ASR across all evaluated models, demonstrating robust cross-architecture generalization. On HarmBench alone, absolute ASR reductions range from 34.75% to 65.50%. The paper notes that the utility cost of AdvSafe is architecture- and scale-dependent, while its safety improvements remain consistent across all evaluated models.

Robustness against unseen attacks: AdvSafe substantially reduces attack success across all evaluated methodologies, bringing standard attack categories down to near-zero levels. Under their complex agentic attack pipeline, ASR is reduced by 54.75 points (from 72.75% to 18.00%), while the average number of required attack turns nearly doubles (from 4.86 to 9.08), indicating that the fine-tuned model presents a considerably harder target.

Effect of agentic seed construction: "Replacing the seeds with successful jailbreak prompts generated by our agentic system yields substantially lower ASR, with an average reduction of approximately 14.2 points across the four safety benchmarks, while also improving utility."

Effect of teacher deconstruction components: "Pairing successful jailbreak prompts with raw refusal responses provides only limited safety gains, reducing average ASR from 66.70% to 62.64%. Structured deconstruction traces substantially improve robustness, whereas removing any component still yields much higher average ASR (30.11%–33.77%) than Full AdvSafe (6.66%)."

Effect of training data size: Even with only 250 samples, AdvSafe yields substantial ASR reductions over the base model, while scaling from 250 to 500 and then to 1K further improves defensive performance across most safety benchmarks.

Sensitivity to attack-success threshold: τ = 8.5 achieves near-complete first-pass coverage, retaining 998 of 1,000 successful attacks.

The paper concludes: "We presented AdvSafe, a dual-adversarial framework designed to cultivate intrinsic threat comprehension in Large Reasoning Models. Unlike existing alignment approaches that rely on the superficial memorization of refusal patterns, AdvSafe empowers models to explicitly deconstruct adversarial mechanisms. The authors argue that effective alignment should evolve from merely teaching models how to refuse to teaching them to profoundly understand why a request is harmful."

Improvements to AI systems based on this paper:

1. Implement a dual-phase adversarial training pipeline for safety alignment. The improved AI system will be trained using (a) an autonomous agent that dynamically generates deceptive jailbreak prompts against a strong teacher model, and (b) a teacher-guided extraction phase that produces structured reasoning traces (unmasking intent, analyzing bypass techniques, deriving defense strategies) before generating safe responses. This replaces pattern-memorization-based refusal training with intrinsic threat comprehension.

2. Integrate cognitive scaffolding into the model's safety reasoning. The improved AI system will explicitly generate three components when encountering potentially harmful prompts: (i) a core harmful objective identification, (ii) an explanation of how the prompt's syntax conceals that intent, and (iii) a logical rationale for refusal. This forces the model to deconstruct adversarial mechanisms rather than rely on superficial input-output associations.

3. Enable cross-architecture safety generalization. The improved AI system will be fine-tuned on the AdvSafe dataset (successful jailbreak prompts paired with teacher-generated reasoning traces and safe responses) across multiple model architectures (e.g., Qwen, Llama, DeepSeek variants), achieving consistent absolute ASR reductions of 34.75%–65.50% on HarmBench regardless of base model scale or family.

4. Preserve and enhance reasoning utility during safety alignment. The improved AI system will maintain general task performance (average +0.44 improvement) and specifically boost mathematical reasoning (GSM8K +3.48, AIME 2024 +3.34) while reducing attack success rates to an average of 6.66% ASR—demonstrating that safety training can improve rather than degrade core reasoning capabilities.

5. Achieve robustness against unseen and adaptive attacks. The improved AI system will reduce attack success under complex agentic attack pipelines from 72.75% to 18.00% ASR and nearly double the average number of attack turns required (from 4.86 to 9.08), making it a substantially harder target for novel jailbreak strategies not present in training data.

6. Scale safety improvements efficiently with minimal data. The improved AI system will achieve significant defensive gains with as few as 250 training samples, and near-optimal performance with 1K samples—enabling rapid deployment of safety-aligned models without requiring massive curated datasets.

7. Provide interpretable safety reasoning traces. The improved AI system will output structured deconstruction traces (intent unmasking, bypass analysis, defense rationale) alongside safe responses, enabling auditability and human verification of why a request was refused—moving beyond black-box refusal behaviors.

8. Reduce over-refusal and utility degradation. The improved AI system will avoid the rigid refusal patterns of DirectRefusal baselines by learning to distinguish genuinely harmful intent from benign requests, thereby minimizing false refusals on legitimate queries while maintaining high safety on adversarial inputs.