SAVeR squared: Reasoning-based Safety Alignment for Large Reasoning Models via Verifiable Rewards
cs.AI
Submitted: 2026-11-05
Updated: 2026-11-07
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large Reasoning Models (LRMs) have achieved impressive performance via explicit Chain-of-Thought (CoT) reasoning, yet this process introduces critical safety risks.
Abstract
Large Reasoning Models (LRMs) have achieved impressive performance via explicit Chain-of-Thought (CoT) reasoning, yet this process introduces critical safety risks. We formalize LRM safety objectives as requiring non-malicious reasoning and answers, safety consistency, and explicit intent identification. We argue that existing alignment methods suffer from poor adversarial generalization, a significant ``safety tax'' on reasoning, and unfaithful coarse-grained rewards, primarily due to their neglect of the LRM's intrinsic reasoning capabilities for safety. To address these, we propose SAVeR squared, a novel Safety Alignment method via Rewards with Reasoning capability. SAVeR squared decomposes the safety objective into four fine-grained verifiable reward signals and utilizes a difficulty-aware data selection strategy to stabilize training. Crucially, we introduce a reasoning-preserving gradient projection mechanism that analytically resolves conflicts between safety and reasoning gradients, effectively eliminating the safety tax. Extensive experiments on models ranging from 1.5B to 671B demonstrate that SAVeR squared achieves superior safety performance and significantly lower over-refusal rates across multiple benchmarks without compromising general reasoning capabilities.
Sources
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- RAJ-PGA: Reasoning-Activated Jailbreak and Principle-Guided Alignment Framework for Large Reasoning Models
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- MindSpeed RL: Distributed Dataflow for Scalable and Efficient RL Training on Ascend NPU Cluster
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Towards Reasoning in Large Language Models: A Survey
- Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable
- GPT-4o System Card
- R1-ACT: Efficient Reasoning Model Safety Alignment by Activating Safety Knowledge
- Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
- OpenAI o1 System Card
- SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities
- RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
- HumanEval on Latest GPT Models -- 2024
- General-Reasoner: Advancing LLM Reasoning Across All Domains
- SaRO: Enhancing LLM Safety through Reasoning-based Alignment
- Optimal Transport for LLM Reward Modeling from Noisy Preference
- Proximal Policy Optimization Algorithms
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents