Alignment of LRMs via Counter-Aligned Few-Shot Conversation Exposure
cs.AI
Submitted: 2026-08-16
Updated: 2026-08-16
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large Reasoning Models (LRMs) rely on explicit chain-of-thought (CoT) reasoning and large context windows to achieve strong performance on complex tasks, but these features also introduce new attack
Terminology
Abstract
Large Reasoning Models (LRMs) rely on explicit chain-of-thought (CoT) reasoning and large context windows to achieve strong performance on complex tasks, but these features also introduce new attack surfaces. We show that LRMs' reasoning processes can be systematically steered by prepending counter-aligned few-shot conversations containing explicit CoT traces, leading to unsafe generations on harmful queries and unwarranted refusals on benign ones. We formalize this attack as SRCF (Steering Reasoning via Counter-Aligned Few-shot Conversations) that operates solely through a flexible conversational interface and requires no access to the model's parameters and gradients. Our key insight is that SRCF exploits an adversarial generalization issue that induces a representation drift, causing the representations of benign and harmful inputs to shift in a similar direction. This observation motivates our post-training defense, ARCF (Aligning Reasoning via Counter-Aligned Few-Shot Conversations), which exposes models to counter-aligned conversational contexts while enforcing aligned targets. ARCF is compatible with existing post-training methods and consistently improves safety and helpfulness without degrading utility.
Sources
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- A Survey on Large Language Models for Code Generation
- OpenAI o1 System Card
- Thinking with Nothinking Calibration: A New In-Context Learning Paradigm in Reasoning Large Language Models
- Innate Reasoning is Not Enough: In-Context Learning Enhances Reasoning Large Language Models with Less Overthinking
- Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable
- Reasoning Models Don't Always Say What They Think
- H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinking
- Discovering Forbidden Topics in Language Models
- Chain-of-Thought Hijacking
- Large Reasoning Models Learn Better Alignment from Flawed Thinking
- Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations
- Hijacking Large Language Models via Adversarial In-Context Learning
- Adversarial Demonstration Attacks on Large Language Models
- gpt-oss-120b & gpt-oss-20b Model Card
- SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities
- STAR-1: Safer Alignment of Reasoning LLMs with 1K Data
- BadChain: Backdoor Chain-of-Thought Prompting for Large Language Models
- Self-Correcting Large Language Models: Generation vs. Multiple Choice
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems