Beyond a Single Direction: Chain-of-Thought Disrupts Simple Steering of Refusal
cs.AI, cs.LG
Submitted: 2026-05-26
Updated: 2026-05-26
Journal ref: ICML 2026 Workshop on Mechanistic Interpretability
Code: https://github.com/kiajueng/cot
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large reasoning models (LRMs) generate chain-of-thought (CoT) traces before producing final outputs, introducing a dynamic internal state that may complicate control mechanisms such as refusal.
Terminology
Abstract
Large reasoning models (LRMs) generate chain-of-thought (CoT) traces before producing final outputs, introducing a dynamic internal state that may complicate control mechanisms such as refusal. Unlike instruction-tuned LLMs, where refusal is mediated by a single directional subspace, refusal in large reasoning models (LRMs) additionally depends on the CoT. In DeepSeek-R1-Distill-LLaMA-8B, activation steering reverses refusal in only 39% of cases when the CoT is kept fixed, but removing the CoT entirely increases this to 70%, indicating that the CoT actively reinforces refusal. In a two-stage intervention where the model regenerates its CoT under activation steering, refusal is reversed in 94% of cases, while the resulting CoT alone retains 48% of this effect even after steering is removed. This suggests that the CoT can carry and reconstruct the compliance signal independently. These findings indicate that refusal in LRMs is jointly encoded in residual stream activations and CoT. This joint activation makes LRM more robust against activation-level interventions alone, but exposes CoT to a possible alternative surface attack.
Sources
- Refusal in Language Models Is Mediated by a Single Direction
- JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- The Llama 3 Herd of Models
- OpenAI o1 System Card
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Where Do Reasoning Models Refuse?
- Chain-of-Thought Hijacking
- LLMs Encode Harmfulness and Refusal Separately
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems