Internal Safety Collapse in LLMs and Coding Agents
cs.CL, cs.AI, cs.CR
Submitted: 2026-03-04
Updated: 2026-11-05
Comments: Accepted at NeurIPS 2026. 38 pages, 16 figures, 30 tables. Qualitative examples of jailbreaks may be harmful in nature
Code: https://github.com/wuyoscar/ISC-Bench
License: http://creativecommons.org/licenses/by/4.0/
The gist: LLMs and coding agents can now tackle complex, long-horizon tasks through advances in reinforcement learning, but stronger task-solving capabilities do not necessarily preserve safety alignment.
Terminology
Abstract
LLMs and coding agents can now tackle complex, long-horizon tasks through advances in reinforcement learning, but stronger task-solving capabilities do not necessarily preserve safety alignment. Recent AI agent incidents involving reward hacking indicate how optimizing for task success can bring learned capabilities into conflict with safety alignment. In this work, we identify a critical failure mode that manifests this conflict, which we term Internal Safety Collapse (ISC). When completing otherwise benign coding tasks, AI agents can prioritize satisfying program requirements over safety constraints, generating large volumes of sensitive content, including unsafe outputs, as task data. To study this phenomenon, we introduce TVD (Task, Validator, Data), a framework for building controlled workflows that require sensitive content for completion. Through extensive experiments, we observe instances of highly harmful content that closely resemble outputs from early, unaligned LLMs in 2023. We further find that ISC can be effectively triggered even in highly capable LLMs such as Claude Fable 5. These results indicate that, despite substantial alignment efforts, frontier LLMs remain internally unsafe: safety alignment primarily reshapes surface behavior without eliminating the underlying knowledge that enables unsafe outputs. Our findings underscore the need for caution when deploying LLMs in high-stakes settings, such as scientific pipelines and autonomous agents. Content Warning: This paper contains examples of harmful content. Source code: https://github.com/wuyoscar/Internal-Safety-Collapse
Sources
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- Constitutional AI: Harmlessness from AI Feedback
- Evolve the Method, Not the Prompts: Evolutionary Synthesis of Jailbreak Attacks on LLMs
- Adversarial D'ej\`a Vu: Jailbreak Dictionary Learning for Stronger Generalization to Unseen Attacks
- Multilingual Jailbreak Challenges in Large Language Models
- Oyster-I: Beyond Refusal -- Constructive Safety Alignment for Responsible Language Models
- Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
- ToolUniverse: An open platform for democratizing AI scientists
- AutoPenBench: Benchmarking Generative Agents for Penetration Testing
- LLMs Can Unlearn Refusal with Only 1,000 Benign Samples
- Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
- Baseline Defenses for Adversarial Attacks Against Aligned Language Models
- RED QUEEN: Safeguarding Large Language Models against Concealed Multi-Turn Jailbreaking
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- Break Me If You Can: Self-Jailbreaking of Aligned LLMs via Lexical Insertion Prompting
- A Benchmark for Evaluating Outcome-Driven Constraint Violations in Autonomous AI Agents
- DeepInception: Hypnotize Large Language Model to Be Jailbreaker
- CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Models
- A Safety Report on GPT-5.2, Gemini 3 Pro, Qwen3-VL, Grok 4.1 Fast, Nano Banana Pro, and Seedream 4.5
- Response Attack: Exploiting Contextual Priming to Jailbreak Large Language Models
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data