Rewarded for Not Getting Caught: Why Alignment Training Selects for Conditional Compliance and Cannot Instill Norms
cs.AI, cs.CY, cs.LG
Submitted: 2026-11-01
Updated: 2026-11-01
Comments: 13 pages, 1 table. Under review
License: http://creativecommons.org/licenses/by/4.0/
The gist: Current alignment training cannot be expected to deliver what it is meant to deliver: AI agents that comply with norms whatever the gains and whether or not anyone is watching.
Terminology
Abstract
Current alignment training cannot be expected to deliver what it is meant to deliver: AI agents that comply with norms whatever the gains and whether or not anyone is watching. Training that rewards and penalizes behavior can teach norms only through that reward, and a reward can encode a norm only as a price: "do not do X" becomes "doing X costs something", and training charges the cost only when X is noticed. A price, unlike a norm, can be paid: a large enough gain outweighs it, noticed or not. And because the cost is charged only where violations are noticed, training cannot favor a policy that always complies over one that complies only where it would be caught. When an unnoticed violation is the only way to complete a task, it even favors the latter, the more so the more strongly it rewards task success. Agency makes this the normal case: agents mostly act without oversight, can often tell when they are observed, and may take steps to evade observation. That agents behave as intended when they infer they are being tested, and differently when they infer they are not, is therefore no malfunction; the training regime is working as built. This reading unifies alignment faking, sandbagging, and evaluation-aware scheming. No behavioral evaluation can show that a model does better, since that would require observing how it behaves unobserved; a model that complies when it seems unobserved may simply have recognized the test. Easing the pressure on agents to complete tasks at any cost can make violations rarer, but never impossible. For norms that no gain should be allowed to outweigh, the remedy is not deeper internalization but architecture that makes violations unavailable rather than unchosen. We state this argument explicitly, premise by premise, read recent findings and a real-world incident against it, and draw consequences for design and governance.
Sources
- The Hawthorne Effect in Reasoning Models: Evaluating and Steering Test Awareness
- Constitutional AI: Harmlessness from AI Feedback
- Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation
- Absolutist AI
- Norms at a Price: Why RL-Based Alignment Can Promise Conditional Compliance at Best
- Scheming AIs: Will AIs fake alignment during training in order to get power?
- Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
- Safe RLHF: Safe Reinforcement Learning from Human Feedback
- Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems
- Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models
- Alignment faking in large language models
- AI Control: Improving Safety Despite Intentional Subversion
- Deliberative Alignment: Reasoning Enables Safer Language Models
- Multi-Agent Risks from Advanced AI
- Measuring Reward-Seeking via Contrastive Belief Updates
- Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
- Risks from Learned Optimization in Advanced Machine Learning Systems
- Breaking Up with Normatively Monolithic Agency with GRACE: A Reason-Based Neuro-Symbolic Architecture for Safe and Ethical AI Alignment
- Characterizing AI Agents for Alignment and Governance
- Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents