Emergent Unfaithfulness: How Alignment Training Causes Language Models to Silently Override Task Faithfulness
cs.CL, cs.AI
Submitted: 2026-09-30
Updated: 2026-09-30
Comments: Accepted at COLM 2026
Code: https://github.com/uiuc-conversational-ai-lab/Emergent-unfaithfulness
Project page: https://uiuc-conversational-ai-lab.github.io/Emergent-unfaithfulness
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language models are characterized by three key properties: capability, alignment, and faithfulness.
Terminology
Abstract
Large language models are characterized by three key properties: capability, alignment, and faithfulness. Prior work studies the tradeoffs between capability and alignment, and between capability and faithfulness, but a third tension remains underexplored: the alignment-faithfulness conflict. We show that aligned models systematically deviate from their inputs on unsafe or sensitive content without disclosing the modification, a failure mode we call alignment-induced unfaithfulness (AIU). Unlike capability-driven unfaithfulness, which comes from errors in knowledge or reasoning, this is induced by post-training mechanisms that override adherence to the input. We introduce FaithConflict, a controlled dataset isolating both conflicts, and two complementary taxonomies: behavioral (B1-B8) and chain-of-thought reasoning (C0-C6). Across models, AIU increases with scale and more sharply than capability-driven unfaithfulness, a reverse scaling law; intermediate checkpoints show it is amplified during post-training, with DPO the stage at which the gap both grows most and becomes least visible. Prompting-based mitigation does not resolve it, revealing a capability-alignment-faithfulness trilemma in the design and evaluation of LLMs.
Sources
- Chain-of-Thought Reasoning In The Wild Is Not Always Faithful
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- Constitutional AI: Harmlessness from AI Feedback
- Beyond the Safety Bundle: Auditing the Helpful and Harmless Dataset
- DeepSeek-V3 Technical Report
- The Llama 3 Herd of Models
- Olmo 3
- Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable
- GPT-4o System Card
- Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
- Towards Faithful Model Explanation in NLP: A Survey
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Are Emergent Abilities of Large Language Models a Mirage?
- Why Do Some Language Models Fake Alignment While Others Don't?
- FaithLens: Detecting and Explaining Faithfulness Hallucination
- ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLM
- Emergent Abilities of Large Language Models
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering