Knowingly Wrong: Training LLMs Against Their Own Beliefs can Induce Misalignment
cs.CL, cs.AI, cs.LG
Submitted: 2026-11-03
Updated: 2026-11-03
Terminology
Sources
- Conditional misalignment: common interventions can hide emergent misalignment behind contextual triggers
- Reinforcement Learning Can Amplify Emergent Misalignment from Harmless Rewards
- Natural Emergent Misalignment from Reward Hacking in Production RL
- Believe It or Not: How Deeply do LLMs Believe Implanted Facts?
- School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs
- The Piggyback Hypothesis of Generalization: Explaining and Mitigating Emergent Misalignment
- Instruction-Following Evaluation for Large Language Models
- Representation Engineering: A Top-Down Approach to AI Transparency
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data