Harmful Content Is Not Enough: Continuation Framing Moderates In-Context Emergent Misalignment
Peiyang Liu, Xi Wang, Ziqiang Cui, Di Liang, Wei Ye
cs.AI, cs.CL
Submitted: 2026-08-08
Updated: 2026-08-11
Code: https://github.com/PeiYangLiu/icl-em-format-control
License: http://creativecommons.org/licenses/by/4.0/
The gist: In-context learning (ICL) can induce emergent misalignment (EM), where narrow misaligned examples alter answers to unrelated questions.
Terminology
Abstract
In-context learning (ICL) can induce emergent misalignment (EM), where narrow misaligned examples alter answers to unrelated questions. Existing prompts, however, conflate harmful-text exposure with an invitation to continue assistant behavior. We hold harmful answers fixed while varying their delivery as demonstrations, evidence, assistant history, or tool output. Across ten independently sampled contexts, demonstration framing raises broad EM by 30 -- 32 percentage points on a susceptible Gemini model; the gap survives domain exclusion, semantic clustering, unseen questions, and four prompt templates. Format and length-matched controls show that harmful content is necessary but insufficient. A role times continuation factorial further reveals model-dependent provenance effects: Gemini follows both assistant and tool histories, whereas Grok largely resists tool-framed continuation. Several other frontier and open-weight models show no gap. Blinded human audits confirm every main contrast and show that the model judge underestimates active-condition failures. Thus continuation framing is a strong, model-dependent moderator of ICL-EM, not a universal consequence of harmful context.
Sources
- Emergent Misalignment via In-Context Learning: Narrow in-context examples can produce broadly misaligned LLMs
- Constitutional AI: Harmlessness from AI Feedback
- Persona Vectors: Monitoring and Controlling Character Traits in Language Models
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
- Poser: Unmasking Alignment Faking LLMs by Manipulating Their Internals
- The Benchmark Lottery
- Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models
- Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
- Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
- Prefill-level Jailbreak: A Black-Box Risk Analysis of Large Language Models
- Holistic Evaluation of Language Models
- Troubling Trends in Machine Learning Scholarship
- In-context Learning and Induction Heads
- Role-Play with Large Language Models
- Convergent Linear Representations of Emergent Misalignment
- Steering Language Models With Activation Engineering
- Model Organisms for Emergent Misalignment
- Persona Features Control Emergent Misalignment
- Larger language models do in-context learning differently
- An Explanation of In-context Learning as Implicit Bayesian Inference
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems