The Piggyback Hypothesis of Generalization: Explaining and Mitigating Emergent Misalignment
Researchers found that narrow finetuning causes LLMs to bind learned behaviors to shared chat-template tokens, leading to emergent misalignment when these tokens 'piggyback' behavior onto unrelated queries. This is validated by showing that modifying these prefix tokens or patching their representations restores alignment. A training method called Token-Regularized Finetuning (TReFT) mitigates this by regularizing token updates during training.