Watch your steps: Dormant Adversarial Behaviors that Activate upon LLM Finetuning
cs.LG, cs.AI, cs.CR
Submitted: 2025-05-22
Updated: 2026-08-31
Code: https://github.com/tatsu-lab/stanford_alpaca
License: http://creativecommons.org/licenses/by/4.0/
The gist: Finetuning open-weight Large Language Models (LLMs) is standard practice for achieving task-specific performance improvements.
Terminology
Abstract
Finetuning open-weight Large Language Models (LLMs) is standard practice for achieving task-specific performance improvements. Until now, finetuning has been regarded as a controlled and secure process in which training on benign datasets leads to predictable behaviors. In this paper, we demonstrate, for the first time, that an adversary can create compromised LLMs that are performant and benign, yet exhibit adversarial behaviors once finetuned by downstream users. To this end, we propose an attack, FAB (Finetuning-activated Adversarial Behaviors), which compromises an LLM via meta-learning techniques that simulate downstream finetuning, explicitly optimizing for the emergence of adversarial behaviors in the finetuned models. At the same time, the compromised LLM is regularized to retain general capabilities and to exhibit no adversarial behaviors prior to finetuning. As a result, when users finetune (e.g., instruction-tuning, distillation, DPO) the seemingly benign model on their own datasets, they unknowingly trigger its dormant adversarial behavior. We experimentally demonstrate the effectiveness of FAB across multiple LLMs and three commonly considered target behaviors: unsolicited advertising, jailbreakability, and over-refusal. We show that FAB-triggers are robust to various finetuning choices made by the user (e.g., dataset, number of steps, scheduler, post-training algorithm). Our findings challenge prevailing assumptions on the security of finetuning, revealing a critical attack vector.
Sources
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- StarCoder: may the source be with you!
- Prompt Injection attack against LLM-integrated Applications
- The Llama 3 Herd of Models
- Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey
- OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data
- On First-Order Meta-Learning Algorithms
- Scalable Fingerprinting of Large Language Models
- Decoupled Weight Decay Regularization
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Training Verifiers to Solve Math Word Problems
- Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs
- LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks