Subliminal Learning Is Steering Vector Distillation
cs.AI
Submitted: 2026-05-31
Updated: 2026-11-07
Code: https://github.com/agu18dec/steering-vector-distillation
License: http://creativecommons.org/licenses/by/4.0/
The gist: Subliminal learning refers to a student language model acquiring a teacher's traits (e.g.
Terminology
Abstract
Subliminal learning refers to a student language model acquiring a teacher's traits (e.g. a system-prompted preference for owls) when fine-tuned on the teacher's outputs, despite the outputs being semantically unrelated to those traits. It remains poorly understood how data without semantic meaning can transfer specific semantic traits. In this work, we show that subliminal learning is mediated by a single steering vector, i.e. a vector added to the model's activations. Across two open-source models, we find that the teacher's system prompt is well approximated by a steering vector, and that the student's behavior is driven by learning an aligned vector over fine-tuning. System prompts that are not well approximated by steering vectors are not subliminally learned. This is a special case of steering vector distillation, in which a student trained on the outputs of a steered teacher learns to imitate that steering. We demonstrate steering vector distillation on a range of semantic and random vectors. Adding a semantic vector to a model's activations can have both model-independent and model-specific (i.e. non-semantic) effects on its behavior, so generated data that is non-semantic can transmit a vector with semantic effects, enabling subliminal learning. This also explains why subliminal learning does not transfer between models. We find that adaptive optimizers are necessary for subliminal learning in language models: activation gradients on steered data carry a small but consistent component along the steering direction, and non-adaptive optimizers impede this by allowing outlier gradients to dominate.
Sources
- Subliminal Effects in Your Data: A General Mechanism via Log-Linearity
- Weird Generalization and Inductive Backdoors: New Ways to Corrupt LLMs
- Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs
- Persona Vectors: Monitoring and Controlling Character Traits in Language Models
- Toy Models of Superposition
- You Didn't Have to Say It like That: Subliminal Learning from Faithful Paraphrases
- The Llama 3 Herd of Models
- Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
- Adam: A Method for Stochastic Optimization
- Subliminal Steering: Stronger Encoding of Hidden Signals
- Subliminal Learning is a LoRA Artifact
- Olmo 3
- Steering Llama 2 via Contrastive Activation Addition
- Qwen2.5 Technical Report
- Convergent Linear Representations of Emergent Misalignment
- Gemma 3 Technical Report
- Steering Language Models With Activation Engineering
- From Data to Behavior: Predicting Unintended Model Behaviors Before Training
- Thought Virus: Viral Misalignment via Subliminal Prompting in Multi-Agent Systems
- Representation Engineering: A Top-Down Approach to AI Transparency
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents