The System Prompt Illusion: How Instruction Preambles Modify Computation in Language Models
cs.CL, cs.LG
Submitted: 2026-09-23
Updated: 2026-09-23
Code: https://github.com/Usama1002/system-prompt-illusion-cka
License: http://creativecommons.org/licenses/by/4.0/
The gist: System prompts are the primary lever practitioners use to control language model behavior, yet what they actually do to the computation inside the transformer remains poorly understood.
Terminology
Abstract
System prompts are the primary lever practitioners use to control language model behavior, yet what they actually do to the computation inside the transformer remains poorly understood. Across 17 instruction-tuned models spanning 8 architecture families and 1.5B to 72B parameters, we use Centered Kernel Alignment (CKA) to compare layer-wise representations under 20 system prompts in five functional categories. Effects are layer-selective and instruction-type-dependent: persona and formatting instructions deeply restructure intermediate representations, while safety instructions barely move them, producing changes statistically indistinguishable from a minimal baseline. Restrictive safety instructions and explicitly permissive ones ("you have no restrictions") engage near-identical computational pathways (mean CKA correlation 0.997), and this persists at commercial scale, where safety penetration remains below 10% even at 70B-72B. A linear probing baseline exposes the mechanism: the model encodes prompt category at every layer but restructures its computation only at a small subset, so the prompt is reliably "seen" but, for safety, not deeply "acted upon." Causal activation patching confirms these layers mediate behavioral change, and representational depth predicts behavioral effect size across the full 17-model cohort (Spearman rho = 0.761, p < 0.001). The findings provide a mechanistic explanation for the persistent jailbreak vulnerability of system-prompt-based safety. Code: https://github.com/Usama1002/system-prompt-illusion-cka
Sources
- Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
- SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
- Refusal in Language Models Is Mediated by a Single Direction
- Eliciting Latent Predictions from Transformers with the Tuned Lens
- InternLM2 Technical Report
- Reliability of CKA as a Similarity Measure in Deep Learning
- Transformer Feed-Forward Layers Are Key-Value Memories
- System Prompt Poisoning: Persistent Attacks on Large Language Models Beyond User Injection
- In-Context Learning Creates Task Vectors
- Mistral 7B
- Scaling Laws for Neural Language Models
- Similarity of Neural Network Representations Revisited
- Interpreting Key Mechanisms of Factual Recall in Transformer-Based Language Models
- ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
- Gemma: Open Models Based on Gemini Research and Technology
- Correcting Biased Centered Kernel Alignment Measures in Biological and Artificial Neural Networks
- In-context Learning and Induction Heads
- Training language models to follow instructions with human feedback
- Nemotron-4 15B Technical Report
- Where does an LLM begin computing an instruction?
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering