Semantic Overlays: Mitigating Prompt Injection with Annotations Beyond Tokens and Steering Vectors

arXiv:2608.23873 · cs.AI, cs.CL, cs.CR, cs.LG · Submitted 2026-08-24 · Read on arXiv

cs.AI, cs.CL, cs.CR, cs.LG

Submitted: 2026-08-24

Updated: 2026-09-03

Comments: 21 pages, 4 figures, 13 tables. Interactive demo: https://semantic-overlays.vercel.app. Code and released adapters: https://github.com/JoshuaSP/semantic-overlays

Code: https://github.com/JoshuaSP/semantic-overlays

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Frozen model — 25× in this paper · explained in 3 episodes
• Benchmark — 20× in this paper · explained in 2 episodes
• Steering — 19× in this paper · explained in 2 episodes
• Embeddings — 8× in this paper · explained in 2 episodes
• Decode — 6× in this paper · explained in 2 episodes
• Activation Steering — 4× in this paper · explained in 4 episodes
• Fine-Tuning — 4× in this paper · explained in 5 episodes
• Provenance — 4× in this paper · explained in 3 episodes
• Generalization — 3× in this paper · explained in 13 episodes
• Transformer — 3× in this paper · explained in 2 episodes
• Adam — 2× in this paper · explained in 2 episodes
• In International Conference on Learning Representations — 2× in this paper
• Activation Addition — 1× in this paper · explained in 1 episode
• Affordance — 1× in this paper · explained in 1 episode
• Alignment — 1× in this paper · explained in 1 episode
• Core — 1× in this paper · explained in 2 episodes
• Distillation — 1× in this paper · explained in 2 episodes
• Fidelity — 1× in this paper · explained in 2 episodes
• KV cache — 1× in this paper · explained in 3 episodes
• LLM judge — 1× in this paper · explained in 2 episodes
• Representation Engineering — 1× in this paper · explained in 2 episodes
• Robustness — 1× in this paper · explained in 5 episodes
• Zero-Shot Transfer — 1× in this paper · explained in 4 episodes

Sources

• StruQ: Defending Against Prompt Injection with Structured Queries
• PIArena: A Platform for Prompt Injection Evaluation
• Stronger Enforcement of Instruction Hierarchy via Augmented Intermediate Representations
• Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
• Epistemic Goggles: A Pretrained Module that Induces an Epistemic Frame via Gradient Editing
• Ignore Previous Prompt: Attack Techniques For Language Models
• Steering Llama 2 via Contrastive Activation Addition
• Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game
• Steering Language Models With Activation Engineering
• The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions
• Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy
• Steering Instruction Hierarchies at Inference Time
• Representation Engineering: A Top-Down Approach to AI Transparency
• Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?
• ASIDE: Architectural Separation of Instructions and Data in Language Models

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• Agents' Last Exam
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems