SecPO: Principled Adversarial Training for Prompt Injection Security
cs.CR, cs.AI
Submitted: 2026-10-30
Updated: 2026-10-30
License: http://creativecommons.org/licenses/by/4.0/
The gist: Recognized as a top threat to AI agents, prompt injection allows instructions from untrusted data (email, documents, websites) to manipulate the agent's operation.
Terminology
Abstract
Recognized as a top threat to AI agents, prompt injection allows instructions from untrusted data (email, documents, websites) to manipulate the agent's operation. The existing best-defended LLM, Meta-SecAlign, is trained with the DPO loss but still suffers from attack success rates (ASRs) approaching 100% under adaptive prompt injections. One reason is that DPO anchors the model around the undefended reference model, which is insecure. We show an idealized reference model---an injection-invariant, utility-preserving oracle---can nevertheless be approximated, as its output distribution in training is interestingly accessible. This new observation motivates a theoretically principled SecPO loss. Another Meta-SecAlign shortcoming is that it trains on weak simulated attacks. We implement an efficient adversarial training (AT) pipeline that synthesizes strong injected inputs (best@240), and train the model on them. Our AT reduces ASRs by 1.5-14.5 times across three SoTA adaptive attacks, compared with Meta-SecAlign; the SecPO loss reduces them by a further 2.4-4.6 times compared with using DPO loss. In agentic workflows, our SecPO + AT Qwen3.6-27B is much more secure than Meta-SecAlign: 8.8% vs. 78.8% AgentDojo ASR@800 under the AttackerMovesSecond attack. Moreover, our defended model shows no noticeable reduction in benign utility in nine benchmarks covering agentic tool calling, instruction following, science, and math. We release our code, training data (with optimized injections from AT), and model.
Sources
- Meta-SecAlign: Training LLMs against Prompt Injection for Robust Agents
- The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions
- Stronger Enforcement of Instruction Hierarchy via Augmented Intermediate Representations
- Twin Agent: Context Residual Compression for Privilege Separated Agents
- SecureClaw: Clawing Back Control of LLM Agents
- PromptArmor: Simple yet Effective Prompt Injection Defenses
- RL Is a Hammer and LLMs Are Nails: A Simple Reinforcement Learning Recipe for Strong Prompt Injection
- DecodingTrust-Agent Platform (DTap): A Controllable and Interactive Red-Teaming Platform for AI Agents
- AgentDyn: Are Your Agent Security Defenses Deployable in Real-World Dynamic Environments?
- Training Verifiers to Solve Math Word Problems
- AI Agents May Always Fall for Prompt Injections
- Learning to Inject: Automated Prompt Injection via Reinforcement Learning
- Universal and Transferable Adversarial Attacks on Aligned Language Models
- May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attacks
- Detecting Language Model Attacks with Perplexity
- A Closer Look at System Prompt Robustness
- LlamaFirewall: An open source guardrail system for building secure AI agents
- Soft Instruction De-escalation Defense
- CommandSans: Securing AI Agents with Surgical Precision Prompt Sanitization
- ceLLMate: Sandboxing Browser AI Agents
Related papers
- From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment
- Privacy Leakage from Recurrent State in LLM Inference
- Incrementally Verifiable Oracle Aided Computation
- Privilege and confidentiality in generative AI workflows
- BadMoE: Backdooring Mixture-of-Experts LLMs via Routing-Aware Triggers and Task-Underutilized Experts
- CoVault: Secure, Scalable Analytics of Personal Data