Treating the Model as an Untrusted Component: Deterministic Policy Enforcement, Human Approval and Taint Tracking Against Prompt Injection in Tool-Using Agents, with Validation on Open-Weights Models
cs.CR, cs.AI
Submitted: 2026-11-08
Updated: 2026-11-08
Code: https://github.com/nawaralseelawi/untrusted-model-agentsec
Terminology
Sources
- Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
- Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?
- The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections
- Ignore Previous Prompt: Attack Techniques For Language Models
- AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
- Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents
- WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks
- AgentDyn: Are Your Agent Security Defenses Deployable in Real-World Dynamic Environments?
- The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions
- Defending Against Indirect Prompt Injection Attacks With Spotlighting
- StruQ: Defending Against Prompt Injection with Structured Queries
- SecAlign: Defending Against Prompt Injection with Preference Optimization
- A Critical Evaluation of Defenses against Prompt Injection Attacks
- On Adaptive Attacks to Adversarial Example Defenses
- Defeating Prompt Injections by Design
- Securing AI Agents with Information-Flow Control
- Progent: Securing AI Agents with Privilege Control
- AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents
- Contextual Agent Security: A Policy for Every Purpose
- IsolateGPT: An Execution Isolation Architecture for LLM-Based Agentic Systems
Related papers
- From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment
- Privacy Leakage from Recurrent State in LLM Inference
- Incrementally Verifiable Oracle Aided Computation
- Privilege and confidentiality in generative AI workflows
- BadMoE: Backdooring Mixture-of-Experts LLMs via Routing-Aware Triggers and Task-Underutilized Experts
- CoVault: Secure, Scalable Analytics of Personal Data