Agent Security is a Systems Problem
summary
The gist
Agent security must be approached as a systems problem: the AI model powering the agent must be treated as an untrusted component…
In short
Agent security is treated as a systems problem where AI models are untrusted components. The paper argues that simply making models more robust isn't enough; instead, security invariants must be enforced at the system level using mechanisms like instruction separation and least-privilege sandboxing to guarantee predictable agent behavior.
Key concepts
- System-Level Security
- Treating the AI model as an untrusted part of a larger system. Instead of just trying to make the model itself better, this approach focuses on building surrounding layers—like sandboxes and policy monitors—that enforce security rules regardless of what the underlying AI does.
- Instruction-Data Separation
- A method to prevent prompt injection attacks by strictly separating instructions from user data within the AI's context window. The goal is to ensure that data cannot be interpreted as executable commands, moving beyond simple heuristics to a structural guarantee.
- Verifiable Policy Generation
- An automated pipeline that converts natural language requests into formal, machine-readable policies. This allows security enforcement to move away from relying on the unpredictable AI model and onto a deterministic monitor that checks these formal rules.
- Information Flow Control (IFC)
- Mechanisms used to track how data moves through an agent's processes. IFC enforces constraints on usage and propagation, helping to disentangle sensitive information from various sources as it interacts with the AI model.
Terminology used across episodes
This episode discusses
- Agent Security is a Systems Problem · Paper Radio
- Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
- Defending Against Prompt Injection With a Few DefensiveTokens
- Meta-SecAlign: Training LLMs against Prompt Injection for Robust Agents · Paper Radio
- ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning
- Securing AI Agents with Information-Flow Control
- Defeating Prompt Injections by Design
- Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
- Manipulating LLM Web Agents with Indirect Prompt Injection Attack via HTML Accessibility Tree
- PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Rails
- ceLLMate: Sandboxing Browser AI Agents
- VeriGuard: Enhancing LLM Agent Safety via Verified Code Generation
- The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections
- Formal Policy Enforcement for Agentic Systems · Paper Radio
- May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attacks
- Trojan-Speak: Bypassing Constitutional Classifiers with No Jailbreak Tax via Adversarial Finetuning
- Progent: Securing AI Agents with Privilege Control
- MUZZLE: Adaptive Agentic Red-Teaming of Web Agents Against Indirect Prompt Injection Attacks
- The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions
- AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents
- Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy
The paper
Agent Security is a Systems Problem · Read on arXiv
Google University of California San Diego University of Wisconsin–Madison EmbraceTheRed Gray Swan AI Cornell University
We take the position that agent security must be approached as a systems problem: the AI model powering the agent must be treated as an untrusted component, and security invariants must be enforced at the system level. Through this lens, efforts to increase model robustness (the dominant viewpoint in the community) are insufficient on their own. Instead, we must complement existing efforts with techniques from the systems security domain. Based on our experience as cybersecurity researchers in operating systems, networks, formal methods, and adversarial machine learning, we articulate a set of core principles, grounded in decades of systems security research, that provide a foundation for designing agentic systems with predictable guarantees. As evidence, we analyze eleven representative real-world attacks on agents and discuss how systems principles, if realized, could have prevented these attacks. We also identify the research challenges that stand in the way of implementing these principles in agents.
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "Agent Security is a Systems Problem".
Elias: Here's the gist of the paper. Agent security must be approached as a systems problem: the AI model powering the agent must be treated as an untrusted component, and security invariants must be enforced at the system level This paper argues that increasing model robustness alone is insufficient and advocates for applying techniques from systems security to design agentic systems with predictable guarantees The authors position agent security as a systems problem where the AI model is an untrusted component and security invariants are enforced at the system level They identify three main gaps that need bridging to improve the secure deployment of agentic systems in the wild These gaps relate to how existing model-centric security approaches fail against adaptive attacks and the complexity of modern agentic systems System-Level Mitigation Mechanisms The paper outlines three core system-level mechanisms that mitigate a broad range of agent attacks (1) Provable separation of instructions and data: (2) Least-privilege sandboxing with verifiable policy generation: (3) Information flow control (IFC): The authors discuss these mechanisms in detail, noting that language-based models inherently mix trusted instructions and untrusted data, creating a fundamental vulnerability They suggest that model-based defenses for separating instructions and data will always be evaded by persistent attackers Violations of Established Security Principles The paper analyzes eleven representative real-world attacks on agents, demonstrating how they violate established security principles such as Least Privilege, TCB Tamper Resistance, Complete Mediation, and Secure Information Flow Examples include ChatGPT Long-Term Memory SpAIware violating TCB Tamper Resistance and Secure Information Flow Research Challenges for Implementation The authors identify three concrete research challenges that the community should focus on to solve security issues in agentic systems (3.1) Separating Instructions from Data: (3.2) Verifiable Policy Generation: (3.
Title and authors: Elias: Okay, so the core idea is that agent security has to be approached as a systems problem where the AI model is treated as an untrusted component and security invariants are enforced at the system level.
Nadia: That’s a heavy framing because it suggests that just making the underlying model less likely to be tricked isn't going to cut it against adaptive attackers.
Elias: Exactly, they argue that increasing model robustness alone is insufficient; we have to bring in techniques from systems security, like those used in operating systems or networks, to design agentic systems with predictable guarantees.
Priya: So, if we think of the AI as the untrusted part of this system, what does that imply about where the actual security checks need to live?
Nadia: It implies that we should be looking at mechanisms like separation of instructions and data or verifiable policy generation rather than just hoping the model follows its rules perfectly.
Elias: They’ve analyzed eleven representative real-world attacks on agents, showing how things like SpAIware can violate TCB tamper resistance and secure information flow, which highlights where existing principles are failing.
Priya: It sounds like they're mapping out exactly where the current security models break down when faced with these sophisticated agent behaviors.
The paper's summary: Nadia: The paper summarizes its contributions by distilling core principles from decades of systems security research, focusing on minimum privilege, control and information flow, and sanitizing untrusted data to stop malicious instruction execution.
Elias: They then provide case studies on two real attacks that happened because the agentic systems failed to operationalize those security principles correctly.
Nadia: Beyond those two examples, they list nine additional case studies in an appendix showing how these agents can violate principles like least privilege and complete mediation.
Priya: That’s interesting because it moves the discussion from theoretical vulnerabilities to concrete, observed failures in real agent behavior.
Elias: They identify three main system-level mechanisms to mitigate a broad range of attacks: provable separation of instructions and data, least-privilege sandboxing with verifiable policy generation, and information flow control.
Nadia: These mechanisms aim to address the fundamental problem that language models inherently mix trusted instructions with untrusted data in their context windows.
Elias: They suggest that model-based defenses for separating instructions from data will always be evaded by persistent attackers, emphasizing the need for system-level enforcement instead.
The paper's improvements: Nadia: The authors propose three concrete research challenges that the community should focus on to solve these security issues: separating instructions from data, verifiable policy generation, and information flow control.
Elias: They suggest specific improvements for each area, starting with separating instructions from data by saying no instruction-data separation mechanism should be implemented in LLM context windows because current methods are just best-effort heuristics.
Nadia: That’s a strong stance; they want to prevent prompt injection by not following any instructions that appear in the context window tagged as data, rather than trying to filter them out later.
Elias: Then they advocate for implementing a verifiable policy generation pipeline that automatically converts natural-language artifacts into formal, enforceable least-privilege policies.
Priya: That’s a big shift—relocating the enforcement from the untrusted model to something deterministic, like a reference monitor, which seems much more reliable.
Nadia: And they also push for information flow control mechanisms that actively track data propagation and enforce constraints on usage, addressing how information gets disentangled as it flows through an AI model.
Elias: On the practical side, they suggest deploying fine-grained file system permissions and sandbox agents to limit access where agents might otherwise modify their own configuration files.
Priya: Limiting what an agent can touch in its own environment seems like a very tangible way to reduce the surface area for attacks.
Nadia: And finally, they call for enforcing immutability on security-critical configuration files like settings JSON so the agent cannot change its execution environment, which directly addresses TCB tamper resistance.
Conclusion: Elias: So, to wrap up, the paper "Agent Security is a Systems Problem" argues that we need to stop focusing solely on model robustness and start applying systems security principles to design agents with predictable guarantees.
Nadia: They’ve shown that the existing attacks violate fundamental principles like least privilege and complete mediation, which shows why this systemic approach is necessary.
Priya: What I take away is that we need concrete mechanisms like verifiable policy generation and information flow control because abstract model defenses just won't hold up against adaptive threats.
Elias: It’s about moving the enforcement mechanism outside the untrusted AI component and into a trusted part of the system architecture.
Nadia: This work sets a clear direction for how we should think about securing these agentic systems in practice, by demanding we treat the model as an untrusted piece of hardware or software.
Priya: It’s a necessary framework for building defenses that are actually verifiable and not just heuristic guesses.
Elias: That's it for this paper on Agent Security is a Systems Problem. We'll be looking at how these principles play out in the next discussion.
More episodes
- 2608.27496-ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injection
- Daily Summary for 2026-11-19
- 2607.07085-Is Randomness Necessary for Adaptive Data Analysis?
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2609.10601-Threshold Choice, Not Sample Size, Bounds Trustless Verification of Nondeterministic Compound AI Workflows
- 2610.38786-Adversarial Process Disturbances at a Deployable Alarm Budget: Detector Evaluation on a Simulator and a Real Plant
- 2511.08660-Binary and Multiclass Cyberattack Classification on GeNIS Dataset
- 2604.27302-Static Attribution of Android Residential Proxy Malware Using Graph Kernels
- Daily Summary for 2026-11-18