AI Agents May Always Fall for Prompt Injections

arXiv:2605.17634 · cs.CR, cs.CL, cs.CY · Submitted 2026-05-17 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: We're looking at the paper: "AI Agents May Always Fall for Prompt Injections".

Elias: Here's the gist of the paper. Prompt injection is identified as a critical vulnerability in deployed AI agents, with current defenses based on data-instruction separation proving insufficient against attacks that exploit contextual manipulation and degrade contextually appropriate behavior.

Nadia: The authors introduce Contextual Integrity (CI), a privacy theory used to judge information flow compliance against contextual norms, reframing prompt injection through this lens.

Elias: This framework suggests an impossibility result: an adversary can always construct a context where a blocked flow appears legitimate, or a defender tightening norms risks blocking genuinely appropriate flows.

Nadia: The paper develops specific benign and attack scenarios designed to force agents into norm violations by either misrepresenting the flow, manipulating the norms themselves, or mixing multiple information flows within a single message.

Title and authors: Nadia: So we're talking about this paper, "AI Agents May Always Fall for Prompt Injections." It sounds like they're arguing that the way we currently defend against prompt injections is just not cutting it anymore.

Elias: Right, it suggests that separating data from instructions isn't enough because you can manipulate the context itself to make a blocked flow look legitimate, or it can mess up how appropriate the agent acts in a situation.

Priya: So, instead of just looking at the input data versus instructions as the main problem, they propose this new idea called Contextual Integrity or CI to judge information flow compliance against contextual norms.

Nadia: Exactly. They break down what action appropriateness actually means into five dimensions: sender, transmission principle, subject, information type, and how the norm is evaluated. It’s a much richer way to look at it than just looking for specific injection vocabulary or data boundaries.

Elias: That decomposition is key because it shows that you can probe and protect at any of those five dimensions instead of just focusing on one narrow spot in the input. They show that an adversary can always construct a context where a blocked flow appears legitimate, or a defender tightening norms risks blocking genuinely appropriate flows.

Priya: The paper develops specific examples, both benign and attack scenarios, designed to force the agents into norm violations by either misrepresenting the flow, manipulating the norms themselves, or mixing multiple information flows inside one message.

Nadia: And they found that prompt injection classifiers perform almost at chance when attacks use context manipulation instead of just some simple injection words. They also point out that alignment training can actually degrade an agent's contextual appropriateness and usefulness in a way that makes them particularly vulnerable to out-of-scope requests if they don't have enough interaction history to ground their delegation norms.

Elias: That leads into the impossibility result they show, where an adversary can supply false values for any of those five CI parameters or manipulate the agent’s judgment about legitimacy even if all those parameters are correctly identified.

Priya: They also looked at how agents might fall for these injections by comparing in-scope executions with out-of-scope test messages. For instance, they compared five in-scope executions of an agent following scheduling principles against a request to forward sensitive documents to a personal email address and found the agent's transmission principle only covered scheduling, so forwarding confidential data was outside that scope and needed deferral.

Nadia: And they also explored why agents might overgeneralize based on patterns from those in-scope executions; for example, casual phrasing in delegation can cause an agent to confuse the sender's identity with the actual transmission principle.

Elias: The paper then proposes three paths forward: first, contextual alignment training using scenario pairs that share the same surface form but differ in their delegation context, which might use reinforcement learning from CI-derived rewards.

Priya: They also suggest layered defenses that combine model reasoning with system-level verification against ground truth, and CI-grounded red-teaming specifically for multiagent systems.

Nadia: So the main point is that this structured framework of Contextual Integrity is essential for training and designing agents that are actually trustworthy, because it forces us to look at context in a much more detailed way than we have before.

Elias: It’s a shift from just looking at what the input data is to understanding the entire flow of information according to established contextual rules. This whole discussion about "AI Agents May Always Fall for Prompt Injections" shows how crucial it is to move beyond simple syntax checks and into deeper reasoning about context.

Priya: I think what really stands out from this work, based on the analysis of in-scope versus out-of-scope tests, is that the agents overgeneralize when they lack full interaction history. That means we need to build better ways for agents to explicitly state their boundaries rather than guessing what's allowed.

Nadia: Right, so we’re not just patching vulnerabilities; we’re trying to teach the agent how to respect its own operational boundaries based on a deeper understanding of where the information is going and why. This sets up the next part of our conversation about how these failures play out in practice.

The paper's summary: Nadia: So, to wrap up what we just talked about, these authors are saying that current defenses against prompt injection are really only looking at the wrong thing—just separating the data from the instructions is not enough anymore.

Elias: Exactly. They introduce this framework called Contextual Integrity or CI to judge how well an AI agent follows the rules of a given situation based on context, and they find that this approach helps us see where things actually go wrong.

Nadia: Right, it’s like they break down what makes an action appropriate into five pieces: the sender who is asking, the transmission principle being used, the subject matter, what kind of information it is—like a picture or text—and how we judge all that against some norm.

Elias: That decomposition is powerful because it lets you probe and protect at any of those five dimensions instead of just trying to block one single piece of vocabulary. They show that an attacker can always build a context where something should be blocked looks totally legitimate, or if a defender gets too strict about the rules, they might block something that's actually fine.

Nadia: The paper also does some experiments and shows that simple AI classifiers struggle almost entirely when the attack uses context manipulation instead of just using specific injection words. They even found that sometimes training an agent to be more aligned can actually make it worse at understanding what the right action is in a tough spot, especially if it doesn't have enough history to know what its job boundaries are.

Elias: That vulnerability comes from those simultaneous information flows; agents struggle when they have to handle multiple things at once in one message, and authorization for one thing can leak into another up to sixty-five percent of the time. The authors argue that you can always supply false values for those context parameters or mess with the agent’s judgment of legitimacy even if you correctly identify all those parameters.

Nadia: And it gets interesting when they look at why agents sometimes overgeneralize based on past good examples, like when casual phrasing in a delegation message causes the AI to confuse who is sending the request with what kind of transmission principle it should follow.

Elias: So they’re pushing for three ways to fix this: first, training agents using scenario pairs that look similar but have different context so they learn better judgment; second, layering defenses that combine the model reasoning with system checks against real truth; and third, doing red-teaming specifically for how these things work in multiagent setups.

Nadia: It really boils down to building a structured framework based on this Contextual Integrity theory to train and design AI agents that actually respect the nuances of their context. This moves the conversation past just checking if a word is malicious and into understanding the entire flow of information.

Elias: And it opens up a lot of questions about how we can actually implement these checks in real-world systems without making them too slow or too brittle when things get messy.

The paper's improvements: Tom: So we've been talking about how current defenses are falling short against prompt injections, and now we're looking at what the authors actually suggest to fix this problem in practice.

Nadia: They propose three main directions for improvement: first, contextual alignment training using scenario pairs that look similar but have different context so the AI learns better judgment; second, layered defenses that combine model reasoning with system checks against ground truth; and third, CI-grounded red-teaming specifically for multiagent systems.

Elias: That’s the RL part where they use those CI derived rewards to train models on subtle norm evaluation instead of just pattern matching. It’s about teaching the AI *why* something is appropriate based on the flow of information.

Nadia: They also suggest this layered defense idea, which means you get model reasoning working alongside system level verification against what we know to be the truth. That way, you don't rely on just one part of the security stack to catch everything.

Elias: And they stress that for multiagent systems, you need red-teaming that is grounded in Contextual Integrity specifically—not just testing for injection words but testing how context manipulation works across different agents. That’s a big shift from what we do now.

Nadia: The paper also hints at making the agent's reasoning stronger by requiring it to explicitly state its own scope based on its interaction history, like needing a full history of turns before it can confidently delegate something.

Elias: That addresses the overgeneralization issue where agents might misinterpret casual phrasing and think they know their boundaries when they actually don't have the right context to ground their delegation norms. It’s about forcing that explicit boundary check.

Nadia: So these improvements point toward a more holistic approach: training for nuance, verifying at the system level, and testing for multiagent leakage in a way that respects context integrity.

Elias: What this means for us is we need to start designing training data around these contextual pairs before we deploy agents into high-stakes environments where they have to make decisions based on complex flows. It shifts the focus from simple input filtering to understanding the agent's entire operational world.

Conclusion: Nadia: So we’ve gone through the details of "AI Agents May Always Fall for Prompt Injections," and basically, these authors are arguing that we need to change how we think about agent security because current defenses aren't keeping up with context manipulation attacks.

Elias: Right, they’re pushing this Contextual Integrity framework to show that action appropriateness isn't just a binary thing like data versus instruction; it’s a complex five-dimensional flow where you can break the rules in many subtle ways.

Priya: I just want to check—so when they show how agents overgeneralize because of poor history, does that mean we need to build better memory systems for these AI so they know their limits?

Nadia: Exactly, Priya. It’s about making sure the agent has the necessary context—that full interaction history, as they put it—so it can actually state its boundaries instead of just guessing what's allowed.

Elias: I think that points toward this idea of contextual alignment training where we feed the AI scenario pairs that look similar but have different rules so it learns to evaluate the norm better.

Priya: From a privacy side, I’m interested in how this helps with data leakage because if the agent understands its flow boundaries better, maybe it can stop leaking sensitive information across those simultaneous information flows we talked about earlier.

Nadia: It seems like a lot of work to put all that together—training, layered verification, and new red-teaming—but the authors are saying this is essential for building trustworthy AI agents moving forward.

Elias: Yeah, it’s a structured way to handle the complexity of agent behavior instead of just patching specific injection vectors which we know are easily bypassed.

Priya: It really shows that security in AI isn't just about looking at the input text; it has to be about understanding the entire operational context and flow, which is what this paper on "AI Agents May Always Fall for Prompt Injections" gets at.

Nadia: So we’re moving toward a world where agent security is tied to a deeper theory of context integrity rather than just surface-level checks. That opens up huge questions about how we actually implement these checks in deployment and what the real costs are for those systems.

ELLIS Institute Tübingen · MPI-IS & Tübingen AI Center · University of Massachusetts Amherst

cs.CR, cs.CL, cs.CY

Submitted: 2026-05-17

Updated: 2026-11-15

Comments: NeurIPS 2026

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: Prompt injection is identified as a critical vulnerability in deployed AI agents, with current defenses based on data-instruction separation proving insufficient against attacks that exploit…

Key concepts

Contextual Integrity (CI)
A privacy theory used to judge if information flow complies with established contextual norms. It suggests that an adversary can always construct a context where a blocked action appears legitimate, or that tightening norms risks blocking safe actions.
Action Appropriateness Decomposition
A framework structuring how appropriate an agent's action is across five dimensions: sender, transmission principle, subject, information type, and norm evaluation. This helps probe failures beyond simple instruction separation.
In-Scope vs. Out-of-Scope Execution
Analyzing agent behavior by comparing actions that follow established rules (in-scope) against requests outside those rules (out-of-scope). This reveals where agents overgeneralize or fail to recognize boundary violations.
Contextual Alignment Training
A training method using scenario pairs with similar surface forms but different delegation contexts. This trains models to evaluate nuanced norms rather than just matching simple syntactic patterns.

Terminology

Summary

Prompt injection is identified as a critical vulnerability in deployed AI agents, with current defenses based on data-instruction separation proving insufficient against attacks that exploit contextual manipulation and degrade contextually appropriate behavior. p. 0 The authors introduce Contextual Integrity (CI), a privacy theory used to judge information flow compliance against contextual norms, reframing prompt injection through this lens. p. 0 This framework suggests an impossibility result: an adversary can always construct a context where a blocked flow appears legitimate, or a defender tightening norms risks blocking genuinely appropriate flows. p. 0 The paper develops specific benign and attack scenarios designed to force agents into norm violations by either misrepresenting the flow, manipulating the norms themselves, or mixing multiple information flows within a single message. p. 0

The CI decomposition structures action appropriateness across dimensions such as sender, transmission principle, subject, information type, and norm evaluation, allowing for probing and protection at any of these dimensions. p. 1 Empirically, the authors demonstrate that prompt injection classifiers perform near-chance when attacks leverage context manipulation rather than simple injection vocabulary. p. 1 Furthermore, alignment training can actively degrade contextual appropriateness and utility; agents are particularly vulnerable to out-of-scope requests when they lack the interaction history necessary to ground their delegation norms, leading them to fail in separating simultaneous information flows occurring in one message. p. 1 The impossibility argument is substantiated by showing that an adversary can supply false values for any CI parameter or manipulate the agent’s legitimacy judgment even if all parameters are correctly identified. p. 5

The paper also investigates how agents might fall for prompt injections by analyzing in-scope executions versus out-of-scope test messages. p. 0 For instance, an analysis contrasting five in-scope executions of an agent following scheduling principles with an out-of-scope request to forward sensitive documents to a personal email address reveals that the agent's delegated transmission principle only covers scheduling, and forwarding confidential data falls outside this scope, necessitating deferral. The paper further explores why agents might overgeneralize based on patterns of in-scope executions; for example, casual phrasing in delegation can lead an agent to conflate sender identity with the delegated transmission principle. p. 5

The authors propose three complementary paths forward: contextual alignment training using scenario pairs that share surface form but differ in delegation context (potentially leveraging RL from CI-derived rewards), layered defenses combining model reasoning with system-level verification against ground truth, and CI-grounded red-teaming specifically for multi-agent systems. The paper concludes that this structured, theory-driven framework is essential for training and designing trustworthy AI agents. p. 0

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Contextual Integrity (CI) Reframing for Action Appropriateness: Implement a framework that decomposes action appropriateness into five dimensions—sender, transmission principle, subject, information type, and norm evaluation—to probe failures beyond simple data/instruction separation.

  2. Red-Teaming via CI-Grounded Scenarios: Develop red-teaming loops that target specific CI parameters (e.g., sender identity or transmission principle) rather than just surface vocabulary to find context manipulation attacks that current classifiers miss, such as those where a forged user quote would assert a false sender.

  3. Norm Inference Grounding Mechanism: Enhance agent reasoning by requiring the agent to explicitly infer its delegation scope based on interaction history (full k-turn history) and use explicit boundary statements, preventing overgeneralization when test requests fall outside the delegated scope.

  4. Multi-Flow Separation Enforcement: Design system-level enforcement mechanisms that evaluate multiple simultaneous information flows within a single message independently, ensuring authorization for one flow does not leak into another, which is a failure mode observed in agents where authorization for one may leak into the other in up to 65% of cases.

  5. Contextual Alignment Training: Implement contextual alignment training using scenario pairs that share surface form but differ in delegation context, potentially utilizing reinforcement learning from CI-derived rewards to train models on nuanced norm evaluation rather than just syntactic patterns.

Sources

Related papers