Beyond Instruction Override: Benchmarking Indirect Prompt Injection with Task-Tampering Attacks
summary
The gist
Indirect prompt injection (IPI) benchmarks predominantly test planted instructions that redirect a tool-using agent toward an attacker-chosen task <ref:2611.07640#pg3> Yet attacker-controlled content…
In short
This research introduces a benchmark for Indirect Prompt Injection (IPI), focusing on attacks that tamper with an agent's ongoing task rather than just redirecting its instructions. The benchmark tests defenses by placing attacker-controlled content in contexts where the source is not authorized to determine a decision, revealing weaknesses in existing instruction-based security measures and suggesting authority-grounded defenses are necessary.
Key concepts
- Indirect Prompt Injection (IPI)
- IPI occurs when external content influences a decision that its source is not authorized to make. Unlike simple instruction overrides, IPI involves planting content that attempts to change a rule, state, or consequence the agent is supposed to follow. It tests the boundary of what an agent can be influenced by from unauthorized sources.
- Authority Boundary
- This concept defines where an agent's authority ends. An attack in IPI succeeds when content tries to influence a decision outside this boundary—such as asserting a false capability or rule—even if the content is not phrased as a direct command. Defenses must be grounded in respecting these established sources of authority.
- Influence Primitives
- These are the specific ways an attacker can try to manipulate an agent's decision-making process. These include targeting beliefs about objectives, rules, consequences, available actions, or the current state of the environment. Understanding these primitives helps in designing defenses that protect against various types of task tampering.
Terminology used across episodes
This episode discusses
- Beyond Instruction Override: Benchmarking Indirect Prompt Injection with Task-Tampering Attacks · Paper Radio
- AI Agents May Always Fall for Prompt Injections · Paper Radio
- Securing AI Agents with Information-Flow Control
- AgentDyn: Are Your Agent Security Defenses Deployable in Real-World Dynamic Environments?
- Ignore Previous Prompt: Attack Techniques For Language Models
- Progent: Securing AI Agents with Privilege Control
- PromptArmor: Simple yet Effective Prompt Injection Defenses
- Agent Security Needs Redefinition through a Holistic Framework · Paper Radio
- The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions
- The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysis
- Aligning Provenance with Authorization: A Dual-Graph Defense for LLM Agents
- ARGUS: Defending LLM Agents Against Context-Aware Prompt Injection
The paper
Beyond Instruction Override: Benchmarking Indirect Prompt Injection with Task-Tampering Attacks · Read on arXiv
Amazon Responsible AI
Indirect prompt-injection (IPI) benchmarks predominantly test planted instructions that redirect a tool-using agent toward an attacker-chosen task. Yet attacker-controlled content can instead tamper with the outcome of the task already in progress. A buyer's dispute evidence can cite a false refund policy to inflate a refund without changing the task. We define IPI by authority rather than instruction-like wording: planted content constitutes an attack when it attempts to influence a decision that its source is not authorized to determine. We introduce a benchmark that makes this boundary explicit: in each of 90 validated cases across three tool-use environments, the user request specifies the source that resolves the targeted decision. Every attack tampers with a decision required by the benign task rather than replacing that task. Each case has attacks with the same harmful outcome that target up to six parts of the agent's decision making: its objective, decision rules, available actions, expected consequences, reading of observations, or beliefs about the current state. Without a defense, attacks on three of the five other targets (decision rules, available actions, and state) have higher mean success than objective attacks across five LLMs. A detection-and-removal defense reduces objective-targeting attacks to 2.4% success but leaves observation- and state-targeting attacks at 9.5% and 12.1%; this gap persists under detection-optimized rewrites. A tool-policy defense still allows 15.4% attack success, close to 17.0% without a defense, because these attacks generally retain expected tools and manipulate values obtained during execution. Evaluations whose attacks redirect the task or target only the agent's objective can therefore overestimate the coverage of these defenses. Code and data are available at https://github.com/amazon-science/task-tampering-prompt-injection.
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "Beyond Instruction Override".
Elias: Here's the gist of the paper. Indirect prompt injection (IPI) benchmarks predominantly test planted instructions that redirect a tool-using agent toward an attacker-chosen task <ref:2611.07640#pg3> Yet attacker-controlled content can instead tamper with the outcome of the task already in progress <ref:2611.07640#pg4>.
Nadia: We define IPI by authority rather than instruction-like wording <ref:2611.07640#pg4>.
Elias: Defining Indirect Prompt Injection Indirect prompt injection is defined by authority rather than instruction-like wording, where planted content constitutes an attack when it attempts to influence a decision that its source is not authorized to determine <ref:2611.07640#pg4>.
Nadia: This authority boundary extends beyond commands, as an unauthorized source can steer a decision by asserting a false state, rule, consequence, capability, or interpretation <ref:2611.07640#pg4>.
Title and authors: Nadia: So, the core of the work is this new benchmark they created. They have ninety validated cases across three different environments, testing how this authority boundary works in real tool-using situations >
Elias: What’s interesting is that in every case, the attacker isn't just trying to switch tasks; they are targeting a decision that the benign task absolutely requires >
Priya: I noticed they emphasize that the attack always tampers with a required decision rather than replacing the whole task, which makes it much closer to what we see happening in real workflows >
Nadia: Yeah. They use this environment-aware case construction where they put attacker content relevant to the main task into a source that the agent has to process >
Elias: And they focus on influence primitives, which are basically ways attackers try to change things like objective rules, or claims about what the current state is >
Priya: That sounds really practical. It means we’re not just looking for weird phrasing; we’re looking for where an external piece of information can trick the agent into using a wrong amount or picking the wrong recipient >
Nadia: It shows that this isn't some abstract linguistic puzzle; it’s about how external context interacts with the agent's ongoing process >
Elias: Exactly. They are testing if an unauthorized source can set a false state, a rule, or even a capability for the agent to use during its work >
The paper's summary: Nadia: Now let’s talk about what they suggest we should do next. They point out that most existing defenses only test against instruction overriding, and that leaves task tampering only partially covered >
Elias: They find that defenses focused on just redirecting instructions don't handle these kinds of attacks very well, especially when it comes to objective attacks, which they call b(U) >
Priya: So they’re saying we need to shift our focus from just filtering commands to actually defending the decisions the agent is making in real time >
Nadia: Right. They suggest two main directions for defense design, both centered around that idea of authority, which is a big shift from what we’ve seen before >
Elias: And they point out that content screening can miss attacks if it reads like normal task data, like b(O) or b(s) attacks >
Priya: That means we can’t just rely on simple keyword filtering for everything; the content could look perfectly normal but still be poisoning a critical piece of data used in a calculation or a tool call >
Nadia: And for runtime defenses, they showed that stripping content before the model reads it only catches about twenty-six to thirty-four percent of these other types of attacks, like b(O) and b(s) >
The paper's improvements: Elias: So to wrap this up, the main implication is that defenses need to be grounded in authority, not just instruction structure >
Nadia: They suggest that tool-level gates can fail because they might block calls that use tools the task actually needs, so we have to think about what the tool call itself is actually doing >
Priya: I think action checks could be better if they only let each argument through if the source has been authorized to set it, which sounds like a much tighter control over data flow >
Elias: That’s the direction they are pushing for, focusing on controlling the arguments themselves rather than just blocking the initial input >
Nadia: So, this paper on "Beyond Instruction Override: Benchmarking Indirect Prompt Injection with Task-Tampering Attacks" shows us that to stop these kinds of attacks, we have to design defenses around who is authorized to make a decision in any given step >
Priya: It’s about controlling the flow of data and ensuring every piece of information used in a tool call comes from a source that has been explicitly cleared for that specific decision >
Elias: Exactly. And that moves us beyond just looking at the prompt text itself to looking at the entire context of who is trying to steer what happens >
Conclusion: Nadia: So we’ve looked at "Beyond Instruction Override: Benchmarking Indirect Prompt Injection with Task-Tampering Attacks" and what it means for how we build defenses against these agent attacks >
Elias: Yeah, they define indirect prompt injection differently now. It’s not just about planting a command that redirects the agent; it's about planting content that tries to influence a decision that the source shouldn't be allowed to touch >
Priya: I think what stuck with me is how they frame the attack. It’s not replacing the task; it’s tampering with a required decision, which makes it much harder to spot because the agent still thinks it’s doing its job >
Nadia: Right. They built this whole benchmark around that idea of authority. It means we need to stop looking at simple instruction commands and start looking at what the external content is trying to change about the agent's internal state or its planned actions >
Elias: And they show that defenses tested mainly against just redirecting instructions leave task tampering only partially covered, which is a big limitation for us right now >
Priya: It sounds like we need to get better at testing those objective attacks, the b(U) attacks, because those are the ones that seem to succeed more often without any defense >
Nadia: They also pointed out that content screening can miss these subtle attacks if they look like regular task data, which is a real problem for us in the field >
Elias: And on the technical side, runtime defenses like stripping content before it's read only caught about twenty-six to thirty-four percent of those other types of tampering, like b(O) and b(s) attacks >
Priya: It means action-level policies, or gates that check each argument when a tool is called, are probably more important than just filtering the initial text >
Nadia: Exactly. We need to enforce rules about which source can set which argument at the moment of execution, not just what text is in the prompt >
Elias: So if we're building defenses, the focus should be on grounding them in that authority boundary they’ve defined for every single decision >
Priya: It’s a shift from just patching input to controlling the whole process of making a choice >
Nadia: That’s what this paper on "Beyond Instruction Override: Benchmarking Indirect Prompt Injection with Task-Tampering Attacks" really shows us about how deep we have to look into agent security >
Elias: It makes me wonder how far this idea of authority control can actually scale when we move to more complex systems where agents make many decisions in a sequence >
Priya: Right, and that's where the real challenge lies for measurement and privacy research as well >
More episodes
- 2608.27496-ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injection
- Daily Summary for 2026-11-19
- 2607.07085-Is Randomness Necessary for Adaptive Data Analysis?
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2609.10601-Threshold Choice, Not Sample Size, Bounds Trustless Verification of Nondeterministic Compound AI Workflows
- 2610.38786-Adversarial Process Disturbances at a Deployable Alarm Budget: Detector Evaluation on a Simulator and a Real Plant
- 2511.08660-Binary and Multiclass Cyberattack Classification on GeNIS Dataset
- 2604.27302-Static Attribution of Android Residential Proxy Malware Using Graph Kernels
- Daily Summary for 2026-11-18