Beyond Instruction Override: Benchmarking Indirect Prompt Injection with Task-Tampering Attacks
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "Beyond Instruction Override".
Elias: Here's the gist of the paper. Indirect prompt injection (IPI) benchmarks predominantly test planted instructions that redirect a tool-using agent toward an attacker-chosen task <ref:2611.07640#pg3> Yet attacker-controlled content can instead tamper with the outcome of the task already in progress <ref:2611.07640#pg4>.
Nadia: We define IPI by authority rather than instruction-like wording <ref:2611.07640#pg4>.
Elias: Defining Indirect Prompt Injection Indirect prompt injection is defined by authority rather than instruction-like wording, where planted content constitutes an attack when it attempts to influence a decision that its source is not authorized to determine <ref:2611.07640#pg4>.
Nadia: This authority boundary extends beyond commands, as an unauthorized source can steer a decision by asserting a false state, rule, consequence, capability, or interpretation <ref:2611.07640#pg4>.
Title and authors: Nadia: So, the core of the work is this new benchmark they created. They have ninety validated cases across three different environments, testing how this authority boundary works in real tool-using situations >
Elias: What’s interesting is that in every case, the attacker isn't just trying to switch tasks; they are targeting a decision that the benign task absolutely requires >
Priya: I noticed they emphasize that the attack always tampers with a required decision rather than replacing the whole task, which makes it much closer to what we see happening in real workflows >
Nadia: Yeah. They use this environment-aware case construction where they put attacker content relevant to the main task into a source that the agent has to process >
Elias: And they focus on influence primitives, which are basically ways attackers try to change things like objective rules, or claims about what the current state is >
Priya: That sounds really practical. It means we’re not just looking for weird phrasing; we’re looking for where an external piece of information can trick the agent into using a wrong amount or picking the wrong recipient >
Nadia: It shows that this isn't some abstract linguistic puzzle; it’s about how external context interacts with the agent's ongoing process >
Elias: Exactly. They are testing if an unauthorized source can set a false state, a rule, or even a capability for the agent to use during its work >
The paper's summary: Nadia: Now let’s talk about what they suggest we should do next. They point out that most existing defenses only test against instruction overriding, and that leaves task tampering only partially covered >
Elias: They find that defenses focused on just redirecting instructions don't handle these kinds of attacks very well, especially when it comes to objective attacks, which they call b(U) >
Priya: So they’re saying we need to shift our focus from just filtering commands to actually defending the decisions the agent is making in real time >
Nadia: Right. They suggest two main directions for defense design, both centered around that idea of authority, which is a big shift from what we’ve seen before >
Elias: And they point out that content screening can miss attacks if it reads like normal task data, like b(O) or b(s) attacks >
Priya: That means we can’t just rely on simple keyword filtering for everything; the content could look perfectly normal but still be poisoning a critical piece of data used in a calculation or a tool call >
Nadia: And for runtime defenses, they showed that stripping content before the model reads it only catches about twenty-six to thirty-four percent of these other types of attacks, like b(O) and b(s) >
The paper's improvements: Elias: So to wrap this up, the main implication is that defenses need to be grounded in authority, not just instruction structure >
Nadia: They suggest that tool-level gates can fail because they might block calls that use tools the task actually needs, so we have to think about what the tool call itself is actually doing >
Priya: I think action checks could be better if they only let each argument through if the source has been authorized to set it, which sounds like a much tighter control over data flow >
Elias: That’s the direction they are pushing for, focusing on controlling the arguments themselves rather than just blocking the initial input >
Nadia: So, this paper on "Beyond Instruction Override: Benchmarking Indirect Prompt Injection with Task-Tampering Attacks" shows us that to stop these kinds of attacks, we have to design defenses around who is authorized to make a decision in any given step >
Priya: It’s about controlling the flow of data and ensuring every piece of information used in a tool call comes from a source that has been explicitly cleared for that specific decision >
Elias: Exactly. And that moves us beyond just looking at the prompt text itself to looking at the entire context of who is trying to steer what happens >
Conclusion: Nadia: So we’ve looked at "Beyond Instruction Override: Benchmarking Indirect Prompt Injection with Task-Tampering Attacks" and what it means for how we build defenses against these agent attacks >
Elias: Yeah, they define indirect prompt injection differently now. It’s not just about planting a command that redirects the agent; it's about planting content that tries to influence a decision that the source shouldn't be allowed to touch >
Priya: I think what stuck with me is how they frame the attack. It’s not replacing the task; it’s tampering with a required decision, which makes it much harder to spot because the agent still thinks it’s doing its job >
Nadia: Right. They built this whole benchmark around that idea of authority. It means we need to stop looking at simple instruction commands and start looking at what the external content is trying to change about the agent's internal state or its planned actions >
Elias: And they show that defenses tested mainly against just redirecting instructions leave task tampering only partially covered, which is a big limitation for us right now >
Priya: It sounds like we need to get better at testing those objective attacks, the b(U) attacks, because those are the ones that seem to succeed more often without any defense >
Nadia: They also pointed out that content screening can miss these subtle attacks if they look like regular task data, which is a real problem for us in the field >
Elias: And on the technical side, runtime defenses like stripping content before it's read only caught about twenty-six to thirty-four percent of those other types of tampering, like b(O) and b(s) attacks >
Priya: It means action-level policies, or gates that check each argument when a tool is called, are probably more important than just filtering the initial text >
Nadia: Exactly. We need to enforce rules about which source can set which argument at the moment of execution, not just what text is in the prompt >
Elias: So if we're building defenses, the focus should be on grounding them in that authority boundary they’ve defined for every single decision >
Priya: It’s a shift from just patching input to controlling the whole process of making a choice >
Nadia: That’s what this paper on "Beyond Instruction Override: Benchmarking Indirect Prompt Injection with Task-Tampering Attacks" really shows us about how deep we have to look into agent security >
Elias: It makes me wonder how far this idea of authority control can actually scale when we move to more complex systems where agents make many decisions in a sequence >
Priya: Right, and that's where the real challenge lies for measurement and privacy research as well >
Amazon Responsible AI
cs.CR, cs.AI
Submitted: 2026-11-05
Updated: 2026-11-05
Comments: 34 pages, 6 figures. Code and data: https://github.com/amazon-science/task-tampering-prompt-injection
Code: https://github.com/amazon-science/task-tampering-prompt-injection
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 81/100
The gist: Indirect prompt injection (IPI) benchmarks predominantly test planted instructions that redirect a tool-using agent toward an attacker-chosen task <ref:2611.07640#pg3> Yet attacker-controlled content…
Key concepts
- Indirect Prompt Injection (IPI)
- IPI occurs when external content influences a decision that its source is not authorized to make. Unlike simple instruction overrides, IPI involves planting content that attempts to change a rule, state, or consequence the agent is supposed to follow. It tests the boundary of what an agent can be influenced by from unauthorized sources.
- Authority Boundary
- This concept defines where an agent's authority ends. An attack in IPI succeeds when content tries to influence a decision outside this boundary—such as asserting a false capability or rule—even if the content is not phrased as a direct command. Defenses must be grounded in respecting these established sources of authority.
- Influence Primitives
- These are the specific ways an attacker can try to manipulate an agent's decision-making process. These include targeting beliefs about objectives, rules, consequences, available actions, or the current state of the environment. Understanding these primitives helps in designing defenses that protect against various types of task tampering.
Terminology
Summary
Indirect prompt injection (IPI) benchmarks predominantly test planted instructions that redirect a tool-using agent toward an attacker-chosen task <ref:2611. p. 0 Preprint
BEYOND INSTRUCTION OVERRIDE:
BENCHMARKING INDIRECT PROMPT INJECTION
WITH TASK-TAMPERING ATTACKS
Rongting Zhang∗
, Martin Bertran, Sandeep Avula†
,
Chia-Jung Lee‡
, Vanessa Murdock
Amazon Responsible AI
ABSTRACT
Indirect prompt-injection (IPI) benchmarks predominantly test planted instructions
that redirect a tool-using agent toward an attacker-chosen task. Yet attackercontrolled content can instead tamper with the outcome of the task already in
progress. Preprint
BEYOND INSTRUCTION OVERRIDE:
BENCHMARKING INDIRECT PROMPT INJECTION
WITH TASK-TAMPERING ATTACKS
Rongting Zhang∗
, Martin Bertran, Sandeep Avula†
,
Chia-Jung Lee‡
, Vanessa Murdock
Amazon Responsible AI
ABSTRACT
Indirect prompt-injection (IPI) benchmarks predominantly test planted instructions
that redirect a tool-using agent toward an attacker-chosen task. Yet attackercontrolled content can instead tamper with the outcome of the task already in
progress. For example, a buyer’s dispute evidence can cite a false refund policy to
inflate the refund without redirecting the agent away from resolving the dispute. We define IPI by authority rather than instruction-like wording: planted content
constitutes an attack when it attempts to influence a decision that its source is not
authorized to determine.Where the paper says it
Where the paper says it
Where the paper says it
Defining Indirect Prompt Injection
Indirect prompt injection is defined by authority rather than instruction-like wording, where planted content constitutes an attack when it attempts to influence a decision that its source is not authorized to determine <ref:2611.07640#pg4>. This authority boundary extends beyond commands, as an unauthorized source can steer a decision by asserting a false state, rule, consequence, capability, or interpretation <ref:2611. p. 1 (2026) frame instruction injection as following
a command from a source unauthorized to issue it. We argue that this authority boundary should
extend beyond commands: an unauthorized source can steer a decision by asserting a false state,
rule, consequence, capability, or interpretation. We argue that this authority boundary should
extend beyond commands: an unauthorized source can steer a decision by asserting a false state,
rule, consequence, capability, or interpretation. External content constitutes IPI when it attempts to
influence a decision that its source is not authorized to determine.Where the paper says it
Where the paper says it
Benchmark Construction and Primitives
The authors introduce a benchmark where in each of 90 validated cases across three tool-use environments, the user request specifies the source that resolves the targeted decision <ref:2611.07640#pg4>. Every attack tampers with a decision required by the benign task rather than replacing that task <ref:2611. p. 0 We introduce a benchmark that makes this boundary
explicit: in each of 90 validated cases across three tool-use environments, the user
request specifies the source that resolves the targeted decision. Every attack tampers
with a decision required by the benign task rather than replacing that task. To measure this gap, we use environment-aware case construction. Each case places attackercontrolled content that is relevant to the benign task in a source the agent must process, identifies a
decision that source is not authorized to determine, and provides an agent-visible basis for resolving
the conflict. Each primitive has a typical form: an objective or rule reads most naturally as a request, a state or
consequence as a claim. The form does not define the primitive, however: an imperative can target
b(O) (“Read the displayed $899.Where the paper says it
Where the paper says it
Where the paper says it
Attack Taxonomy and Evaluation
The benchmark spans 90 cases across productivity, DevOps, and marketplace settings <ref:2611.07640#pg5>. The harmful effect can be structured (calling a different tool or changing a discrete argument) or free-form (writing text like a message body) <ref:2611. p. 3 Table 1: Benchmark size. S / F: cases
whose harmful effect is a structured argument / free-form content. Our benchmark makes this boundary explicit in 90 validated task-tampering
cases across three environments; within each case, the planted content justifies a fixed harmful
outcome through different influence primitives, so defenses can be compared per primitive, not only
by aggregate ASR. Our results show that defenses tested mainly against redirecting instructions leave
task tampering only partly covered: objective attacks (b(U)), the kind prior evaluations mainly test,
already succeed less often without a defense than attacks on decision rules, available actions, and
state, and b(U) remains among the three lowest-ASR primitives under every defense but CaMeL.Where the paper says it
Where the paper says it
Defense Analysis and Findings
The results show that defenses tested mainly against redirecting instructions leave task tampering only partly covered <ref:2611.07640#pg4>. Objective attacks (b(U)), the kind prior evaluations mainly test, already succeed less often without a defense than attacks on decision rules, available actions, and state <ref:2611. p. 8 Our benchmark makes this boundary explicit in 90 validated task-tampering
cases across three environments; within each case, the planted content justifies a fixed harmful
outcome through different influence primitives, so defenses can be compared per primitive, not only
by aggregate ASR. Our results show that defenses tested mainly against redirecting instructions leave
task tampering only partly covered: objective attacks (b(U)), the kind prior evaluations mainly test,
already succeed less often without a defense than attacks on decision rules, available actions, and
state, and b(U) remains among the three lowest-ASR primitives under every defense but CaMeL. Our results show that defenses tested mainly against redirecting instructions leave
task tampering only partly covered: objective attacks (b(U)), the kind prior evaluations mainly test,
already succeed less often without a defense than attacks on decision rules, available actions, and
state, and b(U) remains among the three lowest-ASR primitives under every defense but CaMeL. The results suggest two directions for defense design, both grounded in authority. The results suggest two directions for defense design, both grounded in authority. Content screening
can miss attacks whose content reads as ordinary task data, such as b(O) and b(s) attacks; screening
could therefore also ask whether a source is authorized to determine the targeted decision, not only
whether its content reads as a redirecting instruction.Where the paper says it
Where the paper says it
Where the paper says it
Runtime Defense Efficacy
Runtime defenses can remove content before the model reads it or reject a resulting action <ref:2611.07640#pg9>. Content stripping by DRIFT’s Injection Isolator shows that it fully strips b(U) attacks but only 26. p. 8 Content isolation. By contrast, DRIFT’s selective protection comes principally from its Injection
Isolator (Appendix Table 26), which across both DRIFT configurations and both models fully strips
52. ‡Work done while at Amazon. 1
arXiv:2611. Content screening
can miss attacks whose content reads as ordinary task data, such as b(O) and b(s) attacks; screening
could therefore also ask whether a source is authorized to determine the targeted decision, not only
whether its content reads as a redirecting instruction. Tool-level gates can fail to block attacked calls
that use tools the task needs; action checks could instead admit each consequential argument only
from a source authorized to set it.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion on Defense Design
The results suggest that content screening can miss attacks whose content reads as ordinary task data, such as b(O) and b(s) attacks <ref:2611.07640#pg10>. Tool-level gates can fail to block attacked calls that use tools the task needs <ref:2611. p. 8 Content screening
can miss attacks whose content reads as ordinary task data, such as b(O) and b(s) attacks; screening
could therefore also ask whether a source is authorized to determine the targeted decision, not only
whether its content reads as a redirecting instruction. Tool-level gates can fail to block attacked calls
that use tools the task needs; action checks could instead admit each consequential argument only
from a source authorized to set it. Content screening
can miss attacks whose content reads as ordinary task data, such as b(O) and b(s) attacks; screening
could therefore also ask whether a source is authorized to determine the targeted decision, not only
whether its content reads as a redirecting instruction. Tool-level gates can fail to block attacked calls
that use tools the task needs; action checks could instead admit each consequential argument only
from a source authorized to set it. Our results show that defenses tested mainly against redirecting instructions leave
task tampering only partly covered: objective attacks (b(U)), the kind prior evaluations mainly test,
already succeed less often without a defense than attacks on decision rules, available actions, and
state, and b(U) remains among the three lowest-ASR primitives under every defense but CaMeL. The results suggest two directions for defense design, both grounded in authority.Where the paper says it
Where the paper says it
Where the paper says it
AI Use Statement
We used generative AI tools, principally Claude Code, for two tasks: generating and revising the synthetic benchmark dataset, and supporting qualitative analysis through trajectory annotation <ref:2611.07640#pg11>. We also used these tools to find and summarize related work, create or edit software code (including adapting existing defenses to our benchmark) and research artifacts, generate figures from authors-specified measurements, and draft, organize, and edit the paper <ref:2611. p. 9 Preprint
AI USE STATEMENT
We used generative AI tools, principally Claude Code, for two tasks: generating and revising the
synthetic benchmark dataset, and supporting qualitative analysis through trajectory annotation. We also used these tools to find and summarize related work, create or edit software code (including
adapting existing defenses to our benchmark) and research artifacts, generate figures from authorspecified measurements, and draft, organize, and edit the paper.Where the paper says it
Ethics Statement
This work constructs indirect prompt injections, content that an attacker plants in a resource an agent later reads, and could in principle help an attacker write them <ref:2611.07640#pg10>. Two properties limit this risk: first, the attacks are passive because the attacker plants content but cannot control whether or when an agent reads it <ref:2611. p. 9 Two properties limit this risk. First,
the attacks are passive: the attacker plants content but cannot control whether or when an agent reads
it, and attackers already control the content they own, so a catalogue of attack forms helps defenders
enumerate the attack surface of their deployments more than it helps attackers. First,
the attacks are passive: the attacker plants content but cannot control whether or when an agent reads
it, and attackers already control the content they own, so a catalogue of attack forms helps defenders
enumerate the attack surface of their deployments more than it helps attackers. Second, each case is
written for the tools, records, and decisions of one simulated environment, so its content does not
transfer to a real deployment without redesigning the case.Where the paper says it
Where the paper says it
Reproducibility Statement
The benchmark dataset and code are available at https://github.com/amazon-science/task-tampering-prompt-injection under CC BY-NC 4. p. 0 Code and data are available
at https://github. com/amazon-science/task-tampering-prompt-injection. com/amazon-science/task-tampering-prompt-injection under CC
BY-NC 4. 0, including the defense implementations whose licenses permit redistribution. Every reported number is computed by script from the run records, and literature
claims were checked against the cited papers. The authors designed the methodology, selected
the analyses, and interpreted the results, and take responsibility for the final content of this work,
including text, claims, code, data, and other artifacts produced with generative AI assistance.Where the paper says it
Where the paper says it
Where the paper says it
References
Sahar Abdelnabi and Eugene Bagdasarian. AI agents may always fall for prompt injections <ref:2611. p. 9 REFERENCES
Sahar Abdelnabi and Eugene Bagdasarian. AI agents may always fall for prompt injections. org/abs/2605. Tri Cao, Bennett Lim, Yue Liu, Yuan Sui, Yuexin Li, Shumin Deng, Lin Lu, Nay Oo, Shuicheng
Yan, and Bryan Hooi. Tri Cao, Bennett Lim, Yue Liu, Yuan Sui, Yuexin Li, Shumin Deng, Lin Lu, Nay Oo, Shuicheng
Yan, and Bryan Hooi. VPI-Bench: Visual prompt injection attacks for computer-use agents. id=UMauKu2azg. Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. StruQ: Defending against prompt
injection with structured queries. , 2026a) isolate applications
or execution steps, while CaMeL and FIDES (Debenedetti et al. , 2026; Costa et al. The policy or plan generally does not constrain the content of free-text arguments. Architectural isolation and information-flow control constrain how content can reach agent components and actions. , 2025; Zhan et al. , 2024; Debenedetti et al. Indirect prompt injection (IPI), ranked the top vulnerability for LLM applications
by OWASP (OWASP Foundation, 2025), occurs when adversarial content in data the agent retrieves
causes it to act contrary to the user’s intent (Greshake et al. Existing IPI benchmarks have expanded from single-generation applications to stateful and dynamic
tool-using agents (Yi et al.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Authority-based IPI boundary definition: Define indirect prompt injection by authority rather than instruction-like wording, meaning planted content constitutes an attack when it
attempts to influence a decision that its source is not authorized to determine.
This allows for defenses that target decisions outside of simple command redirection. -
Multi-primitive attack modeling: Use POMDP-inspired views to vary how decision is manipulated by targeting specific belief primitives such as objective, action selection rule, consequence model, available actions, observation semantics, and current state. This enables the creation of attacks beyond direct objective manipulation by targeting
belief in the agent’s decision process.
-
Environment-aware case construction: Implement a system where each case explicitly specifies "the attacker controlled content that is relevant to the benign task in a source the agent must process, identifies a decision that source is not authorized to determine, and provides an agent-visible basis for resolving the conflict." This ensures defenses are evaluated against real-world scenarios where authority boundaries are explicit.
-
Primitive-level defense analysis: Evaluate defenses per primitive rather than aggregate success rates to identify specific weaknesses. This reveals that
attacks on three of the five other targets (decision rules, available actions, and state) have higher mean success than objective attacks across five LLMs
without a defense. -
Runtime enforcement against tool calls: Deploy action-level policies or argument-level gates that
constrain calls whose tools and arguments follow from the request,
specifically blockingattacked calls that use a tool the task needs, and its consequential values are resolved at runtime from retrieved content.
Abstract
Indirect prompt-injection (IPI) benchmarks predominantly test planted instructions that redirect a tool-using agent toward an attacker-chosen task. Yet attacker-controlled content can instead tamper with the outcome of the task already in progress. A buyer's dispute evidence can cite a false refund policy to inflate a refund without changing the task. We define IPI by authority rather than instruction-like wording: planted content constitutes an attack when it attempts to influence a decision that its source is not authorized to determine. We introduce a benchmark that makes this boundary explicit: in each of 90 validated cases across three tool-use environments, the user request specifies the source that resolves the targeted decision. Every attack tampers with a decision required by the benign task rather than replacing that task. Each case has attacks with the same harmful outcome that target up to six parts of the agent's decision making: its objective, decision rules, available actions, expected consequences, reading of observations, or beliefs about the current state. Without a defense, attacks on three of the five other targets (decision rules, available actions, and state) have higher mean success than objective attacks across five LLMs. A detection-and-removal defense reduces objective-targeting attacks to 2.4% success but leaves observation- and state-targeting attacks at 9.5% and 12.1%; this gap persists under detection-optimized rewrites. A tool-policy defense still allows 15.4% attack success, close to 17.0% without a defense, because these attacks generally retain expected tools and manipulate values obtained during execution. Evaluations whose attacks redirect the task or target only the agent's objective can therefore overestimate the coverage of these defenses. Code and data are available at https://github.com/amazon-science/task-tampering-prompt-injection.
Sources
- AI Agents May Always Fall for Prompt Injections
- Securing AI Agents with Information-Flow Control
- AgentDyn: Are Your Agent Security Defenses Deployable in Real-World Dynamic Environments?
- Ignore Previous Prompt: Attack Techniques For Language Models
- Progent: Securing AI Agents with Privilege Control
- PromptArmor: Simple yet Effective Prompt Injection Defenses
- Agent Security Needs Redefinition through a Holistic Framework
- The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions
- The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysis
- Aligning Provenance with Authorization: A Dual-Graph Defense for LLM Agents
- ARGUS: Defending LLM Agents Against Context-Aware Prompt Injection
Related papers
- From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment
- Privacy Leakage from Recurrent State in LLM Inference
- Incrementally Verifiable Oracle Aided Computation
- Privilege and confidentiality in generative AI workflows
- BadMoE: Backdooring Mixture-of-Experts LLMs via Routing-Aware Triggers and Task-Underutilized Experts
- CoVault: Secure, Scalable Analytics of Personal Data