DeepStress: Stress-Testing Deep Search Agents
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "DeepStress: Stress-Testing Deep Search Agents".
Jane: Here's the gist of the paper. DeepStress proposes a simulation framework to stress-test deep search agents by controlling the frequency of challenging evidence through a controlled synthetic environment.
Tom: This framework allows researchers to precisely manipulate document reliability along three orthogonal axes: trustworthiness, relevance, and factuality.
Jane: By intercepting the agent’s search calls and replacing the retrieval module with this synthetic environment, DeepStress evaluates how search agents behave when faced with unreliable information.
Tom: DeepStress Framework and Dimensions The framework introduces a controlled simulation stress test environment that evaluates search agents by intercepting tool calls and returning synthetic documents modified along three axes: trustworthiness, relevance, and factuality.
Paper summary: Tom: So we’re looking at this paper called DeepStress: Stress-Testing Deep Search Agents. Basically, it’s about figuring out how much we can actually trust these deep search agents when the information they find is bad.
Jane: That sounds really practical, Tom. What’s the main idea here? What problem are they trying to solve with this framework?
Tom: They propose a simulation environment to stress-test these agents by controlling how often they see challenging evidence in a controlled setup. It lets researchers manipulate document reliability along three specific axes: trustworthiness, relevance, and factuality.
Lu: I think the core idea is that we need more than just testing on perfect data; we need to see how agents handle things that are deliberately unreliable because that’s where real-world problems happen.
Meng: So they aren't just giving them a fixed set of bad documents, but they’re controlling the probability of each type of unreliability for every single search call. That sounds like a lot of setup work to get right at scale.
Lalam: It sounds like they are intercepting the agent's search calls and replacing the actual retrieval module with this synthetic environment so they can test it under different conditions, DeepStress: Stress-Testing Deep Search Agents.
Tom: Exactly. They use this framework to see how agents behave when faced with information that’s degraded across these three dimensions, and they also introduce new metrics for measuring those outcomes.
Jane: The paper claims that testing several agents on things like HotpotQA and BrowseComp-Plus shows substantial differences in how they handle unreliable information, which is a big deal because it shows where the vulnerabilities are.
Lu: They introduce these final-answer outcomes like correct or incorrect, but they also have concepts like Abstention or Exceed tool budget, which gives a better picture of what’s actually happening when things go wrong.
Meng: And they have this Reliability-Aware Score that incorporates the document quality condition to give full credit to correct answers and partial credit for other behaviors when reliable evidence is unlikely. That feels like a way to properly weight the results.
Lalam: They also look at how agents change their search queries when they face degraded documents, seeing if they get more constrained or more source-aware, or if they adapt at all.
Tom: So it’s not just about the final answer being right or wrong; it’s about the whole process of searching and reasoning under these stressful conditions.
Jane: That leads into what they found: agents show substantial behavior differences depending on which dimension—trustworthiness, relevance, or factuality—they are facing degraded in.
Lu: For example, they find that when trustworthiness decreases for something like DR-Tulu, the agent becomes less reliable overall because the source is weaker than it used to be.
Meng: And GPT-based systems seem more likely to avoid giving a final answer entirely by either abstaining or exhausting their tool-call budget when facing these degraded documents. That suggests some models are more cautious than others under stress.
Lalam: The qualitative analysis also looked at the volume and content of the search queries produced by different systems, showing that some agents react by making their queries more constrained or more source-aware, while others show little adaptation at all.
Tom: It’s interesting to see that different agent architectures react in distinct ways when they encounter unreliable context; it suggests we might need to design agents with specific resilience mechanisms depending on the type of degradation they expect.
Jane: What this means for us is that if we want search agents to be robust in real-world applications, we can’t just test them on clean data anymore; we have to simulate this controlled degradation more rigorously.
Lu: The paper suggests that their framework makes it possible to precisely measure how much parametric knowledge an agent has and how it interacts with contradictory evidence.
Meng: If you’re trying to deploy this at scale, the cost of running this stress test simulation environment is something we’d need to consider, so it’s not just a theoretical exercise.
Lalam: They also suggest further qualitative analysis of chain-of-thought reasoning and search query formulations as future work because seeing that step is crucial for understanding the 'why' behind the behavior.
Tom: So, overall, DeepStress gives us a way to systematically test the robustness of deep search agents by controlling the exact nature of bad evidence they encounter.
Jane: And it points toward a future where we can better understand how these complex systems handle uncertainty and contradiction in information retrieval tasks.
Conclusion: Tom: So, we’re wrapping up our look at DeepStress, which is this paper about stress-testing deep search agents by controlling how much bad evidence they see in a simulated environment.
Jane: It sounds like they built a way to really push these agents to their limits instead of just testing them on perfect data.
Tom: Right, they’re manipulating three things—trustworthiness, relevance, and factuality—and sampling probabilities for each one for every single search call. It lets them precisely control the reliability of the documents feeding the agent.
Lu: What I find interesting is how they set up that synthetic environment to make sure it really tests those agent behaviors under degraded conditions.
Meng: So, instead of just giving them a bad document, they let you choose how much trustworthy or relevant that document is for each question. That’s a different kind of stress test than just throwing garbage at the system.
Lalam: And they have these specific metrics to measure the agent's response, like if it gives a final answer correctly or if it just quits early because the evidence was bad.
Jane: It seems like they found that agents react very differently depending on which quality—trustworthiness or factuality—is being degraded.
Tom: That’s what they showed; for example, when trustworthiness goes down, one agent starts failing much more often than another.
Jane: And the way the agents change their search queries when things are unreliable is also a key part of this study.
Lu: It suggests that some models adapt by searching smarter, while others just get stuck and stop trying altogether.
Meng: From a deployment standpoint, seeing how they react to these kinds of failures is important because it tells you what breaks at scale when the data isn't perfect.
Tom: So the main implication here is that we need better ways to understand how much knowledge an AI has and how it handles conflicting information in real-world scenarios.
Jane: It gives us a new tool, DeepStress, to measure that interaction precisely across different types of evidence degradation. (Sound of a short musical transition)
Orange Research · Aix-Marseille University, CNRS, LIS UMR 7020
cs.CL
Submitted: 2026-07-15
Updated: 2026-11-12
Comments: 8 pages preprint
Code: https://github.com/OpenSourcesGroup/opensources
License: http://creativecommons.org/licenses/by-sa/4.0/
Importance score: 91/100
The gist: DeepStress proposes a simulation framework to stress-test deep search agents by controlling the frequency of challenging evidence through a controlled synthetic environment.
Key concepts
- Trustworthiness (T)
- This measures how reliable or credible a piece of information is. In the simulation, documents are generated with varying levels of trustworthiness, ranging from highly credible evidence to intentionally untrustworthy sources. Agents must navigate these different credibility levels.
- Relevance (R)
- Relevance assesses how pertinent a document is to the specific search query being asked. The framework allows researchers to control this dimension, testing if agents can maintain focus when presented with documents that are either highly relevant or completely off-topic.
- Factuality (F)
- Factuality determines whether the information presented in a document is actually true. Documents can be made factually correct by grounding them in real data or intentionally made incorrect by prompting the LLM to introduce severe errors, allowing testing of agents' ability to detect falsehoods.
Terminology
Summary
DeepStress proposes a simulation framework to stress-test deep search agents by controlling the frequency of challenging evidence through a controlled synthetic environment. p. 0 This phenomenon occurs rarely in realistic benchmarks but can
lead to dramatic failure in real life applications. Therefore in this study we
propose DeepStress, a stress testing framework that controls the frequency
of challenging evidence by replacing the retrieval module of search agents
with a controlled synthetic environment. It intercepts the agent’s
search calls and replaces the retrieval module with a synthetic environment, allowing us to
precisely manipulate the properties of the returned documents. Specifically, we degrade
document quality along three orthogonal axes: trustworthiness, relevance, and factuality. be finely controlled, intercepting the agent’s search calls and replacing the retrieval module with controlled synthetic documents. Our experiments, focused on three reliability
dimensions (trustworthiness, relevance and factuality), have revealed substantial behavior
differences when faced with degraded documents.Where the paper says it
Where the paper says it
Where the paper says it
DeepStress Framework and Dimensions
The framework introduces a controlled simulation stress test environment that evaluates search agents by intercepting tool calls and returning synthetic documents modified along three axes: trustworthiness, relevance, and factuality. A scenario γ specifies a probability for each dimension, chosen by the experimenter. p. 2 , Dn} be the set of dimensions under study. A scenario γ specifies a probability for each dimension, chosen by the experimenter:
πγ =
πγ(D1), . , πγ(Dn)
(1)
For each simulated search call, we sample one binary label per dimension from the scenario
probabilities. These labels define the target properties of the returned document.Where the paper says it
Where the paper says it
Document Reliability Simulation
The paper focuses on manipulating three source attributes: Trustworthiness (T), Relevance (R), and Factuality (F). Documents are generated with an LLM using the sampled values of T, R, and F to control the output. p. 2 • Factuality (F): whether the claims contained in the source are factually correct. We generate documents with an LLM, using the sampled values of T, R, and F to control the
output. Full prompt and generation details are given in Appendix D. Factually correct documents (F=1) are grounded in the selected support context, while
factually incorrect documents (F=0) are prompted to introduce severe factual errors.Where the paper says it
Where the paper says it
Evaluation Metrics
The study defines metrics to reflect various possible agent behaviors, going beyond a binary correct/incorrect evaluation. These include Final-answer outcomes, such as Correct (Ci), Incorrect (Ii), Abstention (Ai), Exceed tool budget (Bi), and Missing answer tag (Mi). p. 8 0%
gpt-5-nano (low) gpt-5-nano (medium) gpt-5-nano (high)
gpt-5-mini (low) gpt-5-mini (medium) gpt-5-mini (high)
SpiqaDR-3B
Search-R1 AutoRefine
ASearcher
DR-Tulu DR-Tulu (no reasoning)
0
20
40
60
80
100
52. 9%
Correct Abstention Missing answer tag Exceed tool budget Incorrect
Figure 2: Outcomes at factuality levels 0% and 50% on HotpotQA and BrowseComp-Plus. (5)
Thus, correct answers receive full credit. Abstentions and tool-budget exhaustion receive
partial credit when reliable evidence is unlikely, while incorrect answers and missing answer
tags receive no credit.Where the paper says it
Where the paper says it
Experimental Setup and Results
The evaluation involves 12 systems across HotpotQA and BrowseComp-Plus datasets, with a maximum budget of 100 tool calls per question. The results demonstrate substantial behavior differences when agents face degraded documents across the three dimensions. p. 9 Our experiments, focused on three reliability
dimensions (trustworthiness, relevance and factuality), have revealed substantial behavior
differences when faced with degraded documents. Our detailed evaluation shows that
agents handle degraded conditions in different ways: final performances, expressing uncertainty, number of tool calls, reaction against degraded documents. GPT-based systems show a different pattern: when trustworthiness is
degraded, they more often avoid giving a final answer, either by abstaining or by exhausting
the tool-call budget. For instance, GPT-5-mini (low) increases its abstention rate from 14.Where the paper says it
Where the paper says it
Qualitative Analysis of Agent Behavior
Additional experiments examine the volume and content of search queries produced by different systems under degraded document conditions. These analyses reveal that some agents react to degraded documents by making their queries more constrained or more source-aware, while others show little adaptation. p. 6 When degrading trustworthiness (Figure 1a), some systems are almost unaffected, while
others change their behavior substantially. Several open-weight search agents, including
SpiqaDR, Search-R1, AutoRefine, and ASearcher, show little or no change in final-answer
outcomes as trustworthiness decreases. We also studied the coherence of reasoning
verbalization of models when faced with degraded conditions in C. This constitutes
preliminary work towards process-based evaluation, that cannot be reported fully in the
core paper but reveals interesting insights towards more qualitative evaluation of search
agents, constituting promising future directions.Where the paper says it
Where the paper says it
Conclusion and Future Directions
The study reveals that agents handle degraded conditions in different ways regarding final performances, expressing uncertainty, and reaction against degraded documents. The framework makes it possible to precisely measure the role of parametric knowledge in search agents and their interaction with contradictory evidence. p. 9 Our detailed evaluation shows that
agents handle degraded conditions in different ways: final performances, expressing uncertainty, number of tool calls, reaction against degraded documents. Additionally, the
framework makes it possible to precisely measure the role of parametric knowledge in
search agents and to measure their interaction with contradictory evidence. Correct answers in this condition suggest that the agent is
able to aggregate evidence across multiple tool calls, even if some may yield contradictory
information. 3 Discussion and future directions
While our experiments allow finer-grained outcome evaluation of search agents faced
with controlled retrieval conditions, further qualitative analysis should complement this
evaluation, looking closer at chain-of-thought reasoning and search query formulations
themselves. 37 in RAS, indicating that it receives little
reliability-aware credit under degradation. Overall, RAS better separates robust non-answer
behavior from overconfident failure when document quality is low.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
DeepStress framework can be used to control three dimensions that affect document reliability: trustworthiness, relevance, and factuality by replacing search agents' retrieval module with a controlled synthetic environment. This allows for precise manipulation of these properties in a laboratory-style setting to stress test search agent behavior.
-
The Reliability-Aware Score (RAS) can be used as an aggregate metric to better distinguish between robust non-answer behavior and overconfident failure when document quality is low, as it gives
full credit to correct answers, no credit to incorrect or malformed answers, and partial credit to abstention or tool-budget exhaustion when reliable evidence is unlikely.
-
Agents can be analyzed based on their query strategies under degraded conditions; for instance, some agents might increase the use of exact-match operators when relevance is degraded to 0%, suggesting they adapt by making their queries more constrained.
Abstract
While search agents demonstrate impressive capabilities in multi-step question answering, their robustness to poor-quality evidence remains under-explored. Such evidence is rare in benchmark corpora, and resulting failures are hard to reproduce. We propose DeepStress, a dynamic stress-testing framework that subjects agents to degraded evidence at controlled rates. It replaces the retrieval module with a synthetic environment where documents are generated dynamically for each query, under independent control of their trustworthiness, relevance, and factuality. To characterize behavior under stress, we go beyond binary accuracy with a five-way classification of final-answer outcomes and two new reliability-aware metrics, RAS and CoPRAS, which credit cautious behavior and account for computational cost. Evaluating 12 systems on questions from HotpotQA and BrowseComp-Plus, we find that several open-weight agents answer confidently when sources contain obvious credibility red flags, and that many HotpotQA questions are answered from parametric knowledge even when all returned documents are factually wrong.
Sources
- DRAGged into Conflicts: Detecting and Addressing Conflicting Sources in Search-Augmented LLMs
- TRAIL: Trace Reasoning and Agentic Issue Localization
- Agentic Search in the Wild: Intents and Trajectory Dynamics from 14M+ Real Search Requests
- DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research
- ZeroSearch: Incentivize the Search Capability of LLMs without Searching
- BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data