DeepStress: Stress-Testing Deep Search Agents
summary
The gist
DeepStress proposes a simulation framework to stress-test deep search agents by controlling the frequency of challenging evidence through a controlled synthetic environment.
In short
DeepStress is a simulation framework designed to stress-test deep search agents by controlling how often they encounter challenging evidence. Researchers manipulate document quality across three axes: trustworthiness, relevance, and factuality. The study shows that agents react differently to unreliable information, affecting their final answers and how they express uncertainty.
Key concepts
- Trustworthiness (T)
- This measures how reliable or credible a piece of information is. In the simulation, documents are generated with varying levels of trustworthiness, ranging from highly credible evidence to intentionally untrustworthy sources. Agents must navigate these different credibility levels.
- Relevance (R)
- Relevance assesses how pertinent a document is to the specific search query being asked. The framework allows researchers to control this dimension, testing if agents can maintain focus when presented with documents that are either highly relevant or completely off-topic.
- Factuality (F)
- Factuality determines whether the information presented in a document is actually true. Documents can be made factually correct by grounding them in real data or intentionally made incorrect by prompting the LLM to introduce severe errors, allowing testing of agents' ability to detect falsehoods.
Terminology used across episodes
This episode discusses
- DeepStress: Stress-Testing Deep Search Agents · Paper Radio
- DRAGged into Conflicts: Detecting and Addressing Conflicting Sources in Search-Augmented LLMs
- TRAIL: Trace Reasoning and Agentic Issue Localization
- Agentic Search in the Wild: Intents and Trajectory Dynamics from 14M+ Real Search Requests
- DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research
- ZeroSearch: Incentivize the Search Capability of LLMs without Searching
- BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents
The paper
DeepStress: Stress-Testing Deep Search Agents · Read on arXiv
Orange Research · Aix-Marseille University, CNRS, LIS UMR 7020
While search agents demonstrate impressive capabilities in multi-step question answering, their robustness to poor-quality evidence remains under-explored. Such evidence is rare in benchmark corpora, and resulting failures are hard to reproduce. We propose DeepStress, a dynamic stress-testing framework that subjects agents to degraded evidence at controlled rates. It replaces the retrieval module with a synthetic environment where documents are generated dynamically for each query, under independent control of their trustworthiness, relevance, and factuality. To characterize behavior under stress, we go beyond binary accuracy with a five-way classification of final-answer outcomes and two new reliability-aware metrics, RAS and CoPRAS, which credit cautious behavior and account for computational cost. Evaluating 12 systems on questions from HotpotQA and BrowseComp-Plus, we find that several open-weight agents answer confidently when sources contain obvious credibility red flags, and that many HotpotQA questions are answered from parametric knowledge even when all returned documents are factually wrong.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "DeepStress: Stress-Testing Deep Search Agents".
Jane: Here's the gist of the paper. DeepStress proposes a simulation framework to stress-test deep search agents by controlling the frequency of challenging evidence through a controlled synthetic environment.
Tom: This framework allows researchers to precisely manipulate document reliability along three orthogonal axes: trustworthiness, relevance, and factuality.
Jane: By intercepting the agent’s search calls and replacing the retrieval module with this synthetic environment, DeepStress evaluates how search agents behave when faced with unreliable information.
Tom: DeepStress Framework and Dimensions The framework introduces a controlled simulation stress test environment that evaluates search agents by intercepting tool calls and returning synthetic documents modified along three axes: trustworthiness, relevance, and factuality.
Paper summary: Tom: So we’re looking at this paper called DeepStress: Stress-Testing Deep Search Agents. Basically, it’s about figuring out how much we can actually trust these deep search agents when the information they find is bad.
Jane: That sounds really practical, Tom. What’s the main idea here? What problem are they trying to solve with this framework?
Tom: They propose a simulation environment to stress-test these agents by controlling how often they see challenging evidence in a controlled setup. It lets researchers manipulate document reliability along three specific axes: trustworthiness, relevance, and factuality.
Lu: I think the core idea is that we need more than just testing on perfect data; we need to see how agents handle things that are deliberately unreliable because that’s where real-world problems happen.
Meng: So they aren't just giving them a fixed set of bad documents, but they’re controlling the probability of each type of unreliability for every single search call. That sounds like a lot of setup work to get right at scale.
Lalam: It sounds like they are intercepting the agent's search calls and replacing the actual retrieval module with this synthetic environment so they can test it under different conditions, DeepStress: Stress-Testing Deep Search Agents.
Tom: Exactly. They use this framework to see how agents behave when faced with information that’s degraded across these three dimensions, and they also introduce new metrics for measuring those outcomes.
Jane: The paper claims that testing several agents on things like HotpotQA and BrowseComp-Plus shows substantial differences in how they handle unreliable information, which is a big deal because it shows where the vulnerabilities are.
Lu: They introduce these final-answer outcomes like correct or incorrect, but they also have concepts like Abstention or Exceed tool budget, which gives a better picture of what’s actually happening when things go wrong.
Meng: And they have this Reliability-Aware Score that incorporates the document quality condition to give full credit to correct answers and partial credit for other behaviors when reliable evidence is unlikely. That feels like a way to properly weight the results.
Lalam: They also look at how agents change their search queries when they face degraded documents, seeing if they get more constrained or more source-aware, or if they adapt at all.
Tom: So it’s not just about the final answer being right or wrong; it’s about the whole process of searching and reasoning under these stressful conditions.
Jane: That leads into what they found: agents show substantial behavior differences depending on which dimension—trustworthiness, relevance, or factuality—they are facing degraded in.
Lu: For example, they find that when trustworthiness decreases for something like DR-Tulu, the agent becomes less reliable overall because the source is weaker than it used to be.
Meng: And GPT-based systems seem more likely to avoid giving a final answer entirely by either abstaining or exhausting their tool-call budget when facing these degraded documents. That suggests some models are more cautious than others under stress.
Lalam: The qualitative analysis also looked at the volume and content of the search queries produced by different systems, showing that some agents react by making their queries more constrained or more source-aware, while others show little adaptation at all.
Tom: It’s interesting to see that different agent architectures react in distinct ways when they encounter unreliable context; it suggests we might need to design agents with specific resilience mechanisms depending on the type of degradation they expect.
Jane: What this means for us is that if we want search agents to be robust in real-world applications, we can’t just test them on clean data anymore; we have to simulate this controlled degradation more rigorously.
Lu: The paper suggests that their framework makes it possible to precisely measure how much parametric knowledge an agent has and how it interacts with contradictory evidence.
Meng: If you’re trying to deploy this at scale, the cost of running this stress test simulation environment is something we’d need to consider, so it’s not just a theoretical exercise.
Lalam: They also suggest further qualitative analysis of chain-of-thought reasoning and search query formulations as future work because seeing that step is crucial for understanding the 'why' behind the behavior.
Tom: So, overall, DeepStress gives us a way to systematically test the robustness of deep search agents by controlling the exact nature of bad evidence they encounter.
Jane: And it points toward a future where we can better understand how these complex systems handle uncertainty and contradiction in information retrieval tasks.
Conclusion: Tom: So, we’re wrapping up our look at DeepStress, which is this paper about stress-testing deep search agents by controlling how much bad evidence they see in a simulated environment.
Jane: It sounds like they built a way to really push these agents to their limits instead of just testing them on perfect data.
Tom: Right, they’re manipulating three things—trustworthiness, relevance, and factuality—and sampling probabilities for each one for every single search call. It lets them precisely control the reliability of the documents feeding the agent.
Lu: What I find interesting is how they set up that synthetic environment to make sure it really tests those agent behaviors under degraded conditions.
Meng: So, instead of just giving them a bad document, they let you choose how much trustworthy or relevant that document is for each question. That’s a different kind of stress test than just throwing garbage at the system.
Lalam: And they have these specific metrics to measure the agent's response, like if it gives a final answer correctly or if it just quits early because the evidence was bad.
Jane: It seems like they found that agents react very differently depending on which quality—trustworthiness or factuality—is being degraded.
Tom: That’s what they showed; for example, when trustworthiness goes down, one agent starts failing much more often than another.
Jane: And the way the agents change their search queries when things are unreliable is also a key part of this study.
Lu: It suggests that some models adapt by searching smarter, while others just get stuck and stop trying altogether.
Meng: From a deployment standpoint, seeing how they react to these kinds of failures is important because it tells you what breaks at scale when the data isn't perfect.
Tom: So the main implication here is that we need better ways to understand how much knowledge an AI has and how it handles conflicting information in real-world scenarios.
Jane: It gives us a new tool, DeepStress, to measure that interaction precisely across different types of evidence degradation. (Sound of a short musical transition)
More episodes
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI