Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents".
Jane: Here's the gist of the paper. Large Language Model (LLM) code agents increasingly resolve repository-level issues by iteratively editing code, invoking tools, and validating candidate patches <ref:2602.07900#pg2> In these workflows, agents often write tests on the fly, but the value of this behavior remains unclear <ref:2602.07900#pg2> The results show that test writing is common, but resolved and unresolved tasks within the same model exhibit similar test-writing frequencies <ref:2602.07900#pg2> Based on these insights, prompt-induced changes in the volume of agent-written tests do not significantly change final outcomes in this setting <ref:2602.07900#pg2> These results suggest that current agent-written testing practices reshape process and cost more than final task outcomes <ref:2602.07900#pg2>.
Tom: Pruning redundant reasoning tokens The paper analyzes trajectories produced by six strong LLMs on SWE-bench Verified to understand the role of agent-written tests <ref:2602.07900#pg2> Test writing is common, but resolved and unresolved tasks within the same model exhibit similar test-writing frequencies <ref:2602.07900#pg2> When tests are written, they mainly serve as observational feedback channels, with value-revealing print statements appearing much more often than assertion-based checks <ref:2602.07900#pg2>.
Jane: The prompt-intervention study suggests that prompt-induced changes in the volume of agent-written tests do not significantly change final outcomes in this setting <ref:2602.07900#pg2>.
Tom: Taken together, these results suggest that current agent-written testing practices reshape process and cost more than final task outcomes <ref:2602.07900#pg2>.
Title and authors: Tom: The paper is called "Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents", and it explores whether these tests actually improve issue resolution or if they are just a familiar software development practice.
Jane: It seems to be focusing on this central idea: do these on-the-fly tests meaningfully improve how much the agent solves, or are they mostly just consuming interaction budget?
Lu: The authors are looking at this behavior across six different models, including Claude Opus four point five and DeepSeek v3 point 2 Reasoner, to see if this behavior is consistent or model dependent.
Meng: I'm curious about the models they picked; are they representative of what we actually run in production environments?
Lalam: The study uses a setup where testing can be optional at runtime, which helps isolate the base model's natural behavior before any test writing gets involved.
Tom: The paper summarizes their findings, and it points out that test writing and execution intensity are mostly process styles that depend on the specific model used rather than a direct measure of task success.
Jane: So, it seems like for some models, like GPT-five point two mentioned in the abstract, they write almost no new tests yet still get performance comparable to top agents <ref:2602.07900#pg1,almost no new tests yet>.
Lu: They also analyzed the feedback signals, finding that value-revealing prints consistently outnumber assertions across all models when looking at the macro average count per task.
Meng: Prints versus assertions... what does that mean for us in terms of what we should be looking at when we're debugging an agent's output?
Lalam: The paper shows that those value-revealing prints are mostly used for inspecting values and content, with exception or execution status signals being the second most common type of print.
Tom: The paper then looked at prompt interventions—changing whether the agent writes tests—and found that these changes don't significantly alter the final outcomes for most tasks on SWE-bench Verified.
Jane: That's a key finding, so even if we try to encourage or discourage test writing, it doesn't seem to flip the success rate for most issues.
Lu: But they did find that prompt changes can materially affect efficiency, which is where things get interesting; discouraging test writing can lead to large cost reductions.
Meng: Cost reductions are what matter to me; if we can cut down on API calls and tokens substantially by just stopping unnecessary testing, that’s a win for deployment.
Lalam: The paper found that for high-test-writing models, like Kimi K2 Thinking or DeepSeek v3 point 2 Reasoner, discouraging test writing resulted in input tokens dropping by forty-nine point zero percent and API calls dropping by thirty five point four percent <ref:2602.07900#pg1>.
Tom: So it seems like the main conclusion is that more agent-written tests don't automatically mean more solves; what they actually change is the process footprint—things like API calls and token usage, not necessarily the final result.
Jane: It suggests that we might need better ways to measure testing value than just looking at whether a task succeeded or failed.
Lu: They suggest that improving the value of testing might require better oracle signals or more actionable feedback, rather than just blindly telling the agent to write more tests.
Meng: If that's true, then we should look at how we can make those test results more useful for debugging, instead of just counting them as a measure of effort.
Lalam: I think the paper points toward self-evolving testing strategies where the agent revises its own prompts based on what it learns from failures and environment feedback.
Tom: That means we should be thinking about building closed-loop systems for testing, where the agent learns how to test better itself instead of just following a static set of instructions.
Jane: It shifts the focus from simply inducing test writing to creating a system that optimizes the testing process itself under cost and safety constraints.
Lu: And they highlight that agents can behave very differently in how often they write tests or how much budget they spend, even when the final resolution of the code change is small.
Meng: That means our deployment strategy needs to account for these different process footprints across different models.
Lalam: The study also found eight recurring issues that appear in both categories of test-sensitive candidates under those intervention runs, suggesting tests might be more helpful for smaller bugs with clear reproduction conditions.
The paper's summary: Tom: We've covered the setup and the initial results, and now we're getting into what the core findings really mean about agent-written tests in this context.
Jane: So, essentially, the paper argues that test writing is more of a model-dependent process style than a simple indicator of eventual success on its own.
Lu: They break down what those tests are actually doing by looking at assertions versus those value-revealing prints, and they found that value-revealing prints are the dominant signal.
Meng: That means when we look at agent behavior, we should prioritize understanding what the model is *showing* us about its internal state during execution rather than just whether it passed a check.
Lalam: The paper also looked at how changing test writing frequency affects outcomes and efficiency, and found that while the final outcome doesn't change much for most tasks, the cost changes are substantial.
Tom: Exactly. They show that encouraging or discouraging test writing only weakly aligns with task success in this setting, but it strongly influences the process footprint of API calls and tokens used.
Jane: It suggests that our current understanding might be too coarse if we only measure final task resolution as the main thing to look for.
The paper's improvements: Tom: Now let's talk about what they suggest we should do next, because the paper isn't just reporting results, it’s suggesting a direction for future work.
Lu: They suggest focusing on self-evolving test-generation strategies where the agent revises its own testing prompt or policy based on feedback and failure modes from the environment.
Meng: That sounds ambitious; we need to figure out how to build a system that lets the AI refine its own testing approach instead of just following a static handwritten instruction set.
Lalam: It's about formalizing closed-loop optimization under cost and safety constraints, meaning the agent learns what works best for *it* in terms of efficiency and resource usage.
Tom: So it's not about writing more tests, but designing a system where the agent learns to write the *right kind* of tests based on what it needs to learn from its failures.
Jane: That sounds like moving toward something where testing becomes an active part of the agent's learning process, rather than just a static step in its workflow.
Lu: And they point out that agents can vary wildly in their testing behavior, so we need to account for that variability when designing these self-evolving systems.
Conclusion: Tom: So to wrap up this paper on Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents, it seems like more tests don't directly equal more successful solves.
Jane: The main point is that we should stop viewing test writing as a guarantee of success and start looking at it as a way to understand the agent's internal process.
Lu: They conclude that improving testing value will likely come from giving the agents better ways to generate or interpret validation signals, not just demanding more tests.
Meng: From an engineering standpoint, this means we should monitor test writing and execution costs separately so teams can know when that extra testing effort stops adding actual value to the deployment.
Lalam: And they also highlight that for those who write a lot of tests, like Kimi K2 Thinking or DeepSeek v3 point 2 Reasoner, discouraging test writing actually yields substantial efficiency savings in tokens and API calls <ref:2602.07900#pg1>.
Tom: Exactly. The paper suggests we need better oracles and more actionable validation signals instead of just inducing agents to write more tests for the sake of it.
Jane: It sounds like testing needs to be studied as a process intervention, not just a final success rate difference when we look at this work on Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents.
Zhi Chen, Zhensu Sun, Yuling Shi, Chao Peng, Xiaodong Gu, David Lo, Lingxiao Jiang
Singapore Management University
cs.SE, cs.AI
Submitted: 2026-02-08
Updated: 2026-11-09
Code: https://github.com/aorwall/moatless-tools
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 83/100
The gist: Large Language Model (LLM) code agents increasingly resolve repository-level issues by iteratively editing code, invoking tools…
Key concepts
- Agent-Written Tests
- These are tests generated by the LLM agent while it is iteratively fixing code. The research found that these tests are often used for observation rather than definitive validation, meaning they reveal runtime values and execution status more frequently than they confirm a pass or fail condition.
- Process Footprint
- This refers to the measurable aspects of how the LLM agent interacts with the system, such as API calls and token consumption. The study indicates that test writing significantly alters this footprint—often decreasing costs—even when it doesn't improve the final solution correctness.
- Value-Revealing Prints
- These are print statements used by agents to inspect runtime values or coarse execution outcomes during testing. The analysis showed these prints are far more common than assertion-based checks, suggesting that agents rely more on observing what happens during execution than on formal pass/fail logic.
- Prompt-Induced Changes
- This refers to intentionally changing the instructions given to the agent regarding test writing. The research concluded that while prompt changes can flip test writing behavior at scale, they have only small observed effects on the final task outcome for most problems.
Terminology
Summary
Large Language Model (LLM) code agents increasingly resolve repository-level issues by iteratively editing code, invoking tools, and validating candidate patches <ref:2602. p. 0 sg
Lingxiao Jiang
Singapore Management University
Singapore, Singapore
lxjiang@smu. sg
Abstract
Large Language Model (LLM) code agents increasingly resolve
repository-level issues by iteratively editing code, invoking tools,
and validating candidate patches. sg
Abstract
Large Language Model (LLM) code agents increasingly resolve
repository-level issues by iteratively editing code, invoking tools,
and validating candidate patches. In these workflows, agents often
write tests on the fly, but the value of this behavior remains unclear. To better understand the role of agent-written tests, we analyze
trajectories produced by six strong LLMs on SWE-bench Verified. Our results show that test writing is common, but resolved and
unresolved tasks within the same model exhibit similar test-writing
frequencies. Based on
these insights, we perform a prompt-intervention study by revising
the prompts used with four models to either increase or reduce
test writing. The results suggest that prompt-induced changes in
the volume of agent-written tests do not significantly change final
outcomes in this setting. The results suggest that prompt-induced changes in
the volume of agent-written tests do not significantly change final
outcomes in this setting. Taken together, these results suggest that
current agent-written testing practices reshape process and cost
more than final task outcomes.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
The paper analyzes trajectories produced by six strong LLMs on SWE-bench Verified to understand the role of agent-written tests <ref:2602.07900#pg2> Test writing is common, but resolved and unresolved tasks within the same model exhibit similar test-writing frequencies <ref:2602. p. 0 To better understand the role of agent-written tests, we analyze
trajectories produced by six strong LLMs on SWE-bench Verified. Our results show that test writing is common, but resolved and
unresolved tasks within the same model exhibit similar test-writing
frequencies. Our results show that test writing is common, but resolved and
unresolved tasks within the same model exhibit similar test-writing
frequencies. When tests are written, they mainly serve as observational feedback channels, with value-revealing print statements
appearing much more often than assertion-based checks. Based on
these insights, we perform a prompt-intervention study by revising
the prompts used with four models to either increase or reduce
test writing. The results suggest that prompt-induced changes in
the volume of agent-written tests do not significantly change final
outcomes in this setting. The results suggest that prompt-induced changes in
the volume of agent-written tests do not significantly change final
outcomes in this setting. Taken together, these results suggest that
current agent-written testing practices reshape process and cost
more than final task outcomes.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
How the training signal is built
The study decomposes the problem into three complementary research questions: RQ1 examines testing behaviors, RQ2 analyzes feedback signals in agent-written tests, and RQ3 studies observed outcome and efficiency changes under test-writing interventions <ref:2602.07900#pg2>. The methodology uses mini-SWE-agent to isolate base-model behavior by making testing optional at runtime <ref:2602. p. 2 2 Agent and its LLMs
While many recent LLM-based agents incorporate curated testing components, such as specialized validation modules, dedicated
test-planning stages, or multi-agent coordination [10, 26, 41, 59],
these frameworks can confound a model’s intrinsic tendencies with
scaffold-induced constraints. To better isolate base-model behavior,
we adopt mini-SWE-agent [45, 46]. When the resulting signals are low-value, this overhead
may dilute the agent’s focus and become net detrimental. To better understand agent-written tests, we conduct a quantitative analysis of agent trajectories on SWE-bench Verified [34] using
mini-SWE-agent [45], where testing is optional and not enforced
by any hard-coded procedure. Agent-written testing is prevalent
for several strong models. For example, Claude Opus 4. 4%), gemini-3-
pro-preview [16] (74. 2%), gpt-5. 5, Gemini 3 Pro, GPT-5. 2, Kimi K2 Thinking, MiniMax M2, and DeepSeek v3. Agent-written tests can be beneficial by surfacing edge
cases and providing actionable feedback for fault localization and
patch refinement. However, they can also be harmful if they embed
incorrect assumptions or oracles, diverting effort toward satisfying
arXiv:2602.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the gains come from
RQ1 shows that agent-written testing is better understood as a model-dependent process behavior than as a simple marker of eventual success <ref:2602.07900#pg2> RQ3 shows that prompt-induced changes in test writing have only small observed effects on task outcomes for most tasks, but can materially change efficiency <ref:2602. p. 1 RQ2 shows that
agent-written tests primarily serve as an observational feedback
channel, with value-revealing prints dominating assert-based
checks. RQ3 shows that prompt-induced changes in test writing
have only small observed effects on task outcomes for most tasks,
but can materially change efficiency. We show that large flips in test-writing status are
accompanied by only small observed changes in resolution outcomes for most tasks, whereas efficiency changes can be substantial. Inducing tests can increase token and interaction overhead
without improving success, while suppressing tests yields large
cost reductions with only modest success drops.Where the paper says it
Where the paper says it
RQ2 shows that agent-written tests function primarily as an observational feedback channel: value-revealing prints dominate, and the assertions that do appear are concentrated in local-property and exact-value checks <ref:2602.07900#pg2>. Value-revealing prints consistently outnumber assertions in the macro-average counts per task across all models <ref:2602. p. 5 Across models, unresolved tasks tend to
show slightly higher total signal counts, driven mainly by more
value-revealing prints, while assertion counts remain comparatively
stable. 1 Test Signal Amount: Key Takeaway
When agents write tests, the feedback is mostly observational:
value-revealing prints consistently outnumber assertions, although total signal volume still varies markedly by model. Overall, value-revealing prints are used
mainly to inspect runtime values and coarse execution outcomes
rather than encode strong pass/fail criteria, reinforcing our interpretation of agent-written tests as observational debugging tools. 3 Print Categorization: Key Takeaway
Across models, most value-revealing prints are used for value/-
content inspection, with exception/execution-status signals as
a distant second.Where the paper says it
Where the paper says it
RQ3.2 Cost changes are larger than outcome changes <ref:2602. p. 8 These paired results therefore strengthen
the interpretation that suppressing test writing yields substantial
efficiency savings in these two high test-writing models, whereas
encouraging test writing produces model-dependent and much less
stable cost effects. 2: Cost changes are larger than outcome changes
Changing whether agents write tests has much larger effects
on efficiency than on task resolution. 0%
(49)
n=322
n=185
n=342
n=376
Unchanged (Stable) Fail Success Success Fail
Figure 3: Outcome-transition distribution on tasks with an
intended test-status change
tasks transition from No test to Has test, respectively. Conversely,
the discourage test writing prompt removes tests at scale for the
high test-writing models Kimi K2 Thinking and DeepSeek v3. 2 Reasoner: input tokens drop by
49. 9%, respectively, and Kimi K2 Thinking also reduces
API calls by 35. 8% output tokens)
without any gain in resolution. The paired analyses support this
increase across all three metrics: API calls rise by +1. 1 Outcome Analysis
(SWE-Bench Results)
RQ3. 2 Efficiency Analysis
(API Calls and Tokens)
Setup
RQ2. Agent-written tests can be beneficial by surfacing edge
cases and providing actionable feedback for fault localization and
patch refinement. However, they can also be harmful if they embed
incorrect assumptions or oracles, diverting effort toward satisfying
arXiv:2602.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The paper concludes that more agent-written tests do not mean more solves; what they more reliably change is the process footprint—API calls, token usage, and interaction patterns <ref:2602.07900#pg2>. Improving the value of testing for code agents may therefore require better oracles and more actionable validation signals, rather than simply inducing agents to write more tests <ref:2602. p. 8 For practitioners,
the goal should not be to simply make agents write more tests, but
to make testing more targeted and budget-aware. One lightweight
approach is to package reusable testing behaviors as focused Claude
Code subagents/skills [2, 3], such as generating one minimal regression test, strengthening a weak oracle by converting prints
into assertions, or running only the smallest relevant test slice. For benchmark users and maintainers, our study points to a narrower measurement concern. In its February 23, 2026 article “Why
SWE-bench Verified no longer measures frontier coding capabilities,”
OpenAI argued that SWE-bench Verified has become harder to
interpret because contamination and residual benchmark-design
issues weaken the meaning of final scores [36].Where the paper says it
Where the paper says it
Future work should focus on self-evolving test-generation strategies, where the agent revises its own testing prompt or policy from environment feedback and failure modes rather than following a static handwritten prompt <ref:2602.07900#pg2>. This could formalize closed-loop optimization under cost and safety constraints <ref:2602. p. 9 A promising
direction isself-evolving [19, 40, 51, 57] testing policies, in which the
agent revises its own testing prompt or policy from environment
feedback and failure modes rather than following a static handwritten prompt [14]. Future work can formalize this as closed-loop
optimization under cost and safety constraints, comparing humanspecified and self-adapted strategies under matched budgets and
controlled scaffolds. We do not evaluate
those benchmark-level claims directly, nor do we test contamination or training-data effects. Our evidence instead supports a more
limited point: even when final resolution changes little, agents can
still differ substantially in how often they write tests, how much
validation budget they spend, and how their software-engineering
behavior unfolds.Where the paper says it
Where the paper says it
The study finds 8 issues that appear in both categories of test-sensitive candidates under the intervention runs <ref:2602.07900#pg2>. These recurring themes suggest that tests may help more for small-to-moderate correctness bugs, often involving explicit reproduction conditions or edge-case triggers <ref:2602. p. 8 4 to help
summarize recurring themes. This exploratory pass suggests three
recurring properties: they are mostly small-to-moderate correctness
bugs, often involve explicit reproduction conditions or edge-case
triggers, and frequently require checking precise expected behavior
or semantics. 1: Outcomes vs. Test-Status Changes
Prompt interventions can flip test-writing status at scale, yet
most tasks keep the same final outcome. Test-Status Changes
Prompt interventions can flip test-writing status at scale, yet
most tasks keep the same final outcome. Even large induced or
suppressed shifts in test writing therefore appear to be a weak
lever on task resolution in this setting. It
is also useful to log test-writing, test-running, and failure-analysis
costs separately so teams can see when extra test-related interaction stops adding value. For researchers, the main implication is to
study testing as a process intervention, not only as a final success-rateWhere the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improve testing strategies by focusing on targeted validation rather than simply increasing test volume, as
More tests do not mean more solves; what they more reliably change is the process footprint—API calls, token usage, and interaction patterns.
Practitioners should adopt approaches likepackage reusable testing behaviors as focused Claude Code subagents/skills.
-
Enhance agent reliability by prioritizing value-revealing prints over assertion-based checks when interpreting test feedback, since
value-revealing prints consistently outnumber assertions in the macro-average counts per task
and they are usedmainly to inspect runtime values and coarse execution outcomes rather than encode strong pass/fail criteria.
-
Optimize efficiency by implementing a system that monitors test writing and execution costs separately so teams can identify when
extra test-related interaction stops adding value,
as interventions show thatDiscourage tests yields large cost reductions with only modest success drops
for high-writing models. -
Refine assertion usage by focusing on property checks and exact-value checks, as these dominate the feedback signals, since
assertions are dominated by property checks and exact-value checks, whereas relational or range-style constraints remain uncommon.
Abstract
Large Language Model (LLM) code agents increasingly resolve repository-level issues by iteratively editing code, invoking tools, and validating candidate patches. In these workflows, agents often write tests on the fly, but the value of this behavior remains unclear. This raises a central question: how does agent-written testing relate to task outcomes and resource use during repository-level issue resolution? To better understand the role of agent-written tests, we analyze 5,000 recorded agent runs produced by six strong LLMs on SWE-bench Verified using the lightweight mini-SWE-agent scaffold. Our results show that test writing is common for most models. GPT-5.2 is an exception: it rarely writes new test files while achieving a resolution rate close to the top-ranked model. Test-writing frequency varies widely across models. Within a given model, it differs less between resolved and unresolved tasks. When tests are written, their files contain substantially more statements that print runtime values than assertions that check conditions. We then compare baseline prompts with revised prompts that encourage or discourage writing new test files for four models. Agents start or stop writing test files as requested on 36.8%-93.2% of tasks. Resolved-task totals differ by 0-13 out of 500 per model, with no statistically significant difference, while API calls or token use differ substantially for three models. Similar resolution totals can therefore hide substantial differences in how agents test and how many resources they use.
Sources
- Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Study
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
- CodeMonkeys: Scaling Test-Time Compute for Software Engineering
- A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence
- Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling
- Towards a Science of Scaling Agent Systems
- An Empirical Study on Failures in Automated Issue Solving
- MarsCode Agent: AI-native Automated Bug Fixing
- EXPEREPAIR: Dual-Memory Enhanced LLM-based Repository-Level Program Repair
- SpecRover: Code Intent Extraction via LLMs
- OpenHands: An Open Platform for AI Software Developers as Generalist Agents
- Agentless: Demystifying LLM-based Software Engineering Agents
- Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?
- Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters