Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents

arXiv:2602.07900 · cs.SE, cs.AI · Submitted 2026-02-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents".

Jane: Here's the gist of the paper. Large Language Model (LLM) code agents increasingly resolve repository-level issues by iteratively editing code, invoking tools, and validating candidate patches <ref:2602.07900#pg2> In these workflows, agents often write tests on the fly, but the value of this behavior remains unclear <ref:2602.07900#pg2> The results show that test writing is common, but resolved and unresolved tasks within the same model exhibit similar test-writing frequencies <ref:2602.07900#pg2> Based on these insights, prompt-induced changes in the volume of agent-written tests do not significantly change final outcomes in this setting <ref:2602.07900#pg2> These results suggest that current agent-written testing practices reshape process and cost more than final task outcomes <ref:2602.07900#pg2>.

Tom: Pruning redundant reasoning tokens The paper analyzes trajectories produced by six strong LLMs on SWE-bench Verified to understand the role of agent-written tests <ref:2602.07900#pg2> Test writing is common, but resolved and unresolved tasks within the same model exhibit similar test-writing frequencies <ref:2602.07900#pg2> When tests are written, they mainly serve as observational feedback channels, with value-revealing print statements appearing much more often than assertion-based checks <ref:2602.07900#pg2>.

Jane: The prompt-intervention study suggests that prompt-induced changes in the volume of agent-written tests do not significantly change final outcomes in this setting <ref:2602.07900#pg2>.

Tom: Taken together, these results suggest that current agent-written testing practices reshape process and cost more than final task outcomes <ref:2602.07900#pg2>.

Title and authors: Tom: The paper is called "Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents", and it explores whether these tests actually improve issue resolution or if they are just a familiar software development practice.

Jane: It seems to be focusing on this central idea: do these on-the-fly tests meaningfully improve how much the agent solves, or are they mostly just consuming interaction budget?

Lu: The authors are looking at this behavior across six different models, including Claude Opus four point five and DeepSeek v3 point 2 Reasoner, to see if this behavior is consistent or model dependent.

Meng: I'm curious about the models they picked; are they representative of what we actually run in production environments?

Lalam: The study uses a setup where testing can be optional at runtime, which helps isolate the base model's natural behavior before any test writing gets involved.

Tom: The paper summarizes their findings, and it points out that test writing and execution intensity are mostly process styles that depend on the specific model used rather than a direct measure of task success.

Jane: So, it seems like for some models, like GPT-five point two mentioned in the abstract, they write almost no new tests yet still get performance comparable to top agents <ref:2602.07900#pg1,almost no new tests yet>.

Lu: They also analyzed the feedback signals, finding that value-revealing prints consistently outnumber assertions across all models when looking at the macro average count per task.

Meng: Prints versus assertions... what does that mean for us in terms of what we should be looking at when we're debugging an agent's output?

Lalam: The paper shows that those value-revealing prints are mostly used for inspecting values and content, with exception or execution status signals being the second most common type of print.

Tom: The paper then looked at prompt interventions—changing whether the agent writes tests—and found that these changes don't significantly alter the final outcomes for most tasks on SWE-bench Verified.

Jane: That's a key finding, so even if we try to encourage or discourage test writing, it doesn't seem to flip the success rate for most issues.

Lu: But they did find that prompt changes can materially affect efficiency, which is where things get interesting; discouraging test writing can lead to large cost reductions.

Meng: Cost reductions are what matter to me; if we can cut down on API calls and tokens substantially by just stopping unnecessary testing, that’s a win for deployment.

Lalam: The paper found that for high-test-writing models, like Kimi K2 Thinking or DeepSeek v3 point 2 Reasoner, discouraging test writing resulted in input tokens dropping by forty-nine point zero percent and API calls dropping by thirty five point four percent <ref:2602.07900#pg1>.

Tom: So it seems like the main conclusion is that more agent-written tests don't automatically mean more solves; what they actually change is the process footprint—things like API calls and token usage, not necessarily the final result.

Jane: It suggests that we might need better ways to measure testing value than just looking at whether a task succeeded or failed.

Lu: They suggest that improving the value of testing might require better oracle signals or more actionable feedback, rather than just blindly telling the agent to write more tests.

Meng: If that's true, then we should look at how we can make those test results more useful for debugging, instead of just counting them as a measure of effort.

Lalam: I think the paper points toward self-evolving testing strategies where the agent revises its own prompts based on what it learns from failures and environment feedback.

Tom: That means we should be thinking about building closed-loop systems for testing, where the agent learns how to test better itself instead of just following a static set of instructions.

Jane: It shifts the focus from simply inducing test writing to creating a system that optimizes the testing process itself under cost and safety constraints.

Lu: And they highlight that agents can behave very differently in how often they write tests or how much budget they spend, even when the final resolution of the code change is small.

Meng: That means our deployment strategy needs to account for these different process footprints across different models.

Lalam: The study also found eight recurring issues that appear in both categories of test-sensitive candidates under those intervention runs, suggesting tests might be more helpful for smaller bugs with clear reproduction conditions.

The paper's summary: Tom: We've covered the setup and the initial results, and now we're getting into what the core findings really mean about agent-written tests in this context.

Jane: So, essentially, the paper argues that test writing is more of a model-dependent process style than a simple indicator of eventual success on its own.

Lu: They break down what those tests are actually doing by looking at assertions versus those value-revealing prints, and they found that value-revealing prints are the dominant signal.

Meng: That means when we look at agent behavior, we should prioritize understanding what the model is *showing* us about its internal state during execution rather than just whether it passed a check.

Lalam: The paper also looked at how changing test writing frequency affects outcomes and efficiency, and found that while the final outcome doesn't change much for most tasks, the cost changes are substantial.

Tom: Exactly. They show that encouraging or discouraging test writing only weakly aligns with task success in this setting, but it strongly influences the process footprint of API calls and tokens used.

Jane: It suggests that our current understanding might be too coarse if we only measure final task resolution as the main thing to look for.

The paper's improvements: Tom: Now let's talk about what they suggest we should do next, because the paper isn't just reporting results, it’s suggesting a direction for future work.

Lu: They suggest focusing on self-evolving test-generation strategies where the agent revises its own testing prompt or policy based on feedback and failure modes from the environment.

Meng: That sounds ambitious; we need to figure out how to build a system that lets the AI refine its own testing approach instead of just following a static handwritten instruction set.

Lalam: It's about formalizing closed-loop optimization under cost and safety constraints, meaning the agent learns what works best for *it* in terms of efficiency and resource usage.

Tom: So it's not about writing more tests, but designing a system where the agent learns to write the *right kind* of tests based on what it needs to learn from its failures.

Jane: That sounds like moving toward something where testing becomes an active part of the agent's learning process, rather than just a static step in its workflow.

Lu: And they point out that agents can vary wildly in their testing behavior, so we need to account for that variability when designing these self-evolving systems.

Conclusion: Tom: So to wrap up this paper on Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents, it seems like more tests don't directly equal more successful solves.

Jane: The main point is that we should stop viewing test writing as a guarantee of success and start looking at it as a way to understand the agent's internal process.

Lu: They conclude that improving testing value will likely come from giving the agents better ways to generate or interpret validation signals, not just demanding more tests.

Meng: From an engineering standpoint, this means we should monitor test writing and execution costs separately so teams can know when that extra testing effort stops adding actual value to the deployment.

Lalam: And they also highlight that for those who write a lot of tests, like Kimi K2 Thinking or DeepSeek v3 point 2 Reasoner, discouraging test writing actually yields substantial efficiency savings in tokens and API calls <ref:2602.07900#pg1>.

Tom: Exactly. The paper suggests we need better oracles and more actionable validation signals instead of just inducing agents to write more tests for the sake of it.

Jane: It sounds like testing needs to be studied as a process intervention, not just a final success rate difference when we look at this work on Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents.

Zhi Chen, Zhensu Sun, Yuling Shi, Chao Peng, Xiaodong Gu, David Lo, Lingxiao Jiang

Singapore Management University

cs.SE, cs.AI

Submitted: 2026-02-08

Updated: 2026-11-09

Code: https://github.com/aorwall/moatless-tools

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: Large Language Model (LLM) code agents increasingly resolve repository-level issues by iteratively editing code, invoking tools…

Key concepts

Agent-Written Tests
These are tests generated by the LLM agent while it is iteratively fixing code. The research found that these tests are often used for observation rather than definitive validation, meaning they reveal runtime values and execution status more frequently than they confirm a pass or fail condition.
Process Footprint
This refers to the measurable aspects of how the LLM agent interacts with the system, such as API calls and token consumption. The study indicates that test writing significantly alters this footprint—often decreasing costs—even when it doesn't improve the final solution correctness.
Value-Revealing Prints
These are print statements used by agents to inspect runtime values or coarse execution outcomes during testing. The analysis showed these prints are far more common than assertion-based checks, suggesting that agents rely more on observing what happens during execution than on formal pass/fail logic.
Prompt-Induced Changes
This refers to intentionally changing the instructions given to the agent regarding test writing. The research concluded that while prompt changes can flip test writing behavior at scale, they have only small observed effects on the final task outcome for most problems.

Terminology

Summary

Large Language Model (LLM) code agents increasingly resolve repository-level issues by iteratively editing code, invoking tools, and validating candidate patches <ref:2602. p. 0

Where the paper says it

sg Lingxiao Jiang Singapore Management University Singapore, Singapore lxjiang@smu. sg Abstract Large Language Model (LLM) code agents increasingly resolve repository-level issues by iteratively editing code, invoking tools, and validating candidate patches.

07900#pg2> In these workflows, agents often write tests on the fly, but the value of this behavior remains unclear <ref:2602. p. 0
Where the paper says it

sg Abstract Large Language Model (LLM) code agents increasingly resolve repository-level issues by iteratively editing code, invoking tools, and validating candidate patches. In these workflows, agents often write tests on the fly, but the value of this behavior remains unclear.

07900#pg2> The results show that test writing is common, but resolved and unresolved tasks within the same model exhibit similar test-writing frequencies <ref:2602. p. 0
Where the paper says it

To better understand the role of agent-written tests, we analyze trajectories produced by six strong LLMs on SWE-bench Verified. Our results show that test writing is common, but resolved and unresolved tasks within the same model exhibit similar test-writing frequencies.

07900#pg2> Based on these insights, prompt-induced changes in the volume of agent-written tests do not significantly change final outcomes in this setting <ref:2602. p. 0
Where the paper says it

Based on these insights, we perform a prompt-intervention study by revising the prompts used with four models to either increase or reduce test writing. The results suggest that prompt-induced changes in the volume of agent-written tests do not significantly change final outcomes in this setting.

07900#pg2> These results suggest that current agent-written testing practices reshape process and cost more than final task outcomes <ref:2602. p. 0
Where the paper says it

The results suggest that prompt-induced changes in the volume of agent-written tests do not significantly change final outcomes in this setting. Taken together, these results suggest that current agent-written testing practices reshape process and cost more than final task outcomes.

07900#pg2>.

Pruning redundant reasoning tokens

The paper analyzes trajectories produced by six strong LLMs on SWE-bench Verified to understand the role of agent-written tests <ref:2602.07900#pg2> Test writing is common, but resolved and unresolved tasks within the same model exhibit similar test-writing frequencies <ref:2602. p. 0

Where the paper says it

To better understand the role of agent-written tests, we analyze trajectories produced by six strong LLMs on SWE-bench Verified. Our results show that test writing is common, but resolved and unresolved tasks within the same model exhibit similar test-writing frequencies.

07900#pg2> When tests are written, they mainly serve as observational feedback channels, with value-revealing print statements appearing much more often than assertion-based checks <ref:2602. p. 0
Where the paper says it

Our results show that test writing is common, but resolved and unresolved tasks within the same model exhibit similar test-writing frequencies. When tests are written, they mainly serve as observational feedback channels, with value-revealing print statements appearing much more often than assertion-based checks.

07900#pg2>. The prompt-intervention study suggests that prompt-induced changes in the volume of agent-written tests do not significantly change final outcomes in this setting <ref:2602. p. 0
Where the paper says it

Based on these insights, we perform a prompt-intervention study by revising the prompts used with four models to either increase or reduce test writing. The results suggest that prompt-induced changes in the volume of agent-written tests do not significantly change final outcomes in this setting.

07900#pg2>. Taken together, these results suggest that current agent-written testing practices reshape process and cost more than final task outcomes <ref:2602. p. 0
Where the paper says it

The results suggest that prompt-induced changes in the volume of agent-written tests do not significantly change final outcomes in this setting. Taken together, these results suggest that current agent-written testing practices reshape process and cost more than final task outcomes.

07900#pg2>.

How the training signal is built

The study decomposes the problem into three complementary research questions: RQ1 examines testing behaviors, RQ2 analyzes feedback signals in agent-written tests, and RQ3 studies observed outcome and efficiency changes under test-writing interventions <ref:2602.07900#pg2>. The methodology uses mini-SWE-agent to isolate base-model behavior by making testing optional at runtime <ref:2602. p. 2

Where the paper says it

2 Agent and its LLMs While many recent LLM-based agents incorporate curated testing components, such as specialized validation modules, dedicated test-planning stages, or multi-agent coordination [10, 26, 41, 59], these frameworks can confound a model’s intrinsic tendencies with scaffold-induced constraints. To better isolate base-model behavior, we adopt mini-SWE-agent [45, 46].

07900#pg2> A quantitative analysis of agent trajectories on SWE-bench Verified was conducted using mini-SWE-agent <ref:2602. p. 1
Where the paper says it

When the resulting signals are low-value, this overhead may dilute the agent’s focus and become net detrimental. To better understand agent-written tests, we conduct a quantitative analysis of agent trajectories on SWE-bench Verified [34] using mini-SWE-agent [45], where testing is optional and not enforced by any hard-coded procedure.

07900#pg2>. The models selected are Claude Opus 4. p. 1
Where the paper says it

Agent-written testing is prevalent for several strong models. For example, Claude Opus 4.

5, Gemini 3 Pro, GPT-5. p. 2
Where the paper says it

4%), gemini-3- pro-preview [16] (74. 2%), gpt-5.

2, Kimi K2 Thinking, MiniMax M2, and DeepSeek v3. p. 2
Where the paper says it

5, Gemini 3 Pro, GPT-5. 2, Kimi K2 Thinking, MiniMax M2, and DeepSeek v3.

2 Reasoner <ref:2602. p. 0
Where the paper says it

Agent-written tests can be beneficial by surfacing edge cases and providing actionable feedback for fault localization and patch refinement. However, they can also be harmful if they embed incorrect assumptions or oracles, diverting effort toward satisfying arXiv:2602.

07900#pg2>.

Where the gains come from

RQ1 shows that agent-written testing is better understood as a model-dependent process behavior than as a simple marker of eventual success <ref:2602.07900#pg2> RQ3 shows that prompt-induced changes in test writing have only small observed effects on task outcomes for most tasks, but can materially change efficiency <ref:2602. p. 1

Where the paper says it

RQ2 shows that agent-written tests primarily serve as an observational feedback channel, with value-revealing prints dominating assert-based checks. RQ3 shows that prompt-induced changes in test writing have only small observed effects on task outcomes for most tasks, but can materially change efficiency.

07900#pg2>. Specifically, discouraging test writing yields large cost reductions with only modest success drops for high test-writing models <ref:2602. p. 2
Where the paper says it

We show that large flips in test-writing status are accompanied by only small observed changes in resolution outcomes for most tasks, whereas efficiency changes can be substantial. Inducing tests can increase token and interaction overhead without improving success, while suppressing tests yields large cost reductions with only modest success drops.

07900#pg2>. The findings suggest that agent-written tests often behave more like a habitual software-development routine than a dependable source of validation in this setting <ref:2602.07900#pg2>.

RQ2 shows that agent-written tests function primarily as an observational feedback channel: value-revealing prints dominate, and the assertions that do appear are concentrated in local-property and exact-value checks <ref:2602.07900#pg2>. Value-revealing prints consistently outnumber assertions in the macro-average counts per task across all models <ref:2602. p. 5

Where the paper says it

Across models, unresolved tasks tend to show slightly higher total signal counts, driven mainly by more value-revealing prints, while assertion counts remain comparatively stable. 1 Test Signal Amount: Key Takeaway When agents write tests, the feedback is mostly observational: value-revealing prints consistently outnumber assertions, although total signal volume still varies markedly by model.

07900#pg2>. Across models, most value-revealing prints are used for value/content inspection, with exception/execution-status signals as a distant second <ref:2602. p. 6
Where the paper says it

Overall, value-revealing prints are used mainly to inspect runtime values and coarse execution outcomes rather than encode strong pass/fail criteria, reinforcing our interpretation of agent-written tests as observational debugging tools. 3 Print Categorization: Key Takeaway Across models, most value-revealing prints are used for value/- content inspection, with exception/execution-status signals as a distant second.

07900#pg2>.

RQ3.2 Cost changes are larger than outcome changes <ref:2602. p. 8

Where the paper says it

These paired results therefore strengthen the interpretation that suppressing test writing yields substantial efficiency savings in these two high test-writing models, whereas encouraging test writing produces model-dependent and much less stable cost effects. 2: Cost changes are larger than outcome changes Changing whether agents write tests has much larger effects on efficiency than on task resolution.

07900#pg2> Discouraging test writing for high test-writing models like Kimi K2 Thinking and DeepSeek v3. p. 7
Where the paper says it

0% (49) n=322 n=185 n=342 n=376 Unchanged (Stable) Fail Success Success Fail Figure 3: Outcome-transition distribution on tasks with an intended test-status change tasks transition from No test to Has test, respectively. Conversely, the discourage test writing prompt removes tests at scale for the high test-writing models Kimi K2 Thinking and DeepSeek v3.

2 Reasoner yields substantial efficiency savings in input tokens and API calls <ref:2602. p. 8
Where the paper says it

2 Reasoner: input tokens drop by 49. 9%, respectively, and Kimi K2 Thinking also reduces API calls by 35.

07900#pg2>. The paired analyses support this, showing that input tokens drop by 49. p. 8
Where the paper says it

8% output tokens) without any gain in resolution. The paired analyses support this increase across all three metrics: API calls rise by +1.

0% and API calls by 35. p. 1
Where the paper says it

1 Outcome Analysis (SWE-Bench Results) RQ3. 2 Efficiency Analysis (API Calls and Tokens) Setup RQ2.

4% for these models under the discourage setting <ref:2602. p. 0
Where the paper says it

Agent-written tests can be beneficial by surfacing edge cases and providing actionable feedback for fault localization and patch refinement. However, they can also be harmful if they embed incorrect assumptions or oracles, diverting effort toward satisfying arXiv:2602.

07900#pg2>.

The paper concludes that more agent-written tests do not mean more solves; what they more reliably change is the process footprint—API calls, token usage, and interaction patterns <ref:2602.07900#pg2>. Improving the value of testing for code agents may therefore require better oracles and more actionable validation signals, rather than simply inducing agents to write more tests <ref:2602. p. 8

Where the paper says it

For practitioners, the goal should not be to simply make agents write more tests, but to make testing more targeted and budget-aware. One lightweight approach is to package reusable testing behaviors as focused Claude Code subagents/skills [2, 3], such as generating one minimal regression test, strengthening a weak oracle by converting prints into assertions, or running only the smallest relevant test slice.

07900#pg2>. The study points to a narrower measurement concern for benchmark users, suggesting that final solve rates on SWE-bench Verified are too coarse as a standalone measure of agent behavior <ref:2602. p. 9
Where the paper says it

For benchmark users and maintainers, our study points to a narrower measurement concern. In its February 23, 2026 article “Why SWE-bench Verified no longer measures frontier coding capabilities,” OpenAI argued that SWE-bench Verified has become harder to interpret because contamination and residual benchmark-design issues weaken the meaning of final scores [36].

07900#pg2>.

Future work should focus on self-evolving test-generation strategies, where the agent revises its own testing prompt or policy from environment feedback and failure modes rather than following a static handwritten prompt <ref:2602.07900#pg2>. This could formalize closed-loop optimization under cost and safety constraints <ref:2602. p. 9

Where the paper says it

A promising direction isself-evolving [19, 40, 51, 57] testing policies, in which the agent revises its own testing prompt or policy from environment feedback and failure modes rather than following a static handwritten prompt [14]. Future work can formalize this as closed-loop optimization under cost and safety constraints, comparing humanspecified and self-adapted strategies under matched budgets and controlled scaffolds.

07900#pg2>. The paper emphasizes that agents may differ substantially in how often they write tests, how much validation budget they spend, and how their software-engineering behavior unfolds even when final resolution changes little <ref:2602. p. 9
Where the paper says it

We do not evaluate those benchmark-level claims directly, nor do we test contamination or training-data effects. Our evidence instead supports a more limited point: even when final resolution changes little, agents can still differ substantially in how often they write tests, how much validation budget they spend, and how their software-engineering behavior unfolds.

07900#pg2>.

The study finds 8 issues that appear in both categories of test-sensitive candidates under the intervention runs <ref:2602.07900#pg2>. These recurring themes suggest that tests may help more for small-to-moderate correctness bugs, often involving explicit reproduction conditions or edge-case triggers <ref:2602. p. 8

Where the paper says it

4 to help summarize recurring themes. This exploratory pass suggests three recurring properties: they are mostly small-to-moderate correctness bugs, often involve explicit reproduction conditions or edge-case triggers, and frequently require checking precise expected behavior or semantics.

07900#pg2>. The overall conclusion is that prompt interventions can flip test-writing status at scale, yet most tasks keep the same final outcome <ref:2602. p. 8
Where the paper says it

1: Outcomes vs. Test-Status Changes Prompt interventions can flip test-writing status at scale, yet most tasks keep the same final outcome.

07900#pg2>. Even large induced or suppressed shifts in test writing therefore appear to be a weak lever on task resolution in this setting <ref:2602. p. 8
Where the paper says it

Test-Status Changes Prompt interventions can flip test-writing status at scale, yet most tasks keep the same final outcome. Even large induced or suppressed shifts in test writing therefore appear to be a weak lever on task resolution in this setting.

07900#pg2>. The paper suggests that testing should be studied as a process intervention, not only as a final success-rate difference <ref:2602. p. 8
Where the paper says it

It is also useful to log test-writing, test-running, and failure-analysis costs separately so teams can see when extra test-related interaction stops adding value. For researchers, the main implication is to study testing as a process intervention, not only as a final success-rate

07900#pg2>.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improve testing strategies by focusing on targeted validation rather than simply increasing test volume, as More tests do not mean more solves; what they more reliably change is the process footprint—API calls, token usage, and interaction patterns. Practitioners should adopt approaches like package reusable testing behaviors as focused Claude Code subagents/skills.

  2. Enhance agent reliability by prioritizing value-revealing prints over assertion-based checks when interpreting test feedback, since value-revealing prints consistently outnumber assertions in the macro-average counts per task and they are used mainly to inspect runtime values and coarse execution outcomes rather than encode strong pass/fail criteria.

  3. Optimize efficiency by implementing a system that monitors test writing and execution costs separately so teams can identify when extra test-related interaction stops adding value, as interventions show that Discourage tests yields large cost reductions with only modest success drops for high-writing models.

  4. Refine assertion usage by focusing on property checks and exact-value checks, as these dominate the feedback signals, since assertions are dominated by property checks and exact-value checks, whereas relational or range-style constraints remain uncommon.

Abstract

Large Language Model (LLM) code agents increasingly resolve repository-level issues by iteratively editing code, invoking tools, and validating candidate patches. In these workflows, agents often write tests on the fly, but the value of this behavior remains unclear. This raises a central question: how does agent-written testing relate to task outcomes and resource use during repository-level issue resolution? To better understand the role of agent-written tests, we analyze 5,000 recorded agent runs produced by six strong LLMs on SWE-bench Verified using the lightweight mini-SWE-agent scaffold. Our results show that test writing is common for most models. GPT-5.2 is an exception: it rarely writes new test files while achieving a resolution rate close to the top-ranked model. Test-writing frequency varies widely across models. Within a given model, it differs less between resolved and unresolved tasks. When tests are written, their files contain substantially more statements that print runtime values than assertions that check conditions. We then compare baseline prompts with revised prompts that encourage or discourage writing new test files for four models. Agents start or stop writing test files as requested on 36.8%-93.2% of tasks. Resolved-task totals differ by 0-13 out of 500 per model, with no statistically significant difference, while API calls or token use differ substantially for three models. Similar resolution totals can therefore hide substantial differences in how agents test and how many resources they use.

Sources

Related papers