Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents

summary

The gist

Large Language Model (LLM) code agents increasingly resolve repository-level issues by iteratively editing code, invoking tools…

In short

The study analyzed how LLM agents use tests during code solving to determine if writing more tests actually improves final task outcomes. Findings show that test writing primarily reshapes the process footprint—like token usage and API calls—rather than significantly changing success rates. Tests mainly serve as observational feedback channels, with value-revealing prints being more common than strict assertions.

Key concepts

Agent-Written Tests
These are tests generated by the LLM agent while it is iteratively fixing code. The research found that these tests are often used for observation rather than definitive validation, meaning they reveal runtime values and execution status more frequently than they confirm a pass or fail condition.
Process Footprint
This refers to the measurable aspects of how the LLM agent interacts with the system, such as API calls and token consumption. The study indicates that test writing significantly alters this footprint—often decreasing costs—even when it doesn't improve the final solution correctness.
Value-Revealing Prints
These are print statements used by agents to inspect runtime values or coarse execution outcomes during testing. The analysis showed these prints are far more common than assertion-based checks, suggesting that agents rely more on observing what happens during execution than on formal pass/fail logic.
Prompt-Induced Changes
This refers to intentionally changing the instructions given to the agent regarding test writing. The research concluded that while prompt changes can flip test writing behavior at scale, they have only small observed effects on the final task outcome for most problems.

Terminology used across episodes

This episode discusses

The paper

Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents · Read on arXiv

Zhi Chen, Zhensu Sun, Yuling Shi, Chao Peng, Xiaodong Gu, David Lo, Lingxiao Jiang

Singapore Management University

Large Language Model (LLM) code agents increasingly resolve repository-level issues by iteratively editing code, invoking tools, and validating candidate patches. In these workflows, agents often write tests on the fly, but the value of this behavior remains unclear. This raises a central question: how does agent-written testing relate to task outcomes and resource use during repository-level issue resolution? To better understand the role of agent-written tests, we analyze 5,000 recorded agent runs produced by six strong LLMs on SWE-bench Verified using the lightweight mini-SWE-agent scaffold. Our results show that test writing is common for most models. GPT-5.2 is an exception: it rarely writes new test files while achieving a resolution rate close to the top-ranked model. Test-writing frequency varies widely across models. Within a given model, it differs less between resolved and unresolved tasks. When tests are written, their files contain substantially more statements that print runtime values than assertions that check conditions. We then compare baseline prompts with revised prompts that encourage or discourage writing new test files for four models. Agents start or stop writing test files as requested on 36.8%-93.2% of tasks. Resolved-task totals differ by 0-13 out of 500 per model, with no statistically significant difference, while API calls or token use differ substantially for three models. Similar resolution totals can therefore hide substantial differences in how agents test and how many resources they use.

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents".

Jane: Here's the gist of the paper. Large Language Model (LLM) code agents increasingly resolve repository-level issues by iteratively editing code, invoking tools, and validating candidate patches <ref:2602.07900#pg2> In these workflows, agents often write tests on the fly, but the value of this behavior remains unclear <ref:2602.07900#pg2> The results show that test writing is common, but resolved and unresolved tasks within the same model exhibit similar test-writing frequencies <ref:2602.07900#pg2> Based on these insights, prompt-induced changes in the volume of agent-written tests do not significantly change final outcomes in this setting <ref:2602.07900#pg2> These results suggest that current agent-written testing practices reshape process and cost more than final task outcomes <ref:2602.07900#pg2>.

Tom: Pruning redundant reasoning tokens The paper analyzes trajectories produced by six strong LLMs on SWE-bench Verified to understand the role of agent-written tests <ref:2602.07900#pg2> Test writing is common, but resolved and unresolved tasks within the same model exhibit similar test-writing frequencies <ref:2602.07900#pg2> When tests are written, they mainly serve as observational feedback channels, with value-revealing print statements appearing much more often than assertion-based checks <ref:2602.07900#pg2>.

Jane: The prompt-intervention study suggests that prompt-induced changes in the volume of agent-written tests do not significantly change final outcomes in this setting <ref:2602.07900#pg2>.

Tom: Taken together, these results suggest that current agent-written testing practices reshape process and cost more than final task outcomes <ref:2602.07900#pg2>.

Title and authors: Tom: The paper is called "Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents", and it explores whether these tests actually improve issue resolution or if they are just a familiar software development practice.

Jane: It seems to be focusing on this central idea: do these on-the-fly tests meaningfully improve how much the agent solves, or are they mostly just consuming interaction budget?

Lu: The authors are looking at this behavior across six different models, including Claude Opus four point five and DeepSeek v3 point 2 Reasoner, to see if this behavior is consistent or model dependent.

Meng: I'm curious about the models they picked; are they representative of what we actually run in production environments?

Lalam: The study uses a setup where testing can be optional at runtime, which helps isolate the base model's natural behavior before any test writing gets involved.

Tom: The paper summarizes their findings, and it points out that test writing and execution intensity are mostly process styles that depend on the specific model used rather than a direct measure of task success.

Jane: So, it seems like for some models, like GPT-five point two mentioned in the abstract, they write almost no new tests yet still get performance comparable to top agents <ref:2602.07900#pg1,almost no new tests yet>.

Lu: They also analyzed the feedback signals, finding that value-revealing prints consistently outnumber assertions across all models when looking at the macro average count per task.

Meng: Prints versus assertions... what does that mean for us in terms of what we should be looking at when we're debugging an agent's output?

Lalam: The paper shows that those value-revealing prints are mostly used for inspecting values and content, with exception or execution status signals being the second most common type of print.

Tom: The paper then looked at prompt interventions—changing whether the agent writes tests—and found that these changes don't significantly alter the final outcomes for most tasks on SWE-bench Verified.

Jane: That's a key finding, so even if we try to encourage or discourage test writing, it doesn't seem to flip the success rate for most issues.

Lu: But they did find that prompt changes can materially affect efficiency, which is where things get interesting; discouraging test writing can lead to large cost reductions.

Meng: Cost reductions are what matter to me; if we can cut down on API calls and tokens substantially by just stopping unnecessary testing, that’s a win for deployment.

Lalam: The paper found that for high-test-writing models, like Kimi K2 Thinking or DeepSeek v3 point 2 Reasoner, discouraging test writing resulted in input tokens dropping by forty-nine point zero percent and API calls dropping by thirty five point four percent <ref:2602.07900#pg1>.

Tom: So it seems like the main conclusion is that more agent-written tests don't automatically mean more solves; what they actually change is the process footprint—things like API calls and token usage, not necessarily the final result.

Jane: It suggests that we might need better ways to measure testing value than just looking at whether a task succeeded or failed.

Lu: They suggest that improving the value of testing might require better oracle signals or more actionable feedback, rather than just blindly telling the agent to write more tests.

Meng: If that's true, then we should look at how we can make those test results more useful for debugging, instead of just counting them as a measure of effort.

Lalam: I think the paper points toward self-evolving testing strategies where the agent revises its own prompts based on what it learns from failures and environment feedback.

Tom: That means we should be thinking about building closed-loop systems for testing, where the agent learns how to test better itself instead of just following a static set of instructions.

Jane: It shifts the focus from simply inducing test writing to creating a system that optimizes the testing process itself under cost and safety constraints.

Lu: And they highlight that agents can behave very differently in how often they write tests or how much budget they spend, even when the final resolution of the code change is small.

Meng: That means our deployment strategy needs to account for these different process footprints across different models.

Lalam: The study also found eight recurring issues that appear in both categories of test-sensitive candidates under those intervention runs, suggesting tests might be more helpful for smaller bugs with clear reproduction conditions.

The paper's summary: Tom: We've covered the setup and the initial results, and now we're getting into what the core findings really mean about agent-written tests in this context.

Jane: So, essentially, the paper argues that test writing is more of a model-dependent process style than a simple indicator of eventual success on its own.

Lu: They break down what those tests are actually doing by looking at assertions versus those value-revealing prints, and they found that value-revealing prints are the dominant signal.

Meng: That means when we look at agent behavior, we should prioritize understanding what the model is *showing* us about its internal state during execution rather than just whether it passed a check.

Lalam: The paper also looked at how changing test writing frequency affects outcomes and efficiency, and found that while the final outcome doesn't change much for most tasks, the cost changes are substantial.

Tom: Exactly. They show that encouraging or discouraging test writing only weakly aligns with task success in this setting, but it strongly influences the process footprint of API calls and tokens used.

Jane: It suggests that our current understanding might be too coarse if we only measure final task resolution as the main thing to look for.

The paper's improvements: Tom: Now let's talk about what they suggest we should do next, because the paper isn't just reporting results, it’s suggesting a direction for future work.

Lu: They suggest focusing on self-evolving test-generation strategies where the agent revises its own testing prompt or policy based on feedback and failure modes from the environment.

Meng: That sounds ambitious; we need to figure out how to build a system that lets the AI refine its own testing approach instead of just following a static handwritten instruction set.

Lalam: It's about formalizing closed-loop optimization under cost and safety constraints, meaning the agent learns what works best for *it* in terms of efficiency and resource usage.

Tom: So it's not about writing more tests, but designing a system where the agent learns to write the *right kind* of tests based on what it needs to learn from its failures.

Jane: That sounds like moving toward something where testing becomes an active part of the agent's learning process, rather than just a static step in its workflow.

Lu: And they point out that agents can vary wildly in their testing behavior, so we need to account for that variability when designing these self-evolving systems.

Conclusion: Tom: So to wrap up this paper on Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents, it seems like more tests don't directly equal more successful solves.

Jane: The main point is that we should stop viewing test writing as a guarantee of success and start looking at it as a way to understand the agent's internal process.

Lu: They conclude that improving testing value will likely come from giving the agents better ways to generate or interpret validation signals, not just demanding more tests.

Meng: From an engineering standpoint, this means we should monitor test writing and execution costs separately so teams can know when that extra testing effort stops adding actual value to the deployment.

Lalam: And they also highlight that for those who write a lot of tests, like Kimi K2 Thinking or DeepSeek v3 point 2 Reasoner, discouraging test writing actually yields substantial efficiency savings in tokens and API calls <ref:2602.07900#pg1>.

Tom: Exactly. The paper suggests we need better oracles and more actionable validation signals instead of just inducing agents to write more tests for the sake of it.

Jane: It sounds like testing needs to be studied as a process intervention, not just a final success rate difference when we look at this work on Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents.

More episodes

← Home