Writing Bug Reports for Software Repair Agents: What Information Matters Most?

summary

The gist

Software development is increasingly moving toward agentic-first workflows This paper studies what makes a bug report agent-ready Our results suggest that agentic-first reports benefit most from…

In short

The study investigated what makes bug reports useful for AI agents tasked with software repair. By analyzing 441 real bug reports, the research found that agent success depends less on traditional details like reproduction steps and more on information that directly guides the agent's search and repair process, such as localization cues and suggested fixes.

Key concepts

Agentic-first workflows
This refers to a new way of software development where AI agents are central to the process. Instead of humans just reading reports, these agents actively inspect code, reason about issues, modify code, and propose solutions based on task specifications provided in bug reports.
Information Types and Annotation
The authors manually categorized sentences in bug reports into specific types like observed behavior or suggested fixes. This detailed labeling allowed them to empirically test which specific pieces of information an agent actually uses to succeed or fail at its repair task.
Ablation Study Findings
This involved removing different information types from the reports and seeing how much the agent's performance dropped. The results showed that removing localization cues and suggested fixes together caused the biggest degradation in agent success, indicating these are crucial for guiding an agent's repair actions.
Repair Specifications
The paper argues that bug reports should evolve from being long descriptions into concise 'repair specifications.' These specifications tell an AI exactly what went wrong, where to focus its search, and what kind of change is needed, making them actionable instructions for the agent.

Terminology used across episodes

This episode discusses

The paper

Writing Bug Reports for Software Repair Agents: What Information Matters Most? · Read on arXiv

USI Universita della Svizzera italiana University of Sannio

Software development is increasingly moving toward agentic-first workflows. This includes AI agents responsible for generating initial fixes for submitted issue reports. In this setting, issue reports are no longer merely documentation for human maintainers; they become the primary task specification for the agent. However, little is known about how such reports should be written to maximize the agent's chances of producing a correct fix. We study what makes a bug report agent-ready. Starting from the SWE-Bench Verified benchmark (i.e. a collection of 500 real repository issues with human-written gold patches and test suites for evaluating generated fixes) we manually classify each issue by change type (e.g. bug fix vs refactoring) and annotate each sentence with its information type, such as observed behavior, expected behavior, reproduction steps, localization cues, and suggested fixes. We focus on the 441 issues representing bug reports, and we run on them mini-swe-agent using three LLM backbones (i.e. GPT-5-mini, MiniMax M2.5, and Gemini 3 Flash). We then fit a binomial regression model to estimate the incremental association between each information type and agent success, controlling for confounding factors. Our results suggest that agentic-first reports benefit most from information that narrows the agent's search and repair space. Suggested fixes, expressed either in code or natural language, show the strongest positive associations with pass probability, while localization cues show positive but non-significant associations. An ablation study removing selected information types confirms that agents benefit less from information traditionally useful to humans, such as reproduction steps, and more from sentences that expose a repair direction.

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Writing Bug Reports for Software Repair Agents".

Jane: Here's the gist of the paper. Software development is increasingly moving toward agentic-first workflows This paper studies what makes a bug report agent-ready Our results suggest that agentic-first reports benefit most from information that narrows the agent’s search and repair space Software development workflows are increasingly incorporating LLM-based agents that can inspect repositories, reason over issues, modify code, and propose patches In this setting, issue reports are no longer merely documentation for human maintainers; they become the primary task specification for the agent The practical question is no longer only what makes a bug report useful for humans but also what makes a bug report effective for agents Information Types and Annotation The authors manually classified each issue by change type and annotated each sentence with its information type, such as observed behavior, expected behavior, reproduction steps, localization cues, and suggested fixes They focused on the 441 issues representing bug reports from the SWE-bench Verified benchmark The taxonomy includes bug summary, observed behavior, expected behavior, reproduction steps, environment details, references to external resources (e.g., other issues, Stack Overflow discussions), localization information at different granularity (i.e., lines/functions/files likely affected by the bug), suggested fixes in code or natural language, and current workarounds Empirical Analysis of Agent Success They ran on them mini-swe-agent using three LLM backbones (i.e., GPT-5-mini, MiniMax M2.5, and Gemini 3 Flash) They then fit a binomial regression model to estimate the incremental association between each information type and agent success Their results suggest that agent-readiness is not simply traditional bug-report quality under a new name Ablation Study Findings An ablation study removing selected information types confirms that agents benefit less from information traditionally useful to humans, such as reproduction steps, and more from sentences that expose a repair direction, either through bug localization or a suggested fix The strongest degradation appears when localization cues and suggested fixes are removed together Implications for Agentic Development The implication for agentic-first development is therefore concrete: issue reports should be written less as exhaustive narratives and more as repair specifications that tell the agent what failed, where to start looking, and what kind of change may be needed Future research should explore multi-stage agent architectures in which dedicated localization or issue-understanding agents enrich reports before patch generation The per-LLM analyses show that different backbones do not exploit issue information in the same way Taken together, the observational analysis suggests that agent-ready reports are those that transform a bug report from a descriptive artifact into an actionable repair specification The results provide several lessons for agentic-first development where issue reports are not only read by developers but also serve as the initial task specification for repair agents Agentic workflows should be incremental, not one-shot Localization may need to be automatically derived Issue templates should be treated as agent-specific interfaces References should be summarized and properly contextualized, not merely linked References

Paper summary: Jane: So, the paper sets up this experiment by manually classifying every sentence in those bug reports into different info types. They looked at things like observed behavior, expected behavior, and reproduction steps >

Lu: They also included stuff like environment details and localization cues. The taxonomy they used was pretty detailed: bug summary, observed behavior, expected behavior, reference links to other issues or Stack Overflow discussions >

Meng: And they also included things that are super useful for us developers like suggested fixes in code or natural language and even current workarounds >

Tom: They ran this through a mini SWE-agent system on those four hundred forty-one reports using those three different LLM backbones and fitted a binomial regression model to see how much each information type actually helped the agent succeed > <ref:2607.09553#pg2>

Jane: The key finding is that localization cues, which are mentions of affected lines or functions in the code, were positively associated with successful patches >

Lu: But even stronger was the information about suggested fixes. This held true whether that fix was provided as a code snippet or just written in natural language >

Meng: They did an ablation study where they took out certain pieces of info to see what happened. Removing things like reproduction steps, which are super helpful for humans, didn't help the agent much >

Tom: The strongest drop in success came when they removed localization cues and suggested fixes at the same time >

Jane: So it really suggests that agent-readiness isn't just about having a nicely written report anymore. It’s about structuring the report so it tells the agent exactly where to start looking and what kind of change is needed >

Conclusion: Tom: The paper, 'Writing Bug Reports for Software Repair Agents: What Information Matters Most?', really shows us that we need to stop writing bug reports as long descriptions of a failure and start writing them as repair specifications >

Jane: Exactly. The authors found that reports need to transform from descriptive artifacts into actionable instructions for the agent, telling it what failed, where to begin searching, and what kind of change is expected >

Lu: What this means for the future is that we might need multi-stage agent architectures. We could have one agent dedicated just to understanding the issue and localizing it, which then hands off to another agent that proposes the actual patch >

Meng: From an engineering standpoint, this implies that we shouldn't treat every bug report as a single input prompt. It suggests reports should be incremental, maybe not one-shot fixes from start to finish >

Tom: And they also pointed out that localization cues might need to be derived automatically instead of being manually written in every report >

Jane: They also said that those reference links we use—to other issues or Stack Overflow discussions—should probably be summarized and contextualized properly, not just thrown in there as a link >

Lu: So the lesson here is that these reports are starting to act like interfaces for an agent. The goal is to make them clear enough for the AI to operate without needing a human intermediary step between every single piece of data >

More episodes

← Home