BRQGen: Towards Adaptive Completeness Assessment and Follow-Up Question Generation for Bug Reports
summary
The gist
BRQGen proposes an iterative framework to automatically assess bug report completeness and generate context-aware follow-up questions for missing details…
In short
BRQGen is an iterative framework that automatically assesses bug report completeness and generates context-aware follow-up questions up to five times. It uses a single LLM call to evaluate gaps, check answers, and generate new questions in a loop. This aims to reduce the delay before authors provide missing details by intelligently prompting them.
Key concepts
- Iterative Framework
- BRQGen operates in a loop where one LLM call performs three distinct roles: evaluating the report for missing fields, assessing if an answer is enough, and generating new questions. This cycle repeats until all critical information is found or a limit is reached, allowing for deeper gap identification over multiple turns.
- Bug Report Evaluator
- This role analyzes the bug report's title, body, and comments to flag specific fields that are missing and prevent reproduction. It identifies the initial set of gaps based on what information is necessary to understand and reproduce the reported issue in detail.
- Answer Evaluator
- In subsequent iterations, this component determines if the author's response adequately addresses a previous question. If an answer is insufficient, this evaluator derives new remaining gaps based on what information is still needed, driving the next round of targeted follow-up questions.
Terminology used across episodes
This episode discusses
- BRQGen: Towards Adaptive Completeness Assessment and Follow-Up Question Generation for Bug Reports · Paper Radio
- DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression · Paper Radio
- Automated Root-Cause Subclassification and No-Code Fix Generation for Invalid Bug Reports · Paper Radio
- Asking Clarifying Questions for Preference Elicitation With Large Language Models
- FeedAIde: Guiding App Users to Submit Rich Feedback Reports by Asking Context-Aware Follow-Up Questions
- Automatically Enhancing the Quality of Android App Bug Reports · Paper Radio
The paper
BRQGen: Towards Adaptive Completeness Assessment and Follow-Up Question Generation for Bug Reports · Read on arXiv
Technical University of Munich · Bilkent University
Many bug reports lack the necessary information required to reproduce issues. In early studies, large language models have been employed to generate follow-up questions to enhance incomplete reports. However, they ask a fixed number of questions regardless of whether the report is incomplete. We propose BRQGen, an iterative framework that identifies missing or ambiguous information in bug reports and generates targeted follow-up questions, only if necessary. Reports with no identified gaps are classified as sufficient. When the author responds, BRQGen evaluates whether the answers provide the requested information and generates further questions for any exposed gaps. We curated a novel dataset of 429 incomplete bug reports from Home Assistant and Brave Browser, each containing at least one maintainer follow-up question. Each question is mapped to the author's answer and assigned a label indicating whether the requested information could have been provided directly through the issue template. Earlier maintainer requests were associated with author responses, whereas delayed requests were associated with no response. Among responding authors, requests within one hour received median replies after 0.52 hours in Home Assistant and 1.45 hours in Brave, versus 9.76 and 13.49 hours after four hours. BRQGen covered 35.4% (Qwen 3.8 Flash) to 51.8% (DeepSeek V4.1 Flash) of the 1,073 ground-truth follow-up questions. DeepSeek V4.1 Flash managed to ask 45.7% of all detailed questions and 63.9% of all generic questions. Gemini 3 Flash Preview reached the highest macro-F1 for completeness assessment of 49.3%, but with a post-hoc analysis by getting only the gaps with high severities, Qwen 3.8 Flash reaches 73.6% macro-F1 score. Completeness assessment improved when at least one identified gap had high severity, and nearly all ground-truth questions were captured within the first two iterations.
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "BRQGen: Towards Adaptive Completeness Assessment and Follow-Up Question Generation for Bug Reports".
Jane: Here's the gist of the paper. BRQGen proposes an iterative framework to automatically assess bug report completeness and generate context-aware follow-up questions for missing details, addressing the information gap that hinders effective bug reporting and triage The first sentence of the paper states that BRQGen is a semi-automated framework that decides whether a bug report is complete to be reproduced and, if not, generates context-aware follow-up questions up to five iterations This approach aims to shorten the delay before missing information is requested from the author by automatically assessing report completeness and generating context-aware questions for the missing details.
Tom: Pruning redundant reasoning tokens The BRQGen framework operates as an iterative loop over a bug report where a single LLM call takes on two roles: Bug Report Evaluator, Answer Evaluator, and Question Generator The Bug Report Evaluator in the first iteration analyzes the report title, body, and preceding comments to flag fields whose absence prevents reproduction Subsequent iterations involve the Answer Evaluator deciding if an answer is sufficient and deriving remaining gaps based on previous answers The Question Generator writes follow-up questions only for the gaps flagged by the evaluator in that call.
Jane: This cycle repeats until all critical fields are determined to be complete or a maximum number of iterations is reached.
Tom: How the training signal is built The authors curated a novel dataset of 429 incomplete bug reports from the Home Assistant and Brave Browser repositories, each containing at least one maintainer follow-up question This dataset includes annotated ground-truth follow-up questions mapped to the author’s answers, labeled with specificity indicating if information could be provided directly through the project’s issue template.
Title and authors: Tom: Let’s talk about the title of this work, "BRQGen: Towards Adaptive Completeness Assessment and Follow-Up Question Generation for Bug Reports". It clearly signals that they aren't just using a standard AI to ask questions; they are focusing on *when* to ask.
Jane: And the authors are Sungur et al., from technical universities in Germany and Turkey, who seem focused on making the system more adaptive instead of just being reactive. The core idea is moving away from fixed sets of questions toward a dynamic assessment of completeness.
Lu: The paper’s main contribution is this iterative framework that uses an AI to decide if a bug report is complete or not, and then generates follow-up questions only for the specific missing details identified in that step.
Meng: If I'm being honest, I’m curious about how they handle the decision point—the "completeness" part. How does the system avoid making a mistake and asking questions when it thinks it's done, but then missing something critical?
Lalam: They address that by having a specific process where the Bug Report Evaluator flags fields that prevent reproduction in the first round, and then subsequent iterations use an Answer Evaluator to decide if those answers were actually sufficient.
Tom: So they’re trying to build a mechanism where the system learns from the author’s responses during the loop, refining its understanding of what information is truly necessary. That sounds like a big step toward more intelligent triage.
Jane: Exactly, it’s about making sure that each follow-up question is highly relevant to whatever detail is still missing at that specific moment in the process. This moves beyond just looking at the initial text and comments.
The paper's summary: Tom: The summary explains that the system works as a loop where one LLM call does all three jobs: evaluating, evaluating answers, and generating new questions based on what was missing in the previous step. It repeats this until everything is complete or they hit a set limit of five iterations.
Lu: They detail the roles clearly: in the first iteration, it flags gaps; later on, it checks if those gaps are filled by the author’s answers; and then it generates new questions only for what's left outstanding based on that check.
Meng: I see a lot of complexity in that loop. It sounds like a lot of sequential thinking is happening within one single model interaction, which could be computationally heavy at scale if you have thousands of reports to process.
Jane: The authors are trying to show that this iterative approach helps shorten the delay before we get the information we need from the author by automatically assessing report completeness and generating context-aware questions for those missing details.
Lalam: They use a dataset they curated, which includes ground-truth follow-up questions mapped to author answers, which is how they train this system to know what a good question looks like.
Tom: So the summary hinges on this cycle being adaptive—it doesn't just run a script; it's actively assessing the quality of the information provided at each stage. That’s where I get excited about the mechanics.
Jane: It’s about moving from a rigid system to one that actually adapts its questioning strategy based on what it learns during the conversation with the author.
The paper's improvements: Tom: The paper highlights several specific improvements, and they focus on making sure the framework doesn't just run wild. They suggest using a deterministic way to decide if a report is complete based on the identified gap set, so it doesn't get stuck in an infinite loop of asking questions when it should have stopped.
Lu: Another improvement they point out is that the system can prioritize these generated questions by ranking them based on how severe or critical they are for reproduction, and then it can stop after a certain number of the highest-ranked ones, say N max.
Meng: Prioritizing by severity makes sense from a production standpoint. If we have limited time or resources, you don't want to ask five questions when one high-severity one will do the job. That truncation based on ranking sounds like practical efficiency.
Jane: They also mention that the system can use a fully automated User Simulator to act as an oracle, matching generated questions against real maintainer questions to get precise scores on how well the generation is actually working compared to human judgment.
Lalam: The evaluation showed that models like DeepSeek V4 point 1 Flash could manage to ask sixty-three point nine percent of all generic questions and forty-five point seven percent of all detailed ground-truth questions, which gives us a concrete measure of the Question Generator’s capability across different question types <ref:2610.15903#pg1>.
Tom: That’s a useful metric because it separates whether the model is good at asking specific technical questions versus just asking general stuff. It shows where its strength lies in generating useful follow-ups.
Conclusion: Tom: So, to wrap up, BRQGen is this semi-automated framework that uses an iterative loop where one model handles evaluation and question generation until the report is complete or the iteration limit is hit. It’s designed to automatically assess completeness and generate context-aware follow-up questions for missing details.
Jane: The implication here for us in software maintenance is that we can significantly reduce the time spent waiting for authors to provide information that would have been obvious if the report was clearer from the start. It shifts the burden of asking clarifying questions from a human maintainer to an intelligent system, guided by real data on what causes delays.
Lu: The study suggests that integrating context engineering, perhaps through Retrieval-Augmented Generation or RAG, could make this framework even more powerful by giving the model access to broader documentation before it starts evaluating.
Meng: From a deployment standpoint, if we can reliably measure completeness this way, it could integrate into platforms like GitHub where a bot posts those precise questions as comments, which would streamline our workflow immensely if the performance holds up in production.
Lalam: Ultimately, this paper shows that by using an adaptive loop and training on real-world incomplete reports, we can create a system that is much more efficient at pinpointing exactly what’s missing and asking the right follow-up question at the right time.
Tom: So we’ve seen how they build the mechanism to make AI smarter about chasing missing bug report details. It's a lot of engineering focused on closing that information gap, and it seems like it’s heading in a direction where systems proactively seek clarity instead of just reacting to what's already there.
More episodes
- 2411.16152-Optimizing Winograd Convolution on ARMv8 processors
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines