BRQGen: Towards Adaptive Completeness Assessment and Follow-Up Question Generation for Bug Reports
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "BRQGen: Towards Adaptive Completeness Assessment and Follow-Up Question Generation for Bug Reports".
Jane: Here's the gist of the paper. BRQGen proposes an iterative framework to automatically assess bug report completeness and generate context-aware follow-up questions for missing details, addressing the information gap that hinders effective bug reporting and triage The first sentence of the paper states that BRQGen is a semi-automated framework that decides whether a bug report is complete to be reproduced and, if not, generates context-aware follow-up questions up to five iterations This approach aims to shorten the delay before missing information is requested from the author by automatically assessing report completeness and generating context-aware questions for the missing details.
Tom: Pruning redundant reasoning tokens The BRQGen framework operates as an iterative loop over a bug report where a single LLM call takes on two roles: Bug Report Evaluator, Answer Evaluator, and Question Generator The Bug Report Evaluator in the first iteration analyzes the report title, body, and preceding comments to flag fields whose absence prevents reproduction Subsequent iterations involve the Answer Evaluator deciding if an answer is sufficient and deriving remaining gaps based on previous answers The Question Generator writes follow-up questions only for the gaps flagged by the evaluator in that call.
Jane: This cycle repeats until all critical fields are determined to be complete or a maximum number of iterations is reached.
Tom: How the training signal is built The authors curated a novel dataset of 429 incomplete bug reports from the Home Assistant and Brave Browser repositories, each containing at least one maintainer follow-up question This dataset includes annotated ground-truth follow-up questions mapped to the author’s answers, labeled with specificity indicating if information could be provided directly through the project’s issue template.
Title and authors: Tom: Let’s talk about the title of this work, "BRQGen: Towards Adaptive Completeness Assessment and Follow-Up Question Generation for Bug Reports". It clearly signals that they aren't just using a standard AI to ask questions; they are focusing on *when* to ask.
Jane: And the authors are Sungur et al., from technical universities in Germany and Turkey, who seem focused on making the system more adaptive instead of just being reactive. The core idea is moving away from fixed sets of questions toward a dynamic assessment of completeness.
Lu: The paper’s main contribution is this iterative framework that uses an AI to decide if a bug report is complete or not, and then generates follow-up questions only for the specific missing details identified in that step.
Meng: If I'm being honest, I’m curious about how they handle the decision point—the "completeness" part. How does the system avoid making a mistake and asking questions when it thinks it's done, but then missing something critical?
Lalam: They address that by having a specific process where the Bug Report Evaluator flags fields that prevent reproduction in the first round, and then subsequent iterations use an Answer Evaluator to decide if those answers were actually sufficient.
Tom: So they’re trying to build a mechanism where the system learns from the author’s responses during the loop, refining its understanding of what information is truly necessary. That sounds like a big step toward more intelligent triage.
Jane: Exactly, it’s about making sure that each follow-up question is highly relevant to whatever detail is still missing at that specific moment in the process. This moves beyond just looking at the initial text and comments.
The paper's summary: Tom: The summary explains that the system works as a loop where one LLM call does all three jobs: evaluating, evaluating answers, and generating new questions based on what was missing in the previous step. It repeats this until everything is complete or they hit a set limit of five iterations.
Lu: They detail the roles clearly: in the first iteration, it flags gaps; later on, it checks if those gaps are filled by the author’s answers; and then it generates new questions only for what's left outstanding based on that check.
Meng: I see a lot of complexity in that loop. It sounds like a lot of sequential thinking is happening within one single model interaction, which could be computationally heavy at scale if you have thousands of reports to process.
Jane: The authors are trying to show that this iterative approach helps shorten the delay before we get the information we need from the author by automatically assessing report completeness and generating context-aware questions for those missing details.
Lalam: They use a dataset they curated, which includes ground-truth follow-up questions mapped to author answers, which is how they train this system to know what a good question looks like.
Tom: So the summary hinges on this cycle being adaptive—it doesn't just run a script; it's actively assessing the quality of the information provided at each stage. That’s where I get excited about the mechanics.
Jane: It’s about moving from a rigid system to one that actually adapts its questioning strategy based on what it learns during the conversation with the author.
The paper's improvements: Tom: The paper highlights several specific improvements, and they focus on making sure the framework doesn't just run wild. They suggest using a deterministic way to decide if a report is complete based on the identified gap set, so it doesn't get stuck in an infinite loop of asking questions when it should have stopped.
Lu: Another improvement they point out is that the system can prioritize these generated questions by ranking them based on how severe or critical they are for reproduction, and then it can stop after a certain number of the highest-ranked ones, say N max.
Meng: Prioritizing by severity makes sense from a production standpoint. If we have limited time or resources, you don't want to ask five questions when one high-severity one will do the job. That truncation based on ranking sounds like practical efficiency.
Jane: They also mention that the system can use a fully automated User Simulator to act as an oracle, matching generated questions against real maintainer questions to get precise scores on how well the generation is actually working compared to human judgment.
Lalam: The evaluation showed that models like DeepSeek V4 point 1 Flash could manage to ask sixty-three point nine percent of all generic questions and forty-five point seven percent of all detailed ground-truth questions, which gives us a concrete measure of the Question Generator’s capability across different question types <ref:2610.15903#pg1>.
Tom: That’s a useful metric because it separates whether the model is good at asking specific technical questions versus just asking general stuff. It shows where its strength lies in generating useful follow-ups.
Conclusion: Tom: So, to wrap up, BRQGen is this semi-automated framework that uses an iterative loop where one model handles evaluation and question generation until the report is complete or the iteration limit is hit. It’s designed to automatically assess completeness and generate context-aware follow-up questions for missing details.
Jane: The implication here for us in software maintenance is that we can significantly reduce the time spent waiting for authors to provide information that would have been obvious if the report was clearer from the start. It shifts the burden of asking clarifying questions from a human maintainer to an intelligent system, guided by real data on what causes delays.
Lu: The study suggests that integrating context engineering, perhaps through Retrieval-Augmented Generation or RAG, could make this framework even more powerful by giving the model access to broader documentation before it starts evaluating.
Meng: From a deployment standpoint, if we can reliably measure completeness this way, it could integrate into platforms like GitHub where a bot posts those precise questions as comments, which would streamline our workflow immensely if the performance holds up in production.
Lalam: Ultimately, this paper shows that by using an adaptive loop and training on real-world incomplete reports, we can create a system that is much more efficient at pinpointing exactly what’s missing and asking the right follow-up question at the right time.
Tom: So we’ve seen how they build the mechanism to make AI smarter about chasing missing bug report details. It's a lot of engineering focused on closing that information gap, and it seems like it’s heading in a direction where systems proactively seek clarity instead of just reacting to what's already there.
Technical University of Munich · Bilkent University
cs.SE
Submitted: 2026-10-11
Updated: 2026-11-10
Code: https://github.com/home-assistant/core
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 85/100
The gist: BRQGen proposes an iterative framework to automatically assess bug report completeness and generate context-aware follow-up questions for missing details…
Key concepts
- Iterative Framework
- BRQGen operates in a loop where one LLM call performs three distinct roles: evaluating the report for missing fields, assessing if an answer is enough, and generating new questions. This cycle repeats until all critical information is found or a limit is reached, allowing for deeper gap identification over multiple turns.
- Bug Report Evaluator
- This role analyzes the bug report's title, body, and comments to flag specific fields that are missing and prevent reproduction. It identifies the initial set of gaps based on what information is necessary to understand and reproduce the reported issue in detail.
- Answer Evaluator
- In subsequent iterations, this component determines if the author's response adequately addresses a previous question. If an answer is insufficient, this evaluator derives new remaining gaps based on what information is still needed, driving the next round of targeted follow-up questions.
Terminology
Summary
BRQGen proposes an iterative framework to automatically assess bug report completeness and generate context-aware follow-up questions for missing details, addressing the information gap that hinders effective bug reporting and triage The first sentence of the paper states that BRQGen is a semi-automated framework that decides whether a bug report is complete to be reproduced and, if not, generates context-aware follow-up questions up to five iterations This approach aims to shorten the delay before missing information is requested from the author by automatically assessing report completeness and generating context-aware questions for the missing details. p. 0 Objective: We aim to shorten the delay before missing information is requested from the author by automatically assessing report completeness and generating context-aware questions for the missing details. Method: We propose BRQGen, an iterative framework that identifies missing or ambiguous information in
bug reports and generates targeted follow-up questions.Where the paper says it
Pruning redundant reasoning tokens
The BRQGen framework operates as an iterative loop over a bug report where a single LLM call takes on two roles: Bug Report Evaluator, Answer Evaluator, and Question Generator The Bug Report Evaluator in the first iteration analyzes the report title, body, and preceding comments to flag fields whose absence prevents reproduction Subsequent iterations involve the Answer Evaluator deciding if an answer is sufficient and deriving remaining gaps based on previous answers The Question Generator writes follow-up questions only for the gaps flagged by the evaluator in that call. This cycle repeats until all critical fields are determined to be complete or a maximum number of iterations is reached. p. 2 (3) Question Generator (every iteration): writes follow-up questions addressed to the author,
only for the gaps flagged by the evaluator in the same call. This cycle repeats until all critical fields for reproduction are determined to be complete and
unambiguous, none of the generated questions are answered, or the maximum number of iterations
is reached.Where the paper says it
How the training signal is built
The authors curated a novel dataset of 429 incomplete bug reports from the Home Assistant and Brave Browser repositories, each containing at least one maintainer follow-up question This dataset includes annotated ground-truth follow-up questions mapped to the author’s answers, labeled with specificity indicating if information could be provided directly through the project’s issue template. The annotation protocol involved three authors jointly annotating candidate reports, finalizing a guideline through majority voting on disagreements to establish final labels. p. 6 Three authors annotated the 649 candidate positive reports. To establish
an initial annotation guideline, they first jointly annotated five randomly selected reports.Where the paper says it
Where the gains come from
The empirical analysis on Home Assistant and Brave Browser showed that maintainers’ earlier information requests are significantly associated with more frequent and faster author responses Specifically, information requests made within one hour received a median reply in 0.52 hours in Home Assistant, compared to 9. p. 0 52 hours in Home Assistant and 1. 45 hours in Brave, compared with 9. 45 hours in Brave, compared with 9. 49 hours for
those made after four hours. (2) Answer Evaluator (subsequent iterations): given the question–answer pairs of the previous
iteration, decides for each pair whether the answer is sufficient, and derives the remaining
gaps that arise from those answers, if any. (3) Question Generator (every iteration): writes follow-up questions addressed to the author,
only for the gaps flagged by the evaluator in the same call. DeepSeek V4. 1
Flash managed to ask 45. 1
Flash managed to ask 45. 7% of all ground-truth detailed questions and 63. 7% of all ground-truth detailed questions and 63. 9% of all generic questions. 9% of all generic questions. Gemini
3 Flash Preview reached the highest macro-F1 for completeness assessment of 49. 6% macro-F1 score. Conclusion: BRQGen performs better at completeness assessment while at least one identified gap has high
severity. 6% macro-F1 score. Conclusion: BRQGen performs better at completeness assessment while at least one identified gap has high
severity.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Evaluating model performance and future directions
The evaluation showed that Gemini 3 Flash Preview reached the highest macro-F1 for completeness assessment of 49.3%, but Qwen 3. p. 0 On the 427 incomplete reports, the generated questions covered 35. 4% (Qwen 3. 8 Flash reaches 73. 6% macro-F1 score. The first two iterations nearly cover all ground-truth questions captured on each run compared
to later iterations, including detailed questions, which cannot be directly answered from the issue template. Future work will focus on integrating context engineering via Retrieval-Augmented Generation (RAG) and
validating the pipeline on industrial software ecosystems. The report is evaluated in its latest state before the first annotated follow-up question 𝑞1 was
asked, denoted 𝐵0 = (𝑏, 𝑡,𝐶0, 𝑢, 𝐷0), where 𝑏 is the report body at that time, 𝑡 its title, 𝑢 its author,𝐶0
the comments posted before the comment that contains 𝑞1, and 𝐷0 the additional media referenced
in 𝑏 and 𝐶0. We remove bot comments from 𝐶0 and mark each remaining comment as written by
the author, maintainer or another participant, so that the model can tell the author’s additions from
maintainers and other people’s comments. After selecting the repositories, we constructed two candidate datasets. Candidate positive samples (B
+
) are reports likely to contain a maintainer follow-up
question and a subsequent author response, whereas negative samples (B
−
) are reports likely to
have required no further clarification.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Threats to validity and limitations
The internal validity relies on a User Simulator to answer generated questions by matching them to ground-truth questions, with an agreement F1 score of 0.84 across models. p. 0 This information gap increases triage time and often leads to valid bugs being closed as non-reproducible. While early studies utilized large language models (LLMs) to enhance low-quality reports, these approaches
introduce incorrect information when the initial input lacks sufficient semantic content. Elements such as S2R, stack traces,
and test cases are frequently omitted from bug reports [50]. Initial reports in major open-source
projects rarely contain actionable data beyond observed and expected behavior [13], industrial
reports particularly lack application configuration and operating data [27], and in web applications
the usage context and diagnostic details are typically absent upon submission and elicited later [45]. To construct the candidate positive set, we applied strict heuristic filtering
to the closed issues of both repositories. We required that the more-information label was applied
by a maintainer and subsequently removed, and that the issue author commented while the label
was active.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Adaptive completeness assessment can be implemented through an iterative framework where a single LLM call takes on roles such as Bug Report Evaluator, Answer Evaluator, and Question Generator to identify missing information and generate targeted follow-up questions.
-
The system can decide whether a bug report is complete by deriving the binary completeness label deterministically from the identified gap set, preventing self-inconsistent outputs where a model declares a report sufficient while simultaneously identifying missing information.
-
The framework can generate context-aware follow-up questions up to five iterations, with each new iteration evaluating the author's answers to previous questions to expose further gaps arising from those answers.
-
The system can prioritize missing information by ranking generated questions in descending order of severity and reproducibility criticality, and it can truncate the question list to the Nmax highest-ranked questions for efficiency.
-
The system can use a fully automated User Simulator that acts as an oracle over ground-truth mappings to match generated questions to real maintainer questions, providing precision, recall, and F1 scores against human judgment.
Abstract
Many bug reports lack the necessary information required to reproduce issues. In early studies, large language models have been employed to generate follow-up questions to enhance incomplete reports. However, they ask a fixed number of questions regardless of whether the report is incomplete. We propose BRQGen, an iterative framework that identifies missing or ambiguous information in bug reports and generates targeted follow-up questions, only if necessary. Reports with no identified gaps are classified as sufficient. When the author responds, BRQGen evaluates whether the answers provide the requested information and generates further questions for any exposed gaps. We curated a novel dataset of 429 incomplete bug reports from Home Assistant and Brave Browser, each containing at least one maintainer follow-up question. Each question is mapped to the author's answer and assigned a label indicating whether the requested information could have been provided directly through the issue template. Earlier maintainer requests were associated with author responses, whereas delayed requests were associated with no response. Among responding authors, requests within one hour received median replies after 0.52 hours in Home Assistant and 1.45 hours in Brave, versus 9.76 and 13.49 hours after four hours. BRQGen covered 35.4% (Qwen 3.8 Flash) to 51.8% (DeepSeek V4.1 Flash) of the 1,073 ground-truth follow-up questions. DeepSeek V4.1 Flash managed to ask 45.7% of all detailed questions and 63.9% of all generic questions. Gemini 3 Flash Preview reached the highest macro-F1 for completeness assessment of 49.3%, but with a post-hoc analysis by getting only the gaps with high severities, Qwen 3.8 Flash reaches 73.6% macro-F1 score. Completeness assessment improved when at least one identified gap had high severity, and nearly all ground-truth questions were captured within the first two iterations.
Sources
- DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
- Automated Root-Cause Subclassification and No-Code Fix Generation for Invalid Bug Reports
- Asking Clarifying Questions for Preference Elicitation With Large Language Models
- FeedAIde: Guiding App Users to Submit Rich Feedback Reports by Asking Context-Aware Follow-Up Questions
- Automatically Enhancing the Quality of Android App Bug Reports
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters