Automatically Enhancing the Quality of Android App Bug Reports
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Automatically Enhancing the Quality of Android App Bug Reports".
Jane: Here's the gist of the paper. Most defects in mobile applications are visually observable on the device screen.
Tom: This paper introduces BUGSCRIBE, an LLM-powered approach that links bug report information with app-specific UI execution information to infer and generate accurate, complete, and correct Observed Behavior (OB), Expected Behavior (EB), and Steps to Reproduce (S2Rs).
Jane: Problem Formulation The core challenge in enhancing these reports is therefore not merely improving the bug description, but recovering the missing connections between the reported bug information and the application’s underlying UI execution needed to produce a complete and accurate description of the defect.
Tom: The authors formulate automatic bug report enhancement as a problem of connecting user-written bug reports with application execution.
Paper summary: Jane: So Tom, we're talking about this paper called "Automatically Enhancing the Quality of Android App Bug Reports." It seems to be tackling the problem that most bugs in mobile apps are just visually observable on the screen, but users often write bug reports that are incomplete or ambiguous because they miss the crucial app-specific execution information needed to reproduce them.
Tom: Yeah, exactly. The authors set out to solve this by connecting what a user writes in a bug report with the actual way an app runs on the device and using that connection to generate better versions of things like Observed Behavior, Expected Behavior, and Steps to Reproduce.
Lu: The core idea is that it's not just about making the text sound better; it's about recovering those missing links between what a user reports and the technical execution details of the app. They frame it as connecting user-written bug reports with application execution information to infer and generate accurate descriptions of Observed Behavior, Expected Behavior, and Steps to Reproduce.
Meng: So, if I’m running this on my end at scale, what's the big deal? Is this just a nice way to write reports for testers?
Jane: It's more than that. The paper points out that the challenge is connecting the reported bug information with the application’s underlying UI execution needed to produce a complete and accurate description of the defect, and they formulate automatic bug report enhancement as a problem of connecting user-written bug reports with application execution.
Tom: Right. So instead of just editing the text, they're building something that looks at both sides—the report and the app’s behavior simultaneously—to pull out those missing connections using an LLM and some app-specific context.
Lalam: The architecture they use is called BUGSCRIBE, which employs a component-specific grounding strategy to give the LLM the most relevant context for generating each part of the bug report. It builds a graph-based execution model of app screens and GUI interactions, enriching it with UI metadata, and then links sentences from the original report to those relevant app execution elements.
Lu: That component-specific grounding is key. They discovered that different parts of the bug report need different kinds of app context. For instance, generating Steps to Reproduce depends mainly on GUI interactions and the buggy screen itself, while generating Observed Behavior and Expected Behavior benefits from generated Steps to Reproduce and buggy screen descriptions.
Paper summary: Meng: Hmm, so they're not just feeding one giant blob of app data to the LLM for everything? That sounds like it could be computationally heavy if we have millions of screens and interactions.
Tom: That’s where their methodology gets smart. They developed a data-driven way to figure out which combinations of app-specific information actually work best for generating those specific components, finding that the combination "Interac" plus "Scrn" plus "BuggyScrn" was the best for generating Steps to Reproduce, achieving an F1 score of eighty-eight point four nine on that metric <ref:2604.01148#pg1>.
Jane: That's a concrete number showing they found a sweet spot for their strategy, and they found that different components benefit from different pieces of context, which is really important because it stops them from just guessing what information the LLM needs.
Lalam: They use LLM-based reasoning through zero-shot task decomposition prompting to generate the final high-quality bug reports after linking all that execution information together. This selective combination of data is their main contribution, allowing the model to focus on what matters for each specific output, which is what they call a novel LLM-guided reasoning strategy.
Tom: The results they got are pretty compelling too. BUGSCRIBE generated higher quality bug report components than the original reports and three other LLM-based baselines. They saw improvements in S2R quality by forty-four point one percent to over eighty-two point three percent, and OB and EB quality improved by about four point eight percent to thirty-five point two percent <ref:2604.01148#pg1>.
Meng: Those numbers sound significant, but how do we know those results hold up when we look at real traffic? I worry that performance on a test set doesn't always translate to production environments where reports are much messier.
Lu: The authors admit that the performance gains are sensitive to low-quality user reports, specifically vague steps, which can cause BUGSCRIBE to produce a shorter but less optimal path in those cases. They also flag that ambiguities in buggy screen references pop up because BUGSCRIBE often uses generic names instead of the exact UI wording.
Jane: So even though it's powerful, it has limitations where poor input quality can degrade the output, and they are worried about using generic screen names instead of the precise UI text. That makes sense for real-world app data where naming conventions aren't perfect.
Paper summary: Tom: Their future work is to tackle those naming issues by incorporating visual information from UI screenshots using multimodal models, which should help improve how the system names screens more accurately, and they also acknowledge that running BSGPT costs about three minutes per bug report.
Meng: Three minutes per report? That's a significant operational cost if we have high volume reporting. What does that mean for deployment speed?
Jane: It means the computational cost is there, but they are showing that the output quality is substantially better than existing methods, especially in Steps to Reproduce where they generated five hundred thirty-two correct steps compared to about one hundred ninety-six of those from other baselines <ref:2604.01148#pg1>.
Lalam: They really emphasize that their approach consistently produces higher-quality bug reports than existing methods that rely mainly on the text in the bug report because it connects the report content with relevant app execution elements through this component-specific grounding strategy.
Tom: So, to wrap up, BUGSCRIBE is an LLM approach that uses a graph of screen interactions and UI metadata to ground its reasoning for generating accurate Observed Behavior, Expected Behavior, and Steps to Reproduce by connecting them directly to the app's execution details.
Jane: And it shows that tailoring the context you give the LLM for different report components makes a real difference in quality, even if user reports are imperfect. That connects back to how we can build better systems that understand user intent more deeply.
Lu: The implication is that we need to move beyond just text analysis and start grounding language generation with structured execution data from the application itself, which is exactly what this paper proposes with BUGSCRIBE.
Meng: It sounds like it’s a good proof of concept for how LLMs can be used as reasoning engines when you have the right kind of external, structured context available to them.
Tom: Yeah, it's about making sure the AI doesn't just sound fluent; it has to actually know where the bug is happening in the app’s actual running state.
Jane: We can see this approach being applied wherever complex user input needs to be translated into structured, executable steps for automated testing or developer workflows.
Conclusion: Jane: So we've been talking about BUGSCRIBE, which is this LLM approach for automatically improving Android app bug reports by connecting user descriptions to actual app execution data like screen interactions and UI metadata.
Tom: Right, it basically takes a messy bug report and uses that technical context—the graph of screens and how they connect—to generate better versions of the observed behavior, expected behavior, and those steps to reproduce.
Lu: The authors found that you need different kinds of app data for different parts of the report; like they used GUI interactions mainly for writing the steps to reproduce.
Meng: But what's the actual impact here if I'm just shipping models at scale? How does this translate from a research paper to something I can actually deploy without it eating up my entire infrastructure budget?
Jane: The main thing is that even if you have low-quality reports, BUGSCRIBE still produces higher quality components than standard methods or other LLM baselines.
Tom: They showed improvements in steps to reproduce quality of about forty point one percent to over eighty two point three percent. That's a real jump in how accurately the bug can be described for a developer.
Lu: But they also flagged that if the user's original report is really vague, like they just write "it doesn't work," BUGSCRIBE might produce a shorter but less useful set of steps because it can’t find enough good context.
Meng: So it’s not a perfect fix for every single bad report yet. What about those limitations? What are the authors worried about when they talk about this system?
Jane: They're worried that the system often uses generic screen names instead of the exact wording on the screen, and that’s another hurdle for getting super accurate context.
Tom: Their future plan is to use visual information from screenshots with multimodal models to fix those naming issues so it can be more precise.
Lu: And they did admit the computational cost for running this system, which they call BSGPT, is about three minutes per bug report on average.
Jane: So that's a big time investment, but the core message is that grounding the LLM with actual app execution data makes it much better at generating those critical bug details than just reading the text alone.
Tom: It shows us how to make AI reasoning actually tethered to the physical reality of how an application runs on a device.
Jane: And if this works out, it means we can move past just analyzing text and start grounding language generation with structured data that comes directly from the app itself.
Antu Saha, Atish Kumar Dipongkor, Sam Bennett, Kevin Moran, Andrian Marcus, Oscar Chaparro
William & Mary University · University of Central Florida Department of Computer Science
cs.SE
Submitted: 2026-04-01
Updated: 2026-10-30
Comments: 12 pages, 6 figures
Code: https://github.com/netmackan/ATimeTracker
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 92/100
The gist: Most defects in mobile applications are visually observable on the device screen.
Key concepts
- BUGSCRIBE
- This is an LLM-powered approach designed to enhance Android bug reports. It works by connecting information from a user's bug report with actual execution data from the application's UI. The goal is to automatically infer and generate complete and correct descriptions of what happened, what should have happened, and how to reproduce the issue.
- Component-Specific Grounding Strategy
- This strategy involves giving different parts of the bug report generation task specific, relevant context from the app. Instead of feeding all data to the LLM at once, it focuses on providing only the most pertinent app execution information for generating each part (like S2R or OB), leading to better results.
- Execution Model Graph
- This is a structure built by mapping out all possible screens and GUI interactions within an application, complete with metadata. This graph helps the system understand how different parts of the app connect, allowing it to find the most relevant execution elements when trying to link them to a bug report.
Terminology
Summary
Most defects in mobile applications are visually observable on the device screen. p. 0 edu
Oscar Chaparro
William & Mary
Williamsburg, Virginia, USA
oscarch@wm. edu
Abstract—Most defects in mobile applications are visually
observable on the device screen. We formulate automatic bug report enhancement as the
problem of connecting user-written bug reports with application
execution. We present BUGSCRIBE, an LLM-powered approach
that links bug report information with app-specific UI execution
information to infer and generate accurate, complete, and correct
Observed Behavior (OB), Expected Behavior (EB), and Steps to
Reproduce (S2Rs).Where the paper says it
Where the paper says it
Problem Formulation
The core challenge in enhancing these reports is therefore not merely improving the bug description, but recovering the missing connections between the reported bug information and the application’s underlying UI execution needed to produce a complete and accurate description of the defect. The authors formulate automatic bug report enhancement as a problem of connecting user-written bug reports with application execution. p. 0 This challenge is particularly
prominent for UI-centric defects, where the relevant application
behavior is difficult for end users to describe precisely. We formulate automatic bug report enhancement as the
problem of connecting user-written bug reports with application
execution. We formulate automatic bug report enhancement as the
problem of connecting user-written bug reports with application
execution. We present BUGSCRIBE, an LLM-powered approach
that links bug report information with app-specific UI execution
information to infer and generate accurate, complete, and correct
Observed Behavior (OB), Expected Behavior (EB), and Steps to
Reproduce (S2Rs).Where the paper says it
Where the paper says it
BUGSCRIBE Architecture
BUGSCRIBE employs a component-specific grounding strategy that provides the most relevant context to an LLM for generating each bug report component. The system constructs a graph-based execution model of app screens and GUI interactions enriched with UI metadata. p. 1 BUGSCRIBE augments existing reports with
accurate, correct, and complete OB, EB, and S2R descriptions
through a pipeline that connects bug report content with
app-specific execution information. Specifically, BUGSCRIBE
constructs a graph-based execution model of app screens
and GUI interactions enriched with UI metadata; identifies
OB, EB, and S2R sentences in the original report; links this
information to relevant app execution elements; and generates
high-quality bug report components using component-specific
prompts. BUGSCRIBE augments existing reports with
accurate, correct, and complete OB, EB, and S2R descriptions
through a pipeline that connects bug report content with
app-specific execution information. Specifically, BUGSCRIBE
constructs a graph-based execution model of app screens
and GUI interactions enriched with UI metadata; identifies
OB, EB, and S2R sentences in the original report; links this
information to relevant app execution elements; and generates
high-quality bug report components using component-specific
prompts. , GUI interactions) used
as context for the LLM (Section III-B). 3) Bug Report Generation: it generates high-quality bug
reports, including a title, OB, EB, and S2Rs, leveraging
the app-specific information and LLM-based reasoning via
zero-shot, task-decomposition prompting (Section III-C).Where the paper says it
Where the paper says it
Where the paper says it
Contextual Information Strategy
The authors developed a data-driven methodology to identify which combinations of app-specific information are most effective for generating OB, EB, and S2Rs. They found that different components benefit from different app-specific information: S2R generation depends primarily on GUI interactions and the buggy screen, whereas OB and EB generation benefit from generated S2Rs, buggy screen descriptions, and screen-level context. p. 1 Using this quality model
and manually curated ground truth for 10 bug reports from 9
Android apps, we evaluated eight context configurations for
S2R and OB/EB generation. Our results show that different
components benefit from different app-specific information:
S2R generation depends primarily on GUI interactions and the
buggy screen, whereas OB and EB generation benefit from
generated S2Rs, buggy screen descriptions, and screen-level
context. Figure 3 shows the S2R quality results of the four configurations for S2R generation, averaged across the three executions. All three context-aware configurations substantially outperform
the no-information configuration, with F1 increasing from 72. 4) Interac. +BuggyScrn. 4) Interac. +BuggyScrn. over three runs)
achieved consistent performance across the runs. For instance,
the F1 score’s std.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Empirical Evaluation Results
BUGSCRIBE generates higher-quality bug report components than the original reports and three LLM-based baselines, improving S2R quality by 44.1%–82.3% and OB/EB quality by 3. p. 0 Automatically Enhancing the Quality of Android
App Bug Reports
Antu Saha
William & Mary
Williamsburg, Virginia, USA
asaha02@wm. edu
Atish Kumar Dipongkor
University of Central Florida
Orlando, Florida, USA
atish. The
table shows that BUGSCRIBE (both BSGPT and BSCLAUDE)
clearly outperforms all baselines and the original bug reports
across all metrics. In particular, it generates substantially
more correct steps (532, 538 vs. In particular, it generates substantially
more correct steps (532, 538 vs. 196, 243, 345, 312) while
significantly reducing both extra and missing steps (e. Figure 3 shows the S2R quality results of the four configurations for S2R generation, averaged across the three executions. All three context-aware configurations substantially outperform
the no-information configuration, with F1 increasing from 72. For S2Rs, we assigned three quality labels:
1) Correct Step (CS): the generated step is found in the ground
truth S2Rs (True Positive or TP), 2) Extra Step (ES): the
generated step is not found in the ground truth S2Rs (False
Positive or FP, and 3) Missing Step (MS): a ground truth S2R
is not found in the generated steps (False Negative or FN). Using the quality labels assigned to the S2Rs, we computed
precision, recall, and F1 score to measure S2R quality.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Limitations and Future Work
The analysis revealed that the performance gains are sensitive to low-quality user reports, such as vague steps, which can lead BUGSCRIBE to produce a shorter but suboptimal path. Furthermore, ambiguities in buggy screen references arise because BUGSCRIBE often uses generic names rather than the exact UI wording. p. 9 These ambiguities
arise because BUGSCRIBE often uses generic, functionalitybased names rather than the exact UI wording. For instance, in
bug report #154 for the Camfahrplan app [41], the correct
buggy screen is “schedule screen”, whereas BSCLAUDE
refers to it as “event details screen”. This discrepancy stems
from limitations in the app metadata used to build the app
execution model, which may omit or differ from the uservisible screen names, prompting BUGSCRIBE to infer names
from the UI screen descriptions used as context. To address
this, our future work will incorporate visual information from
UI screenshots via multimodal models to improve screen
naming. 3) Computational Cost: We measured the end-to-end cost
of BUGSCRIBE. On average, BSGPT processes 121.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
BUGSCRIBE enables LLMs to generate accurate, complete, and correct OB, EB, and S2Rs by connecting report information to relevant app execution elements through a component-specific grounding strategy. The approach consistently produces higher-quality bug reports than existing methods that rely primarily on the bug report text. p. 0 Prior work has proposed techniques to assess bug report
quality [9], [6], identify missing information [10], [11], provide
feedback to reporters [9], [6], support interactive report construction [2], [12], [13], [14], and restructure bug reports [15]. While these approaches help identify quality problems or
rewrite existing reports, they either leave users responsible
for correcting the report or generate content primarily from
the original report text.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Automated bug report enhancement system that connects user-written bug reports with application execution information to generate accurate, complete, and correct Observed Behavior (OB), Expected Behavior (EB), and Steps to Reproduce (S2Rs). This system can infer and generate high-quality OB/EB/S2R descriptions by leveraging a component-specific grounding strategy.
-
A data-driven methodology for identifying the specific app information elements needed by the LLM’s reasoning strategy to link report and app execution data. This allows the AI to dynamically select which combination of app context—such as GUI interactions, screen descriptions, or buggy screens—is most effective for generating specific components like S2Rs or OB/EB.
-
An empirical evaluation framework that demonstrates BUGSCRIBE's superiority by comparing it against LLM-based baselines that do not use app-specific information. This system can achieve significant improvements in S2R quality, showing relative increases of 44.1% to 82.3% and OB/EB quality improvements of 3.8% to 35.2%.
-
A pipeline capable of constructing a graph-based execution model of app screens and GUI interactions enriched with UI metadata for every application analyzed. This model allows the system to represent the app's flow as a directed graph where nodes are screens and edges are user interactions, enabling precise pathfinding for S2R generation.
-
A context extraction module that annotates user-submitted bug reports to specify OB, EB, and S2Rs sentences while simultaneously extracting three app information elements: app screen information, GUI interactions, and the buggy screen. This module feeds the most relevant context to the LLM for generating specific report components based on a component-specific grounding strategy.
-
A generation engine that employs zero-shot task decomposition prompting to synthesize atomic S2Rs from identified GUI interactions within an execution model and map them to plausible reproduction scenarios leading to the buggy screen. This enables the system to generate detailed, enumerated lists of reproduction steps that correspond directly to specific GUI actions.
-
A generation engine that uses GPT-5.4 or CLAUDEOPUS-4.6 as a state-of-the-art reasoning LLM, guided by context, to generate high-quality bug reports including a one line bug summary, OB, EB, and S2Rs in a structured format. This system produces outputs that are more complete and accurate than original reports or other LLM baselines.
Abstract
Most defects in mobile applications are visually observable on the device screen. Since automated mechanisms for detecting and reporting such defects are often unavailable, users, testers, and developers must manually submit bug reports. However, these reports are frequently incomplete, ambiguous, or inaccurate, often lacking the information needed to understand, reproduce, and diagnose defects. We formulate automatic bug report enhancement as the problem of connecting user-written bug reports with application execution. We present BUGSCRIBE, an LLM-powered approach that links bug report information with app-specific UI execution information to infer and generate accurate, complete, and correct Observed Behavior (OB), Expected Behavior (EB), and Steps to Reproduce (S2Rs). To support BUGSCRIBE's design and evaluation, we develop a bug report quality model and use it to identify the most effective context for each component. We evaluate BUGSCRIBE on 48 bug reports from 26 Android applications with manually constructed ground truth. Our results show that BUGSCRIBE generates higher-quality bug report components than the original reports and three LLM-based baselines, improving S2R quality by 44.2%--84.7%, and OB/EB quality by up to 35.2%. We further show that BUGSCRIBE-generated reports help an LLM-based agent reproduce Android bugs, improving the reproduction rate by 28.1%--57.7% compared to the original reports and the baselines.
Sources
- Combining Language and App UI Analysis for the Automated Assessment of Bug Reproduction Steps
- Can We Enhance Bug Report Quality Using LLMs?: An Empirical Study of LLM-Based Bug Report Generation
- Decomposed Prompting: A Modular Approach for Solving Complex Tasks
- Successive Prompting for Decomposing Complex Questions
- Which Prompting Technique Should I Use? An Empirical Investigation of Prompting Techniques for Software Engineering Tasks
- The Prompt Report: A Systematic Survey of Prompt Engineering Techniques
- Meta Prompting for AI Systems
- Studying and Automating Issue Resolution for Software Quality
- Evaluating Language Model Applications for Identifying Solution-Related Content in Issue Report Discussions
- ImageR: Enhancing Bug Report Clarity by Screenshots
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters