Automatically Enhancing the Quality of Android App Bug Reports

arXiv:2604.01148 · cs.SE · Submitted 2026-04-01 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Automatically Enhancing the Quality of Android App Bug Reports".

Jane: Here's the gist of the paper. Most defects in mobile applications are visually observable on the device screen.

Tom: This paper introduces BUGSCRIBE, an LLM-powered approach that links bug report information with app-specific UI execution information to infer and generate accurate, complete, and correct Observed Behavior (OB), Expected Behavior (EB), and Steps to Reproduce (S2Rs).

Jane: Problem Formulation The core challenge in enhancing these reports is therefore not merely improving the bug description, but recovering the missing connections between the reported bug information and the application’s underlying UI execution needed to produce a complete and accurate description of the defect.

Tom: The authors formulate automatic bug report enhancement as a problem of connecting user-written bug reports with application execution.

Paper summary: Jane: So Tom, we're talking about this paper called "Automatically Enhancing the Quality of Android App Bug Reports." It seems to be tackling the problem that most bugs in mobile apps are just visually observable on the screen, but users often write bug reports that are incomplete or ambiguous because they miss the crucial app-specific execution information needed to reproduce them.

Tom: Yeah, exactly. The authors set out to solve this by connecting what a user writes in a bug report with the actual way an app runs on the device and using that connection to generate better versions of things like Observed Behavior, Expected Behavior, and Steps to Reproduce.

Lu: The core idea is that it's not just about making the text sound better; it's about recovering those missing links between what a user reports and the technical execution details of the app. They frame it as connecting user-written bug reports with application execution information to infer and generate accurate descriptions of Observed Behavior, Expected Behavior, and Steps to Reproduce.

Meng: So, if I’m running this on my end at scale, what's the big deal? Is this just a nice way to write reports for testers?

Jane: It's more than that. The paper points out that the challenge is connecting the reported bug information with the application’s underlying UI execution needed to produce a complete and accurate description of the defect, and they formulate automatic bug report enhancement as a problem of connecting user-written bug reports with application execution.

Tom: Right. So instead of just editing the text, they're building something that looks at both sides—the report and the app’s behavior simultaneously—to pull out those missing connections using an LLM and some app-specific context.

Lalam: The architecture they use is called BUGSCRIBE, which employs a component-specific grounding strategy to give the LLM the most relevant context for generating each part of the bug report. It builds a graph-based execution model of app screens and GUI interactions, enriching it with UI metadata, and then links sentences from the original report to those relevant app execution elements.

Lu: That component-specific grounding is key. They discovered that different parts of the bug report need different kinds of app context. For instance, generating Steps to Reproduce depends mainly on GUI interactions and the buggy screen itself, while generating Observed Behavior and Expected Behavior benefits from generated Steps to Reproduce and buggy screen descriptions.

Paper summary: Meng: Hmm, so they're not just feeding one giant blob of app data to the LLM for everything? That sounds like it could be computationally heavy if we have millions of screens and interactions.

Tom: That’s where their methodology gets smart. They developed a data-driven way to figure out which combinations of app-specific information actually work best for generating those specific components, finding that the combination "Interac" plus "Scrn" plus "BuggyScrn" was the best for generating Steps to Reproduce, achieving an F1 score of eighty-eight point four nine on that metric <ref:2604.01148#pg1>.

Jane: That's a concrete number showing they found a sweet spot for their strategy, and they found that different components benefit from different pieces of context, which is really important because it stops them from just guessing what information the LLM needs.

Lalam: They use LLM-based reasoning through zero-shot task decomposition prompting to generate the final high-quality bug reports after linking all that execution information together. This selective combination of data is their main contribution, allowing the model to focus on what matters for each specific output, which is what they call a novel LLM-guided reasoning strategy.

Tom: The results they got are pretty compelling too. BUGSCRIBE generated higher quality bug report components than the original reports and three other LLM-based baselines. They saw improvements in S2R quality by forty-four point one percent to over eighty-two point three percent, and OB and EB quality improved by about four point eight percent to thirty-five point two percent <ref:2604.01148#pg1>.

Meng: Those numbers sound significant, but how do we know those results hold up when we look at real traffic? I worry that performance on a test set doesn't always translate to production environments where reports are much messier.

Lu: The authors admit that the performance gains are sensitive to low-quality user reports, specifically vague steps, which can cause BUGSCRIBE to produce a shorter but less optimal path in those cases. They also flag that ambiguities in buggy screen references pop up because BUGSCRIBE often uses generic names instead of the exact UI wording.

Jane: So even though it's powerful, it has limitations where poor input quality can degrade the output, and they are worried about using generic screen names instead of the precise UI text. That makes sense for real-world app data where naming conventions aren't perfect.

Paper summary: Tom: Their future work is to tackle those naming issues by incorporating visual information from UI screenshots using multimodal models, which should help improve how the system names screens more accurately, and they also acknowledge that running BSGPT costs about three minutes per bug report.

Meng: Three minutes per report? That's a significant operational cost if we have high volume reporting. What does that mean for deployment speed?

Jane: It means the computational cost is there, but they are showing that the output quality is substantially better than existing methods, especially in Steps to Reproduce where they generated five hundred thirty-two correct steps compared to about one hundred ninety-six of those from other baselines <ref:2604.01148#pg1>.

Lalam: They really emphasize that their approach consistently produces higher-quality bug reports than existing methods that rely mainly on the text in the bug report because it connects the report content with relevant app execution elements through this component-specific grounding strategy.

Tom: So, to wrap up, BUGSCRIBE is an LLM approach that uses a graph of screen interactions and UI metadata to ground its reasoning for generating accurate Observed Behavior, Expected Behavior, and Steps to Reproduce by connecting them directly to the app's execution details.

Jane: And it shows that tailoring the context you give the LLM for different report components makes a real difference in quality, even if user reports are imperfect. That connects back to how we can build better systems that understand user intent more deeply.

Lu: The implication is that we need to move beyond just text analysis and start grounding language generation with structured execution data from the application itself, which is exactly what this paper proposes with BUGSCRIBE.

Meng: It sounds like it’s a good proof of concept for how LLMs can be used as reasoning engines when you have the right kind of external, structured context available to them.

Tom: Yeah, it's about making sure the AI doesn't just sound fluent; it has to actually know where the bug is happening in the app’s actual running state.

Jane: We can see this approach being applied wherever complex user input needs to be translated into structured, executable steps for automated testing or developer workflows.

Conclusion: Jane: So we've been talking about BUGSCRIBE, which is this LLM approach for automatically improving Android app bug reports by connecting user descriptions to actual app execution data like screen interactions and UI metadata.

Tom: Right, it basically takes a messy bug report and uses that technical context—the graph of screens and how they connect—to generate better versions of the observed behavior, expected behavior, and those steps to reproduce.

Lu: The authors found that you need different kinds of app data for different parts of the report; like they used GUI interactions mainly for writing the steps to reproduce.

Meng: But what's the actual impact here if I'm just shipping models at scale? How does this translate from a research paper to something I can actually deploy without it eating up my entire infrastructure budget?

Jane: The main thing is that even if you have low-quality reports, BUGSCRIBE still produces higher quality components than standard methods or other LLM baselines.

Tom: They showed improvements in steps to reproduce quality of about forty point one percent to over eighty two point three percent. That's a real jump in how accurately the bug can be described for a developer.

Lu: But they also flagged that if the user's original report is really vague, like they just write "it doesn't work," BUGSCRIBE might produce a shorter but less useful set of steps because it can’t find enough good context.

Meng: So it’s not a perfect fix for every single bad report yet. What about those limitations? What are the authors worried about when they talk about this system?

Jane: They're worried that the system often uses generic screen names instead of the exact wording on the screen, and that’s another hurdle for getting super accurate context.

Tom: Their future plan is to use visual information from screenshots with multimodal models to fix those naming issues so it can be more precise.

Lu: And they did admit the computational cost for running this system, which they call BSGPT, is about three minutes per bug report on average.

Jane: So that's a big time investment, but the core message is that grounding the LLM with actual app execution data makes it much better at generating those critical bug details than just reading the text alone.

Tom: It shows us how to make AI reasoning actually tethered to the physical reality of how an application runs on a device.

Jane: And if this works out, it means we can move past just analyzing text and start grounding language generation with structured data that comes directly from the app itself.

Antu Saha, Atish Kumar Dipongkor, Sam Bennett, Kevin Moran, Andrian Marcus, Oscar Chaparro

William & Mary University · University of Central Florida Department of Computer Science

cs.SE

Submitted: 2026-04-01

Updated: 2026-10-30

Comments: 12 pages, 6 figures

Code: https://github.com/netmackan/ATimeTracker

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: Most defects in mobile applications are visually observable on the device screen.

Key concepts

BUGSCRIBE
This is an LLM-powered approach designed to enhance Android bug reports. It works by connecting information from a user's bug report with actual execution data from the application's UI. The goal is to automatically infer and generate complete and correct descriptions of what happened, what should have happened, and how to reproduce the issue.
Component-Specific Grounding Strategy
This strategy involves giving different parts of the bug report generation task specific, relevant context from the app. Instead of feeding all data to the LLM at once, it focuses on providing only the most pertinent app execution information for generating each part (like S2R or OB), leading to better results.
Execution Model Graph
This is a structure built by mapping out all possible screens and GUI interactions within an application, complete with metadata. This graph helps the system understand how different parts of the app connect, allowing it to find the most relevant execution elements when trying to link them to a bug report.

Terminology

Summary

Most defects in mobile applications are visually observable on the device screen. p. 0

Where the paper says it

edu Oscar Chaparro William & Mary Williamsburg, Virginia, USA oscarch@wm. edu Abstract—Most defects in mobile applications are visually observable on the device screen.

This paper introduces BUGSCRIBE, an LLM-powered approach that links bug report information with app-specific UI execution information to infer and generate accurate, complete, and correct Observed Behavior (OB), Expected Behavior (EB), and Steps to Reproduce (S2Rs). p. 0
Where the paper says it

We formulate automatic bug report enhancement as the problem of connecting user-written bug reports with application execution. We present BUGSCRIBE, an LLM-powered approach that links bug report information with app-specific UI execution information to infer and generate accurate, complete, and correct Observed Behavior (OB), Expected Behavior (EB), and Steps to Reproduce (S2Rs).

Problem Formulation

The core challenge in enhancing these reports is therefore not merely improving the bug description, but recovering the missing connections between the reported bug information and the application’s underlying UI execution needed to produce a complete and accurate description of the defect. The authors formulate automatic bug report enhancement as a problem of connecting user-written bug reports with application execution. p. 0

Where the paper says it

This challenge is particularly prominent for UI-centric defects, where the relevant application behavior is difficult for end users to describe precisely. We formulate automatic bug report enhancement as the problem of connecting user-written bug reports with application execution.

This approach leverages LLM reasoning together with app-specific execution information to infer and generate accurate, complete, and correct OB, EB, and S2Rs. p. 0
Where the paper says it

We formulate automatic bug report enhancement as the problem of connecting user-written bug reports with application execution. We present BUGSCRIBE, an LLM-powered approach that links bug report information with app-specific UI execution information to infer and generate accurate, complete, and correct Observed Behavior (OB), Expected Behavior (EB), and Steps to Reproduce (S2Rs).

BUGSCRIBE Architecture

BUGSCRIBE employs a component-specific grounding strategy that provides the most relevant context to an LLM for generating each bug report component. The system constructs a graph-based execution model of app screens and GUI interactions enriched with UI metadata. p. 1

Where the paper says it

BUGSCRIBE augments existing reports with accurate, correct, and complete OB, EB, and S2R descriptions through a pipeline that connects bug report content with app-specific execution information. Specifically, BUGSCRIBE constructs a graph-based execution model of app screens and GUI interactions enriched with UI metadata; identifies OB, EB, and S2R sentences in the original report; links this information to relevant app execution elements; and generates high-quality bug report components using component-specific prompts.

It then identifies OB, EB, and S2R sentences in the original report and links this information to relevant app execution elements. p. 1
Where the paper says it

BUGSCRIBE augments existing reports with accurate, correct, and complete OB, EB, and S2R descriptions through a pipeline that connects bug report content with app-specific execution information. Specifically, BUGSCRIBE constructs a graph-based execution model of app screens and GUI interactions enriched with UI metadata; identifies OB, EB, and S2R sentences in the original report; links this information to relevant app execution elements; and generates high-quality bug report components using component-specific prompts.

The final step generates high-quality bug reports by leveraging LLM-based reasoning via zero-shot, task-decomposition prompting. p. 2
Where the paper says it

, GUI interactions) used as context for the LLM (Section III-B). 3) Bug Report Generation: it generates high-quality bug reports, including a title, OB, EB, and S2Rs, leveraging the app-specific information and LLM-based reasoning via zero-shot, task-decomposition prompting (Section III-C).

Contextual Information Strategy

The authors developed a data-driven methodology to identify which combinations of app-specific information are most effective for generating OB, EB, and S2Rs. They found that different components benefit from different app-specific information: S2R generation depends primarily on GUI interactions and the buggy screen, whereas OB and EB generation benefit from generated S2Rs, buggy screen descriptions, and screen-level context. p. 1

Where the paper says it

Using this quality model and manually curated ground truth for 10 bug reports from 9 Android apps, we evaluated eight context configurations for S2R and OB/EB generation. Our results show that different components benefit from different app-specific information: S2R generation depends primarily on GUI interactions and the buggy screen, whereas OB and EB generation benefit from generated S2Rs, buggy screen descriptions, and screen-level context.

Specifically, for S2R generation, the configuration Interac. p. 6
Where the paper says it

Figure 3 shows the S2R quality results of the four configurations for S2R generation, averaged across the three executions. All three context-aware configurations substantially outperform the no-information configuration, with F1 increasing from 72.

+Scrn.+BuggyScrn. p. 5
Where the paper says it

4) Interac. +BuggyScrn.

p. 5
Where the paper says it

4) Interac. +BuggyScrn.

achieved the best performance in terms of F1 score. p. 6
Where the paper says it

over three runs) achieved consistent performance across the runs. For instance, the F1 score’s std.

Empirical Evaluation Results

BUGSCRIBE generates higher-quality bug report components than the original reports and three LLM-based baselines, improving S2R quality by 44.1%–82.3% and OB/EB quality by 3. p. 0

Where the paper says it

Automatically Enhancing the Quality of Android App Bug Reports Antu Saha William & Mary Williamsburg, Virginia, USA asaha02@wm. edu Atish Kumar Dipongkor University of Central Florida Orlando, Florida, USA atish.

8%–35.2%. In the test set evaluation, BUGSCRIBE clearly outperforms all baselines and the original bug reports across all metrics, particularly in S2R generation where it generates substantially more correct steps (532, 538 vs. p. 7
Where the paper says it

The table shows that BUGSCRIBE (both BSGPT and BSCLAUDE) clearly outperforms all baselines and the original bug reports across all metrics. In particular, it generates substantially more correct steps (532, 538 vs.

196, 243, 345, 312). p. 7
Where the paper says it

In particular, it generates substantially more correct steps (532, 538 vs. 196, 243, 345, 312) while significantly reducing both extra and missing steps (e.

The best configuration for S2R generation was Interac. p. 6
Where the paper says it

Figure 3 shows the S2R quality results of the four configurations for S2R generation, averaged across the three executions. All three context-aware configurations substantially outperform the no-information configuration, with F1 increasing from 72.

+Scrn.+BuggyScrn., achieving an F1 score of 88. p. 5
Where the paper says it

For S2Rs, we assigned three quality labels: 1) Correct Step (CS): the generated step is found in the ground truth S2Rs (True Positive or TP), 2) Extra Step (ES): the generated step is not found in the ground truth S2Rs (False Positive or FP, and 3) Missing Step (MS): a ground truth S2R is not found in the generated steps (False Negative or FN). Using the quality labels assigned to the S2Rs, we computed precision, recall, and F1 score to measure S2R quality.

49.

Limitations and Future Work

The analysis revealed that the performance gains are sensitive to low-quality user reports, such as vague steps, which can lead BUGSCRIBE to produce a shorter but suboptimal path. Furthermore, ambiguities in buggy screen references arise because BUGSCRIBE often uses generic names rather than the exact UI wording. p. 9

Where the paper says it

These ambiguities arise because BUGSCRIBE often uses generic, functionalitybased names rather than the exact UI wording. For instance, in bug report #154 for the Camfahrplan app [41], the correct buggy screen is “schedule screen”, whereas BSCLAUDE refers to it as “event details screen”.

Future work will incorporate visual information from UI screenshots via multimodal models to improve screen naming. p. 9
Where the paper says it

This discrepancy stems from limitations in the app metadata used to build the app execution model, which may omit or differ from the uservisible screen names, prompting BUGSCRIBE to infer names from the UI screen descriptions used as context. To address this, our future work will incorporate visual information from UI screenshots via multimodal models to improve screen naming.

The computational cost for BSGPT is approximately 3 minutes per bug report. p. 9
Where the paper says it

3) Computational Cost: We measured the end-to-end cost of BUGSCRIBE. On average, BSGPT processes 121.

Conclusion

BUGSCRIBE enables LLMs to generate accurate, complete, and correct OB, EB, and S2Rs by connecting report information to relevant app execution elements through a component-specific grounding strategy. The approach consistently produces higher-quality bug reports than existing methods that rely primarily on the bug report text. p. 0

Where the paper says it

Prior work has proposed techniques to assess bug report quality [9], [6], identify missing information [10], [11], provide feedback to reporters [9], [6], support interactive report construction [2], [12], [13], [14], and restructure bug reports [15]. While these approaches help identify quality problems or rewrite existing reports, they either leave users responsible for correcting the report or generate content primarily from the original report text.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Automated bug report enhancement system that connects user-written bug reports with application execution information to generate accurate, complete, and correct Observed Behavior (OB), Expected Behavior (EB), and Steps to Reproduce (S2Rs). This system can infer and generate high-quality OB/EB/S2R descriptions by leveraging a component-specific grounding strategy.

  2. A data-driven methodology for identifying the specific app information elements needed by the LLM’s reasoning strategy to link report and app execution data. This allows the AI to dynamically select which combination of app context—such as GUI interactions, screen descriptions, or buggy screens—is most effective for generating specific components like S2Rs or OB/EB.

  3. An empirical evaluation framework that demonstrates BUGSCRIBE's superiority by comparing it against LLM-based baselines that do not use app-specific information. This system can achieve significant improvements in S2R quality, showing relative increases of 44.1% to 82.3% and OB/EB quality improvements of 3.8% to 35.2%.

  4. A pipeline capable of constructing a graph-based execution model of app screens and GUI interactions enriched with UI metadata for every application analyzed. This model allows the system to represent the app's flow as a directed graph where nodes are screens and edges are user interactions, enabling precise pathfinding for S2R generation.

  5. A context extraction module that annotates user-submitted bug reports to specify OB, EB, and S2Rs sentences while simultaneously extracting three app information elements: app screen information, GUI interactions, and the buggy screen. This module feeds the most relevant context to the LLM for generating specific report components based on a component-specific grounding strategy.

  6. A generation engine that employs zero-shot task decomposition prompting to synthesize atomic S2Rs from identified GUI interactions within an execution model and map them to plausible reproduction scenarios leading to the buggy screen. This enables the system to generate detailed, enumerated lists of reproduction steps that correspond directly to specific GUI actions.

  7. A generation engine that uses GPT-5.4 or CLAUDEOPUS-4.6 as a state-of-the-art reasoning LLM, guided by context, to generate high-quality bug reports including a one line bug summary, OB, EB, and S2Rs in a structured format. This system produces outputs that are more complete and accurate than original reports or other LLM baselines.

Abstract

Most defects in mobile applications are visually observable on the device screen. Since automated mechanisms for detecting and reporting such defects are often unavailable, users, testers, and developers must manually submit bug reports. However, these reports are frequently incomplete, ambiguous, or inaccurate, often lacking the information needed to understand, reproduce, and diagnose defects. We formulate automatic bug report enhancement as the problem of connecting user-written bug reports with application execution. We present BUGSCRIBE, an LLM-powered approach that links bug report information with app-specific UI execution information to infer and generate accurate, complete, and correct Observed Behavior (OB), Expected Behavior (EB), and Steps to Reproduce (S2Rs). To support BUGSCRIBE's design and evaluation, we develop a bug report quality model and use it to identify the most effective context for each component. We evaluate BUGSCRIBE on 48 bug reports from 26 Android applications with manually constructed ground truth. Our results show that BUGSCRIBE generates higher-quality bug report components than the original reports and three LLM-based baselines, improving S2R quality by 44.2%--84.7%, and OB/EB quality by up to 35.2%. We further show that BUGSCRIBE-generated reports help an LLM-based agent reproduce Android bugs, improving the reproduction rate by 28.1%--57.7% compared to the original reports and the baselines.

Sources

Related papers