Automatically Enhancing the Quality of Android App Bug Reports

summary

The gist

Most defects in mobile applications are visually observable on the device screen.

In short

BUGSCRIBE is an LLM-powered system that improves Android bug reports by linking user descriptions with app execution data. It connects reported issues to specific UI interactions to automatically generate accurate Observed Behavior, Expected Behavior, and Steps to Reproduce. This means the system creates much higher quality bug reports than existing methods.

Key concepts

BUGSCRIBE
This is an LLM-powered approach designed to enhance Android bug reports. It works by connecting information from a user's bug report with actual execution data from the application's UI. The goal is to automatically infer and generate complete and correct descriptions of what happened, what should have happened, and how to reproduce the issue.
Component-Specific Grounding Strategy
This strategy involves giving different parts of the bug report generation task specific, relevant context from the app. Instead of feeding all data to the LLM at once, it focuses on providing only the most pertinent app execution information for generating each part (like S2R or OB), leading to better results.
Execution Model Graph
This is a structure built by mapping out all possible screens and GUI interactions within an application, complete with metadata. This graph helps the system understand how different parts of the app connect, allowing it to find the most relevant execution elements when trying to link them to a bug report.

Terminology used across episodes

This episode discusses

The paper

Automatically Enhancing the Quality of Android App Bug Reports · Read on arXiv

Antu Saha, Atish Kumar Dipongkor, Sam Bennett, Kevin Moran, Andrian Marcus, Oscar Chaparro

William & Mary University · University of Central Florida Department of Computer Science

Most defects in mobile applications are visually observable on the device screen. Since automated mechanisms for detecting and reporting such defects are often unavailable, users, testers, and developers must manually submit bug reports. However, these reports are frequently incomplete, ambiguous, or inaccurate, often lacking the information needed to understand, reproduce, and diagnose defects. We formulate automatic bug report enhancement as the problem of connecting user-written bug reports with application execution. We present BUGSCRIBE, an LLM-powered approach that links bug report information with app-specific UI execution information to infer and generate accurate, complete, and correct Observed Behavior (OB), Expected Behavior (EB), and Steps to Reproduce (S2Rs). To support BUGSCRIBE's design and evaluation, we develop a bug report quality model and use it to identify the most effective context for each component. We evaluate BUGSCRIBE on 48 bug reports from 26 Android applications with manually constructed ground truth. Our results show that BUGSCRIBE generates higher-quality bug report components than the original reports and three LLM-based baselines, improving S2R quality by 44.2%--84.7%, and OB/EB quality by up to 35.2%. We further show that BUGSCRIBE-generated reports help an LLM-based agent reproduce Android bugs, improving the reproduction rate by 28.1%--57.7% compared to the original reports and the baselines.

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Automatically Enhancing the Quality of Android App Bug Reports".

Jane: Here's the gist of the paper. Most defects in mobile applications are visually observable on the device screen.

Tom: This paper introduces BUGSCRIBE, an LLM-powered approach that links bug report information with app-specific UI execution information to infer and generate accurate, complete, and correct Observed Behavior (OB), Expected Behavior (EB), and Steps to Reproduce (S2Rs).

Jane: Problem Formulation The core challenge in enhancing these reports is therefore not merely improving the bug description, but recovering the missing connections between the reported bug information and the application’s underlying UI execution needed to produce a complete and accurate description of the defect.

Tom: The authors formulate automatic bug report enhancement as a problem of connecting user-written bug reports with application execution.

Paper summary: Jane: So Tom, we're talking about this paper called "Automatically Enhancing the Quality of Android App Bug Reports." It seems to be tackling the problem that most bugs in mobile apps are just visually observable on the screen, but users often write bug reports that are incomplete or ambiguous because they miss the crucial app-specific execution information needed to reproduce them.

Tom: Yeah, exactly. The authors set out to solve this by connecting what a user writes in a bug report with the actual way an app runs on the device and using that connection to generate better versions of things like Observed Behavior, Expected Behavior, and Steps to Reproduce.

Lu: The core idea is that it's not just about making the text sound better; it's about recovering those missing links between what a user reports and the technical execution details of the app. They frame it as connecting user-written bug reports with application execution information to infer and generate accurate descriptions of Observed Behavior, Expected Behavior, and Steps to Reproduce.

Meng: So, if I’m running this on my end at scale, what's the big deal? Is this just a nice way to write reports for testers?

Jane: It's more than that. The paper points out that the challenge is connecting the reported bug information with the application’s underlying UI execution needed to produce a complete and accurate description of the defect, and they formulate automatic bug report enhancement as a problem of connecting user-written bug reports with application execution.

Tom: Right. So instead of just editing the text, they're building something that looks at both sides—the report and the app’s behavior simultaneously—to pull out those missing connections using an LLM and some app-specific context.

Lalam: The architecture they use is called BUGSCRIBE, which employs a component-specific grounding strategy to give the LLM the most relevant context for generating each part of the bug report. It builds a graph-based execution model of app screens and GUI interactions, enriching it with UI metadata, and then links sentences from the original report to those relevant app execution elements.

Lu: That component-specific grounding is key. They discovered that different parts of the bug report need different kinds of app context. For instance, generating Steps to Reproduce depends mainly on GUI interactions and the buggy screen itself, while generating Observed Behavior and Expected Behavior benefits from generated Steps to Reproduce and buggy screen descriptions.

Paper summary: Meng: Hmm, so they're not just feeding one giant blob of app data to the LLM for everything? That sounds like it could be computationally heavy if we have millions of screens and interactions.

Tom: That’s where their methodology gets smart. They developed a data-driven way to figure out which combinations of app-specific information actually work best for generating those specific components, finding that the combination "Interac" plus "Scrn" plus "BuggyScrn" was the best for generating Steps to Reproduce, achieving an F1 score of eighty-eight point four nine on that metric <ref:2604.01148#pg1>.

Jane: That's a concrete number showing they found a sweet spot for their strategy, and they found that different components benefit from different pieces of context, which is really important because it stops them from just guessing what information the LLM needs.

Lalam: They use LLM-based reasoning through zero-shot task decomposition prompting to generate the final high-quality bug reports after linking all that execution information together. This selective combination of data is their main contribution, allowing the model to focus on what matters for each specific output, which is what they call a novel LLM-guided reasoning strategy.

Tom: The results they got are pretty compelling too. BUGSCRIBE generated higher quality bug report components than the original reports and three other LLM-based baselines. They saw improvements in S2R quality by forty-four point one percent to over eighty-two point three percent, and OB and EB quality improved by about four point eight percent to thirty-five point two percent <ref:2604.01148#pg1>.

Meng: Those numbers sound significant, but how do we know those results hold up when we look at real traffic? I worry that performance on a test set doesn't always translate to production environments where reports are much messier.

Lu: The authors admit that the performance gains are sensitive to low-quality user reports, specifically vague steps, which can cause BUGSCRIBE to produce a shorter but less optimal path in those cases. They also flag that ambiguities in buggy screen references pop up because BUGSCRIBE often uses generic names instead of the exact UI wording.

Jane: So even though it's powerful, it has limitations where poor input quality can degrade the output, and they are worried about using generic screen names instead of the precise UI text. That makes sense for real-world app data where naming conventions aren't perfect.

Paper summary: Tom: Their future work is to tackle those naming issues by incorporating visual information from UI screenshots using multimodal models, which should help improve how the system names screens more accurately, and they also acknowledge that running BSGPT costs about three minutes per bug report.

Meng: Three minutes per report? That's a significant operational cost if we have high volume reporting. What does that mean for deployment speed?

Jane: It means the computational cost is there, but they are showing that the output quality is substantially better than existing methods, especially in Steps to Reproduce where they generated five hundred thirty-two correct steps compared to about one hundred ninety-six of those from other baselines <ref:2604.01148#pg1>.

Lalam: They really emphasize that their approach consistently produces higher-quality bug reports than existing methods that rely mainly on the text in the bug report because it connects the report content with relevant app execution elements through this component-specific grounding strategy.

Tom: So, to wrap up, BUGSCRIBE is an LLM approach that uses a graph of screen interactions and UI metadata to ground its reasoning for generating accurate Observed Behavior, Expected Behavior, and Steps to Reproduce by connecting them directly to the app's execution details.

Jane: And it shows that tailoring the context you give the LLM for different report components makes a real difference in quality, even if user reports are imperfect. That connects back to how we can build better systems that understand user intent more deeply.

Lu: The implication is that we need to move beyond just text analysis and start grounding language generation with structured execution data from the application itself, which is exactly what this paper proposes with BUGSCRIBE.

Meng: It sounds like it’s a good proof of concept for how LLMs can be used as reasoning engines when you have the right kind of external, structured context available to them.

Tom: Yeah, it's about making sure the AI doesn't just sound fluent; it has to actually know where the bug is happening in the app’s actual running state.

Jane: We can see this approach being applied wherever complex user input needs to be translated into structured, executable steps for automated testing or developer workflows.

Conclusion: Jane: So we've been talking about BUGSCRIBE, which is this LLM approach for automatically improving Android app bug reports by connecting user descriptions to actual app execution data like screen interactions and UI metadata.

Tom: Right, it basically takes a messy bug report and uses that technical context—the graph of screens and how they connect—to generate better versions of the observed behavior, expected behavior, and those steps to reproduce.

Lu: The authors found that you need different kinds of app data for different parts of the report; like they used GUI interactions mainly for writing the steps to reproduce.

Meng: But what's the actual impact here if I'm just shipping models at scale? How does this translate from a research paper to something I can actually deploy without it eating up my entire infrastructure budget?

Jane: The main thing is that even if you have low-quality reports, BUGSCRIBE still produces higher quality components than standard methods or other LLM baselines.

Tom: They showed improvements in steps to reproduce quality of about forty point one percent to over eighty two point three percent. That's a real jump in how accurately the bug can be described for a developer.

Lu: But they also flagged that if the user's original report is really vague, like they just write "it doesn't work," BUGSCRIBE might produce a shorter but less useful set of steps because it can’t find enough good context.

Meng: So it’s not a perfect fix for every single bad report yet. What about those limitations? What are the authors worried about when they talk about this system?

Jane: They're worried that the system often uses generic screen names instead of the exact wording on the screen, and that’s another hurdle for getting super accurate context.

Tom: Their future plan is to use visual information from screenshots with multimodal models to fix those naming issues so it can be more precise.

Lu: And they did admit the computational cost for running this system, which they call BSGPT, is about three minutes per bug report on average.

Jane: So that's a big time investment, but the core message is that grounding the LLM with actual app execution data makes it much better at generating those critical bug details than just reading the text alone.

Tom: It shows us how to make AI reasoning actually tethered to the physical reality of how an application runs on a device.

Jane: And if this works out, it means we can move past just analyzing text and start grounding language generation with structured data that comes directly from the app itself.

More episodes

← Home