Automated Root-Cause Subclassification and No-Code Fix Generation for Invalid Bug Reports

arXiv:2605.17561 · cs.SE, cs.AI, cs.MA · Submitted 2026-05-17 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Automated Root-Cause Subclassification and No-Code Fix Generation for Invalid Bug Reports".

Jane: Here's the gist of the paper. Our goal is to introduce a standardized taxonomy for root-cause oriented invalid bug report subclassification and perform experiments to test the accuracy of various approaches on invalid subclassification based on root causes and no-code fix generation; this study shows that retrieval augmented generation improves invalid subclassification performance, while no-code fix generation benefits more from agentic web search Pruning redundant reasoning tokens The paper addresses the waste of resources caused by manually determining the root cause of invalid bug reports and providing actionable resolutions by the customer support team The objective is to introduce a standardized taxonomy for root-cause oriented invalid bug report subclassification and perform experiments to test the accuracy of various approaches on invalid subclassification based on root causes and no-code fix generation The study evaluates vanilla large language models (LLMs), Retrieval Augmented Generation (RAG), and agentic web search to identify invalid subclasses and generate no-code fixes How the training signal is built The methodology involves designing a multi-step evaluation framework called IssueSupport that leverages LLMs to subclassify invalid bug reports and generate no-code fixes This framework first establishes a baseline (vanilla) setting, evaluating the inherent capabilities of both proprietary and open-weight LLMs to map bug reports to predefined seven invalid subclasses The taxonomy organizes these reports by three primary dimensions: Invalid Subclass (Root Cause), Description, and High-Level No-Code Fix Guideline Where the gains come from The results show that retrieval augmented generation achieves the highest overall performance with 0.66 weighted F1, slightly outperforming vanilla LLMs at 0.65 and agentic web search at 0.64 For no-code fix generation, agentic web search achieves the highest overall Judge LLM success rate at 68.9%, compared to 64.4% for RAG applications and 64.9% for vanilla LLMs The generated no-code fixes demonstrate the potential to automate customer support workflows, reduce manual triage effort, and provide immediate, actionable guidance to users within bug tracking systems Evaluation of experimental settings The study evaluates four distinct methodologies: (1) Vanilla LLM Pipeline, (2) Vanilla LLM Pipeline Without Prior Invalid Subclass Information, (3) RAG Pipeline, and (4) Agentic Web Search Pipeline The evaluation for RQ2 uses a dual-metric approach employing BERTScore and a Judge-LLM success rate to assess the semantic fidelity and functional utility of generated no-code fixes The comparison between the Without Priors and Vanilla pipelines reveals that omitting the root-cause subclass information causes only a marginal decrease in overall performance for most models Implications for practitioners Practitioners should start with project-grounded retrieval when reliable internal data is available, such as issue histories, internal wikis, or support playbooks Use Context Engineering in Stages by anchoring the model in trusted project-specific corpora before adding live web search for time-sensitive facts For auditability, retrieved URLs and chunk identifiers should be logged for both RAG and agentic web search outputs The lack of a unified evaluation framework for no-code fix generation remains a critical gap, suggesting the need for standardized benchmarks that combine invalid subclassification accuracy with no-code fix quality The study highlights the importance of manual assessment to capture readability and practical usefulness in AI-generated no-code fixes Threats to validity External validity is limited because the study is conducted exclusively on the Brave Browser repository, which may limit the generalizability of the findings Internal validity is threatened by LLM training processes and potential exposure to benchmark data during pretraining The use of RAG introduces the risk of incorporating information that became available after the original bug report was created, necessitating temporal filtering Agentic web search may suffer from data retrieval mismatches between issue creation dates and live web results, posing a threat to validity Construct validity is also questioned because the no-code fixes are inferred from issue discussions rather than being verified through controlled reproduction in a sandbox environment The BERTScore metric may overestimate the alignment between generated and reference fixes, leading to overly optimistic evaluations This study introduces and evaluates an automated framework for subclassifying invalid bug reports based on root causes and generating corresponding no-code fixes For subclassification, RAG achieves the highest overall performance with 0.66 weighted F1, slightly outperforming vanilla LLMs at 0.65 and agentic web search at 0.64 For no-code fix generation, agentic web search achieves the highest overall success rate at 68.9%, outperforming vanilla LLMs at 63.1% For future work, the authors aim to evaluate the proposed framework across a broader range of repositories to assess generalizability The study suggests that practitioners should use project-grounded retrieval first, and then add live web search when the issue depends on upstream systems or time-sensitive facts The authors call for standardized evaluation frameworks that combine invalid subclassification accuracy with no-code fix quality The study concludes by summarizing key outcomes and suggesting directions for future work in automated no-code fix systems REFERENCES

1: H.

Tom: Krasner, “The cost of poor software quality in the us: A 2022 report,” Consortium for Information & Software Quality (CISQ), Tech.

Jane: Rep., Dec.

Tom: 2022, accessed 2025-11-07

3: M.

Paper summary: Tom: So, we're talking about this paper today, "Automated Root-Cause Subclassification and No-Code Fix Generation for Invalid Bug Reports." Basically, the main problem they're tackling is that customer support teams waste a lot of time manually figuring out if a bug report is actually valid or not.

Jane: Right. They introduce this standardized system for classifying those invalid reports based on their root cause and then generate no-code fixes to help reporters solve those issues. It seems like the big idea here is to stop that manual waste of resources.

Lu: Exactly, the objective they set out is to create a standardized taxonomy for root-cause oriented invalid bug report subclassification and test different approaches on both that classification and the no-code fix generation based on those root causes.

Meng: So they're not just looking at one way to do this; they're comparing vanilla large language models, Retrieval Augmented Generation, and agentic web search to see which ones are best at identifying those invalid subclasses and creating the fixes.

Lalam: I see. And the results show that retrieval augmented generation improves the invalid subclassification performance with a weighted F1 score of zero point six six, slightly better than vanilla large language models with zero point six five, and agentic web search at zero point six four >

Tom: That's interesting. So for the no-code fix generation part, agentic web search gets the highest overall Judge LLM success rate at sixty eight point nine percent compared to sixty four point four percent for retrieval augmented generation and sixty four point nine percent for vanilla large language models >

Jane: The implication there is that agentic web search seems really good at generating those actual no-code fixes, which could seriously help automate customer support workflows and give users immediate guidance right in their bug tracking systems.

Lu: And the study evaluates four different settings: a vanilla pipeline, one without prior invalid subclass information, an RAG pipeline, and that agentic web search pipeline >

Meng: The ablation study comparing the version without priors to the vanilla baseline shows that omitting the root cause subclass information only causes a marginal decrease in overall performance for most models >

Tom: Which means knowing the root cause helps a bit with classification, but it's not like it's completely useless if you don't have those specific labels ready >

Jane: And they used a dual-metric approach for the no-code fix generation part, using BERTScore and a Judge LLM success rate to check both the meaning and the actual usefulness of what was generated >

Lu: The study points out that for no-code fix generation, agentic web search achieves the highest overall success rate at sixty eight point nine percent, while RAG applications get sixty four point four percent and vanilla models get sixty three point one percent >

Meng: That suggests agentic web search is particularly strong when it comes to generating those helpful fixes compared to the other methods tested >

Tom: So what does this mean for us in practice? The authors suggest starting with project-grounded retrieval if you have reliable internal data like issue histories or wikis, and then adding live web search only when the issue depends on upstream systems or time-sensitive facts >

Jane: They also mention that for auditability, you should log those retrieved URLs and chunk identifiers for both the RAG and agentic web search outputs >

Lu: They also flag a critical gap in the field: there's no unified evaluation framework specifically for no-code fix generation, which suggests we need standardized benchmarks that combine invalid subclassification accuracy with no-code fix quality >

Meng: I agree. And the study itself notes that you should do manual assessment to capture how readable and practically useful those AI generated no-code fixes actually are, because the metrics alone don't always catch that nuance >

Tom: To wrap this up, it seems like for classification, RAG is leading with zero point six six weighted F1, but for the actual fix generation part, agentic web search is showing its strength with a success rate of sixty eight point nine percent >

Jane: So the main message from this paper, "Automated Root-Cause Subclassification and No-Code Fix Generation for Invalid Bug Reports," is that retrieval augmented generation provides the best overall performance for classifying those invalid reports based on their root causes >

Lu: And for generating no-code fixes, agentic web search performs best when you need external context, but they also warn that external validity is limited because the study was only done on the Brave Browser repository >

Meng: I agree with that. Internal validity is also a concern because of how the LLMs were trained and whether they might have seen benchmark data during pretraining >

Tom: So for anyone looking to use this, you should start by using project-grounded retrieval when you have internal data, and then consider adding live web search later >

Jane: And remember that the authors stress that manual assessment is still necessary to truly judge the quality of those no-code fixes they produce >

Lu: The future work mentioned is testing this framework across a broader range of repositories to check how generally applicable it really is >

Meng: It sounds like they are setting up a path toward more automated support systems, but we still need those standardized benchmarks you mentioned to really trust the results on a large scale.

Conclusion: Tom: So, we're wrapping up this look at the paper "Automated Root-Cause Subclassification and No-Code Fix Generation for Invalid Bug Reports." The authors are trying to get rid of that manual work support teams do when they have a ton of invalid bug reports.

Jane: Exactly. It’s about using AI to standardize how we categorize these bad reports by their root cause and then automatically suggesting fixes that don't require coding.

Tom: Right. They tested several methods, like vanilla large language models, retrieval augmented generation, and agentic web search, to see what works best for both the classification and the fixing part.

Jane: And the results show that retrieval augmented generation is the strongest for classifying those reports, with a weighted score of zero point six six. But then they look at generating those no-code fixes.

Tom: That’s where agentic web search comes out on top for fix generation, with a success rate of sixty eight point nine percent compared to vanilla models. It seems like getting external information helps the AI write better guidance than just using what it already knows.

Jane: So what does this actually mean for support teams? It suggests that instead of manually sorting through thousands of reports, you can use this framework to get a much more accurate initial classification and a usable fix suggestion right away.

Tom: The authors also give some advice on how to start using this stuff in the real world, suggesting project-grounded retrieval first when you have internal data like issue histories.

Jane: And they’re also clear about the limitations, saying that you need to keep an eye out for things like external validity issues since they only tested it on one specific repository.

Tom: It boils down to this: we have a new way to automate the triage and fixing of buggy reports, but we still need better standards before everyone can trust it at scale.

Mahmut Furkan Gonän, Emre Dincı, Tevfik Emre Sungur, Eray Tuzຜn uຜ

Bilkent University

cs.SE, cs.AI, cs.MA

Submitted: 2026-05-17

Updated: 2026-11-10

Comments: Submitted to IEEE Transactions on Software Engineering (TSE) and currently under review

Code: https://github.com/brave/brave-browser6https:

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 74/100

The gist: Our goal is to introduce a standardized taxonomy for root-cause oriented invalid bug report subclassification and perform experiments to test the accuracy of various approaches on invalid…

Key concepts

Invalid Bug Report Subclassification
This process involves categorizing a bug report based on its underlying root cause. The paper proposes a standardized taxonomy with three dimensions: the specific invalid subclass (root cause), the description of the issue, and a high-level guideline for an automated no-code fix.
Retrieval Augmented Generation (RAG)
RAG enhances LLMs by allowing them to use external, reliable data sources—like internal issue histories or wikis—to improve their classification accuracy. The study found RAG provided the highest overall performance in identifying the correct invalid bug subclass.
Agentic Web Search
This approach uses an agent that can actively search the live web to find current, external information. This method proved superior for generating no-code fixes, suggesting that real-time external knowledge is crucial for creating actionable solutions.

Terminology

Summary

Our goal is to introduce a standardized taxonomy for root-cause oriented invalid bug report subclassification and perform experiments to test the accuracy of various approaches on invalid subclassification based on root causes and no-code fix generation; this study shows that retrieval augmented generation improves invalid subclassification performance, while no-code fix generation benefits more from agentic web search

Pruning redundant reasoning tokens

The paper addresses the waste of resources caused by manually determining the root cause of invalid bug reports and providing actionable resolutions by the customer support team The objective is to introduce a standardized taxonomy for root-cause oriented invalid bug report subclassification and perform experiments to test the accuracy of various approaches on invalid subclassification based on root causes and no-code fix generation The study evaluates vanilla large language models (LLMs), Retrieval Augmented Generation (RAG), and agentic web search to identify invalid subclasses and generate no-code fixes

How the training signal is built

The methodology involves designing a multi-step evaluation framework called IssueSupport that leverages LLMs to subclassify invalid bug reports and generate no-code fixes This framework first establishes a baseline (vanilla) setting, evaluating the inherent capabilities of both proprietary and open-weight LLMs to map bug reports to predefined seven invalid subclasses The taxonomy organizes these reports by three primary dimensions: Invalid Subclass (Root Cause), Description, and High-Level No-Code Fix Guideline

Where the gains come from

The results show that retrieval augmented generation achieves the highest overall performance with 0.66 weighted F1, slightly outperforming vanilla LLMs at 0. p. 0

Where the paper says it

Results: For subclassification, RAG achieves the highest overall performance with 0. 66 weighted F1, slightly outperforming vanilla LLMs at 0.

p. 0
Where the paper says it

Results: For subclassification, RAG achieves the highest overall performance with 0. 66 weighted F1, slightly outperforming vanilla LLMs at 0.

65 and agentic web search at 0. p. 0
Where the paper says it

We study how different configurations perform on a gold-standard benchmark we have created. Method: Using a manually curated benchmark for higher quality analysis, we experimented with vanilla large language models (LLMs), Retrieval Augmented Generation (RAG), and agentic web search to identify invalid subclasses and generate no-code fixes.

p. 0
Where the paper says it

We study how different configurations perform on a gold-standard benchmark we have created. Method: Using a manually curated benchmark for higher quality analysis, we experimented with vanilla large language models (LLMs), Retrieval Augmented Generation (RAG), and agentic web search to identify invalid subclasses and generate no-code fixes.

64 For no-code fix generation, agentic web search achieves the highest overall Judge LLM success rate at 68. p. 0
Where the paper says it

81 for Feature Request in Vanilla LLM, while Wrong Version remains the most challenging with scores between 0. For no-code fix generation, agentic web search achieves the highest overall Judge LLM success rate at 68.

9%, compared to 64. p. 0
Where the paper says it

For no-code fix generation, agentic web search achieves the highest overall Judge LLM success rate at 68. 9%, compared to 64.

4% for RAG applications and 64. p. 0
Where the paper says it

9%, compared to 64. 4% for RAG applications and 64.

9% for vanilla LLMs The generated no-code fixes demonstrate the potential to automate customer support workflows, reduce manual triage effort, and provide immediate, actionable guidance to users within bug tracking systems

Evaluation of experimental settings

The study evaluates four distinct methodologies: (1) Vanilla LLM Pipeline, (2) Vanilla LLM Pipeline Without Prior Invalid Subclass Information, (3) RAG Pipeline, and (4) Agentic Web Search Pipeline The evaluation for RQ2 uses a dual-metric approach employing BERTScore and a Judge-LLM success rate to assess the semantic fidelity and functional utility of generated no-code fixes The comparison between the Without Priors and Vanilla pipelines reveals that omitting the root-cause subclass information causes only a marginal decrease in overall performance for most models

Implications for practitioners

Practitioners should start with project-grounded retrieval when reliable internal data is available, such as issue histories, internal wikis, or support playbooks Use Context Engineering in Stages by anchoring the model in trusted project-specific corpora before adding live web search for time-sensitive facts For auditability, retrieved URLs and chunk identifiers should be logged for both RAG and agentic web search outputs The lack of a unified evaluation framework for no-code fix generation remains a critical gap, suggesting the need for standardized benchmarks that combine invalid subclassification accuracy with no-code fix quality The study highlights the importance of manual assessment to capture readability and practical usefulness in AI-generated no-code fixes

Threats to validity

External validity is limited because the study is conducted exclusively on the Brave Browser repository, which may limit the generalizability of the findings Internal validity is threatened by LLM training processes and potential exposure to benchmark data during pretraining The use of RAG introduces the risk of incorporating information that became available after the original bug report was created, necessitating temporal filtering Agentic web search may suffer from data retrieval mismatches between issue creation dates and live web results, posing a threat to validity Construct validity is also questioned because the no-code fixes are inferred from issue discussions rather than being verified through controlled reproduction in a sandbox environment The BERTScore metric may overestimate the alignment between generated and reference fixes, leading to overly optimistic evaluations

Conclusion

This study introduces and evaluates an automated framework for subclassifying invalid bug reports based on root causes and generating corresponding no-code fixes For subclassification, RAG achieves the highest overall performance with 0.66 weighted F1, slightly outperforming vanilla LLMs at 0.65 and agentic web search at 0.64 For no-code fix generation, agentic web search achieves the highest overall success rate at 68. p. 0

Where the paper says it

81 for Feature Request in Vanilla LLM, while Wrong Version remains the most challenging with scores between 0. For no-code fix generation, agentic web search achieves the highest overall Judge LLM success rate at 68.

9%, outperforming vanilla LLMs at 63. p. 0
Where the paper says it

Results: For subclassification, RAG achieves the highest overall performance with 0. 66 weighted F1, slightly outperforming vanilla LLMs at 0.

1% For future work, the authors aim to evaluate the proposed framework across a broader range of repositories to assess generalizability The study suggests that practitioners should use project-grounded retrieval first, and then add live web search when the issue depends on upstream systems or time-sensitive facts The authors call for standardized evaluation frameworks that combine invalid subclassification accuracy with no-code fix quality The study concludes by summarizing key outcomes and suggesting directions for future work in automated no-code fix systems

REFERENCES

[1] H. Krasner, “The cost of poor software quality in the us: A 2022 report,” Consortium for Information & Software Quality (CISQ), Tech. p. 2

Where the paper says it

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING 3 actionable no-code fixes. To isolate the impact of subclassification on the quality of generated resolutions (RQ2.

Rep., Dec. 2022, accessed 2025-11-07

[3] M. Laiq, N. bin Ali, J. Borstler, and E. Engstr ¨ om, “A data-driven ¨ approach for understanding invalid bug reports: An industrial case study,” Information and Software Technology, vol. p. 9

Where the paper says it

Since approach (2) is for ablation study purposes, it lacks subclass descriptions and guidelines for the no-code fixes in the system prompt. Conversely, the user prompt serves strictly as the data input layer, injecting only the raw markdowns of title and body of the bug report for vanilla LLM pipelines, and additionally retrieved information for RAG and agentic web search pipelines.

164, p. 107305

[5] K. Herzig, S. Just, and A. Zeller, “It’s not a bug, it’s a feature: How misclassification impacts bug prediction,” in 2013 35th International Conference on Software Engineering (ICSE), 2013

[7] J. Sun, “Why are bug reports invalid? p. 2

Where the paper says it

In order to define these root causes, researchers have conducted empirical analyses and proposed various taxonomies to subclassify invalid bug reports. Sun [7] attributed many invalid bug reports to testing errors or a lack of system knowledge.

” in 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation. IEEE, 2011

[8] Y. Su, P. Luarn, Y.-S. Lee, and S.-J. Yen, “Creating an invalid defect classification model using text mining on server development,” Journal of Systems and Software, vol. p. 8

Where the paper says it

The curated benchmark serves as a valuable asset for the software engineering research community. By providing a ground truth set of invalid bug reports paired with their specific no-code solutions, it enables the rigorous evaluation of automated classification models and solution recommendation systems, facilitating the development of tools that can alleviate the maintenance burden in open-source projects.

125, pp. 197–206

[9] S. Panichella, G. p. 3

Where the paper says it

(2023) [3] Su et al. (2017) [8] Panichella et al.

Canfora, and A. Di Sorbo, ““won’t we fix this issue? p. 0
Where the paper says it

In contrast, an external system dependency issue could be resolved by explaining that the issue originates from an upstream external issue (see Figure 1b). The customer support team can also consider these root causes while providing no-code fix solutions to the reporters.

” qualitative characterization and automated identification of wontfix issues on github,” Information and Software Technology, vol. p. 3
Where the paper says it

Panichella et al. [9] found that GitHub ”wontfix” issues are often just support questions or out-of-scope requests.

139, p. 106665

[10] U. B. Torun, M. T. Demircan, M. F. Gon, and E. p. 0

Where the paper says it

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING 1 Automated Root-Cause Subclassification and No-Code Fix Generation for Invalid Bug Reports Mahmut Furkan Gon¨ * , Emre Dinc¸* , Tevfik Emre Sungur* , and Eray Tuz¨ un¨ Abstract—Context: Issues faced when using software are reported in the form of bug reports. However, many bug reports are invalid, meaning they do not require code changes and are resolved with a no-code fix.

T ¨ uz¨ un, “Past, present, ¨ and future of bug tracking in the generative ai era,” ACM Transactions on Software Engineering and Methodology, 2026

[11] G. Yang, J. Ji, and J. Kim, “Enhanced bug priority prediction via priority-sensitive long short-term memory–attention mechanism,” Applied Sciences, vol. 15, no. 2, p. 633, 2025

[12] J. He, L. Xu, Y. Fan, Z. Xu, M. Yan, and Y. Lei, “Deep learning based valid bug reports determination and explanation,” in 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE), 2020

[13] Z. Li, M. Pan, Y. Pei, T. Zhang, L. Wang, and X. Li, “Deeplabel: Automated issue classification for issue tracking systems,” in Proceedings of the 13th Asia-Pacific Symposium on Internetware, 2022

[14] M. Laiq, N. bin Ali, J. Borstler, and E. Engstr ¨ om, “A comparative ¨ analysis of ml techniques for bug report classification,” Journal of Systems and Software, vol. p. 8

Where the paper says it

The curated benchmark serves as a valuable asset for the software engineering research community. By providing a ground truth set of invalid bug reports paired with their specific no-code solutions, it enables the rigorous evaluation of automated classification models and solution recommendation systems, facilitating the development of tools that can alleviate the maintenance burden in open-source projects.

227, p. 112457

[16] E. Dinc¸ and E. p. 0

Where the paper says it

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING 1 Automated Root-Cause Subclassification and No-Code Fix Generation for Invalid Bug Reports Mahmut Furkan Gon¨ * , Emre Dinc¸* , Tevfik Emre Sungur* , and Eray Tuz¨ un¨ Abstract—Context: Issues faced when using software are reported in the form of bug reports. However, many bug reports are invalid, meaning they do not require code changes and are resolved with a no-code fix.

Tuz¨ un, “Judge the votes: A system to classify bug reports ¨ and give suggestions,” in Proceedings of the 2nd ACM International Conference on AI-powered Software (AIWare ’25), 2025

[17] X. Li, J. Jin, G. Dong, H. Qian, Y. Wu, J.-R. Wen, Y. Zhu, and Z. Dou, “Webthinker: Empowering large reasoning models with deep research capability,” 2025

[18] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, “Bertscore: Evaluating text generation with bert,” arXiv preprint arXiv:1904. p. 0

Where the paper says it

com/software/jira 2https://github. com/features/issues arXiv:2605.

p. 0
Where the paper says it

com/software/jira 2https://github. com/features/issues arXiv:2605.

09675, 2019

[28] Z. Wang, J. Li, M. Ma, Z. Li, Y. Kang, C. Zhang, C. Bansal, M. Chintalapati, S. Rajmohan, Q. Lin et al., “Large language models can provide accurate and interpretable incident triage,” in 2024 IEEE 35th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 2024

[31] Y. Tao, Y. Qin, and Y. Liu, “Retrieval-augmented code generation: A survey with focus on repository-level approaches,” arXiv preprint arXiv:2510. p. 0

Where the paper says it

We study how different configurations perform on a gold-standard benchmark we have created. Method: Using a manually curated benchmark for higher quality analysis, we experimented with vanilla large language models (LLMs), Retrieval Augmented Generation (RAG), and agentic web search to identify invalid subclasses and generate no-code fixes.

04905, 2025

[34] W. Zhang, Y. Li, Y. Bei, J. Luo, G. Wan, L. Yang, C. Xie, Y. Yang, W.-C. Huang et al., “From web search towards agentic deep research: Incentivizing search with reasoning agents,” 2025

[38] Y. Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “Autocoderover: Autonomous program improvement,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024

[41] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “Swe-bench: Can language models resolve real-world github issues? p. 4

Where the paper says it

Because consumer issues and invalid bug reports share these underlying root causes, they naturally necessitate a similar resolution process. Recent benchmarks like ECom-Bench [40] evaluate the capacity of multimodal LLM agents to resolve real-world customer support issues by simulating user interactions and providing immediate solutions.

” arXiv preprint arXiv:2310. p. 0
Where the paper says it

com/software/jira 2https://github. com/features/issues arXiv:2605.

06770

[43] A. Patil, “Gitbugs: Bug reports for duplicate detection, retrieval augmented generation, triage, and more,” arXiv e-prints, pp. p. 0

Where the paper says it

The generated no-code fixes demonstrate the potential to automate customer support workflows, reduce manual triage effort, and provide immediate, actionable guidance to users within bug tracking systems. Index Terms—Bug Report, Invalid Bug Reports, Invalid Bug Report Subclassification, No-Code Fixes, LLM Evaluation, LLM Agents, Customer Support Agents, Retrieval Augmented Generation I.

arXiv–2504, 2025

[67] M. Hanna and O. Bojar, “A fine-grained analysis of BERTScore,” in Proceedings of the Sixth Conference on Machine Translation, L. Barrault, O. Bojar, F. Bougares, R. Chatterjee, M. R. Costa-jussa, C. Federmann, M. Fishel, A. Fraser, M. Freitag, Y. Graham, R. Grundkiewicz, P.-Guzmanis, B Haddow, M HuckA Martins Morishita and C Monz Eds., Online: Association for Computational Linguistics

[68] Google Search Help, “Why your google search results differ from others,” https://support.google.com/websearch/answer/12412910? p. 0

Where the paper says it

Early detection and resolution of invalid bug reports could significantly reduce the workload of all stakeholders involved in the bug tracking lifecycle, including users, customer support, 1https://www. com/software/jira 2https://github.

hl=en, 2025

[73] Class Model Judge (%) BERTScore F1 Judge (%) BERTScore F1 Judge (%) BERTScore F1 Judge (%) BERTScore F1 Judge (%) BERTScore F1 Judge (%) BERTScore F1

[74] Gemini 3.1 Pro 62.4% 0.82 63.3% 0.82 63.3% 0.82

[75] GPT-5.4 73.3% 0.82

[79] Gemini 3.1 Pro 64.1% 0.83

[80] Working as Designed (Conflicting Expectations) Gemini 3.1 Pro 85.4% 0.83

[77] Kimi K2.5 77.1% 0.82

[81] Wrong Version (Already Fixed) Gemini 3.1 Pro 25.0% 0.81

[63] “Brave issue 34522, wrong version no-code fix example,” https://github.com/brave/brave-browser/issues/34522

[64] “Brave issue 31299, working as designed no-code fix example,” https://github.com/brave/brave-browser/issues/31299

[58] Serper. (2026) Serper: The world’s fastest and cheapest Google Search API, 2026

[59] OpenRouter, “Openrouter api reference,” https://openrouter.ai/docs/api/reference/overview, 2026

[60] J. Niimi, “A simple ensemble strategy for llm inference: Towards more stable text classification,” in International Conference on Applications of Natural Language to Information Systems. p. 8

Where the paper says it

By providing a ground truth set of invalid bug reports paired with their specific no-code solutions, it enables the rigorous evaluation of automated classification models and solution recommendation systems, facilitating the development of tools that can alleviate the maintenance burden in open-source projects. IssueSupport Methodology We experimented with four distinct methodologies: (1) Vanilla LLM Pipeline, (2) Vanilla LLM Pipeline Without Prior Invalid Subclass Information, (3) RAG Pipeline, and (4) Agentic Web Search Pipeline.

Springer, 2025

[61] M. Zhang, O. Press, W. Merrill, A. Liu, and N. A. Smith, “How language model hallucinations can snowball,” in Proceedings of the 41st International Conference on Machine Learning, ser ICML’24

[62] J. Wu, J. Zhu, Y. Liu, M. Xu, and Y. Jin, “Agentic reasoning: A streamlined framework for enhancing llm reasoning with agentic tools,” arXiv preprint arXiv:2502. p. 8

Where the paper says it

By providing a ground truth set of invalid bug reports paired with their specific no-code solutions, it enables the rigorous evaluation of automated classification models and solution recommendation systems, facilitating the development of tools that can alleviate the maintenance burden in open-source projects. IssueSupport Methodology We experimented with four distinct methodologies: (1) Vanilla LLM Pipeline, (2) Vanilla LLM Pipeline Without Prior Invalid Subclass Information, (3) RAG Pipeline, and (4) Agentic Web Search Pipeline.

04644v2

[63] “Brave issue 34522, wrong version no-code fix example,” https://github.com/brave/brave-browser/issues/34522

[64] “Brave issue 31299, working as designed no-code fix example,” https://github.com/brave/brave-browser/issues/31299

[55] A. Zeng, X. Lv, Z. Hou, Z. Du, Q. Zheng, B. Chen, D. Yin et al., “Glm-5: from vibe coding to agentic engineering,” arXiv preprint arXiv:2602. p. 0

Where the paper says it

com/software/jira 2https://github. com/features/issues arXiv:2605.

15763

[57] Kimi k2 5: Visual agentic intelligence, arXiv preprint arXiv:2602.02276

[18] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, “Bertscore: Evaluating text generation with bert,” arXiv preprint arXiv:1904.09675

[2605.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Automated Root-Cause Subclassification and No-Code Fix Generation: The system can automatically identify root causes of invalid bug reports, such as "External System & Dependency Issues or Faulty Configuration & Environment Setup," and generate specific no-code fixes tailored to that exact cause, moving beyond generic suggestions.

  2. Context Engineering for Classification: The system should dynamically adjust its context engineering approach based on the bug report type; for instance, it can use Retrieval Augmented Generation (RAG) when classifying Working as Designed issues and Agentic Web Search when addressing Wrong Version issues to maximize accuracy, as shown by the results in Table V.

  3. Multi-Metric Evaluation for Fix Quality: Instead of relying solely on similarity metrics like BERTScore, the system should use a dual-metric approach combining Judge LLM evaluation and BERTScore to assess no-code fix generation quality, ensuring suggestions are not just lexically similar but also functionally useful and aligned with successful resolution patterns.

  4. Prior Information Sensitivity Tuning: The system's performance on no-code fix generation should be tuned based on the specific subclass; for example, providing prior knowledge generally improves performance in Non-reproducibility but may cause a drop in "Faulty Configuration & Environment Setup," indicating a need for category-specific prior injection strategies.

  5. Constraint Implementation for Agentic Search: To stabilize agentic web search performance, the system should implement predefined logical transitions (Search -> Verify -> Summarize) instead of fully autonomous loops, which mitigates the risk of recursive noise and potential hallucinations in less capable models.

Abstract

Issues faced when using software are reported in the form of bug reports. However, many bug reports are invalid, meaning they do not require code changes, and are resolved with a no-code fix. Manually determining the root cause of the invalid bug reports and providing actionable resolutions by the customer support causes a serious waste of resources. Our goal is to introduce a standardized taxonomy for root-cause oriented invalid bug report subclassification, and perform experiments to test the accuracy of various approaches on invalid subclassification and no-code fix generation. We study how different configurations perform on a gold-standard benchmark we have created. Using a manually curated benchmark for higher quality analysis, we experimented with vanilla LLMs, Retrieval Augmented Generation, and agentic web search to identify invalid subclasses and generate no-code fixes. We evaluated the results against manually labeled ground truth data that includes the invalid subclass and no-code fixes from the original bug reports. We measured subclass detection performance with weighted F1-Score, and assessed no-code fix suggestions using BERTScore and Judge LLM success rates. For subclassification, retrieval augmented generation achieves the highest overall performance with 0.66 weighted F1, slightly outperforming vanilla LLMs at 0.65 and agentic web search at 0.64. At the subclass level, performance peaks at 0.85 F1 for Non-reproducibility and 0.79 for Feature Request and Question, while Wrong Version remains the most challenging with scores between 0.00 and 0.29. For no-code fix generation, agentic web search achieves the highest overall Judge LLM success rate at 68.9%, compared to 64.4% for RAG applications and 64.9% for vanilla LLMs, with subclass-level peaks of 87.4% for Working as Designed and 72.2% for Question.

Sources

Related papers