ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS

summary

The gist

Automated program repair benchmarks largely cover mature languages such as Python and Java, leaving limited support for ArkTS and OpenHarmony.

In short

ArkEval is a new executable repository-level benchmark for testing automated code repair on ArkTS and OpenHarmony. It contains 502 real issue-resolution tasks with tests, allowing researchers to evaluate models based on compilation success and behavioral correctness. Results show that official knowledge retrieval significantly improves behavioral repair metrics.

Key concepts

ArkEval
An executable benchmark containing 502 real issue-resolution instances for ArkTS/OpenHarmony. It includes traceable buggy and fixed revisions, standardized problem statements, and executable reproduction tests to rigorously evaluate automated code repair models.
Fault Localization vs. Applicability
A protocol that separates the process of finding where a bug is located (fault localization) from the process of generating a fix (patch generation). This allows researchers to study how well models can pinpoint errors versus how accurately they can apply solutions.
Official-knowledge RAG
Integrating official knowledge context during the patch generation phase. This Retrieval-Augmented Generation technique uses domain-specific rules to guide the model, which was shown to increase both compilation success and behavioral repair scores.
Pass@1
A metric measuring the percentage of generated patches that pass a set of behavioral tests. It assesses whether a proposed fix actually solves the original issue in terms of system behavior, rather than just compiling successfully.

Terminology used across episodes

This episode discusses

The paper

ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS · Read on arXiv

Bang Xie, Senjian Zhang, Zhiyuan Peng, Wei Chen, Xin Yin, Chenhao Ying, Yuan Luo

Shanghai Jiao Tong University

Large language models have transformed code generation, enabling unprecedented automation in software development. As mobile ecosystems evolve, HarmonyOS has emerged as a critical platform requiring robust development tools. Software development for the HarmonyOS ecosystem relies heavily on ArkTS, a statically typed extension of TypeScript. Despite its growing importance, the ecosystem lacks robust tools for automated code repair, primarily due to the absence of a high-quality benchmark for evaluation. To address this gap, we present ArkEval, a unified framework for ArkTS automated repair workflow evaluation and benchmark construction. It provides the first comprehensive benchmark specifically designed for ArkTS automated program repair. We constructed this benchmark by mining issues from a large-scale official Huawei repository containing over 400 independent ArkTS applications. Through a rigorous multi-stage filtering process, we curated 502 reproducible issues. To ensure testability, we employed a novel LLM-based test generation and voting mechanism involving Claude and other models. Furthermore, we standardized problem statements to facilitate fair evaluation. Finally, we evaluated four state-of-the-art Large Language Models (LLMs) on our benchmark using a retrieval-augmented repair workflow. Our results highlight the current capabilities and limitations of LLMs in repairing ArkTS code, paving the way for future research in this low-resource language domain.

DOI: 10.1145/3832783.3837514

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS".

Jane: Here's the gist of the paper. Automated program repair benchmarks largely cover mature languages such as Python and Java, leaving limited support for ArkTS and OpenHarmony.

Tom: We present ArkEval, an executable repositorylevel benchmark for evaluating repair on real ArkTS/OpenHarmony issue-resolution tasks.

Jane: The best model compiled 54.38% of its patches, whereas the highest behavioral Pass@1 was 19.32%.

Tom: Pruning redundant reasoning tokens ArkEval is the first executable repository-level benchmark for automated repair on real ArkTS/OpenHarmony issue-resolution tasks.

Title and authors: Tom: So we're looking at this paper called ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS. Basically, it's trying to create a proper test for when AI systems can fix code in ArkTS, which is this language used for native HarmonyOS applications.

Jane: That sounds really practical. What's the big picture here? We all know that most automated repair benchmarks are focused on languages like Python and Java, but this paper says there’s really limited support for things like ArkTS and OpenHarmony, so they needed something specific.

Lu: The main contribution is this executable repositorylevel benchmark. It includes five hundred two issue-resolution instances with traceable buggy and fixed revisions, standardized problem statements, and tests that actually run. That makes it a real test of repair on actual real-world tasks.

Meng: I’m interested in the scale of that data set. How big is this mining effort? We need to know if these fifty two examples are representative or just covering a tiny corner of the problem space for ArkTS and OpenHarmony.

Lalam: They mined more than six thousand pull request and merge request records available through July three two thousand twenty-five from nine public repositories, which is a pretty big starting point for this benchmark <ref:2602.08866#pg1,pull request and merge request records available through July 3, 2025, from>.

Tom: And they did some screening too. They used automated profiling and unanimous screening by three ArkTS/OpenHarmony developers to produce those five hundred two instances. That suggests a level of human input to make sure the issues are real and relevant before testing the AI models on them.

Jane: Before we talk about how they built this, let's look at what they actually did with it. They introduced a configurable baseline that separates finding the fault from making the fix, and then separating that from whether it actually compiles or behaves correctly. That’s a smart way to test different parts of the repair process separately.

Lu: Yeah, they separated applicability and compilation from behavioral testing. This protocol lets you see if just compiling something is enough to know if a patch works, or if you have to run the whole thing through tests that check for behavioral issues too.

Meng: So what are the actual results looking like? Are we seeing models that are good at just getting the code to compile, or are we seeing models that actually fix things in a way that makes sense when you run them?

Tom: The results show two different metrics. The best model compiled fifty-four point three eight percent of its patches, while the highest behavioral Pass at one was nineteen point three two percent. That tells us compilation alone isn't enough for ArkTS repair.

Jane: And they found that using official knowledge retrieval augmentation, which is like giving the AI access to specific documentation for ArkTS and OpenHarmony rules, improves both those metrics in their four-model comparison. That suggests that knowing the specific language context helps models do better on both compiling and behaving correctly.

Title and authors: Lu: They also saw a big jump when they introduced a localization intervention, which increased the behavioral Pass at one from four point two percent up to forty five point eight percent on a subset of those instances. That points toward where most of the repair difficulty is actually located.

Meng: So if we look at why things are failing, what's the main bottleneck they found? Is it just that it’s hard to get the right code, or something else?

Tom: They identified localization as a major bottleneck. The analysis showed that replacing predicted files with files modified by developers increased the Pass at one from one twenty four percent up to eleven twenty four percent on a subset of instances. That suggests focusing on finding those specific files is critical for success.

Jane: That makes sense when you think about complex codebases, where knowing which file needs changing is often harder than knowing what to write in it. They also suggested that the official knowledge retrieval RAG points toward needing domain-specific context, meaning the AI needs rules specific to ArkTS and OpenHarmony to make better decisions.

Lu: The study characterizes this as localized issue resolution rather than trying to do big architectural repairs across the whole system. It’s focused on fixing specific problems in open-source ArkTS and OpenHarmony projects. That really sets the scope for what they were testing.

Meng: What are some of the risks or limitations they pointed out? Because when you're deploying this kind of benchmark, you need to know where it might fall short in a real production environment.

Tom: They acknowledged six main limitations. One is that there are correlated agent errors within each review batch, which means some errors might be grouped together and not fully isolated. Another limitation is the necessity of two-state semantics for semantic validity, which adds complexity to making sure their tests are actually checking the right thing.

Jane: And they also mentioned potential recall issues with localization constraints, like limiting ArkTS-aware chunks to at most two thousand forty-eight characters and only retrieving from the top ten files. That means if a crucial piece of context is outside that small window or those ten files, the AI might miss it.

Lu: Plus, their results are limited to nine repositories and they only tested the patch-only protocol, which keeps them focused on localized fixes rather than testing broader architectural repair capabilities. And they noted that four hundred fourteen of those five hundred two instances came from the firstparty OpenHarmony sample repository, which is a big piece of context for understanding the data source.

Meng: So to summarize, this ArkEval benchmark gives us a concrete way to measure if AI can handle real ArkTS and OpenHarmony code repair tasks. It shows that improving how the AI finds the right files seems more important than just training it on more general data.

Title and authors: Tom: Exactly. The finding that localization is a major bottleneck means that future work should probably focus on better file selection mechanisms before we expect big gains in behavioral success rates for this kind of work, as shown in ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS.

Jane: It’s a very useful tool for anyone working on AI tools for specific ecosystems like HarmonyOS because it moves the evaluation away from just looking at whether the code compiles to seeing if the fix actually works as expected by a user.

Lu: And it helps us understand that when you are dealing with specialized languages, like ArkTS, you need context that goes beyond just general programming knowledge; you need domain-specific rules for those systems.

Meng: For deployment purposes, knowing these failure modes helps us know what to test for in our own pipelines—we can specifically target localization improvements if that’s where the Pass at one metric is lagging.

Lalam: I just want to say that the way they constructed this, from mining records through curation and testing, gives a really traceable path for anyone trying to improve these kinds of repair systems. It’s not just a black box result; you can see how it got there.

Tom: Traceability is key. It lets us pinpoint exactly where the process breaks down when we want to build better AI tools for mobile platforms like OpenHarmony and ArkTS, which is what this paper does with ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS.

Jane: So, if you're listening and you work on code that needs fixing in these systems, knowing that the real challenge isn't always the writing of the fix but finding the right place to write it is a significant piece of information.

Lu: That’s what they are pointing toward—localized issue resolution. It’s a much more realistic goal than expecting a general AI model to suddenly understand the entire architecture of HarmonyOS and fix complex bugs everywhere at once.

Meng: It keeps expectations grounded, which is good for shipping models; we don't want to spend all our time chasing impossible architectural fixes when localized ones are where we actually see improvement.

Lalam: It’s about focusing on the small, verifiable parts of the problem that have a measurable impact on user experience in these specific applications.

Tom: So, ArkEval gives us a concrete way to measure if AI can handle real ArkTS and OpenHarmony code repair tasks by breaking down the process into steps like localization and testing.

Jane: It’s a very useful tool for anyone working on AI tools for specific ecosystems because it moves the evaluation away from just looking at whether the code compiles to seeing if the fix actually works as expected by a user.

Tom: So, ArkEval gives us a concrete way to measure if AI can handle real ArkTS and OpenHarmony code repair tasks by breaking down the process into steps like localization and testing.

The paper's summary: Tom: So, ArkEval is basically an executable test you run against real ArkTS and OpenHarmony code repair tasks to see how good AI is at fixing bugs in that specific ecosystem.

Jane: It’s like giving an AI a real coding job instead of just showing it a bunch of textbook examples. The big takeaway here is they set up this whole system to separate the process into stages, so you can actually look at where the AI is failing—is it finding the right file, or is it writing bad code?

Tom: Exactly. They built this five hundred and two instances with traceable buggy and fixed revisions, standardized problem statements, and tests that run on those actual fixes. It’s not just a paper; it's a benchmark.

Jane: And they found that compiling the code isn't enough to tell you if the fix actually works when you run it in real-world scenarios. The best model only compiled fifty-four point three eight percent of its patches, but the highest behavioral pass at one was only nineteen point three two percent.

Tom: That’s a big gap between compiling and behaving correctly, right? And they found that using official knowledge retrieval augmentation actually helps both metrics improve in their four-model comparison. So feeding the AI specific documentation for ArkTS and OpenHarmony rules makes it better at both compiling and behaving right.

Jane: They also pinpointed that localization—figuring out which exact files need changing—is a huge hurdle. They saw the behavioral pass at one jump from four point two percent up to forty five point eight percent when they added that localization intervention on a subset of tasks.

Tom: So, the main bottleneck isn't just generating code; it’s figuring out *where* to generate it in the codebase. The analysis showed that swapping predicted files with files modified by developers increased the pass at one from one twenty four percent up to eleven twenty four percent on some instances.

Jane: It means for these specific systems, focusing on getting the file selection right is way more important than just having a good patch generation engine. It’s not about fixing everything architecturally at once; it's about fixing the localized spots where things break.

Tom: And they flagged some limitations, like having correlated errors in each review batch and the risk that their localization setup might miss context if it’s too restrictive, like limiting file chunks to two thousand forty-eight characters.

Jane: That means while ArkEval gives us a really concrete way to measure performance on this specific platform, we still need to watch those smaller details—like context retrieval limits—to see if the results hold up in a much bigger deployment.

Tom: So, ArkEval gives us a concrete way to measure if AI can handle real ArkTS and OpenHarmony code repair tasks by breaking down the process into steps like localization and testing.

Jane: It’s a very useful tool for anyone working on AI tools for specific ecosystems because it moves the evaluation away from just looking at whether the code compiles to seeing if the fix actually works as expected by a user.

Tom: So, if you're listening and you work on code that needs fixing in these systems, knowing that the real challenge isn't always writing the fix but finding the right place to write it is a significant piece of information.

The paper's improvements: Tom: So they’re not just stopping at measuring things; they’re suggesting ways to actually improve how we test these AI repair systems going forward.

Jane: They've pointed out that localization is a major bottleneck, and their suggestion is to use a more specific protocol, called ArkFix, which uses syntax-aware chunking and embedding retrieval to figure out exactly which files are relevant for the fix.

Tom: That means they want the AI to use smarter file finding methods rather than guessing based on general context. It’s about guiding the AI's search process more precisely.

Jane: And they also emphasized separating fault localization and patch generation from applicability, compilation, and behavioral testing through this baseline protocol. This lets us test different parts of the repair pipeline on their own terms.

Tom: That separation is key because it shows you exactly where the failure is coming from—if it fails at the localization stage, you know to focus your efforts there before worrying about compilation errors later on.

Jane: Plus, they’re looking at how official knowledge retrieval augmentation works during the patch generation phase. They want to see how feeding in domain-specific context actually affects those compilation and behavioral metrics in a more controlled way.

Tom: It seems like their main goal is to move from just getting a number on a final result to understanding the actual mechanism of repair better, which requires these fine-grained controls they introduced.

Jane: It shifts the focus from building the biggest model possible to building models that are better at understanding and applying specific domain knowledge within that language structure.

Tom: So, it’s less about brute force training and more about engineering smarter evaluation protocols so we can diagnose why a model is failing before we even deploy it widely.

Jane: It gives us a blueprint for how to set up benchmarks for other specialized languages too, showing that the method of separating localization from execution is something that works across different domains.

Tom: And if you look at the limitations they mentioned, like the constraint on retrieval to just the top ten files, it tells us that even with these smart methods, we still have to worry about whether we’re missing critical context because of those hard limits.

Conclusion: Tom: So, to wrap up, ArkEval gives us a concrete way to measure if AI can handle real ArkTS and OpenHarmony code repair tasks by breaking down the process into steps like localization and testing.

Jane: It’s a very useful tool for anyone working on AI tools for specific ecosystems because it moves the evaluation away from just looking at whether the code compiles to seeing if the fix actually works as expected by a user.

Tom: Exactly. The finding that localization is a major bottleneck means that future work should probably focus on better file selection mechanisms before we expect big gains in behavioral success rates for this kind of work, as shown in ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS.

Jane: It’s a very useful tool for anyone working on AI tools for specific ecosystems because it moves the evaluation away from just looking at whether the code compiles to seeing if the fix actually works as expected by a user.

Lu: I think what this paper really shows is that when you’re dealing with specialized languages, like ArkTS, you need context that goes beyond just general programming knowledge; you need domain-specific rules for those systems.

Meng: For deployment purposes, knowing these failure modes helps us know what to test for in our own pipelines—we can specifically target localization improvements if that’s where the Pass at one metric is lagging.

Lalam: I just want to say that the way they constructed this, from mining records through curation and testing, gives a really traceable path for anyone trying to improve these kinds of repair systems. It’s not just a black box result; you can see how it got there.

More episodes

← Home