Large Language Models for Multi-Lingual Equivalent Mutant Detection: An Extended Empirical Study

arXiv:2607.00511 · cs.SE · Submitted 2026-07-01 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Large Language Models for Multi-Lingual Equivalent Mutant Detection".

Jane: Here's the gist of the paper. Large Language Models (LLMs) are being explored as a powerful technique for equivalent mutant detection, addressing limitations in pure-code analysis methods constrained by compiler infrastructure and existing machine learning approaches hampered by scarce training data and poor generalization to unseen mutants.

Tom: This paper presents the first comprehensive empirical study on LLMs for Equivalent Mutant Detection (EMD), benchmarking them against state-of-the-art methods across both Java and C environments using 3,302 Java and 1,088 C mutant pairs.

Jane: The research systematically evaluates 13 state-of-the-art LLMs across five research questions: RQ1 (Performance): LLM-based techniques significantly outperform traditional baselines, achieving average F1-score improvements of 75.18% and 557.79% over Compiler-based techniques in Java and C, respectively, and showing substantial gains over ML-based and Tree-based NN techniques.

Tom: RQ2 (Best Strategy): Fine-tuned UniXCoder achieved the highest F1-scores on Java environments, while fine-tuned CodeT5+ demonstrated superior effectiveness in C environments.

Paper summary: Jane: So to recap, this paper, Large Language Models for Multi-Lingual Equivalent Mutant Detection: An Extended Empirical Study, is testing whether large language models can detect equivalent mutants across different programming languages without being tied down by specific compiler setups or limited training data <ref:2607.00511#pg1>.

Tom: They set up this comprehensive study by benchmarking thirteen state-of-the-art LLMs against traditional methods using three thousand three hundred twenty-two Java mutant pairs and one thousand eight hundred eighty eight C mutant pairs to get their numbers <ref:2607.00511#pg2>.

Lu: The main thesis they’re pushing is that deep semantic code understanding, which LLMs have because of their pre-training on diverse repositories, gives them a superior edge over traditional learning-based EMD techniques that suffer from data scarcity <ref:2607.00511#pg3>.

Meng: They found significant performance gains, showing average F1 score improvements of seventy-five point one eight percent in Java and five hundred fifty seven point seventy nine percent in C when using LLM-based approaches over compiler based techniques <ref:2607.00511#pg3>.

Lalam: They also identified that the best specific model for Java was fine-tuned UniXCoder, while for C, fine-tuned CodeT5 plus showed superior effectiveness <ref:2607.00511#pg3>.

Tom: The study also looked at how these LLMs stack up against other EMD categories and strategies, finding that the LLM-based methods generally achieved higher accuracy than alternatives in both languages <ref:2607.00511#pg3>.

Jane: And they looked at efficiency too; the best performing technique had inference times comparable to existing machine learning models, with the fastest being zero point zero four three one seconds for Java and zero point zero two one five seconds for C <ref:2607.00511#pg3>.

Lu: They also explored how instruction fine-tuning helped; they found that it provided measurable F1 improvements of three point five five percent to three point seven zero percent in effectiveness <ref:2607.00511#pg3>.

Meng: So, while the numbers look strong, I’m always thinking about what this actually means for us when we try to deploy these things on real traffic where things get messy <ref:2607.00511#pg3>.

Tom: The study found that cross-lingual fine-tuning enhances LLM effectiveness, but they noted that code embedding did not perform as well under that specific approach <ref:2607.00511#pg3>.

Jane: They also highlighted the limitations they found

Conclusion: Tom: So we’ve been digging into this paper, "Large Language Models for Multi-Lingual Equivalent Mutant Detection: An Extended Empirical Study," and what you get is a look at how these big language models actually handle finding those equivalent mutants across different programming languages.

Jane: Right, so the main thing they did was take thirteen different large language models and test them against established methods using three thousand three hundred twenty-two Java mutant pairs and one thousand eight hundred eighty eight C mutant pairs.

Lu: What’s interesting is that the research wasn't just about picking one model; they systematically answered five specific questions to see which strategy worked best for each language, which was a big part of their approach.

Meng: I saw some numbers earlier suggesting huge performance boosts over compiler-based tools, but the real question for me is how much effort goes into actually fine-tuning these models to get those results.

Lalam: I can tell you that while the general LLM approach showed a significant jump in accuracy—like seventy-five point one eight percent improvement in Java—the specific model that won depended heavily on whether it was UniXCoder for Java or CodeT5 plus for C.

Tom: And the efficiency part, which is crucial for production, they found that the best models were still running at inference times similar to other machine learning models, though there were some slight speed differences between Java and C environments.

Jane: They also looked at how cross-lingual fine-tuning helps these things perform better; instruction fine-tuning gave a measurable boost of about three point five five percent to three point seven zero percent in the final score.

Lu: The study’s conclusion is that LLMs are definitely a viable tool for this detection task, but they aren't completely replacing the traditional structural checks done by compilers or other techniques.

Meng: What it means for us in shipping models is that we still need those initial filters—the compiler stuff—and then we route the really tricky cases over to the LLMs for that deeper analysis.

Tom: The authors pinpointed where the LLMs make mistakes, which was mostly down to very specific structural things like type coercion subtleties and operator coverage mismatches in code.

Jane: So, while they show a lot of promise for semantic understanding, they flag that these models still aren't perfect when it comes to catching those tiny coding errors.

Lu: That points toward the next step in research—figuring out how to make the fine-tuning process more robust so these models can handle those structural subtleties even better.

Tianjin University · Kyushu University

cs.SE

Submitted: 2026-07-01

Updated: 2026-11-05

Comments: 38 pages, 8 figures

Code: https://github.com/SpanShu96/Large-Language-Models-for-Multi-Lingual-Equivalent-MutantDetection-An-Extended-Empirical-Study

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: Large Language Models (LLMs) are being explored as a powerful technique for equivalent mutant detection…

Key concepts

Equivalent Mutant Detection (EMD)
This is the task of finding pairs of code changes (mutants) that produce the exact same program behavior. The goal is to identify these functionally equivalent mutants, which helps in analyzing code robustness and correctness without introducing bugs.
Large Language Models (LLMs)
These are advanced AI models trained on vast amounts of text data. In this study, they were used to understand the semantic meaning of code to determine if two different versions of a program perform the same function, moving beyond simple structural comparisons.
Fine-tuning
This is the process of taking a pre-trained LLM and training it further on specific data relevant to the task, like mutant detection. This makes the model much better at performing that specific job on code, leading to higher accuracy in detecting equivalent mutants.

Terminology

Summary

Large Language Models (LLMs) are being explored as a powerful technique for equivalent mutant detection, addressing limitations in pure-code analysis methods constrained by compiler infrastructure and existing machine learning approaches hampered by scarce training data and poor generalization to unseen mutants. p. 0

Where the paper says it

Although numerous equivalent mutant detection (EMD) methods have been proposed, they often face distinct challenges: pure-code analysis methods can be limited by their reliance on specific compiler infrastructures, while existing machinelearning approaches remain constrained by scarce training data and limited generalization to unseen mutants. Large language models (LLMs) have recently demonstrated remarkable performance across diverse coderelated tasks by better capturing program semantics.

This paper presents the first comprehensive empirical study on LLMs for Equivalent Mutant Detection (EMD), benchmarking them against state-of-the-art methods across both Java and C environments using 3,302 Java and 1,088 C mutant pairs. p. 0
Where the paper says it

This paper presents the first comprehensive empirical study on LLMs for EMD, using 3,302 Java and 1,088 C mutant pairs to benchmark against state-of-the-art methods, explore strategy variations, assess efficiency, and evaluate cross-lingual generalization. Experimental results show that LLM-based approaches achieve higher F1-scores than the evaluated traditional methods, with fine-tuned code embedding yielding the highest detection accuracy among the tested strategies.

The research systematically evaluates 13 state-of-the-art LLMs across five research questions:

  • RQ1 (Performance): LLM-based techniques significantly outperform traditional baselines, achieving average F1-score improvements of 75.18% and 557. p. 2

    Where the paper says it

    Result: LLM-based techniques achieve higher F1-scores than all ten EMD baselines in equivalent mutant detection, achieving average F1-score improvements of 75. 18% and 557.

    79% over Compiler-based techniques in Java and C, respectively, and showing substantial gains over ML-based and Tree-based NN techniques. p. 1
    Where the paper says it

    While these pure-code analysis methods are foundational and highly beneficial because they do not require training data, their scope can sometimes be constrained by a reliance on specific compiler infrastructures or the need for manually crafted heuristics. To complement these approaches, researchers have explored learning-based EMD techniques, including conventional machine learning classifiers such as KNN and SVM [12, 16, 64], as well as tree-based neural network models [79].

  • RQ2 (Best Strategy): Fine-tuned UniXCoder achieved the highest F1-scores on Java environments, while fine-tuned CodeT5+ demonstrated superior effectiveness in C environments.

  • RQ3 (Orthogonality): LLM-based techniques and the fine-tuned code embedding strategy generally achieve higher accuracy than alternative EMD categories and LLM strategies in both languages.

  • RQ4 (Efficiency): The best-performing LLM technique demonstrated inference times comparable to existing ML models, with the fastest being 0.0431 s for Java and 0. p. 3

    Where the paper says it

    Result: The inference time of the best-performing LLM-based technique (0. 0431 s for Java and 0.

    0215 s for C, outperforming the best Compiler-based technique but being slightly slower than some ML and Tree-based NN approaches. p. 3
    Where the paper says it

    3537 s for Java and 0. 6613 s for C), while being slightly slower than the best-performing ML-based technique (0.

  • RQ5 (Cross-Lingual Effectiveness): Cross-lingual fine-tuning enhances LLM effectiveness; instruction fine-tuning showed measurable benefits (3.55%–3.70% F1 improvement), although code embedding performed worse under this cross-lingual approach. p. 3

    Where the paper says it

    Result: Cross-lingual fine-tuning enhances LLMs’ effectiveness in equivalent mutant detection across Java and C, with fine-tuning with instruction showing measurable benefits (3. 70% F1-score improvement), while fine-tuning code embedding performs worse under this approach.

The study establishes LLMs as a viable and efficient approach for equivalent mutant detection, showing measurable generalization across programming languages when fine-tuned. p. 0

Where the paper says it

Importantly, fine-tuned LLMs demonstrate measurable generalization across programming languages. These findings establish LLMs as a viable and efficient approach for tackling the longstanding challenge of equivalent mutant detection, offering new directions for advancing mutation testing in practice.

The key contribution is the first large-scale empirical evaluation in a multi-lingual context, demonstrating that while LLMs possess superior semantic understanding, they do not entirely replace the structural rigor of conventional EMD techniques. p. 3
Where the paper says it

Contributions. To sum up, the contributions of this study are: • Our study presents the first comprehensive, large-scale empirical evaluation of LLMs for equivalent mutant detection in the multi-lingual context, by examining five key dimensions: effectiveness relative to existing EMD techniques, the impact of different LLM strategies, orthogonality between various EMD approaches, computational efficiency, and generalization of fine-tuned LLMs on cross-lingual EMD.

Failure modes for incorrect LLM detections were identified, concentrating in structural subtleties such as Type-Coercion/LanguageSpecific Semantic Subtlety and Operator Coverage Mismatch. The study concludes by recommending a multi-stage hybrid pipeline where compiler-based techniques act as initial filters, routing ambiguous cases to LLM-based techniques for rigorous dynamic analysis. p. 4
Where the paper says it

To address this issue, researchers have developed various EMD techniques. Traditional EMD approaches include genetic algorithms [2], constraint-based testing [9, 51, 66], coverage analysis [74, 77, 86, 87], automata language equivalence [17], software behavior graphs [24], structural pattern matching [50], dynamic subsumption relations [23, 28], and compiler optimization [37, 46, 75].

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. The LLM-based approach can achieve higher F1-scores than all ten EMD baselines in equivalent mutant detection, with average F1-score improvements of 75.18% and 557.79% over Compiler-based techniques and ML-based techniques, respectively. This capability allows the system to be adopted for automated mutant filtering or developer warning systems due to its higher effectiveness compared to conventional methods.

  2. The system can be configured with a fine-tuned code embedding strategy, which yields the highest F1-scores among all LLM combinations and strategies on Java environments, improving F1-scores by 1.16%–84.10%. This demonstrates that fine-tuned code embedding strategy yields the highest F1-scores among all LLM combinations and strategies on Java environments.

  3. The system can be deployed with a fine-tuned LLM using the instruction strategy, which achieves higher scores across all metrics compared to both zero-shot and few-shot prompting approaches across all decoder-only LLMs. For example, in the C environment, this strategy demonstrated remarkable F1-score enhancements of 172.04% and 194.90% compared to prompting strategies.

  4. The system can perform cross-lingual equivalent mutant detection by being fine-tuned on both Java and C simultaneously, showing that cross-lingual fine-tuning enhances LLMs’ effectiveness in equivalent mutant detection across Java and C, with an average F1-score improvement of 3.55%–3.70% over single-language fine-tuning.

  5. The system can be used to select the best EMD technique based on the specific engineering workflow, as fine-tuned code embedding strategy consistently achieves higher F1-scores than the pre-trained approach across almost all studied LLMs in terms of F1-score. This allows for tailoring detection sensitivity based on whether precision or recall is prioritized.

  6. The system can provide a qualitative analysis of unique incorrect detections to identify specific code structures where LLMs fail, such as Type-Coercion / LanguageSpecific Semantic Subtlety and Operator Coverage Mismatch. This helps engineers understand the limitations of the LLM's semantic understanding compared to structural rigor.

  7. The system can be integrated into a multi-stage hybrid pipeline, allowing Compiler-based techniques to act as scalable initial filters to prune obvious equivalent mutants, routing ambiguous or structurally complex cases to LLM-based techniques for rigorous dynamic analysis.

Abstract

Mutation testing is a powerful technique for ensuring software quality. However, the presence of equivalent mutants introduces unnecessary costs and biases, limiting its practical effectiveness. Although numerous equivalent mutant detection (EMD) methods have been proposed, they often face distinct challenges: pure-code analysis methods can be limited by their reliance on specific compiler infrastructures, while existing machine-learning approaches remain constrained by scarce training data and limited generalization to unseen mutants. Large language models (LLMs) have recently demonstrated remarkable performance across diverse code-related tasks by better capturing program semantics. Yet their potential for EMD remains largely unexplored, particularly in the multi-lingual context. This paper presents the first comprehensive empirical study on LLMs for EMD, using 3,302 Java and 1,088 C mutant pairs to benchmark against state-of-the-art methods, explore strategy variations, assess efficiency, and evaluate cross-lingual generalization. Experimental results show that LLM-based approaches achieve higher F1-scores than the evaluated traditional methods, with fine-tuned code embedding yielding the highest detection accuracy among the tested strategies. Moreover, LLM-based approaches strike a practical balance between effectiveness and efficiency with inference times comparable to existing machine-learning models. Importantly, fine-tuned LLMs demonstrate measurable generalization across programming languages. These findings establish LLMs as a viable and efficient approach for tackling the longstanding challenge of equivalent mutant detection, offering new directions for advancing mutation testing in practice.

Sources

Related papers