Identifying and Mitigating API Misuse in Large Language Models

arXiv:2503.22821 · cs.SE · Submitted 2025-03-28 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Identifying and Mitigating API Misuse in Large Language Models".

Jane: Here's the gist of the paper. API misuse in code generated by large language models (LLMs) presents a serious and growing challenge in software development This work conducts a large-scale study of API misuse patterns in LLM-generated code, analyzing both method selection and parameter usage across Python and Java, using three representative LLMs (StarCoder-7B, Qwen2.5-Coder-7B, and GitHub Copilot) Based on extensive manual annotation of 3,209 methodlevel and 3,492 parameter-level misuses, four recurring misuse types are identified The evaluation reveals persistent challenges in API usage, particularly hallucination and intent misalignment To address these issues, Dr.Fix, an LLM-based automatic repair approach guided by the taxonomy, is proposed Misuse Taxonomy Development The authors adapt and refine existing API misuse taxonomies to the context of LLM-generated code They retain violation types that can be reliably identified from snippet-level context, such as Missing and Redundant, and introduce two additional categories: Intent misuse and Hallucination misuse The four recurring misuse patterns identified are (1) Intent misuse, where a model selects an API that is syntactically valid but semantically inappropriate for the intended task; (2) Hallucination misuse, where the model produces non-existent or fabricated API methods or parameters; (3) Missing item misuse, where required API methods or parameters are omitted; and (4) Redundancy misuse, where unnecessary API calls or arguments are introduced Model Performance Comparison The study evaluates three representative decoder-only LLMs used in IDE settings: StarCoder-7B, Qwen2.5-Coder-7B, and Copilot The performance comparison shows that Qwen2.5-Coder achieves the highest overall accuracy across both languages, combining strong fluency with precise matching Specifically, in Python Method Infilling, Qwen2.5-Coder has 15 unique errors compared to StarCoder's 126 and Copilot's 466 The results indicate that while Qwen2.5-Coder shows fewer unique errors, all three models converge on many of the same failure cases, reflecting persistent limitations in LLM reasoning about API semantics Dr.Fix Repair Efficacy Dr.Fix is proposed as an LLM-based APR method that improves repair accuracy compared to baseline prompting and existing repair methods Dr.Fix achieves substantial improvements, with BLEU scores increasing by up to 38.4 points and exact match rates improving by up to 40 percentage points across different models and languages The refusal rate metric evaluates the models’ ability to avoid hallucinating API misuses when no misuse exists, and Dr.Fix significantly enhances these rates across all tested models Key Findings on Misuse Patterns Finding 1 reveals that Qwen2.5-Coder achieves the highest overall accuracy across both languages, combining strong fluency with precise matching Finding 2 states that Hallucination remains the most common misuse overall, although Qwen2.5-Coder reduces its occurrence compared to StarCoder and Copilot Missing Item errors remain prominent in Python parameter prediction for all models Intent misuses continue to appear frequently but are less severe in Qwen2.5-Coder, suggesting improved grounding and fewer redundant errors Mitigation Strategy Dr.Fix is designed to be backbone-agnostic and taxonomy-driven, making it applicable to general-purpose third-party APIs across different programming languages The workflow of Dr.Fix integrates detection and reasoning capabilities through multiple stages of LLM prompting It combines detection, classification, and reasoning to produce candidate fixes that aim to be both syntactically correct and semantically meaningful The results demonstrate that Dr.Fix significantly enhances API misuse repair across all models, achieving up to 130% improvement in exact match rates compared to baseline approaches Limitations and Future Work A threat to internal validity is the reliance on subjective human annotations for the initial classification of API misuse A significant threat to external validity is the generalizability of the findings to other programming languages, APIs, and LLMs Future work should focus on extending Dr.Fix’s capabilities to handle more complex API misuse scenarios and investigating its applicability to other programming languages The authors plan to expand their taxonomy and dataset as new patterns of API misuse emerge with the evolution of code generation models In this work, LLMs struggle significantly with hallucination and intent misalignment, with distinct patterns emerging between programming languages The shared error patterns between Copilot and StarCoder highlight fundamental challenges in API usage that persist across different model architectures Dr.

Title and authors: Tom: So what's the main takeaway from this paper, beyond just saying LLMs make mistakes? It’s about how specific those mistakes are, and which kinds are happening most often.

Jane: They’ve identified four recurring misuse types by adapting old taxonomies they already had for API misuse, but they added two new ones: intent misuse and hallucination misuse.

Lu: Intent misuse means the model picks an API that looks correct syntactically but doesn't actually match what you were trying to achieve semantically.

Meng: And halluation misuse is when the model just makes up API methods or parameters that don't exist at all in the actual library documentation.

Lalam: They found four main patterns: intent misuse, hallucination misuse, missing item misuse where they forget a required part, and redundancy misuse where they call things unnecessarily.

The paper's summary: Tom: So what do these findings tell us about the models themselves? It seems like Qwen2 point 5-Coder actually performed best overall in their tests <ref:2503.22821#pg1>.

Jane: That’s right, Qwen2 point 5-Coder showed the highest overall accuracy across both Python and Java, combining good fluency with precise matching compared to StarCoder and Copilot <ref:2503.22821#pg1>.

Lu: But even though Qwen2 point 5-Coder had fewer unique errors than the others in some specific tasks—like Python method infilling—all three models ended up struggling with the same kinds of failures <ref:2503.22821#pg1>.

Meng: It suggests that whatever these models are doing, there’s a fundamental problem with how they reason about what a library actually expects semantically.

Lalam: Finding one shows that Qwen2 point 5-Coder is the most accurate overall, but finding two states that hallucination misuse is still the most common issue across the board <ref:2503.22821#pg1>.

The paper's improvements: Tom: So what’s the fix they propose? They aren't just pointing out problems; they’re suggesting how to actually repair that broken code automatically.

Jane: They propose something called Dr.Fix, which is an LLM-based automatic repair approach guided by their new taxonomy of misuse types.

Lu: This Dr.Fix system integrates detection and reasoning through multiple stages of prompting, combining classification and reasoning to suggest fixes that are both syntactically correct and semantically meaningful.

Meng: And the results on Dr.Fix are pretty strong; it improved BLEU scores by up to thirty-eight point, which is a big jump over baseline methods.

Lalam: They also showed that Dr.Fix significantly enhances the refusal rate metric, which checks if the models can avoid hallucinating API misuses even when no misuse is actually present in the code snippet.

Conclusion: Tom: So to wrap up, we see that while there are persistent challenges like intent misalignment and hallucinations, a taxonomy-guided repair system like Dr.Fix can make a real difference in how we handle these errors.

Jane: It’s really about moving beyond just detecting the error to actually fixing it using structured reasoning, which is what they show works better than just prompting the model again.

Lu: The paper notes that Dr.Fix is backbone-agnostic and taxonomy-driven, meaning it should apply to general third-party APIs across different languages.

Meng: It’s not a silver bullet, though; the authors admit that their initial classification relies on subjective human annotations, and they also point out that generalizing these results to totally new programming languages is still a big question.

Lalam: They are planning to expand their taxonomy and dataset as new patterns of misuse pop up with newer code generation models.

Tom: That’s the status on "Identifying and Mitigating API Misuse in Large Language Models." It shows that we need better ways to make sure the AI isn't just guessing what a library intends for it to do.

Terry Yue Zhuo, Junda He, Jiamou Sun, Zhenchang Xing, David Lo, John Grundy, Xiaoning Du

Monash University · CSIRO's Data61

cs.SE

Submitted: 2025-03-28

Updated: 2026-11-13

Comments: Accepted at IEEE Transactions on Software Engineering (TSE)

Code: https://github.com/huggingface/llm-ls

Project page: https://terryyz.github.io

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 82/100

The gist: API misuse in code generated by large language models (LLMs) presents a serious and growing challenge in software development This work conducts a large-scale study of API misuse patterns in…

Key concepts

Misuse Taxonomy
The authors adapted existing taxonomies to specifically categorize errors in LLM-generated code. They retained known issues like 'Missing' and 'Redundant' but added new categories: 'Intent misuse' (wrong API choice), 'Hallucination misuse' (fabricating APIs), and the two others, providing a structured way to classify API mistakes.
Dr.Fix
Dr.Fix is an LLM-based Automatic Program Repair approach guided by the developed taxonomy. It uses multiple stages of LLM prompting—detection, classification, and reasoning—to propose fixes that are both syntactically correct and semantically meaningful, aiming to fix complex errors like hallucinations.
Intent Misuse
This misuse occurs when an LLM selects an API that is technically valid for the code but does not match the actual intended task. It means the model misunderstands the user's goal and chooses a semantically inappropriate function, leading to incorrect logic or output.
Hallucination Misuse
This refers to when an LLM invents API methods or parameters that do not actually exist in the real library documentation. Hallucinations are a major problem, as they lead to completely fabricated code elements that cause runtime errors when executed.

Terminology

Summary

API misuse in code generated by large language models (LLMs) presents a serious and growing challenge in software development This work conducts a large-scale study of API misuse patterns in LLM-generated code, analyzing both method selection and parameter usage across Python and Java, using three representative LLMs (StarCoder-7B, Qwen2. p. 05-Coder-7B, and GitHub Copilot) Based on extensive manual annotation of 3,209 methodlevel and 3,492 parameter-level misuses, four recurring misuse types are identified The evaluation reveals persistent challenges in API usage, particularly hallucination and intent misalignment To address these issues, Dr. p. 0Fix, an LLM-based automatic repair approach guided by the taxonomy, is proposed

Misuse Taxonomy Development

The authors adapt and refine existing API misuse taxonomies to the context of LLM-generated code They retain violation types that can be reliably identified from snippet-level context, such as Missing and Redundant, and introduce two additional categories: Intent misuse and Hallucination misuse The four recurring misuse patterns identified are (1) Intent misuse, where a model selects an API that is syntactically valid but semantically inappropriate for the intended task; (2) Hallucination misuse, where the model produces non-existent or fabricated API methods or parameters; (3) Missing item misuse, where required API methods or parameters are omitted; and (4) Redundancy misuse, where unnecessary API calls or arguments are introduced

Model Performance Comparison

The study evaluates three representative decoder-only LLMs used in IDE settings: StarCoder-7B, Qwen2.5-Coder-7B, and Copilot The performance comparison shows that Qwen2. p. 75-Coder achieves the highest overall accuracy across both languages, combining strong fluency with precise matching Specifically, in Python Method Infilling, Qwen2. p. 95-Coder has 15 unique errors compared to StarCoder's 126 and Copilot's 466 The results indicate that while Qwen2.5-Coder shows fewer unique errors, all three models converge on many of the same failure cases, reflecting persistent limitations in LLM reasoning about API semantics

Dr.Fix Repair Efficacy

Dr.Fix is proposed as an LLM-based APR method that improves repair accuracy compared to baseline prompting and existing repair methods Dr. p. 0Fix achieves substantial improvements, with BLEU scores increasing by up to 38. p. 14 points and exact match rates improving by up to 40 percentage points across different models and languages The refusal rate metric evaluates the models’ ability to avoid hallucinating API misuses when no misuse exists, and Dr. p. 1Fix significantly enhances these rates across all tested models

Key Findings on Misuse Patterns

Finding 1 reveals that Qwen2.5-Coder achieves the highest overall accuracy across both languages, combining strong fluency with precise matching Finding 2 states that Hallucination remains the most common misuse overall, although Qwen2. p. 95-Coder reduces its occurrence compared to StarCoder and Copilot Missing Item errors remain prominent in Python parameter prediction for all models Intent misuses continue to appear frequently but are less severe in Qwen2. p. 95-Coder, suggesting improved grounding and fewer redundant errors

Mitigation Strategy

Dr.Fix is designed to be backbone-agnostic and taxonomy-driven, making it applicable to general-purpose third-party APIs across different programming languages The workflow of Dr. p. 2Fix integrates detection and reasoning capabilities through multiple stages of LLM prompting It combines detection, classification, and reasoning to produce candidate fixes that aim to be both syntactically correct and semantically meaningful The results demonstrate that Dr.Fix significantly enhances API misuse repair across all models, achieving up to 130% improvement in exact match rates compared to baseline approaches

Limitations and Future Work

A threat to internal validity is the reliance on subjective human annotations for the initial classification of API misuse A significant threat to external validity is the generalizability of the findings to other programming languages, APIs, and LLMs Future work should focus on extending Dr.Fix’s capabilities to handle more complex API misuse scenarios and investigating its applicability to other programming languages The authors plan to expand their taxonomy and dataset as new patterns of API misuse emerge with the evolution of code generation models

Conclusion

In this work, LLMs struggle significantly with hallucination and intent misalignment, with distinct patterns emerging between programming languages The shared error patterns between Copilot and StarCoder highlight fundamental challenges in API usage that persist across different model architectures Dr.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. DrFix can perform an LLM-based Automatic Program Repair that improves repair accuracy by up to 38.4 BLEU and 40% in exact match on benchmark datasets compared to baseline prompting and existing repair methods.

  2. The improved system can detect four recurring misuse types in LLM-generated API code: Intent misuse, Hallucination misuse, Missing item misuse, and Redundancy misuse by building on an adapted taxonomy.

  3. The system can perform API Method Infilling and Parameter Completion tasks with higher accuracy across different models and languages than current baselines, as evidenced by the performance tables.

  4. The DrFix approach significantly enhances the refusal rates of LLMs across all tested architectures, demonstrating its ability to avoid hallucinating an API misuse when none exists in the code snippet.

  5. The system can provide a taxonomy-guided LLM-based APR method that leverages semantic understanding and structured reasoning to propose fixes, which is more suitable for addressing issues like hallucinations or incorrect intent selections prevalent in LLM outputs.

Sources

Related papers