Identifying and Mitigating API Misuse in Large Language Models
summary
The gist
API misuse in code generated by large language models (LLMs) presents a serious and growing challenge in software development This work conducts a large-scale study of API misuse patterns in…
In short
This study analyzed API misuse patterns in code generated by three LLMs (StarCoder-7B, Qwen2.5-Coder-7B, and Copilot) across Python and Java. It identified four recurring misuse types: Intent misuse, Hallucination misuse, Missing item misuse, and Redundancy misuse. A proposed LLM-based repair method called Dr.Fix was shown to significantly improve API usage accuracy by up to 38.4 BLEU points.
Key concepts
- Misuse Taxonomy
- The authors adapted existing taxonomies to specifically categorize errors in LLM-generated code. They retained known issues like 'Missing' and 'Redundant' but added new categories: 'Intent misuse' (wrong API choice), 'Hallucination misuse' (fabricating APIs), and the two others, providing a structured way to classify API mistakes.
- Dr.Fix
- Dr.Fix is an LLM-based Automatic Program Repair approach guided by the developed taxonomy. It uses multiple stages of LLM prompting—detection, classification, and reasoning—to propose fixes that are both syntactically correct and semantically meaningful, aiming to fix complex errors like hallucinations.
- Intent Misuse
- This misuse occurs when an LLM selects an API that is technically valid for the code but does not match the actual intended task. It means the model misunderstands the user's goal and chooses a semantically inappropriate function, leading to incorrect logic or output.
- Hallucination Misuse
- This refers to when an LLM invents API methods or parameters that do not actually exist in the real library documentation. Hallucinations are a major problem, as they lead to completely fabricated code elements that cause runtime errors when executed.
Terminology used across episodes
This episode discusses
- Identifying and Mitigating API Misuse in Large Language Models · Paper Radio
- LLaMA: Open and Efficient Foundation Language Models
- GPT-4 Technical Report
- StarCoder 2 and The Stack v2: The Next Generation
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions · Paper Radio
- GitChameleon: Unmasking the Version-Switching Capabilities of Code Generation Models
- LibEvolutionEval: A Benchmark and Study for Version-Specific Code Generation
- Evaluating Pre-trained Language Models for Repairing API Misuses
- Code Llama: Open Foundation Models for Code
- Type Prediction With Program Decomposition and Fill-in-the-Type Training
- Qwen2.5-Coder Technical Report
- Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code
- LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation
- CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification
- The Llama 3 Herd of Models · Paper Radio
- From Code to Courtroom: LLMs as the New Software Judges
- Evaluating Large Language Models Trained on Code
- Bugs in Large Language Models Generated Code: An Empirical Study
- What's Wrong with Your Code Generated by Large Language Models? An Extensive Study
The paper
Identifying and Mitigating API Misuse in Large Language Models · Read on arXiv
Terry Yue Zhuo, Junda He, Jiamou Sun, Zhenchang Xing, David Lo, John Grundy, Xiaoning Du
Monash University · CSIRO's Data61
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "Identifying and Mitigating API Misuse in Large Language Models".
Jane: Here's the gist of the paper. API misuse in code generated by large language models (LLMs) presents a serious and growing challenge in software development This work conducts a large-scale study of API misuse patterns in LLM-generated code, analyzing both method selection and parameter usage across Python and Java, using three representative LLMs (StarCoder-7B, Qwen2.5-Coder-7B, and GitHub Copilot) Based on extensive manual annotation of 3,209 methodlevel and 3,492 parameter-level misuses, four recurring misuse types are identified The evaluation reveals persistent challenges in API usage, particularly hallucination and intent misalignment To address these issues, Dr.Fix, an LLM-based automatic repair approach guided by the taxonomy, is proposed Misuse Taxonomy Development The authors adapt and refine existing API misuse taxonomies to the context of LLM-generated code They retain violation types that can be reliably identified from snippet-level context, such as Missing and Redundant, and introduce two additional categories: Intent misuse and Hallucination misuse The four recurring misuse patterns identified are (1) Intent misuse, where a model selects an API that is syntactically valid but semantically inappropriate for the intended task; (2) Hallucination misuse, where the model produces non-existent or fabricated API methods or parameters; (3) Missing item misuse, where required API methods or parameters are omitted; and (4) Redundancy misuse, where unnecessary API calls or arguments are introduced Model Performance Comparison The study evaluates three representative decoder-only LLMs used in IDE settings: StarCoder-7B, Qwen2.5-Coder-7B, and Copilot The performance comparison shows that Qwen2.5-Coder achieves the highest overall accuracy across both languages, combining strong fluency with precise matching Specifically, in Python Method Infilling, Qwen2.5-Coder has 15 unique errors compared to StarCoder's 126 and Copilot's 466 The results indicate that while Qwen2.5-Coder shows fewer unique errors, all three models converge on many of the same failure cases, reflecting persistent limitations in LLM reasoning about API semantics Dr.Fix Repair Efficacy Dr.Fix is proposed as an LLM-based APR method that improves repair accuracy compared to baseline prompting and existing repair methods Dr.Fix achieves substantial improvements, with BLEU scores increasing by up to 38.4 points and exact match rates improving by up to 40 percentage points across different models and languages The refusal rate metric evaluates the models’ ability to avoid hallucinating API misuses when no misuse exists, and Dr.Fix significantly enhances these rates across all tested models Key Findings on Misuse Patterns Finding 1 reveals that Qwen2.5-Coder achieves the highest overall accuracy across both languages, combining strong fluency with precise matching Finding 2 states that Hallucination remains the most common misuse overall, although Qwen2.5-Coder reduces its occurrence compared to StarCoder and Copilot Missing Item errors remain prominent in Python parameter prediction for all models Intent misuses continue to appear frequently but are less severe in Qwen2.5-Coder, suggesting improved grounding and fewer redundant errors Mitigation Strategy Dr.Fix is designed to be backbone-agnostic and taxonomy-driven, making it applicable to general-purpose third-party APIs across different programming languages The workflow of Dr.Fix integrates detection and reasoning capabilities through multiple stages of LLM prompting It combines detection, classification, and reasoning to produce candidate fixes that aim to be both syntactically correct and semantically meaningful The results demonstrate that Dr.Fix significantly enhances API misuse repair across all models, achieving up to 130% improvement in exact match rates compared to baseline approaches Limitations and Future Work A threat to internal validity is the reliance on subjective human annotations for the initial classification of API misuse A significant threat to external validity is the generalizability of the findings to other programming languages, APIs, and LLMs Future work should focus on extending Dr.Fix’s capabilities to handle more complex API misuse scenarios and investigating its applicability to other programming languages The authors plan to expand their taxonomy and dataset as new patterns of API misuse emerge with the evolution of code generation models In this work, LLMs struggle significantly with hallucination and intent misalignment, with distinct patterns emerging between programming languages The shared error patterns between Copilot and StarCoder highlight fundamental challenges in API usage that persist across different model architectures Dr.
Title and authors: Tom: So what's the main takeaway from this paper, beyond just saying LLMs make mistakes? It’s about how specific those mistakes are, and which kinds are happening most often.
Jane: They’ve identified four recurring misuse types by adapting old taxonomies they already had for API misuse, but they added two new ones: intent misuse and hallucination misuse.
Lu: Intent misuse means the model picks an API that looks correct syntactically but doesn't actually match what you were trying to achieve semantically.
Meng: And halluation misuse is when the model just makes up API methods or parameters that don't exist at all in the actual library documentation.
Lalam: They found four main patterns: intent misuse, hallucination misuse, missing item misuse where they forget a required part, and redundancy misuse where they call things unnecessarily.
The paper's summary: Tom: So what do these findings tell us about the models themselves? It seems like Qwen2 point 5-Coder actually performed best overall in their tests <ref:2503.22821#pg1>.
Jane: That’s right, Qwen2 point 5-Coder showed the highest overall accuracy across both Python and Java, combining good fluency with precise matching compared to StarCoder and Copilot <ref:2503.22821#pg1>.
Lu: But even though Qwen2 point 5-Coder had fewer unique errors than the others in some specific tasks—like Python method infilling—all three models ended up struggling with the same kinds of failures <ref:2503.22821#pg1>.
Meng: It suggests that whatever these models are doing, there’s a fundamental problem with how they reason about what a library actually expects semantically.
Lalam: Finding one shows that Qwen2 point 5-Coder is the most accurate overall, but finding two states that hallucination misuse is still the most common issue across the board <ref:2503.22821#pg1>.
The paper's improvements: Tom: So what’s the fix they propose? They aren't just pointing out problems; they’re suggesting how to actually repair that broken code automatically.
Jane: They propose something called Dr.Fix, which is an LLM-based automatic repair approach guided by their new taxonomy of misuse types.
Lu: This Dr.Fix system integrates detection and reasoning through multiple stages of prompting, combining classification and reasoning to suggest fixes that are both syntactically correct and semantically meaningful.
Meng: And the results on Dr.Fix are pretty strong; it improved BLEU scores by up to thirty-eight point, which is a big jump over baseline methods.
Lalam: They also showed that Dr.Fix significantly enhances the refusal rate metric, which checks if the models can avoid hallucinating API misuses even when no misuse is actually present in the code snippet.
Conclusion: Tom: So to wrap up, we see that while there are persistent challenges like intent misalignment and hallucinations, a taxonomy-guided repair system like Dr.Fix can make a real difference in how we handle these errors.
Jane: It’s really about moving beyond just detecting the error to actually fixing it using structured reasoning, which is what they show works better than just prompting the model again.
Lu: The paper notes that Dr.Fix is backbone-agnostic and taxonomy-driven, meaning it should apply to general third-party APIs across different languages.
Meng: It’s not a silver bullet, though; the authors admit that their initial classification relies on subjective human annotations, and they also point out that generalizing these results to totally new programming languages is still a big question.
Lalam: They are planning to expand their taxonomy and dataset as new patterns of misuse pop up with newer code generation models.
Tom: That’s the status on "Identifying and Mitigating API Misuse in Large Language Models." It shows that we need better ways to make sure the AI isn't just guessing what a library intends for it to do.
More episodes
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines
- 2601.15318-On the closest balanced game
- 1905.10490-Finding new routes for integrating Multi-Agent Systems using Apache Camel