PluginEval: A Diagnostic Benchmark for Fine-Grained Error Attribution in Function Calling
Dongjie Xu, Julius, Hanchi Dong, Minghua Tang, Yuxuan Sun, Ziwei Nie, Zicheng Liu, Dujun Qing, Jiajie Xu
cs.AI
Submitted: 2026-08-09
Updated: 2026-08-11
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reliable evaluation of tool routing is critical as Large Language Models increasingly operate as autonomous agents.
Terminology
Abstract
Reliable evaluation of tool routing is critical as Large Language Models increasingly operate as autonomous agents. Current benchmarks face three structural limitations: data distributions that follow a power law leave rare scenarios underrepresented; the absence of adversarial hard negatives obscures performance differences across models; and annotation pipelines depend on LLM judgments that have not been validated through execution. In this paper, we introduce PluginEval, a benchmark constructed through a two-stage framework that systematically mitigates these limitations. First, we formulate tool routing as a sequence of three decisions and separate generation from verification. LLMs propose candidate calls, while deterministic validation and real API execution provide reliable quality signals. Second, we decompose each plugin by capability, intent, and boundary to identify trigger and exclusion scenarios. We then generate queries at different difficulty levels to fill coverage gaps, including adversarial negatives targeting three failure modes, and return them to the first stage for annotation. This process creates a closed loop that iterates until coverage converges. For evaluation, we move beyond aggregate accuracy. An LLM judge anchored to gold annotations classifies failures as missed calls, spurious calls, or parameter errors, producing a detailed error profile for each model. We evaluate five model families, including proprietary models and models with open weights, analyze their performance across difficulty levels and error categories, and validate the judge through agreement with human annotations.
Sources
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
- Beyond Perfect APIs: A Comprehensive Evaluation of LLM Agents Under Real-World API Complexity
- EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents
- APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets
- Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Models
- Towards Reliable Benchmarking: A Contamination Free, Controllable Evaluation Framework for Multi-step LLM Function Calling
- tinyBenchmarks: evaluating LLMs with fewer examples
- OpenAI GPT-5 System Card
- Gemini: A Family of Highly Capable Multimodal Models
- HammerBench: Fine-Grained Function-Calling Evaluation in Real Mobile Device Scenarios
- Self-Consistency Improves Chain of Thought Reasoning in Language Models
- Self-Preference Bias in LLM-as-a-Judge
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- Qwen3 Technical Report
- $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
- Enhancing Tool Calling in LLMs with the International Tool Calling Dataset
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems