DOCSCHISEL: Adaptive Tool Documentation Optimization Framework for LLM Agents

Large language models (LLMs) increasingly rely on external tools to accomplish complex real-world tasks, making tool documentation a critical grounding resource for LLM agents. Existing studies mainly focus on improving the tool-use capabilities of LLM agents, while largely treating tool documentation as a fixed input. Although several recent works attempt to optimize tool documentation through rewriting or compression, little is known about how the information contained in tool documentation affects agent performance across different settings.

To bridge this gap, we conduct a large-scale empirical study on tool documentation for LLM agents. Our study reveals substantial heterogeneity in the information fields provided by existing tool documentation. Moreover, the effectiveness of different information fields is highly dependent on the task domain, LLM backbone, and agent paradigm, indicating that no fixed tool documentation can consistently generalize across diverse agent settings.

Motivated by these findings, we propose DOCS CHISEL, an adaptive tool documentation optimization framework for LLM agents. DOCS CHISEL analyzes failed execution traces of a target LLM agent to identify documentation-related issues, and iteratively optimizes tool documentation by adding, removing, and refining information fields for each tool. We evaluate DOCS CHISEL against two state-of-the-art baselines, i.e., EASYTOOL and DRAFT. Experimental results show that DOCS CHISEL improves the task success rate of LLM agents by 95.89% over the original tool documentation and by 75.15%, on average, over existing baselines, while incurring limited optimization time and token overhead.

In the empirical study, we collect tool documentation from 14 representative datasets, covering diverse tool-use scenarios, e.g., workflow execution, mobile automation, and software engineering tasks. In total, we collect 24,955 tools along with their corresponding tool documentation. We identify 17 common information fields, and report their prevalence across the 14 datasets. Tool name and functionality description appear in all datasets, while input parameter name, input parameter description, and input parameter type appear in most datasets. In contrast, fields supporting task-oriented tool use and result interpretation, such as default parameter value, return parameter name, return parameter type, usage example and response template, are provided much less consistently. Usage guidance and invocation constraint appear in only two datasets.

For the impact analysis, we evaluate the task success rate of two different LLM agent paradigms (i.e., ReAct and Multi-Agent) on WorkBench, using three different LLM backbones (i.e., GPT-4o, GLM-5 and Claude Haiku 4.5). The results show that the same information field may have different or even opposite effects under different agent settings. On average, adding or removing a single information field changes the task success rate of LLM agents by 6.34 percentage points in absolute terms. For example, removing task domain decreases task success rate by 3.75 to 13.75 percentage points, while adding invocation constraint improves it by 0.75 to 11.25 percentage points. When fixing the domain and agent paradigm, 12 of the 17 fields exhibit different effect directions across GPT-4o, GLM-5, and Claude Haiku 4.5. A similar variation is observed across agent paradigms, as replacing ReAct with Multi-Agent reverses the effects of several fields, including functionality description, input parameter description, parameter required flag, and response template.

The empirical study provides two insights for tool documentation optimization. First, no fixed documentation convention can consistently generalize across different LLM agent settings, since the effectiveness of information fields may vary across task domains, LLM backbones, and agent paradigms. Second, an effective tool documentation optimizer should adaptively adjust the information field composition based on the LLM agent setting according to different execution feedback.

DOCS CHISEL performs domain-aware execution profiling with the original tool documentation. When the target agent is equipped with tools from multiple task domains, DOCS CHISEL organizes the tools and queries by task domain. Within each domain, it associates each tool with the queries whose execution requires that tool, and partitions these tool-associated queries and their execution ground truth into optimization and validation sets. It then executes the target agent to collect traces on each tool's optimization set and obtain its initial task success rate on the corresponding validation set.

Then, DOCS CHISEL enters an iterative optimization loop for each tool in the current task domain. At each iteration, DOCS CHISEL performs memory-guided documentation optimization for the target tool using the queries associated with that tool in the optimization set. Instead of directly asking an LLM to rewrite the current documentation, DOCS CHISEL decomposes the optimization process into three LLM-assisted steps, i.e., failed trace diagnosis, field operation planning, and optimized documentation generation. The diagnosis step identifies why the LLM agent fails under the current tool documentation. The planning step maps the diagnosed issues to possible field-level operations. The generation step applies the planned operations to produce new tool documentation.

After generating the candidate documentation, DOCS CHISEL evaluates whether it improves the task success rate of the target LLM agent through agent re-execution, and updates the optimization feedback for subsequent iterations. Since the optimized documentation may improve some failed queries while harming previously successful ones, DOCS CHISEL considers both validation performance and regression risk during the evaluation. When optimization for a tool reaches the maximum number of iterations, DOCS CHISEL selects the candidate with the highest validation task success rate, breaking ties by choosing the shortest documentation to reduce execution context cost.

We implement a prototype of DOCS CHISEL with 2,687 lines of Python code. We select two tool-use benchmarks, i.e., WorkBench and API-Bank, covering 9 task domains, involving 74 tools and 2,072 user queries. We split the user queries into optimization, validation, and test sets with a ratio of 5:1:4. The optimization and validation sets are used during documentation optimization, while the test set is held out throughout the optimization process and used only for the final evaluation of the optimized tool documentation.

For effectiveness evaluation, DOCS CHISEL achieves the highest average tool invocation correctness and task success rate in all evaluated settings. Compared with the original documentation, DOCS CHISEL improves tool invocation correctness by 34.69% and task success rate by 95.89% on average. Compared with the two baseline approaches, it achieves an average tool invocation correctness improvement of 30.83% and task success rate improvement of 75.15%. Even against the strongest baseline in each setting, DOCS CHISEL improves tool invocation correctness and task success rate by 26.23% and 46.11% on average, respectively. DOCS CHISEL also reduces the average interquartile range of tool invocation correctness by 41.56% and 52.93%, respectively, and that of task success rate by 36.71% and 44.07%, compared with EASYTOOL and DRAFT.

For efficiency evaluation, DOCS CHISEL produces documentation containing 189.23 tokens on average, which is 24.02% longer than the original documentation but comparable to DRAFT. DOCS CHISEL consumes 6.35% fewer tokens than DRAFT for each tool during optimization. However, it requires 12.65 minutes on average, longer than EASYTOOL and DRAFT, because DOCS CHISEL repeatedly diagnoses failed traces, generates candidate documentation, and evaluates candidates through agent re-execution.

For sensitivity analysis, Claude Haiku 4.5 achieves the best performance as the optimization model, improving tool invocation correctness and task success rate over GLM-5 by 27.12% and 57.87%, respectively, and over GPT-4o by 97.37% and 164.71%. Increasing the number of optimization iterations improves both tool invocation correctness and task success rate, with gains most pronounced in the early rounds. Compared with one iteration, five iterations improve task success rate by 80.00% and tool invocation correctness by 14.50%. After five iterations, the marginal gains become small, suggesting that the optimization process has largely converged.

For the ablation study, DOCS CHISEL-NoMem, which disables the memory mechanism, reduces tool invocation correctness and task success rate by 13.64% and 76.47% compared with DOCS CHISEL, respectively. The relative task success rate improvement between DOCS CHISEL and DOCS CHISEL-NoMem increases from 4.17% at Iteration-1 to 76.47% at Iteration-5, indicating that the benefit of memory becomes more pronounced as optimization experience accumulates. Without memory, both metrics largely plateau after the second iteration. In contrast, DOCS CHISEL continues to improve by reusing previously observed documentation deficiencies, effective field operations, and editing experience.

Improvements to AI systems:

1. Adaptive tool-documentation optimization: AI agents can now automatically diagnose their own failed execution traces, identify documentation-related issues (e.g., missing parameter descriptions, ambiguous return formats), and iteratively add, remove, or refine information fields in tool documentation to maximize task success rates for their specific deployment context (task domain, LLM backbone, agent paradigm).

2. Context-aware documentation personalization: Instead of using a fixed tool documentation template, AI systems can dynamically tailor documentation per tool based on the specific LLM backbone and agent architecture (e.g., ReAct vs. Multi-Agent), since the empirical study shows that the same information field can have opposite effects across different backbones and paradigms.

3. Memory-guided iterative optimization: AI systems can maintain a memory of previously observed documentation deficiencies, effective field-level operations, and editing experiences across optimization iterations, enabling continuous improvement beyond what single-pass rewriting achieves—the memory mechanism alone improves task success rate by 76.47% over memory-less optimization.

4. Regression-aware documentation updates: AI systems can evaluate candidate documentation changes by considering both validation performance gains and regression risk (i.e., whether previously successful queries break), then select the candidate with the highest validation success rate while preferring shorter documentation to reduce execution context cost.

5. Domain-aware execution profiling: AI systems can organize tools and queries by task domain, associate each tool with the queries requiring it, and partition optimization/validation sets per domain—enabling targeted documentation fixes that respect domain-specific information needs.

What the improved AI system can do:

• Achieve 95.89% higher task success rate on tool-use benchmarks (e.g., WorkBench, API-Bank) compared to using original tool documentation, and 75.15% higher than state-of-the-art baselines (EASYTOOL, DRAFT).

• Improve tool invocation correctness by 34.69% over original documentation and 30.83% over baselines, with 26.23% improvement even against the strongest baseline in each setting.

• Reduce performance variance by 36.71–52.93% in tool invocation correctness and task success rate across different settings, making agent behavior more consistent and reliable.

• Adapt documentation in under 13 minutes per tool with only 189 tokens of optimized documentation (24% longer than original but comparable to DRAFT), consuming 6.35% fewer optimization tokens than DRAFT.

• Converge effectively within 5 optimization iterations, with most gains in early rounds, and continue improving via memory reuse rather than plateauing.

• Handle heterogeneous tool ecosystems across 14 datasets and 9 task domains (workflow execution, mobile automation, software engineering), covering 24,955 tools and 2,072 queries, by automatically deciding which of 17 information fields (e.g., default values, return types, usage examples, invocation constraints) to include or exclude per tool.

Large language models (LLMs) increasingly rely on external tools to accomplish complex real-world tasks, making tool documentation a critical grounding resource for LLM agents. Existing studies mainly focus on improving the tool-use capabilities of LLM agents, while largely treating tool documentation as a fixed input. Although several recent works attempt to optimize tool documentation through rewriting or compression, little is known about how the information contained in tool documentation affects agent performance across different settings. To bridge this gap, we conduct a large-scale empirical study on tool documentation for LLM agents. Our study reveals substantial heterogeneity in the information fields provided by existing tool documentation. Moreover, the effectiveness of different information fields is highly dependent on the task domain, LLM backbone, and agent paradigm, indicating that no fixed tool documentation can consistently generalize across diverse agent settings. Motivated by these findings, we propose DocsChisel, an adaptive tool documentation optimization framework for LLM agents. DocsChisel analyzes failed execution traces of a target LLM agent to identify documentation-related issues, and iteratively optimizes tool documentation by adding, removing, and refining information fields for each tool. We evaluate DocsChisel against two state-of-the-art baselines, i.e., EasyTool and DRAFT. Experimental results show that DocsChisel improves the task success rate of LLM agents by 95.89% over the original tool documentation and by 75.15%, on average, over existing baselines, while incurring limited optimization time and token overhead