Harness Engineering in LLM Tool Use via Agent-Native Reusable Tool Primitives
cs.SE, cs.AI, cs.CL, cs.LG, cs.MA
Submitted: 2026-09-01
Updated: 2026-09-01
Comments: 21 pages
Code: https://github.com/tatsu-lab/stanford_alpaca
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language models (LLMs) augmented with external tools have demonstrated remarkable capability in solving complex real-world tasks.
Terminology
Abstract
Large language models (LLMs) augmented with external tools have demonstrated remarkable capability in solving complex real-world tasks. However, existing approaches suffer from two key challenges: brittle multi-step and multi-turn reasoning caused by incompatible tool output types and API schemas, and performance degradation under large tool catalogues. To address these, we introduce Tool Primitives, a design that replaces rigid API schema-based invocation with natural language as the interface for tool calling, where each tool is wrapped with an LLM interface that handles schema resolution and execution internally, enabling natural inter-tool communication for nested and multi-turn tool calling. Building on Tool Primitives, we host ToolFace, a centralized repository of 25,519 functions from which LLMs dynamically retrieve only the relevant tools at inference time, eliminating the need to enumerate raw API schemas in context. To orchestrate Tool Primitives and ToolFace reliably in complex settings, we further propose HEART, a Harness Engineering framework via Agent-native, Reusable Tool Primitives, comprising a Planner, Router, and Verifier that jointly support dynamic tool invocation planning, multi-step execution, and feedback-driven recovery. Experiments on five benchmarks demonstrate that HEART outperforms SFT-based models by 10% on average and surpasses GPT-5.4, Claude-4.6-Sonnet, and Gemini-3.1-Pro by 6% on average while reducing API cost by up to 85%. On 50 real-world tasks, HEART achieves 84% task completion, 3.8 times the average of three frontier commercial models (22%).
Sources
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs
- Hammer: Robust Function-Calling for On-Device Language Models via Function Masking
- ToolACE: Winning the Points of LLM Function Calling
- APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
- LongFuncEval: Measuring the effectiveness of long context models for function calling
- Agent Primitives: Reusable Latent Building Blocks for Multi-Agent Systems
- $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
- GPT-4 Technical Report
- The Llama 3 Herd of Models
- Mixtral of Experts
- DeepSeek-V3 Technical Report
- GPT-4o System Card
- Qwen2.5 Technical Report
- Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
- Qwen3 Technical Report
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
- Prompt Injection Attack to Tool Selection in LLM Agents
Related papers
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces
- PackMonitor: Enabling Zero Package Hallucinations Through Decoding-Time Monitoring
- IntentCoding: Amplifying User Intent in Code Generation
- Incentives and Outcomes in Bug Bounties