Can your AI agent be cheaper? Investigating the effects of task specifications on token spend in agentic coding tasks
cs.AI
Submitted: 2026-08-26
Updated: 2026-08-26
Code: https://github.com/github/spec-kit
License: http://creativecommons.org/licenses/by/4.0/
The gist: Agentic coding workflows are now widely deployed in real-world systems.
Terminology
Abstract
Agentic coding workflows are now widely deployed in real-world systems. With long-horizon reasoning and tool use, token usage has become an important consideration for both cost and efficiency. Two engineers using AI will solve the same problem differently. How the specification of a task shapes an agent's token spend, and whether that spend can be predicted in advance, are open questions. Here, we study the effects of different task specifications on agentic token spend with the Kimi K3 model at three thinking efforts. Across 2,700 runs, we show that reducing a full task specification to a bare user story raises token spend by 29.7%, while run-to-run variance remains unaffected by any prompt changes. We show that prompt-sensitivity is task-dependent, running from 13% to 115%. We fit a simple predictor that can price a full distribution of task specifications and thinking effort configurations from a single cheap probe on an unseen task within 36%, improving over prior work in predicting token spend. Our work provides initial results quantifying the effects of task specification on agentic token spend and introduces a method that can be used to systematically evaluate the cost of AI coding workflows.
Sources
- OptimalThinkingBench: Evaluating Over and Underthinking in LLMs
- When Prompt Under-Specification Improves Code Correctness: An Exploratory Study of Prompt Wording and Structure Effects on LLM-Based Code Generation
- Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMs
- How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks
- SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
- Value of Information: A Framework for Human-Agent Communication
- SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?
- AI Agents That Matter
- What Makes a Good Bug Report for an AI Agent?
- ClarEval: A Benchmark for Evaluating Clarification Skills of Code Agents under Ambiguous Instructions
- SelfBudgeter: Adaptive Token Allocation for Efficient LLM Reasoning
- Re-Evaluating Code LLM Benchmarks Under Semantic Mutation
- Agentless: Demystifying LLM-based Software Engineering Agents
- SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
- Adaptive Test-Time Compute Allocation for Reasoning LLMs via Constrained Policy Optimization
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems