The AI Era's Lines of Code? A Case Study on What an Engineer's Token Spend Measures
cs.SE, cs.AI
Submitted: 2026-11-10
Updated: 2026-11-10
Code: https://github.com/CMUSTRUDEL/token-study-replication
License: http://creativecommons.org/licenses/by/4.0/
The gist: Organizations use engineers' token spend to monitor AI adoption, set budgets, and measure productivity, but the metric's relationship with engineering output is unclear.
Abstract
Organizations use engineers' token spend to monitor AI adoption, set budgets, and measure productivity, but the metric's relationship with engineering output is unclear. We link five weeks of token usage in coding agents to the commits of 404 engineers at JetBrains, and analyze a company survey of 412 employees. We find that token spend varied mostly between engineers, and which engineers spent the most stayed largely the same over the five weeks. Among weeks with similar commit counts, the 90th percentile of spend was 39 to 240 times the 10th. Within an engineer, doubling spend was associated with about 10% more commits. Available work metrics explained 38% of the variation in spend between engineers and 13% of the week-to-week variation within an engineer. In the survey, respondents in technical roles reported in nearly nine of ten agent evaluations using agents for tasks beyond coding, such as analysis, research, and planning. Token spend therefore measures how much engineers use agents across many kinds of work and not the outcome and value of that work. We recommend using token spend at most for resource accounting, and evaluating agent-assisted work with evidence about tasks and outcomes.
Sources
- AI Writes Faster Than Humans Can Review: A Longitudinal Study of an Enterprise 2x Mandate
- Adoption and Impact of Command-Line AI Coding Agents: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI
- How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks
- SWE-Effi: Re-Evaluating Software AI Agent System Effectiveness Under Resource Constraints
- The Best Programming Language for Tokenmaxxing: An Investigation of Coding Agent Behavior Across Programming Languages
- GitHub Copilot and Developer Productivity: An Observational Dose-Response Analysis
- A Few Pages of Markdown: Committed AI Configuration and Lower Quality Cost after Coding-Agent Adoption
- Tokenomics: Quantifying Where Tokens Are Used in Agentic Software Engineering
- Token Reduction Is Not Cost Reduction
- Babbling Suppression: Making LLMs Greener One Token at a Time
- The Token Efficiency Index: A Peer-Benchmarked Composite Indicator for AI Token Efficiency
- Inference Economics of Enterprise Coding Agents: A Case Study of Cloud vs. On-Premise LLMs
- Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters