The Reasoning Tax: Token Economics of LLM Reasoning Across Task Types and Deployment Contexts
cs.AI, cs.PF
Submitted: 2026-08-26
Updated: 2026-08-26
Comments: 15 pages, 8 figures and tables, accepted at the 2026 TPCTC Conference and will be published at Performance Evaluation and Benchmarking (Springer)
Code: https://github.com/Sachin-Wani/reasoning_tax
License: http://creativecommons.org/licenses/by/4.0/
The gist: Accuracy-only benchmarking of reasoning-capable large language models misses a central deployment question: when do extended thinking tokens earn their cost? We introduce the Token Economy Score
Terminology
Abstract
Accuracy-only benchmarking of reasoning-capable large language models misses a central deployment question: when do extended thinking tokens earn their cost? We introduce the Token Economy Score (TES), a marginal benchmarking metric that measures the accuracy gain of a reasoning model over a non-reasoning baseline, normalized by the generated-token multiplier. We define paired and approximated TES variants for model families with reasoning toggles and frontier models without direct non-reasoning counterparts. We then conduct an empirical benchmarking analysis across 151 model-benchmark evaluation runs on seven benchmarks spanning mathematics, code generation, science reasoning, instruction following, expert knowledge, knowledge recall, and research-level physics. The analysis examines three deployment-facing dimensions: which task structures yield positive marginal reasoning efficiency, how increasing reasoning effort changes TES within model families, and how deployment context changes economic viability. Results show that task structure predicts reasoning efficiency better than nominal difficulty: sequential inferencechain tasks such as AIME 2025 and LiveCodeBench show high TES, while knowledge-recall tasks such as MMLU-Pro show low TES despite their difficulty. We also find systematic diminishing returns at higher reasoning effort levels, including cases where additional thinking reduces accuracy. Finally, Reasoning Cost Share (RCS) shows that inference spend is often dominated by internal thinking, while Deployment Cost Multiplier (DCM) shows how on-premises deployment can change the economics of otherwise costly reasoning workloads. These findings support a benchmarking-driven model-selection rule: enable reasoning selectively by task type, effort level, and deployment context rather than treating it as a universally beneficial mode.
Sources
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Qwen3 Technical Report
- OckBench: Measuring the Efficiency of LLM Reasoning
- ReEfBench: Quantifying the Reasoning Efficiency of LLMs
- Measuring Massive Multitask Language Understanding
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark
- Humanity's Last Exam
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- Generalizing Verifiable Instruction Following
- Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark
- Benchmarking Distilled Language Models: Performance and Efficiency in Resource-Constrained Settings
- MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
- MathArena: Evaluating LLMs on Uncontaminated Math Competitions
- Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
- GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks
- APEX-Agents
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems