Harder Than the Task: From Technical to Organizational Difficulty in AI Benchmarking
cs.CY, cs.AI
Submitted: 2026-11-02
Updated: 2026-11-02
Comments: 43 pages, 6 figures, 9 tables
License: http://creativecommons.org/licenses/by/4.0/
The gist: AI benchmarks have made remarkable progress in measuring the ability of AI systems to perform increasingly demanding cognitive and technical tasks.
Terminology
Abstract
AI benchmarks have made remarkable progress in measuring the ability of AI systems to perform increasingly demanding cognitive and technical tasks. Yet, many of these benchmarks treat technical tasks as the unit of analysis, while much economically valuable work takes place within organizations, where tasks are embedded in a system of relations. Within organizations, successful task performance depends not only on solving the technical demands of the task but also on navigating the interdependence of activities, distributed knowledge, differentiated authority, temporal constraints, conflicting interests, and changing environmental conditions while the task is underway. Accordingly, we argue that organizational task difficulty constitutes a distinct and insufficiently recognized dimension of AI capability, and that existing approaches to benchmarking provide an incomplete basis for assessing how effectively AI agents can perform real-world organizational work. By drawing on organization theory, we develop a taxonomy of the structure of work that characterizes the conditions that agents face in organizational work. Applying this taxonomy to analyze existing benchmarks, we find limited and uneven coverage of organizational dimensions, with fewer than one quarter of prominent benchmarks claiming any of them. We therefore argue that AI benchmarks should increasingly move beyond the evaluation of technically difficult tasks toward the systematic assessment of agents' ability to navigate the organizational conditions under which economically valuable work takes place.
Sources
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
- How will Language Modelers like ChatGPT Affect Occupations and Industries?
- Agentic Misalignment: How LLMs Could Be Insider Threats
- Remote Labor Index: Measuring AI Automation of Remote Work
- Crashing Waves vs. Rising Tides: Findings on AI Automation from Thousands of Worker Evaluations of Labor Market Tasks
- SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?
- GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks
- Humanity's Last Exam
- TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks
- OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks
Related papers
- AI-Mediated Communication Can Steer Collective Opinion
- Epistemic Infrastructures of Science in AI Era Should Rebalance Costs of Generation and Verification
- AI out-persuades expert humans in text-based dialogue
- Evaluating Ambient Clinical Scribes in India: The Need for Multilingual Real-World Clinical Conversation Data
- AI Value Alignment for Evolving Social Norms
- Voting Biases in Decentralized Autonomous Organization (DAO) Governance