READY or Not: Reliable Enterprise Agent Deployment
cs.AI
Submitted: 2026-09-02
Updated: 2026-09-02
Code: https://github.com/UKGovernmentBEIS/inspect_ai
License: http://creativecommons.org/licenses/by/4.0/
The gist: An AI agent can perform well on benchmarks and still be unsuitable for deployment.
Terminology
Abstract
An AI agent can perform well on benchmarks and still be unsuitable for deployment. Existing AI-agent benchmarks measure whether an agent can complete realistic professional work, whereas enterprise deployment asks a different question: whether an agent can meet a required reliability level, under acceptable human oversight, and at tolerable cost. We introduce Reliable Enterprise Agent Deployment (READY), a framework for qualifying AI agents for deployment on enterprise workflows. READY preserves each workflow's own definition of successful execution while applying a common qualification procedure. Given an agent, a workflow, and a class of candidate oversight policies, READY measures the reliability and operating cost of the human-AI system, selects the minimum-cost policy that satisfies a specified reliability target, and statistically qualifies it on held-out cases. The resulting deployment profile characterizes the supported operating point: reliability, human-oversight burden, and cost. READY is implemented as an open testbed that decouples workflow specification, execution, evaluation, and qualification, and runs on existing agent-evaluation infrastructure. In an end-to-end clinical-audit case study spanning 16 agent systems and 750 cases, READY reveals differences hidden by autonomous performance: two systems separated by only 0.3 points in autonomous accuracy (72.8% vs. 72.5%) require 39.2% versus 29.6% human review, respectively, to qualify at the same 76% reliability target under the evaluated oversight policy. READY thus shifts enterprise agent evaluation from how well can the agent perform the work? to under what conditions, and at what cost, can it be reliably deployed? By making those conditions explicit and statistically testable, READY provides a basis for comparing agent systems, setting oversight requirements, and making evidence-based deployment decisions.
Sources
- Agents' Last Exam
- GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks
- xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluations
- CliniCARE-Bench: Clinical Calibrated Audit of Medical Reasoning in EHR
- AI Agents That Matter
- Humanity's Last Exam
- Language Models (Mostly) Know What They Know
- Designing Benchmarks for Knowledge Work
- Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
- BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents
- DRBench: A Realistic Benchmark for Enterprise Deep Research
- Evaluation and Benchmarking of LLM Agents: A Survey
- TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks
- WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Setting
- Measuring Faithfulness in Chain-of-Thought Reasoning
- Selective Classification for Deep Neural Networks
- Coverage-Constrained Human-AI Cooperation with Multiple Experts
- A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification
- Conformal Prediction with Large Language Models for Multi-Choice Question Answering
- Language Models Prefer What They Know: Relative Confidence Estimation via Confidence Preferences
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems