Agents' Last Exam

arXiv:2606.05405 · cs.AI, cs.CL, cs.LG · Submitted 2026-06-03 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Agents' Last Exam".

Jane: Detailed Research Summary of "Agents’ Last Exam (ALE)" Benchmark Paper As a diligent researcher,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we’re talking about this paper called Agents' Last Exam, and the main thing is that current AI systems are getting good scores on many tests, but those scores don't actually translate into agents being used in real jobs yet.

Jane: Right. The authors of Agents' Last Exam argue that most benchmarks are missing the mark because they don’t measure performance on tasks that actually matter economically in the real world.

Lu: They introduce this benchmark, ALE, which is specifically designed to test AI agents on long horizon, economically valuable real world tasks with verifiable outcomes.

Meng: So it's not just about whether an agent can answer a question; it’s about whether it can complete a complex job that takes time and has actual value for a company.

Lalam: The abstract mentions they developed this benchmark by working with over two hundred fifty industry experts to make sure the tasks are real <ref:2606.05405#pg1>.

Tom: Exactly. These tasks cover non-physical industries, and they define those industries using something called O*NET or SOC two thousand eighteen which is basically the U <ref:2606.05405#pg1>.S. federal occupational taxonomy for jobs.

Jane: That means they aren't testing some made-up scenarios; they are testing things that look like actual professional workflows from industries like manufacturing or healthcare.

Lu: The structure of this benchmark is organized around a task taxonomy with fifty-five subfields grouped into thirteen industry clusters, covering over one thousand tasks <ref:2606.05405#pg1>.

Meng: That's a lot of variety, and the point they are making is that the hardest tier of these tasks still isn't saturated.

Tom: That’s what’s interesting because across mainstream harness and backbone configurations, the average full pass rate for those hardest tasks is below one percent.

Jane: So when you hear that number, it suggests that even powerful AI agents are struggling to consistently complete these professional level jobs.

Lu: ALE is designed to be a living benchmark, meaning their task pool keeps growing continuously as new workflows and industries get added over time.

Paper summary: Meng: And they say the bigger picture here is that ALE isn't just another leaderboard for bragging rights; it’s meant to be a tool for closing that gap between what benchmarks measure and what actually drives Gross Domestic Product.

Tom: So the core thesis of Agents' Last Exam is that we need better ways to measure if AI agents can actually make an economic impact, not just if they can pass a test.

Jane: And the authors are pointing out that this gap between benchmark success and real impact is largely an evaluation problem.

Lu: This paper introduces ALE as the answer to that by focusing on tasks with verifiable outcomes instead of just subjective judging.

Meng: The methodology they use is focused on determinism, which means they rely on scoring based on observable artifacts or rubrics rather than letting a large language model make a purely subjective judgment.

Tom: They define rigor around three pillars: representativeness, complexity, and verifiability for these tasks.

Jane: Representativeness means the tasks have to truly reflect how professionals work and the software environments agents use day to day.

Lu: Complexity means we’re looking at end-to-end deliverable work that requires sustained planning instead of just a single quick answer.

Meng: And verifiability is key, so they try to avoid open-ended judging whenever there’s a way to check the answer deterministically, like by scoring based on bytes or geometry.

Tom: They organize the workload into three difficulty tiers: near-term, full-spectrum, and the hardest tier which is called Last-Exam Tier.

Jane: And that last tier has thirty-eight tasks specifically designed to test the limits of long horizon, tool-intensive work in professional practice.

Lu: The total benchmark includes nine hundred sixty expert authored task workflows, totaling one thousand four hundred ninety individual task instances.

Meng: So when you look at the current results, they show that even frontier agents using configurations like "Codex with GPT-five point five" only score below fifty percent on the easiest tier and under ten percent on the hardest tier <ref:2606.05405#pg1>.

Tom: That’s a pretty stark picture for those who are trying to build these complex agents right now.

Paper summary: Jane: And mainstream agents, like Claude Code, are seeing near zero pass rates at that last exam difficulty level.

Lu: The paper also details the task distribution across different engineering and science fields, listing things like Manufacturing and Indust. Sys with one hundred seventy-three tasks and Health and Medicine with one hundred fifty-five tasks <ref:2606.05405#pg1>.

Meng: It's interesting how they map out these specific industrial categories, which gives us a clearer idea of where the current agent capabilities are falling short in terms of real-world application.

Tom: The authors also emphasize that ALE is intended to be a living benchmark, meaning their task pool will just keep growing as new workflows and industries get onboarded.

Jane: So it’s not a static thing; it’s meant to evolve with the industry and the technology itself.

Lu: They stress that this isn't just another leaderboard for showing off scores; ALE is positioned as an instrument for closing that gap between benchmark success and GDP relevant impact.

Meng: That means they are trying to figure out if AI agents have reached a capability level sufficient to sustain the kind of long horizon, tool-intensive work professional practices actually demand.

Tom: So the big question here is whether we can get an agent that consistently passes those hard tests and starts showing real economic value.

Jane: The authors are essentially saying that if we want AI to be truly useful in professional domains, our way of measuring success needs to change from simple performance metrics to something tied directly to economic output.

Lu: This benchmark, Agents' Last Exam, is presented as the definitive tool for assessing whether current AI agents can transition from impressive demonstrations into sustaining economically valuable professional impact.

Meng: It’s a very practical framing of the problem that we need to address right now in terms of deployment and real-world utility.

Tom: So for anyone listening, think of Agents' Last Exam as a way to check if the AI you’re using can actually do the hard, long-term work that professionals in those specific industries are paying for.

Conclusion: Tom: So we're wrapping up our look at Agents' Last Exam, and we need to talk about what that title means and who actually wrote this thing.

Jane: It really boils down to this paper being a test for agents in the real world, not just another abstract leaderboard.

Lu: The authors developed Agents' Last Exam by collaborating with over two hundred industry experts to make sure the tasks they picked were genuinely hard and valuable.

Meng: So they weren't just guessing problems; these tasks are based on actual professional work from industries like manufacturing or healthcare, using that O*NET taxonomy.

Lalam: And what this means is that we are finally getting a benchmark that tries to measure if an AI agent can actually do a sustained job with long-term planning and real economic value.

Tom: Exactly. The title itself, Agents' Last Exam, suggests they're looking for the final test before an agent is ready for professional deployment.

Jane: It implies that we need to stop focusing on how many simple tasks an AI can do and start focusing on whether it can handle the complex stuff that actually drives GDP.

Lu: This moves the focus away from just raw performance numbers and toward something more useful—a measure of economic impact.

Meng: For someone listening, this means if you're building an agent, you need to ask yourself if it can pass these kinds of tough, long-horizon tests before you expect it to be useful in a real business setting.

Lalam: And the results they’re showing—where even powerful agents struggle with the hardest tier—suggest that we still have a lot to do before this gap between benchmark success and actual impact closes.

Tom: Yeah, so the paper is pushing us toward a much more practical way of evaluating these systems if we actually want to see AI used in high-value professional fields.

Jane: And that brings us nicely into what the authors found about those specific performance numbers across different agent configurations.

University of California, Berkeley

cs.AI, cs.CL, cs.LG

Submitted: 2026-06-03

Updated: 2026-10-09

Comments: Project website: https://agents-last-exam.org Code: https://github.com/rdi-berkeley/agents-last-exam

Code: https://github.com/tailcallhq/forgecode

Project page: https://google-gemini.github.io/gemini-cli

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 95/100

The gist: Detailed Research Summary of "Agents’ Last Exam (ALE)" Benchmark Paper As a diligent researcher, I have thoroughly analyzed the provided text snippets from the arXiv paper concerning Agents’ Last

Key concepts

Generalist Computer-Use Agent (GCUA)
This agent paradigm requires a sophisticated combination of skills: visual perception, code execution, tool use, and long-horizon planning. It must operate all these capabilities within a single action loop to be considered capable of complex tasks.
O*NET / SOC 2018 Taxonomy
This is the U.S. federal occupational taxonomy used to define the industries and tasks covered by ALE. By grounding the benchmark in this system, ALE ensures that the evaluated workflows are drawn from authentic professional practices rather than synthetic scenarios.
Verifiability
ALE prioritizes deterministic checking over subjective LLM judging. Evaluation relies on unambiguous rubrics tied to observable artifacts, such as code output or geometric relationships. This ensures performance measurement is objective and directly measurable against real deliverables.

Terminology

Summary

Detailed Research Summary of Agents’ Last Exam (ALE) Benchmark Paper

As a diligent researcher, I have thoroughly analyzed the provided text snippets from the arXiv paper concerning Agents’ Last Exam (ALE). This benchmark is designed to rigorously evaluate Artificial Intelligence agents on complex, real-world tasks that possess long horizons and significant economic value. The core philosophy of ALE is not merely to create another leaderboard but to serve as a critical instrument for bridging the gap between current benchmark success metrics and actual impact on Gross Domestic Product (GDP).

Core Concept and Scope

ALE is fundamentally structured around evaluating the Generalist Computer-Use Agent (GCUA). This agent paradigm requires a sophisticated combination of capabilities, specifically integrating visual perception, code execution, tool utilization, and long-horizon planning all within a single action loop. The benchmark demands agents demonstrate full capability across five functional layers: Brain, Eyes (perception), Body (interaction), Hands (manipulation/tool use), and Feet (navigation/planning).

The scope of ALE is exceptionally broad and deeply grounded in professional reality. It covers 1K+ task instances spanning 55 subfields grouped into 13 industry clusters. Crucially, the domain coverage is anchored by the OOTNET / SOC 2018 (U.S. federal occupational taxonomy), ensuring that the tasks are not synthetic but are drawn from authentic professional workflows. The task instances themselves are sourced directly from real projects already completed by over 300 domain experts, which undergo multi-round quality control before inclusion in the benchmark.

Defining Benchmark Rigor: The Three Pillars

ALE establishes a high bar for agent performance based on three stringent, interconnected requirements:

  1. Representativeness: Tasks must accurately reflect real professional practices and the actual software environments agents operate within.

  2. Complexity: The evaluations must assess end-to-end deliverable work, demanding sustained, long-horizon planning rather than simple single-step responses.

  3. Verifiability: This is a cornerstone of ALE's design philosophy. Evaluation relies on deterministic checking or unambiguous rubrics tied to observable artifacts, explicitly avoiding open-ended LLM judging whenever a deterministic alternative exists (e.g., scoring based on bytes, fields, geometry, executable behavior).

Task Structure and Difficulty Tiers

ALE organizes its workload into three distinct difficulty tiers to assess agent capability across a spectrum of challenge:

  • Near-Term Tier (67 tasks): Represents more straightforward applications.

  • Full-Spectrum Tier (55 tasks): Assesses complex, multi-faceted challenges.

  • Last-Exam Tier (38 tasks): Represents the hardest tier, designed to test the limits of long-horizon, tool-intensive work professional practice.

The total benchmark comprises 960 expert-authored task workflows, resulting in 1,490 individual task instances.

Evaluation Methodology: Deterministic Scoring

A key methodological innovation of ALE is its move away from subjective human evaluation toward automated verification. The pipeline involves:

  1. Task Specification: Defining the input assets, necessary software, reference materials, and precise evaluation criteria.

  2. Agent Execution: The agent operates in an action loop within a standardized virtual environment (a remote VM with a fixed directory layout).

  3. Scoring: Evaluation is performed by scoring the agent’s output artifacts against expert-provided references or rubrics using structured, deliverable-based, or milestone-based checks.

The evaluation modes are highly granular and deterministic: Exact/hashed values, Structured tabular/numeric data, Geometric/spatial relationships, Visual appearance (using a vision-LLM judge), Behavioral/world state analysis, and Free-text semantic rubric checks.

Key Findings and Significance

The current results underscore the difficulty of achieving mastery at the highest levels. The paper reports that frontier agents—even those utilizing powerful configurations like Codex with GPT-5.5 (which already scores 82% on Terminal-Bench)—perform poorly on ALE, clearing only a small fraction of tasks: scoring below 50% on the easiest tier and under 10% on the hardest tier. Mainstream agents, such as Claude Code, record near-zero pass rates at the Last-Exam difficulty level.

ALE's ultimate purpose is explicitly stated: to serve as an instrument for closing the gap between benchmark success and GDP-relevant impact. The authors argue that saturation in ALE results would finally signal that AI agents have reached a capability level sufficient to sustain the kind of long-horizon, tool-intensive work professional practices actually demand.

Conclusion Synthesis

In summary, Agents’ Last Exam (ALE) is a meticulously constructed, expert-validated benchmark designed to test the practical utility and robustness of Generalist Computer-Use Agents. By grounding its tasks in real professional workflows (OOTNET/SOC 2018 taxonomy), enforcing strict representativeness, and utilizing deterministic verification methods instead of subjective LLM judging, ALE aims to provide a meaningful measure of agent capability that correlates with real-world economic productivity. It is positioned as the definitive tool for assessing whether current AI agents can transition from impressive demonstrations to sustaining economically valuable professional impact.

Improvements for AI systems

  1. Improve agent capability through Generalist CUA-agent architecture by implementing GUI-as-Tool mode to enable agents to read GUI screens and interact with desktop applications, addressing the gap between CLI/GUI-agents and Generalist CUA-agents. This allows agents like Claude Code or OpenClaw to perform complex tasks requiring both shell commands and visual perception, as noted in Figure 7.

  2. Enhance long-horizon planning reliability by integrating a context manager that implements a three-tier compaction strategy (microcompaction, LLM summarization, truncation) to handle context exceeding model limits. This ensures the agent maintains long-range planning state while preserving recent detail, as described in Section C.4.

  3. Increase task success rate by enforcing deterministic verification using Gate-and-score patterns where a hard precondition must pass before continuous scoring begins, preventing reward hacking and ensuring fidelity to professional workflows, as detailed in Section C.3.3 and exemplified by the manufacturing/gcode task workflow.

  4. Improve domain knowledge robustness by prioritizing Understanding failures over other error types in the failure taxonomy, focusing on closing Domain Knowledge Gap errors where agents fail due to missing specialized expertise, which accounts for 31% of classifiable failures.

  5. Optimize resource utilization and cost efficiency by leveraging findings from Figure 13, specifically by selecting agent harness–backbone configurations that achieve a high score with low API costs and wall-clock time, such as ALE-Claw with GPT-5.5 at 45.8% score for only a fraction of the cost compared to higher-spending configurations.

  6. Improve visual reasoning accuracy in creative tasks by utilizing LLM vision yes no judge probes, which target specific artifacts like UV seams or texture artifacts, rather than relying on general holistic judgments, as suggested in Section C.3.4 and Task Card 4 analysis for game/uv reproduction.

Abstract

Recent AI systems have achieved strong results on a wide range of benchmarks, yet these gains have not translated into economically meaningful deployment across many professional domains. We argue that this gap is largely an evaluation problem: widely used benchmarks lack sustained performance measurement on real and economically valuable workflows. This paper introduces Agents' Last Exam (ALE), a benchmark designed to evaluate AI agents on long horizon, economically valuable, real world tasks with verifiable outcomes. Developed in collaboration with 250+ industry experts, ALE covers non-physical industries defined with reference to O*NET / SOC 2018 (the U.S. federal occupational taxonomy). It is organized around a task taxonomy with 55 sub fields grouped into 13 industry clusters covering 1K+ tasks. Current results show that the hardest tier remains far from saturated: across mainstream harness and backbone configurations, the average full pass rate is below 1%. ALE is designed as a living benchmark: its task pool grows continuously as new workflows and industries are onboarded. More broadly, ALE is intended not merely as another leaderboard, but as an instrument for closing the gap between benchmark success and GDP relevant impact.

Sources

Related papers