Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows

arXiv:2605.27922 · cs.AI · Submitted 2026-05-27 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows".

Jane: The gist The benchmark introduces Harness-Bench, a diagnostic protocol for evaluating configuration-level harness effects in realistic agent workflows.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: The paper we just discussed is "Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows." It really focuses on the harness as a variable, not just the model.

Jane: That’s right. The authors are showing that you need to look at the entire configuration of the agent system—the model plus its tools and state management—to understand performance accurately > Page one --- <ref:2605.27922#pg1>

Lu: They use this benchmark to show that when you have different models, they can perform very differently depending on how much control or structure is imposed by the harness > Page two --- <ref:2605.27922#pg1>

Meng: It means if we want reliable, efficient agent systems, we have to stop focusing only on the model weights and start looking at the entire execution stack setup > Page two --- <ref:2605.27922#pg1>

Lalam: The overall implication is that reporting agent capability should be tied to this model-harness configuration rather than just the base model alone > Page two --- <ref:2605.27922#pg1>

Tom: So, if you’re building an agent system, you need to evaluate it against a harness and see how that pairing performs across different models > Page one --- <ref:2605.27922#pg1>

Conclusion: Tom: So we’ve been looking at how different models can perform differently depending on the setup, but this paper is about that setup itself, focusing on something called a harness > Jane, what is the main thing they are trying to measure with this Harness-Bench thing?

Jane: Well, they're testing how much a specific tool or state management system—the harness—actually changes what an AI can do across different models > Lu, you know about how crucial that layer is for making sure the model’s brain actually connects to the real world actions > Meng, from your production side, when you deploy an agent system, does that harness configuration ever cause a weird failure mode that just doesn't show up in standard testing?

Meng: Exactly. Standard testing doesn't catch execution drift where the model thinks it’s doing something right because of its reasoning, but the harness messes up the workspace state or ignores a tool output > Tom, what kind of "execution alignment failure" are they seeing most often in these runs?

Tom: They find that plausible reasoning gets completely disconnected from tool feedback or what’s recorded in the workspace state > Lalam, you see if that decoupling happens when you try to run these one hundred six tasks > Lalam, what does that decoupling look like concretely?

Lalam: It looks like the model reasons about a plan, but when it tries to execute the tool call, it uses an outdated piece of information from the workspace or ignores a constraint set by the harness > Jane, so if we put this into practice for someone just listening to the show, what does that mean for how we trust these AI agents?

Jane: It means you can’t just look at one model and say it’s good; you have to look at the whole package—the model plus the harness—because that configuration determines if an agent is reliable enough to be used in a real workflow > Lu, what is the authors' big conclusion about how we should report on AI capabilities now?

Lu: They are saying capability isn't something you can just assign to the base model weights anymore; it has to be reported at the model-harness configuration level because that’s where the actual execution happens > Tom, so what does this imply for building better tools or interfaces for these agents?

Peking University

cs.AI

Submitted: 2026-05-27

Updated: 2026-10-15

Comments: 36 pages, 10 figures, 28 tables

Code: https://github.com/Qihoo360/harness-bench

Project page: http://www.harness-bench.ai

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 85/100

The gist: The gist The benchmark introduces Harness-Bench, a diagnostic protocol for evaluating configuration-level harness effects in realistic agent workflows.

Key concepts

Harness: The harness is the system layer that manages a complex AI agent's execution. It controls how context, tools, state (like workspace data), constraints, permissions, tracing, and recovery mechanisms work together to mediate between the base language model's raw output and its actual external actions.
Harness-Bench: This is a diagnostic benchmark used to study the effects of different harness configurations. It fixes the task environment while systematically varying which harness surrounds the model. It records detailed execution traces and artifacts to measure performance across these various setups.
Execution Alignment Failure: This occurs when an agent's logical reasoning becomes disconnected from its operational reality. Specifically, this means the model's internal thoughts no longer match what is happening in the external environment, such as tool feedback, changes in the workspace state, or verifiable output requirements.
Model-Harness Configuration Level: This refers to evaluating agent capability based on the specific combination of a base model and its surrounding harness. The research suggests that performance metrics should be reported at this combined level rather than attributing success solely to the underlying language model.

Terminology

Benchmark — 3× in this paper · explained in 2 episodes
Alignment — 1× in this paper
Trace — 1× in this paper · explained in 2 episodes

Summary

The gist The benchmark introduces Harness-Bench, a diagnostic protocol for evaluating configuration-level harness effects in realistic agent workflows.

Harness Definition and Importance

LLM agents are increasingly deployed as executable systems that use tools, modify workspaces, and produce concrete artifacts <ref:2605.27922#pg2> The paper refers to this system layer as a harness: the mechanism that organizes context, tools, state, constraints, permissions, tracing, and recovery to mediate between model outputs and external actions <ref:2605.27922#pg2>. Harnesses are therefore central to agent system design: they shape how model capability is exposed, constrained, and realized <ref:2605.27922#pg2>. They affect completion, cost, safety, robustness, and auditability <ref:2605.27922#pg2>.

Harness-Bench Design

Harness-Bench is a diagnostic benchmark for studying model–harness configurations in executable agent workflows <ref:2605.27922#pg2>. The benchmark fixes external task conditions while varying the harness surrounding the model and records both final artifacts and execution traces <ref:2605.27922#pg2>. It evaluates representative harness configurations across multiple model backends under shared task environments, budgets, and evaluation protocols while preserving each harness’s native execution behavior <ref:2605.27922#pg2>. The benchmark contains 106 sandboxed offline tasks constructed from practical agent-use patterns and manually reviewed for realism, solvability, oracle-checkability, and integrity <ref:2605.27922#pg2>.

Evaluation Protocol and Metrics

The evaluation protocol fixes the user-facing task, initial sandbox state, budget, timeout, and evaluator while varying the harness configuration <ref:2605.27922#pg2>. Each run records final artifacts, execution traces, usage statistics, and validator outputs <ref:2605.27922#pg2>. The overall score is calculated as TaskScorei = Securityi · Completioni · Processi, where Securityi ∈ 0 or 1 and Processi = Robustnessi + ToolUsei + Consistencyi <ref:2605.27922#pg2>. Completion measures task-specific output quality, while the process score is computed from the reconstructed trace covering robustness, tool-use appropriateness, and consistency <ref:2605.27922#pg2>.

Key Findings on Configuration Effects

Across 5,194 execution trajectories, substantial variation in completion, process quality, efficiency, and failure behavior was observed across model–harness pairings <ref:2605.27922#pg2>. Results suggest that agent capability should be reported at the model–harness configuration level rather than attributed to the base model alone <ref:2605.27922#pg2>. Performance varies across model–harness pairings, with NanoBot obtaining the highest aggregate score (76.2) and OpenClaw obtaining the lowest score (52.4) under the same task set and model-backend pool <ref:2605.27922#pg2>.

Diagnostic Analysis of Failures

Analysis identifies recurring execution-alignment failures where plausible reasoning becomes decoupled from tool feedback, workspace state, evidence, or verifiable output contracts <ref:2605.27922#pg2>. These failures are symptoms of execution drift where model reasoning loses alignment with the conditions of success <ref:2605.27922#pg2>. Harness-Bench evaluates whether the harness preserves correspondence between what the agent reasons about, what the workspace records, and what the evaluator ultimately checks <ref:2605.27922#pg2>.

Harness Dependence and Limitations

Stronger model backends tend to achieve higher mean scores while exhibiting lower cross-harness variance <ref:2605.27922#pg2>. This suggests that stronger models may be more tolerant of differences in prompting, tool interfaces, state management, and recovery behavior <ref:2605.27922#pg2>. Limitations include the focus on controlled, sandboxed offline workflows at the cost of coverage of live services and long-term production memory <ref:2605.27922#pg2>. The results are descriptive measurements over a fixed task suite rather than statistical claims about all possible agent workflows <ref:2605.27922#pg2>.

The paper presents Harness-Bench, a diagnostic benchmark for studying configuration-level harness effects in realistic executable agent workflows.

Improvements for AI systems

System performance should be reported at the model–harness configuration level rather than attributed to the base model alone, allowing users to diagnose specific execution-layer deficiencies. This enables developers to identify exactly which aspects of the system—such as tool use appropriateness, consistency, and robustness—are failing for a given model and harness pairing.
The AI system can be improved by developing a diagnostic protocol that uses execution alignment to detect execution drift, where plausible reasoning becomes decoupled from tool feedback, workspace state, evidence, or verifiable output contracts. This allows the system to pinpoint precisely where the connection between internal thought and external action breaks down during complex workflows.
Harnesses can be engineered to preserve correspondence between reasoning and execution by ensuring that what counts as a pending obligation is clearly defined within the harness's structure, thereby making failures at the boundary between intention and completion detectable before they result in final oracle failures.

Abstract

LLM agents are increasingly deployed as executable systems that use tools, modify workspaces, and produce concrete artifacts. In such workflows, performance depends not only on the base model, but also on the harness: the system layer that manages context, tools, state, constraints, permissions, tracing, and recovery. However, existing benchmarks typically abstract away execution, compare complete agent systems, or hold the harness fixed, making execution-layer variation difficult to study. We introduce Harness-Bench, a diagnostic benchmark for evaluating configuration-level harness effects in realistic agent workflows. Harness-Bench evaluates representative harness configurations across multiple model backends under shared task environments, budgets, and evaluation protocols, while preserving each harness's native execution behavior. The benchmark contains 106 sandboxed offline tasks constructed from practical agent-use patterns and manually reviewed for realism, solvability, oracle-checkability, and integrity. Each run records final artifacts, execution traces, usage statistics, and validator outputs, enabling analysis beyond final completion. Across 5,194 execution trajectories, we observe substantial variation in completion, process quality, efficiency, and failure behavior across model-harness pairings. These results suggest that agent capability should be reported at the model-harness configuration level rather than attributed to the base model alone. Our analysis further identifies recurring execution-alignment failures, where plausible reasoning becomes decoupled from tool feedback, workspace state, evidence, or verifiable output contracts. Harness-Bench provides a reproducible foundation for diagnosing and improving reliable, efficient, and auditable agent execution stacks.