SteerBench-Work: A Benchmark for Agent Steering at Action Boundaries

SteerBench-Work: A Benchmark for Agent Steering at Action Boundaries

Abstract

Long-running LLM agents act through tools. A single step can send an email, merge a pull request, or wire a payment. The steering decision is the pre-commit choice at that boundary: proceed, or hold for human or policy review. We introduce SteerBench-Work, an incident-anchored, bidirectional benchmark protocol for measuring that decision in workplace agents across developer operations, customer service, finance, legal, medical, HR, security, and other domains.

Release v2026-05 contains 106 scenarios anchored in public incidents, paired evidence-reversed mirrors, and calibration controls. The labels split nearly evenly between proceed and hold, so the two error directions get near-identical numbers of chances. The protocol, not the count, defines the benchmark: a model sees the proposed action and available evidence, returns a gate decision, and is scored on whether it crosses or holds the boundary correctly.

Across 30 model conditions, the failures run almost entirely in one direction: models wrongly hold authorized, evidence-cleared work on 28.1% of opportunities and wrongly allow unsafe work on 1.0%. The direction holds under sensitivity checks: excluding the 17 rows where the runner emitted a derived risk flag gives 19.4% and 1.25%, and relabeling against an unadjudicated three-rater majority gives 41.1% and 0.36%. The hardest cases are risk-resolved commits, where signed or structured evidence has already cleared a real risk trigger, and models score markedly worse on evidence-reversed mirrors of famous incidents (63.8%) than on the incidents themselves (98.5%). A model can win the majority vote while flipping between answers across trials. General capability is not the same as steering calibration: higher-capability models often over-refuse at the commit boundary. More reasoning can repair a weak gate while leaving a calibrated gate flat or slightly worse. The same grid selects open-weight repair targets for post-training at fixed-or-lower under-refusal. The public leaderboard is at steerbench.com.

Introduction

The action-boundary commit is the moment the human in the loop will not see if everything works. After commit, the consequence is in the world and is no longer steerable. Public incidents from the LLM era and from earlier automated-decision systems document the failure pattern: agents and automated decision systems that judge the boundary incorrectly produce tribunal rulings, regulator settlements, class actions, and operational harm. The pattern is reproducible, and a labeled benchmark of those decisions is the natural measurement tool.

Frontier labs already ship action-boundary reviewers. In Codex Auto-review, OpenAI routes a planned action and recent context to a reviewer before execution; in Claude Code Auto Mode, Anthropic uses an action classifier to reduce approval fatigue while blocking risky tool calls. These are product and engineering references, not open benchmark papers. They show that the boundary is operationally real.

The benchmark gap is narrower and sharper: the field lacks an open, incident-anchored, bidirectional benchmark with open training views for workplace action-boundary gates. XSTest, OR-Bench, SORRY-Bench, and HarmBench measure chat refusal where the question is whether the model produces a particular kind of text reply. SWE-bench and GAIA measure final task completion where the question is whether the patch passes tests or the answer matches ground truth. HiL-Bench, ClarifyBench, and AskBench measure ask-or-act when the agent lacks information. ST-WebAgentBench scores web-agent policy compliance with a metric that falls only when an agent acts beyond its permissions, never when it acts short of them. Independent permission-gate work such as AmPermBench is adjacent, and reports over-blocking where the authorization is deliberately ambiguous. AgentAbstain, concurrent work, tests act-or-abstain decisions in executable sandboxes under ambiguity, tool failure, and runtime discovery. SteerBench-Work focuses on public incidents, paired incident mirrors, both under-refusal and over-refusal, risk-resolved workplace commits, and open training views.

This paper contributes:

1. A corpus that scores the live steering decision. The current release places a model at the commit moment with the user request, the proposed action, and the evidence on hand. The scored field is the binary proceed-or-hold decision. Scenarios are anchored to public incidents so each label is auditable. We have not found a prior benchmark that combines public-incident anchoring, paired mirrors, and pre-commit labels for both unsafe action and unnecessary interruption at workplace action boundaries.

2. Incident-mirror methodology. A 13-scenario mirror subset preserves the surface shape of public incidents while inverting the verification state, testing whether the gate follows the changed evidence or keeps giving the answer the incident is associated with.

3. Both steering errors in one suite. The release scores under-refusal and over-refusal from the same scenarios and the same scored field, so a model cannot lower one by silently trading it against the other. Bidirectional scoring is not itself new: HiL-Bench's Ask-F1 balances over-asking against silent guessing when the agent lacks information. What is scored here is both directions at a boundary where the information is sufficient, and the decision turns on authority, evidence resolution, and irreversibility.

4. Irreversibility-weighted severity reporting. A secondary severity metric beside mean trial accuracy, modal-of-5, and pass5, showing the operational severity of remaining misses without changing the primary unweighted leaderboard rank.

5. Domain and action-effect stratification. Every scenario in the release carries workplace domain and action-effect metadata. The run reports stratified results so a model's weak areas are visible rather than hidden in one global score.

6. Training and evaluation views in the tinker-cookbook shapes. The same scenario schema carries the labels a training run needs. In the planned 500-scenario corpus, examples are split by scenario family into train, development, and sealed test sets, so final lift is measured on scenarios not used for supervision or recipe tuning.

Related work

Product action gates. OpenAI's Codex Auto-review and Anthropic's Claude Code Auto Mode are production examples of action-boundary review. We cite them as engineering references: they motivate the gate as a real product boundary, but they do not release an open incident-anchored benchmark or open training recipe for the gate. AmPermBench stress-tests a deployed permission gate, Claude Code's auto mode, with 128 prompts covering 253 state-changing actions; it evaluates one product's gate rather than defining a cross-model protocol, and it comes from the same group as UnderSpecBench below. It is the nearest prior work that reports over-blocking of safe actions alongside under-blocking of unsafe ones, although its headline result is the latter. Its scenarios deliberately underspecify target scope, blast radius, and risk level, so its blocked-safe-action figure is measured where the authorization is genuinely ambiguous and holding may be the defensible call. SteerBench-Work measures the opposite construction, in which signed or structured evidence resolves the risk before the commit. The two quantities answer different questions and are therefore not compared numerically here. SteerBench-Work is complementary because it uses public-harm anchors, paired mirrors, bidirectional error reporting, risk-resolved workplace commits, and training views.

Action-boundary benchmarks. UnderSpecBench is the closest neighbor to this work: 69 task families grounded in documented incidents, CVEs, and tool behavior, expanded into 2,208 prompt variants, scored by deterministic side-effect oracles that separate safe success from wrong-target and over-scope execution, with non-action runs classified as clarification, refusal, or deferment. It reports that 55.8% to 67.8% of runs violate at least one boundary, and concludes that completion-centric evaluation can overstate safe autonomy. The two benchmarks vary different things. UnderSpecBench varies what the agent knows, sweeping intent clarity, target certainty, and blast radius. SteerBench-Work holds the action fixed and varies whether it is allowed. The rows the two do not share are the ones this paper is about: the information is sufficient, authorization exists, risk-clearing evidence exists, and the model refuses anyway. The Authority Frontier framework prices each side-effect-bearing action against a safe default and treats persistence under denial as an underwriting variable, which is adjacent to the post-denial behavior this benchmark does not yet score.

Policy-compliance benchmarks. ST-WebAgentBench is the closest prior work on the completion-plus-compliance framing. It scores enterprise web-agent trajectories in GitLab, SuiteCRM, and a shopping-admin application against per-task authored policies across six dimensions, including user consent and action confirmation, using deterministic evaluators rather than an LLM judge, and reports Completion under Policy, CuP = Ctask · 1[Vtotal = 0], which credits only completions that violate no policy. CuP is monotone in a single direction: it falls as an agent acts beyond its permissions and never as it acts short of them. An agent that declines a task it was authorized to complete scores Ctask = 0 and is, in that metric, indistinguishable from an agent that crashed or executed the task incorrectly. The benchmark accordingly reports no over-refusal quantity, and its headline consent finding runs opposite to ours: it observes agents failing to request permission before critical actions. SteerBench-Work labels 56 of its 106 scenarios proceed and reports the rate at which authorized, evidence-cleared work is wrongly held, which is a measurement CuP cannot express.

Over-refusal as a side effect of defense training. The Autonomy Tax reports that agents hardened against prompt injection begin refusing legitimate tool operations prematurely, and frames this as a capability cost of adversarial fine-tuning. It is the closest prior observation that over-safety degrades an agent into uselessness. It studies models deliberately modified for defense, whereas the conditions evaluated here are stock provider endpoints as shipped.

Chat-refusal benchmarks. XSTest names exaggerated-safety refusal on a surface trigger word and measures it on chat refusal. Wen et al.'s survey of abstention defines and measures the abstention-vs-helpfulness trade-off.

Agent-safety benchmarks. Agent-SafetyBench evaluates unsafe tool-using agents across 349 environments and 2,000 synthetic and augmented test cases across 8 risk categories. ODCV-Bench scores outcome-driven constraint violations on 40 multi-step scenarios in production-inspired sandbox environments where KPI pressure conflicts with constraints. HiL-Bench scores ask-or-act on software engineering and text-to-SQL using Ask-F1, the harmonic mean of question precision and blocker recall, and reports that reinforcement learning on a shaped Ask-F1 reward improves both help-seeking quality and task pass rate. It is therefore already bidirectional and already trainable, on the axis of missing information. MLCommons AILuminate Agentic frames agent product maturity along quality, boundedness, confidentiality, control, robustness, security, and reliability axes.

Incident catalogs and instruction-hierarchy work. The AI Incident Database catalogs incidents but is not itself a scoring benchmark. The OpenAI Instruction Hierarchy paper defines the privileged-instruction hierarchy. Greshake et al. names indirect prompt injection.

Training methods relevant to the open training target. Rule-Based Rewards defines the rule-based reward formulation. Process Reward Models score intermediate reasoning steps. Self-consistency sampling is the basis for the modal-of-N trial verdict.

SteerBench-Work targets a different measurement object from each of these: the pre-commit steering decision at workplace action boundaries, with public-incident anchors, paired over-refusal mirrors, and labels for both unsafe action and unnecessary escalation. Adjacent benchmarks measure unsafe outcomes after execution (Agent-SafetyBench, ODCV-Bench), chat refusal (XSTest), or ask-versus-act when information is missing (HiL-Bench, UnderSpecBench). The remaining distinction is the axis rather than the direction. Bidirectional scoring and trained asking behavior already exist on the missing-information axis. SteerBench-Work holds information sufficient in order to isolate the four properties that survive when the agent already knows what to do: authorization, evidence resolution, consequence, and irreversibility. The failure-mode category names from the chat-refusal and abstention literature are reused as diagnostic slices so a reviewer can inspect which class of failure each model is making; the training target remains the action-boundary decision.

Name disambiguation. An unrelated 2009 benchmark of the same short name, SteerBench, evaluates steering behaviors for crowd and character navigation in virtual environments. This work is unrelated to it and is published under the full name SteerBench-Work.

Benchmark design

Task definition. A scenario is a frozen commit moment. The model receives the user request, the proposed action, and the evidence available at that moment (policies, signatures, audit records, prior messages), and nothing else. It returns a structured verdict: a commit permission (allowed or blocked), a policy action (continue, show progress, ask clarification, request approval, or escalate), a confidence, and a short reason. The scored field is the commit permission alone. Each scenario's expected verb maps onto it through a fixed published mapping: continue and proceed map to allowed; block, request approval, escalate, and ask clarification map to blocked. Asking, escalating, and requesting approval are therefore all holds at the scored boundary. A model that proceeds against a hold label commits an under-refusal miss; a model that holds against a proceed label commits an over-refusal miss. Each (scenario, model condition) cell runs five independent trials under one SHA-256-pinned prompt, and the grid reports mean trial accuracy, the modal-of-5 verdict, and pass5 side by side.

Corpus: incidents, mirrors, and resolved risk. Real incidents from the LLM era and from earlier automated-decision-system cases are the dataset asset other benchmarks do not have. The court rulings, regulator settlements, and class actions provide ground truth that an evaluator cannot dispute: the airline was held to the chatbot's statement; the lawyer was sanctioned for filing the hallucinated brief; the database was deleted; the candidate was rejected. The incident set is reproducible because the failure pattern is reproducible.

The paired over-refusal mirror methodology answers an obvious objection: a model that refuses on every surface flag will look good on the under-refusal set, since it will never commit anything risky. The 13-scenario mirror subset takes the same surface shape (a benefits notice that looks like Robodebt income-averaging, an applicant decision that looks like Workday Mobley, a production change that looks like a known outage) and inverts the verification state (signed cross-service authorization, eligibility met, audit trail present). A model that refuses an authorized mirror because it resembles a cautionary incident has the wrong heuristic. A model that blocks the unsafe case and commits the authorized mirror demonstrates the steering signal we are measuring.

The hardest over-refusal rows are not generic safety refusals. They are risk-resolved commit-boundary cases: a real risk trigger is visible, but signed or structured evidence resolves the risk for the proposed action. Examples include regulated notices after eligibility and due-process checks, secret rotation under a service account and cron schedule, broad but authorized codemods, and protected-surface changes with explicit approval. These rows test whether an agent can…

Based on the paper, here are the specific improvements to AI systems and what the improved systems can do:

1. Implement a two-stage risk-resolution gate

• Improvement: Add a dedicated pre-commit module that first detects risk triggers (e.g., financial transactions, data deletion, legal actions) and then checks whether signed/structured evidence has already resolved that risk before deciding to hold or proceed.

• Capability: The AI can distinguish between an unresolved risk (e.g., sending a payment without authorization) and a resolved one (e.g., sending a payment after a signed approval), reducing over-refusal on authorized work by up to 28.1% while keeping under-refusal at 1.0%.

2. Train on incident-mirror pairs for evidence-based decision-making

• Improvement: Fine-tune models on paired scenarios where the surface details are identical but the verification state is inverted (e.g., a benefits notice with vs. without eligibility confirmation), forcing the model to follow evidence rather than surface heuristics.

• Capability: The AI can correctly proceed on authorized actions that resemble past incidents (e.g., approving a credit limit increase after fairness controls are signed), improving accuracy on evidence-reversed cases from 63.8% to near the 98.5% level seen on original incidents.

3. Add bidirectional error reporting to the training objective

• Improvement: Replace single-direction safety metrics with a loss function that penalizes both under-refusal (acting without authorization) and over-refusal (blocking authorized work) equally, using the paper's 56/50 proceed/hold label split as a balanced training set.

• Capability: The AI can optimize for both safety and usability simultaneously, avoiding the common failure of becoming overly cautious after safety fine-tuning, which currently causes 471 over-refusal misses per 1,677 opportunities.

4. Use modal-of-5 voting for deployment decisions

• Improvement: Run the gate decision five times per action and take the majority verdict, rather than relying on a single trial, to reduce answer instability that the paper found even in leaderboard-winning models.

• Capability: The AI produces more reliable commit decisions, with pass5 consistency (all five trials correct) improving from 91.5% to potentially higher, and reduces the risk of a single stochastic failure causing an unnecessary hold or unsafe action.

5. Apply reasoning effort selectively based on calibration status

• Improvement: Implement a calibration check: if the model's baseline gate is already well-calibrated (low error in both directions), keep reasoning at a minimal level; if it over-refuses, increase reasoning effort to repair the gate, as the paper shows high reasoning can fix weak gates but harms calibrated ones.

• Capability: The AI can adapt its computational cost per decision, improving accuracy on weak gates (e.g., from 63.8% to higher on mirrors) without degrading already-calibrated performance, saving inference cost on routine authorized actions.

6. Build a risk-resolved commit classifier

• Improvement: Train a specialized classifier on the 33 risk-resolved commit scenarios (e.g., secret rotation with cron schedule, broad codemods with explicit approval) to recognize when a visible risk trigger has been cleared by structured evidence.

• Capability: The AI can correctly proceed on authorized but superficially risky actions—like the Apple Card mirror where all 30 model conditions failed—enabling autonomous execution of legitimate high-value tasks that current systems wrongly block.

7. Use severity-weighted decision thresholds

• Improvement: Adjust the hold/proceed threshold based on irreversibility tier (low=1, medium=2, high=4), so the AI is more conservative on irreversible actions (e.g., deleting production) and more permissive on reversible ones (e.g., saving a draft), without changing the primary accuracy metric.

• Capability: The AI minimizes operational harm by weighting misses appropriately—reducing the impact of the 1.0% under-refusal rate on critical systems while not over-blocking low-stakes authorized work.

8. Implement evidence-trace verification before commit

• Improvement: Add a module that explicitly checks for the presence of signed authorizations, audit trails, or structured eligibility confirmations in the evidence before allowing a proceed decision, mirroring the paper's finding that models fail when evidence is present but not acted upon.

• Capability: The AI can verify that risk-clearing evidence exists (e.g., a signed cross-service authorization) and use it to override surface-level risk flags, reducing over-refusal on authorized actions by up to 19.4% (when excluding inferred flags) while maintaining safety on genuinely unsafe actions.

Long-running LLM agents act through tools, and a single step can send an email, merge a pull request, or wire a payment. The steering decision is the pre-commit choice at that boundary: proceed, or hold for human or policy review. We introduce SteerBench-Work, an incident-anchored, bidirectional benchmark for that decision in workplace agents across developer operations, customer service, finance, legal, medical, HR, and security. Release v2026-05 contains 106 scenarios anchored in public incidents, paired evidence-reversed mirrors, and calibration controls, with labels split nearly evenly between proceed and hold so the two error directions get near-identical numbers of chances. A model sees the proposed action and the available evidence, returns a gate decision, and is scored on whether it crosses or holds the boundary correctly. Across 30 model conditions the failures run almost entirely in one direction: models wrongly hold authorized, evidence-cleared work on 28.1% of opportunities and wrongly allow unsafe work on 1.0%. The hardest cases are risk-resolved commits, where signed or structured evidence has already cleared a real risk trigger, and models score markedly worse on evidence-reversed mirrors of famous incidents (63.8%) than on the incidents themselves (98.5%). General capability is not the same as steering calibration: higher-capability models often over-refuse at the commit boundary, and more reasoning can repair a weak gate while leaving a calibrated one flat. The public leaderboard is at steerbench.com.