Safety, or Just Capability? A Validity Audit of Agent-Safety Benchmarks
cs.AI, cs.IR
Submitted: 2026-07-30
Updated: 2026-11-04
License: http://creativecommons.org/licenses/by/4.0/
The gist: Agent-safety benchmarks measure different behaviors, and their scores get quoted interchangeably as an agent's safety.
Terminology
Abstract
Agent-safety benchmarks measure different behaviors, and their scores get quoted interchangeably as an agent's safety. We treat four of them (R-Judge, InjecAgent, AgentHarm, AgentDojo) as measurements to be validated, running each under its official implementation and author-provided scorer on up to 22 models, with MMLU and GPQA measured by us under one protocol as a capability composite. The metric is the first problem. On any binary trace-judgment benchmark scored by F 1, an ``always positive'' policy attains F 1 = 2π/(1+π); on R-Judge that is 0.690, above five of the 21 models that actually discriminate. The three broad-coverage benchmarks then rank the same 18 models differently, and the trade-off behind that disagreement is a small-panel artifact: R-Judge specificity against AgentHarm safety correlates-0.64 at n = 7 and +0.02 at n = 18, and a quarter of random size-7 subsets reach ρ at least 0.5 around that near-zero value. Held-out validity turns on which outcome you pick. Capability predicts task success (ρ = + 0.60) but correlates negatively with misalignment safety (ρ = - 0.44, n = 21). On their paired n = 20 panel, the corresponding contrast is Δ = - 1.00 (95% CI [-1.48, -0.49], p<0.001), and it survives leave-one-organization-out and organization-clustered bootstrap analyses. On an expanded 41-model panel, the misalignment correlation weakens to-0.16 (95% CI [-0.54, +0.22]) and jailbreak strengthens to +0.34, though neither change is significant. AgentHarm shows the strongest held-out association, ρ = + 0.72 with three-template jailbreak safety after controlling capability. But both instruments score harmful compliance, so this is evidence of convergent validity rather than general safety. Naming the benchmark, metric, target behavior, and model panel is the minimum a safety claim needs.
Sources
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
- Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluation
- Evidence-Bound Autonomous Research (EviBound): A Governance Framework for Eliminating False Claims
- OR-Bench: An Over-Refusal Benchmark for Large Language Models
- Quantifying construct validity in large language model evaluations
- Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agents
- Agentic Misalignment: How LLMs Could Be Insider Threats
- Do These LLM Benchmarks Agree? Fixing Benchmark Evaluation with BenchBench
- A StrongREJECT for Empty Jailbreaks
- Jailbroken: How Does LLM Safety Training Fail?
- $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents