Are LLM Safety Judges Policy-Invariant? A Three-Principle Stress-Test
cs.AI, cs.SE
Submitted: 2026-05-07
Updated: 2026-11-05
Comments: Accepted to NeurIPS 2026 (Spotlight)
License: http://creativecommons.org/licenses/by/4.0/
The gist: LLM-as-a-Judge pipelines have become the de facto evaluator for agent safety, yet existing benchmarks treat their verdicts as ground-truth proxies without checking whether the verdicts depend on the…
Terminology
Abstract
LLM-as-a-Judge pipelines have become the de facto evaluator for agent safety, yet existing benchmarks treat their verdicts as ground-truth proxies without checking whether the verdicts depend on the agent's behavior or merely on how the evaluation policy happens to be worded. We argue that any trustworthy safety judge must satisfy a basic property we call policy invariance, and we operationalize it as three testable principles: rubric-semantics invariance under certified-equivalent rewrites, rubric-threshold invariance under intentional strict-to-lenient shifts, and ambiguity-aware calibration so that verdict instability concentrates on genuinely ambiguous cases. Instantiating these principles as a stress-test protocol with four agent-class judges on trajectories drawn from ASSEBench and R-Judge, we surface a previously unmeasured failure mode: today's judges respond to meaningful normative shifts and to meaningless structural rewrites with comparable strength, and cannot tell the two apart. Content-preserving policy rewrites flip up to 9.1% of verdicts above baseline jitter, and 18-43% of all observed flips occur on unambiguous cases under such rewrites, so safety scores produced by these judges partly conflate what the agent did with how the evaluator was prompted. These findings persist with a frontier-scale judge, a third benchmark, and a second base policy. Beyond the diagnosis, we contribute the Policy Invariance Score and the Judge Card reporting protocol, which expose an order-of-magnitude spread in judge reliability that is invisible to accuracy-only leaderboards. We release the protocol and code so that future agent-safety benchmarks can audit their own evaluators rather than trust them by default: https://github.com/wengshihao/policy-invariance-judge
Sources
- AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
- ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
- Judge Reliability Harness: Stress Testing the Reliability of LLM Judges
- EvalCards: A Framework for Standardized Evaluation Reporting
- Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
- Validating LLM-as-a-Judge Systems under Rating Indeterminacy
- From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judges
- Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement
- ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents
- Evaluating Scoring Bias in LLM-as-a-Judge
- LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judge
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
- AgentBench: Evaluating LLMs as Agents
- AgentAuditor: Human-Level Safety and Security Evaluation for LLM Agents
- Identifying the Risks of LM Agents with an LM-Emulated Sandbox
- Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting
- OpenAI GPT-5 System Card
- Evil Geniuses: Delving into the Safety of LLM-based Agents
- Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models
- Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents