When Evaluators of AI Agents Disagree: An Empirical Study of Native Scoring, LLM Judges, and Commit-Boundary Checks on tau2-bench
cs.SE, cs.AI, cs.CR
Submitted: 2026-11-10
Updated: 2026-11-10
Code: https://github.com/sierra-research/tau2-bench
Terminology
Sources
- From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
- A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluation
- ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning
- AgentBench: Evaluating LLMs as Agents
- AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories
- Self-Refine: Iterative Refinement with Self-Feedback
- trajectory-judge: What Outcome-Only LLM Judges Miss on Agent Trajectories
- Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias
- LLM Evaluators Recognize and Favor Their Own Generations
- Counsel: A Meta-Evaluation Dataset for Agentic Tasks
- Identifying the Risks of LM Agents with an LM-Emulated Sandbox
- Toolformer: Language Models Can Teach Themselves to Use Tools
- Progent: Securing AI Agents with Privilege Control
- Reflexion: Language Agents with Verbal Reinforcement Learning
- AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling
- Voyager: An Open-Ended Embodied Agent with Large Language Models
- AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents
- Self-Consistency Improves Chain of Thought Reasoning in Language Models
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters