Does the Judge See What the Evaluation Requires? Input Assurance for LLM Agent Evaluation
cs.SE
Submitted: 2026-11-02
Updated: 2026-11-02
Terminology
Sources
- AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories
- Lessons from the Trenches on Reproducible Evaluation of Language Models
- Establishing Best Practices for Building Rigorous Agentic Benchmarks
- Judge Reliability Harness: Stress Testing the Reliability of LLM Judges
- One Token to Fool LLM-as-a-Judge
- AgentChaos: Chaos Engineering for Agent Systems via Programmatic Fault Injection
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters