Consistency as a Testable Property: Statistical Methods to Evaluate AI Agent Reliability
cs.AI
Submitted: 2026-05-11
Updated: 2026-11-07
Code: https://github.com/harbor-framework/harbor
Terminology
Sources
- Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows
- T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step
- Reasoning Robustness of LLMs to Adversarial Typographical Errors
- MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
- Certifying LLM Safety against Adversarial Prompting
- Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
- Consistency in Language Models: Current Landscape, Challenges, and Future Directions
- Gorilla: Large Language Model Connected with Massive APIs
- ChatDev: Communicative Agents for Software Development
- Towards a Science of AI Agent Reliability
- Measuring Reliability of Large Language Models through Semantic Consistency
- Quantifying perturbation impacts for large language models
- Identifying the Risks of LM Agents with an LM-Emulated Sandbox
- E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testing
- Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
- PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change
- $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
- PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents