CAREFUL-Bench: Effect-Aware, Judge-Free Evaluation of Multi-Turn LLM Function Calling
cs.AI, cs.SE
Submitted: 2026-11-02
Updated: 2026-11-03
Terminology
Sources
- AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
- Contract2Tool: Learning Preconditions and Effects for Reliable Tool-Augmented LLM Agents
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
- NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls
- Do Agent Benchmarks Do What They Say? An Executable-Contract Audit of Tool-Using Agent Environments
- Towards Verifiably Safe Tool Use for LLM Agents
- NEXUS: Structured Runtime Safety for Tool-Using LLM Agents
- POLAR: Ontology-Guided Risk Prevention for Tool-Calling LLM Agents
- Small agents should know what their tools do
- ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents
- Where Does Exactly-Once Live? Model, Harness, and Tool-Contract Effects on Duplicate Side Effects in LLM Agents
- ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces
- ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities
- AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories
- Atomix: Timely, Transactional Tool Use for Reliable Agentic Workflows
- Tracking Capabilities for Safer Agents
- Identifying the Risks of LM Agents with an LM-Emulated Sandbox
- WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Setting
- AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents