CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies
cs.AI
Submitted: 2026-06-15
Updated: 2026-11-13
Comments: Accepted to NeurIPS 2026 (Evaluations and Datasets Track)
Code: https://github.com/SakanaAI/CoffeeBench
License: http://creativecommons.org/licenses/by/4.0/
The gist: LLM agents are increasingly capable of handling complex, long-horizon tasks and may eventually take on business management, a role traditionally performed by humans.
Terminology
Abstract
LLM agents are increasingly capable of handling complex, long-horizon tasks and may eventually take on business management, a role traditionally performed by humans. Deploying them safely requires monitoring their capabilities and behavior in environments that reflect real economies. We introduce CoffeeBench, a benchmark built on a compact simulation of a coffee supply chain with realistic business mechanics. Six autonomous LLM coding agents run its farmers, roasters, and retailers, and negotiate and trade with one another to maximize their own net income over 90 simulated days. We evaluate eight recent proprietary and open-weight models by letting each run one roaster while a fixed background model runs the other five firms, and scoring that roaster's net income. Claude Sonnet 5.5 earns the highest net income, closely followed by GPT-6 Astra. The leaders sell at higher prices, and the best model also buys its inputs far more cheaply, whereas weaker models trade at thin margins. Since performance targets are a common cause of accounting fraud in human firms, we also test whether agents under KPI pressure inflate their numbers. Under a revenue target, circular trades occur for some pairs of models: one agent proposes that the two rival firms sell goods back and forth at inflated prices so that both book revenue, and the rival accepts. We hope CoffeeBench serves as an early step toward a future in which LLM agents manage firms without misconduct. We release our code and agent trajectories.
Sources
- HealthBench: Evaluating Large Language Models Towards Improved Human Health
- Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents
- GLM-5: from Vibe Coding to Agentic Engineering
- FutureSim: Replaying World Events to Evaluate Adaptive Agents
- Kimi K2: Open Agentic Intelligence
- A Benchmark for Evaluating Outcome-Driven Constraint Violations in Autonomous AI Agents
- BAGEN: Are LLM Agents Budget-Aware?
- Agentic Misalignment: How LLMs Could Be Insider Threats
- WebGPT: Browser-assisted question-answering with human feedback
- PostTrainBench: Can LLM Agents Automate LLM Post-Training?
- TradingAgents: Multi-Agents LLM Financial Trading Framework
- CodeClash: Benchmarking Goal-Oriented Software Engineering
- ProgramBench: Can Language Models Rebuild Programs From Scratch?
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents