TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks
cs.AI
Submitted: 2026-05-21
Updated: 2026-11-04
Comments: Accepted at NeurIPS 2026 Evaluations and Datasets Track. Project website: https://terminalworld.ai
Code: https://github.com/EuniAI/TerminalWorld
License: http://creativecommons.org/licenses/by/4.0/
The gist: We introduce TerminalWorld, a scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from "in-the-wild" terminal recordings.
Terminology
Abstract
We introduce TerminalWorld, a scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from "in-the-wild" terminal recordings. Processing 80,870 terminal recordings, the engine yields a full benchmark of 1,530 validated tasks, spanning 18 real-world categories, ranging from short everyday operations to workflows exceeding 50 steps, and covering 1,280 unique commands. From these, we curate a Verified subset of 200 diverse, manually reviewed tasks. Comprehensive benchmarking on TerminalWorld-Verified across eight frontier models and six agents reveals that current systems still struggle with authentic terminal workflows, achieving a maximum pass rate of only 62.5%. Moreover, TerminalWorld captures real-world terminal capabilities distinct from existing expert-curated benchmarks (e.g., Terminal-Bench), with only a weak correlation to their scores (Pearson r=0.20). The automated engine makes TerminalWorld authentic and scalable by construction, enabling it to evaluate agents in real-world terminal environments as developer practices evolve.
Sources
- LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces
- TermiGen: High-Fidelity Environment and Robust Trajectory Synthesis for Terminal Agents
- CLI-Gym: Scalable CLI Task Generation via Agentic Environment Inversion
- Endless Terminals: Scaling RL Environments for Terminal Agents
- On Data Engineering for Scaling LLM Terminal Capabilities
- Large-Scale Terminal Agentic Trajectory Generation from Dockerized Environments
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents