Is Your Long-Horizon Benchmark Really Deep? Measuring and Synthesizing Depth in Terminal Tasks
cs.AI
Submitted: 2026-11-01
Updated: 2026-11-01
Comments: 28 pages, 14 figures
Code: https://github.com/harbor-framework/harbor
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Evaluating Large Language Models Trained on Code
- Terminal-World: Scaling Terminal-Agent Environments via Agent Skills
- TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks
- DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
- SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?
- Toward Scalable Terminal Task Synthesis via Skill Graphs
- Tmax: A simple recipe for terminal agents
- Measuring AI Ability to Complete Long Software Tasks
- Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading
- Recursive Synthesis for Long-Horizon Terminal Tasks
- CLI-Gym: Scalable CLI Task Generation via Agentic Environment Inversion
- SecProbe: Adaptive Evaluation of Coding Agents on Cybersecurity Vulnerabilities
- Synthetic Interaction Data for Scalable Personalization in Large Language Models
- CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks
- Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
- LiteCoder-Terminal: Scaling Long-Horizon Terminal Environments for Learning Language Agents
- On Data Engineering for Scaling LLM Terminal Capabilities
- OpenThoughts-Agent: Data Recipes for Agentic Models
- SETA: Scaling Environments for Terminal Agents
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents