Multi-step SWE: Can Coding Agents Keep Building on Their Own Code Over Long Horizons?
cs.SE, cs.AI, cs.LG
Submitted: 2026-11-04
Updated: 2026-11-04
Code: https://github.com/harbor-framework/harbor
Terminology
Sources
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
- DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks
- Commit0: Library Generation from Scratch
- ProgramBench: Can Language Models Rebuild Programs From Scratch?
- SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
- SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades
- EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions
- SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution
- Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution
- Concrete Problems in AI Safety
- Defining and Characterizing Reward Hacking
- Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation
- SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
- Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving
- SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
- MigrationBench: Repository-Level Code Migration Benchmark from Java 8
- TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks
- SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories?
- SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters