Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement
cs.AI
Submitted: 2026-09-01
Updated: 2026-09-01
Comments: Github: https://github.com/Flesymeb/HarnessOfHarness Project Page: https://flesymeb.github.io/HarnessOfHarness/
Code: https://github.com/openai/codex
License: http://creativecommons.org/licenses/by/4.0/
The gist: This paper studies autonomous software development, in which LLM-based coding agents transform high-level requirements into complete, functional, and usable software systems without human
Terminology
Abstract
This paper studies autonomous software development, in which LLM-based coding agents transform high-level requirements into complete, functional, and usable software systems without human intervention. We introduce Harness-of-Harness (HoH), a framework that enables coding agents to continually improve software during autonomous development. HoH operates on existing coding-agent harnesses, and organizes their executions into iterative planning-coding-testing loops. To sustain improvement across loops, HoH balances repair with capability growth, scopes development into small and verifiable increments, separates implementation-time testing from independent evaluation, and constrains verifiable outputs rather than prescribing agent workflows. It progressively exposes deliverables, role-specific tools, and skills, encourages reuse rather than recreation, and maintains versioned project histories. On GameCraft-Bench, FrontierSWE, and ProgramBench, three harness-model pairs (Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, and Pi with MiniMax-M3), HoH consistently outperforms the corresponding standalone harnesses, achieving an average relative gain of 52.25 percent and a maximum gain of 82.86 percent after three iterations. In a multi-day deployment with more than 70 iterations, HoH autonomously develops a first-person-shooter game, featuring a coherent storyline, fully implemented core mechanics, human-playable experience, polished visuals and integrated audio. Github: https://github.com/Flesymeb/HarnessOfHarness Project Page: https://flesymeb.github.io/HarnessOfHarness/
Sources
- Evaluating Large Language Models Trained on Code
- Proof-or-Stop: Don't Trust the Agent, Trust the Evidence -- Loop Engineering for Verifiable Evidence-Gated Lifecycle Control
- SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios
- Meta-Harness: End-to-End Optimization of Model Harnesses
- Dive into Claude Code: The Design Space of Today's and Future AI Agent Systems
- Towards Iterative End-to-End Software Development: A Feature-Driven Multi-Agent Framework
- AutoHarness: improving LLM agents by automatically synthesizing a code harness
- ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development
- GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?
- Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
- SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
- RefAgent: A Multi-agent LLM-based Framework for Automatic Software Refactoring
- MemGPT: Towards LLMs as Operating Systems
- ProgramBench: Can Language Models Rebuild Programs From Scratch?
- Advancing DRL Agents in Commercial Fighting Games: Training, Integration, and Agent-Human Alignment
- Self-Harness: Harnesses That Improve Themselves
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems