Adaptive Auto-Harness: Sustained Self-Improvement for Agentic System Deployment on Open-Ended Task Streams
cs.LG, cs.AI
Submitted: 2026-06-01
Updated: 2026-11-01
Comments: version 2
License: http://creativecommons.org/licenses/by/4.0/
The gist: Auto-harness systems improve LLM agents by adapting prompts, skills, tools, and supporting infrastructure from execution feedback.
Terminology
Abstract
Auto-harness systems improve LLM agents by adapting prompts, skills, tools, and supporting infrastructure from execution feedback. Sustaining these gains on open-ended task streams requires handling growing histories, heterogeneous tasks, and shifting distributions. We introduce Adaptive Auto-Harness, a unified framework that separates persistent evolution workspaces from deployable harnesses and supports interchangeable evolution procedures. We compare configurations adapted from five prior approaches on recorded streams of prediction markets, security challenges, and event forecasts. With the same Sonnet solver, our full system reaches 80.9% market all-task accuracy and 50.2% security Pass@1, versus 22.2% and 37.2% without evolution; market accuracy also exceeds a market-following control's 72.0%. Our analyses show that artifact accumulation does not guarantee sustained gains, and knowledge that transfers across tasks can still lose value as the task distribution changes. Continued evolution can therefore help preserve earlier gains through revision of accumulated knowledge. Code: https://github.com/A-EVO-Lab/a-evolve.
Sources
- GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning
- PolyBench: Benchmarking LLM Forecasting and Trading Capabilities on Live Prediction Market Data
- Continual Harness: Online Adaptation for Self-Improving Foundation Agents
- Meta-Harness: End-to-End Optimization of Model Harnesses
- Position: Agentic Evolution is the Path to Evolving LLMs
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
- OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoning
- SkillOS: Learning Skill Curation for Self-Evolving Agents
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
- Kimi K2.5: Visual Agentic Intelligence
- Tent: Fully Test-time Adaptation by Entropy Minimization
- Voyager: An Open-Ended Embodied Agent with Large Language Models
- FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction
- Training Language Model Agents to Find Vulnerabilities with CTF-Dojo
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis