TTHE: Test-Time Harness Evolution

arXiv:2607.08124 · cs.SE, cs.LG · Submitted 2026-07-09 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "TTHE: Test-Time Harness Evolution".

Tom: The gist The behavior of an LLM agent is determined not only by its underlying model, but also by its harness: the executable program that constructs context, invokes tools,

Jane: First, who's behind it and why it matters.

Paper summary: Tom: So, we're talking about this paper called "TTHE: Test-Time Harness Evolution." The main idea here is that the way an AI agent actually behaves isn't just about the model itself, but also the executable program that controls it—the harness.

Jane: Exactly. This paper argues that existing methods optimize this harness before deployment, searching for a fixed workflow in training data and freezing it at test time. But this is limited when the real test inputs or failure modes are different from what they saw during development.

Tom: So TTHE suggests we can optimize that harness right during evaluation, using only the traces the agent produces on its test inputs. It treats the executable harness as if it's adapting in real-time during testing.

Lu: That’s really interesting because it moves the optimization into the evaluation process itself rather than searching for a fixed setup on labeled development instances and freezing it. This means we don't need gold labels or training a separate adaptation model.

Meng: So, instead of finding the perfect workflow beforehand, we use unlabeled test batches to refine a population of candidate harnesses through an agentic proposer that reasons over the execution traces they generate without any supervision.

Lalam: The adaptation target is this executable control program that wraps the frozen solver M and decides how the agent builds context, invokes tools, validates intermediate results, and recovers from failures. It’s about evolving that whole control structure on-the-fly during testing.

Tom: Right. And to guide this evolution they use execution traces—things like model prompts and completions, tool calls and outputs, stdout or stderr—to derive some lightweight proxy signals to show what reliable execution looks like.

Jane: They use these proxy signals, such as "execution health," which is a score from zero to one indicating whether the agent runs without error and returns a well-formed result. They also have round-trip consistency, which checks if the output can be inverted back into a description and scored against the input.

Tom: And they use public test pass rates as another signal, which just shows how often an output passes some public examples or repository tests—it's direct but incomplete evidence of success.

Lu: The methodology involves a judge loop where the solver, proposers, and judge are all different roles and harnesses around the same frozen LLM. They demonstrate gains across five execution-grounded domains including Text-to-SQL, competitive programming, software engineering, and data science coding.

Meng: So they show tangible improvements in these areas. For instance, in Text-to-SQL for the BIRD task, they improved it from twenty percent to fifty percent by evolving the harness this way. And for SWE-bench Verified they got that up from twenty percent to thirty five percent as well.

Paper summary: Jane: The evolved harnesses aren't just better; they are inspectable source code, meaning you can actually look at the policy improvements they made, like value grounding through database probes or explicit output-shape checks for Text-to-SQL.

Tom: It sounds like this entire approach is focused on turning evaluation time execution into actual updates to the program that controls an agent without touching the model's weights or needing gold labels. This is a big shift away from adaptation methods that fine-tune model parameters during testing.

Lu: The paper points out that this transductive test-time adaptation setting, where a batch supplies traces used to select a harness, doesn't automatically establish forward generalization for future inputs. That’s an important caveat they bring up.

Meng: And the decisive test left for future work is prequential scoring—evaluating the harness committed on one batch against the next batch before it adapts to that next one. That sounds like where they need to go next.

Jane: They also highlighted that selection is made by a learned agentic judge, and their selection regret above shows there’s still room for improvement in how that judge picks the best harness from the candidate set.

Tom: So, looking at "TTHE: Test-Time Harness Evolution," it’s about using evaluation traces to evolve the control program of an AI agent on the fly, rather than pre-optimizing a fixed workflow. The authors show it improves baseline ReAct-style harnesses and gives us inspectable policies for grounding and verification.

Lu: What this means for the world is that we can start building agents whose operational procedures are robustly shaped by their actual failures during testing, rather than relying solely on training data patterns.

Meng: Practically speaking, if we want an agent to reliably interact with a database or write correct code, having its control program evolve based on real execution traces sounds like a much more stable way to build those systems up.

Jane: It gives us a path toward creating agents that are inherently better at things like verifying intermediate results or fixing errors because the harness itself is being tuned by experience.

Tom: So, to wrap up this part of the paper, TTHE turns evaluation execution into updates for the program controlling an agent without changing model weights or using gold labels, improving those baseline harnesses we rely on.

Lu: The key challenge they identify is that proxy reliability—those signals like execution health and round-trip consistency—is what makes this evolution robust across different tasks.

Meng: I think that means the next step for engineers will be figuring out how to make sure those proxy signals accurately reflect whether the agent is actually doing what we want it to do.

Jane: And for listeners, this paper shows us a new way of thinking about AI evaluation—moving from looking at the model's final output to inspecting and improving the entire decision-making process behind that output.

Conclusion: Tom: So we’re wrapping up "TTHE: Test-Time Harness Evolution." The authors are looking at how we can fix the control program of an AI agent directly during testing, instead of just before it goes live.

Jane: It turns the entire evaluation process into a way to update that program based on what happens when the agent actually tries to solve a problem.

Lu: Basically, they aren't training new model weights or using gold labels; they’re evolving an executable harness that wraps the frozen solver M.

Meng: So if I understand right, instead of finding the perfect workflow in development and freezing it, you let the agent run on test data and use its execution traces to refine that control program iteratively?

Lalam: Exactly. They maintain a population of candidate harnesses and use an agentic proposer to reason over those traces, which is how they decide what the next harness should look like.

Tom: And they do this using these proxy signals—things like execution health or round-trip consistency—instead of relying on some perfect reward function to tell them what’s good.

Jane: That’s a big deal because it shows that you don't need perfect human feedback or gold data to improve the agent’s operational logic.

Lu: The results show solid gains across five different kinds of coding and problem-solving tasks, like Text-to-SQL and competitive programming.

Meng: From an engineering standpoint, if this means we can generate inspectable policies for verification and repair, that’s practical stuff for building more reliable systems.

Lalam: And for culture, it means the AI isn't just following a fixed script; it’s actually learning how to ground its actions in real execution traces.

Tom: It points toward a future where we tune the agent's behavior by letting it fail and learn from those failures right on the test bench.

Jane: But there are still things they flag as not fully solved, like making sure that selection process—the judge choosing the next harness—is totally reliable.

Lu: That’s their main limitation, they need to figure out how to make that selection more robust before it can scale up across every task imaginable.

Hong Kong Baptist University · University of Science and Technology of China

cs.SE, cs.LG

Submitted: 2026-07-09

Updated: 2026-10-12

Comments: 22 pages, 5 figures

Code: https://github.com/junnie00/TTHE

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 89/100

The gist: The gist The behavior of an LLM agent is determined not only by its underlying model, but also by its harness: the executable program that constructs context, invokes tools, verifies intermediate

Key concepts

Test-Time Harness Evolution (TTHE): TTHE treats the executable program that controls an AI agent's behavior (the harness) as the state being adapted during testing. It refines this harness by running it on test data and using automated agents to propose and select better versions based purely on execution traces, not external rewards or labels.
Proxy Signals: Since TTHE avoids relying on gold labels or scalar rewards, it derives lightweight signals from execution traces to judge harness quality. These include 'Execution health' (whether the output runs without error), 'Round-trip consistency' (checking if the output matches a description derived from it), and 'Public-test pass rate,' which uses existing public tests as incomplete evidence.
Agentic Proposers: These are agents that operate within TTHE to refine the harness. They reason over the execution traces generated by various candidate harnesses, identifying weaknesses in their logic or tool usage. Their job is to propose edits to the parent harness, effectively guiding its evolution toward a more reliable control program.
Harness (Executable Program): The harness is the specific executable code that dictates how an LLM agent interacts with tools and context. TTHE focuses on optimizing this program—how it constructs prompts, calls tools, validates results, and recovers from errors—rather than changing the frozen LLM itself.

Terminology

Adaptation — 5× in this paper · explained in 1 episode
Test-Time Harness Evolution — 2× in this paper
Trace — 2× in this paper · explained in 2 episodes
Failure Modes — 1× in this paper · explained in 3 episodes
Text-to-SQL — 1× in this paper · explained in 3 episodes

Summary

The gist The behavior of an LLM agent is determined not only by its underlying model, but also by its harness: the executable program that constructs context, invokes tools, verifies intermediate results, and recovers from failures.

How it works

Test-Time Harness Evolution (TTHE) treats the executable harness as the state of test-time adaptation During evaluation, TTHE maintains a population of candidate harnesses and refines them through an agentic proposer that reasons over their execution traces, without gold labels or task-specific supervision. A judge then commits an improved harness from execution-derived proxy signals, and the selected program persists to govern subsequent inputs. Crucially, TTHE does not update model weights, require gold labels, or train a separate adaptation model. The solver, proposers, and judge are different roles and harnesses around the same frozen LLM.

Adaptation Mechanism

TTHE moves harness optimization into the evaluation process itself rather than searching for a harness on labeled development instances and freezing it for evaluation. On each unlabeled test batch, TTHE maintains a population of candidate harnesses, executes them under instrumentation, and iteratively refines them through agentic proposers that reason over their unlabeled execution traces and proxy-indicated weaknesses. The adaptation target is the harness: an executable control program that wraps the frozen solver M and determines how the agent constructs context, invokes tools, validates intermediate results, and recovers from failures.

Optimization Signals

TTHE assumes no trusted scalar reward. Running a harness H on an input x to produce o = H(x) yields a detailed execution trace: the model prompts and completions, tool calls and outputs, stdout/stderr, intermediate artifacts, runtime states, errors, and probe results. From each trace TTHE also derives a few lightweight, task-dependent proxy signals that encode prior knowledge of what reliable execution looks like. These proxies include Execution health s ∈ 0, 1: o runs without error and returns a well-formed, shapeconsistent result, Round-trip consistency b ∈ [0, 1]: the output is inverted into a description x˜ = RT(o) and scored against the input, b = match(x, x˜); because x˜ is derived from the output rather than from gold, and Public-test pass rate p ∈ [0, 1]: where a task exposes public examples or repository tests, the fraction o passes—direct but incomplete evidence.

Methodology and Results

TTHE processes an unlabeled stream x1,..., xT in batches with a frozen solver M and an executable harness H, maintaining a single committed harness across the stream. For batch Xt it starts from the current harness Ht, evolves a population of candidate harnesses over R rounds, and commits one harness Ht+1 that carries forward to Xt+1. The search loop involves three labelfree operators: OBSERVE(H, Xt) executes every harness in H on every input of Xt and returns their detailed traces, cached outputs, and derived proxy signals, PROPOSE(G), which runs G proposers in parallel to edit their own parent branch H into one child harness, and JUDGE(F, T, Xt) is an agentic selector over the final branch set F that commits one harness.

Performance Across Tasks

TTHE improves fixed ReAct-style baseline harnesses across the evaluated settings and produces inspectable policies for grounding, verification, and repair. The results show significant gains across diverse execution-grounded tasks, including Text-to-SQL (BIRD), competitive programming (LiveCodeBench), software engineering (SWE-bench Verified), and data-science coding (DS-1000). For the Text-to-SQL slice, TTHE improved BIRD from 12.0% to 50.0%, and for SWE-bench Verified from 20.0% to 35.0%. The evolved harnesses are inspectable source code, with recurring policies including value grounding through database probes, date-format repair, explicit output-shape checks, and removal of accidental limits on list queries for Text-to-SQL.

Limitations

The main protocol follows a transductive test-time adaptation setting: a batch supplies the unlabeled traces used to select a harness and is then evaluated after adaptation, but this does not by itself establish forward generalization. The decisive test left for future work is prequential scoring, evaluating the harness committed on batch t against batch t+1 before it adapts. Furthermore, selection is made by a learned agentic judge; the selection regret above locates the bottleneck there.

Conclusion

TTHE turns evaluation-time execution into updates to the program that controls an agent without changing model weights or revealing gold, improving baseline harnesses and producing inspectable grounding, verification, and repair policies. The findings identify the executable harness as a practical test-time adaptation state and proxy reliability as the key challenge for making such evolution robust.

Optimization Signals

Improvements for AI systems

Test-Time Harness Evolution (TTHE) optimizes executable control programs during evaluation using unlabeled execution traces, which allows adaptation to test distribution differences without needing gold labels or task-specific supervision. This enables persistent, inspectable improvements by treating the harness as the state of test-time adaptation.
The agent can develop robust grounding and verification policies by leveraging execution-derived proxy signals such as Execution health s ∈ [0, 1] and Round-trip consistency b ∈ [0, 1]. These signals allow the system to identify when an output is operationally plausible but benchmark-incompatible or exhibits semantic drift.
The system can achieve complex behavioral changes, such as revised context construction, multi-sample generation, tool-mediated grounding, deterministic contract checks, and conditional recovery branches, which are difficult to encode in a single textual reflection. This is realized through the population search where proposers edit its own harness into one child harness based on reading other branches' traces for context.
The system can learn sophisticated debugging and repair strategies, such as the SWE-bench policy of reproduce-first debugging, root-cause localization before editing, which is discovered from label-free execution traces alone. This results in an evolved harness that includes a post-rollout repair branch that re-runs the agent with a sharper prompt whenever the first rollout returns an empty patch.
The system can adapt its behavior to specific workloads, leading to domain-specific policies like Text-to-SQL harnesses that implement value grounding through database probes and explicit output-shape checks. This shows how test-time evolution produces inspectable source code policies tailored to the task's execution environment.

Abstract

The behavior of an LLM agent is determined not only by the underlying model, but also by its harness: the executable program that constructs context, invokes tools, verifies intermediate results, and recovers from failures. Existing approaches typically optimize harnesses on training or development data and freeze the resulting programs at test time, limiting adaptation to shifts in task distributions, failure modes, and tool interactions. We ask whether the harness can instead be optimized during evaluation using unlabeled execution traces. We introduce Test-Time Harness Evolution (TTHE), which treats the executable harness as the state of test-time adaptation. On each unlabeled test batch, TTHE iteratively evolves a population of candidate harnesses through agentic proposers that reason over execution traces and explore diverse edit directions. A judge selects a candidate using execution traces and execution-derived proxy signals. The selected harness determines the current batch's outputs and initializes adaptation on the next. Crucially, the solver, proposers, and judge operate through different harnesses around the same frozen LLM, without weight updates, access to final-answer labels, or a separately trained adaptation model. Across six execution-grounded benchmarks spanning text-to-SQL, competitive programming, software engineering, data-science coding, terminal-based tasks, and agentic tool use, TTHE improves fixed ReAct-style baseline harnesses through persistent, inspectable program adaptation. These results recast test-time adaptation for LLM agents as evolution over executable control programs and identify execution-derived proxy reliability as a central challenge for robust unsupervised agent improvement.