Build-and-Find: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases

summary

The gist

Most coding-agent benchmarks ask whether generated code behaves correctly <ref:2605.06136#pg2> That remains essential…

In short

BUILD-AND-FIND evaluates whether AI agents can recover intended design choices from generated code repositories. It introduces a protocol separating behavioral correctness from artifact recovery, measuring accuracy, stability, and inspection effort to assess how well generated code communicates its intent.

Key concepts

Artifact-Evidenced Recovery
This is the core evaluation layer that checks if an agent can find the original design decisions within the repository itself before making changes. It distinguishes between checking if code *works* and checking if the *intent* is visible in the structure, tests, or documentation.
Builder–Finder Protocol
This defines two roles: a builder creates a codebase from hidden specifications, and a finder inspects only the resulting repository and question bank. The protocol measures how accurately a finder can recover those original intentions from the artifact.
Effort Interpretation
Inspection effort is only considered meaningful after recovery succeeds reliably. It is measured in 'novel agent inspection bytes'—the amount of new code or information retrieved by the finder—and serves as a proxy for how easy or hard it was to locate the intended design choice.

Terminology used across episodes

This episode discusses

The paper

Build-and-Find: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases · Read on arXiv

National Yang Ming Chiao Tung University

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Build-and-Find: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases".

Jane: Here's the gist of the paper. Most coding-agent benchmarks ask whether generated code behaves correctly <ref:2605.06136#pg2> That remains essential, but repository-level engineering is increasingly agentmanaged <ref:2605.06136#pg3> In that setting, a generated repository is not only an answer to a task but also a communication artifact for future work <ref:2605.06136#pg4> BUILD-AND-FIND introduces a protocol for evaluating whether downstream agents can recover those intended choices from generated repositories, and how much inspection that recovery requires <ref:2605.06136#pg5> The protocol separates behavioral correctness from artifact-side recovery and reports recovery accuracy, repeatability, implementation coverage, and inspection effort <ref:2605.06136#pg6> Accuracy and stability act as gates: effort is interpreted only when recovery succeeds reliably <ref:2605.06136#pg6> Among artifacts from which the same intent can be recovered, lower effort by the same finder suggests that the artifact makes that intent easier to locate <ref:2605.06136#pg6> Evaluation Target Definition BUILD-AND-FIND defines artifact-evidenced recovery of intended repository behavior and design choices as a distinct layer between behavioral correctness and downstream modification success <ref:2605.06136#pg8> Functional correctness asks whether the repository behaves correctly; downstream modification success asks whether a later agent can safely change it; artifact-evidenced recovery asks whether intended decisions are visible in the artifact before any modification is attempted <ref:2605.06136#pg8> In high-recovery regimes, accuracy alone is a coarse measure of artifact legibility <ref:2605.06136#pg9> An artifact that exposes intent through clear structure, tests, configuration, or documentation should require less finder inspection than one that buries the same intent <ref:2605.06136#pg9> The central idea is to treat generated code as external memory and as a communication medium <ref:2605.06136#pg10> A hidden specification defines intended behaviors and design choices; a builder encodes them in a codebase; and a finder, seeing only the codebase and a question bank, tries to recover them Builder–Finder Protocol The protocol involves two roles: builders see hidden specifications and produce repositories; finders see only those repositories plus specification-traced question banks The builder constructs a self-contained codebase Ab,t,r on trial r The finder inspects files, makes tool calls, and writes a structured answer file The intended construct is artifact-evidenced recovery: a correct answer should be distinguishable from plausible alternatives by evidence in source files, tests, configuration, or documentation Exact-match grading keeps the formal score deterministic Measurement Contract and Effort Interpretation The protocol reports recovery accuracy, stability, implementation coverage, and inspection effort; effort is interpreted as a proxy for artifact legibility only after recovery and stability gates The reported effort unit is novel agent inspection bytes: new artifact bytes retrieved or requested by the finder through retrieval and tool calls Conditional Rb is the geometric mean, over contributing all-correct finder-task cells, of the builder’s mean all-correct effort divided by the minimum contributing builder effort in that same finder-task cell The task-level recovery gate asks whether the cell is recoverable enough for effort to be interpreted Empirical Validation and Findings The empirical scope treats controls as part of the evaluation contract: they separate ordinary engineering priors, specification access, and artifact-conditioned recovery The empirical section starts with prior and stress calibration, then reports repeatability, implementation-aware recovery, conditional effort, rank stability, affinity, and low-prior evidence audits Table 9 shows the aggregate artifact-evidenced legibility score (AELS0.1), where higher is better The resulting benchmark does not replace behavioral correctness tests; it adds an artifact-side view of whether generated repositories expose intent clearly enough for later agents to use Known biases include the fact that among source-bearing final build cells, the median artifact contains roughly 4.2k source LOC BUILD-AND-FIND treats generated repositories as communication artifacts for future agents The builder–finder protocol asks whether a downstream agent can recover intended behavior and design choices from the repository alone In the released high-prior task pack, recovery accuracy is already near saturation, so inspection effort provides the main panel-local comparison after recovery and stability gates The resulting benchmark does not replace behavioral correctness tests; it adds an artifact-side view of whether generated repositories expose intent clearly enough for later agents to use References Mubashara Akhtar, Omar Benjelloun, Costanza Conforti, Luca Foschini, Pieter Gijsbers, Joan Giner-Miguelez, Sujata Goswami, Nitisha Jain, Michalis Karamousadakis, Satyapriya Krishna, Michael Kuchnik, Sylvain Lesage, Quentin Lhoest.

Tom: Croissant: A metadata format for ML-ready datasets Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang.

Jane: Program synthesis with large language models Jürgen Börstler, Kwabena E.

Tom: Bennin, Sara Hooshangi, Johan Jeuring, Hieke Keuning.

Paper summary: Jane: To summarize, the paper "Build-and-Find: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases" introduces this protocol to evaluate artifact-evidenced recovery of intended repository behavior and design choices. They are separating behavioral correctness from artifact legibility >

Tom: They propose a builder who sees a hidden specification and creates a codebase, and a finder who only sees that codebase and the specification-traced question bank, trying to recover those intended choices >

Lu: The protocol establishes this as the middle layer of evaluation, targeting artifact-side intent recoverability before moving on to modification success or downstream modification success >

Meng: I see they are focusing on what's visible in the source files, tests, configuration files, or documentation when evaluating if an agent can recover those design choices >

Lalam: They report recovery accuracy, stability, implementation coverage, and inspection effort as metrics; they even have a conditional Rb calculation that involves the builder’s mean all-correct effort divided by the minimum contributing builder effort in that same finder-task cell >

Tom: The main claim is that accuracy and stability act as gates; you only interpret the inspection effort once recovery succeeds reliably, and lower effort suggests easier intent location >

Jane: So, for someone listening who doesn't read this, it means they are creating a test where we check if an AI can figure out the original blueprint just by looking at the finished product >

Lu: It treats generated code like external memory and a communication medium; the goal is to see if that communication medium is clear enough for downstream agents to use as context when making future changes >

Meng: I'm still worried about the practical implications of this effort metric, though. How do you get an honest measurement of inspection effort when the finder has to use tool calls and retrieval?

Lalam: The unit they use is novel agent inspection bytes; this measures new artifact bytes retrieved or requested by the finder through those tool calls >

Conclusion: Tom: So, looking at "Build-and-Find: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases," the authors are essentially saying that generated repositories should be seen as communication artifacts for agents who will build on top of them >

Jane: And the protocol they developed, the builder and finder setup, asks whether a downstream agent can recover those intended behaviors and design choices using only the repository itself >

Lu: It adds an artifact-side view to evaluation that sits between checking if the code works correctly and checking if you can safely modify it later >

Meng: What this means for production is that we need benchmarks that test not just correctness, but how well the generated code communicates its design intent so our maintenance agents don't have to spend all their time reverse-engineering everything >

Lalam: The main implication is that we are moving toward evaluating how clearly generated repositories expose intent so later agents can actually use them as context >

Tom: They stress that this benchmark doesn't replace behavioral correctness tests; it just adds this artifact-side view to see if the repository exposes its design choices clearly enough for agents to use >

More episodes

← Home