Build-and-Find: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases

arXiv:2605.06136 · cs.SE, cs.AI · Submitted 2026-05-07 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Build-and-Find: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases".

Jane: Here's the gist of the paper. Most coding-agent benchmarks ask whether generated code behaves correctly <ref:2605.06136#pg2> That remains essential, but repository-level engineering is increasingly agentmanaged <ref:2605.06136#pg3> In that setting, a generated repository is not only an answer to a task but also a communication artifact for future work <ref:2605.06136#pg4> BUILD-AND-FIND introduces a protocol for evaluating whether downstream agents can recover those intended choices from generated repositories, and how much inspection that recovery requires <ref:2605.06136#pg5> The protocol separates behavioral correctness from artifact-side recovery and reports recovery accuracy, repeatability, implementation coverage, and inspection effort <ref:2605.06136#pg6> Accuracy and stability act as gates: effort is interpreted only when recovery succeeds reliably <ref:2605.06136#pg6> Among artifacts from which the same intent can be recovered, lower effort by the same finder suggests that the artifact makes that intent easier to locate <ref:2605.06136#pg6> Evaluation Target Definition BUILD-AND-FIND defines artifact-evidenced recovery of intended repository behavior and design choices as a distinct layer between behavioral correctness and downstream modification success <ref:2605.06136#pg8> Functional correctness asks whether the repository behaves correctly; downstream modification success asks whether a later agent can safely change it; artifact-evidenced recovery asks whether intended decisions are visible in the artifact before any modification is attempted <ref:2605.06136#pg8> In high-recovery regimes, accuracy alone is a coarse measure of artifact legibility <ref:2605.06136#pg9> An artifact that exposes intent through clear structure, tests, configuration, or documentation should require less finder inspection than one that buries the same intent <ref:2605.06136#pg9> The central idea is to treat generated code as external memory and as a communication medium <ref:2605.06136#pg10> A hidden specification defines intended behaviors and design choices; a builder encodes them in a codebase; and a finder, seeing only the codebase and a question bank, tries to recover them Builder–Finder Protocol The protocol involves two roles: builders see hidden specifications and produce repositories; finders see only those repositories plus specification-traced question banks The builder constructs a self-contained codebase Ab,t,r on trial r The finder inspects files, makes tool calls, and writes a structured answer file The intended construct is artifact-evidenced recovery: a correct answer should be distinguishable from plausible alternatives by evidence in source files, tests, configuration, or documentation Exact-match grading keeps the formal score deterministic Measurement Contract and Effort Interpretation The protocol reports recovery accuracy, stability, implementation coverage, and inspection effort; effort is interpreted as a proxy for artifact legibility only after recovery and stability gates The reported effort unit is novel agent inspection bytes: new artifact bytes retrieved or requested by the finder through retrieval and tool calls Conditional Rb is the geometric mean, over contributing all-correct finder-task cells, of the builder’s mean all-correct effort divided by the minimum contributing builder effort in that same finder-task cell The task-level recovery gate asks whether the cell is recoverable enough for effort to be interpreted Empirical Validation and Findings The empirical scope treats controls as part of the evaluation contract: they separate ordinary engineering priors, specification access, and artifact-conditioned recovery The empirical section starts with prior and stress calibration, then reports repeatability, implementation-aware recovery, conditional effort, rank stability, affinity, and low-prior evidence audits Table 9 shows the aggregate artifact-evidenced legibility score (AELS0.1), where higher is better The resulting benchmark does not replace behavioral correctness tests; it adds an artifact-side view of whether generated repositories expose intent clearly enough for later agents to use Known biases include the fact that among source-bearing final build cells, the median artifact contains roughly 4.2k source LOC BUILD-AND-FIND treats generated repositories as communication artifacts for future agents The builder–finder protocol asks whether a downstream agent can recover intended behavior and design choices from the repository alone In the released high-prior task pack, recovery accuracy is already near saturation, so inspection effort provides the main panel-local comparison after recovery and stability gates The resulting benchmark does not replace behavioral correctness tests; it adds an artifact-side view of whether generated repositories expose intent clearly enough for later agents to use References Mubashara Akhtar, Omar Benjelloun, Costanza Conforti, Luca Foschini, Pieter Gijsbers, Joan Giner-Miguelez, Sujata Goswami, Nitisha Jain, Michalis Karamousadakis, Satyapriya Krishna, Michael Kuchnik, Sylvain Lesage, Quentin Lhoest.

Tom: Croissant: A metadata format for ML-ready datasets Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang.

Jane: Program synthesis with large language models Jürgen Börstler, Kwabena E.

Tom: Bennin, Sara Hooshangi, Johan Jeuring, Hieke Keuning.

Paper summary: Jane: To summarize, the paper "Build-and-Find: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases" introduces this protocol to evaluate artifact-evidenced recovery of intended repository behavior and design choices. They are separating behavioral correctness from artifact legibility >

Tom: They propose a builder who sees a hidden specification and creates a codebase, and a finder who only sees that codebase and the specification-traced question bank, trying to recover those intended choices >

Lu: The protocol establishes this as the middle layer of evaluation, targeting artifact-side intent recoverability before moving on to modification success or downstream modification success >

Meng: I see they are focusing on what's visible in the source files, tests, configuration files, or documentation when evaluating if an agent can recover those design choices >

Lalam: They report recovery accuracy, stability, implementation coverage, and inspection effort as metrics; they even have a conditional Rb calculation that involves the builder’s mean all-correct effort divided by the minimum contributing builder effort in that same finder-task cell >

Tom: The main claim is that accuracy and stability act as gates; you only interpret the inspection effort once recovery succeeds reliably, and lower effort suggests easier intent location >

Jane: So, for someone listening who doesn't read this, it means they are creating a test where we check if an AI can figure out the original blueprint just by looking at the finished product >

Lu: It treats generated code like external memory and a communication medium; the goal is to see if that communication medium is clear enough for downstream agents to use as context when making future changes >

Meng: I'm still worried about the practical implications of this effort metric, though. How do you get an honest measurement of inspection effort when the finder has to use tool calls and retrieval?

Lalam: The unit they use is novel agent inspection bytes; this measures new artifact bytes retrieved or requested by the finder through those tool calls >

Conclusion: Tom: So, looking at "Build-and-Find: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases," the authors are essentially saying that generated repositories should be seen as communication artifacts for agents who will build on top of them >

Jane: And the protocol they developed, the builder and finder setup, asks whether a downstream agent can recover those intended behaviors and design choices using only the repository itself >

Lu: It adds an artifact-side view to evaluation that sits between checking if the code works correctly and checking if you can safely modify it later >

Meng: What this means for production is that we need benchmarks that test not just correctness, but how well the generated code communicates its design intent so our maintenance agents don't have to spend all their time reverse-engineering everything >

Lalam: The main implication is that we are moving toward evaluating how clearly generated repositories expose intent so later agents can actually use them as context >

Tom: They stress that this benchmark doesn't replace behavioral correctness tests; it just adds this artifact-side view to see if the repository exposes its design choices clearly enough for agents to use >

National Yang Ming Chiao Tung University

cs.SE, cs.AI

Submitted: 2026-05-07

Updated: 2026-11-06

Comments: 36 pages, 8 figures. Accepted at NeurIPS 2026. Code: https://github.com/JacobLinCool/Build-and-Find

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 87/100

The gist: Most coding-agent benchmarks ask whether generated code behaves correctly <ref:2605.06136#pg2> That remains essential…

Key concepts

Artifact-Evidenced Recovery
This is the core evaluation layer that checks if an agent can find the original design decisions within the repository itself before making changes. It distinguishes between checking if code *works* and checking if the *intent* is visible in the structure, tests, or documentation.
Builder–Finder Protocol
This defines two roles: a builder creates a codebase from hidden specifications, and a finder inspects only the resulting repository and question bank. The protocol measures how accurately a finder can recover those original intentions from the artifact.
Effort Interpretation
Inspection effort is only considered meaningful after recovery succeeds reliably. It is measured in 'novel agent inspection bytes'—the amount of new code or information retrieved by the finder—and serves as a proxy for how easy or hard it was to locate the intended design choice.

Terminology

Summary

Most coding-agent benchmarks ask whether generated code behaves correctly <ref:2605. p. 006136#pg2> That remains essential, but repository-level engineering is increasingly agentmanaged <ref:2605. p. 006136#pg3> In that setting, a generated repository is not only an answer to a task but also a communication artifact for future work <ref:2605. p. 006136#pg4> BUILD-AND-FIND introduces a protocol for evaluating whether downstream agents can recover those intended choices from generated repositories, and how much inspection that recovery requires <ref:2605. p. 006136#pg5> The protocol separates behavioral correctness from artifact-side recovery and reports recovery accuracy, repeatability, implementation coverage, and inspection effort <ref:2605. p. 006136#pg6> Accuracy and stability act as gates: effort is interpreted only when recovery succeeds reliably <ref:2605. p. 006136#pg6> Among artifacts from which the same intent can be recovered, lower effort by the same finder suggests that the artifact makes that intent easier to locate <ref:2605. p. 006136#pg6>

Evaluation Target Definition

BUILD-AND-FIND defines artifact-evidenced recovery of intended repository behavior and design choices as a distinct layer between behavioral correctness and downstream modification success <ref:2605.06136#pg8> Functional correctness asks whether the repository behaves correctly; downstream modification success asks whether a later agent can safely change it; artifact-evidenced recovery asks whether intended decisions are visible in the artifact before any modification is attempted <ref:2605. p. 106136#pg8> In high-recovery regimes, accuracy alone is a coarse measure of artifact legibility <ref:2605. p. 106136#pg9> An artifact that exposes intent through clear structure, tests, configuration, or documentation should require less finder inspection than one that buries the same intent <ref:2605. p. 106136#pg9> The central idea is to treat generated code as external memory and as a communication medium <ref:2605. p. 106136#pg10> A hidden specification defines intended behaviors and design choices; a builder encodes them in a codebase; and a finder, seeing only the codebase and a question bank, tries to recover them

Builder–Finder Protocol

The protocol involves two roles: builders see hidden specifications and produce repositories; finders see only those repositories plus specification-traced question banks The builder constructs a self-contained codebase Ab,t,r on trial r The finder inspects files, makes tool calls, and writes a structured answer file The intended construct is artifact-evidenced recovery: a correct answer should be distinguishable from plausible alternatives by evidence in source files, tests, configuration, or documentation Exact-match grading keeps the formal score deterministic

Measurement Contract and Effort Interpretation

The protocol reports recovery accuracy, stability, implementation coverage, and inspection effort; effort is interpreted as a proxy for artifact legibility only after recovery and stability gates The reported effort unit is novel agent inspection bytes: new artifact bytes retrieved or requested by the finder through retrieval and tool calls Conditional Rb is the geometric mean, over contributing all-correct finder-task cells, of the builder’s mean all-correct effort divided by the minimum contributing builder effort in that same finder-task cell The task-level recovery gate asks whether the cell is recoverable enough for effort to be interpreted

Empirical Validation and Findings

The empirical scope treats controls as part of the evaluation contract: they separate ordinary engineering priors, specification access, and artifact-conditioned recovery The empirical section starts with prior and stress calibration, then reports repeatability, implementation-aware recovery, conditional effort, rank stability, affinity, and low-prior evidence audits Table 9 shows the aggregate artifact-evidenced legibility score (AELS0.1), where higher is better The resulting benchmark does not replace behavioral correctness tests; it adds an artifact-side view of whether generated repositories expose intent clearly enough for later agents to use Known biases include the fact that among source-bearing final build cells, the median artifact contains roughly 4. p. 82k source LOC

Conclusion

BUILD-AND-FIND treats generated repositories as communication artifacts for future agents The builder–finder protocol asks whether a downstream agent can recover intended behavior and design choices from the repository alone In the released high-prior task pack, recovery accuracy is already near saturation, so inspection effort provides the main panel-local comparison after recovery and stability gates The resulting benchmark does not replace behavioral correctness tests; it adds an artifact-side view of whether generated repositories expose intent clearly enough for later agents to use

References

Mubashara Akhtar, Omar Benjelloun, Costanza Conforti, Luca Foschini, Pieter Gijsbers, Joan Giner-Miguelez, Sujata Goswami, Nitisha Jain, Michalis Karamousadakis, Satyapriya Krishna, Michael Kuchnik, Sylvain Lesage, Quentin Lhoest. Croissant: A metadata format for ML-ready datasets Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang. p. 8 Program synthesis with large language models Jürgen Börstler, Kwabena E. p. 8 Bennin, Sara Hooshangi, Johan Jeuring, Hieke Keuning. p. 8 Developers talking about code quality Raymond P. p. 8 L. Buse and Westley Weimer. p. 8 Learning a metric for code readability Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu. p. 8 MLEbench: Evaluating machine learning agents on machine learning engineering Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan. p. 8 Evaluating large language models trained on code Jane Cleland-Huang, Orlena C. p. 8 Z. Gotel, Jane Huffman Hayes, Patrick Mäder, Andrea Zisman. p. 9 Software traceability: Trends and future directions Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo. p. 9 Measuring coding challenge competence with APPS Ruida Hu, Chao Peng, Jingyi Ren, Bo Jiang. p. 9 Understanding large language model performance in software engineering: A large-scale question answering benchmark Naman Jain, King Han, Alex Gu, Wen-Ding Li. p. 9 LiveCodeBench: Holistic and contamination-free evaluation of large language models for code Carlos E. p. 9 Jimenez, John Yang, Alexander Wettig, Shunyu Yao. p. 9 SWE-bench: Can language models resolve real-world GitHub issues? p. 9 Andrew J. Ko and Brad A. p. 9 p. 9 Myers. Debugging reinvented: Asking and answering why and why not questions about program behavior Andrew J. p. 9 Ko and Brad A. Myers. Extracting and answering why and why not questions about Java program output Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih. p. 9 DS-1000: A natural and reliable benchmark for data science code generation Linyi Li, Shijie Geng, Zhenwen Li, Yibo He. p. 9 InfiBench: Evaluating the question-answering capabilities of code large language models Tianyang Liu, Canwen Xu, Julian McAuley. p. 9 RepoBench: Benchmarking repository-level code auto-completion systems Yue Liu, Thanh Le-Cong, Ratnadira Widyasari, Chakkrit Tantithamthavorn, Li Li. p. 9 Refining ChatGPT-generated code: Characterizing and mitigating code quality issues Shuai Lu, Daya Guo, Shuo Ren. p. 9 CodeXGLUE: A machine learning benchmark dataset for code understanding and generation Mike A. p. 9 Merrill, Alexander G. p. 9 Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj. p. 9 Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces Delano Oliveira, Reydne Bruno, Fernanda Madeiral, Fernando Castor. p. 9

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Evaluation target refinement: Introduce artifact-evidenced recovery of intended repository behavior and design choices as a distinct layer between behavioral correctness and downstream modification success to focus evaluation on whether the repository exposes intended behavior and design choices clearly enough for later agents to inspect, audit, and use it as context.

  2. Builder–finder protocol separation: Implement a protocol where builders see hidden specifications while finders only see the codebase and a specification-traced multiple-choice question bank to evaluate artifact-evidenced recovery of intended repository behavior and design choices from the artifact itself.

  3. Measurement contract gating: Interpret inspection effort as a proxy for artifact legibility only after recovery and stability gates, ensuring that effort is only considered when recovery succeeds reliably to prevent treating a low-effort but unstable artifact as equivalent to a consistently recoverable one.

  4. Prior and specification controls: Quantify generic priors with question-only controls and specification access with spec-only controls, allowing the protocol to separate ordinary engineering priors, specification access, and artifact-conditioned recovery.

  5. Conditional inspection effort metric: Report conditional builder conditioned artifact effort diagnostic Rb to measure agent-facing artifact legibility, which is interpreted as a proxy for legibility after accounting for recovery accuracy, repeatability, and coverage.

Sources

Related papers