SaltBench: A Referee-Gated Protocol for Measuring Method Effects in Machine-Checked Software Work

arXiv:2609.11076 · cs.SE, cs.LO · Submitted 2026-09-10 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "SaltBench: A Referee-Gated Protocol for Measuring Method Effects in Machine-Checked Software Work".

Jane: Here's the gist of the paper. SaltBench is a benchmark protocol for one question: How does a machine referee change the way a coding agent works?

Tom: The outcome is decided by a machine referee the agent cannot argue with — a proof kernel, a program verifier, or a withheld test suite — run outside the agent’s own toolchain and the comparison is fixed before the agent runs, so that it cannot be narrated afterwards and every run is authorized by a dated freeze with its predictions registered and a budget stop is a halt, never a failure.

Jane: Benchmark Design and Protocol SaltBench is built on two refusals The outcome is decided by a machine referee the agent cannot argue with — a proof kernel, a program verifier, or a withheld test suite run outside the agent’s own toolchain and the comparison is fixed before the agent runs, so that it cannot be narrated afterwards.

Tom: The protocol is pre-registered

12: : the population is drawn by a committed seed, the arms are byte-pinned, the predictions are written down, and the freeze commit is the authorization Everything that happens afterwards is a dated amendment appended to the record.

Paper summary: Tom: Right, so they’re talking about this benchmark protocol called SaltBench. They set up four different arms to test how the agent performs depending on whether it’s just a plain agent, or if it's instructed to write a specification and verify its code against that spec.

Jane: And they also look at where that specification comes from—either the agent writes it, or someone hands it over beforehand as the formal statement of the task. It’s comparing those different setups.

Lu: The substrate they chose is important because they made sure all four arms are comparable, using Rust under a pinned Verus toolchain and scoring them with the same withheld test suite across the board.

Meng: But what's the actual finding here? Are we seeing some huge difference in performance, or is it more about how much work each setup takes? I need to know what this means for deploying models that rely on these kinds of checks.

Lalam: The measurement section tests five different systems components, all written in Rust using that pinned Verus toolchain and scored by the withheld test suite. They found that the arm instructed to specify and verify code cost more across those five components.

Tom: And they quantified it pretty clearly—across those five components, no premium exceeded two point eight thousand eight hundred seventy-nine times under either reading of the declared set. And three of them sat below a floor of two point seventy-two times in both readings.

Jane: That number is interesting, because it suggests that while specifying and verifying costs more, it’s not an astronomical jump for these specific five components they tested. So what does that imply about the actual cost premium?

Lu: They also have a primary reading which is a cross-problem sign test on something called P equals median of diet bare versus median of plain bare, which gives one premium per problem.

Meng: And what did that sign test show? I want to know if it’s showing a real trend or just noise in the data.

Lalam: Five out of five problems gave a p value of zero point three hundred twelve four out of five gave p value of zero point one thousand eight hundred seventy-five and three out of five gave p value of zero point five thousand. And those three premiums below the floor are the ones that suggest the design can’t say that the true value is above one point zero at all.

Tom: So, to summarize, they claim they've measured how changing the way a machine referee works alters agent performance, and in this case, it mostly affects how much effort is needed for specification and verification.

Conclusion: Jane: Thinking about the whole SaltBench paper again—"SaltBench: A Referee-Gated Protocol for Measuring Method Effects in Machine-Checked Software Work"—what’s the main thing we should get out of this, beyond just the numbers?

Tom: The main thing is that they’ve built a rigorous way to measure method effects. By fixing the referee outside the agent's toolchain and freezing predictions with dated commits, they create a protocol where you can actually say what changed and you can trust it later.

Lu: It establishes a framework for comparing different coding methods by making sure the comparison itself is fixed before the work even starts. It’s about measuring how much of a method's effort is attributable to the agent versus other factors.

Meng: For someone trying to actually use this in production, does this mean anything practical right now? Can I use this structure when my models are running on real traffic instead of these controlled components?

Lalam: They also suggest some instrument findings that they think apply to any benchmark of this kind. The paper says the instrument findings, as results, were found by a gate, a probe, or a re-check in the record; each changed the protocol and taught them what it was.

Jane: So even if the specific measurement didn't separate some arms, their findings from those checks are still valuable for understanding how these agents operate. It’s about learning from what breaks when you put these checks in place.

Tom: Exactly. The paper lays out a method for controlled comparison that isn't just a quick snapshot; it’s a system where the measurement process itself is recorded and authorized. It sets the standard for how we should measure these complex agent interactions moving forward.

cs.SE, cs.LO

Submitted: 2026-09-10

Updated: 2026-11-05

Code: https://github.com/jyh/salt

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 91/100

The gist: SaltBench is a benchmark protocol for one question: How does a machine referee change the way a coding agent works? The outcome is decided by a machine referee the agent cannot argue with…

Key concepts

SaltBench
A benchmark protocol designed to measure how a machine referee affects a coding agent's work. The outcome is fixed before the agent runs by an unarguable external checker, ensuring comparisons are reliable and reproducible.
Machine Referee
An external mechanism used to decide the outcome of the comparison between two agents. This could be a proof kernel, a program verifier, or a withheld test suite. The agent cannot argue against this decision.
Arms (Method vs. Specification)
The benchmark has four arms testing different aspects of the coding process: how the agent works (plain vs. specification-driven) and where the task specification comes from (written by the agent versus provided beforehand). These arms are run under identical conditions.
Cost Premium
A measure of how much more time or resources an agent method takes compared to a baseline method. The study found that specifying and verifying code increased costs, but most premiums were below a floor of 2.0072x.

Terminology

Summary

SaltBench is a benchmark protocol for one question: How does a machine referee change the way a coding agent works? p. 0 The outcome is decided by a machine referee the agent cannot argue with — a proof kernel, a program verifier, or a withheld test suite — run outside the agent’s own toolchain and the comparison is fixed before the agent runs, so that it cannot be narrated afterwards and every run is authorized by a dated freeze with its predictions registered and a budget stop is a halt, never a failure. p. 0

Benchmark Design and Protocol

SaltBench is built on two refusals The outcome is decided by a machine referee the agent cannot argue with — a proof kernel, a program verifier, or a withheld test suite run outside the agent’s own toolchain and the comparison is fixed before the agent runs, so that it cannot be narrated afterwards. The protocol is pre-registered [12]: the population is drawn by a committed seed, the arms are byte-pinned, the predictions are written down, and the freeze commit is the authorization Everything that happens afterwards is a dated amendment appended to the record. p. 0 The benchmark design has four arms, two-by-two One axis is the method: a plain agent, against an agent instructed to write a specification and verify its code against it The other is where the specification comes from: written by the agent, or handed to it a priori as the formal statement of the task. p. 1 The substrate is chosen so that the four arms are comparable Every arm writes Rust under the same pinned Verus toolchain and is scored by the same withheld suite. p. 1

Measurement of Method Effects

The study tested five systems components, all authored in Rust under a pinned Verus toolchain, with a withheld test suite as the referee for each Four arms are tested: a plain agent; an agent that is also instructed to create a specification and verify the code against it, in a reduced rendering of the method, as registered, and two arms where the specification is provided a priori, extended under a dated amendment to k = 4 We found that the arm instructed to specify and verify cost more on all five components, and by a practical margin: across these five components no premium exceeded 2.8879× under either reading of the declared set, and the three cheapest sat below 1. p. 04×.

Key Findings on Cost Premium

The v3 reading returns a registered cost premium on all five problems, and a post-hoc correctness pass that did not separate the arms The primary reading is a cross-problem sign test on premium(P) = median(diet-bare, P)/median(plain-bare, P), one premium per problem. The outcome table was enumerated before the first cell Five of five gives p = 0. p. 70312, four of five gives p = 0. p. 71875 and is registered in advance as not a positive result, three of five gives p = 0. p. 75000. Three of the five premiums sit below the design’s resolvable floor of 2. p. 80072× under both readings. p. 8

Instrument Findings and Limitations

The instrument did not separate the arms The post-hoc pass below did not separate the arms on that question, and it remains the question the next wave exists to answer. A price difference is not a capability difference, and nothing in that design licenses reading one from the other. p. 6 The three premiums below the floor are the ones such an interval would carry, and for those three the design cannot say that the true value is above 1. p. 80 at all.

Instrument Findings as Results

The instrument findings, as results Each was found by a gate, a probe, or a re-check in the record; each changed the protocol; and each is stated as the rule it taught, with the episode that taught it beside it. A censored cap returns the cap The statistic is a cap’s price only in the cap’s unit. p. 9

Reproducibility and Future Work

Everything in this paper can be re-derived, and this section says from what The code is released under Apache-2.0 and the data and documents under CC BY 4. p. 60. Version 2’s remedy for the last of these is under way and is described in Section 7. p. 1

Acknowledgements

CLEVER is by the Trishul group at UT Austin (MIT licence) VeruSAGE-Bench and lynette are by Microsoft (MIT licence) The five systems components of S3, their withheld suites and their mutant sets were authored for this benchmark by the author’s assistant heads under the author’s direction The runs were executed by Claude Code with Anthropic’s Sonnet 5 and Opus 5 models.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Precise cost measurement for agent work is established by using machine referees such as a proof kernel, a program verifier, or a withheld test suite to decide what an agent’s work is worth. This allows for measuring the method effects in machine-checked software work.

  2. Agent instruction files must be byte-pinned and the comparison must be fixed before the agent runs so that it cannot be narrated afterwards, ensuring that the comparison is not based on an agent's unreliable account of its own work.

  3. The system can differentiate between writing a specification and verifying code by testing arms where an agent is instructed to create a specification and verify the code against it, in a reduced rendering of the method, as registered. This reveals how much the arm instructed to specify and verify cost more on all five components.

  4. The system can distinguish between specifications written by the agent versus those provided a priori by testing arms where the specification is provided a priori, isolating the cost of verifying against a specification from the cost of writing one.

  5. The system can be evaluated under multiple readings, such as Reading A and Reading B, which are defined differently to show how different metrics capture different aspects, allowing for a comparison between the continuity reading and what was produced by the run.

  6. The system can identify when a post-hoc correctness pass does not separate arms, noting that the post-hoc pass did not separate the arms on that question. This prevents misinterpretation of a lack of separation as an absence of capability difference.

  7. The system can be used to determine if a method changes what a referee accepts by observing that no cell of this matrix carried a referee verdict on a withheld suite when it was scored.

  8. The system can identify when the budget stop is not a failure, noting that a budget stop is not a correctness failure and that the resulting sample is easier on exactly one arm.

  9. The system can be used to measure box-load terms by observing how the harness reaps the client and the watcher and does not reap what the subject forked, quantifying potential concurrency confounds.

  10. The system can establish a bound on dataset growth by showing that a bound does not cover a set that grew after it was written, and the growth is silent, preventing misinterpretation of changing datasets as new findings.

Sources

Related papers