Verifiers in the Loop: Decoding Time Verification for Code Translation

arXiv:2605.17626 · cs.LG, cs.SE · Submitted 2026-05-17 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Verifiers in the Loop".

Jane: The gist The Decoding Time Verification (DTV) framework introduces an interleaving decoding strategy that treats structural boundaries as meta steps for verifier-guided generation,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at this paper called "Verifiers in the Loop: Decoding Time Verification for Code Translation." It’s about how to make large language models better at translating code, and they introduce something called Decoding Time Verification or DTV.

Jane: Exactly. The main idea is that instead of waiting until the whole piece of code is written before checking if it's correct, you check during the writing process itself. They treat structural boundaries as special steps in how the AI generates text, which helps stop errors from building up in a way that ruins the whole generation sequence.

Lu: It’s like instead of writing a whole book and then sending it to an editor to find all the typos, you have a system that checks every chapter as you write it. This allows for test-time scaling of models like Qwen3-4B to improve pass rates from seventy-two point three percent on C-to-Rust up to eighty-two point zero percent, which is pretty significant because it uses fewer tokens per case than the standard self-refine baselines one.

Meng: Fewer tokens per case is a big deal for practical use, right? It means faster processing for users translating code, and that's what I care about from an engineering standpoint. What does this DTV framework actually do step-by-step?

Tom: Well, it uses a deterministic state machine controller. They have these steps: GENERATE extends the prefix to the next structural boundary—like a statement or a block on Page two—then VERIFICATION renders that part and runs checks on Page two <ref:2605.17626#pg1>.

Jane: And if those checks pass, they COMMIT the prefix as a rollback target, meaning it’s safe to keep going. If something fails during VERIFICATION, they ROLLBACK to that checkpoint with diagnostics attached on Page two <ref:2605.17626#pg1>.

Lu: Then there's FEEDBACK, which is where they take those diagnostic summaries and inject them right back into the model context for retries on Page five which is really smart because it lets the AI learn from its mistakes immediately. This whole loop prevents errors from propagating through the autoregressive context on Page three.

Meng: So this structure-aware rollback sounds like it’s not just a simple retry mechanism. They say they escalate the scope on repeated failures, rolling back to the smallest scope implicated by the error and then checking larger scopes if it keeps failing again on Page two <ref:2605.17626#pg1>.

Title and authors: Tom: That escalation is key because it means they aren't just stuck re-trying the same small section over and over; they are actively trying to find where the structural problem started. This whole setup seems designed to keep generation moving forward while ensuring the resulting code is actually valid according to the verifiers.

Jane: And on Page three they show that this in-loop verification alone can improve one-shot pass rates by about thirty point seven percent over just direct decoding, which shows a distinct scaling mechanism is at play here. It’s not just an extra layer on top of existing methods.

Lu: The ablation study confirms that all three key design components—the boundary checks, the rollback with escalation, and the feedback—all contribute to the final performance gain on Page three. And they highlight that in-loop recovery is actually providing the biggest benefit across those different parts one <ref:2605.17626#pg1>.

Meng: It’s good to see that. From an engineer's view, knowing exactly which part of the DTV mechanism gives you the most bang for your buck helps when you try to implement this on a new model. What about how this compares to other methods they looked at?

Tom: They compared it against post-hoc verification and sampling-based scaling. The results show that DTV achieves a better pass rate versus cost tradeoff than both of those in the cost-matched grid on Page three. It sits on the upper left of the cost versus pass-rate frontier for both C-to-Rust and JavaScript-to-TypeScript tasks one <ref:2605.17626#pg1>.

Jane: That suggests it’s a more efficient way to get high quality results without necessarily needing massive increases in compute, which is always a nice thing when you're working with large models. It also notes that DTV is more token-efficient than the naive self-refine baseline on Page ten.

Lu: And they found that this gain tends to concentrate on cases where the fix is local, meaning early intervention helps a lot, but it weakens when the error is nonlocal, which shows where its effectiveness has limits one <ref:2605.17626#pg1>.

Meng: So it’s not a perfect solution for every kind of bug. It works best when the issue is close to where it happens. That’s a crucial piece of information for deployment planning because you can anticipate where the system will actually succeed and where you might need more brute force scaling or different techniques.

Title and authors: Tom: Exactly. And they mentioned that DTV/self-refine dominates naïve self-refine, sitting above the best-of-N and S* frontiers at matched cost on Page three one. This means for a given amount of computing power, you get a better pass rate with this method than other common scaling techniques.

Jane: It’s clear that the paper "Verifiers in the Loop: Decoding Time Verification for Code Translation" is focused on making code translation more robust by integrating verification directly into the decoding loop rather than treating it as an afterthought.

Lu: The real potential here, I think, is not just translation but applying this concept to any task with deterministic constraints—like generating specific structured data or writing specific configuration files. It’s a way to inject hard correctness checks into the generative process itself one <ref:2605.17626#pg1>.

Meng: That's an interesting thought for me. If we can use this structure-aware rollback idea for, say, ensuring a generated JSON file adheres to a strict schema, that seems like it could translate really well into other domains. It moves the verification from post-generation to real-time guidance during generation.

Tom: That’s the shift they are pushing for. And while they focus on code translation now, the implication is that we can start building systems where generation doesn't just *guess* what's right, but actively steers itself toward structures that satisfy known rules throughout the process one <ref:2605.17626#pg1>.

Jane: So to wrap up, DTV introduces a framework where generation and verification run in a tight loop guided by structural boundaries. It uses checks on things like statements and blocks to decide whether to continue generating or backtrack and fix the problem immediately.

Lu: And it does this by using feedback loops that update the model's understanding of what went wrong, rather than just discarding the failed attempt one <ref:2605.17626#pg1>.

Meng: The main thing is that it lowers the cost for legacy code translation because you get better results for less token usage per successful case compared to simply scaling up other methods one <ref:2605.17626#pg1>.

Tom: That’s a solid summary of where they landed. It’s definitely a method that shows how to handle the constraints of deterministic tasks more efficiently in generative AI models. We'll take a quick break and come back after we talk about how this applies broadly to other AI applications.

The paper's summary: Tom: So, we're looking at how this Decoding Time Verification framework works in plain English now that we know it’s about checking code while the AI is writing it.

Jane: Basically, they've built a deterministic state machine controller that forces the AI to check its work every time it hits a big structural point like a statement or a block.

Lu: Think of it like this: you can't just keep writing code in total darkness; you have to pause, check the last bit you wrote, and make sure it follows all the rules before moving on.

Meng: And if that check fails, they have this "rollback" feature where they don't just start over from scratch; they roll back to the last safe spot and fix only that small section.

Tom: That rollback is smart because it means they aren't wasting tokens on re-writing the whole program; it’s localized error correction.

Jane: And when those checks fail, they don't just stop there; they use feedback to take what went wrong and inject that into the AI's context so it tries again with new information.

Lu: So, this whole loop—generate, verify, commit or rollback—is designed specifically to stop errors from creeping through the generation process on Page three.

Tom: It’s about enforcing a legal prefix invariant; only code prefixes that pass all the verifiers get to be used as context for the next part of the AI’s work.

Jane: And they have this structure-aware rollback with escalation, which means if it fails repeatedly, it rolls back to the tiniest spot that caused the problem and then checks bigger scopes on subsequent failures.

Meng: That escalation helps prevent wasted tokens; you’re not just retrying small sections over and over again when the issue is deeper.

Tom: The big win they show is in C-to-Rust translation, where this method boosts the pass rate from seventy-two point three percent up to eighty-two point zero percent.

Jane: It also shows a better trade off on cost; they're using fewer tokens per case compared to just running a standard self-refine baseline.

Lu: The paper’s finding is that all these parts—the boundary checks, the rollback with escalation, and the feedback—all work together to give you that performance boost.

Tom: And they found in-loop recovery is actually providing the biggest benefit across those different components when they tested it.

Jane: It means this technique isn't just adding a layer; it's fundamentally changing how the AI handles deterministic tasks by integrating verification directly into its generation timing.

Lu: This moves the focus from just generating text to actively steering that generation process toward structures that are known to be correct according to specific rules.

Meng: It’s interesting because the paper mentions this gain is most effective for local fixes, meaning it’s really good at catching small errors right where they happen.

Tom: But the limitation they point out is that when the error is nonlocal, this method loses some of its effectiveness and you might need a different approach.

Jane: So, while it's fantastic for finding and fixing local mistakes efficiently, we still have to deal with those larger structural problems where the DTV framework struggles a bit.

Lu: That leads us into how they tackle those larger issues by using prompt augmentation to surface unresolved oracle failures back into the model context for better retries.

The paper's improvements: Tom: So, we've been talking about how DTV works now that we know it lets you check code while the AI is writing it, and now let’s see what they suggest next to make it even better.

Jane: They aren't just stopping at verification during generation; they propose a few ways to actually improve the system itself.

Lu: The first big suggestion is using a multi-oracle composition strategy, which means combining different types of checks for different languages or tasks.

Meng: For C-to-Rust, they suggest mixing compiler checks with differential testing against the output in the end; it captures both syntax and how the code actually behaves.

Tom: And for JavaScript to TypeScript, they focus on static type checking and linting verification to nail those type annotations specifically.

Jane: That’s a smart move because it lets you tailor the verification layer to what that specific language needs most.

Lu: Then they talk about feedback via prompt augmentation again, but this time, they emphasize summarizing those diagnostic results into a text block that gets injected back into the model context for retries.

Tom: So it’s not just feeding raw errors back in; it's giving the AI a summarized report on what went wrong so it can learn from the failure more effectively.

Jane: And they also introduced something called a fix-shape classifier, which tries to analyze the differences between failed and successful repairs.

Meng: It classifies fixes as "LOCAL," "NONLOCAL," or "MIXED." That’s useful because it tells the system whether a fix is likely to be rescued by in-loop verification.

Lu: The idea is that DTV’s incremental work tends to concentrate on those local error codes, which means the system learns exactly where to focus its effort for immediate gains.

Tom: That makes sense; if you can pinpoint the type of fix you’re looking for, the AI knows it has a better chance of getting it right in that single attempt.

Jane: They also look at how this plays out in terms of how much it changes things for someone just listening to the show.

Meng: For practical use, this means if you're working on code and you know the error is localized, DTV is going to be way more efficient than just hitting the standard self-refine baseline.

Tom: It’s about moving from a broad search for a solution to a targeted fix based on where the failure actually occurred during generation.

Jane: But they also acknowledge that this approach still has its limits, especially when dealing with errors that are nonlocal, because that's where it struggles most.

Lu: So the paper lays out a path forward: tailor the verifiers to the task and use structured feedback to guide retries on what went wrong.

Tom: It’s about making the verification process smarter, not just faster, which is a really interesting direction for this kind of research.

Conclusion: Tom: So we’ve been walking through the whole thing on "Verifiers in the Loop: Decoding Time Verification for Code Translation," and to wrap up, they’ve shown that this deterministic verification approach significantly boosts code translation pass rates without needing massive scaling efforts.

Jane: Exactly. The core idea is weaving checks into the generation process itself, which cuts down on errors propagating through the whole sequence.

Lu: It’s a neat way to handle those constraints in code translation by treating structural boundaries as explicit steps in the AI's workflow instead of just letting it run free.

Meng: From an engineering standpoint, the practical implication is that you can get higher quality results on legacy code using fewer tokens per case than when you rely on standard self-refinement methods.

Tom: That token efficiency is huge for deployment; it means faster processing and lower running costs for companies translating code.

Jane: And they’ve proven this method beats other scaling techniques in the cost-matched grid, sitting above the best of N and S star frontiers when you look at pass rate versus cost.

Lu: The paper suggests that the future for this is taking that verification concept and applying it to any task with strict rules, not just programming languages.

Tom: Right. So we’ve seen how DTV tackles code translation, but the bigger question is where else this deterministic control can be applied across different AI problems?

Jane: That’s the next big thing. We need to explore how this structure-aware rollback and feedback loop can be used for things like generating complex structured data or specific configuration files.

Meng: I think if we can get that level of reliable, real-time checking, it could really improve the stability of AI systems when they’re producing output for critical applications.

Lu: And Lalam sees this as a way to build more trustworthy and predictable generation capabilities into the very foundation of how large models operate.

Tom: So while "Verifiers in the Loop: Decoding Time Verification for Code Translation" is a specific result, the underlying technique seems like it could be a foundational piece for building more robust AI systems overall.

Jane: It definitely points toward a future where generative models aren't just guessing; they are actively steering themselves toward correctness during their generation time.

University of Illinois Urbana-Champaign · University of Wisconsin–Madison and Google

cs.LG, cs.SE

Submitted: 2026-05-17

Updated: 2026-10-13

Comments: Accepted at NeurIPS 2026. 37 pages, 9 figures

Code: https://github.com/qsdrqs/dtv-translation

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: The gist The Decoding Time Verification (DTV) framework introduces an interleaving decoding strategy that treats structural boundaries as meta steps for verifier-guided generation, effectively

Key concepts

DTV Framework: A decoding strategy that interleaves generation with verifier calls using a deterministic state machine. It systematically generates code up to structural boundaries, verifies it, commits checkpoints on success, and rolls back or escalates on failure. This structure guides the model's generation process based on verification results.
Structural-Boundary Verification: A mechanism that enforces a legal prefix invariant by checking generated prefixes against all applicable verifiers before they are used as context for further decoding. This ensures that only valid code segments satisfying all necessary checks are admitted into the autoregressive context, preventing incorrect sequences from being built.
Feedback via Prompt Augmentation: A technique where unresolved oracle failures are summarized and injected back into the model's prompt context. This allows subsequent generation steps to condition themselves on what went wrong during verification, effectively surfacing and addressing specific errors in a targeted manner.

Terminology

Verifier — 6× in this paper · explained in 2 episodes
Verification — 3× in this paper · explained in 2 episodes
Decoding Time Verification — 2× in this paper
Test-time Scaling — 1× in this paper · explained in 2 episodes

Summary

The gist The Decoding Time Verification (DTV) framework introduces an interleaving decoding strategy that treats structural boundaries as meta steps for verifier-guided generation, effectively improving code translation performance by preventing error propagation through autoregressive context on Page 1.

How it works

The DTV framework operates as a deterministic state-machine controller that interleaves generation with verifier calls under explicit control on Page 2. This process involves several key steps:

GENERATE extends the prefix to the next structural boundary, such as a statement or block on Page 2.
VERIFICATION renders the prefix and runs applicable oracles to check for correctness on Page 2.
COMMIT checkpoints the prefix as a rollback target when all checks pass on Page 2.
ROLLBACK restores a checkpoint, augmented with diagnostics, when verification fails on Page 2.
FEEDBACK augments generation by summarizing diagnostics and injecting them back into the model context for retries on Page 5.

Key Mechanisms

DTV combines three key mechanisms to achieve its goal of preventing verifier-detectable errors from propagating through the autoregressive context on Page 3. These mechanisms include:

structural-boundary verification enforces a legal prefix invariant, ensuring only prefixes satisfying all applicable verifiers are admitted as conditioning context for subsequent decoding on Page 4.
structure-aware rollback with escalation rolls back to the smallest scope implicated by the verifier diagnostic and escalates to larger scopes on repeated failures on Page 5.
feedback via prompt augmentation surfaces unresolved oracle failures back to the model so subsequent generation conditions on what went wrong on Page 5.

Experimental Evaluation

DTV was evaluated across two translation settings: C-to-Rust and JavaScript-to-TypeScript on Page 6. Using Qwen3-4B as the primary generator under matched token budgets, DTV improves pass rates from 72.3% to 82.0% on C-to-Rust and from 33.3% to 46.0% on JavaScript-to-TypeScript relative to matched self-refinement baselines on Page 10. DTV achieves a more favorable pass-rate-cost tradeoff than post-hoc verification or sampling-based scaling in the cost-matched grid on Page 3.

Performance and Analysis

The results show that DTV/self-refine Pareto-dominates na¨ıve/self-refine and sits on the upper-left of the cost-versus-pass-rate frontier on both tasks on Page 6. DTV is also more token-efficient than na¨ıve/self-refine while attaining a higher pass rate on Page 6. The ablation study shows that all three key design components of DTV contribute, with in-loop recovery providing the largest benefit on Page 7. DTV’s gain concentrates on cases with local fixes while weakens on nonlocal ones, where early intervention prevents derivative errors and reduces self-refine rounds on Page 9.

Conclusion

DTV/self-refine raises compile pass rate from 72.3% to 82.0% on C to Rust and from 33.3% to 46.0% on JavaScript-to-TypeScript using fewer tokens per case than na¨ıve/self-refine on Page 10, Pareto-dominating na¨ıve/self-refine and sitting above the best-of-N and S∗ frontiers at matched cost on Page 3. Gains transfer to Gemma4-E4B on three of four pairs on Page 6. Key challenges remain scaling to larger models, extending the approach beyond translation, and reducing the implementation effort for new models and tasks on Page 13. DTV is a decoding-time algorithm for code translation under deterministic verifiers; it does not train new models or release new generative capabilities on Page 14. Its most direct positive impact is lowering the cost of translating legacy code into safer target languages on Page 14. The most direct negative impact is the risk of misplaced confidence in oracle-passing translations on Page 13.

Improvements for AI systems

A decoding framework that interleaves generation with deterministic verifier calls under a state-machine controller can be implemented to prevent error propagation by ensuring generation continues only from committed verified prefixes. This allows for test-time scaling of code translation models, such as Qwen3-4B, to improve pass rates from 72.3% to 82.0% on C-to-Rust and 33.3% to 46.0% on JavaScript-to-TypeScript relative to matched self-refinement baselines while using fewer tokens per case.
The system can employ structure-aware rollback with escalation where the controller rolls back to the smallest scope implicated by the verifier diagnostic, escalating to larger scopes on repeated failures rather than retrying more. This mechanism prevents wasted tokens and ensures that verification signals actively influence generation trajectory by updating context via feedback via prompt augmentation.
The improved AI system can utilize a multi-oracle composition strategy, combining compiler checks with differential testing against the output in the end (for C-to-Rust) or static type and lint verification for the type annotation task (for JS-to-TypeScript). This enables a robust verification layer that captures both syntactic correctness and behavioral equivalence simultaneously.
The system can leverage feedback via prompt augmentation, where diagnostics are summarized into a text block, and this is injected back into the model context so that retries proceed under updated information rather than blind resampling, as described by the mechanism where S remains fixed while the conditioning context shifts to incorporate failures.
The system can integrate a fix-shape classifier based on analyzing edit differences between failed and successful repair attempts (LOCAL, NONLOCAL, MIXED), allowing it to learn which types of fixes are most likely to be rescued by in-loop verification, specifically noting that DTV’s incremental work is concentrated on the same error codes na¨ıve’s R1 hit rather than redirected to a different error class.

Abstract

Test-time scaling is an important mechanism for improving large language models, especially on tasks with deterministic verifiers. Code translation is a canonical example: the source program constrains valid outputs, while compilers, type checkers, and behavioral checks provide exact pass/fail feedback. Outcome-only approaches apply these verifiers only after generation, which is inefficient because early errors corrupt the autoregressive context and are rarely corrected later. We introduce Decoding Time Verification (DTV), a framework that treats structural boundaries as meta steps for verifier-guided decoding. DTV interleaves generation with verifier calls under a state-machine controller that commits verified prefixes, using structural-boundary checks and structure-aware rollback to prevent error propagation while reducing wasted tokens. We evaluate DTV on C-to-Rust and JavaScript-to-TypeScript translation. Using Qwen3-4B as the primary generator under matched token budgets, DTV improves pass rates from 72.3% to 82.0% on C-to-Rust and from 33.3% to 46.0% on JavaScript-to-TypeScript relative to matched self-refinement baselines, while using fewer tokens per case; the same trend largely transfers to Gemma-4-E4B and Qwen3-30B-A3B. In the evaluated cost-matched grid, DTV achieves a more favorable tradeoff between pass rate and generated-token cost than post-hoc verification or sampling-based scaling. These results show that verifier-guided decoding is an effective use of inference-time compute for code translation.