How Many Tries Does It Take? Iterative Self-Repair in LLM Code Generation Across Model Scales and Benchmarks

arXiv:2604.10508 · cs.SE, cs.AI · Submitted 2026-04-12 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "How Many Tries Does It Take? Iterative Self-Repair in LLM Code Generation Across Model Scales and Benchmarks".

Jane: Here's the gist of the paper. Iterative self-repair across various LLM scales and architectures demonstrates that feeding execution errors back to models universally improves code generation pass rates The results show that self-repair universally improves pass rates: +4.9 to +17.1 pp on HumanEval and +16.0 to +30.0 pp on MBPP Universal Effectiveness of Self-Repair The study investigates iterative self-repair across seven models spanning three families and both openweight and proprietary providers This investigation addresses whether self-repair now works universally, even for smaller models The authors demonstrate that prompt-based self-repair is now effective across seven models from three families, even at 8B scale This finding updates the finding by Olausson et al.

5: that weaker models are harmed by self-repair, and contrasting with Chen et al.

12: who found limited gains for a 7B model The cross-benchmark consistency of our findings strengthens this conclusion Self-repair improves every model on both HumanEval and MBPP, with largely consistent model rankings across benchmarks.

Title and authors: Tom: So we're talking about this paper today, "How Many Tries Does It Take? Iterative Self-Repair in LLM Code Generation Across Model Scales and Benchmarks." It sounds a bit like it’s asking how many times you have to try before the AI finally gets the code right.

Jane: That’s a really relatable title. The authors, Johin Johny Arimbur, they've put this into practice across seven different models and both openweight and proprietary ones, which is pretty broad coverage for this kind of study.

Lu: What’s interesting about the scope here is that it isn't just looking at one type of model or one specific benchmark; they are testing across different families, like Llama and Gemini, so we get a better idea of where this self-repair ability holds up.

Meng: I'm curious if this means these gains are consistent, or if it only works for the bigger models that have more training data to draw from. We need to know how much lifting this actually costs in terms of compute.

Lalam: My internal analysis shows that the paper is focused on showing that feeding execution errors back into the model universally improves code generation pass rates across these different systems, and they find significant gains on HumanEval and MBPP Sanitized.

Tom: Right, so we’re looking at how much better things get when the AI gets a chance to fix its own mistakes instead of just failing once. It seems like a big step because most evaluations treat it like a single shot attempt, which is pretty limiting for real-world coding tasks.

Jane: The main point here is that iterative self-repair works universally, even for smaller models, and the results show improvements ranging from about four point nine to seventeen points on HumanEval and sixteen to thirty points on MBPP Sanitized depending on the model <ref:2604.10508#pg1>.

The paper's summary: Tom: So, what’s the core finding here is that feeding execution errors back into a model for correction universally improves code generation pass rates, and they measured this across seven models spanning three families and both openweight and proprietary providers.

Lu: They are showing that this works even for models at the eight billion parameter scale, which is important because previously some of the literature suggested that weaker models were actually harmed by self-repair mechanisms.

Jane: The study confirms that across all these different systems, including Llama three point one 8B and Gemini two point five Flash, self-repair improves the pass rates on both HumanEval and MBPP Sanitized, with fairly consistent rankings across the benchmarks as well <ref:2604.10508#pg1>.

Meng: If this is true for smaller models too, then the deployment strategy changes a lot because we can’t just rely on using the largest model available; we can use smaller ones if they have this feature enabled. But what about when things break at scale?

Lalam: The paper highlights that most of these gains concentrate in the first two repair rounds, suggesting a practical guideline for deployment, which is that you probably don't need to do too much iteration to see most of the benefit.

Tom: And they also provide some error type analysis, showing that assertion errors—which are logical mistakes—are actually the hardest ones to fix at about forty-five percent success rate, while things like name errors are repaired at a much higher rate of about seventy-seven percent <ref:2604.10508#pg1,name errors are repaired at>.

The paper's improvements: Jane: The authors suggest several ways we can improve this iterative process, and one big idea is to integrate error-type analysis into the repair routing mechanism so the system can prioritize fixing name errors because they are repaired at a much higher rate than assertion errors.

Lu: They also suggest using a chain-of-thought prompting strategy for the more capable models, specifically those with thirty-two billion parameters or larger, because they found that this yields about five point five percentage points of additional repair gain over just minimal prompting.

Meng: I’m thinking practically about deployment. The paper suggests adopting a two-round repair policy as the default because those two rounds capture the bulk, about seventy six to ninety five percent of all achievable gains, which makes sense for saving on computational overhead when you’re running things in production.

Tom: That makes sense from a cost perspective. And they also propose a hybrid strategy where you use self-repair for errors that can be diagnosed and resampling for assertion errors, which might be optimal for those weaker models where the logical mistakes are the toughest to fix sequentially.

Jane: They also discuss optimizing model selection based on your specific performance goals, suggesting favoring models like Gemini two point five Flash or Llama four Maverick if you need a high final pass rate, or perhaps a larger model like Gemini two point five Pro if you’re prioritizing the actual repair gain per token <ref:2604.10508#pg1>.

Lalam: And they also look at how to handle the output generation itself, with a pipeline to remove chain-of-thought traces and extract code from markdown fences, which helps clean up the responses before they are used by downstream systems.

Conclusion: Tom: To wrap this up, the paper "How Many Tries Does It Take? Iterative Self-Repair in LLM Code Generation Across Model Scales and Benchmarks" shows that iterative self-repair is now effective across all tested models and benchmarks, even for smaller ones.

Jane: The implication for us is that we shouldn't treat the first attempt as final; instead, we should build a pipeline where execution errors get fed back into the AI to correct them, and there are clear rules on when to stop iterating based on the diminishing returns curve they measured.

Lu: From a research standpoint, it confirms that error category is a strong predictor of repair success; knowing whether you’re dealing with an assertion error versus a name error lets you choose your strategy much more efficiently.

Meng: For me, the practical implication is that we can start designing systems that use two repair rounds as the default because it saves compute, and maybe only run more if those initial two rounds don't solve the problem completely.

Lalam: I’m confident in this direction; based on what I’ve seen across all literature, this iterative approach is a solid way to boost accuracy without needing massive increases in model size for every single task.

Tom: So, we’re looking at a framework that lets the AI learn from its own mistakes and scale that learning up, and the results show concrete improvements on metrics like HumanEval and MBPP Sanitized.

Jane: That’s what it boils down to; moving away from single-shot generation toward a system with built-in, intelligent self-correction loops. That gives us a lot to think about for how we design code generation tools in the future.

Johin Johny Arimbur

Meta · Google

cs.SE, cs.AI

Submitted: 2026-04-12

Updated: 2026-04-12

Comments: 11 pages, 7 figures, 8 tables

Code: https://github.com/Johin2/iterative-code-repair

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 88/100

The gist: Iterative self-repair across various LLM scales and architectures demonstrates that feeding execution errors back to models universally improves code generation pass rates The results show that…

Key concepts

Iterative Self-Repair
This process involves a model generating code, running it, identifying execution errors, and then feeding those specific errors back into the model as new input to correct its previous output. It is an automated feedback loop designed to make the model self-correct its mistakes.
Universal Effectiveness
The research shows that this self-repair technique works consistently across a wide variety of different LLMs, including smaller models (like 8B scale) and various architectures. This proves the technique is not limited to larger or more capable models.
Benchmark Improvement
The study measures success using standard coding benchmarks like HumanEval and MBPP. The results consistently show that implementing self-repair leads to measurable improvements in how often the generated code passes automated tests on these specific problems.

Terminology

Summary

Iterative self-repair across various LLM scales and architectures demonstrates that feeding execution errors back to models universally improves code generation pass rates The results show that self-repair universally improves pass rates: +4. p. 7

Where the paper says it

DISCUSSION A. Self-Repair as a Universal Improvement Strategy Our results demonstrate that iterative self-repair with error feedback is a universally effective strategy for improving LLM code generation across all seven models tested, spanning three families (Meta, Alibaba, Google), both open-weight and proprietary models, and both dense and MoE architectures.

9 to +17.1 pp on HumanEval and +16. p. 0
Where the paper says it

5 Flash, and Gemini 2. On HumanEval (164 problems) and MBPP Sanitized (257 problems) with up to five attempts, self-repair universally improves pass rates: +4.

0 to +30.0 pp on MBPP

Universal Effectiveness of Self-Repair

The study investigates iterative self-repair across seven models spanning three families and both openweight and proprietary providers This investigation addresses whether self-repair now works universally, even for smaller models The authors demonstrate that prompt-based self-repair is now effective across seven models from three families, even at 8B scale This finding updates the finding by Olausson et al. [5] that weaker models are harmed by self-repair, and contrasting with Chen et al. p. 0

Where the paper says it

Our contributions are as follows: 1) We demonstrate that prompt-based self-repair is now effective across seven models from three families, even at 8B scale, updating the finding by Olausson et al. [5] that weaker models are harmed by self-repair, and contrasting with Chen et al.

[12] who found limited gains for a 7B model The cross-benchmark consistency of our findings strengthens this conclusion Self-repair improves every model on both HumanEval and MBPP, with largely consistent model rankings across benchmarks. p. 7
Where the paper says it

The cross-benchmark consistency of our findings strengthens this conclusion. Self-repair improves every model on both HumanEval and MBPP, with largely consistent model rankings across benchmarks.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Implement iterative self-repair as a standard pipeline for code generation, allowing models to replicate this workflow within an LLM pipeline by feeding execution errors back for correction. This enables systems to achieve universal improvement across all tested models and benchmarks, including gains up to +30.0 pp on MBPP for Gemini 2.5 Flash.

  2. Integrate error-type analysis into the repair routing mechanism, specifically prioritizing name errors because Name errors are repaired at high rates (∼77%), while assertion errors are the hardest to repair at ∼45%). This allows the system to select a more efficient strategy, such as sampling multiple independent solutions when assertion-dominated failures occur.

  3. Employ a chain-of-thought prompting strategy for capable models (≥32B) during repair rounds, as CoT prompting yields +5.5 pp additional self-repair gain over minimal prompting for the 70B model. This provides an explicit step where the model analyzes what does the error tell us and formulates a root cause before generating a fix.

  4. Adopt a two-round repair policy as the default for cost efficiency, since two repair rounds capture the majority (76–95%) of achievable gains. This reduces computational overhead while still capturing the bulk of the benefit, which is crucial for resource-constrained applications.

  5. Develop a hybrid strategy that uses self-repair for diagnosable errors and resampling for assertion errors, as this approach may be optimal for weaker models where assertion errors are the hardest to repair. This balances the efficiency of sequential repair with the diversity of independent samples.

  6. Optimize model selection based on performance goals, favoring Gemini 2.5 Flash (96.3% HumanEval) or Llama 4 Maverick (93.9% HumanEval) for final pass rate requirements and Gemini 2.5 Pro or Llama 3.3 70B for the best repair gain per token.

Abstract

Large language models frequently fail to produce correct code on their first attempt, yet most benchmarks evaluate them in a single-shot setting. We investigate iterative self-repair (feeding execution errors back to the model for correction) across seven models spanning three families and both open-weight and proprietary providers: Llama 3.1 8B, Llama 3.3 70B, Llama 4 Scout (MoE, 16 experts), Llama 4 Maverick (MoE, 128 experts), Qwen3 32B, Gemini 2.5 Flash, and Gemini 2.5 Pro. On HumanEval (164 problems) and MBPP Sanitized (257 problems) with up to five attempts, self-repair universally improves pass rates: +4.9 to +17.1 pp on HumanEval and +16.0 to +30.0 pp on MBPP. Gemini 2.5 Flash achieves the highest final pass rates (96.3% HumanEval, 93.8% MBPP). Most gains concentrate in the first two rounds.Error-type analysis shows assertion errors (logical mistakes) are the hardest to repair at 45%, while syntax and name errors are repaired at substantially higher rates, connecting to broader findings on the limits of LLM self-correction. Prior work found that weaker models fail at self-repair or require fine-tuning; we show that modern instruction-tuned models succeed with prompting alone, even at 8B scale. We also provide the first comparison of dense and MoE architectures for self-repair, and extend the repair-vs-resampling tradeoff analysis to modern models. A prompt ablation reveals chain-of-thought repair yields up to +5.5 pp additional self-repair gain (measured as improvement in repair delta) over minimal prompting for capable models.

Sources

Related papers