Verification-Gated Redraw and Repair for Local Small Language Models: A Paired Evaluation
cs.SE
Submitted: 2026-10-31
Updated: 2026-10-31
Comments: 21 pages, 3 figures. Includes appendices. Data and scientific reproduction code: https://github.com/fedir/arxiv-experiments-slm-recover
Code: https://github.com/fedir/arxiv-experiments-slm-recover
License: http://creativecommons.org/licenses/by/4.0/
The gist: Executable checks can reject incorrect model-generated programs, but rejection alone does not determine whether fresh sampling or diagnostic repair is preferable.
Terminology
Abstract
Executable checks can reject incorrect model-generated programs, but rejection alone does not determine whether fresh sampling or diagnostic repair is preferable. We present a controlled, resource-bounded conceptual replication and extension of established recovery policies: a single answer (N), one verification-triggered redraw (R), and one diagnostic repair (D), for Ornith 1.5 9B and Gemma 4 E4B served locally in nominal 4-bit MLX format. The frozen evaluation contains 72 tasks: 40 HumanEval+ problems, 16 SQL tasks and 16 shell tasks. Policies share initial responses and use only public checks for online decisions; every candidate is model-generated. Each task has one stochastic trajectory per model, so the study does not estimate variation across independent sampling blocks. Redraw increases observed final-test success from 51.4% to 59.7% for Ornith and from 55.6% to 62.5% for Gemma. Diagnostic repair reaches 52.8% and 63.9%, respectively. None of four internally prespecified paired contrasts is significant after Holm correction (minimum adjusted p = 0.125). Public checks falsely accept 25 Ornith and 11 Gemma candidates, demonstrating that acceptance is not a correctness guarantee. A post-review audit of initial answers gives perfect-retry upper bounds of 76.4% and 87.5%, respectively: accepted initial errors cannot trigger recovery under the fixed gate. This one-block, resource-bounded study provides a finite-cohort experimental record, not policy equivalence, a general model ranking or universal reliability
Sources
- How Many Tries Does It Take? Iterative Self-Repair in LLM Code Generation Across Model Scales and Benchmarks
- Evaluating Large Language Models Trained on Code
- Teaching Large Language Models to Self-Debug
- Large Language Models Cannot Self-Correct Reasoning Yet
- Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
- Self-Refine: Iterative Refinement with Self-Feedback
- Is Self-Repair a Silver Bullet for Code Generation?
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters