Towards Functional Correctness of Code Models with Selective Generation
cs.SE, cs.LG
Submitted: 2025-05-19
Updated: 2026-10-29
Comments: ICML 2026
Code: https://github.com/trustml-lab/s
License: http://creativecommons.org/licenses/by/4.0/
The gist: The hallucination of code generation models hinders their applicability to systems requiring higher safety standards.
Terminology
Abstract
The hallucination of code generation models hinders their applicability to systems requiring higher safety standards. One critical bottleneck in addressing code hallucination is the difficulty of identifying the functional correctness of generated code, due to its unnatural form. We address this core bottleneck by automatically generating unit tests using dynamic code analysis tools, leveraging the executable nature of code. Accordingly, we propose a selective code generator that abstains from uncertain generations -- based on the functional correctness evaluated by generated unit tests -- to theoretically control the correctness among non-abstained answers, the false discovery rate. Finally, we propose to use generated unit tests in evaluation as well as in learning for precise code evaluation, calling this paradigm FuzzEval. We demonstrate the efficacy of our method along with the controllability of code hallucination and reasonable selection efficiency.
Sources
- MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation
- Evaluating Large Language Models Trained on Code
- GPT-4 Technical Report
- Competitive Programming with Large Reasoning Models
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters