VeriContest: A Competitive-Programming Benchmark for Verifiable Code Generation
cs.SE, cs.AI, cs.LG
Submitted: 2026-05-08
Updated: 2026-11-07
Comments: Accepted to NeurIPS 2026 (Evaluations & Datasets Track)
Code: https://github.com/HIPREL-Group/VeriContest
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language models can generate useful code from natural language, but their outputs come without correctness guarantees.
Terminology
Abstract
Large language models can generate useful code from natural language, but their outputs come without correctness guarantees. Verifiable code generation offers a path beyond testing by requiring models to produce not only executable code, but also formal specifications and machine-checkable proofs. Progress in this direction, however, is difficult to measure: existing benchmarks are often small, cover only part of the pipeline, lack ground-truth proofs or rigorous specification validation, or target verification settings far from mainstream software development. We present VeriContest, a benchmark of 1,006 competitive-programming problems from LeetCode and Codeforces for verifiable code generation in Rust with Verus. Each problem includes a natural language description, expert-validated formal specifications, judge-accepted Rust code, Verus-checked proofs, and positive and negative test suites. VeriContest is constructed through a three-phase pipeline that scales from manually verified seed problems to semi-automated expansion with human-in-the-loop review. To strengthen benchmark quality, we also use testing to validate postcondition completeness. VeriContest supports both isolated and compositional evaluation of specification, code, and proof generation, and end-to-end verified program synthesis. Evaluating ten state-of-the-art models reveals a sharp gap between ordinary coding and verifiable code generation: the strongest model reaches 97.51% on natural-language-to-code generation, but only 57.16% on specification generation, 13.82% on proof generation, and 6.76% end-to-end. These results identify proof and specification generation as the central bottlenecks for current models and establish VeriContest as a rigorous platform for evaluating and training future systems that generate code with machine-checkable correctness.
Sources
- AI IDEs or Autonomous Agents? Measuring the Impact of Coding Agents on Software Development
- Program Synthesis with Large Language Models
- A benchmark for vericoding: formally verified program synthesis
- Can LLMs Generate Reliable Test Case Generators? A Study on Competition-Level Programming Problems
- Evaluating Large Language Models Trained on Code
- VerifyThisBench: Generating Code, Specifications, and Proofs All at Once
- HARDTESTGEN: A High-Quality RL Verifier Generation Pipeline for LLM Algorithmic Coding
- Measuring Coding Challenge Competence With APPS
- Large Language Models for Code Generation: A Comprehensive Survey of Challenges, Techniques, Evaluation, and Applications
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- miniCodeProps: a Minimal Benchmark for Proving Code Properties
- DafnyBench: A Benchmark for Formal Software Verification
- Beyond Postconditions: Can Large Language Models infer Formal Contracts for Automatic Software Verification?
- CodeHacker: Automated Test Case Generation for Detecting Vulnerabilities in Competitive Programming Solutions
- GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models
- CLEVER: A Curated Benchmark for Formally Verified Code Generation
- Can Large Language Models Write Good Property-Based Tests?
- CodeContests+: High-Quality Test Case Generation for Competitive Programming
- VeruSAGE-Bench: A Benchmark Suite for Rust System Verification
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters