Isomorphism Is Not Enough: An Agentic Audit of Verified Code Generation in Lean
cs.AI, cs.LO, cs.PL, cs.SE
Submitted: 2026-05-22
Updated: 2026-10-30
Code: https://github.com/trishullab/clever
Project page: https://rocq-prover.org
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: Agentic systems have recently emerged as state-of-the-art approaches for automated theorem proving in formal mathematics.
Terminology
Abstract
Agentic systems have recently emerged as state-of-the-art approaches for automated theorem proving in formal mathematics. To assess how far these capabilities extend to program verification and how reliably existing benchmarks can measure them, we audit CLEVER, a Lean 4 benchmark for verifiable code generation, using three independent agentic stacks based on Claude Opus 4.6, GPT-5.4, and Kimi K2.5. Our audit finds arguably erroneous reference specifications in 80 of 161 entries, each accepted by the benchmark's authors, distills a taxonomy of benchmark errors, and exposes structural limitations of isomorphism-based scoring, whose signal breaks down when generated and reference specifications formalize different yet defensible readings. In the process, the stacks generate arguably valid specifications for up to 98.8% of entries, certify implementations against sound reference specifications for up to 92.3%, and reach up to 96.2% end-to-end success over entries with sound premises, suggesting that CLEVER no longer stresses modern agentic provers as much as its design intended. Looking ahead, we translate our findings into recommendations for future work, with our error analysis doubling as a repair list for CLEVER and a heuristic checklist for the next generation of benchmarks.
Sources
- Aristotle: IMO-level Automated Theorem Proving
- A Case Study on the Effectiveness of LLMs in Verification with Proof Assistants
- Evaluating Large Language Models Trained on Code
- Seed-Prover: Deep and Broad Reasoning for Automated Theorem Proving
- Certified Program Synthesis with a Multi-Modal Verifier
- Winning Gold at IMO 2025 with a Model-Agnostic Verification-and-Refinement Pipeline
- Intent Formalization: A Grand Challenge for Reliable Coding in the Age of AI Agents
- Goedel-Code-Prover: Hierarchical Proof Search for Open State-of-the-Art Code Verification
- FVEL: Interactive Formal Verification Environment with Large Language Models via Theorem Proving
- Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correction
- Numina-Lean-Agent: An Open and General Agentic Reasoning System for Formal Mathematics
- Canonical for Automated Theorem Proving in Lean
- DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decomposition
- An In-Context Learning Agent for Formal Theorem-Proving
- CLEVER: A Curated Benchmark for Formally Verified Code Generation
- MiniF2F in Rocq: Automatic Translation Between Proof Assistants -- A Case Study
- Kimina-Prover Preview: Towards Large Formal Reasoning Models with Reinforcement Learning
- A Minimalist Proof Language for Neural Theorem Proving over Isabelle/HOL
- VERINA: Benchmarking Verifiable Code Generation
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents