FORALL-LEAN-AGENT for Auditable Reasoning in Formal Mathematics and Software Verification
cs.SE, cs.AI, cs.LO
Submitted: 2026-10-01
Updated: 2026-10-01
Comments: Accepted to NeurIPS 2026 VeriCodeGen
Code: https://github.com/SkidanovAlex/putnambench-deepseek
Project page: https://trishullab.github.io/PutnamBench/leaderboard.html
License: http://creativecommons.org/licenses/by/4.0/
The gist: Coding agents increasingly automate Lean proof development, but successful compilation alone does not establish that a candidate proves the intended statement under acceptable assumptions.
Terminology
Abstract
Coding agents increasingly automate Lean proof development, but successful compilation alone does not establish that a candidate proves the intended statement under acceptable assumptions. We present FORALL-LEAN-AGENT, a frontend-agnostic framework for auditable reasoning in formal mathematics and software verification. The framework combines isolated workspaces, Lean tools, and fresh review with statement comparison, axiom audits, and independent proof checking where supported. Verification evidence and reviewer decisions are bound to the same candidate artifact, making acceptance traceable. We evaluate the framework on VeriSoftBench, PutnamBench, and both problems in the Lean Eval softwareverification track. On the 100-task VeriSoftBench subset, integration with FORALLLEAN-AGENT raises benchmark-rule success from 93 to 100 for GPT-5.6 Sol at low effort while reducing cost from 69 to 62. The PutnamBench evaluation accepts all 672 problems at an average of 4.72 each. These results show that agent harness design can improve correctness and efficiency while providing evidence beyond aggregate solve counts.
Sources
- Aristotle: IMO-level Automated Theorem Proving
- Pantograph: A Machine-to-Machine Interaction Interface for Advanced Theorem Proving, High Level Reasoning, and Data Extraction in Lean 4
- Ax-Prover: A Deep Reasoning Agentic Framework for Theorem Proving in Mathematics and Quantum Physics
- A benchmark for vericoding: formally verified program synthesis
- Seed-Prover 1.5: Mastering Undergraduate-Level Theorem Proving via Learning from Experience
- Goedel-Architect: Streamlining Formal Theorem Proving with Blueprint Generation and Refinement
- miniCTX: Neural Theorem Proving with (Long-)Contexts
- Harnessing Code Agents for Automatic Software Verification
- Goedel-Code-Prover: Hierarchical Proof Search for Open State-of-the-Art Code Verification
- Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correction
- Numina-Lean-Agent: An Open and General Agentic Reasoning System for Formal Mathematics
- A Minimal Agent for Automated Theorem Proving
- Agentic Proving for Program Verification
- An In-Context Learning Agent for Formal Theorem-Proving
- PutnamBench: Evaluating Neural Theorem-Provers on the Putnam Mathematical Competition
- Hilbert: Recursively Building Formal Proofs with Informal Reasoning
- VeriSoftBench: Repository-Scale Formal Verification Benchmarks for Lean
Related papers
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces
- PackMonitor: Enabling Zero Package Hallucinations Through Decoding-Time Monitoring
- IntentCoding: Amplifying User Intent in Code Generation
- Incentives and Outcomes in Bug Bounties