Formal Conjectures: An Open and Evolving Benchmark for Verified Discovery in Mathematics
cs.AI
Submitted: 2026-05-13
Updated: 2026-11-11
Comments: 24 pages, 4 figures, 5 tables, Accepted at NeurIPS 2026 Datasets and Benchmarks Track
Code: https://github.com/google-deepmind/formal-conjectures
Project page: https://google-deepmind.github.io/formal-conjectures/20
License: http://creativecommons.org/licenses/by/4.0/
The gist: As automated reasoning systems advance rapidly, there is a growing need for research-level formal mathematical problems to accurately evaluate their capabilities.
Terminology
Abstract
As automated reasoning systems advance rapidly, there is a growing need for research-level formal mathematical problems to accurately evaluate their capabilities. To address this, we present Formal Conjectures (https://github.com/google-deepmind/formal-conjectures), an evolving benchmark of currently 7559 mathematical problem statements formalized in Lean 4. Sourced from areas of active mathematical research, the dataset features 1642 open research conjectures without known solutions for mathematical proof discovery, 2923 solved problems for proof auto-formalization, and 2994 supporting statements. Notably, the repository provides a structured interface connecting mathematicians who formalize and clarify problems with the AI systems and humans attempting to solve them. Demonstrating its immediate utility, the benchmark has already been leveraged for new mathematical discoveries, including the resolution of open research conjectures, and for the formal confirmation of major newly announced results. We describe our approach to ensuring the correctness of these formalizations in a collaborative open-source project where contributions stem from an active community. In this framework, AI-generated proofs and disproofs serve as a valuable auditing mechanism to iteratively improve the fidelity of the benchmark. Finally, we provide a standardized evaluation setup and report baseline results on frozen evaluation subsets, demonstrating a climbable signal that measures the current frontier of automated reasoning on research-level mathematics.
Sources
- Aristotle: IMO-level Automated Theorem Proving
- Forbidden Sidon subsets of perfect difference sets, featuring a human-assisted proof
- MathArena: Evaluating LLMs on Uncontaminated Math Competitions
- Humanity's Last Exam
- Seed-Prover 1.5: Mastering Undergraduate-Level Theorem Proving via Learning from Experience
- Seed-Prover: Deep and Broad Reasoning for Automated Theorem Proving
- ARC Prize 2024: Technical Report
- ARC Prize 2025: Technical Report
- FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI
- FATE: A Formal Benchmark Series for Frontier Algebra of Multiple Difficulty Levels
- Unsolved Problems in Group Theory. The Kourovka Notebook
- SorryDB: Can AI Provers Complete Real-World Lean Theorems?
- Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correction
- Numina-Lean-Agent: An Open and General Agentic Reasoning System for Formal Mathematics
- PutnamBench: Evaluating Neural Theorem-Provers on the Putnam Mathematical Competition
- FormalMATH: Benchmarking Formal Mathematical Reasoning of Large Language Models
- MiniF2F: a cross-system benchmark for formal Olympiad-level mathematics
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents