Parametric CAD Bench v3: Evaluating Agentic Parametric CAD Authoring from Text and Engineering Drawings
cs.AI, cs.CV, cs.GR, cs.LG
Submitted: 2026-11-02
Updated: 2026-11-02
Comments: 31 pages, 12 figures, 11 tables. Benchmark: https://cadbench.ai/ ; tasks and leaderboard: https://hub.harborframework.com/datasets/gnucleus-ai/cad-bench/v3 ; scoring code: https://github.com/gNucleus-AI/freecad-validator
Code: https://github.com/gNucleus-AI/freecad-validator
Project page: https://harborframework.com
License: http://creativecommons.org/licenses/by/4.0/
The gist: Producing an executable CAD script is not sufficient to establish that an agent has satisfied an engineering design request.
Abstract
Producing an executable CAD script is not sufficient to establish that an agent has satisfied an engineering design request. Evaluation must also consider the resulting geometry, specified dimensions, and native CAD modeling history. We present Parametric CAD Bench v3, a 100-task benchmark comprising 30 text-to-CAD creation tasks, 30 text-to-CAD create-and-edit tasks, and 40 tasks that reconstruct mechanical parts from engineering drawings. Agents operate in a pinned FreeCAD environment and submit native CAD artifacts to a separate programmatic verifier. The primary metric is mean continuous task reward, combining geometry similarity and specification consistency rather than binary task success. A September 24, 2026 snapshot covers 13 model-agent systems; Claude Code with Claude Opus 5.5 achieves the highest overall point estimate, 61.03%, and an image-to-CAD mean reward of 84.66%. These measurements describe complete systems, not isolated model capabilities. We release versioned tasks, evaluation configurations, and inspectable result artifacts.
Sources
- CADBench: A Multimodal Benchmark for AI-Assisted CAD Program Generation
- 3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code
- Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
- BenchCAD: A Comprehensive, Industry-Standard Benchmark for Programmatic CAD
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents