Done Is Not Correct: Measuring Silent Failures and Self-Verification Calibration When LLM Agents Take CAD Actions
cs.SE, cs.LG
Submitted: 2026-11-06
Updated: 2026-11-06
Code: https://github.com/rodriguescarson/cad-silent-failure-bench
Terminology
Sources
- ReAct: Synergizing Reasoning and Acting in Language Models
- Toolformer: Language Models Can Teach Themselves to Use Tools
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench
- Establishing Best Practices for Building Rigorous Agentic Benchmarks
- $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
- AI Agents That Matter
- An Illusion of Progress? Assessing the Current State of Web Agents
- Agentic Uncertainty Reveals Agentic Overconfidence
- Text2CAD: Generating Sequential CAD Models from Beginner-to-Expert Level Text Prompts
- Text-to-CadQuery: A New Paradigm for CAD Generation with Scalable Large Model Capabilities
- Text2CAD-Bench: A Benchmark for LLM-based Text-to-Parametric CAD Generation
- CAD-Assistant: Tool-Augmented VLLMs as Generic CAD Task Solvers
- Clarify Before You Draw: Proactive Agents for Robust Text-to-CAD Generation
- CADSmith: Multi-Agent CAD Generation with Programmatic Geometric Validation
- From Idea to CAD: A Language Model-Driven Multi-Agent System for Collaborative Design
- Large Language Models for Computer-Aided Design: A Survey
- CAD-Judge: Toward Efficient Morphological Grading and Verification for Text-to-CAD Generation
- Generating CAD Code with Vision-Language Models for 3D Designs
- A Contradiction Judge Counted Reported Teammate Errors as Hallucinations: An Assertion-Aware Re-Score of One Multi-Agent LLM Harness
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters