3D-DefectBench: A Controlled Factorial Study of Vision-Language Model Evaluation Pipelines for Fine-Grained 3D Generation Defects
This study created a benchmark, 3D-DefectBench, to rigorously test how different evaluation pipelines affect AI judge reliability for 3D generation defects. By systematically varying four pipeline factors (VLM, camera protocol, input visual, and prompt schema) across 84 designs and comparing results against human labels from two different sources (vendor vs. expert), the research shows that while the VLM is most important, how evidence is constructed and specified significantly influences judge performance.