The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Models
This paper argues that predictive machine learning benchmarks are measurement tools, not just scores. To make scientific claims about these scores valid across fields like vision or sociology, researchers must explicitly define and test five types of construct validity: internal, external, content, consequential, and auxiliary. This framework ensures benchmark results support meaningful inferences rather than just descriptive statistics.