On Challenges in Evaluation Methodology: A Review
This scoping review synthesizes four decades of critiques regarding evaluation methods for language models. It develops a taxonomy of 22 recurring concerns across data, metrics, hypotheses, and reporting. The resulting framework provides evaluators with a structured lens to understand historical debates and identify critical areas for future research.