Dimensionality and Measurement Precision in HLE's Multiple-Choice Subset
The study analyzed 29 language models' responses to HLE's text-only multiple-choice subset using an IRT model. Results show HLE measures one general reasoning factor, not eight distinct domains. Measurement precision is highest for models at moderate ability levels, and the benchmark struggles to discriminate among the strongest models in certain areas.