Autoscoring Anticlimax: A Meta-analytic Understanding of AI's Short-answer Shortcomings and Wording Weaknesses
A meta-analysis of LLM short-answer scoring shows progress is stalled because autoregressive training favors surface form over construct validity. Findings reveal that human difficulty does not affect LLM performance, and models are highly sensitive to tokenization and wording. The research urges a shift from prompt engineering to designing robust systems for educational assessment.