LLMs Do Not Grade Essays Like Humans
cs.AI, cs.CL
Submitted: 2026-03-24
Updated: 2026-03-24
Journal ref: Computers and Education: Artificial Intelligence, Volume 11, 2026, 100666
DOI: 10.1016/j.caeai.2026.100666
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Large language models have recently been proposed as tools for automated essay scoring, but their agreement with human grading remains unclear.
Terminology
Abstract
Large language models have recently been proposed as tools for automated essay scoring, but their agreement with human grading remains unclear. In this work, we evaluate how LLM-generated scores compare with human grades and analyze the grading behavior of several models from the GPT and Llama families in an out-of-the-box setting, without task-specific training. Our results show that agreement between LLM and human scores remains relatively weak and varies with essay characteristics. In particular, compared to human raters, LLMs tend to assign higher scores to short or underdeveloped essays, while assigning lower scores to longer essays that contain minor grammatical or spelling errors. We also find that the scores generated by LLMs are generally consistent with the feedback they generate: essays receiving more praise tend to receive higher scores, while essays receiving more criticism tend to receive lower scores. These results suggest that LLM-generated scores and feedback follow coherent patterns but rely on signals that differ from those used by human raters, resulting in limited alignment with human grading practices. Nevertheless, our work shows that LLMs produce feedback that is consistent with their grading and that they can be reliably used in supporting essay scoring.
Sources
- Assessing the Reliability and Validity of Large Language Models for Automated Assessment of Student Essays in Higher Education
- Are Large Language Models Good Essay Graders?
- Large Language Models (GPT) for automating feedback on programming assignments
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents