Evaluating open-weight LLMs for scoring L2 essays: Accuracy, prompt transfer, and deployment of AiAWE
cs.CY
Submitted: 2026-06-11
Updated: 2026-10-28
Comments: 27 pages with 14 tables and 1 figure
DOI: 10.1016/j.rmal.2026.100390
Code: https://github.com/hiyouga/LLaMA-Factory
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: This study presents AiAWE, an open-source automated writing evaluation system that scores argumentative essays with a LoRA-adapted large language model deployed on local hardware.
Terminology
Abstract
This study presents AiAWE, an open-source automated writing evaluation system that scores argumentative essays with a LoRA-adapted large language model deployed on local hardware. Using 480 TOEFL Independent Writing essays from a restricted-access Educational Testing Service (ETS) dataset, we fine-tuned Gemma-3-27B and LLaMA-3.3-70B under identical LoRA configurations on 120 essays from one prompt, with all hyperparameters fixed in advance, and evaluated them on the remaining 360 essays. Gemma achieved a root mean square error (RMSE) of 0.474 and a quadratic weighted kappa (QWK) of 0.828 on the half-point grid against the ETS scores. In paired comparisons, it significantly outperformed a fine-tuned GPT-3.5 baseline from prior work on every metric tested, showing that a locally deployable open-weight model can achieve competitive scoring performance in this setting. It also had significantly lower error than the larger LLaMA configuration, although, because the two models differ in family as well as size, this comparison does not isolate the effect of scale. Neither open-weight model showed a statistically detectable increase in error on the 240 essays written to a second, unseen prompt from the same task family. On a single consumer-grade GPU, the scoring model returned a complete response in a median of 14.2 s, demonstrating technical feasibility under the tested conditions. Only the scoring component was evaluated, not the generated feedback. The system is publicly accessible at https://app.awade.gec.waseda.ac.jp, and its code, LoRA adapters, and training configurations are openly released.
Sources
- Automated Text Scoring in the Age of Generative AI for the GPU-poor
- LoRA: Low-Rank Adaptation of Large Language Models
- Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adaptation
- How Much is Too Much? Exploring LoRA Rank Trade-offs for Retaining Knowledge and Domain Robustness
- Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct
Related papers
- AI-Mediated Communication Can Steer Collective Opinion
- Epistemic Infrastructures of Science in AI Era Should Rebalance Costs of Generation and Verification
- AI out-persuades expert humans in text-based dialogue
- Evaluating Ambient Clinical Scribes in India: The Need for Multilingual Real-World Clinical Conversation Data
- AI Value Alignment for Evolving Social Norms
- Voting Biases in Decentralized Autonomous Organization (DAO) Governance