Evaluating the Evaluator: Summarization Metrics and LLM-Judges beyond English
cs.CL, cs.AI
Submitted: 2025-03-21
Updated: 2026-09-02
Comments: SEPLN 2026
Code: https://github.com/hitz-zentroa/summarization
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Automatic text summarization relies on automatic evaluation to quickly determine the quality of summarization models via automatic metrics and LLM-as-a-Judge models.
Terminology
Abstract
Automatic text summarization relies on automatic evaluation to quickly determine the quality of summarization models via automatic metrics and LLM-as-a-Judge models. However, these techniques require meta-evaluation to ensure that they capture human judgments correctly. In this paper, we explore this meta-evaluation beyond English by generating a new multilingual summary meta-evaluation dataset (BASSE), which comprises human judgments on 2,040 abstractive summaries, generated either manually or by five Large Language Models (LLMs) with four different prompts. For each summary, annotators evaluate five criteria on a 5-point Likert scale: coherence, consistency, fluency, relevance, and 5W1H. We then benchmark automatic summarization metrics and LLM-as-a-Judge models. Our results show that currently proprietary judge LLMs have the highest correlation with human judgments, followed by criteria-specific automatic metrics, while open-sourced judge LLMs perform poorly.
Sources
- GPT-4 Technical Report
- Atla Selene Mini: A General Purpose Evaluation Model
- The Llama 3 Herd of Models
- A Survey on LLM-as-a-Judge
- An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4
- Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning
- Qwen2.5 Technical Report
- Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models
- Language Models are Multilingual Chain-of-Thought Reasoners
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering