A Grounded and Decomposed Framework for Relation-Level Hallucination Evaluation in Abstractive Summarization
Praveen Kumar Katwe, Rakesh Chandra Balabantaray, Kali Prasad Vittala, Naman Kabadi
cs.CL, cs.AI
Submitted: 2026-08-08
Updated: 2026-08-11
Comments: 6 pages, 4 figures, 6 tables
License: http://creativecommons.org/publicdomain/zero/1.0/
The gist: Abstractive text summarization systems frequently generate fluent yet unfaithful summaries by fabricating or distorting relationships between entities and events.
Terminology
Abstract
Abstractive text summarization systems frequently generate fluent yet unfaithful summaries by fabricating or distorting relationships between entities and events. Such relation-level hallucinations undermine the reliability of generated summaries, particularly in high-stakes domains. In this work, we present a refined and grounded framework for evaluating relation hallucination in abstractive summarization. We present the empirical Relation Hallucination Index (RHI) by introducing a dependency-aware relation extraction algorithm that incorporates lemmatization-based normalization, named entity grounded subject resolution, passive agent recovery, negation-aware verb modeling, reporting verb filtering, nominal relation fallback, clausal propagation, and systematic deduplication. These enhancements improve the structural fidelity of extracted relation triples and reduce spurious matches during evaluation. In addition, we introduce a normalized formulation of RHI to ensure scale-invariant comparison between datasets and models. The revised metric decomposes hallucination into interpretable components, aggregates relation hallucination metric into a normalized relation faithfulness score. Extensive evaluation across multiple state-of-the-art summarization models demonstrates that the grounded extraction process yields more stable and discriminative hallucination measurements. The proposed framework advances automated relation-level faithfulness evaluation and supports coherence-aware, hallucination-sensitive model analysis.
Sources
- A Survey of Automatic Hallucination Evaluation on Natural Language Generation
- Stanza: A Python Natural Language Processing Toolkit for Many Human Languages
- Precisely the Point: Adversarial Augmentations for Faithful and Informative Text Generation
- KGPT: Knowledge-Grounded Pre-Training for Data-to-Text Generation
- Text2KGBench: A Benchmark for Ontology-Driven Knowledge Graph Generation from Text
- SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
- About Evaluation of F1 Score for RECENT Relation Extraction System
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering