Enhancing Knowledge Entity Relation Extraction in Abstracts and Full-text Content of Academic Articles Using Large Language Models
cs.CL, cs.DL, cs.IR
Submitted: 2026-10-30
Updated: 2026-11-02
Journal ref: Scientometrics, 2026
DOI: 10.1007/s11192-026-05892-8
Code: https://github.com/YingyiZhang/Knowledge-Entity-Relation-Extraction-with-LLM
License: http://creativecommons.org/licenses/by/4.0/
The gist: This study investigates the effectiveness of large pre-trained language models (LLMs) for knowledge entity relation extraction (KERE) from academic papers.
Terminology
Abstract
This study investigates the effectiveness of large pre-trained language models (LLMs) for knowledge entity relation extraction (KERE) from academic papers. Existing studies have examined individual types of LLM-based strategies and evaluated model performance using metrics such as precision and recall. To provide a broader evaluation, this study investigates multiple strategies, including closed-source LLM-based CoT prompting, LoRA-adapted open-source LLMs, LLM-enhanced small pre-trained language models (SLMs), and SLM-enhanced closed-source LLMs. In addition, this study introduces two strategies, namely SLM-enhanced open-source LLMs and closed-source LLM-enhanced open-source LLMs. These strategies are evaluated from multiple perspectives, including metrics, their ability to mitigate errors observed in SLMs, and potential directions for model optimization. This study manually constructs two annotated datasets in the NLP domain, covering Abstract and Full-Text datasets. Experimental results show that the LoRA-adapted T5 model achieves the highest macro F1 scores of 0.7408 and 0.7013 on the Abstract and Full-Text datasets, respectively. SLMs also benefit from LLM-generated entity descriptions under the LLM-enhanced SLM strategy. Error analysis further reveals that the effectiveness of LLM-based strategies in mitigating SLM errors varies across error types. Based on the identified limitations and error patterns of these strategies, several optimization directions are proposed. In addition, combining strategies with different error patterns further improves KERE performance.
Sources
- Longformer: The Long-Document Transformer
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Relation Extraction with Fine-Tuned Large Language Models in Retrieval Augmented Generation Frameworks
- ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools
- Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
- Qwen2.5 Technical Report
- Small Language Models as Effective Guides for Large Language Models in Chinese Relation Extraction
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- Baichuan 2: Open Large-scale Language Models
- An evaluation of DeepSeek Models in Biomedical Natural Language Processing
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data
- BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigation
- Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking