Towards AI-Assisted Research Writing: Benchmarking LLMs for AI/ML Introduction Generation

arXiv:2508.14273 · cs.CL · Submitted 2025-08-19 · Read on arXiv

cs.CL

Submitted: 2025-08-19

Updated: 2026-08-31

Comments: 25 pages, 5 figures, 13 Tables

Code: https://github.com/danielnsilva/semanticscholar

License: http://creativecommons.org/licenses/by/4.0/

The gist: As researchers increasingly adopt LLMs as writing assistants, generating high-quality research paper introductions remains both challenging and essential.

Terminology

Abstract

As researchers increasingly adopt LLMs as writing assistants, generating high-quality research paper introductions remains both challenging and essential. We introduce Scientific Introduction Generation (SciIG), a task that evaluates LLMs' ability to produce coherent introductions from titles, abstracts, and related works. Curating new datasets from NAACL 2025 and ICLR 2025 papers, we assess five state-of-the-art models, including both open-source (DeepSeek-v3, Gemma-3-12B, LLaMA 4-Maverick, MistralAI Small 3.1) and closed-source GPT-4o systems, across multiple dimensions: lexical overlap, semantic similarity, content coverage, faithfulness, consistency, citation correctness, and narrative quality. Our comprehensive framework combines automated metrics with LLM-as-a-judge evaluations. Results demonstrate LLaMA-4 Maverick's superior performance on most metrics, particularly in semantic similarity and faithfulness. Moreover, three-shot prompting consistently outperforms fewer-shot approaches. These findings provide practical insights into developing effective research writing assistants and set realistic expectations for LLM-assisted academic writing. To foster re- producibility and future research, we publicly release all code and datasets.

Sources

Related papers