Generating Edit-Inducing Questions for AI Research Manuscripts

arXiv:2609.36617 · cs.CL · Submitted 2026-09-29 · Read on arXiv

cs.CL

Submitted: 2026-09-29

Updated: 2026-10-09

Comments: Accepted at the DocInsights Workshop @ EMNLP 2026

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

The gist: We study the ability of LLMs to generate edit-inducing questions whose answer will improve a paper draft.

Terminology

Abstract

We study the ability of LLMs to generate edit-inducing questions whose answer will improve a paper draft. On a dataset of paired submission and camera-ready papers from ICLR and NeurIPS, we compare the helpfulness of questions from GPT models with or without full paper context to that of human reviewers. GPT produces more edit-inducing questions and its questions are associated with more extensive edits and cover a broader range of edited content compared to questions from reviewers. However, a much smaller percentage of the GPT questions are edit-inducing. Our analyses confirm that automated questions can be beneficial to authors and highlight an example task where proper attending to long context deteriorates reasoning model ability to produce helpful output.

Sources

Related papers