GUEST: Preliminary Guidelines for Using and Evaluating GenAI Tools to Support Systematic Literature Reviews

arXiv:2607.24991 · cs.SE, cs.AI · Submitted 2026-07-27 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "GUEST: Preliminary Guidelines for Using and Evaluating GenAI Tools to Support Systematic Literature Reviews".

Jane: Here's the gist of the paper. To support researchers intending to conduct systematic literature reviews (SLRs) using Generative AI (GenAI) or those conducting empirical studies evaluating how well GenAI supports SLR tasks.

Tom: The paper discusses the problems researchers face when evaluating GenAI for SLRs, identifies and explains process issues to consider when planning, conducting, and reporting both SLRs using GenAI and evaluations of GenAI tools.

Jane: Finally, it summarizes its results as a set of process recommendations named GUEST (GenAI Use and Evaluation in SLR Tasks).

Tom: The authors argue that GenAI requires human oversight and is not currently capable of unsupervised systematic studies.

Paper summary: Tom: So what's the core problem they’re identifying? It seems like the biggest issue isn't just that the AI makes mistakes sometimes, but that it creates a very unstable environment for evaluation. They point out these two levels of change, which is something we see constantly in AI development.

Jane: That’s right. They say there’s the major model updates, like moving from GPT-four to GPT-4o or whatever comes next, which is trackable because they announce version numbers and documentation <ref:2607.24991#pg1>. But then there's the infrastructure level, which is much more problematic for reproducibility.

Lu: Exactly. That infrastructure level includes things like safety filters and content policies, system prompts and guardrails that can change daily without public notice, input preprocessing, output filtering, rate limiting abuse detection, and even general UI wrapper changes that can happen hourly without warning. This lack of notice leads to serious reproducibility problems.

Meng: That sounds like a nightmare for any quality control process. If the rules governing how the AI processes your data change overnight because of an unannounced prompt update or a new filter kicking in, you can’t trust your results from yesterday, right?

Lalam: The paper highlights that these Level two changes are often undocumented, which is what they call reproducibility problems, and it makes it really hard to evaluate the tool consistently <ref:2607.24991#pg1>.

Tom: And they tie this instability directly into the methodological issues they found in existing work. They classify these problems into four categories covering adjustments to SLR practice, general secondary study standards, specific GenAI evaluation practices, and core empirical rigour.

Jane: So it’s not just a technical glitch; it’s a breakdown in the process itself when we try to apply AI to systematic reviews. They are trying to define how researchers should adjust their actual SLR tasks based on what these tools are capable of.

Lu: They also bring in some historical context, mentioning things like the Mean Magnitude Relative Error or MMRE metric from earlier work that showed how biased some performance metrics can be, and they apply that cautionary lens to GenAI tool performance.

Meng: It’s interesting they brought up those older metrics because it shows that the risks aren't new; we’re just applying old scrutiny to a new technology.

Lalam: And what they really want is for researchers to be explicit about their choice of tools, the version numbers, and the prompts they use so that others can actually understand what you did.

Conclusion: Tom: So wrapping up this paper, "GUEST: Preliminary Guidelines for Using and Evaluating GenAI Tools to Support Systematic Literature Reviews" is about establishing a baseline for how we talk about using AI in this specific research area. The authors argue that right now, Generative AI cannot be trusted to perform unsupervised systematic reviews on its own.

Jane: That’s the big message: human oversight is absolutely required. Even if an AI tool can give you a summary of the literature, that summary isn't the same as a systematic review because it doesn't have all the associated documentation needed for someone else to replicate your exact process.

Lu: They emphasize that this requirement for human supervision won't disappear even if we develop GenAI tools that are theoretically capable of delivering a complete SLR with all its associated documentation. The paper stresses that the process matters more than just having the tool itself.

Meng: From a deployment standpoint, this means we can’t just throw an AI script at a large dataset and expect it to run autonomously without someone checking every step and validating the output against our established metrics.

Lalam: The GUEST framework they propose is designed to be this comprehensive set of fifty-one recommendations meant to guide researchers through planning, executing, and reporting these GenAI-assisted studies in a trustworthy way. It’s meant to be a baseline for rigorous evaluation.

Tom: So the implication for the wider research community is that if you want to use AI in an SLR, you have to follow this structure—just justify your tool choice and report your version details clearly. It shifts the focus from "can this AI do it?" to "how rigorously are we managing this AI process?"

Jane: It’s a necessary shift. They aren't saying give up on using GenAI for tasks like repetitive screening or secondary validation, but they are saying we need to standardize *how* we use these tools so that the results remain trustworthy in software engineering research.

Keele University · Universidad de la Repíblica Universidad de la Repíblica · Wroclaw University of Science and Technology, Poland · University of Calgary Canada · Brunel University of London United Kingdom Durham University United Kingdom

cs.SE, cs.AI

Submitted: 2026-07-27

Updated: 2026-10-29

Comments: v2: revised version. 49 pages, 2 figures, 15 tables

Project page: https://icasr.github.io

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 91/100

The gist: To support researchers intending to conduct systematic literature reviews (SLRs) using Generative AI (GenAI) or those conducting empirical studies evaluating how well GenAI supports SLR tasks.

Key concepts

GUEST
An acronym representing preliminary process recommendations for using and evaluating GenAI in SLRs. It emphasizes that a GenAI tool should always operate under human oversight rather than replacing the researcher.
Level 2 Changes
Infrastructure-level changes in GenAI tools that can occur daily or hourly without public notice. These include things like safety filters, system prompts, input preprocessing, and UI/API wrapper changes. These rapid updates create significant reproducibility problems for users.
SLR Practice Changes
Specific adjustments to standard SLR tasks—like literature screening metrics, data extraction performance assessment, and qualitative synthesis—that arise when using GenAI tools. These are the paper's main contribution to improving SLR conduct.
Human Oversight
The fundamental requirement that GenAI tools must be used under human supervision in academic research. Current GenAI summaries cannot yet be equated to a full systematic review because they lack the necessary associated documentation for replication.

Terminology

Summary

To support researchers intending to conduct systematic literature reviews (SLRs) using Generative AI (GenAI) or those conducting empirical studies evaluating how well GenAI supports SLR tasks. p. 0 The paper discusses the problems researchers face when evaluating GenAI for SLRs, identifies and explains process issues to consider when planning, conducting, and reporting both SLRs using GenAI and evaluations of GenAI tools. p. 0 Finally, it summarizes its results as a set of process recommendations named GUEST (GenAI Use and Evaluation in SLR Tasks). p. 0 The authors argue that GenAI requires human oversight and is not currently capable of unsupervised systematic studies. p. 0 The GUEST recommendations should help software engineering researchers both to conduct and report trustworthy SLRs using GenAI and to provide rigorous independent evaluation studies. p. 0

"Our goal is to provide process recommendations to assist researchers who want either to evaluate or to use GenAI tools for conducting SLRs. We discuss a number of methodological issues concerning the use of GenAI tools and identify process recommendations for planning, conducting, and reporting that we hope will be of value to researchers using and evaluating GenAI tools to support SLRs. p. 1 Our recommendations are primarily intended to support software engineering (SE) research, which is the focus of our own research, but may be of interest to the wider research community. p. 1 For brevity, we refer to this preliminary set of recommendations as GUEST (GenAI Use and Evaluation in SLR Tasks): like a guest, a GenAI tool should make its contributions under the researchers’ oversight, never replace them. p. 1"

"In Section 2.1 The Nature of GenAI tools: (1) They create something of a moving target for evaluators and users, and this happens at two levels. p. 2 The first level is the major model updates that occur (e. p. 2g., GPT-4 to GPT-4o to GPT-5. p. 26), though fortunately, these releases are differentiated by formal release announcements, version numbers, and some documentation. p. 2 More problematic is the second, or infrastructure level. p. 2 This includes: (i) safety filters and content policies, (ii) system prompts and guardrails, (iii) input preprocessing and output filtering, which can be highly relevant for say PDF processing capabilities, (iv) rate limiting and abuse detection, and finally (v) general UI/API wrapper changes. p. 2 Unfortunately, Level 2 changes can happen daily or even hourly without any public notice, leading to reproducibility problems. p. 2"

"In Section 2.3 Related Work: While early work on SLR automation showed promise, it was only in the last few years, with the emergence of GenAI, that broad automation began to seem realistically achievable. p. 5 In an SLR on the topic, Van Dinter et al. p. 5 [94] found only 41 papers on SLR automation up to June 2020, and these studies were confined to just two domains: software engineering and medicine. p. 5 Since around that time, there has been a rapid increase in both the number and the diversity of studies investigating how LLMs and other GenAI techniques can automate one or more stages of the SLR process (see, e. p. 5g., [21, 32, 33]). Several commercial tools tailored to SLRs have also emerged (see, e. p. 5g., Elicit3). p. 5 However, in the absence of clear guidelines and recommendations, and given the limited technical understanding of GenAI among many users, some initiatives pursue SLR automation in an overly naive way, which can compromise the quality of both the intermediate results and the final outputs of SLRs conducted with these technologies. p. 5"

"In Section 4.4 Summary of Recommendations: Our recommendations are of four kinds, summarized in Table 4. p. 6 The first kind covers SLR practice changes due to GenAI tool use. p. 6 Such changes arise in the tasks that belong to systematic reviews in particular: literature screening, data extraction, qualitative synthesis, and risk-of-bias and strength-of-evidence assessment. p. 9 These recommendations are the distinctive contribution of this paper, and most of them form the SLR-process recommendations below. p. 6"

"In Section 6 Discussion: Our recommendations support the viewpoint that the use of GenAI tools in the context of academic research must be based on human oversight. Although GenAI tools are already capable of providing accurate summaries of the research literature, such summaries cannot be equated to systematic literature reviews because currently they do not deliver associated documentation, allowing their summaries to be replicated by other human researchers using a trustworthy systematic literature review process. p. 0 The requirement for human supervision will not disappear even if we have GenAI tools that are theoretically capable of delivering a complete SLR, together with associated documentation. p. 9"

"In Section 7 Open Research Questions: For literature screening and qualitative synthesis we can draw on empirical evidence (Section 4.3). For three further tasks — data extraction, risk-of-bias (RoB) assessment, and strength-of-evidence (SoE) assessment — there is little or no empirical evidence on GenAI tool performance for the heterogeneous study types found in SE, and what evidence exists (largely from medical randomized controlled trials) is contradictory. p. 9 Our recommendations for these tasks are therefore preliminary and process-oriented, and we set out the main questions we believe must be answered before firmer, evidence-based guidance is possible. p. 6"

"In Section 8 Conclusions: In this paper, we have presented GUEST, a preliminary set of recommendations that extend existing research processes to address some of the problems faced by researchers using, testing, and evaluating GenAI tools in the context of SLR conduct. We argue that GenAI tools are unable to produce SLRs, but that the ability of such tools to produce summaries of research papers provides a strong incentive for further research. p. 7 These recommendations will help researchers to develop more rigorous evaluation methods that properly address both the limitations and the potential of GenAI tools. p. 0"

"Table 4 Classification of our recommendations: which are specific to SLRs, and which existing guidance we align the general ones with. Category What it covers (examples) Related existing guidance SLR practice changes due to GenAI tool use (i) Tasks particular to SLRs: literature-screening metrics, data-extraction performance assessment, qualitative synthesis, and risk-of-bias and strength-of-evidence assessment. p. 9 The distinctive contribution of this paper. p. 6 General secondary study practice (ii) Good practice for any secondary study, with or without GenAI: documenting exclusion decisions and guarding against lost evidence (general aim; SLR-specific control under screening metrics). p. 6 SEGRESS [51]; PRISMA 2020 [73] General GenAI evaluation (iii) Practice generic to almost any empirical evaluation of a GenAI tool: justifying tool choice, reporting model version, parameters and prompts, and using an open tool as a baseline. p. 0 Baltes et al. p. 5 [3]; Wagner et al. p. 6 [97]; RAISE [90] General evaluation practice (iv) Good practice for almost any evaluation study whether or not GenAI-based: defining goals and hypotheses, using an appropriate study design and data analysis methods, and supporting reproducibility, including replication packages and open data. Kitchenham et al. p. 0 [49, 52]; Ralph [80]; Huotala et al. [39]"

"Table 6 GUEST SLR-process recommendations: Literature screening (Scr1 Use Matthews correlation coefficient (MCC) to ensure results are better than chance.a Both (i) Scr2 Use WMCC to properly account for cost differences associated with False Positives (FPs) and False Negatives (FNs). Both (i) Scr3 Report full confusion matrix counts to support meta-analysis and re-analysis of results. p. 1a Both (i)"

"Table 5 GUEST planning, conduct and reporting recommendations: Planning Plan1 Justify the choice of GenAI tools and critically appraise them. Both (iii) Plan2 Justify the choice of study design. p. 6 Both (iv) Plan3 Define the methods to be used to evaluate GenAI tool performance. p. 7 Both (iv) Plan4 Ensure that the evaluation processes are designed to detect tool errors and to minimize risks to validity from built-in tool biases and from data contamination caused by repeated analyses of the same dataset within the study. Both (iii) Plan5 Identify the data needed to assess tool performance and plan the data collection and storage process and any quality control procedures. p. 4 Both (iv) Plan6 Base comparisons on results from at least two human researchers. p. 2 Both (ii)"

"Table 6 GUEST SLR-process recommendations: Data extraction Ext1 Plan for data extraction to include descriptive textual information such as study type, study goals, study setting, publication date, first author, and publication type. Both (ii) Ext2 For mapping studies requiring classifying primary studies against pre-defined categories, use MCC (with an extension if multiple outcomes or multiple categories are required). p. 8 Both (i) Ext3 For mapping studies requiring classifying primary studies against pre-defined categories, avoid WMCC unless there is an explicit justification for assuming differential costs for data extraction errors. Both (i) Ext4 For quantitative data extraction, correctness metrics should use the values explicitly reported in the primary study text as the gold standard whether the extraction was performed by human researchers, or by a tool. p. 9 Both (iv) Ext5 For more complex data extraction, such as that supporting qualitative analysis, define which sub-tasks in the data extraction process are being automated. p. 8 Both (i) Ext6 For more complex data extraction, such as that supporting qualitative analysis, define the starting point of the automated process in terms of knowledge about primary studies. p. 4 Both (i)"

"Table 6 GUEST SLR-process recommendations: Qualitative synthesis Syn1 Specify and report the qualitative synthesis method being adopted and explain why it is suitable given the SLR research questions and the SLR primary studies. Both (ii) Syn2 Specify and report the criteria used to assess GenAI tool performance, given the qualitative synthesis method being adopted. p. 7 Both (i) Syn3 Specify and report how the synthesis of GenAI tools will be compared with the synthesis provided by human researchers, in particular, how relative performance will be measured and assessed. p. 5

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Development of a process-oriented framework for GenAI tool use in Systematic Literature Reviews (GUEST) that addresses planning, conducting, and reporting issues for both Tool Evaluators and Reviewers by defining four kinds of recommendations.

  2. Implementation of cost-sensitive metrics like Weighted Matthews Correlation Coefficient (WMCC) for literature screening to properly account for the cost differences associated with False Positives (FPs) and False Negatives (FNs) as noted in the paper.

  3. Integration of a validation step into the SLR process where human researchers can conduct prospective studies of GenAI performance, especially for tasks like literature screening, to avoid data contamination.

  4. Development of task-specific briefing notes for data extraction, risk-of-bias assessment (RoB), and strength-of-evidence assessment (SoE) to guide GenAI tools in these complex areas.

  5. Creation of a reporting standard that requires the explicit reporting of prompt strategy, model versions, parameters, and costs for any GenAI tool use during an SLR to ensure traceability.

Sources

Related papers