GUEST: Preliminary Guidelines for Using and Evaluating GenAI Tools to Support Systematic Literature Reviews

summary

The gist

To support researchers intending to conduct systematic literature reviews (SLRs) using Generative AI (GenAI) or those conducting empirical studies evaluating how well GenAI supports SLR tasks.

In short

The paper provides process recommendations, GUEST, for researchers using Generative AI (GenAI) in Systematic Literature Reviews (SLRs) or evaluating GenAI tools for SLRs. It argues that GenAI requires human oversight and is not yet capable of unsupervised systematic studies. The recommendations focus on planning, conducting, and reporting trustworthy SLR tasks.

Key concepts

GUEST
An acronym representing preliminary process recommendations for using and evaluating GenAI in SLRs. It emphasizes that a GenAI tool should always operate under human oversight rather than replacing the researcher.
Level 2 Changes
Infrastructure-level changes in GenAI tools that can occur daily or hourly without public notice. These include things like safety filters, system prompts, input preprocessing, and UI/API wrapper changes. These rapid updates create significant reproducibility problems for users.
SLR Practice Changes
Specific adjustments to standard SLR tasks—like literature screening metrics, data extraction performance assessment, and qualitative synthesis—that arise when using GenAI tools. These are the paper's main contribution to improving SLR conduct.
Human Oversight
The fundamental requirement that GenAI tools must be used under human supervision in academic research. Current GenAI summaries cannot yet be equated to a full systematic review because they lack the necessary associated documentation for replication.

Terminology used across episodes

This episode discusses

The paper

GUEST: Preliminary Guidelines for Using and Evaluating GenAI Tools to Support Systematic Literature Reviews · Read on arXiv

Keele University · Universidad de la Repíblica Universidad de la Repíblica · Wroclaw University of Science and Technology, Poland · University of Calgary Canada · Brunel University of London United Kingdom Durham University United Kingdom

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "GUEST: Preliminary Guidelines for Using and Evaluating GenAI Tools to Support Systematic Literature Reviews".

Jane: Here's the gist of the paper. To support researchers intending to conduct systematic literature reviews (SLRs) using Generative AI (GenAI) or those conducting empirical studies evaluating how well GenAI supports SLR tasks.

Tom: The paper discusses the problems researchers face when evaluating GenAI for SLRs, identifies and explains process issues to consider when planning, conducting, and reporting both SLRs using GenAI and evaluations of GenAI tools.

Jane: Finally, it summarizes its results as a set of process recommendations named GUEST (GenAI Use and Evaluation in SLR Tasks).

Tom: The authors argue that GenAI requires human oversight and is not currently capable of unsupervised systematic studies.

Paper summary: Tom: So what's the core problem they’re identifying? It seems like the biggest issue isn't just that the AI makes mistakes sometimes, but that it creates a very unstable environment for evaluation. They point out these two levels of change, which is something we see constantly in AI development.

Jane: That’s right. They say there’s the major model updates, like moving from GPT-four to GPT-4o or whatever comes next, which is trackable because they announce version numbers and documentation <ref:2607.24991#pg1>. But then there's the infrastructure level, which is much more problematic for reproducibility.

Lu: Exactly. That infrastructure level includes things like safety filters and content policies, system prompts and guardrails that can change daily without public notice, input preprocessing, output filtering, rate limiting abuse detection, and even general UI wrapper changes that can happen hourly without warning. This lack of notice leads to serious reproducibility problems.

Meng: That sounds like a nightmare for any quality control process. If the rules governing how the AI processes your data change overnight because of an unannounced prompt update or a new filter kicking in, you can’t trust your results from yesterday, right?

Lalam: The paper highlights that these Level two changes are often undocumented, which is what they call reproducibility problems, and it makes it really hard to evaluate the tool consistently <ref:2607.24991#pg1>.

Tom: And they tie this instability directly into the methodological issues they found in existing work. They classify these problems into four categories covering adjustments to SLR practice, general secondary study standards, specific GenAI evaluation practices, and core empirical rigour.

Jane: So it’s not just a technical glitch; it’s a breakdown in the process itself when we try to apply AI to systematic reviews. They are trying to define how researchers should adjust their actual SLR tasks based on what these tools are capable of.

Lu: They also bring in some historical context, mentioning things like the Mean Magnitude Relative Error or MMRE metric from earlier work that showed how biased some performance metrics can be, and they apply that cautionary lens to GenAI tool performance.

Meng: It’s interesting they brought up those older metrics because it shows that the risks aren't new; we’re just applying old scrutiny to a new technology.

Lalam: And what they really want is for researchers to be explicit about their choice of tools, the version numbers, and the prompts they use so that others can actually understand what you did.

Conclusion: Tom: So wrapping up this paper, "GUEST: Preliminary Guidelines for Using and Evaluating GenAI Tools to Support Systematic Literature Reviews" is about establishing a baseline for how we talk about using AI in this specific research area. The authors argue that right now, Generative AI cannot be trusted to perform unsupervised systematic reviews on its own.

Jane: That’s the big message: human oversight is absolutely required. Even if an AI tool can give you a summary of the literature, that summary isn't the same as a systematic review because it doesn't have all the associated documentation needed for someone else to replicate your exact process.

Lu: They emphasize that this requirement for human supervision won't disappear even if we develop GenAI tools that are theoretically capable of delivering a complete SLR with all its associated documentation. The paper stresses that the process matters more than just having the tool itself.

Meng: From a deployment standpoint, this means we can’t just throw an AI script at a large dataset and expect it to run autonomously without someone checking every step and validating the output against our established metrics.

Lalam: The GUEST framework they propose is designed to be this comprehensive set of fifty-one recommendations meant to guide researchers through planning, executing, and reporting these GenAI-assisted studies in a trustworthy way. It’s meant to be a baseline for rigorous evaluation.

Tom: So the implication for the wider research community is that if you want to use AI in an SLR, you have to follow this structure—just justify your tool choice and report your version details clearly. It shifts the focus from "can this AI do it?" to "how rigorously are we managing this AI process?"

Jane: It’s a necessary shift. They aren't saying give up on using GenAI for tasks like repetitive screening or secondary validation, but they are saying we need to standardize *how* we use these tools so that the results remain trustworthy in software engineering research.

More episodes

← Home