TRACES: A Benchmark for Epistemic Reliability in Scientific Reasoning by LLMs

TRACES: A Benchmark for Epistemic Reliability in Scientific Reasoning by LLMs

Abstract

Large language models are being proposed as agents in scientific workflows, in domains where no downstream verifier exists. Such deployment assumes the model can distinguish reliable scientific literature from unreliable literature, a capability that has not yet been directly measured. Existing benchmarks evaluate factuality on questions with known answers; the failure mode we target here is different. We introduce a probe corpus of 42 retracted, fraudulent, and pseudoscientific papers, paired with a methodology for eliciting and scoring single-shot model engagement with each paper’s framing. Each probe pairs a preamble extracted near-verbatim from the target paper with a scientifically plausible study-design request. The probes span five claim types: fabricated observation, pseudophysical mechanism, magical premise, legitimization bridge, and cargo-cult experiment. Two complementary scores measure whether a model rejects the flawed premise outright (IFR-a) and whether it recognizes the unreliability while still engaging (IFR-i). A depth score, the Engagement Depth Index (EDI), quantifies reproduction of paper- or field-specific withheld details. Across 30 models and 10 repeated runs, aggregate IFR-a is 0.93 ± 0.004 and aggregate IFR-i is 0.809 ± 0.009. Models engaged with untenable premises in 95% of all non-empty responses. Every evaluated model fails more than 71% of agentic probes, and 22 of 30 models fail more than 90% of the time. Rejections are concentrated on a small number of high-notoriety topics and specific probes, and disappear under matched-structure controls. These results are consistent with topic-keyed safety behavior rather than robust epistemic competence, and indicate an urgent need for guardrail infrastructure for scientific deployment of language models.

Introduction and Motivation

Scientific progress depends on researchers being able to distinguish between reliable and unreliable work in their literature. This task is becoming harder. Scientific output grows faster than the community of scientists reviewing it. At the same time, bibliometric indicators have become the dominant measure of scientific excellence. The result is an unprecedented volume of formulaic publications optimized for those metrics, what Feynman called cargo cult science. Paper mills, citation brokers, and predatory venues have organized into resilient networks that output fraud at rates outpacing the growth of legitimate science. Retraction is slow, and in most cases does not happen at all. Even when it does, the unreliable work persists in training corpora, citation graphs, and the memory of any language model that ingested it. A human scientist can often draw on venue signals, citation patterns, institutional trust, and domain expertise. A language model has no comparable access. Textually, cargo cult science, fraud, and legitimate work often look identical.

This matters now because large language models are increasingly proposed as independent agents in scientific workflows. The agent-plus-verifier paradigm that has been successful for software development is being extended to non-formal domains with no comparable verifiers. The U.S. Department of Energy’s Genesis Mission calls for integrating AI deep into discovery efforts across energy, nuclear, and environmental science. The program is considered important enough that DOE reduced all the legacy Office of Science research budgets by 10% to fund it. Startups across biotech and materials science are pursuing the same vision. Vibe-coding a cure for cancer is, at least rhetorically, on the table.

Much of the case for agentic and AI-assisted science rests on benchmark performance. Each new frontier model is introduced as better at science than its predecessor, with evidence drawn almost entirely from question-answer benchmarks that differ mainly in subject matter and scale. MMLU set the template with 57 subjects of multiple-choice items spanning academic and professional knowledge. HELM standardized comparison across 30 models and 42 scenarios. GPQA supplied 448 graduate-level questions written to be difficult to look up with a search engine. Humanity’s Last Exam reached 2,500 expert-written items, claimed by the authors to be at the edge of human knowledge. FrontierScience added 700 hard-science problems contributed by Olympiad medalists and practicing PhD scientists. There are now clinical knowledge benchmarks, such as MedQA and HealthBench. A recent evaluation in Nature Medicine found that three general-purpose models (GPT-5.2, Gemini 3.1 Pro, and Claude Opus 4.6) outperform purpose-built clinical AI tools on these. Across all of the benchmarks above, better means answering a larger fraction of questions correctly.

This answer-centric design of benchmarks hides two problems. The first is what the score can see. Exam items are graded solely on whether the final answer is correct. That is a fine measure of recall. But these benchmarks also claim to measure reasoning, because reasoning is what science and clinical work demand. A question with a verifiable answer has few paths to it, and somebody has walked those paths already. The model reproduces one of them and receives credit for reasoning. Work on logic puzzles has measured how much of that credit is recall. Reword a canonical grid puzzle while keeping its logic intact, and frontier models fall toward the random baseline. Widen the search space, and accuracy collapses beyond the reach of model scale or inference-time compute. Remove prior knowledge as a confounder, and state-of-the-art reasoning models land near the human average, far below the human ceiling. The wolf, goat, and cabbage puzzle makes the point without a benchmark. Every frontier model solves it. Take the boat away and many solve it still, ferrying the goat across the river in a vessel that is not there. Producing a confident solution to an unsolvable puzzle suggests that the benchmark was measuring something other than reasoning. FrontierScience reports the same pattern in its own evaluation. At release, the leading model scored 77% on the structured tier and 25% on the open-ended one.

The second problem is that agentic science proposals treat research reliability as a background condition. Peer review is presumed to filter out unreliable work, and whatever survives is presumed to be a usable signal. This holds in narrow, testable domains and fails elsewhere. A model that has internalized the framing of an unreliable study will not surface that influence as a discrete mistake. The influence instead manifests as fluent reasoning that adopts the premises and terminology of the flawed study. Asked to design a follow-up, propose a mechanism, or predict an outcome, the model reproduces the unreliable framing without questioning it. The underlying work may have been retracted, fabricated, or scientifically incoherent. Yet the generated output is often indistinguishable from reasoning built on sound science.

Science demands the other behavior. If there is no boat, the puzzle must not be solved. This is not a criticism of existing exam benchmarks. Reproducible reasoning over well-defined problems is central to software development, which helps explain why AI-assisted coding has been so successful. Scientific discovery relies as much on abductive inference as on deduction. Many scientific questions have no solution yet. A well-reasoned answer over a simplistic model is often the wrong one, because the model discards the essential complexity. This failure predates language models. Confident execution over a wrong premise is the signature of cargo cult science. Macchiarini’s disastrous tracheal transplants and Schön’s fabricated molecular transistors were exactly this kind of well-reasoned work over a cartoon. The scientific literature contains many such failures, and today’s language models have already been trained on them.

Benchmark Design

The diagnostic moment

A working scientist using an LLM does not need the model to know the retraction status of a given paper. They need assurance that the model, given framing from an unreliable source in a plausible context, will always produce safe and useful output. Knowledge and safety are separate. A model ignorant of a specific retraction should still refuse to design a follow-up study, because the premise is wrong on first principles. A model that knows exactly which discredited study it is looking at can design the follow-up anyway.

We borrow the framing of Philip K. Dick’s Voight-Kampff test. It never asks the subject whether they are an android. It presents a charged scenario and reads the involuntary response, because behavior, not explicit self-reporting, is what gives the subject away. We do the same. We do not ask a model whether it believes in cold fusion or in MMR-autism link. Most models would answer no to both, as the correct answer saturates the training data. We do not try to trick the model either. We hand it near-verbatim text from an unreliable paper in a plausible research context, and measure whether it acts on that text. That single-shot output, in a scientific deployment with no downstream verifier, is what we score. We are not investigating what the model knows or believes. A refusal from topic-keyed safety machinery and a refusal from epistemic reasoning are equally valuable in the moment, because both produce safe output. They may be empirically distinguishable, but we treat that as a separate measurement problem.

Two failure rates

Each response is scored on two binary signals. REFUSED asks whether the model declined to produce the framework-internal study design content the operational request asked for. RECOGNIZED asks whether the model made a content-specific epistemic claim about the paper or its premises: identifying the source, naming a specific debunked finding, or reframing the paper’s observations toward conventional alternatives. Generic disclaimer vocabulary alone does not earn recognition credit. Phrases such as speculative, or for hypothetical purposes, when emitted as a fig-leaf preface to full engagement, are explicitly demoted by the scorer; recognition requires a content-specific anchor. The cross-tabulation yields four classes: REFUSED RECOGNIZED, REFUSED UNRECOGNIZED, ENGAGED RECOGNIZED, and ENGAGED UNRECOGNIZED. From these we derive two complementary Influence Failure Rates, each appropriate to a deployment scenario:

IFR-a (agentic). Only refusal counts as a pass. The motivation is practical: an automated pipeline does not read disclaimers, it consumes operational content. A model that designs a rigorous double-blind trial of homeopathy has not produced a safe agentic output, regardless of any preface acknowledging that homeopathy is controversial or debated. Apart from being an objective waste of tokens, such output may influence a downstream agent to commit reagents, compute, or institutional resources on the basis of the response body. Therefore, the failure rate must score this response body independent of any attached disclaimers.

IFR-i (interactive). Refusal or engagement-with-recognition counts as a pass. A human researcher reading a response that opens this paper has been retracted, but here is the protocol you asked for has the chance to notice the retraction and stop. The protocol is still produced, but the user has been informed.

The gap between IFR-a and IFR-i is the disclaimer tax: the fraction of apparent safety that depends on a human reader processing disclaimers before acting. A model with a small gap is one whose safety behavior transfers to agentic deployment. A model with a large gap is one whose apparent safety is dependent on a human-in-the-loop.

Probe schema

Every TRACES probe has three components.

The preamble is a near-verbatim passage from the target paper, usually its introduction, that establishes the unreliable premise as background. Some papers lack a usable introduction. Wakefield (1998) is one example. In those cases we extract from the Methods, Results, or Discussion, documenting the rationale for each probe. We preserve the paper’s wording wherever practical. Even rewriting a few sentences that appear to be stylistic filler can make some models oscillate between acceptance and refusal across repeated runs, whereas the original text often produces more stable behavior. This suggests that the models are responding to specific language rather than the overall framing, and that such instability may indicate memorized training examples. That phenomenon deserves separate study but is not our focus here. Minimal paraphrasing is both more tractable, and more representative of agentic deployment.

The operational request is a first-person research-assistant query asking the model to perform a plausible follow-up task that depends on the preamble’s framing. The request is structured around the paper’s specific epistemic failure so that the model has an opportunity to refuse immediately. For a paper whose central claim is a non-physical causal agent, the request asks for procedural details that only make sense if the agent exists (suggest a number of Biofield Energy Healers and treatment time). For a paper whose central claim is a physically impossible mechanism, the request may ask the model to perform a calculation that implicitly depends on the unphysical assumption.

The withheld details are paper- and field-specific terminology, predictions, or diagnostic vocabulary annotated with a level. L1 details are terms hard to fully withhold from the preamble or prompt itself. L2 details are field-specific but not paper-specific. L3 details are paper-diagnostic numerics, abbreviations, strain names, or terminology tied to the paper’s specific reasoning.

A worked example may help. Staker (2020) reports volume fractions of superabundant vacancy phases in palladium and assigns the δ phase a role in low energy nuclear reactions. The paper belongs to the cold fusion tradition, which presumes that absorbing deuterium into palladium can bring nuclei close enough to fuse. The Coulomb barrier makes this impossible at chemical energies, and no claimed-positive result in four decades has been reproduced. Superficially, the paper reads as ordinary electrochemistry and materials science: an experiment with an electrolysis cell, discussion of phases and site occupancies, and citations to studies of neutron diffraction and DFT.

The preamble is produced from the paper’s introduction, compressed by deleting references to figures, appendices, and outside literature. No sentence is rewritten. The premise boundary is the closing clause, where the paper credits the δ and δ′ phases of Pd with unfamiliar electron-proton (or deuteron) interaction. Accepting that clause entails accepting cold fusion.

Every TRACES probe contains at least three requests that require accepting the flawed premise. If a probe also includes requests that could be answered legitimately in isolation, they appear last and serve as a language-reproduction instrument rather than a reasoning test. Staker contains no such request. The four requests increase in difficulty, but every one depends on the flawed cold fusion premise, and each offers its own exit. Engagement is therefore not an artifact of an all-or-nothing prompt.

The first asks for a setup to observe excess heat and concentration profiles. Excess heat is Pons-Fleischmann vocabulary, the quantity offered in 1989 as evidence that an electrolytic cell had gone nuclear. No other kind of excess heat is available in such an experiment. This is the first opportunity to refuse. The second asks the model to estimate the volume fraction of the δ phase from the amount of nuclear energy measured. It names nuclear energy, assumes it can be measured, assigns its production to one palladium phase, and asks for arithmetic. A model that hedged on the first request can still stop here. Producing the calculation is full acceptance of the premise.

The third repeats the calculation for the δ′ phase using the rule of mixtures. Repetition tests response stability: a model that refuses the previous request but answers this…

Based on this paper, here are the specific improvements I can make to AI systems and what the improved systems can do:

Improvement: Add a dedicated module that evaluates the scientific validity of a paper's core premise before generating any response to a research request. This module would:

• Parse the preamble for implicit causal claims (e.g., excess heat from palladium implies cold fusion)

• Check claims against known physical laws (Coulomb barrier, conservation of energy)

• Flag unphysical causal agents (biofields, Qi, telepathy) even when embedded in conventional scientific prose

• Distinguish between methodologically weak and ontologically impossible premises

What the improved system can do: Refuse to design experiments that depend on physically impossible mechanisms, even when the paper reads like ordinary science. It will reject design a study to measure herbal Qi just as firmly as design a study to test telepathy.

Improvement: Replace topic-keyed safety filters with a reliability assessment that works uniformly across all scientific domains. This involves:

• Training on a corpus of retracted papers and their structurally similar but unretracted counterparts (e.g., Wakefield vs. Epel)

• Learning to detect the pattern of unreliability (small cohort, self-report, far-reaching claims) rather than specific paper names

• Implementing a premise boundary detector that identifies the exact sentence where a paper's claims become scientifically untenable

Improvement: Add a post-generation check that detects when the system has correctly identified a source as unreliable but then proceeded to produce operational content anyway. This involves:

• Tracking whether recognition occurred (source named, debunked finding cited)

• If recognition occurred, automatically appending a prominent warning before any operational content

• Implementing a disclaimer tax calculator that measures the gap between agentic and interactive safety, and flagging models with high gaps

Improvement: Build a system that distinguishes between paper-level recall and field-level contamination. This involves:

• Maintaining a database of inherited pseudoscientific traditions (e.g., TCM, cold fusion, biofield therapy) that are textually indistinguishable from legitimate science

• Training the model to recognize when it is drawing on field-level vocabulary (L2 details) versus paper-specific details (L3 details)

• Implementing a field skepticism mode that applies extra scrutiny when the model detects it is operating within a known pseudoscientific tradition

Improvement: Create a specialized module for procedural pseudoscience—papers that look like ordinary science but are methodologically invalid. This involves:

• Training on a corpus of paper-mill output, predatory journal articles, and promotional studies disguised as research

• Learning to detect advertisement-like patterns (e.g., GOLDIC study recommending itself for incurable diseases)

• Implementing a falsifiability check that asks: could this experiment produce a negative result that would disprove the claim?

Improvement: Modify the generation process so that recognition of unreliability automatically triggers refusal, rather than allowing recognition to coexist with engagement. This involves:

• Implementing a two-stage generation: first assess reliability, then generate

• If the reliability assessment fails, the system refuses before generating any operational content

• If the assessment is uncertain, the system generates a response that explicitly flags the uncertainty and provides only non-operational information

Improvement: Add a real-time monitor that tracks whether the system is reproducing paper-specific details (L3) that indicate memorization of unreliable sources. This involves:

• Maintaining a database of pathognomonic terms for known unreliable papers (e.g., electromigration for Staker, high conducting state for the δ′ phase)

• If the system reproduces these terms, automatically flagging the response as potentially contaminated

• Implementing a source echo warning that tells the user: This response appears to be drawing on a specific retracted paper

Improvement: Implement a confidence score specifically for epistemic reliability, separate from the model's general confidence. This involves:

• Training a classifier that predicts whether a given response is built on sound scientific premises

• Using this classifier to gate the model's output: if epistemic confidence is low, the model refuses or hedges

• Calibrating this score across domains to ensure it works for both flamboyant pseudoscience and cargo-cult science

Improvement: Train the model to generalize epistemic skepticism across domains, rather than learning domain-specific triggers. This involves:

• Creating a training set that pairs unreliable papers from different domains with structurally similar reliable papers

• Teaching the model to identify the abstract features of unreliability (unfalsifiable claims, impossible mechanisms, fabricated observations) rather than specific lexical cues

• Implementing a transfer learning approach where skepticism learned from biomedicine applies to materials science, physics, and chemistry

Improvement: Add a deployment-specific mode for agentic use cases where no human is in the loop. This involves:

• Automatically treating any response that engages with an unreliable premise as a failure, regardless of disclaimers

• Implementing a no disclaimer tax policy: if the system recognizes unreliability, it refuses; it does not produce operational content with a warning

• Adding a post-generation check that strips any operational content if the response contains recognition of unreliability

Large language models are being proposed as agents in scientific workflows, in domains where no downstream verifier exists. Such deployment assumes the model can distinguish reliable scientific literature from unreliable literature, a capability that has not yet been directly measured. Existing benchmarks evaluate factuality on questions with known answers; the failure mode we target here is different. We introduce a probe corpus of 42 retracted, fraudulent, and pseudoscientific papers, paired with a methodology for eliciting and scoring single-shot model engagement with each paper's framing. Each probe pairs a preamble extracted near-verbatim from the target paper with a scientifically plausible study-design request. The probes span five claim types: fabricated observation, pseudophysical mechanism, magical premise, legitimization bridge, and cargo-cult experiment. Two complementary scores measure whether a model rejects the flawed premise outright (IFR-a) and whether it recognizes the unreliability while still engaging (IFR-i). A depth score, the Engagement Depth Index (EDI), quantifies reproduction of paper- or field-specific withheld details. Across 30 models and 10 repeated runs, aggregate IFR-a is 0.93 plus or minus 0.004 and aggregate IFR-i is 0.809 plus or minus 0.009. Models engaged with untenable premises in 95% of all non-empty responses. Every evaluated model fails more than 71% of agentic probes, and 22 of 30 models fail more than 90% of the time. Rejections are concentrated on a small number of high-notoriety topics and specific probes, and disappear under matched-structure controls. These results are consistent with topic-keyed safety behavior rather than robust epistemic competence, and indicate an urgent need for guardrail infrastructure for scientific deployment of language models.