Frontier Lag: A Bibliometric Audit of Capability Misrepresentation in Academic AI Evaluation

arXiv:2605.04135 · cs.CY, cs.AI, cs.CL · Submitted 2026-05-05 · Read on arXiv

cs.CY, cs.AI, cs.CL

Submitted: 2026-05-05

Updated: 2026-09-30

Comments: 52 pages, 6 figures, 7 tables. v4 completes the revision begun in v3: registered primary-model rule and frontier applied; coder-agreement and adjudication details updated; registered sensitivity analyses added. Pre-registered: https://doi.org/10.17605/OSF.IO/7XM3D. Code: https://doi.org/10.5281/zenodo.20060458. VERSIO-AI v1.2: https://doi.org/10.5281/zenodo.20060459. Tool: https://frontierlag.org

Project page: https://davidgringras.github.io/safety-under-scaffolding

License: http://creativecommons.org/licenses/by/4.0/

The gist: LLM evaluations in applied domains tend to reflect models that were already outclassed at time of publication.

Terminology

Abstract

LLM evaluations in applied domains tend to reflect models that were already outclassed at time of publication. We observe a publication elicitation gap: the distance between the AI systems generating the results reported in an academic paper and the AI systems that a current reader of that paper would reasonably assume are being referenced. We systematically sweep OpenAlex from 2022-01-01 to 2026-04-01 (n = 112,303 LLM keyword matches). Then, we identify what models were evaluated (n = 18,574 admissible records). We then rank each evaluated LLM against a frontier LLM based on the Epoch AI Capabilities Index (ECI), an aggregate LLM capability score. We find that the median paper's models are worse than the frontier LLM at the time of evaluation (a median gap of +10.45 ECI; H1, n = 12,668). The gap is increasing at a rate of +4.07 ECI per year (H2, nominal 95% CI [+3.75, +4.45]). An explicitly stated evaluation date can be found in only 18.4% of full-text papers. A Bayes-corrected 52.5% (95% CI: [47.3, 57.9]) of the abstracts audited discuss their conclusions in terms of "AI" as a category, rather than specific models. Just 2.2% of abstracts and 21.2% of full-text articles evaluating reasoning models disclose whether the models were tested with reasoning turned on or off (H4). We propose a solution to this problem that is distributed among authors, editors, and funders. First, reporting from authors; VERSIO-AI v1.2 is a proposed 13-item checklist to cover the configuration surface described herein. Second, enforcement from journal editors and peer reviewers. Third, conditioning grants on disclosure and providing API access.

Sources

Related papers