Towards LLM-Powered Automation of a Dark Matter Constraint Repository

arXiv:2606.21658 · hep-ex, astro-ph.IM, hep-ph · Submitted 2026-06-19 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: High Energy Physics: Experiment. The day in high-energy experiment: detectors, searches and limits.

Kai: I'm Kai.

Mira: And I'm Mira, here with Lev.

Kai: We're looking at the paper: "Towards LLM-Powered Automation of a Dark Matter Constraint Repository".

Mira: Here's the gist of the paper. Dark matter constraint repositories are critical community infrastructure, yet the most widely-used are volunteer-maintained, a sustainability risk as results accelerate We present a large language model (LLM) pipeline that monitors arXiv, extracts limit curves, integrates them as code, and opens pull requests for human review On a 329-paper benchmark of measured limits graded against the upstream-curated repository, it identifies the coupling type for 98.1% of gradable papers The LLM is confined to perception; arithmetic, unit conversion, and the guards that reject mis-extractions are deterministic, testable code Deployed, the pipeline has generated limit proposals, but none have merged: governance of AI-generated scientific data is itself unsolved System Architecture Figure 1 shows the end-to-end pipeline Its design principle, arrived at empirically (Appendix A), is to keep the LLM confined to perception and semantics (reading text, tables, axes, and curves, and declaring what it read) while arithmetic, unit conversion, and candidate selection are deterministic, testable code The pipeline flows from arXiv (top-left) through keyword and LLM triage into five competing extraction channels Rule-based stages also use model-generated information, so the colours do not imply a strict separation of LLM use and code Multi-channel Extraction An extraction channel is a method for obtaining limit values from a paper’s text, tables, figures, or accompanying data files The agent produces candidates through up to five channels of differing semantic trust (table 1), staged as: a cheap particle physics interaction type pre-classification pass; a text/table read of the parsed PDF (PyMuPDF

13: ); a cheap axis-calibration pass that identifies the relevant figure and reads its axis labels and ranges; and a full vision trace of the rendered figure pages The channel selector considers tier only after all validity criteria, preferring higher-tier channels among equally valid candidates Validity and Canonicalization The selector compares candidates using a fixed sequence of checks The first check on which two candidates differ determines which is preferred Physical validity and interpretable units and conventions take priority Convention canonicalization converts quantities to the repository’s standard variable and convention Any quantity or convention outside the reviewed set requires human review Evaluation Methodology We evaluate on 329 papers spanning the repository’s coupling types, chosen as curated repository data files

5: that carry a known arXiv ID (the full registry holds 437 file entries over 347 distinct arXiv IDs) The ground truth (GT) for each paper is the upstream repository’s own digitized curve We compare in log10–log10 space, using g to denote the coupling variable after both curves have been converted to the same convention A paper’s residual is the median of ri over the compared mass points Model Performance and Lessons The production configuration achieves a median per-paper residual of 0.17 dex across 267 compared papers The key gains came from the system built around the model: extraction methods and checks implemented in code, centrally managed physical variable conversions with human review for unsupported conventions, and an evaluation that separates model effects from code changes and measures variation between runs The remaining large errors come from interpreting figures and selecting the correct curve, mass range, or coupling type The pipeline simplifies maintenance rather than replaces the maintainer: it turns curation from authoring into reviewing, and with 51% of papers within a factor of two end-to-end the expert’s accept-or-reject judgement remains the load-bearing step, by design and by necessity Limitations and Future Work The largest errors came from mishandling units and variables, rather than misreading figures The micro–macro gap is carried by coupling types with few papers Further improvement requires better interpretation of these rare limits and their conventions The gap is not tooling but adoption in the light-dark-matter community, where results are still published predominantly as figures The infrastructure for machinereadable limit publication already exists and is in routine use: HEPData

24: archives digitized collider results, and the LHC Reinterpretation Forum has published recommendations for reusable result presentation

25: We delivered an LLM-powered pipeline with deterministic guards that curates the live dark matter limit literature, with every output reviewed by a human before anything merges On a 329-paper benchmark against curator ground truth, it reaches a median accuracy of 0.17±0.04 dex among compared measured-limit papers The key gains came from the system built around the model: extraction methods and checks implemented in code, centrally managed physical variable conversions with human review for unsupported conventions, and an evaluation that separates model effects from code changes and measures variation between runs The benchmark makes that claim measurable: swapping the extraction model for another frontier model a generation apart moves the paired per-paper median by 0.000 dex, while the architectural work, at a fixed model, cut the per-type mean from 1.46 to 0.43 dex (0.38 in the production configuration) The remaining large errors come from interpreting figures and selecting the correct curve, mass range, or coupling type We offer the pipeline, the benchmark, and the measured design lessons collected in Appendix A to the community whose infrastructure problem motivated them, as a bridge until machine-readable limit publication makes extraction unnecessary Appendix A.

Kai: Lessons Learned Validity-first candidate selection is helpful Reading supplied data and vector coordinates directly allows exact or near-exact extraction with little model cost Deterministic guards on the model’s declared outputs stopped incorrect extractions before selection on 49 of 329 papers in the production benchmark Managing conversions in one place is helpful Requiring review for unknown conventions makes the uncertainty visible to reviewers Checking curves against values in the text provides an inexpensive, independent check on a traced curve Expert review before merging is important Separating model effects from pipeline changes and variation between runs is needed to establish the cause of an apparent improvement and whether it exceeds ordinary run-to-run variation Appendix B.

Mira: Selection Criteria and Scoring Details The lexicographic quality tuple records the criteria below in a fixed order The first four check validity, so later preferences cannot outweigh a difference on these checks Agreement with an independent channel is preferred if another channel produces a comparable extraction agreeing within a factor of three (a ratio in

1/3, 3: , or about 0.48 dex) Appendix C.

Kai: Convention Canonicalization in Detail Three rules for conversion are detailed Identifying the output quantity is necessary so later processing does not have to infer them from their magnitudes Preserving the plotted values means the vision channel reads values without converting them and identifies the plotted quantity Managing conversions in one place ensures all pipeline conversions use one reviewed conversion registry Handling unusual data files involves removing very large values using a threshold of 1019 Requiring human review for unknown conventions flags the paper for human review and excludes them from numerical evaluation as convention gaps Appendix D.

Paper summary: Kai: The core idea here is to use an LLM pipeline that reads arXiv papers, extracts the limit curves, and integrates them as code so they can be reviewed by humans before anything gets merged into the main repository.

Mira: They found that on a benchmark of three hundred twenty-nine papers graded against the curated repository, this system identified the coupling type for ninety-eight point one percent of those gradable papers <ref:2606.21658#pg1>.

Lev: That's a lot of coverage, but we need to know if that extraction is reliable when we talk about real physics constraints.

Kai: The paper emphasizes that the LLM itself is only good at perception, like reading text and axes, while the actual math and unit conversions are handled by deterministic code.

Mira: They show that this setup has a median residual of zero point one seven dex for compared papers, which means it’s relatively accurate when you compare its output to the ground truth.

Lev: And they stress that there are these deterministic guards built in to reject bad extractions before anything gets selected.

Kai: The paper claims that this system has generated limit proposals, but none of them have been merged yet, which points to the next big question about governance of AI-generated scientific data.

Mira: It’s about turning the curation work from being done by authors into something that is reviewed by experts, with fifty-one percent of all papers landing within a factor of two of the curator's curve.

Lev: From an engineering standpoint, they built this to be testable and confined; the model doesn't do arithmetic or unit conversions, so you know exactly what it’s doing.

Kai: It sounds like they’re trying to solve a sustainability problem for this community infrastructure by automating the tedious parts of digitizing and converting data.

Conclusion: Mira: So, to wrap up this paper, "Towards LLM-Powered Automation of a Dark Matter Constraint Repository" it presents an end-to-end LLM pipeline with deterministic guards that curates the live dark matter literature, making every output human reviewed before merging.

Kai: The main thing they found on their benchmark is that among compared measured limit papers against the curator ground truth, this system reaches a median accuracy of zero point one seven dex among those papers.

Lev: It’s important to note that while the pipeline generated proposals, none have merged yet, which highlights that governance of AI-generated scientific data is still an open question.

Mira: The real gain they highlight isn't just the model itself, but the system built around it: extraction methods and checks implemented in code.

Kai: Swapping out the extraction model for another frontier model a generation apart only moved their paired per-paper median by zero point zero zero zero dex, while changing the architectural work cut that mean from one point four six to a three point eight in production configuration.

Lev: So it sounds like the architecture and how you handle conversions is more important than just picking the latest big language model for this job.

Kai: The paper offers the pipeline, and the benchmark, and all those measured design lessons to the community that needed this infrastructure problem solved as a bridge until machine-readable limit publication makes extraction unnecessary.

Mira: It’s about making sure that even though we have these powerful new tools, the expert's accept or reject judgment remains the necessary step by design and by necessity.

Department of Physics, Washington University in St Louis

hep-ex, astro-ph.IM, hep-ph

Submitted: 2026-06-19

Updated: 2026-11-04

Code: https://github.com/FaroutYLq/AutoAxionLimits

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 88/100

The gist: Dark matter constraint repositories are critical community infrastructure, yet the most widely-used are volunteer-maintained…

Key concepts

Multi-channel Extraction
This involves using up to five different methods to pull limit values from a single paper, such as reading text, parsing tables, checking axis labels, or performing a full vision trace of figures. The system compares these different methods and selects the best candidate based on predefined validity checks.
Validity and Canonicalization
This is the process of rigorously checking extracted data against physical reality and standard conventions. It involves converting all quantities to a single, standardized format, flagging any unknown conventions for human review, and prioritizing physically valid extractions over others.
Model-Confined Perception
The LLM's role is strictly limited to understanding the semantics of the text—reading what it sees—not performing complex calculations or unit conversions. All arithmetic and critical checks are handled by deterministic, testable code, ensuring the model only contributes perception.
Governance of AI-Generated Data
The paper highlights that while an LLM can generate proposals, human governance remains unsolved. The pipeline's success depends on human review before any data is merged into the official repository, emphasizing that expert judgment is the final necessary step.

Terminology

Summary

Dark matter constraint repositories are critical community infrastructure, yet the most widely-used are volunteer-maintained, a sustainability risk as results accelerate We present a large language model (LLM) pipeline that monitors arXiv, extracts limit curves, integrates them as code, and opens pull requests for human review On a 329-paper benchmark of measured limits graded against the upstream-curated repository, it identifies the coupling type for 98. p. 0

Where the paper says it

We present a large language model (LLM) pipeline that monitors arXiv, extracts limit curves, integrates them as code, and opens pull requests for human review. On a 329-paper benchmark of measured limits graded against the upstream-curated repository, it identifies the coupling type for 98.

1% of gradable papers The LLM is confined to perception; arithmetic, unit conversion, and the guards that reject mis-extractions are deterministic, testable code Deployed, the pipeline has generated limit proposals, but none have merged: governance of AI-generated scientific data is itself unsolved

System Architecture

Figure 1 shows the end-to-end pipeline Its design principle, arrived at empirically (Appendix A), is to keep the LLM confined to perception and semantics (reading text, tables, axes, and curves, and declaring what it read) while arithmetic, unit conversion, and candidate selection are deterministic, testable code The pipeline flows from arXiv (top-left) through keyword and LLM triage into five competing extraction channels Rule-based stages also use model-generated information, so the colours do not imply a strict separation of LLM use and code

Multi-channel Extraction

An extraction channel is a method for obtaining limit values from a paper’s text, tables, figures, or accompanying data files The agent produces candidates through up to five channels of differing semantic trust (table 1), staged as: a cheap particle physics interaction type pre-classification pass; a text/table read of the parsed PDF (PyMuPDF [13]); a cheap axis-calibration pass that identifies the relevant figure and reads its axis labels and ranges; and a full vision trace of the rendered figure pages The channel selector considers tier only after all validity criteria, preferring higher-tier channels among equally valid candidates

Validity and Canonicalization

The selector compares candidates using a fixed sequence of checks The first check on which two candidates differ determines which is preferred Physical validity and interpretable units and conventions take priority Convention canonicalization converts quantities to the repository’s standard variable and convention Any quantity or convention outside the reviewed set requires human review

Evaluation Methodology

We evaluate on 329 papers spanning the repository’s coupling types, chosen as curated repository data files [5] that carry a known arXiv ID (the full registry holds 437 file entries over 347 distinct arXiv IDs) The ground truth (GT) for each paper is the upstream repository’s own digitized curve We compare in log10–log10 space, using g to denote the coupling variable after both curves have been converted to the same convention A paper’s residual is the median of ri over the compared mass points

Model Performance and Lessons

The production configuration achieves a median per-paper residual of 0.17 dex across 267 compared papers The key gains came from the system built around the model: extraction methods and checks implemented in code, centrally managed physical variable conversions with human review for unsupported conventions, and an evaluation that separates model effects from code changes and measures variation between runs The remaining large errors come from interpreting figures and selecting the correct curve, mass range, or coupling type The pipeline simplifies maintenance rather than replaces the maintainer: it turns curation from authoring into reviewing, and with 51% of papers within a factor of two end-to-end the expert’s accept-or-reject judgement remains the load-bearing step, by design and by necessity

Limitations and Future Work

The largest errors came from mishandling units and variables, rather than misreading figures The micro–macro gap is carried by coupling types with few papers Further improvement requires better interpretation of these rare limits and their conventions The gap is not tooling but adoption in the light-dark-matter community, where results are still published predominantly as figures The infrastructure for machinereadable limit publication already exists and is in routine use: HEPData [24] archives digitized collider results, and the LHC Reinterpretation Forum has published recommendations for reusable result presentation [25]

Conclusions

We delivered an LLM-powered pipeline with deterministic guards that curates the live dark matter limit literature, with every output reviewed by a human before anything merges On a 329-paper benchmark against curator ground truth, it reaches a median accuracy of 0.17±0.04 dex among compared measured-limit papers The key gains came from the system built around the model: extraction methods and checks implemented in code, centrally managed physical variable conversions with human review for unsupported conventions, and an evaluation that separates model effects from code changes and measures variation between runs The benchmark makes that claim measurable: swapping the extraction model for another frontier model a generation apart moves the paired per-paper median by 0. p. 1

Where the paper says it

We treat this as a data-processing and software-methods problem rather than an “LLM demo”: the questions that matter are reproducibility, quantified accuracy, and auditable provenance. We present a pipeline that partially automates the curation workflow: (1) discovery via daily arXiv monitoring with keyword filtering and LLM relevance classification; (2) extraction through five competing channels (distinct routes that read a limit from the paper’s text, tables, figure image, or vector paths), ranked by how much of the output rests on model interpretation, with deterministic guard rails and validityfirst candidate selection; (3) convention canonicalization in a single vetted conversion registry that converts only recognized conventions and flags the rest for human review; (4) integration through automated code generation 2

000 dex, while the architectural work, at a fixed model, cut the per-type mean from 1.46 to 0.43 dex (0.38 in the production configuration) The remaining large errors come from interpreting figures and selecting the correct curve, mass range, or coupling type We offer the pipeline, the benchmark, and the measured design lessons collected in Appendix A to the community whose infrastructure problem motivated them, as a bridge until machine-readable limit publication makes extraction unnecessary

Appendix A. Lessons Learned

Validity-first candidate selection is helpful Reading supplied data and vector coordinates directly allows exact or near-exact extraction with little model cost Deterministic guards on the model’s declared outputs stopped incorrect extractions before selection on 49 of 329 papers in the production benchmark Managing conversions in one place is helpful Requiring review for unknown conventions makes the uncertainty visible to reviewers Checking curves against values in the text provides an inexpensive, independent check on a traced curve Expert review before merging is important Separating model effects from pipeline changes and variation between runs is needed to establish the cause of an apparent improvement and whether it exceeds ordinary run-to-run variation

Appendix B. Selection Criteria and Scoring Details

The lexicographic quality tuple records the criteria below in a fixed order The first four check validity, so later preferences cannot outweigh a difference on these checks Agreement with an independent channel is preferred if another channel produces a comparable extraction agreeing within a factor of three (a ratio in [1/3, 3], or about 0.48 dex)

Appendix C. Convention Canonicalization in Detail

Three rules for conversion are detailed Identifying the output quantity is necessary so later processing does not have to infer them from their magnitudes Preserving the plotted values means the vision channel reads values without converting them and identifies the plotted quantity Managing conversions in one place ensures all pipeline conversions use one reviewed conversion registry Handling unusual data files involves removing very large values using a threshold of 1019 Requiring human review for unknown conventions flags the paper for human review and excludes them from numerical evaluation as convention gaps

Appendix D. Operational Safeguards

Four safeguards support routine operation Stopping when the service is unavailable prevents marking the current paper as processed or failed Checking which model is running at startup verifies the identity of the model actually in use before extraction begins Controlling costs involves retrying model calls with progressively longer delays when a rate limit is reached

Appendix E. Cost and Model Choice

The small model, Haiku 4.5, costs about ten times less per extraction than Fable 5 The estimated additional cost is less than a dollar or two per day at production volume We choose Fable 5 to reduce review work at this modest additional cost; for bulk evaluation, the cheaper frontier model would be preferable Running the production workflow with one read per paper (N=1) on five papers using Opus 4. p. 3

Where the paper says it

Each stage below is checked against the collaboration’s own limit data, with no curator digitization in between. Discovery and classification A daily Github workflow pulls new submissions from the arXiv feed and narrows them in three steps of increasing cost: a cheap2 keyword filter against a curated set of experiment names and coupling jargon; a short LLM 2We call a step “cheap” when it costs at most a fraction of a cent per paper, set against the cents-to-a-dollar of a full extraction 4

8 at standard prices cost approximately one dollar per day

Appendix F. Confidence Calibration and Repeatability

Confidence determines review priority, not whether review occurs The pipeline uses confidence to rank extractions for review and to break remaining ties in candidate selection Success rates under this criterion rise across the upper four confidence groups The ±0.04 dex run-to-run scale estimates the variation between repeated runs Confidence determines review priority, not whether review occurs

Data and Software Availability

The pipeline source code, the evaluation benchmark, the scoring code, the extraction prompts, and the benchmark artifacts from which every number and figure in this paper is computed are openly available at https://github.com/FaroutYLq/AutoAxionLimits The benchmark ran the pinned model snapshots claude-fable-5, claude-opus-4-8, and claude-haiku-4-5 as comparison arms in July–August 2026

Funding

Work at Washington University in St. Louis was supported in part by a McDonnell Center for the Space Sciences (MCSS) seed grant (PI: K. p. 0

Where the paper says it

Towards LLM-Powered Automation of a Dark Matter Constraint Repository Lanqing Yuan a,∗ , Karthik Ramanathan a aDepartment of Physics, Washington University in St. Louis, One Brookings Drive, St.

R.) This work was also supported by Dianzhuo Wang and TwentyTwo

CRediT Authorship Contribution Statement

Lanqing Yuan contributed Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Data curation, Visualization, Funding acquisition, Project administration, Writing – original draft Karthik Ramanathan contributed Supervision and Formal analysis

Declaration of Competing Interest

Lanqing Yuan reports financial support from Dianzhuo Wang and TwentyTwo The authors declare no other known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper

Declaration of Generative AI and AI-Assisted Technologies in the Manuscript Preparation Process

Under human direction, Claude models (through Claude Code, based on Claude Opus 4.8) assisted with writing and editing the pipeline code and with drafting and editing this manuscript OpenAI Codex assisted with revising the manuscript’s language, formatting, and references The authors reviewed and edited the resulting content and take full responsibility for all scientific claims, numerical results, figures, references, and the final text AI tools also form part of the research methods described in this paper

Acknowledgements

We thank Ciaran O’Hare, whose creation and years-long maintenance of the community AxionLimits repository [5] provide both the foundation this work builds on and the curated ground truth it is measured against We thank Yue Ma and Aobo Li for useful discussions

References

[1] C. B. Adams, et al., Axion dark matter, Snowmass 2021 White Paper (2022). p. 1

Where the paper says it

15 distinct coupling types (ways dark matter can interact with the Standard Model, e. axion–photon or dark-photon kinetic mixing) [4], each with its own units, parameterisations, and physics conventions.

arXiv:2203. p. 0
Where the paper says it

Limits are typically published as figures rather than machinereadable records, compounding the burden. We present a large language model (LLM) pipeline that monitors arXiv, extracts limit curves, integrates them as code, and opens pull requests for human review.

14923, doi:10.48550/arXiv. p. 0
Where the paper says it

Limits are typically published as figures rather than machinereadable records, compounding the burden. We present a large language model (LLM) pipeline that monitors arXiv, extracts limit curves, integrates them as code, and opens pull requests for human review.

2203.14923 [7] A Rohatgi, WebPlotDigitizer, https://automeris. p. 2
Where the paper says it

The system also includes a weekly preprint version checker. 1 The task sits in known territory: axion physicists digitise plots today with WebPlotDigitizer [7], and vision–language models such as DePlot [8] translate chart images into tables.

io, concept DOI; resolves to the latest version; accessed 9 September 2026 (2024). p. 2
Where the paper says it

with Abstract Syntax Tree (AST)-based insertion, Jupyter notebook updates, and plot regeneration; and (5) review via pull requests (PRs) with highlighted constraint plots; nothing merges without a human expert. The system also includes a weekly preprint version checker.

doi:10.5281/zenodo.592175 [8] F Liu, J M Eisenschlos, F Piccinno, S Krichene, C Pang, K Lee, M Joshi, W Chen, N Collier, Y Altun DePlot: One-shot visual language reasoning by plot-to-table translation in: Findings of the Association for Computational Linguistics: ACL 2023 [9] A Cakir, A Yerlikaya, From Experimental Limits to Physical Insight: A Retrieval-Augmented Multi-Agent Framework for Interpreting Searches Beyond the Standard Model, arXiv e-prints (2026) [10] E A Moreno, S Bright-Thonney, A Novak D Garcia P Harris, AI agents can already autonomously perform experimental high energy physics (2026). p. 2
Where the paper says it

1 The task sits in known territory: axion physicists digitise plots today with WebPlotDigitizer [7], and vision–language models such as DePlot [8] translate chart images into tables. The nearest particle physics system is a retrieval-augmented multi-agent framework for interpreting searches beyond the Standard Model [9], which operates on already-structured HEPData records; agentic LLM systems have also executed substantial portions of a collider analysis autonomously [10].

arXiv:2603. p. 0
Where the paper says it

Limits are typically published as figures rather than machinereadable records, compounding the burden. We present a large language model (LLM) pipeline that monitors arXiv, extracts limit curves, integrates them as code, and opens pull requests for human review.

20179 [11] C Albert, L Yuan J Harris R Basu Thakur A Bear K Berggren C Cappiello C Curwen P Day B Eom et al. p. 0
Where the paper says it

Towards LLM-Powered Automation of a Dark Matter Constraint Repository Lanqing Yuan a,∗ , Karthik Ramanathan a aDepartment of Physics, Washington University in St. Louis, One Brookings Drive, St.

, Dark matter searches with a 13 meV threshold superconducting sensor array, arXiv e-prints (2026) [12] H An, S Ge J Liu Z Lu Direct detection of dark photon dark matter with the James Webb Space Telescope, arXiv e-prints (2024) [13] Artifex Software, PyMuPDF, https://github. p. 3
Where the paper says it

Nothing merges without human approval. We illustrate each step below with one recent arXiv manuscript by the authors, run in the release configuration exactly as deployed: the QUALIPHIDE far-infrared hidden-photon search [11], a 13–500 meV dark-photon exclusion limit.

com/pymupdf/PyMuPDF [14] Physical Superintelligence PBC, Get Physics Done (GPD), https://github. p. 7
Where the paper says it

We prevent this through three rules, detailed in Appendix C: each extraction channel must accurately identify the quantity and convention of its output; the vision channel must return values exactly as plotted, without converting them; and all pipeline conversions must use one reviewed set of conversion definitions. The conversion factors were derived with the agentic physics assistant Get Physics Done (GPD) [14], rather than simply requested from a model without harness, then checked in code against the repository’s plotting source and verified against the cited primary literature [15, 16].

com/psi-oss/get-physics-done, version 1. p. 7
Where the paper says it

We prevent this through three rules, detailed in Appendix C: each extraction channel must accurately identify the quantity and convention of its output; the vision channel must return values exactly as plotted, without converting them; and all pipeline conversions must use one reviewed set of conversion definitions. The conversion factors were derived with the agentic physics assistant Get Physics Done (GPD) [14], rather than simply requested from a model without harness, then checked in code against the repository’s plotting source and verified against the cited primary literature [15, 16].

2.2; accessed 9 September 2026 (2026) [15] T Damour, J F Donoghue, Equivalence principle violations and couplings of a light dilaton, Phys. p. 0
Where the paper says it

edu (Lanqing Yuan ) arXiv:2606. 21658v2 [hep-ex] 9 Sep 2026

Rev. D 82 (2010) 084033. arXiv:1007. p. 0
Where the paper says it

Limits are typically published as figures rather than machinereadable records, compounding the burden. We present a large language model (LLM) pipeline that monitors arXiv, extracts limit curves, integrates them as code, and opens pull requests for human review.

2792 [16] L Di Luzio, M Giannotti, E Nardi L Visinelli The landscape of QCD axion models, Phys. p. 2
Where the paper says it

The system also includes a weekly preprint version checker. 1 The task sits in known territory: axion physicists digitise plots today with WebPlotDigitizer [7], and vision–language models such as DePlot [8] translate chart images into tables.

Rept. 870 (2020) 1–117. arXiv:2003. p. 0
Where the paper says it

Limits are typically published as figures rather than machinereadable records, compounding the burden. We present a large language model (LLM) pipeline that monitors arXiv, extracts limit curves, integrates them as code, and opens pull requests for human review.

01100 [17] G Cowan, K Cranmer E Gross O Vitells Power-Constrained Limits, arXiv e-prints (2011). p. 5
Where the paper says it

Because that figure is a vector graphic, the deterministic vector_trace channel could extract the curve geometry directly from its PDF paths: among three high-confidence path candidates the single cheap identification call picked “the dark solid observed-limit curve spanning 13–487 meV,” yielding a 124-point candidate at confidence 0. The text channel independently contributed the paper’s only numeric bound (the power-constrained χ = 1.

arXiv:1105. p. 0
Where the paper says it

Limits are typically published as figures rather than machinereadable records, compounding the burden. We present a large language model (LLM) pipeline that monitors arXiv, extracts limit curves, integrates them as code, and opens pull requests for human review.

3166 [27] C O’Hare, L Yuan FaroutYLq/AutoAxionLimits: AutoAxionLimit v0. p. 2
Where the paper says it

1Code, evaluation benchmark, and prompts: https://github. com/FaroutYLq/ AutoAxionLimits 3

2.0, Zenodo, version AutoAxionLimit v0. p. 2
Where the paper says it

with Abstract Syntax Tree (AST)-based insertion, Jupyter notebook updates, and plot regeneration; and (5) review via pull requests (PRs) with highlighted constraint plots; nothing merges without a human expert. The system also includes a weekly preprint version checker.

2.0 [software] (Aug 2026). p. 1
Where the paper says it

This is why curation has stayed manual: it is closer to an afternoon of a doctoral student’s careful physics bookkeeping than to a parsing task, and it must be redone for every new paper. We treat this as a data-processing and software-methods problem rather than an “LLM demo”: the questions that matter are reproducibility, quantified accuracy, and auditable provenance.

doi:10.5281/zenodo.21929941 [38] Anthropic, The Claude model family: Fable 5, Opus 4. p. 0
Where the paper says it

Limits are typically published as figures rather than machinereadable records, compounding the burden. We present a large language model (LLM) pipeline that monitors arXiv, extracts limit curves, integrates them as code, and opens pull requests for human review.

8, and Haiku 4.5 model documentation, https://docs. p. 2
Where the paper says it

Its design principle, arrived at empirically (Appendix A), is to keep the LLM confined to perception and semantics (reading text, tables, axes, and curves, and declaring what it read) while arithmetic, unit conversion, and candidate selection are deterministic, testable code. 1Code, evaluation benchmark, and prompts: https://github.

anthropic.com, models used: claude-fable-5 claude-opus-4-8 and claude-haiku-4-5; accessed 9 September 2026 (2026) [18]

Table B.1 Summary

Comparison was attempted for 271 papers, recorded under the software status compared Of these, 267 yielded finite residuals and are called “compared papers” in the main text; only these enter the residual statistics

Table C.1 Representative Examples of Per-File Convention Conversions

The factors were derived with GPD [14], checked in code against the repository’s plotting source, and verified against the cited literature

Figure 2 Reviewing the Highlighted Plot Produced by the Case-Study Run

A reviewer can judge the proposal’s plausibility in seconds without reading code The inset zooms into the QUALIPHIDE-FIR band and overlays the run’s artifacts directly, all paper-native at ρDM = 0.45 GeV/cm3 This figure demonstrates the curation workflow and should not be cited as a particle-physics result

Figure 4 Cumulative Distribution of Per-Paper Median Residuals for the Three Arms

The two frontier arms are nearly superimposed; the model-tier gap to the small model is visible as a rightward shift, and the small model’s heavier tail is the operationally relevant difference

Figure 5 Median Residual by Coupling Type with Bootstrap 95% CIs (Fable 5)

The micro–macro gap is concentrated in rare, convention-heavy types Small-sample types (N < 5) are greyed The pattern is not simply that rare is hard

Figure F.1 Confidence and Measured Accuracy for the Production Configuration

Accuracy rises across the upper four confidence groups, and the two highest groups lie close to the line where confidence equals accuracy Confidence determines review priority, not whether review occurs

Figure 3 Per-Paper Run-to-Run Change in the Median Coupling Residual

The distribution is bimodal: a stable core (blue; 49 papers reproduce to <0.03 dex) and a heavy routing-flip tail (orange; 47 papers, of which 12 route to a different extraction channel between runs)

**Figure F.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Precision extraction through multi-channel architecture: The system can utilize five competing channels (text, table, figure image, vector paths) ranked by semantic trust tiers to generate candidates, as described in section 2.2 of the paper.

  2. Deterministic guardrails for physical plausibility: The pipeline includes deterministic guards that reject unsuitable candidates or lower their priority based on the mass range stated in the abstract and checks that curves extracted visually or from PDF vector paths are also rejected if they differ from an in-range numerical bound in the text by more than a factor of 100 (2 dex).

  3. Automated convention canonicalization: The system can standardize physics parameters by using a single vetted conversion registry and flag cases requiring human review, as detailed in section 2.5 of the paper, preventing a silent multi-decade error.

  4. Validated candidate selection logic: The system employs a lexicographic quality tuple for candidate scoring, prioritizing checks in order: Physical validity and interpretable units and conventions take priority, followed by channel tier and agreement with an independent channel.

  5. Model effect isolation: The system can isolate the LLM's impact from pipeline changes by running identical code on different models to measure the effect, as described in section 4.2 of the paper, allowing for a clean model measurement.

  6. Run-to-run variation measurement: The system can measure stochasticity by comparing paired runs and identifying channel switching using metrics like |Delta log10 g| [dex], providing a noise floor that is used to assess repeatability.

  7. Human review prioritization: The system can route extracted results to human reviewers based on confidence tiers, where low-confidence extractions still receive human review, and the final decision rests with the expert before merging.

Abstract

Dark matter constraint repositories are critical community infrastructure, yet the most widely-used are volunteer-maintained, a sustainability risk as results accelerate. Limits are typically published as figures rather than machine-readable records, compounding the burden. We present a large language model (LLM) system that monitors arXiv, extracts limit curves, integrates them as code, and opens pull requests for human review. Extraction uses a tool-using agent that interprets physical conventions and checks its results, with a fixed multi-channel pipeline evaluated as an alternative. We evaluate both approaches on a 329-paper benchmark of measured limits graded against the upstream-curated repository. An extraction is considered successful when the extracted and reference mass ranges overlap by more than 90%, with at least 90% of compared values agreeing within a factor of two. For Fable 5, 52.6% of eligible curve comparisons meet this criterion with the agent, compared with 20.9% with the pipeline. Deployed, the pipeline has generated limit proposals, but none have merged: governance of AI-generated scientific data is itself unsolved.

Sources

Related papers