Towards LLM-Powered Automation of a Dark Matter Constraint Repository
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: High Energy Physics: Experiment. The day in high-energy experiment: detectors, searches and limits.
Kai: I'm Kai.
Mira: And I'm Mira, here with Lev.
Kai: We're looking at the paper: "Towards LLM-Powered Automation of a Dark Matter Constraint Repository".
Mira: Here's the gist of the paper. Dark matter constraint repositories are critical community infrastructure, yet the most widely-used are volunteer-maintained, a sustainability risk as results accelerate We present a large language model (LLM) pipeline that monitors arXiv, extracts limit curves, integrates them as code, and opens pull requests for human review On a 329-paper benchmark of measured limits graded against the upstream-curated repository, it identifies the coupling type for 98.1% of gradable papers The LLM is confined to perception; arithmetic, unit conversion, and the guards that reject mis-extractions are deterministic, testable code Deployed, the pipeline has generated limit proposals, but none have merged: governance of AI-generated scientific data is itself unsolved System Architecture Figure 1 shows the end-to-end pipeline Its design principle, arrived at empirically (Appendix A), is to keep the LLM confined to perception and semantics (reading text, tables, axes, and curves, and declaring what it read) while arithmetic, unit conversion, and candidate selection are deterministic, testable code The pipeline flows from arXiv (top-left) through keyword and LLM triage into five competing extraction channels Rule-based stages also use model-generated information, so the colours do not imply a strict separation of LLM use and code Multi-channel Extraction An extraction channel is a method for obtaining limit values from a paper’s text, tables, figures, or accompanying data files The agent produces candidates through up to five channels of differing semantic trust (table 1), staged as: a cheap particle physics interaction type pre-classification pass; a text/table read of the parsed PDF (PyMuPDF
13: ); a cheap axis-calibration pass that identifies the relevant figure and reads its axis labels and ranges; and a full vision trace of the rendered figure pages The channel selector considers tier only after all validity criteria, preferring higher-tier channels among equally valid candidates Validity and Canonicalization The selector compares candidates using a fixed sequence of checks The first check on which two candidates differ determines which is preferred Physical validity and interpretable units and conventions take priority Convention canonicalization converts quantities to the repository’s standard variable and convention Any quantity or convention outside the reviewed set requires human review Evaluation Methodology We evaluate on 329 papers spanning the repository’s coupling types, chosen as curated repository data files
5: that carry a known arXiv ID (the full registry holds 437 file entries over 347 distinct arXiv IDs) The ground truth (GT) for each paper is the upstream repository’s own digitized curve We compare in log10–log10 space, using g to denote the coupling variable after both curves have been converted to the same convention A paper’s residual is the median of ri over the compared mass points Model Performance and Lessons The production configuration achieves a median per-paper residual of 0.17 dex across 267 compared papers The key gains came from the system built around the model: extraction methods and checks implemented in code, centrally managed physical variable conversions with human review for unsupported conventions, and an evaluation that separates model effects from code changes and measures variation between runs The remaining large errors come from interpreting figures and selecting the correct curve, mass range, or coupling type The pipeline simplifies maintenance rather than replaces the maintainer: it turns curation from authoring into reviewing, and with 51% of papers within a factor of two end-to-end the expert’s accept-or-reject judgement remains the load-bearing step, by design and by necessity Limitations and Future Work The largest errors came from mishandling units and variables, rather than misreading figures The micro–macro gap is carried by coupling types with few papers Further improvement requires better interpretation of these rare limits and their conventions The gap is not tooling but adoption in the light-dark-matter community, where results are still published predominantly as figures The infrastructure for machinereadable limit publication already exists and is in routine use: HEPData
24: archives digitized collider results, and the LHC Reinterpretation Forum has published recommendations for reusable result presentation
25: We delivered an LLM-powered pipeline with deterministic guards that curates the live dark matter limit literature, with every output reviewed by a human before anything merges On a 329-paper benchmark against curator ground truth, it reaches a median accuracy of 0.17±0.04 dex among compared measured-limit papers The key gains came from the system built around the model: extraction methods and checks implemented in code, centrally managed physical variable conversions with human review for unsupported conventions, and an evaluation that separates model effects from code changes and measures variation between runs The benchmark makes that claim measurable: swapping the extraction model for another frontier model a generation apart moves the paired per-paper median by 0.000 dex, while the architectural work, at a fixed model, cut the per-type mean from 1.46 to 0.43 dex (0.38 in the production configuration) The remaining large errors come from interpreting figures and selecting the correct curve, mass range, or coupling type We offer the pipeline, the benchmark, and the measured design lessons collected in Appendix A to the community whose infrastructure problem motivated them, as a bridge until machine-readable limit publication makes extraction unnecessary Appendix A.
Kai: Lessons Learned Validity-first candidate selection is helpful Reading supplied data and vector coordinates directly allows exact or near-exact extraction with little model cost Deterministic guards on the model’s declared outputs stopped incorrect extractions before selection on 49 of 329 papers in the production benchmark Managing conversions in one place is helpful Requiring review for unknown conventions makes the uncertainty visible to reviewers Checking curves against values in the text provides an inexpensive, independent check on a traced curve Expert review before merging is important Separating model effects from pipeline changes and variation between runs is needed to establish the cause of an apparent improvement and whether it exceeds ordinary run-to-run variation Appendix B.
Mira: Selection Criteria and Scoring Details The lexicographic quality tuple records the criteria below in a fixed order The first four check validity, so later preferences cannot outweigh a difference on these checks Agreement with an independent channel is preferred if another channel produces a comparable extraction agreeing within a factor of three (a ratio in
1/3, 3: , or about 0.48 dex) Appendix C.
Kai: Convention Canonicalization in Detail Three rules for conversion are detailed Identifying the output quantity is necessary so later processing does not have to infer them from their magnitudes Preserving the plotted values means the vision channel reads values without converting them and identifies the plotted quantity Managing conversions in one place ensures all pipeline conversions use one reviewed conversion registry Handling unusual data files involves removing very large values using a threshold of 1019 Requiring human review for unknown conventions flags the paper for human review and excludes them from numerical evaluation as convention gaps Appendix D.
Paper summary: Kai: The core idea here is to use an LLM pipeline that reads arXiv papers, extracts the limit curves, and integrates them as code so they can be reviewed by humans before anything gets merged into the main repository.
Mira: They found that on a benchmark of three hundred twenty-nine papers graded against the curated repository, this system identified the coupling type for ninety-eight point one percent of those gradable papers <ref:2606.21658#pg1>.
Lev: That's a lot of coverage, but we need to know if that extraction is reliable when we talk about real physics constraints.
Kai: The paper emphasizes that the LLM itself is only good at perception, like reading text and axes, while the actual math and unit conversions are handled by deterministic code.
Mira: They show that this setup has a median residual of zero point one seven dex for compared papers, which means it’s relatively accurate when you compare its output to the ground truth.
Lev: And they stress that there are these deterministic guards built in to reject bad extractions before anything gets selected.
Kai: The paper claims that this system has generated limit proposals, but none of them have been merged yet, which points to the next big question about governance of AI-generated scientific data.
Mira: It’s about turning the curation work from being done by authors into something that is reviewed by experts, with fifty-one percent of all papers landing within a factor of two of the curator's curve.
Lev: From an engineering standpoint, they built this to be testable and confined; the model doesn't do arithmetic or unit conversions, so you know exactly what it’s doing.
Kai: It sounds like they’re trying to solve a sustainability problem for this community infrastructure by automating the tedious parts of digitizing and converting data.
Conclusion: Mira: So, to wrap up this paper, "Towards LLM-Powered Automation of a Dark Matter Constraint Repository" it presents an end-to-end LLM pipeline with deterministic guards that curates the live dark matter literature, making every output human reviewed before merging.
Kai: The main thing they found on their benchmark is that among compared measured limit papers against the curator ground truth, this system reaches a median accuracy of zero point one seven dex among those papers.
Lev: It’s important to note that while the pipeline generated proposals, none have merged yet, which highlights that governance of AI-generated scientific data is still an open question.
Mira: The real gain they highlight isn't just the model itself, but the system built around it: extraction methods and checks implemented in code.
Kai: Swapping out the extraction model for another frontier model a generation apart only moved their paired per-paper median by zero point zero zero zero dex, while changing the architectural work cut that mean from one point four six to a three point eight in production configuration.
Lev: So it sounds like the architecture and how you handle conversions is more important than just picking the latest big language model for this job.
Kai: The paper offers the pipeline, and the benchmark, and all those measured design lessons to the community that needed this infrastructure problem solved as a bridge until machine-readable limit publication makes extraction unnecessary.
Mira: It’s about making sure that even though we have these powerful new tools, the expert's accept or reject judgment remains the necessary step by design and by necessity.
Department of Physics, Washington University in St Louis
hep-ex, astro-ph.IM, hep-ph
Submitted: 2026-06-19
Updated: 2026-11-04
Code: https://github.com/FaroutYLq/AutoAxionLimits
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 88/100
The gist: Dark matter constraint repositories are critical community infrastructure, yet the most widely-used are volunteer-maintained…
Key concepts
- Multi-channel Extraction
- This involves using up to five different methods to pull limit values from a single paper, such as reading text, parsing tables, checking axis labels, or performing a full vision trace of figures. The system compares these different methods and selects the best candidate based on predefined validity checks.
- Validity and Canonicalization
- This is the process of rigorously checking extracted data against physical reality and standard conventions. It involves converting all quantities to a single, standardized format, flagging any unknown conventions for human review, and prioritizing physically valid extractions over others.
- Model-Confined Perception
- The LLM's role is strictly limited to understanding the semantics of the text—reading what it sees—not performing complex calculations or unit conversions. All arithmetic and critical checks are handled by deterministic, testable code, ensuring the model only contributes perception.
- Governance of AI-Generated Data
- The paper highlights that while an LLM can generate proposals, human governance remains unsolved. The pipeline's success depends on human review before any data is merged into the official repository, emphasizing that expert judgment is the final necessary step.
Terminology
Summary
Dark matter constraint repositories are critical community infrastructure, yet the most widely-used are volunteer-maintained, a sustainability risk as results accelerate We present a large language model (LLM) pipeline that monitors arXiv, extracts limit curves, integrates them as code, and opens pull requests for human review On a 329-paper benchmark of measured limits graded against the upstream-curated repository, it identifies the coupling type for 98. p. 0 We present a large language
model (LLM) pipeline that monitors arXiv, extracts limit curves, integrates
them as code, and opens pull requests for human review. On a 329-paper
benchmark of measured limits graded against the upstream-curated repository,
it identifies the coupling type for 98.Where the paper says it
System Architecture
Figure 1 shows the end-to-end pipeline Its design principle, arrived at empirically (Appendix A), is to keep the LLM confined to perception and semantics (reading text, tables, axes, and curves, and declaring what it read) while arithmetic, unit conversion, and candidate selection are deterministic, testable code The pipeline flows from arXiv (top-left) through keyword and LLM triage into five competing extraction channels Rule-based stages also use model-generated information, so the colours do not imply a strict separation of LLM use and code
Multi-channel Extraction
An extraction channel is a method for obtaining limit values from a paper’s text, tables, figures, or accompanying data files The agent produces candidates through up to five channels of differing semantic trust (table 1), staged as: a cheap particle physics interaction type pre-classification pass; a text/table read of the parsed PDF (PyMuPDF [13]); a cheap axis-calibration pass that identifies the relevant figure and reads its axis labels and ranges; and a full vision trace of the rendered figure pages The channel selector considers tier only after all validity criteria, preferring higher-tier channels among equally valid candidates
Validity and Canonicalization
The selector compares candidates using a fixed sequence of checks The first check on which two candidates differ determines which is preferred Physical validity and interpretable units and conventions take priority Convention canonicalization converts quantities to the repository’s standard variable and convention Any quantity or convention outside the reviewed set requires human review
Evaluation Methodology
We evaluate on 329 papers spanning the repository’s coupling types, chosen as curated repository data files [5] that carry a known arXiv ID (the full registry holds 437 file entries over 347 distinct arXiv IDs) The ground truth (GT) for each paper is the upstream repository’s own digitized curve We compare in log10–log10 space, using g to denote the coupling variable after both curves have been converted to the same convention A paper’s residual is the median of ri over the compared mass points
Model Performance and Lessons
The production configuration achieves a median per-paper residual of 0.17 dex across 267 compared papers The key gains came from the system built around the model: extraction methods and checks implemented in code, centrally managed physical variable conversions with human review for unsupported conventions, and an evaluation that separates model effects from code changes and measures variation between runs The remaining large errors come from interpreting figures and selecting the correct curve, mass range, or coupling type The pipeline simplifies maintenance rather than replaces the maintainer: it turns curation from authoring into reviewing, and with 51% of papers within a factor of two end-to-end the expert’s accept-or-reject judgement remains the load-bearing step, by design and by necessity
Limitations and Future Work
The largest errors came from mishandling units and variables, rather than misreading figures The micro–macro gap is carried by coupling types with few papers Further improvement requires better interpretation of these rare limits and their conventions The gap is not tooling but adoption in the light-dark-matter community, where results are still published predominantly as figures The infrastructure for machinereadable limit publication already exists and is in routine use: HEPData [24] archives digitized collider results, and the LHC Reinterpretation Forum has published recommendations for reusable result presentation [25]
Conclusions
We delivered an LLM-powered pipeline with deterministic guards that curates the live dark matter limit literature, with every output reviewed by a human before anything merges On a 329-paper benchmark against curator ground truth, it reaches a median accuracy of 0.17±0.04 dex among compared measured-limit papers The key gains came from the system built around the model: extraction methods and checks implemented in code, centrally managed physical variable conversions with human review for unsupported conventions, and an evaluation that separates model effects from code changes and measures variation between runs The benchmark makes that claim measurable: swapping the extraction model for another frontier model a generation apart moves the paired per-paper median by 0. p. 1 We treat this as a data-processing and software-methods problem rather
than an “LLM demo”: the questions that matter are reproducibility, quantified
accuracy, and auditable provenance. We present a pipeline that partially
automates the curation workflow: (1) discovery via daily arXiv monitoring
with keyword filtering and LLM relevance classification; (2) extraction through
five competing channels (distinct routes that read a limit from the paper’s
text, tables, figure image, or vector paths), ranked by how much of the output
rests on model interpretation, with deterministic guard rails and validityfirst candidate selection; (3) convention canonicalization in a single vetted
conversion registry that converts only recognized conventions and flags the
rest for human review; (4) integration through automated code generation
2Where the paper says it
Appendix A. Lessons Learned
Validity-first candidate selection is helpful Reading supplied data and vector coordinates directly allows exact or near-exact extraction with little model cost Deterministic guards on the model’s declared outputs stopped incorrect extractions before selection on 49 of 329 papers in the production benchmark Managing conversions in one place is helpful Requiring review for unknown conventions makes the uncertainty visible to reviewers Checking curves against values in the text provides an inexpensive, independent check on a traced curve Expert review before merging is important Separating model effects from pipeline changes and variation between runs is needed to establish the cause of an apparent improvement and whether it exceeds ordinary run-to-run variation
Appendix B. Selection Criteria and Scoring Details
The lexicographic quality tuple records the criteria below in a fixed order The first four check validity, so later preferences cannot outweigh a difference on these checks Agreement with an independent channel is preferred if another channel produces a comparable extraction agreeing within a factor of three (a ratio in [1/3, 3], or about 0.48 dex)
Appendix C. Convention Canonicalization in Detail
Three rules for conversion are detailed Identifying the output quantity is necessary so later processing does not have to infer them from their magnitudes Preserving the plotted values means the vision channel reads values without converting them and identifies the plotted quantity Managing conversions in one place ensures all pipeline conversions use one reviewed conversion registry Handling unusual data files involves removing very large values using a threshold of 1019 Requiring human review for unknown conventions flags the paper for human review and excludes them from numerical evaluation as convention gaps
Appendix D. Operational Safeguards
Four safeguards support routine operation Stopping when the service is unavailable prevents marking the current paper as processed or failed Checking which model is running at startup verifies the identity of the model actually in use before extraction begins Controlling costs involves retrying model calls with progressively longer delays when a rate limit is reached
Appendix E. Cost and Model Choice
The small model, Haiku 4.5, costs about ten times less per extraction than Fable 5 The estimated additional cost is less than a dollar or two per day at production volume We choose Fable 5 to reduce review work at this modest additional cost; for bulk evaluation, the cheaper frontier model would be preferable Running the production workflow with one read per paper (N=1) on five papers using Opus 4. p. 3 Each stage
below is checked against the collaboration’s own limit data, with no curator
digitization in between. Discovery and classification
A daily Github workflow pulls new submissions from the arXiv feed
and narrows them in three steps of increasing cost: a cheap2 keyword filter
against a curated set of experiment names and coupling jargon; a short LLM
2We call a step “cheap” when it costs at most a fraction of a cent per paper, set against
the cents-to-a-dollar of a full extraction
4Where the paper says it
Appendix F. Confidence Calibration and Repeatability
Confidence determines review priority, not whether review occurs The pipeline uses confidence to rank extractions for review and to break remaining ties in candidate selection Success rates under this criterion rise across the upper four confidence groups The ±0.04 dex run-to-run scale estimates the variation between repeated runs Confidence determines review priority, not whether review occurs
Data and Software Availability
The pipeline source code, the evaluation benchmark, the scoring code, the extraction prompts, and the benchmark artifacts from which every number and figure in this paper is computed are openly available at https://github.com/FaroutYLq/AutoAxionLimits The benchmark ran the pinned model snapshots claude-fable-5, claude-opus-4-8, and claude-haiku-4-5 as comparison arms in July–August 2026
Funding
Work at Washington University in St. Louis was supported in part by a McDonnell Center for the Space Sciences (MCSS) seed grant (PI: K. p. 0 Towards LLM-Powered Automation of a Dark Matter
Constraint Repository
Lanqing Yuan a,∗
, Karthik Ramanathan a
aDepartment of Physics, Washington University in St. Louis, One Brookings Drive, St.Where the paper says it
CRediT Authorship Contribution Statement
Lanqing Yuan contributed Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Data curation, Visualization, Funding acquisition, Project administration, Writing – original draft Karthik Ramanathan contributed Supervision and Formal analysis
Declaration of Competing Interest
Lanqing Yuan reports financial support from Dianzhuo Wang and TwentyTwo The authors declare no other known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper
Declaration of Generative AI and AI-Assisted Technologies in the Manuscript Preparation Process
Under human direction, Claude models (through Claude Code, based on Claude Opus 4.8) assisted with writing and editing the pipeline code and with drafting and editing this manuscript OpenAI Codex assisted with revising the manuscript’s language, formatting, and references The authors reviewed and edited the resulting content and take full responsibility for all scientific claims, numerical results, figures, references, and the final text AI tools also form part of the research methods described in this paper
Acknowledgements
We thank Ciaran O’Hare, whose creation and years-long maintenance of the community AxionLimits repository [5] provide both the foundation this work builds on and the curated ground truth it is measured against We thank Yue Ma and Aobo Li for useful discussions
References
[1] C. B. Adams, et al., Axion dark matter, Snowmass 2021 White Paper (2022). p. 1 15 distinct coupling types (ways dark matter can interact with the Standard
Model, e. axion–photon or dark-photon kinetic mixing) [4], each with its own
units, parameterisations, and physics conventions. Limits are typically published as figures rather than machinereadable records, compounding the burden. We present a large language
model (LLM) pipeline that monitors arXiv, extracts limit curves, integrates
them as code, and opens pull requests for human review. Limits are typically published as figures rather than machinereadable records, compounding the burden. We present a large language
model (LLM) pipeline that monitors arXiv, extracts limit curves, integrates
them as code, and opens pull requests for human review. The system also
includes a weekly preprint version checker. 1
The task sits in known territory: axion physicists digitise plots today
with WebPlotDigitizer [7], and vision–language models such as DePlot [8]
translate chart images into tables. with Abstract Syntax Tree (AST)-based insertion, Jupyter notebook updates,
and plot regeneration; and (5) review via pull requests (PRs) with highlighted
constraint plots; nothing merges without a human expert. The system also
includes a weekly preprint version checker. 1
The task sits in known territory: axion physicists digitise plots today
with WebPlotDigitizer [7], and vision–language models such as DePlot [8]
translate chart images into tables. The nearest particle physics system
is a retrieval-augmented multi-agent framework for interpreting searches
beyond the Standard Model [9], which operates on already-structured HEPData
records; agentic LLM systems have also executed substantial portions of a
collider analysis autonomously [10]. Limits are typically published as figures rather than machinereadable records, compounding the burden. We present a large language
model (LLM) pipeline that monitors arXiv, extracts limit curves, integrates
them as code, and opens pull requests for human review. Towards LLM-Powered Automation of a Dark Matter
Constraint Repository
Lanqing Yuan a,∗
, Karthik Ramanathan a
aDepartment of Physics, Washington University in St. Louis, One Brookings Drive, St. Nothing merges without
human approval. We illustrate each step below with one recent arXiv manuscript by the authors, run in the release configuration exactly as deployed: the QUALIPHIDE
far-infrared hidden-photon search [11], a 13–500 meV dark-photon exclusion
limit. We prevent this through three rules,
detailed in Appendix C: each extraction channel must accurately identify the
quantity and convention of its output; the vision channel must return values
exactly as plotted, without converting them; and all pipeline conversions
must use one reviewed set of conversion definitions. The conversion factors
were derived with the agentic physics assistant Get Physics Done (GPD) [14],
rather than simply requested from a model without harness, then checked in
code against the repository’s plotting source and verified against the cited
primary literature [15, 16]. We prevent this through three rules,
detailed in Appendix C: each extraction channel must accurately identify the
quantity and convention of its output; the vision channel must return values
exactly as plotted, without converting them; and all pipeline conversions
must use one reviewed set of conversion definitions. The conversion factors
were derived with the agentic physics assistant Get Physics Done (GPD) [14],
rather than simply requested from a model without harness, then checked in
code against the repository’s plotting source and verified against the cited
primary literature [15, 16]. edu (Lanqing Yuan )
arXiv:2606. 21658v2 [hep-ex] 9 Sep 2026 Limits are typically published as figures rather than machinereadable records, compounding the burden. We present a large language
model (LLM) pipeline that monitors arXiv, extracts limit curves, integrates
them as code, and opens pull requests for human review. The system also
includes a weekly preprint version checker. 1
The task sits in known territory: axion physicists digitise plots today
with WebPlotDigitizer [7], and vision–language models such as DePlot [8]
translate chart images into tables. Limits are typically published as figures rather than machinereadable records, compounding the burden. We present a large language
model (LLM) pipeline that monitors arXiv, extracts limit curves, integrates
them as code, and opens pull requests for human review. Because that figure is a
vector graphic, the deterministic vector_trace channel could extract the
curve geometry directly from its PDF paths: among three high-confidence
path candidates the single cheap identification call picked “the dark solid
observed-limit curve spanning 13–487 meV,” yielding a 124-point candidate
at confidence 0. The text channel independently contributed the paper’s
only numeric bound (the power-constrained χ = 1. Limits are typically published as figures rather than machinereadable records, compounding the burden. We present a large language
model (LLM) pipeline that monitors arXiv, extracts limit curves, integrates
them as code, and opens pull requests for human review. 1Code, evaluation benchmark, and prompts: https://github. com/FaroutYLq/
AutoAxionLimits
3 with Abstract Syntax Tree (AST)-based insertion, Jupyter notebook updates,
and plot regeneration; and (5) review via pull requests (PRs) with highlighted
constraint plots; nothing merges without a human expert. The system also
includes a weekly preprint version checker. This is why curation has stayed manual: it is closer to an afternoon of a
doctoral student’s careful physics bookkeeping than to a parsing task, and it
must be redone for every new paper. We treat this as a data-processing and software-methods problem rather
than an “LLM demo”: the questions that matter are reproducibility, quantified
accuracy, and auditable provenance. Limits are typically published as figures rather than machinereadable records, compounding the burden. We present a large language
model (LLM) pipeline that monitors arXiv, extracts limit curves, integrates
them as code, and opens pull requests for human review. Its design principle, arrived at
empirically (Appendix A), is to keep the LLM confined to perception and
semantics (reading text, tables, axes, and curves, and declaring what it read)
while arithmetic, unit conversion, and candidate selection are deterministic,
testable code. 1Code, evaluation benchmark, and prompts: https://github.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Table B.1 Summary
Comparison was attempted for 271 papers, recorded under the software status compared Of these, 267 yielded finite residuals and are called “compared papers” in the main text; only these enter the residual statistics
Table C.1 Representative Examples of Per-File Convention Conversions
The factors were derived with GPD [14], checked in code against the repository’s plotting source, and verified against the cited literature
Figure 2 Reviewing the Highlighted Plot Produced by the Case-Study Run
A reviewer can judge the proposal’s plausibility in seconds without reading code The inset zooms into the QUALIPHIDE-FIR band and overlays the run’s artifacts directly, all paper-native at ρDM = 0.45 GeV/cm3 This figure demonstrates the curation workflow and should not be cited as a particle-physics result
Figure 4 Cumulative Distribution of Per-Paper Median Residuals for the Three Arms
The two frontier arms are nearly superimposed; the model-tier gap to the small model is visible as a rightward shift, and the small model’s heavier tail is the operationally relevant difference
Figure 5 Median Residual by Coupling Type with Bootstrap 95% CIs (Fable 5)
The micro–macro gap is concentrated in rare, convention-heavy types Small-sample types (N < 5) are greyed The pattern is not simply that rare is hard
Figure F.1 Confidence and Measured Accuracy for the Production Configuration
Accuracy rises across the upper four confidence groups, and the two highest groups lie close to the line where confidence equals accuracy Confidence determines review priority, not whether review occurs
Figure 3 Per-Paper Run-to-Run Change in the Median Coupling Residual
The distribution is bimodal: a stable core (blue; 49 papers reproduce to <0.03 dex) and a heavy routing-flip tail (orange; 47 papers, of which 12 route to a different extraction channel between runs)
**Figure F.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Precision extraction through multi-channel architecture: The system can utilize five competing channels (text, table, figure image, vector paths) ranked by semantic trust tiers to generate candidates, as described in section 2.2 of the paper.
-
Deterministic guardrails for physical plausibility: The pipeline includes deterministic guards that reject unsuitable candidates or lower their priority based on
the mass range stated in the abstract
and checks thatcurves extracted visually or from PDF vector paths are also rejected if they differ from an in-range numerical bound in the text by more than a factor of 100 (2 dex).
-
Automated convention canonicalization: The system can standardize physics parameters by using a
single vetted conversion registry
and flag cases requiring human review, as detailed in section 2.5 of the paper, preventinga silent multi-decade error.
-
Validated candidate selection logic: The system employs a lexicographic quality tuple for candidate scoring, prioritizing checks in order:
Physical validity and interpretable units and conventions take priority,
followed by channel tier and agreement with an independent channel. -
Model effect isolation: The system can isolate the LLM's impact from pipeline changes by running identical code on different models to measure the effect, as described in section 4.2 of the paper, allowing for a
clean model measurement.
-
Run-to-run variation measurement: The system can measure stochasticity by comparing paired runs and identifying channel switching using metrics like
|Delta log10 g| [dex],
providing anoise floor
that is used to assess repeatability. -
Human review prioritization: The system can route extracted results to human reviewers based on confidence tiers, where low-confidence extractions
still receive human review,
and the final decision rests with the expert before merging.
Abstract
Dark matter constraint repositories are critical community infrastructure, yet the most widely-used are volunteer-maintained, a sustainability risk as results accelerate. Limits are typically published as figures rather than machine-readable records, compounding the burden. We present a large language model (LLM) system that monitors arXiv, extracts limit curves, integrates them as code, and opens pull requests for human review. Extraction uses a tool-using agent that interprets physical conventions and checks its results, with a fixed multi-channel pipeline evaluated as an alternative. We evaluate both approaches on a 329-paper benchmark of measured limits graded against the upstream-curated repository. An extraction is considered successful when the extracted and reference mass ranges overlap by more than 90%, with at least 90% of compared values agreeing within a factor of two. For Fable 5, 52.6% of eligible curve comparisons meet this criterion with the agent, compared with 20.9% with the pipeline. Deployed, the pipeline has generated limit proposals, but none have merged: governance of AI-generated scientific data is itself unsolved.
Sources
- Axion Dark Matter
- New experimental approaches in the search for axion-like particles
- Dark photon limits: a handbook
- Cosmology of axion dark matter
- DePlot: One-shot visual language reasoning by plot-to-table translation
- From Experimental Limits to Physical Insight: A Retrieval-Augmented Multi-Agent Framework for Interpreting Searches Beyond the Standard Model
- AI Agents Can Already Autonomously Perform Experimental High Energy Physics
- Dark matter searches with a 13 meV threshold superconducting sensor array
- Direct Detection of Dark Photon Dark Matter with the James Webb Space Telescope
- Equivalence Principle Violations and Couplings of a Light Dilaton
- The landscape of QCD axion models
- Power-Constrained Limits
- Properties of Ultralight Bosons from Spins of Heavy Quasars via Superradiance
- First Constraints on Nuclear Coupling of Axionlike Particles from the Binary Neutron Star Gravitational Wave Event GW170817
- Oscillating nuclear charge radii as sensors for ultralight dark matter
- Pi in the Sky: Neutron Stars with Exceptionally Light QCD Axions
- HEPData: a repository for high energy physics data
- Reinterpretation of LHC Results for New Physics: Status and Recommendations after Run 2
- Short-range spin-dependent interactions of electrons: a probe for exotic pseudo-Goldstone bosons
Related papers
- DrSai: an agentic AI system toward autonomous physics analysis workflows at BESIII
- CANTON- mu Proposal: A Next-Generation Muon g-2 Measurement at Sub-0.1 ppm Precision
- Search for dark photon dark matter near 1,meV using a superconducting mixer
- ANUBIS detector and its sensitivity to neutral long-lived particles
- How to scale your HEP ML models: A recipe for robust architecture comparisons at scale
- Search for the 16 O(ppp) to 13 C pi+ pi+ e+ Decay Mode in Super-Kamiokande Using Machine Learning Techniques