Towards LLM-Powered Automation of a Dark Matter Constraint Repository
summary
The gist
Dark matter constraint repositories are critical community infrastructure, yet the most widely-used are volunteer-maintained…
In short
A large language model pipeline was built to automate monitoring and extracting dark matter limit curves from arXiv papers. The system uses multiple extraction channels and deterministic code guards to ensure data quality, leading to a median accuracy of 0.17 dex on a benchmark. The core lesson is that the infrastructure—code for extraction and checks—is more important than the model itself.
Key concepts
- Multi-channel Extraction
- This involves using up to five different methods to pull limit values from a single paper, such as reading text, parsing tables, checking axis labels, or performing a full vision trace of figures. The system compares these different methods and selects the best candidate based on predefined validity checks.
- Validity and Canonicalization
- This is the process of rigorously checking extracted data against physical reality and standard conventions. It involves converting all quantities to a single, standardized format, flagging any unknown conventions for human review, and prioritizing physically valid extractions over others.
- Model-Confined Perception
- The LLM's role is strictly limited to understanding the semantics of the text—reading what it sees—not performing complex calculations or unit conversions. All arithmetic and critical checks are handled by deterministic, testable code, ensuring the model only contributes perception.
- Governance of AI-Generated Data
- The paper highlights that while an LLM can generate proposals, human governance remains unsolved. The pipeline's success depends on human review before any data is merged into the official repository, emphasizing that expert judgment is the final necessary step.
Terminology used across episodes
This episode discusses
- Towards LLM-Powered Automation of a Dark Matter Constraint Repository · Paper Radio
- Axion Dark Matter
- New experimental approaches in the search for axion-like particles
- Dark photon limits: a handbook
- Cosmology of axion dark matter
- DePlot: One-shot visual language reasoning by plot-to-table translation
- From Experimental Limits to Physical Insight: A Retrieval-Augmented Multi-Agent Framework for Interpreting Searches Beyond the Standard Model
- AI Agents Can Already Autonomously Perform Experimental High Energy Physics
- Dark matter searches with a 13 meV threshold superconducting sensor array
- Direct Detection of Dark Photon Dark Matter with the James Webb Space Telescope
- Equivalence Principle Violations and Couplings of a Light Dilaton
- The landscape of QCD axion models
- Power-Constrained Limits
- Properties of Ultralight Bosons from Spins of Heavy Quasars via Superradiance
- First Constraints on Nuclear Coupling of Axionlike Particles from the Binary Neutron Star Gravitational Wave Event GW170817
- Oscillating nuclear charge radii as sensors for ultralight dark matter
- Pi in the Sky: Neutron Stars with Exceptionally Light QCD Axions
- HEPData: a repository for high energy physics data
- Reinterpretation of LHC Results for New Physics: Status and Recommendations after Run 2
- Short-range spin-dependent interactions of electrons: a probe for exotic pseudo-Goldstone bosons
The paper
Towards LLM-Powered Automation of a Dark Matter Constraint Repository · Read on arXiv
Department of Physics, Washington University in St Louis
Dark matter constraint repositories are critical community infrastructure, yet the most widely-used are volunteer-maintained, a sustainability risk as results accelerate. Limits are typically published as figures rather than machine-readable records, compounding the burden. We present a large language model (LLM) system that monitors arXiv, extracts limit curves, integrates them as code, and opens pull requests for human review. Extraction uses a tool-using agent that interprets physical conventions and checks its results, with a fixed multi-channel pipeline evaluated as an alternative. We evaluate both approaches on a 329-paper benchmark of measured limits graded against the upstream-curated repository. An extraction is considered successful when the extracted and reference mass ranges overlap by more than 90%, with at least 90% of compared values agreeing within a factor of two. For Fable 5, 52.6% of eligible curve comparisons meet this criterion with the agent, compared with 20.9% with the pipeline. Deployed, the pipeline has generated limit proposals, but none have merged: governance of AI-generated scientific data is itself unsolved.
Transcript
Introduction to the show: ident: High Energy Physics: Experiment. The day in high-energy experiment: detectors, searches and limits.
Kai: I'm Kai.
Mira: And I'm Mira, here with Lev.
Kai: We're looking at the paper: "Towards LLM-Powered Automation of a Dark Matter Constraint Repository".
Mira: Here's the gist of the paper. Dark matter constraint repositories are critical community infrastructure, yet the most widely-used are volunteer-maintained, a sustainability risk as results accelerate We present a large language model (LLM) pipeline that monitors arXiv, extracts limit curves, integrates them as code, and opens pull requests for human review On a 329-paper benchmark of measured limits graded against the upstream-curated repository, it identifies the coupling type for 98.1% of gradable papers The LLM is confined to perception; arithmetic, unit conversion, and the guards that reject mis-extractions are deterministic, testable code Deployed, the pipeline has generated limit proposals, but none have merged: governance of AI-generated scientific data is itself unsolved System Architecture Figure 1 shows the end-to-end pipeline Its design principle, arrived at empirically (Appendix A), is to keep the LLM confined to perception and semantics (reading text, tables, axes, and curves, and declaring what it read) while arithmetic, unit conversion, and candidate selection are deterministic, testable code The pipeline flows from arXiv (top-left) through keyword and LLM triage into five competing extraction channels Rule-based stages also use model-generated information, so the colours do not imply a strict separation of LLM use and code Multi-channel Extraction An extraction channel is a method for obtaining limit values from a paper’s text, tables, figures, or accompanying data files The agent produces candidates through up to five channels of differing semantic trust (table 1), staged as: a cheap particle physics interaction type pre-classification pass; a text/table read of the parsed PDF (PyMuPDF
13: ); a cheap axis-calibration pass that identifies the relevant figure and reads its axis labels and ranges; and a full vision trace of the rendered figure pages The channel selector considers tier only after all validity criteria, preferring higher-tier channels among equally valid candidates Validity and Canonicalization The selector compares candidates using a fixed sequence of checks The first check on which two candidates differ determines which is preferred Physical validity and interpretable units and conventions take priority Convention canonicalization converts quantities to the repository’s standard variable and convention Any quantity or convention outside the reviewed set requires human review Evaluation Methodology We evaluate on 329 papers spanning the repository’s coupling types, chosen as curated repository data files
5: that carry a known arXiv ID (the full registry holds 437 file entries over 347 distinct arXiv IDs) The ground truth (GT) for each paper is the upstream repository’s own digitized curve We compare in log10–log10 space, using g to denote the coupling variable after both curves have been converted to the same convention A paper’s residual is the median of ri over the compared mass points Model Performance and Lessons The production configuration achieves a median per-paper residual of 0.17 dex across 267 compared papers The key gains came from the system built around the model: extraction methods and checks implemented in code, centrally managed physical variable conversions with human review for unsupported conventions, and an evaluation that separates model effects from code changes and measures variation between runs The remaining large errors come from interpreting figures and selecting the correct curve, mass range, or coupling type The pipeline simplifies maintenance rather than replaces the maintainer: it turns curation from authoring into reviewing, and with 51% of papers within a factor of two end-to-end the expert’s accept-or-reject judgement remains the load-bearing step, by design and by necessity Limitations and Future Work The largest errors came from mishandling units and variables, rather than misreading figures The micro–macro gap is carried by coupling types with few papers Further improvement requires better interpretation of these rare limits and their conventions The gap is not tooling but adoption in the light-dark-matter community, where results are still published predominantly as figures The infrastructure for machinereadable limit publication already exists and is in routine use: HEPData
24: archives digitized collider results, and the LHC Reinterpretation Forum has published recommendations for reusable result presentation
25: We delivered an LLM-powered pipeline with deterministic guards that curates the live dark matter limit literature, with every output reviewed by a human before anything merges On a 329-paper benchmark against curator ground truth, it reaches a median accuracy of 0.17±0.04 dex among compared measured-limit papers The key gains came from the system built around the model: extraction methods and checks implemented in code, centrally managed physical variable conversions with human review for unsupported conventions, and an evaluation that separates model effects from code changes and measures variation between runs The benchmark makes that claim measurable: swapping the extraction model for another frontier model a generation apart moves the paired per-paper median by 0.000 dex, while the architectural work, at a fixed model, cut the per-type mean from 1.46 to 0.43 dex (0.38 in the production configuration) The remaining large errors come from interpreting figures and selecting the correct curve, mass range, or coupling type We offer the pipeline, the benchmark, and the measured design lessons collected in Appendix A to the community whose infrastructure problem motivated them, as a bridge until machine-readable limit publication makes extraction unnecessary Appendix A.
Kai: Lessons Learned Validity-first candidate selection is helpful Reading supplied data and vector coordinates directly allows exact or near-exact extraction with little model cost Deterministic guards on the model’s declared outputs stopped incorrect extractions before selection on 49 of 329 papers in the production benchmark Managing conversions in one place is helpful Requiring review for unknown conventions makes the uncertainty visible to reviewers Checking curves against values in the text provides an inexpensive, independent check on a traced curve Expert review before merging is important Separating model effects from pipeline changes and variation between runs is needed to establish the cause of an apparent improvement and whether it exceeds ordinary run-to-run variation Appendix B.
Mira: Selection Criteria and Scoring Details The lexicographic quality tuple records the criteria below in a fixed order The first four check validity, so later preferences cannot outweigh a difference on these checks Agreement with an independent channel is preferred if another channel produces a comparable extraction agreeing within a factor of three (a ratio in
1/3, 3: , or about 0.48 dex) Appendix C.
Kai: Convention Canonicalization in Detail Three rules for conversion are detailed Identifying the output quantity is necessary so later processing does not have to infer them from their magnitudes Preserving the plotted values means the vision channel reads values without converting them and identifies the plotted quantity Managing conversions in one place ensures all pipeline conversions use one reviewed conversion registry Handling unusual data files involves removing very large values using a threshold of 1019 Requiring human review for unknown conventions flags the paper for human review and excludes them from numerical evaluation as convention gaps Appendix D.
Paper summary: Kai: The core idea here is to use an LLM pipeline that reads arXiv papers, extracts the limit curves, and integrates them as code so they can be reviewed by humans before anything gets merged into the main repository.
Mira: They found that on a benchmark of three hundred twenty-nine papers graded against the curated repository, this system identified the coupling type for ninety-eight point one percent of those gradable papers <ref:2606.21658#pg1>.
Lev: That's a lot of coverage, but we need to know if that extraction is reliable when we talk about real physics constraints.
Kai: The paper emphasizes that the LLM itself is only good at perception, like reading text and axes, while the actual math and unit conversions are handled by deterministic code.
Mira: They show that this setup has a median residual of zero point one seven dex for compared papers, which means it’s relatively accurate when you compare its output to the ground truth.
Lev: And they stress that there are these deterministic guards built in to reject bad extractions before anything gets selected.
Kai: The paper claims that this system has generated limit proposals, but none of them have been merged yet, which points to the next big question about governance of AI-generated scientific data.
Mira: It’s about turning the curation work from being done by authors into something that is reviewed by experts, with fifty-one percent of all papers landing within a factor of two of the curator's curve.
Lev: From an engineering standpoint, they built this to be testable and confined; the model doesn't do arithmetic or unit conversions, so you know exactly what it’s doing.
Kai: It sounds like they’re trying to solve a sustainability problem for this community infrastructure by automating the tedious parts of digitizing and converting data.
Conclusion: Mira: So, to wrap up this paper, "Towards LLM-Powered Automation of a Dark Matter Constraint Repository" it presents an end-to-end LLM pipeline with deterministic guards that curates the live dark matter literature, making every output human reviewed before merging.
Kai: The main thing they found on their benchmark is that among compared measured limit papers against the curator ground truth, this system reaches a median accuracy of zero point one seven dex among those papers.
Lev: It’s important to note that while the pipeline generated proposals, none have merged yet, which highlights that governance of AI-generated scientific data is still an open question.
Mira: The real gain they highlight isn't just the model itself, but the system built around it: extraction methods and checks implemented in code.
Kai: Swapping out the extraction model for another frontier model a generation apart only moved their paired per-paper median by zero point zero zero zero dex, while changing the architectural work cut that mean from one point four six to a three point eight in production configuration.
Lev: So it sounds like the architecture and how you handle conversions is more important than just picking the latest big language model for this job.
Kai: The paper offers the pipeline, and the benchmark, and all those measured design lessons to the community that needed this infrastructure problem solved as a bridge until machine-readable limit publication makes extraction unnecessary.
Mira: It’s about making sure that even though we have these powerful new tools, the expert's accept or reject judgment remains the necessary step by design and by necessity.
More episodes
- Daily Summary for 2026-11-19
- 2512.21290-Model-independent ZH production cross section at FCC-ee
- 2610.21982-First Observation of the Radiative Decay D 0 to gamma omega
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- Daily Summary for 2026-11-18
- 2512.08065-Searches for Light Dark Matter and Evidence of Coherent Elastic Neutrino-Nucleus Scattering of Solar Neutrinos with the LUX-ZEPLIN (LZ) Experiment
- Daily Summary for 2026-11-18
- 2508.18022-Precision Measurements of the Electroweak Mixing Angle in the Region of the Z pole at the Large Hadron Collider
- 2512.23992-Prospect for measurement of CP-violating parameters of B s 0 to phi gamma at the Tera Z factory