A semi-autonomous multi-agent pipeline for high-energy astrophysics research with enforced provenance and verifiable self-extension
summary
The gist
Large language model agents are now applied to astronomical data analysis Under pressure to complete a task…
In short
auto_research is a multi-agent pipeline for astronomical data analysis that retrieves raw data from HEASARC and writes necessary analysis tools. It enforces strict verification by testing code determinism, scanning for invented values, and requiring provenance links to ensure results are trustworthy.
Key concepts
- Provenance Enforcement
- This system tracks the origin of every result by linking it back to the specific tool used, the original research request, or a cited paper. A number in a final report is only accepted if its source can be traced through these established links.
- Tool Verification Gate
- Before new analysis tools are trusted, they must pass seven checks: testing for deterministic results on real data, checking for banned phrases indicating fabricated numbers, verifying code structure, and confirming that tests correctly call other tools.
- Verification over Trust
- The pipeline is designed to verify agent outputs rather than blindly trusting them. Verification involves running code on real data to ensure it produces reproducible output and checking results against trusted routines and sanity bounds.
Terminology used across episodes
This episode discusses
- A semi-autonomous multi-agent pipeline for high-energy astrophysics research with enforced provenance and verifiable self-extension · Paper Radio
- The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
- AI Agents Can Already Autonomously Perform Experimental High Energy Physics
- Automating High Energy Physics Data Analysis with LLM-Powered Agents
- The Denario project: Deep knowledge AI agents for scientific discovery · Paper Radio
- Kosmos: An AI Scientist for Autonomous Discovery
- SciIntegrity-Bench: A Benchmark for Evaluating Academic Integrity in AI Scientist Systems
- Collider-Bench: Benchmarking AI Agents with Particle Physics Analysis Reproduction
- gwBenchmarks: Stress-Testing LLM Agents on High-Precision Gravitational Wave Astronomy · Paper Radio
- ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases
- Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents
- How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks · Paper Radio
- Open Source Planning & Control System with Language Agents for Autonomous Scientific Discovery
- The AI Cosmologist I: An Agentic System for Automated Data Analysis
- Domain-Specific Agents for Cherenkov Telescope Array Control Software and Gamma-Ray Data Analysis
- The Imaging X-Ray Polarimetry Explorer (IXPE): Pre-Launch
- A Pipeline for the AI-Assisted Analysis of Imaging X-ray Polarimetry Explorer Observations of Black Hole X-ray Binaries and Limits on Their Polarization Variability · Paper Radio
- TheoPhysEval: Assessing AI agents at autonomous theoretical physics problem solving · Paper Radio
- Voyager: An Open-Ended Embodied Agent with Large Language Models
- ToolUniverse: An open platform for democratizing AI scientists
- Beyond Task Completion: A Verification-vs.-Conformance Gap in Tool-Evolving Agents
The paper
A semi-autonomous multi-agent pipeline for high-energy astrophysics research with enforced provenance and verifiable self-extension · Read on arXiv
Fang Zhoua
Washington University in St. Louis
Large language model (LLM) agents are now applied to astronomical data analysis. They retrieve the literature, reduce observations, and draft conclusions. Under pressure to complete a task, such agents can fabricate results, and the numbers they report may not come from the analysis they ran. We present auto research, a semi-autonomous multi-agent pipeline that retrieves and analyses raw data from the High Energy Astrophysics Science Archive Research Center (HEASARC) and writes the analysis tools that it needs. We check the work of the agents at two points. A new tool enters the toolkit after it passes automatic checks, including repeated test runs and a scan for phrases that accompany invented values, and a code review by a model from a different provider; a human decides whether to promote it to the trusted branch. A number in a report passes only if a recorded output of a toolkit tool or the request contains it, or if the report gives its location in a cited paper. We test the pipeline with data from the Imaging X-ray Polarimetry Explorer (IXPE). Of 65 defects that we planted in the toolkit, 47 pass the automatic checks, and one passes the code review as well; the review also rejects 40 of 69 correct changes. The pipeline reproduces the published IXPE results for Cyg X-1: all 14 compared quantities agree within 3 sigma, and the polarization degrees are lower than the published values. With weaker models in the main roles, the pipeline recovers 12 of the 14 quantities and reports the other two as not measured. We also used the pipeline for a polarization-stability analysis of six observations, a survey of polarization against hardness in 29 observations, and a search for a time lag between the Stokes parameters q and u.
Transcript
Introduction to the show: ident: Astrophysics. The day in astrophysics: observations, models and what the data forces.
Vera: We're looking at the paper: "A semi-autonomous multi-agent pipeline for high-energy astrophysics research with enforced provenance and verifiable self-extension".
Jocelyn: Here's the gist of the paper. Large language model agents are now applied to astronomical data analysis Under pressure to complete a task, such agents can fabricate results and the numbers they report may not come from the analysis they ran The auto research pipeline is a semi-autonomous multi-agent system designed to retrieve and analyze raw data from HEASARC while writing the necessary analysis tools It enforces provenance by ensuring a result passes only if its provenance links it to a toolkit tool, the research request, or a location in a cited paper The pipeline tests new tools through automatic checks including repeated test runs and scans for phrases that accompany invented values, followed by code review from a model from a different provider before human promotion Pipeline Design Principles The design principles of auto research center on verifying rather than trusting the agent's results This verification involves three methods: first, testing the code where tests check that results are deterministic and agree with trusted routines and fall within stated sanity bounds; second, running the code on real data to produce real output files for human reproduction; and third, ensuring every output carries its provenance which names earlier outputs so a human can check the input The system utilizes two repositories, auto research for orchestration code and research toolkit for analysis tools Each tool consists of two halves, lib/ holding the analysis code and skills/ holding a manifest and wrappers that tell agents how to call functions Tool Verification Mechanisms The gate applies seven checks to new tools before they enter the trusted branch These checks include: 1) Determinism, where a tool's tests must run twice on real data and yield the same outcome; 2) Tool-local tests, requiring at least one passing test; 3) Banned vocabulary, scanning comments for phrases accompanying invented results; 4) Data-fabricating code rejection of placeholder function bodies and binary data; 5) Parseability to ensure Python files parse correctly; 6) Structure, ensuring the tool carries a description and a test file on allowed paths; and 7) Provenance, verifying that tests call existing figures and that numbers returned appear in recorded pipeline outputs The co-lead then judges whether the code follows the brief and if tests are adequate Report Validation Provenance enforcement dictates that a result passes from one agent to another or to a human only if its provenance links it to a toolkit tool, the research request, or a location in a cited paper The validator admits a number in the report only if it appears in the cited output of a toolkit tool, appears in the cited request, or gives its location in a cited paper Conversely, shell output and files read by an agent do not support numbers Analysis Outcomes and Benchmarking The pipeline was tested on four research requests, including reproducing the IXPE polarization degree (PD) and polarization angle (PA) of Cyg X-1 results with all 14 quantities agreeing within 3sigma The pipeline reproduced the geometrical conclusions of the original paper but reported a systematic offset in PD values In testing tool writing, checks stopped 64 of the 65 planted defects, and weaker models recovered 12 of the 14 quantities in one run The co-lead model was found to reject 40 of the 69 defect-free controls in a benchmark Limitations and Outlook The study identifies three limitations for auto research, including provenance enforcement establishing origin rather than correctness, the fallibility of the language model co-lead who rejected many controls in testing, and the validator only verifying appearance in a cited output, not correct placement within it The pipeline's design keeps tools that it writes available for later analyses, unlike systems that write new code for each task Future plans involve building these agents from a reusable design and allowing them to start new agents during a run while the validator checks every report The code of the pipeline and toolkit is public, available at specific GitHub repositories Page 1 --- A semi-autonomous multi-agent pipeline for high-energy astrophysics research with enforced provenance and verifiable self-extensionLarge language model (LLM) agents are now applied to astronomical data analysis They retrieve the literature, reduce observations, and draft conclusions Under pressure to complete a task, such agents can fabricate results, and the numbers they report may not come from the analysis they ran We present auto research, a semi-autonomous multi-agent pipeline that retrieves and analyses raw data from the High Energy Astrophysics Science Archive Research Center (HEASARC) and writes the analysis tools that it needs We check the work of the agents at two points A new tool enters the toolkit after it passes automatic checks, including repeated test runs and a scan for phrases that accompany invented values, and a code review by a model from a different provider; a human decides whether to promote it to the trusted branch A number in a report passes only if a recorded output of a toolkit tool or the request contains it, or if the report gives its location in a cited paper We test the pipeline with data from the Imaging X-ray Polarimetry Explorer (IXPE) Of 65 defects that we planted in the toolkit, 47 pass the automatic checks, and one passes the code review as well; the review also rejects 40 of 69 correct changes The pipeline reproduces the published IXPE results for Cyg X-1 all 14 compared quantities agree within 3sigma, and the polarization degrees are lower than the published values With weaker models in the main roles, the pipeline recovers 12 of the 14 quantities and reports the other two as not measured We also used the pipeline for a polarization-stability analysis of six observations, a survey of polarization against hardness in 29 observations, and a search for a time lag between the Stokes parameters q and u Keywords: large language model agents, data analysis, software verification, provenance, X-ray polarimetry, black hole X-ray binaries Page 2 --- Scientists increasingly use large language models (LLMs) in their research An LLM agent is an LLM that calls tools, such as a code interpreter, and acts on their output Agent systems can retrieve the literature, write and run analysis code, and draft the text that reports the results Some assist a scientist at each step, while others take a research question and return a manuscript Tests of LLM agents have revealed three recurring failure modes First, agents fabricate results Second, agents game the checks that are meant to constrain them Third, agents do not act on their own reviews Because of these failure modes, a scientist needs to verify the results of an agent Most systems write new code for each task rather than add it to a reviewed, reusable library Thus the scientist has to check new code for each result Even when a system links each statement to its source, the scientist still has to check the statement In astronomy, agent systems have been applied to cosmological data
12: , galaxy images and cosmological simulations
13: , and to gamma-ray data analysis
14: This project grew as a separate branch from the participation of the author in a group working on the development of an agentic AI pipeline for the analysis of the data from the Imaging X-ray Polarimetry Explorer (IXPE) satellite
15: That pipeline is described in
16, called Paper I hereafter: The auto research pipeline differs in several design principles from the one of Paper I (i) auto research takes project analysis ideas, but does not develop its own ideas
17: ; (ii) the pipeline focuses on the development of a toolkit A new tool is added after automatic checks and an LLM code review A scientist then decides whether to promote the tool to the trusted branch (iii) the automatic checks include quantitative tests of new analysis tools; (iv) the auto research pipeline writes reports about the results Many safeguards act on the reasoning of the agents, for example instructions in the prompt, debate between agents, and critique by a reviewing agent
1, 16: The third failure mode shows their limit because an agent can identify a flaw and still report the unrevised conclusion We therefore check what the agents produce at two points: when the pipeline adds a new tool, and when an agent reports a number For new tools, earlier tool libraries let the model that wrote a tool grade it
18: , or relied on tests and human review
19: Checks on results can miss defective tools in one benchmark where 215 of 222 tools gave wrong results on every held-out input Dong and Chen applied scripted checks followed by an audit by a model from a different provider auto research uses the same combination Each new tool passes automatic checks (the “gate”) and then a review by a model from a provider other than its author’s A human then decides whether to promote the tool to the trusted branch of the toolkit At the second point, we enforce provenance The provenance of an output is the record of the tool and data that produced it Provenance enforcement means a result passes only if its provenance links it to a toolkit tool, research request, or location in a cited paper A toolkit tool is either written by a human or committed after checks on new tools In auto research, each agent ends its task with a report, and a validator admits a number only if it appears in recorded output of a toolkit tool or the request, or if the report gives its location in a cited paper Page 3 --- Section 2 describes auto research which retrieves and analyses raw data from the High Energy Astrophysics Science Archive Research Center (HEASARC) and writes the analysis tools that it needs Section 3 describes its checks on reported numbers and on new tools Section 4 applies it to IXPE data, including a reproduction of the Cyg X-1 results of Krawczynski et al.
25: Section 5 measures how well the checks perform, and Sect.
Jocelyn: 6 tests whether the pipeline still works with weaker models in the main roles Page 4 --- 2.
Vera: The pipeline 2.1.
Paper summary: Vera: We've been looking at this paper, "A semi-autonomous multi-agent pipeline for high-energy astrophysics research with enforced provenance and verifiable self-extension," and it really centers on making AI agents reliable in astronomy. The authors argue that they’re not just letting LLMs write reports; they’re building a whole system around verifying the tools those agents use.
Jocelyn: They focus heavily on this concept of verifiable self-extension, meaning the pipeline can keep writing and updating its own analysis tools as it goes, but only after passing a rigorous gate. This gate involves automated checks like testing for determinism and scanning for invented values before a human co-lead makes the final call to promote the tool.
Subrahmanyan: From my perspective on cosmology and structure formation, this points to how we might use AI to handle the heavy lifting of data reduction and analysis pipelines where human intervention is needed for guidance, but where automation needs strong checks against hallucination. The enforcement of provenance—linking every number back to a specific tool or request—is what makes it scientifically useful in our field.
Vera: So, it's not about the AI being perfect; it's about making the process auditable so we know exactly where any potential error came from when we look at the final scientific conclusion.
Jocelyn: It’s an attempt to give us a tool that can handle complex data analysis tasks autonomously while maintaining a strong link back to established, trusted sources in the literature and our own experimental runs.
Subrahmanyan: The implication is that we might see these pipelines being used for things like polarization stability analyses, where you need consistency over long observation periods, because the system is designed to track those quantities precisely across different steps.
Vera: It’s a step toward integrating LLMs into complex research workflows in a way that respects the need for accountability in scientific reporting.
Conclusion: Vera: So we’re wrapping up this session on "A semi-autonomous multi-agent pipeline for high-energy astrophysics research with enforced provenance and verifiable self-extension."
Jocelyn: It sounds like this paper is about building an AI system that doesn't just give you a finished answer, but a whole process where every single step is traceable back to its original source.
Subrahmanyan: Exactly. They're tackling the problem of AI making up numbers or inventing analysis tools on its own when dealing with massive astronomical datasets like those from HEASARC.
Vera: It’s about verification rather than just trusting the agent’s output, which is crucial because in astronomy, getting a result wrong can have serious consequences for what we actually learn about the sky.
Jocelyn: And they do this by setting up these strict checks—like tests running multiple times on real data and scanning the code for any suspicious language accompanying invented numbers.
Subrahmanyan: The concept of enforced provenance is key here. It means a final result can only be accepted if it can point back to a specific tool used, the original research request, or even a specific paper they cited.
Vera: It’s about creating a system that can self-extend—meaning the pipeline can write its own analysis tools—but only after those rigorous checks are passed by an AI reviewer and then by a human co-lead.
Jocelyn: So, the implication is that we might see research workflows where AI handles the tedious data reduction and tool writing, but a human is still in charge of approving what that tool actually does before it touches the final science.
Subrahmanyan: It shifts the focus from just getting an output to ensuring there's a verifiable chain of custody for every piece of analysis.
Vera: It’s not about replacing the scientist; it’s about giving them a very powerful, self-checking assistant that keeps our work honest and auditable.
Jocelyn: That kind of accountability is what we need when we start relying more on these complex tools for discovering things in deep space.
More episodes
- 2607.17929-gevolution 2.0: GPU-accelerated relativistic N-body simulations for cosmology
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2509.02657-On the synergetic use of Ariel and JWST for exoplanet atmospheric science
- 2609.10042-Modelling Fourier filtering wavefront sensors for PSD-based methods: the AOPERA tool
- 2511.05639-Performance of the AstroPix Prototype Module for the Barrel Imaging Calorimeter at the ePIC Detector and in Space-Based Payloads
- 2610.19909-RPCA: a Resolved PSF reconstruction method for undersampled images
- 2512.03440-An Analysis of LIGO Glitches Using t-SNE During the First Part of the Fourth LIGO-Virgo-KAGRA Observing Run
- 2608.10284-Lunar Reflective Interferometry
- 2610.16633-Institutional broadening of the Japanese astronomy and astrophysics research community during 1996-2025