A semi-autonomous multi-agent pipeline for high-energy astrophysics research with enforced provenance and verifiable self-extension
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Astrophysics. The day in astrophysics: observations, models and what the data forces.
Vera: We're looking at the paper: "A semi-autonomous multi-agent pipeline for high-energy astrophysics research with enforced provenance and verifiable self-extension".
Jocelyn: Here's the gist of the paper. Large language model agents are now applied to astronomical data analysis Under pressure to complete a task, such agents can fabricate results and the numbers they report may not come from the analysis they ran The auto research pipeline is a semi-autonomous multi-agent system designed to retrieve and analyze raw data from HEASARC while writing the necessary analysis tools It enforces provenance by ensuring a result passes only if its provenance links it to a toolkit tool, the research request, or a location in a cited paper The pipeline tests new tools through automatic checks including repeated test runs and scans for phrases that accompany invented values, followed by code review from a model from a different provider before human promotion Pipeline Design Principles The design principles of auto research center on verifying rather than trusting the agent's results This verification involves three methods: first, testing the code where tests check that results are deterministic and agree with trusted routines and fall within stated sanity bounds; second, running the code on real data to produce real output files for human reproduction; and third, ensuring every output carries its provenance which names earlier outputs so a human can check the input The system utilizes two repositories, auto research for orchestration code and research toolkit for analysis tools Each tool consists of two halves, lib/ holding the analysis code and skills/ holding a manifest and wrappers that tell agents how to call functions Tool Verification Mechanisms The gate applies seven checks to new tools before they enter the trusted branch These checks include: 1) Determinism, where a tool's tests must run twice on real data and yield the same outcome; 2) Tool-local tests, requiring at least one passing test; 3) Banned vocabulary, scanning comments for phrases accompanying invented results; 4) Data-fabricating code rejection of placeholder function bodies and binary data; 5) Parseability to ensure Python files parse correctly; 6) Structure, ensuring the tool carries a description and a test file on allowed paths; and 7) Provenance, verifying that tests call existing figures and that numbers returned appear in recorded pipeline outputs The co-lead then judges whether the code follows the brief and if tests are adequate Report Validation Provenance enforcement dictates that a result passes from one agent to another or to a human only if its provenance links it to a toolkit tool, the research request, or a location in a cited paper The validator admits a number in the report only if it appears in the cited output of a toolkit tool, appears in the cited request, or gives its location in a cited paper Conversely, shell output and files read by an agent do not support numbers Analysis Outcomes and Benchmarking The pipeline was tested on four research requests, including reproducing the IXPE polarization degree (PD) and polarization angle (PA) of Cyg X-1 results with all 14 quantities agreeing within 3sigma The pipeline reproduced the geometrical conclusions of the original paper but reported a systematic offset in PD values In testing tool writing, checks stopped 64 of the 65 planted defects, and weaker models recovered 12 of the 14 quantities in one run The co-lead model was found to reject 40 of the 69 defect-free controls in a benchmark Limitations and Outlook The study identifies three limitations for auto research, including provenance enforcement establishing origin rather than correctness, the fallibility of the language model co-lead who rejected many controls in testing, and the validator only verifying appearance in a cited output, not correct placement within it The pipeline's design keeps tools that it writes available for later analyses, unlike systems that write new code for each task Future plans involve building these agents from a reusable design and allowing them to start new agents during a run while the validator checks every report The code of the pipeline and toolkit is public, available at specific GitHub repositories Page 1 --- A semi-autonomous multi-agent pipeline for high-energy astrophysics research with enforced provenance and verifiable self-extensionLarge language model (LLM) agents are now applied to astronomical data analysis They retrieve the literature, reduce observations, and draft conclusions Under pressure to complete a task, such agents can fabricate results, and the numbers they report may not come from the analysis they ran We present auto research, a semi-autonomous multi-agent pipeline that retrieves and analyses raw data from the High Energy Astrophysics Science Archive Research Center (HEASARC) and writes the analysis tools that it needs We check the work of the agents at two points A new tool enters the toolkit after it passes automatic checks, including repeated test runs and a scan for phrases that accompany invented values, and a code review by a model from a different provider; a human decides whether to promote it to the trusted branch A number in a report passes only if a recorded output of a toolkit tool or the request contains it, or if the report gives its location in a cited paper We test the pipeline with data from the Imaging X-ray Polarimetry Explorer (IXPE) Of 65 defects that we planted in the toolkit, 47 pass the automatic checks, and one passes the code review as well; the review also rejects 40 of 69 correct changes The pipeline reproduces the published IXPE results for Cyg X-1 all 14 compared quantities agree within 3sigma, and the polarization degrees are lower than the published values With weaker models in the main roles, the pipeline recovers 12 of the 14 quantities and reports the other two as not measured We also used the pipeline for a polarization-stability analysis of six observations, a survey of polarization against hardness in 29 observations, and a search for a time lag between the Stokes parameters q and u Keywords: large language model agents, data analysis, software verification, provenance, X-ray polarimetry, black hole X-ray binaries Page 2 --- Scientists increasingly use large language models (LLMs) in their research An LLM agent is an LLM that calls tools, such as a code interpreter, and acts on their output Agent systems can retrieve the literature, write and run analysis code, and draft the text that reports the results Some assist a scientist at each step, while others take a research question and return a manuscript Tests of LLM agents have revealed three recurring failure modes First, agents fabricate results Second, agents game the checks that are meant to constrain them Third, agents do not act on their own reviews Because of these failure modes, a scientist needs to verify the results of an agent Most systems write new code for each task rather than add it to a reviewed, reusable library Thus the scientist has to check new code for each result Even when a system links each statement to its source, the scientist still has to check the statement In astronomy, agent systems have been applied to cosmological data
12: , galaxy images and cosmological simulations
13: , and to gamma-ray data analysis
14: This project grew as a separate branch from the participation of the author in a group working on the development of an agentic AI pipeline for the analysis of the data from the Imaging X-ray Polarimetry Explorer (IXPE) satellite
15: That pipeline is described in
16, called Paper I hereafter: The auto research pipeline differs in several design principles from the one of Paper I (i) auto research takes project analysis ideas, but does not develop its own ideas
17: ; (ii) the pipeline focuses on the development of a toolkit A new tool is added after automatic checks and an LLM code review A scientist then decides whether to promote the tool to the trusted branch (iii) the automatic checks include quantitative tests of new analysis tools; (iv) the auto research pipeline writes reports about the results Many safeguards act on the reasoning of the agents, for example instructions in the prompt, debate between agents, and critique by a reviewing agent
1, 16: The third failure mode shows their limit because an agent can identify a flaw and still report the unrevised conclusion We therefore check what the agents produce at two points: when the pipeline adds a new tool, and when an agent reports a number For new tools, earlier tool libraries let the model that wrote a tool grade it
18: , or relied on tests and human review
19: Checks on results can miss defective tools in one benchmark where 215 of 222 tools gave wrong results on every held-out input Dong and Chen applied scripted checks followed by an audit by a model from a different provider auto research uses the same combination Each new tool passes automatic checks (the “gate”) and then a review by a model from a provider other than its author’s A human then decides whether to promote the tool to the trusted branch of the toolkit At the second point, we enforce provenance The provenance of an output is the record of the tool and data that produced it Provenance enforcement means a result passes only if its provenance links it to a toolkit tool, research request, or location in a cited paper A toolkit tool is either written by a human or committed after checks on new tools In auto research, each agent ends its task with a report, and a validator admits a number only if it appears in recorded output of a toolkit tool or the request, or if the report gives its location in a cited paper Page 3 --- Section 2 describes auto research which retrieves and analyses raw data from the High Energy Astrophysics Science Archive Research Center (HEASARC) and writes the analysis tools that it needs Section 3 describes its checks on reported numbers and on new tools Section 4 applies it to IXPE data, including a reproduction of the Cyg X-1 results of Krawczynski et al.
25: Section 5 measures how well the checks perform, and Sect.
Jocelyn: 6 tests whether the pipeline still works with weaker models in the main roles Page 4 --- 2.
Vera: The pipeline 2.1.
Paper summary: Vera: We've been looking at this paper, "A semi-autonomous multi-agent pipeline for high-energy astrophysics research with enforced provenance and verifiable self-extension," and it really centers on making AI agents reliable in astronomy. The authors argue that they’re not just letting LLMs write reports; they’re building a whole system around verifying the tools those agents use.
Jocelyn: They focus heavily on this concept of verifiable self-extension, meaning the pipeline can keep writing and updating its own analysis tools as it goes, but only after passing a rigorous gate. This gate involves automated checks like testing for determinism and scanning for invented values before a human co-lead makes the final call to promote the tool.
Subrahmanyan: From my perspective on cosmology and structure formation, this points to how we might use AI to handle the heavy lifting of data reduction and analysis pipelines where human intervention is needed for guidance, but where automation needs strong checks against hallucination. The enforcement of provenance—linking every number back to a specific tool or request—is what makes it scientifically useful in our field.
Vera: So, it's not about the AI being perfect; it's about making the process auditable so we know exactly where any potential error came from when we look at the final scientific conclusion.
Jocelyn: It’s an attempt to give us a tool that can handle complex data analysis tasks autonomously while maintaining a strong link back to established, trusted sources in the literature and our own experimental runs.
Subrahmanyan: The implication is that we might see these pipelines being used for things like polarization stability analyses, where you need consistency over long observation periods, because the system is designed to track those quantities precisely across different steps.
Vera: It’s a step toward integrating LLMs into complex research workflows in a way that respects the need for accountability in scientific reporting.
Conclusion: Vera: So we’re wrapping up this session on "A semi-autonomous multi-agent pipeline for high-energy astrophysics research with enforced provenance and verifiable self-extension."
Jocelyn: It sounds like this paper is about building an AI system that doesn't just give you a finished answer, but a whole process where every single step is traceable back to its original source.
Subrahmanyan: Exactly. They're tackling the problem of AI making up numbers or inventing analysis tools on its own when dealing with massive astronomical datasets like those from HEASARC.
Vera: It’s about verification rather than just trusting the agent’s output, which is crucial because in astronomy, getting a result wrong can have serious consequences for what we actually learn about the sky.
Jocelyn: And they do this by setting up these strict checks—like tests running multiple times on real data and scanning the code for any suspicious language accompanying invented numbers.
Subrahmanyan: The concept of enforced provenance is key here. It means a final result can only be accepted if it can point back to a specific tool used, the original research request, or even a specific paper they cited.
Vera: It’s about creating a system that can self-extend—meaning the pipeline can write its own analysis tools—but only after those rigorous checks are passed by an AI reviewer and then by a human co-lead.
Jocelyn: So, the implication is that we might see research workflows where AI handles the tedious data reduction and tool writing, but a human is still in charge of approving what that tool actually does before it touches the final science.
Subrahmanyan: It shifts the focus from just getting an output to ensuring there's a verifiable chain of custody for every piece of analysis.
Vera: It’s not about replacing the scientist; it’s about giving them a very powerful, self-checking assistant that keeps our work honest and auditable.
Jocelyn: That kind of accountability is what we need when we start relying more on these complex tools for discovering things in deep space.
Fang Zhoua
Washington University in St. Louis
astro-ph.IM, astro-ph.HE
Submitted: 2026-11-03
Updated: 2026-11-03
Code: https://github.com/langchain-ai/langgraph
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 82/100
The gist: Large language model agents are now applied to astronomical data analysis Under pressure to complete a task…
Key concepts
- Provenance Enforcement
- This system tracks the origin of every result by linking it back to the specific tool used, the original research request, or a cited paper. A number in a final report is only accepted if its source can be traced through these established links.
- Tool Verification Gate
- Before new analysis tools are trusted, they must pass seven checks: testing for deterministic results on real data, checking for banned phrases indicating fabricated numbers, verifying code structure, and confirming that tests correctly call other tools.
- Verification over Trust
- The pipeline is designed to verify agent outputs rather than blindly trusting them. Verification involves running code on real data to ensure it produces reproducible output and checking results against trusted routines and sanity bounds.
Terminology
Summary
Large language model agents are now applied to astronomical data analysis Under pressure to complete a task, such agents can fabricate results and the numbers they report may not come from the analysis they ran The auto research pipeline is a semi-autonomous multi-agent system designed to retrieve and analyze raw data from HEASARC while writing the necessary analysis tools It enforces provenance by ensuring a result passes only if its provenance links it to a toolkit tool, the research request, or a location in a cited paper The pipeline tests new tools through automatic checks including repeated test runs and scans for phrases that accompany invented values, followed by code review from a model from a different provider before human promotion
Pipeline Design Principles
The design principles of auto research center on verifying rather than trusting the agent's results This verification involves three methods: first, testing the code where tests check that results are deterministic and agree with trusted routines and fall within stated sanity bounds; second, running the code on real data to produce real output files for human reproduction; and third, ensuring every output carries its provenance which names earlier outputs so a human can check the input The system utilizes two repositories, auto research for orchestration code and research toolkit for analysis tools Each tool consists of two halves, lib/ holding the analysis code and skills/ holding a manifest and wrappers that tell agents how to call functions
Tool Verification Mechanisms
The gate applies seven checks to new tools before they enter the trusted branch These checks include: 1) Determinism, where a tool's tests must run twice on real data and yield the same outcome; 2) Tool-local tests, requiring at least one passing test; 3) Banned vocabulary, scanning comments for phrases accompanying invented results; 4) Data-fabricating code rejection of placeholder function bodies and binary data; 5) Parseability to ensure Python files parse correctly; 6) Structure, ensuring the tool carries a description and a test file on allowed paths; and 7) Provenance, verifying that tests call existing figures and that numbers returned appear in recorded pipeline outputs The co-lead then judges whether the code follows the brief and if tests are adequate
Report Validation
Provenance enforcement dictates that a result passes from one agent to another or to a human only if its provenance links it to a toolkit tool, the research request, or a location in a cited paper The validator admits a number in the report only if it appears in the cited output of a toolkit tool, appears in the cited request, or gives its location in a cited paper Conversely, shell output and files read by an agent do not support numbers
Analysis Outcomes and Benchmarking
The pipeline was tested on four research requests, including reproducing the IXPE polarization degree (PD) and polarization angle (PA) of Cyg X-1 results with all 14 quantities agreeing within 3sigma The pipeline reproduced the geometrical conclusions of the original paper but reported a systematic offset in PD values In testing tool writing, checks stopped 64 of the 65 planted defects, and weaker models recovered 12 of the 14 quantities in one run The co-lead model was found to reject 40 of the 69 defect-free controls in a benchmark
Limitations and Outlook
The study identifies three limitations for auto research, including provenance enforcement establishing origin rather than correctness, the fallibility of the language model co-lead who rejected many controls in testing, and the validator only verifying appearance in a cited output, not correct placement within it The pipeline's design keeps tools that it writes available for later analyses, unlike systems that write new code for each task Future plans involve building these agents from a reusable design and allowing them to start new agents during a run while the validator checks every report The code of the pipeline and toolkit is public, available at specific GitHub repositories
--- Page 1 ---
A semi-autonomous multi-agent pipeline for high-energy astrophysics research with enforced provenance and verifiable self-extensionLarge language model (LLM) agents are now applied to astronomical data analysis They retrieve the literature, reduce observations, and draft conclusions Under pressure to complete a task, such agents can fabricate results, and the numbers they report may not come from the analysis they ran We present auto research, a semi-autonomous multi-agent pipeline that retrieves and analyses raw data from the High Energy Astrophysics Science Archive Research Center (HEASARC) and writes the analysis tools that it needs We check the work of the agents at two points A new tool enters the toolkit after it passes automatic checks, including repeated test runs and a scan for phrases that accompany invented values, and a code review by a model from a different provider; a human decides whether to promote it to the trusted branch A number in a report passes only if a recorded output of a toolkit tool or the request contains it, or if the report gives its location in a cited paper We test the pipeline with data from the Imaging X-ray Polarimetry Explorer (IXPE) Of 65 defects that we planted in the toolkit, 47 pass the automatic checks, and one passes the code review as well; the review also rejects 40 of 69 correct changes The pipeline reproduces the published IXPE results for Cyg X-1 all 14 compared quantities agree within 3sigma, and the polarization degrees are lower than the published values With weaker models in the main roles, the pipeline recovers 12 of the 14 quantities and reports the other two as not measured We also used the pipeline for a polarization-stability analysis of six observations, a survey of polarization against hardness in 29 observations, and a search for a time lag between the Stokes parameters q and u Keywords: large language model agents, data analysis, software verification, provenance, X-ray polarimetry, black hole X-ray binaries
--- Page 2 ---
Scientists increasingly use large language models (LLMs) in their research An LLM agent is an LLM that calls tools, such as a code interpreter, and acts on their output Agent systems can retrieve the literature, write and run analysis code, and draft the text that reports the results Some assist a scientist at each step, while others take a research question and return a manuscript Tests of LLM agents have revealed three recurring failure modes First, agents fabricate results Second, agents game the checks that are meant to constrain them Third, agents do not act on their own reviews Because of these failure modes, a scientist needs to verify the results of an agent Most systems write new code for each task rather than add it to a reviewed, reusable library Thus the scientist has to check new code for each result Even when a system links each statement to its source, the scientist still has to check the statement In astronomy, agent systems have been applied to cosmological data [12], galaxy images and cosmological simulations [13], and to gamma-ray data analysis [14] This project grew as a separate branch from the participation of the author in a group working on the development of an agentic AI pipeline for the analysis of the data from the Imaging X-ray Polarimetry Explorer (IXPE) satellite [15] That pipeline is described in [16, called Paper I hereafter] The auto research pipeline differs in several design principles from the one of Paper I (i) auto research takes project analysis ideas, but does not develop its own ideas [17]; (ii) the pipeline focuses on the development of a toolkit A new tool is added after automatic checks and an LLM code review A scientist then decides whether to promote the tool to the trusted branch (iii) the automatic checks include quantitative tests of new analysis tools; (iv) the auto research pipeline writes reports about the results Many safeguards act on the reasoning of the agents, for example instructions in the prompt, debate between agents, and critique by a reviewing agent [1, 16] The third failure mode shows their limit because an agent can identify a flaw and still report the unrevised conclusion We therefore check what the agents produce at two points: when the pipeline adds a new tool, and when an agent reports a number For new tools, earlier tool libraries let the model that wrote a tool grade it [18], or relied on tests and human review [19] Checks on results can miss defective tools in one benchmark where 215 of 222 tools gave wrong results on every held-out input Dong and Chen applied scripted checks followed by an audit by a model from a different provider auto research uses the same combination Each new tool passes automatic checks (the “gate”) and then a review by a model from a provider other than its author’s A human then decides whether to promote the tool to the trusted branch of the toolkit At the second point, we enforce provenance The provenance of an output is the record of the tool and data that produced it Provenance enforcement means a result passes only if its provenance links it to a toolkit tool, research request, or location in a cited paper A toolkit tool is either written by a human or committed after checks on new tools In auto research, each agent ends its task with a report, and a validator admits a number only if it appears in recorded output of a toolkit tool or the request, or if the report gives its location in a cited paper
--- Page 3 ---
Section 2 describes auto research which retrieves and analyses raw data from the High Energy Astrophysics Science Archive Research Center (HEASARC) and writes the analysis tools that it needs Section 3 describes its checks on reported numbers and on new tools Section 4 applies it to IXPE data, including a reproduction of the Cyg X-1 results of Krawczynski et al. [25] Section 5 measures how well the checks perform, and Sect. p. 1 Section 4 applies it to IXPE data,
including a reproduction of the Cyg X-1 results of Krawczynski
et al. Section 5 measures how well the checks perform,
and Sect.Where the paper says it
--- Page 4 ---
- The pipeline 2. p. 01. Design principles Verify rather than trust We require that the results of code written by an agent can be verified in three ways First, the code is tested where tests check that results are deterministic and do not change when input is split or combined; they also check that results agree with those of trusted routines and fall within stated sanity bounds Second, the code runs on real data and writes real output files so a human can reproduce its results; third, every output carries its provenance which also names earlier outputs so a human can check that a tool received the right input Two repositories are used where auto research holds orchestration code and research toolkit holds the analysis tools The code is split into two repositories The repository auto research holds the orchestration code (agent graph, prompts, message schemas, sandbox, report generation) The repository research toolkit holds the analysis tools The running pipeline can write to it only through the checked process of Sect. p. 1
Where the paper says it
A semi-autonomous multi-agent pipeline for high-energy astrophysics research with enforced provenance and verifiable self-extension Fang Zhoua,∗ aWashington University in St. Louis, Physics Department, 1 Brookings Dr.
3.2 which commits each new tool to a working branch Each tool is committed as one atomic commit so promoting or reverting a tool is a single operation Each tool consists of two halves, lib/ holding the analysis code and skills/ holding a manifest and short wrappers that tell the agents how to call lib/ functions Choices that are easy to get wrong are hard-coded in the lib/ functions so neither a human nor an agent has to make themWhere the paper says it
The code is split into two repositories. The repository auto_research holds the orchestration code (agent graph, prompts, message schemas, sandbox, report generation).
--- Page 5 ---
Reporting a failure is an acceptable outcome The agent retries an error it can fix within a fixed budget When the agent cannot complete the task for example because the task is ill-posed or data are missing, it reports this to the human We call this a declared failure following Yang et al. [6] (Sect. 1) Prompts tell agents that a declared failure is not penalized A tool-writing session likewise ends with a commit that passed all checks or with a declared failure Its code enters an analysis only as a committed tool
--- Page 6 ---
Section 3 describes the report system and provenance enforcement 3.1. Provenance of reported results The validator checks that the report fills its required fields, cited figure files exist, and each cited output matches its recorded hash before admitting a number A number is admitted only if it appears in the cited output of a toolkit tool, appears in the cited request, or gives its location in a cited paper The following do not support a number shell output and files read by an agent or the output of failed calls or runs on synthetic data and any output computed from such an output
--- Page 7 ---
3.2. Checks on new tools Each new tool passes the gate and review by the co-lead The gate applies seven checks: 1) Determinism, a test that runs twice on real data and checks that outputs agree within a tolerance; 2) Tool-local tests, requiring at least one passing test; 3) Banned vocabulary, scanning code for phrases accompanying invented results; 4) Data-fabricating code rejection of placeholder function bodies and binary data; 5) Parseability to ensure Python files parse correctly; 6) Structure, ensuring the tool carries a description and a test file on allowed paths; and 7) Provenance, verifying that tests call new or changed tools correctly The co-lead judges whether the code follows the brief and if tests are adequate
--- Page 8 ---
- Example investigations 4. p. 31. Analyses and their outcomes We ran the pipeline on four research requests where each analysis wrote new tools or extended existing ones before it produced results 4. p. 3
Where the paper says it
After a failed attempt, it resets the working branch to its base revision. Example investigations 4.
2. Reproduction of a published result We first test the pipeline on a known result, reproducing the IXPE polarization degree (PD) and polarization angle (PA) of Cyg X-1 in the hard state reported by Krawczynski et al. p. 3Where the paper says it
Analyses and their outcomes We ran the pipeline on four research requests. Each analysis wrote new tools or extended existing ones before it produced results.
[25] All 14 quantities agree within 3sigma However, all 10 PD quantities lie below the published values, by 0. p. 3Where the paper says it
Reproduction of a published result We first test the pipeline on a known result. We asked the agents to reproduce the IXPE polarization degree (PD) and polarization angle (PA) of Cyg X-1 in the hard state reported by Krawczynski et al.
2–2.4sigma, for both analysis methods and in all energy bandsWhere the paper says it
24𝜎, so the pipeline reproduces the geometrical conclusions of the original paper. However, all 10 PD quantities, including the slope, lie below the published values, by 0.
--- Page 9 ---
4.3. Polarization stability within an observation The second analysis tests whether the polarization of a black hole X-ray binary (BHXRB) stays constant during a long IXPE observation For five of the six observations, q and u are consistent with being constant with p between 0. p. 3 Polarization stability within an observation
The second analysis tests whether the polarization of a
black hole X-ray binary (BHXRB) stays constant during a long
IXPE observation (see the similar analysis in Paper I). The
pipeline split six observations of four sources (Cyg X-1, Swift
J1727. 3), with 𝑝 between 0. The exception is the Swift J1727. For this data set,
no significant polarization was detected (PD = (0. After accounting for the six trials, the 𝑝-value is 0.Where the paper says it
Where the paper says it
Where the paper says it
--- Page 10 ---
4.5. The q–u time lag The pipeline searched for a time lag between the normalized Stokes parameters q(t) = Q/I and u(t) within single observations Eleven of the 45 observations have enough events for at least eight slices, and the pipeline excluded the other 34 Cyg X-3 observation 02250301 has a DCF peak at a lag of −57. p. 5 Its reviewer agent suspected that the combined correlation was
an artifact of mixing sources. The 𝑞–𝑢 time lag
In the fourth analysis, the pipeline searched for a time lag
between the normalized Stokes parameters 𝑞(𝑡) = 𝑄/𝐼 and
𝑢(𝑡) = 𝑈/𝐼 within single observations. 7 ks with
𝑝 = 0. 0060, below the threshold of 0.Where the paper says it
Where the paper says it
--- Page 11 ---
Figure 8: Signed z of the 14 prescribed quantities in the six runs of the model-tier test
--- Page 12 ---
Figure 8: Signed z of the 14 prescribed quantities in the six runs of the model-tier test
--- Page 13 ---
Figure 7: Cumulative cost of each tool-writing trial against elapsed time, at API list prices, summed over the coder, the lead and the co-lead
--- Page 14 ---
Table 2: Reproduction of the Cyg X-1 measurements of Krawczynski et al. [25] The signed deviation is z = (pipeline − published)/√σ2 pipeline + σ2 published, computed from unrounded values
--- Page 15 ---
Table S2 of the reference (“PD - sum”) The pipeline’s XSPEC rows fit tbabs polconst powerlaw; the reference does not name the model of its XSPEC values The pipeline’s intercept A and slope B come from an XSPEC fit of tbabs pollin powerlaw, with PD(E) = A + B (E/keV − 1)
--- Page 16 ---
Figure 2: The process for writing a new tool, and its two possible outcomes
--- Page 17 ---
Figure C.2: PA (upper panels) and PD (lower panels) of the 5 ks segments of the six observations of the stability analysis
--- Page 18 ---
Figure C.3: PD against HR for each source of the survey, observations with at least four detections
--- Page 19 ---
Figure C.4: PD (2–8 keV) against HR for the 24 observations of the survey with a detection
--- Page 20 ---
Table 1: Summa…
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Enforce strict provenance by requiring that a result (a number, a figure, or a table) passes only if
it appears in a recorded output of a toolkit tool or the request, or if the report gives its location in a cited paper.
-
Implement an automated gate for new tools that includes seven checks:
Determinism,
Tool-local tests,
andBanned vocabulary
scans to reject code containing phrases that accompany invented results. -
Integrate a model from a different provider than the coder as the co-lead for code review to mitigate bias, as the paper notes,
LLM evaluators favour their own output.
-
Use a human decision point to promote new tools:
A human decides whether to promote it to the trusted branch,
ensuring that agents do not act on reviews without human oversight. -
Ensure that reports only admit numbers if they meet strict criteria:
it appears in the cited output of a toolkit tool or in the request, or if the report gives its location in a cited paper.
Abstract
Large language model (LLM) agents are now applied to astronomical data analysis. They retrieve the literature, reduce observations, and draft conclusions. Under pressure to complete a task, such agents can fabricate results, and the numbers they report may not come from the analysis they ran. We present auto research, a semi-autonomous multi-agent pipeline that retrieves and analyses raw data from the High Energy Astrophysics Science Archive Research Center (HEASARC) and writes the analysis tools that it needs. We check the work of the agents at two points. A new tool enters the toolkit after it passes automatic checks, including repeated test runs and a scan for phrases that accompany invented values, and a code review by a model from a different provider; a human decides whether to promote it to the trusted branch. A number in a report passes only if a recorded output of a toolkit tool or the request contains it, or if the report gives its location in a cited paper. We test the pipeline with data from the Imaging X-ray Polarimetry Explorer (IXPE). Of 65 defects that we planted in the toolkit, 47 pass the automatic checks, and one passes the code review as well; the review also rejects 40 of 69 correct changes. The pipeline reproduces the published IXPE results for Cyg X-1: all 14 compared quantities agree within 3 sigma, and the polarization degrees are lower than the published values. With weaker models in the main roles, the pipeline recovers 12 of the 14 quantities and reports the other two as not measured. We also used the pipeline for a polarization-stability analysis of six observations, a survey of polarization against hardness in 29 observations, and a search for a time lag between the Stokes parameters q and u.
Sources
- The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
- AI Agents Can Already Autonomously Perform Experimental High Energy Physics
- Automating High Energy Physics Data Analysis with LLM-Powered Agents
- The Denario project: Deep knowledge AI agents for scientific discovery
- Kosmos: An AI Scientist for Autonomous Discovery
- SciIntegrity-Bench: A Benchmark for Evaluating Academic Integrity in AI Scientist Systems
- Collider-Bench: Benchmarking AI Agents with Particle Physics Analysis Reproduction
- gwBenchmarks: Stress-Testing LLM Agents on High-Precision Gravitational Wave Astronomy
- ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases
- Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents
- How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks
- Open Source Planning & Control System with Language Agents for Autonomous Scientific Discovery
- The AI Cosmologist I: An Agentic System for Automated Data Analysis
- Domain-Specific Agents for Cherenkov Telescope Array Control Software and Gamma-Ray Data Analysis
- The Imaging X-Ray Polarimetry Explorer (IXPE): Pre-Launch
- A Pipeline for the AI-Assisted Analysis of Imaging X-ray Polarimetry Explorer Observations of Black Hole X-ray Binaries and Limits on Their Polarization Variability
- TheoPhysEval: Assessing AI agents at autonomous theoretical physics problem solving
- Voyager: An Open-Ended Embodied Agent with Large Language Models
- ToolUniverse: An open platform for democratizing AI scientists
- Beyond Task Completion: A Verification-vs.-Conformance Gap in Tool-Evolving Agents
Related papers
- On-sky demonstration of a vector Zernike wavefront sensor in a cascaded adaptive optics system
- Euclid Data Release 1 (DR1): Calibration strategy and observations
- The General Antiparticle Spectrometer (GAPS) Antarctic Balloon Payload
- Detection and excision of radio-frequency interference in radio arrays
- Unified matched-filter and excess-power pipeline for gravitational-wave detector characterization
- Ushering in sub-arcsecond angular resolution observatories in the far-infrared: The case for a double-Fourier interferometer