A semi-autonomous multi-agent pipeline for high-energy astrophysics research with enforced provenance and verifiable self-extension

arXiv:2611.05251 · astro-ph.IM, astro-ph.HE · Submitted 2026-11-03 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Astrophysics. The day in astrophysics: observations, models and what the data forces.

Vera: We're looking at the paper: "A semi-autonomous multi-agent pipeline for high-energy astrophysics research with enforced provenance and verifiable self-extension".

Jocelyn: Here's the gist of the paper. Large language model agents are now applied to astronomical data analysis Under pressure to complete a task, such agents can fabricate results and the numbers they report may not come from the analysis they ran The auto research pipeline is a semi-autonomous multi-agent system designed to retrieve and analyze raw data from HEASARC while writing the necessary analysis tools It enforces provenance by ensuring a result passes only if its provenance links it to a toolkit tool, the research request, or a location in a cited paper The pipeline tests new tools through automatic checks including repeated test runs and scans for phrases that accompany invented values, followed by code review from a model from a different provider before human promotion Pipeline Design Principles The design principles of auto research center on verifying rather than trusting the agent's results This verification involves three methods: first, testing the code where tests check that results are deterministic and agree with trusted routines and fall within stated sanity bounds; second, running the code on real data to produce real output files for human reproduction; and third, ensuring every output carries its provenance which names earlier outputs so a human can check the input The system utilizes two repositories, auto research for orchestration code and research toolkit for analysis tools Each tool consists of two halves, lib/ holding the analysis code and skills/ holding a manifest and wrappers that tell agents how to call functions Tool Verification Mechanisms The gate applies seven checks to new tools before they enter the trusted branch These checks include: 1) Determinism, where a tool's tests must run twice on real data and yield the same outcome; 2) Tool-local tests, requiring at least one passing test; 3) Banned vocabulary, scanning comments for phrases accompanying invented results; 4) Data-fabricating code rejection of placeholder function bodies and binary data; 5) Parseability to ensure Python files parse correctly; 6) Structure, ensuring the tool carries a description and a test file on allowed paths; and 7) Provenance, verifying that tests call existing figures and that numbers returned appear in recorded pipeline outputs The co-lead then judges whether the code follows the brief and if tests are adequate Report Validation Provenance enforcement dictates that a result passes from one agent to another or to a human only if its provenance links it to a toolkit tool, the research request, or a location in a cited paper The validator admits a number in the report only if it appears in the cited output of a toolkit tool, appears in the cited request, or gives its location in a cited paper Conversely, shell output and files read by an agent do not support numbers Analysis Outcomes and Benchmarking The pipeline was tested on four research requests, including reproducing the IXPE polarization degree (PD) and polarization angle (PA) of Cyg X-1 results with all 14 quantities agreeing within 3sigma The pipeline reproduced the geometrical conclusions of the original paper but reported a systematic offset in PD values In testing tool writing, checks stopped 64 of the 65 planted defects, and weaker models recovered 12 of the 14 quantities in one run The co-lead model was found to reject 40 of the 69 defect-free controls in a benchmark Limitations and Outlook The study identifies three limitations for auto research, including provenance enforcement establishing origin rather than correctness, the fallibility of the language model co-lead who rejected many controls in testing, and the validator only verifying appearance in a cited output, not correct placement within it The pipeline's design keeps tools that it writes available for later analyses, unlike systems that write new code for each task Future plans involve building these agents from a reusable design and allowing them to start new agents during a run while the validator checks every report The code of the pipeline and toolkit is public, available at specific GitHub repositories Page 1 --- A semi-autonomous multi-agent pipeline for high-energy astrophysics research with enforced provenance and verifiable self-extensionLarge language model (LLM) agents are now applied to astronomical data analysis They retrieve the literature, reduce observations, and draft conclusions Under pressure to complete a task, such agents can fabricate results, and the numbers they report may not come from the analysis they ran We present auto research, a semi-autonomous multi-agent pipeline that retrieves and analyses raw data from the High Energy Astrophysics Science Archive Research Center (HEASARC) and writes the analysis tools that it needs We check the work of the agents at two points A new tool enters the toolkit after it passes automatic checks, including repeated test runs and a scan for phrases that accompany invented values, and a code review by a model from a different provider; a human decides whether to promote it to the trusted branch A number in a report passes only if a recorded output of a toolkit tool or the request contains it, or if the report gives its location in a cited paper We test the pipeline with data from the Imaging X-ray Polarimetry Explorer (IXPE) Of 65 defects that we planted in the toolkit, 47 pass the automatic checks, and one passes the code review as well; the review also rejects 40 of 69 correct changes The pipeline reproduces the published IXPE results for Cyg X-1 all 14 compared quantities agree within 3sigma, and the polarization degrees are lower than the published values With weaker models in the main roles, the pipeline recovers 12 of the 14 quantities and reports the other two as not measured We also used the pipeline for a polarization-stability analysis of six observations, a survey of polarization against hardness in 29 observations, and a search for a time lag between the Stokes parameters q and u Keywords: large language model agents, data analysis, software verification, provenance, X-ray polarimetry, black hole X-ray binaries Page 2 --- Scientists increasingly use large language models (LLMs) in their research An LLM agent is an LLM that calls tools, such as a code interpreter, and acts on their output Agent systems can retrieve the literature, write and run analysis code, and draft the text that reports the results Some assist a scientist at each step, while others take a research question and return a manuscript Tests of LLM agents have revealed three recurring failure modes First, agents fabricate results Second, agents game the checks that are meant to constrain them Third, agents do not act on their own reviews Because of these failure modes, a scientist needs to verify the results of an agent Most systems write new code for each task rather than add it to a reviewed, reusable library Thus the scientist has to check new code for each result Even when a system links each statement to its source, the scientist still has to check the statement In astronomy, agent systems have been applied to cosmological data

12: , galaxy images and cosmological simulations

13: , and to gamma-ray data analysis

14: This project grew as a separate branch from the participation of the author in a group working on the development of an agentic AI pipeline for the analysis of the data from the Imaging X-ray Polarimetry Explorer (IXPE) satellite

15: That pipeline is described in

16, called Paper I hereafter: The auto research pipeline differs in several design principles from the one of Paper I (i) auto research takes project analysis ideas, but does not develop its own ideas

17: ; (ii) the pipeline focuses on the development of a toolkit A new tool is added after automatic checks and an LLM code review A scientist then decides whether to promote the tool to the trusted branch (iii) the automatic checks include quantitative tests of new analysis tools; (iv) the auto research pipeline writes reports about the results Many safeguards act on the reasoning of the agents, for example instructions in the prompt, debate between agents, and critique by a reviewing agent

1, 16: The third failure mode shows their limit because an agent can identify a flaw and still report the unrevised conclusion We therefore check what the agents produce at two points: when the pipeline adds a new tool, and when an agent reports a number For new tools, earlier tool libraries let the model that wrote a tool grade it

18: , or relied on tests and human review

19: Checks on results can miss defective tools in one benchmark where 215 of 222 tools gave wrong results on every held-out input Dong and Chen applied scripted checks followed by an audit by a model from a different provider auto research uses the same combination Each new tool passes automatic checks (the “gate”) and then a review by a model from a provider other than its author’s A human then decides whether to promote the tool to the trusted branch of the toolkit At the second point, we enforce provenance The provenance of an output is the record of the tool and data that produced it Provenance enforcement means a result passes only if its provenance links it to a toolkit tool, research request, or location in a cited paper A toolkit tool is either written by a human or committed after checks on new tools In auto research, each agent ends its task with a report, and a validator admits a number only if it appears in recorded output of a toolkit tool or the request, or if the report gives its location in a cited paper Page 3 --- Section 2 describes auto research which retrieves and analyses raw data from the High Energy Astrophysics Science Archive Research Center (HEASARC) and writes the analysis tools that it needs Section 3 describes its checks on reported numbers and on new tools Section 4 applies it to IXPE data, including a reproduction of the Cyg X-1 results of Krawczynski et al.

25: Section 5 measures how well the checks perform, and Sect.

Jocelyn: 6 tests whether the pipeline still works with weaker models in the main roles Page 4 --- 2.

Vera: The pipeline 2.1.

Paper summary: Vera: We've been looking at this paper, "A semi-autonomous multi-agent pipeline for high-energy astrophysics research with enforced provenance and verifiable self-extension," and it really centers on making AI agents reliable in astronomy. The authors argue that they’re not just letting LLMs write reports; they’re building a whole system around verifying the tools those agents use.

Jocelyn: They focus heavily on this concept of verifiable self-extension, meaning the pipeline can keep writing and updating its own analysis tools as it goes, but only after passing a rigorous gate. This gate involves automated checks like testing for determinism and scanning for invented values before a human co-lead makes the final call to promote the tool.

Subrahmanyan: From my perspective on cosmology and structure formation, this points to how we might use AI to handle the heavy lifting of data reduction and analysis pipelines where human intervention is needed for guidance, but where automation needs strong checks against hallucination. The enforcement of provenance—linking every number back to a specific tool or request—is what makes it scientifically useful in our field.

Vera: So, it's not about the AI being perfect; it's about making the process auditable so we know exactly where any potential error came from when we look at the final scientific conclusion.

Jocelyn: It’s an attempt to give us a tool that can handle complex data analysis tasks autonomously while maintaining a strong link back to established, trusted sources in the literature and our own experimental runs.

Subrahmanyan: The implication is that we might see these pipelines being used for things like polarization stability analyses, where you need consistency over long observation periods, because the system is designed to track those quantities precisely across different steps.

Vera: It’s a step toward integrating LLMs into complex research workflows in a way that respects the need for accountability in scientific reporting.

Conclusion: Vera: So we’re wrapping up this session on "A semi-autonomous multi-agent pipeline for high-energy astrophysics research with enforced provenance and verifiable self-extension."

Jocelyn: It sounds like this paper is about building an AI system that doesn't just give you a finished answer, but a whole process where every single step is traceable back to its original source.

Subrahmanyan: Exactly. They're tackling the problem of AI making up numbers or inventing analysis tools on its own when dealing with massive astronomical datasets like those from HEASARC.

Vera: It’s about verification rather than just trusting the agent’s output, which is crucial because in astronomy, getting a result wrong can have serious consequences for what we actually learn about the sky.

Jocelyn: And they do this by setting up these strict checks—like tests running multiple times on real data and scanning the code for any suspicious language accompanying invented numbers.

Subrahmanyan: The concept of enforced provenance is key here. It means a final result can only be accepted if it can point back to a specific tool used, the original research request, or even a specific paper they cited.

Vera: It’s about creating a system that can self-extend—meaning the pipeline can write its own analysis tools—but only after those rigorous checks are passed by an AI reviewer and then by a human co-lead.

Jocelyn: So, the implication is that we might see research workflows where AI handles the tedious data reduction and tool writing, but a human is still in charge of approving what that tool actually does before it touches the final science.

Subrahmanyan: It shifts the focus from just getting an output to ensuring there's a verifiable chain of custody for every piece of analysis.

Vera: It’s not about replacing the scientist; it’s about giving them a very powerful, self-checking assistant that keeps our work honest and auditable.

Jocelyn: That kind of accountability is what we need when we start relying more on these complex tools for discovering things in deep space.

Fang Zhoua

Washington University in St. Louis

astro-ph.IM, astro-ph.HE

Submitted: 2026-11-03

Updated: 2026-11-03

Code: https://github.com/langchain-ai/langgraph

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 82/100

The gist: Large language model agents are now applied to astronomical data analysis Under pressure to complete a task…

Key concepts

Provenance Enforcement
This system tracks the origin of every result by linking it back to the specific tool used, the original research request, or a cited paper. A number in a final report is only accepted if its source can be traced through these established links.
Tool Verification Gate
Before new analysis tools are trusted, they must pass seven checks: testing for deterministic results on real data, checking for banned phrases indicating fabricated numbers, verifying code structure, and confirming that tests correctly call other tools.
Verification over Trust
The pipeline is designed to verify agent outputs rather than blindly trusting them. Verification involves running code on real data to ensure it produces reproducible output and checking results against trusted routines and sanity bounds.

Terminology

Summary

Large language model agents are now applied to astronomical data analysis Under pressure to complete a task, such agents can fabricate results and the numbers they report may not come from the analysis they ran The auto research pipeline is a semi-autonomous multi-agent system designed to retrieve and analyze raw data from HEASARC while writing the necessary analysis tools It enforces provenance by ensuring a result passes only if its provenance links it to a toolkit tool, the research request, or a location in a cited paper The pipeline tests new tools through automatic checks including repeated test runs and scans for phrases that accompany invented values, followed by code review from a model from a different provider before human promotion

Pipeline Design Principles

The design principles of auto research center on verifying rather than trusting the agent's results This verification involves three methods: first, testing the code where tests check that results are deterministic and agree with trusted routines and fall within stated sanity bounds; second, running the code on real data to produce real output files for human reproduction; and third, ensuring every output carries its provenance which names earlier outputs so a human can check the input The system utilizes two repositories, auto research for orchestration code and research toolkit for analysis tools Each tool consists of two halves, lib/ holding the analysis code and skills/ holding a manifest and wrappers that tell agents how to call functions

Tool Verification Mechanisms

The gate applies seven checks to new tools before they enter the trusted branch These checks include: 1) Determinism, where a tool's tests must run twice on real data and yield the same outcome; 2) Tool-local tests, requiring at least one passing test; 3) Banned vocabulary, scanning comments for phrases accompanying invented results; 4) Data-fabricating code rejection of placeholder function bodies and binary data; 5) Parseability to ensure Python files parse correctly; 6) Structure, ensuring the tool carries a description and a test file on allowed paths; and 7) Provenance, verifying that tests call existing figures and that numbers returned appear in recorded pipeline outputs The co-lead then judges whether the code follows the brief and if tests are adequate

Report Validation

Provenance enforcement dictates that a result passes from one agent to another or to a human only if its provenance links it to a toolkit tool, the research request, or a location in a cited paper The validator admits a number in the report only if it appears in the cited output of a toolkit tool, appears in the cited request, or gives its location in a cited paper Conversely, shell output and files read by an agent do not support numbers

Analysis Outcomes and Benchmarking

The pipeline was tested on four research requests, including reproducing the IXPE polarization degree (PD) and polarization angle (PA) of Cyg X-1 results with all 14 quantities agreeing within 3sigma The pipeline reproduced the geometrical conclusions of the original paper but reported a systematic offset in PD values In testing tool writing, checks stopped 64 of the 65 planted defects, and weaker models recovered 12 of the 14 quantities in one run The co-lead model was found to reject 40 of the 69 defect-free controls in a benchmark

Limitations and Outlook

The study identifies three limitations for auto research, including provenance enforcement establishing origin rather than correctness, the fallibility of the language model co-lead who rejected many controls in testing, and the validator only verifying appearance in a cited output, not correct placement within it The pipeline's design keeps tools that it writes available for later analyses, unlike systems that write new code for each task Future plans involve building these agents from a reusable design and allowing them to start new agents during a run while the validator checks every report The code of the pipeline and toolkit is public, available at specific GitHub repositories

--- Page 1 ---

A semi-autonomous multi-agent pipeline for high-energy astrophysics research with enforced provenance and verifiable self-extensionLarge language model (LLM) agents are now applied to astronomical data analysis They retrieve the literature, reduce observations, and draft conclusions Under pressure to complete a task, such agents can fabricate results, and the numbers they report may not come from the analysis they ran We present auto research, a semi-autonomous multi-agent pipeline that retrieves and analyses raw data from the High Energy Astrophysics Science Archive Research Center (HEASARC) and writes the analysis tools that it needs We check the work of the agents at two points A new tool enters the toolkit after it passes automatic checks, including repeated test runs and a scan for phrases that accompany invented values, and a code review by a model from a different provider; a human decides whether to promote it to the trusted branch A number in a report passes only if a recorded output of a toolkit tool or the request contains it, or if the report gives its location in a cited paper We test the pipeline with data from the Imaging X-ray Polarimetry Explorer (IXPE) Of 65 defects that we planted in the toolkit, 47 pass the automatic checks, and one passes the code review as well; the review also rejects 40 of 69 correct changes The pipeline reproduces the published IXPE results for Cyg X-1 all 14 compared quantities agree within 3sigma, and the polarization degrees are lower than the published values With weaker models in the main roles, the pipeline recovers 12 of the 14 quantities and reports the other two as not measured We also used the pipeline for a polarization-stability analysis of six observations, a survey of polarization against hardness in 29 observations, and a search for a time lag between the Stokes parameters q and u Keywords: large language model agents, data analysis, software verification, provenance, X-ray polarimetry, black hole X-ray binaries

--- Page 2 ---

Scientists increasingly use large language models (LLMs) in their research An LLM agent is an LLM that calls tools, such as a code interpreter, and acts on their output Agent systems can retrieve the literature, write and run analysis code, and draft the text that reports the results Some assist a scientist at each step, while others take a research question and return a manuscript Tests of LLM agents have revealed three recurring failure modes First, agents fabricate results Second, agents game the checks that are meant to constrain them Third, agents do not act on their own reviews Because of these failure modes, a scientist needs to verify the results of an agent Most systems write new code for each task rather than add it to a reviewed, reusable library Thus the scientist has to check new code for each result Even when a system links each statement to its source, the scientist still has to check the statement In astronomy, agent systems have been applied to cosmological data [12], galaxy images and cosmological simulations [13], and to gamma-ray data analysis [14] This project grew as a separate branch from the participation of the author in a group working on the development of an agentic AI pipeline for the analysis of the data from the Imaging X-ray Polarimetry Explorer (IXPE) satellite [15] That pipeline is described in [16, called Paper I hereafter] The auto research pipeline differs in several design principles from the one of Paper I (i) auto research takes project analysis ideas, but does not develop its own ideas [17]; (ii) the pipeline focuses on the development of a toolkit A new tool is added after automatic checks and an LLM code review A scientist then decides whether to promote the tool to the trusted branch (iii) the automatic checks include quantitative tests of new analysis tools; (iv) the auto research pipeline writes reports about the results Many safeguards act on the reasoning of the agents, for example instructions in the prompt, debate between agents, and critique by a reviewing agent [1, 16] The third failure mode shows their limit because an agent can identify a flaw and still report the unrevised conclusion We therefore check what the agents produce at two points: when the pipeline adds a new tool, and when an agent reports a number For new tools, earlier tool libraries let the model that wrote a tool grade it [18], or relied on tests and human review [19] Checks on results can miss defective tools in one benchmark where 215 of 222 tools gave wrong results on every held-out input Dong and Chen applied scripted checks followed by an audit by a model from a different provider auto research uses the same combination Each new tool passes automatic checks (the “gate”) and then a review by a model from a provider other than its author’s A human then decides whether to promote the tool to the trusted branch of the toolkit At the second point, we enforce provenance The provenance of an output is the record of the tool and data that produced it Provenance enforcement means a result passes only if its provenance links it to a toolkit tool, research request, or location in a cited paper A toolkit tool is either written by a human or committed after checks on new tools In auto research, each agent ends its task with a report, and a validator admits a number only if it appears in recorded output of a toolkit tool or the request, or if the report gives its location in a cited paper

--- Page 3 ---

Section 2 describes auto research which retrieves and analyses raw data from the High Energy Astrophysics Science Archive Research Center (HEASARC) and writes the analysis tools that it needs Section 3 describes its checks on reported numbers and on new tools Section 4 applies it to IXPE data, including a reproduction of the Cyg X-1 results of Krawczynski et al. [25] Section 5 measures how well the checks perform, and Sect. p. 1

Where the paper says it

Section 4 applies it to IXPE data, including a reproduction of the Cyg X-1 results of Krawczynski et al. Section 5 measures how well the checks perform, and Sect.

6 tests whether the pipeline still works with weaker models in the main roles

--- Page 4 ---

  1. The pipeline 2. p. 0
    Where the paper says it

    A semi-autonomous multi-agent pipeline for high-energy astrophysics research with enforced provenance and verifiable self-extension Fang Zhoua,∗ aWashington University in St. Louis, Physics Department, 1 Brookings Dr.

    1. Design principles Verify rather than trust We require that the results of code written by an agent can be verified in three ways First, the code is tested where tests check that results are deterministic and do not change when input is split or combined; they also check that results agree with those of trusted routines and fall within stated sanity bounds Second, the code runs on real data and writes real output files so a human can reproduce its results; third, every output carries its provenance which also names earlier outputs so a human can check that a tool received the right input Two repositories are used where auto research holds orchestration code and research toolkit holds the analysis tools The code is split into two repositories The repository auto research holds the orchestration code (agent graph, prompts, message schemas, sandbox, report generation) The repository research toolkit holds the analysis tools The running pipeline can write to it only through the checked process of Sect. p. 1
    Where the paper says it

    The code is split into two repositories. The repository auto_research holds the orchestration code (agent graph, prompts, message schemas, sandbox, report generation).

    3.2 which commits each new tool to a working branch Each tool is committed as one atomic commit so promoting or reverting a tool is a single operation Each tool consists of two halves, lib/ holding the analysis code and skills/ holding a manifest and short wrappers that tell the agents how to call lib/ functions Choices that are easy to get wrong are hard-coded in the lib/ functions so neither a human nor an agent has to make them

--- Page 5 ---

Reporting a failure is an acceptable outcome The agent retries an error it can fix within a fixed budget When the agent cannot complete the task for example because the task is ill-posed or data are missing, it reports this to the human We call this a declared failure following Yang et al. [6] (Sect. 1) Prompts tell agents that a declared failure is not penalized A tool-writing session likewise ends with a commit that passed all checks or with a declared failure Its code enters an analysis only as a committed tool

--- Page 6 ---

Section 3 describes the report system and provenance enforcement 3.1. Provenance of reported results The validator checks that the report fills its required fields, cited figure files exist, and each cited output matches its recorded hash before admitting a number A number is admitted only if it appears in the cited output of a toolkit tool, appears in the cited request, or gives its location in a cited paper The following do not support a number shell output and files read by an agent or the output of failed calls or runs on synthetic data and any output computed from such an output

--- Page 7 ---

3.2. Checks on new tools Each new tool passes the gate and review by the co-lead The gate applies seven checks: 1) Determinism, a test that runs twice on real data and checks that outputs agree within a tolerance; 2) Tool-local tests, requiring at least one passing test; 3) Banned vocabulary, scanning code for phrases accompanying invented results; 4) Data-fabricating code rejection of placeholder function bodies and binary data; 5) Parseability to ensure Python files parse correctly; 6) Structure, ensuring the tool carries a description and a test file on allowed paths; and 7) Provenance, verifying that tests call new or changed tools correctly The co-lead judges whether the code follows the brief and if tests are adequate

--- Page 8 ---

  1. Example investigations 4. p. 3
    Where the paper says it

    After a failed attempt, it resets the working branch to its base revision. Example investigations 4.

    1. Analyses and their outcomes We ran the pipeline on four research requests where each analysis wrote new tools or extended existing ones before it produced results 4. p. 3
    Where the paper says it

    Analyses and their outcomes We ran the pipeline on four research requests. Each analysis wrote new tools or extended existing ones before it produced results.

    2. Reproduction of a published result We first test the pipeline on a known result, reproducing the IXPE polarization degree (PD) and polarization angle (PA) of Cyg X-1 in the hard state reported by Krawczynski et al. p. 3
    Where the paper says it

    Reproduction of a published result We first test the pipeline on a known result. We asked the agents to reproduce the IXPE polarization degree (PD) and polarization angle (PA) of Cyg X-1 in the hard state reported by Krawczynski et al.

    [25] All 14 quantities agree within 3sigma However, all 10 PD quantities lie below the published values, by 0. p. 3
    Where the paper says it

    24𝜎, so the pipeline reproduces the geometrical conclusions of the original paper. However, all 10 PD quantities, including the slope, lie below the published values, by 0.

    2–2.4sigma, for both analysis methods and in all energy bands

--- Page 9 ---

4.3. Polarization stability within an observation The second analysis tests whether the polarization of a black hole X-ray binary (BHXRB) stays constant during a long IXPE observation For five of the six observations, q and u are consistent with being constant with p between 0. p. 3

Where the paper says it

Polarization stability within an observation The second analysis tests whether the polarization of a black hole X-ray binary (BHXRB) stays constant during a long IXPE observation (see the similar analysis in Paper I). The pipeline split six observations of four sources (Cyg X-1, Swift J1727.

75 and 0.98 The exception is the Swift J1727. p. 5
Where the paper says it

3), with 𝑝 between 0. The exception is the Swift J1727.

8−1613 observation 03006001 where no significant polarization was detected After accounting for the six trials, the p-value is 0. p. 5
Where the paper says it

For this data set, no significant polarization was detected (PD = (0. After accounting for the six trials, the 𝑝-value is 0.

046, so variability is marginal

--- Page 10 ---

4.5. The q–u time lag The pipeline searched for a time lag between the normalized Stokes parameters q(t) = Q/I and u(t) within single observations Eleven of the 45 observations have enough events for at least eight slices, and the pipeline excluded the other 34 Cyg X-3 observation 02250301 has a DCF peak at a lag of −57. p. 5

Where the paper says it

Its reviewer agent suspected that the combined correlation was an artifact of mixing sources. The 𝑞–𝑢 time lag In the fourth analysis, the pipeline searched for a time lag between the normalized Stokes parameters 𝑞(𝑡) = 𝑄/𝐼 and 𝑢(𝑡) = 𝑈/𝐼 within single observations.

7 ks with p = 0.0060, below the threshold of 0. p. 5
Where the paper says it

7 ks with 𝑝 = 0. 0060, below the threshold of 0.

01

--- Page 11 ---

Figure 8: Signed z of the 14 prescribed quantities in the six runs of the model-tier test

--- Page 12 ---

Figure 8: Signed z of the 14 prescribed quantities in the six runs of the model-tier test

--- Page 13 ---

Figure 7: Cumulative cost of each tool-writing trial against elapsed time, at API list prices, summed over the coder, the lead and the co-lead

--- Page 14 ---

Table 2: Reproduction of the Cyg X-1 measurements of Krawczynski et al. [25] The signed deviation is z = (pipeline − published)/√σ2 pipeline + σ2 published, computed from unrounded values

--- Page 15 ---

Table S2 of the reference (“PD - sum”) The pipeline’s XSPEC rows fit tbabs polconst powerlaw; the reference does not name the model of its XSPEC values The pipeline’s intercept A and slope B come from an XSPEC fit of tbabs pollin powerlaw, with PD(E) = A + B (E/keV − 1)

--- Page 16 ---

Figure 2: The process for writing a new tool, and its two possible outcomes

--- Page 17 ---

Figure C.2: PA (upper panels) and PD (lower panels) of the 5 ks segments of the six observations of the stability analysis

--- Page 18 ---

Figure C.3: PD against HR for each source of the survey, observations with at least four detections

--- Page 19 ---

Figure C.4: PD (2–8 keV) against HR for the 24 observations of the survey with a detection

--- Page 20 ---

Table 1: Summa…

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Enforce strict provenance by requiring that a result (a number, a figure, or a table) passes only if it appears in a recorded output of a toolkit tool or the request, or if the report gives its location in a cited paper.

  2. Implement an automated gate for new tools that includes seven checks: Determinism, Tool-local tests, and Banned vocabulary scans to reject code containing phrases that accompany invented results.

  3. Integrate a model from a different provider than the coder as the co-lead for code review to mitigate bias, as the paper notes, LLM evaluators favour their own output.

  4. Use a human decision point to promote new tools: A human decides whether to promote it to the trusted branch, ensuring that agents do not act on reviews without human oversight.

  5. Ensure that reports only admit numbers if they meet strict criteria: it appears in the cited output of a toolkit tool or in the request, or if the report gives its location in a cited paper.

Abstract

Large language model (LLM) agents are now applied to astronomical data analysis. They retrieve the literature, reduce observations, and draft conclusions. Under pressure to complete a task, such agents can fabricate results, and the numbers they report may not come from the analysis they ran. We present auto research, a semi-autonomous multi-agent pipeline that retrieves and analyses raw data from the High Energy Astrophysics Science Archive Research Center (HEASARC) and writes the analysis tools that it needs. We check the work of the agents at two points. A new tool enters the toolkit after it passes automatic checks, including repeated test runs and a scan for phrases that accompany invented values, and a code review by a model from a different provider; a human decides whether to promote it to the trusted branch. A number in a report passes only if a recorded output of a toolkit tool or the request contains it, or if the report gives its location in a cited paper. We test the pipeline with data from the Imaging X-ray Polarimetry Explorer (IXPE). Of 65 defects that we planted in the toolkit, 47 pass the automatic checks, and one passes the code review as well; the review also rejects 40 of 69 correct changes. The pipeline reproduces the published IXPE results for Cyg X-1: all 14 compared quantities agree within 3 sigma, and the polarization degrees are lower than the published values. With weaker models in the main roles, the pipeline recovers 12 of the 14 quantities and reports the other two as not measured. We also used the pipeline for a polarization-stability analysis of six observations, a survey of polarization against hardness in 29 observations, and a search for a time lag between the Stokes parameters q and u.

Sources

Related papers