The Agentic Observer Hackathon: Autonomous Decision-making in Simulated Astronomical Surveys

summary

The gist

Astronomical surveys must allocate observing time while responding to changing conditions and competing scientific priorities Within a synthetic multi-object survey with prescribed exposure responses…

In short

This work studies autonomous decision-making in synthetic multi-object astronomical surveys. It tests decision systems that select pointings and exposure configurations under changing conditions, instrumental faults, and competing scientific priorities. The leading systems combine numerical planning with language models for better operational decisions.

Key concepts

Synthetic Multi-Object Survey
This is a simulated environment designed to mimic a real astronomical survey. It includes shared rules for observing geometry, target assignment, and exposure responses. The simulation evolves atmospheric conditions and instrumental performance during the campaign.
Decision System Components
The system being tested selects pointings, assigns targets, and chooses exposure configurations. It must respond dynamically to external inputs like observing requests or unexpected instrumental faults while optimizing a defined survey score.
Survey Score (S)
A mathematical formula used to evaluate the performance of the decision system. The score balances retained target response with completeness, coverage, and operational outcomes, using specific penalties for missed targets and incomplete requirements.

Terminology used across episodes

This episode discusses

The paper

The Agentic Observer Hackathon: Autonomous Decision-making in Simulated Astronomical Surveys · Read on arXiv

Haosong Wang, Ting Tan, Yao Li, Huanyuan Shan, Dezi Liu, Xiangkun Liu, Yifei Luo, Wei Wang, Zonghuan Wu,  Zhongxu Zhai

Shanghai Astronomical Observatory, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing · £IRFU, CEA, Université Paris-Saclay · Upstream Labs (Shenzhen Upstream Labs Technology Co., Ltd) · South-Western Institute for Astronomy Research, Yunnan University · Lawrence Berkeley National Laboratory · State Key Laboratory of Dark Matter Physics, Shanghai Key Laboratory for Particle Physics and Cosmology (MOE), Physics-AI Institute, School of Physics and Astronomy, Shanghai Jiao Tong University · Southern Center for Nuclear-Science Theory (SCNT), Institute of Modern Physics, Chinese Academy of Sciences · Relevant Study LLC · State Key Laboratory of Dark Matter Physics, Tsung-Dao Lee Institute & School of Physics and Astronomy, Shanghai Jiao Tong University · £Waterloo Center for Astrophysics, University of Waterloo

Astronomical surveys must allocate observing time while responding to changing conditions and competing scientific priorities. We present the Agentic Observer Hackathon (AOH), a community challenge for studying autonomous survey operations. Within a synthetic multi-object survey with prescribed exposure responses and assignment constraints, decision systems select pointings, assign targets and choose exposure configurations while responding to observing requests, instrumental faults and data loss. We evaluate systems submitted by 102 teams on campaigns reserved for evaluation and analyse target completion in a reference campaign. The leading systems combine numerical exposure planning with language models for interpreting operational information and supporting longer-term decisions. Exploratory tests show improved fault reporting with model access in several systems, while revised scientific priorities produce substantial changes in ranking. The reference campaign traces target completion and diagnoses errors in the baseline's inference of prescribed quality bands. The released simulator and reference strategies provide common survey conditions for comparing exposure allocation, target selection and recovery from interruptions.

Transcript

Introduction to the show: ident: Astrophysics. The day in astrophysics: observations, models and what the data forces.

Vera: We're looking at the paper: "The Agentic Observer Hackathon".

Jocelyn: Here's the gist of the paper. Astronomical surveys must allocate observing time while responding to changing conditions and competing scientific priorities Within a synthetic multi-object survey with prescribed exposure responses and assignment constraints, decision systems select pointings, assign targets and choose exposure configurations while responding to observing requests, instrumental faults and data loss The leading systems combine numerical exposure planning with language models for interpreting operational information and supporting longer-term decisions Simulation Environment Description The Agentic Observer Hackathon (AOH) presents a community challenge for studying autonomous survey operations For AOH, we define a common survey environment with shared rules for observing geometry, target assignment and exposure response All systems evaluated on a given task receive the same catalogue and observing calendar, and encounter the same weather and event histories The simulation models coupled field selection, target assignment and exposure planning in a synthetic multiobject spectroscopic survey Atmospheric conditions, instrumental performance and operational events evolve during the campaign Survey Modeling Components The simulation environment is structured into six modules representing different stages of a survey operation These modules include Sky and targets, Observing operations, Data processing, Scientific analysis and Survey state Target properties and observing conditions determine each exposure response The catalogue includes stars and four extragalactic populations such as bright galaxies, luminous red galaxies, star-forming emission-line galaxies, and quasars Decision Making and Scoring Objective The decision system receives bulletins, forecasts and exposure feedback as the campaign advances The objective combines retained target response with completeness, coverage and operational outcomes The settled campaign score is defined by the equation S = X Ci − 50Nmiss − 200(1 − J) + Rreq + Rrep This score defines a common set of survey priorities for the benchmark Evaluation and Performance The evaluation compares participant systems with two baseline models, Baseline 1 and Baseline 2, that differ in how they estimate observing conditions and plan exposures The final comparison uses the Python implementations of both baseline models published with the final leaderboard The leading system scores 33 576 and has the highest mean score on each campaign Exploratory Test Findings Exploratory tests show improved fault reporting with model access in several systems, while revised scientific priorities produce substantial changes in ranking Model-enabled runs route calls to a common model service The median score differences Son − Soff is approximately +3 400, and the median difference between the structuredsummary and original model-enabled configurations is approximately +200 Model-enabled runs route calls to a common model service The Spearman correlation with the final evaluation are ρ = 0.85, 0.38 and 0.87 for X1, Y1 and Z1, respectively Model-enabled runs route calls to a common model service The largest rank changes occur under Y1’s combined configuration, where large penalties for incomplete required targets accompany the largest declines The final comparison shows that 46 of 102 systems improve on both baselines under the common survey objective A facility application requires a science objective, calibrated exposure and operating models, and constraints appropriate to its instrument The released simulation and reference strategies support comparisons of observing decisions under these specified models The leading system scores 33 576 and has the highest mean score on each campaign References Andreoni, I., Margutti, R., Banovetz, J., et al.

Vera: 2024, Rubin ToO 2024: Envisioning the Vera C.

Jocelyn: Rubin Observatory Bianco, F.

Vera: B., Ivezić, Ž., Jones, R.

Paper summary: Vera: To recap the paper "The Agentic Observer Hackathon: Autonomous Decision-making in Simulated Astronomical Surveys," the core idea is building decision systems that can autonomously select pointings, assign targets, and choose exposure configurations while constantly reacting to operational changes in a survey.

Jocelyn: The authors set up a common survey environment with shared rules for geometry and target assignment so all the tested systems are evaluated on the exact same conditions, including weather and event histories. They are testing how these agentic systems handle observing requests, instrumental faults, and data loss within this context.

Subrahmanyan: The paper claims that combining numerical exposure planning with language models for interpreting operational information is a way to improve the performance of these decision systems for longer term decisions. This is important because real surveys don't just make one decision; they have to make a sequence of decisions over time.

Vera: It matters because the objective they are trying to hit is complex, combining retained target response with completeness, coverage and operational outcomes, which means the system has to be smart about making trade-offs.

Jocelyn: They show that these agentic systems can handle a lot of this complexity in this simulated environment, providing a benchmark for how these types of planning capabilities work when faced with dynamic operational realities. It's a way to study autonomous survey operations.

Subrahmanyan: The simulation itself has six modules covering the stages of an operation, from the sky and targets all the way through data processing and scientific analysis, which links everything together in a structured flow.

Vera: The catalogue they use is quite rich, featuring stars plus four extragalactic populations: bright galaxies, luminous red galaxies, star-forming emission-line galaxies, and quasars. That diversity tests the system's ability to manage different types of targets simultaneously.

Jocelyn: They are using this setup to compare different scheduling methods against baselines like AstroSA and DeepSurveySim which are already used in real surveys, like the DESI experiment. This gives them a way to see if their new agentic approach can actually do better than existing scheduling techniques.

Subrahmanyan: The paper also touches on how this architecture is promising for applications outside of pure astronomy, suggesting that language model agents combining reasoning with actions are useful in any field where decisions need to respond dynamically.

Vera: So essentially, the paper argues that agentic systems are a viable way to approach the scheduling and planning problems inherent in large astronomical surveys by incorporating reasoning capabilities into the decision-making process.

Jocelyn: It provides a concrete benchmark for how these models perform when used in this specific kind of environment, which is valuable because it moves the discussion from just theoretical capability to practical performance.

Subrahmanyan: The structure of this simulation helps us understand the pipeline better; seeing how inputs flow into execution and feedback across those six modules gives a better picture of the operational reality.

Vera: It really shows that having a robust framework for simulating these environments allows researchers to rigorously test and compare different decision-making strategies in a controlled setting before they try anything on real telescope time.

Conclusion: Jocelyn: So wrapping up the discussion on "The Agentic Observer Hackathon: Autonomous Decision-making in Simulated Astronomical Surveys," the authors are focusing on how this agentic approach handles dynamic operational realities in survey planning.

Vera: They’re essentially showing that by combining numerical exposure planning with language models, you can create decision systems that perform well under these complex, changing conditions, which is what they claim for their benchmark.

Subrahmanyan: It implies that for future large-scale surveys, we might need to think about scheduling not just as a fixed calculation but as a continuous process where the system adapts its priorities based on incoming data and evolving survey status.

Jocelyn: The implications are that agentic systems could be useful tools for automating the planning of complex astronomical observations, moving beyond static methods to something more responsive to real-time feedback.

Vera: It means that we need to ensure our models and constraints are properly calibrated for the specific instrument we are using before deploying these kinds of autonomous decision-making strategies on actual telescope time.

Subrahmanyan: The work gives us a reference point, which is helpful because it shows what a leading system can achieve in terms of performance metrics like that score of thirty-three thousand five hundred seventy-six across campaigns. It’s not a magic solution, but it’s a very strong baseline for comparison against other methods.

Jocelyn: So the paper demonstrates that these agentic systems are capable of handling the planning challenges in this simulated survey environment, and it lays out how to evaluate them fairly against existing scheduling techniques.

Vera: It's about using these simulations to rigorously test if a particular approach actually works under realistic constraints before moving forward with expensive real-world observations.

Subrahmanyan: The main point is that the combination of reasoning and action, as shown in this paper, is a promising direction for how we manage the planning of vast astronomical datasets.

Jocelyn: It's about creating systems that can reason about what needs to be done next based on what has happened so far. That’s where the future work seems to be pointing.

More episodes

← Home