The Agentic Observer Hackathon: Autonomous Decision-making in Simulated Astronomical Surveys
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Astrophysics. The day in astrophysics: observations, models and what the data forces.
Vera: We're looking at the paper: "The Agentic Observer Hackathon".
Jocelyn: Here's the gist of the paper. Astronomical surveys must allocate observing time while responding to changing conditions and competing scientific priorities Within a synthetic multi-object survey with prescribed exposure responses and assignment constraints, decision systems select pointings, assign targets and choose exposure configurations while responding to observing requests, instrumental faults and data loss The leading systems combine numerical exposure planning with language models for interpreting operational information and supporting longer-term decisions Simulation Environment Description The Agentic Observer Hackathon (AOH) presents a community challenge for studying autonomous survey operations For AOH, we define a common survey environment with shared rules for observing geometry, target assignment and exposure response All systems evaluated on a given task receive the same catalogue and observing calendar, and encounter the same weather and event histories The simulation models coupled field selection, target assignment and exposure planning in a synthetic multiobject spectroscopic survey Atmospheric conditions, instrumental performance and operational events evolve during the campaign Survey Modeling Components The simulation environment is structured into six modules representing different stages of a survey operation These modules include Sky and targets, Observing operations, Data processing, Scientific analysis and Survey state Target properties and observing conditions determine each exposure response The catalogue includes stars and four extragalactic populations such as bright galaxies, luminous red galaxies, star-forming emission-line galaxies, and quasars Decision Making and Scoring Objective The decision system receives bulletins, forecasts and exposure feedback as the campaign advances The objective combines retained target response with completeness, coverage and operational outcomes The settled campaign score is defined by the equation S = X Ci − 50Nmiss − 200(1 − J) + Rreq + Rrep This score defines a common set of survey priorities for the benchmark Evaluation and Performance The evaluation compares participant systems with two baseline models, Baseline 1 and Baseline 2, that differ in how they estimate observing conditions and plan exposures The final comparison uses the Python implementations of both baseline models published with the final leaderboard The leading system scores 33 576 and has the highest mean score on each campaign Exploratory Test Findings Exploratory tests show improved fault reporting with model access in several systems, while revised scientific priorities produce substantial changes in ranking Model-enabled runs route calls to a common model service The median score differences Son − Soff is approximately +3 400, and the median difference between the structuredsummary and original model-enabled configurations is approximately +200 Model-enabled runs route calls to a common model service The Spearman correlation with the final evaluation are ρ = 0.85, 0.38 and 0.87 for X1, Y1 and Z1, respectively Model-enabled runs route calls to a common model service The largest rank changes occur under Y1’s combined configuration, where large penalties for incomplete required targets accompany the largest declines The final comparison shows that 46 of 102 systems improve on both baselines under the common survey objective A facility application requires a science objective, calibrated exposure and operating models, and constraints appropriate to its instrument The released simulation and reference strategies support comparisons of observing decisions under these specified models The leading system scores 33 576 and has the highest mean score on each campaign References Andreoni, I., Margutti, R., Banovetz, J., et al.
Vera: 2024, Rubin ToO 2024: Envisioning the Vera C.
Jocelyn: Rubin Observatory Bianco, F.
Vera: B., Ivezić, Ž., Jones, R.
Paper summary: Vera: To recap the paper "The Agentic Observer Hackathon: Autonomous Decision-making in Simulated Astronomical Surveys," the core idea is building decision systems that can autonomously select pointings, assign targets, and choose exposure configurations while constantly reacting to operational changes in a survey.
Jocelyn: The authors set up a common survey environment with shared rules for geometry and target assignment so all the tested systems are evaluated on the exact same conditions, including weather and event histories. They are testing how these agentic systems handle observing requests, instrumental faults, and data loss within this context.
Subrahmanyan: The paper claims that combining numerical exposure planning with language models for interpreting operational information is a way to improve the performance of these decision systems for longer term decisions. This is important because real surveys don't just make one decision; they have to make a sequence of decisions over time.
Vera: It matters because the objective they are trying to hit is complex, combining retained target response with completeness, coverage and operational outcomes, which means the system has to be smart about making trade-offs.
Jocelyn: They show that these agentic systems can handle a lot of this complexity in this simulated environment, providing a benchmark for how these types of planning capabilities work when faced with dynamic operational realities. It's a way to study autonomous survey operations.
Subrahmanyan: The simulation itself has six modules covering the stages of an operation, from the sky and targets all the way through data processing and scientific analysis, which links everything together in a structured flow.
Vera: The catalogue they use is quite rich, featuring stars plus four extragalactic populations: bright galaxies, luminous red galaxies, star-forming emission-line galaxies, and quasars. That diversity tests the system's ability to manage different types of targets simultaneously.
Jocelyn: They are using this setup to compare different scheduling methods against baselines like AstroSA and DeepSurveySim which are already used in real surveys, like the DESI experiment. This gives them a way to see if their new agentic approach can actually do better than existing scheduling techniques.
Subrahmanyan: The paper also touches on how this architecture is promising for applications outside of pure astronomy, suggesting that language model agents combining reasoning with actions are useful in any field where decisions need to respond dynamically.
Vera: So essentially, the paper argues that agentic systems are a viable way to approach the scheduling and planning problems inherent in large astronomical surveys by incorporating reasoning capabilities into the decision-making process.
Jocelyn: It provides a concrete benchmark for how these models perform when used in this specific kind of environment, which is valuable because it moves the discussion from just theoretical capability to practical performance.
Subrahmanyan: The structure of this simulation helps us understand the pipeline better; seeing how inputs flow into execution and feedback across those six modules gives a better picture of the operational reality.
Vera: It really shows that having a robust framework for simulating these environments allows researchers to rigorously test and compare different decision-making strategies in a controlled setting before they try anything on real telescope time.
Conclusion: Jocelyn: So wrapping up the discussion on "The Agentic Observer Hackathon: Autonomous Decision-making in Simulated Astronomical Surveys," the authors are focusing on how this agentic approach handles dynamic operational realities in survey planning.
Vera: They’re essentially showing that by combining numerical exposure planning with language models, you can create decision systems that perform well under these complex, changing conditions, which is what they claim for their benchmark.
Subrahmanyan: It implies that for future large-scale surveys, we might need to think about scheduling not just as a fixed calculation but as a continuous process where the system adapts its priorities based on incoming data and evolving survey status.
Jocelyn: The implications are that agentic systems could be useful tools for automating the planning of complex astronomical observations, moving beyond static methods to something more responsive to real-time feedback.
Vera: It means that we need to ensure our models and constraints are properly calibrated for the specific instrument we are using before deploying these kinds of autonomous decision-making strategies on actual telescope time.
Subrahmanyan: The work gives us a reference point, which is helpful because it shows what a leading system can achieve in terms of performance metrics like that score of thirty-three thousand five hundred seventy-six across campaigns. It’s not a magic solution, but it’s a very strong baseline for comparison against other methods.
Jocelyn: So the paper demonstrates that these agentic systems are capable of handling the planning challenges in this simulated survey environment, and it lays out how to evaluate them fairly against existing scheduling techniques.
Vera: It's about using these simulations to rigorously test if a particular approach actually works under realistic constraints before moving forward with expensive real-world observations.
Subrahmanyan: The main point is that the combination of reasoning and action, as shown in this paper, is a promising direction for how we manage the planning of vast astronomical datasets.
Jocelyn: It's about creating systems that can reason about what needs to be done next based on what has happened so far. That’s where the future work seems to be pointing.
Haosong Wang, Ting Tan, Yao Li, Huanyuan Shan, Dezi Liu, Xiangkun Liu, Yifei Luo, Wei Wang, Zonghuan Wu, Zhongxu Zhai
Shanghai Astronomical Observatory, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing · £IRFU, CEA, Université Paris-Saclay · Upstream Labs (Shenzhen Upstream Labs Technology Co., Ltd) · South-Western Institute for Astronomy Research, Yunnan University · Lawrence Berkeley National Laboratory · State Key Laboratory of Dark Matter Physics, Shanghai Key Laboratory for Particle Physics and Cosmology (MOE), Physics-AI Institute, School of Physics and Astronomy, Shanghai Jiao Tong University · Southern Center for Nuclear-Science Theory (SCNT), Institute of Modern Physics, Chinese Academy of Sciences · Relevant Study LLC · State Key Laboratory of Dark Matter Physics, Tsung-Dao Lee Institute & School of Physics and Astronomy, Shanghai Jiao Tong University · £Waterloo Center for Astrophysics, University of Waterloo
astro-ph.IM
Submitted: 2026-11-01
Updated: 2026-11-01
Comments: 28 pages, 11 figures
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 94/100
The gist: Astronomical surveys must allocate observing time while responding to changing conditions and competing scientific priorities Within a synthetic multi-object survey with prescribed exposure responses…
Key concepts
- Synthetic Multi-Object Survey
- This is a simulated environment designed to mimic a real astronomical survey. It includes shared rules for observing geometry, target assignment, and exposure responses. The simulation evolves atmospheric conditions and instrumental performance during the campaign.
- Decision System Components
- The system being tested selects pointings, assigns targets, and chooses exposure configurations. It must respond dynamically to external inputs like observing requests or unexpected instrumental faults while optimizing a defined survey score.
- Survey Score (S)
- A mathematical formula used to evaluate the performance of the decision system. The score balances retained target response with completeness, coverage, and operational outcomes, using specific penalties for missed targets and incomplete requirements.
Terminology
Summary
Astronomical surveys must allocate observing time while responding to changing conditions and competing scientific priorities Within a synthetic multi-object survey with prescribed exposure responses and assignment constraints, decision systems select pointings, assign targets and choose exposure configurations while responding to observing requests, instrumental faults and data loss The leading systems combine numerical exposure planning with language models for interpreting operational information and supporting longer-term decisions
Simulation Environment Description
The Agentic Observer Hackathon (AOH) presents a community challenge for studying autonomous survey operations For AOH, we define a common survey environment with shared rules for observing geometry, target assignment and exposure response All systems evaluated on a given task receive the same catalogue and observing calendar, and encounter the same weather and event histories The simulation models coupled field selection, target assignment and exposure planning in a synthetic multiobject spectroscopic survey Atmospheric conditions, instrumental performance and operational events evolve during the campaign
Survey Modeling Components
The simulation environment is structured into six modules representing different stages of a survey operation These modules include Sky and targets, Observing operations, Data processing, Scientific analysis and Survey state Target properties and observing conditions determine each exposure response The catalogue includes stars and four extragalactic populations such as bright galaxies, luminous red galaxies, star-forming emission-line galaxies, and quasars
Decision Making and Scoring Objective
The decision system receives bulletins, forecasts and exposure feedback as the campaign advances The objective combines retained target response with completeness, coverage and operational outcomes The settled campaign score is defined by the equation S = X Ci − 50Nmiss − 200(1 − J) + Rreq + Rrep This score defines a common set of survey priorities for the benchmark
Evaluation and Performance
The evaluation compares participant systems with two baseline models, Baseline 1 and Baseline 2, that differ in how they estimate observing conditions and plan exposures The final comparison uses the Python implementations of both baseline models published with the final leaderboard The leading system scores 33 576 and has the highest mean score on each campaign
Exploratory Test Findings
Exploratory tests show improved fault reporting with model access in several systems, while revised scientific priorities produce substantial changes in ranking Model-enabled runs route calls to a common model service The median score differences Son − Soff is approximately +3 400, and the median difference between the structuredsummary and original model-enabled configurations is approximately +200 Model-enabled runs route calls to a common model service The Spearman correlation with the final evaluation are ρ = 0.85, 0.38 and 0.87 for X1, Y1 and Z1, respectively Model-enabled runs route calls to a common model service The largest rank changes occur under Y1’s combined configuration, where large penalties for incomplete required targets accompany the largest declines
Conclusion and Implications
The final comparison shows that 46 of 102 systems improve on both baselines under the common survey objective A facility application requires a science objective, calibrated exposure and operating models, and constraints appropriate to its instrument The released simulation and reference strategies support comparisons of observing decisions under these specified models The leading system scores 33 576 and has the highest mean score on each campaign
References
Andreoni, I., Margutti, R., Banovetz, J., et al. 2024, Rubin ToO 2024: Envisioning the Vera C. p. 0 Operational surveys couple scheduling to measured
conditions. The Rubin scheduler was designed to balance survey objectives (E.Where the paper says it
Bianco, F. B., Ivezić, Ž., Jones, R. p. 6 Schaefer 1991; A. Jones et al.Where the paper says it
Bonaldi, A., Hartley, P., Braun, R., Purser, S., et al. 2025, MNRAS, 543, 1092
Box, G. E. P., Jenkins, G. M., Reinsel, G. C., & Ljung, G. M. 2015, Time Series Analysis: Forecasting and Control
Burleigh, K. J., Landriau, M., Dey, A., et al. 2020, AJ, 160, 61
Calabretta, M. R., & Greisen, E. W. 2002, A&A, 395, 1077
Chaussidon, E., Yèche, C., Palanque-Delabrouille, N., et al. 2023, ApJ, 944, 107
Delgado, F., Saha, A., Chandrasekharan, S., et al. 2014, in Proc. SPIE
DESI Collaboration, Abareshi, B., Aguilar, J., et al. 2022, AJ, 164, 207
DESI Collaboration, Aghamousa, A., Aguilar, J., et al. 2016, The DESI Experiment Part I: Science, Targeting and Survey Design
Guy, J., Bailey, S., Kremin, A., et al. 2023, AJ, 165, 144
Hahn, C., Wilson, M. J., Ruiz-Macias, O., et al. 2023, AJ, 165, 253
Jacobson, A., Kavan, L., & Sorkine-Hornung, O. 2013, ACM Transactions on Graphics
Jain, R. K., Chiu, D.-M. W., & Hawe, W. R. 1984, A Quantitative Measure of Fairness and Discrimination for Resource Allocation in Shared Computer Systems
Jones, A., Noll, S., Kausch, W., Szyszka, C., & Kimeswenger, S. 2013, A&A, 560, A91
Jones, R. L., Yoachim, P., Chandrasekharan, S., et al. 2014, in Proc. SPIE
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. 1998, Artificial Intelligence, 101, 99
Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., & Narayanan, A. 2025, Transactions on Machine Learning Research
Kasten, F., & Young, A. p. 6 Kasten &
A. Young (1989), evaluated with the simulator’s geometric zenith angle and normalised at the zenith (Appendix A. Kasten &
A. Young (1989), evaluated with the simulator’s geometric zenith angle and normalised at the zenith (Appendix A.Where the paper says it
Where the paper says it
Krisciunas, K., & Schaefer, B. p. 6 Krisciunas & B. Schaefer 1991; A.Where the paper says it
Liu, X., Yu, H., Zhang, H., et al. 2024, in International Conference on Learning Representations
Mandelbaum, R., Rowe, B., Bosch, J., et al. 2014, ApJS, 212, 5
Myers, A. D., Moustakas, J., Bailey, S., et al. 2023, AJ, 165, 50
Naghib, E., Yoachim, P., Vanderbei, R. J., Connolly, A. J., & Jones, R. p. 6 Schaefer 1991; A. Jones et al.Where the paper says it
Neilsen, Jr., E. H., Annis, J. T., Diehl, H. T., et al. 2019, Dark Energy Survey’s Observation Strategy, Tactics, and Exposure Scheduler
Neilsen, Jr., E. H., Bernstein, G., Gruendl, R., & Kent, S. 2016, Limiting magnitude, τ, teff, and image quality in DES Year 1
NIST/SEMATECH. 2012, e-Handbook of Statistical Methods
Raichoor, A., Moustakas, J., Newman, J. A., et al. 2023, AJ, 165, 126
Schlafly, E. F., Kirkby, D., Schlegel, D. J., et al. 2023, AJ, 166, 259
Silber, J. H., Fagrelius, P., Fanning, K., et al. 2023, AJ, 165, 9
Singh, S., Nan, Y., Wang, A., et al. 2025, in Advances in Neural Information Processing Systems
Smart, W. M. 1977, Textbook on Spherical Astronomy, 6th edn. p. 2 A penalty for required targets that
remain incomplete gives this subset an additional completion priority. We construct the survey footprints from separated
spherical polygons with irregular boundaries and specified areas.Where the paper says it
Smith, A., He, J.-h., Cole, S., et al. 2019, Monthly Notices of the Royal Astronomical Society, 484, 1285
Terranova, F., Voetberg, M. p. 1 AstroSA and DeepSurveySim provide environments for comparing scheduling
methods (H. Voetberg & B.Where the paper says it
van Oosterom, A., & Strackee, J. 1983, IEEE Transactions on Biomedical Engineering, BME-30, 125
Voetberg, M., & Nord, B. p. 1 Voetberg & B. Nord
2023).Where the paper says it
Wang, C., Zhang, Y., Li, Y., et al. 2025, Communications Engineering, 4, 184
Wang, H., Tan, T., Yao, J., et al. 2026, Executable verification through formalized expert reasoning in astronomical spectroscopy
Wang, R., Jansen, P., Côté, M.-A., & Ammanabrolu, P. 2022, in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (Association for Computational Linguistics), 11279–11298
Xie, H., Kang, Z., & Jiang, X. 2024, Astron. Comput., 47, 100806
Yao, S., Zhao, J., Yu, D., et al. 2023, in International Conference on Learning Representations
Zhou, R., Dey, B., Newman, J. A., et al. 2023, AJ, 165, 58
Zhu, Y., Jin, T., Pruksachatkun, Y., et al.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Adaptive Target Selection based on Dynamic Priorities: The system can select pointings, assign targets, and choose exposure configurations while responding to observing requests, instrumental faults and data loss by combining numerical exposure planning with language models for interpreting operational information and supporting longer-term decisions.
-
Fault Reporting Improvement via Model Access: Systems can improve fault reporting by accessing a model that interprets operational information, which exploratory tests show leads to
improved fault reporting with model access in several systems.
-
Strategic Adaptation to Revised Priorities: The agentic system can demonstrate substantial changes in ranking when revised scientific priorities are introduced, indicating its ability to adapt its decision-making based on new scientific goals.
-
Comprehensive Survey Outcome Scoring: The system can score campaigns by a metric S defined as
S = X i Ci − 50Nmiss − 200(1 − J) + Rreq + Rrep,
which incorporates retained target response, completeness, coverage across right-ascension bands, and operational outcomes. -
Robust Target Completion Diagnosis: The system can trace
target completion through changing conditions and diagnoses errors in the baseline’s inference of synthetic quality bands,
allowing for comparison against independent measurements ofthroughput, sky background and image quality.
-
Efficient Time Management Under Uncertainty: The decision system can manage time by choosing an action from a set including observe, wait, report a fault, or stop, advancing simulated time with
zero transition overhead between exposures.
-
State Estimation for Operational Monitoring: The decision system maintains an internal state throughout a campaign including
the clock, task inputs, complete observation record, telescope and instrument status,
allowing it to map itsobservation–action history to an action.
Abstract
Astronomical surveys must allocate observing time while responding to changing conditions and competing scientific priorities. We present the Agentic Observer Hackathon (AOH), a community challenge for studying autonomous survey operations. Within a synthetic multi-object survey with prescribed exposure responses and assignment constraints, decision systems select pointings, assign targets and choose exposure configurations while responding to observing requests, instrumental faults and data loss. We evaluate systems submitted by 102 teams on campaigns reserved for evaluation and analyse target completion in a reference campaign. The leading systems combine numerical exposure planning with language models for interpreting operational information and supporting longer-term decisions. Exploratory tests show improved fault reporting with model access in several systems, while revised scientific priorities produce substantial changes in ranking. The reference campaign traces target completion and diagnoses errors in the baseline's inference of prescribed quality bands. The released simulator and reference strategies provide common survey conditions for comparing exposure allocation, target selection and recovery from interruptions.
Sources
- Rubin ToO 2024: Envisioning the Vera C. Rubin Observatory LSST Target of Opportunity program
- The DESI Experiment Part I: Science,Targeting, and Survey Design
- AgentBench: Evaluating LLMs as Agents
- Self-Driving Telescopes: Autonomous Scheduling of Astronomical Observation Campaigns with Offline Reinforcement Learning
- DeepSurveySim: Simulation Software and Benchmark Challenges for Astronomical Observation Scheduling
- Executable verification through formalized expert reasoning in astronomical spectroscopy
- ReAct: Synergizing Reasoning and Acting in Language Models
Related papers
- On-sky demonstration of a vector Zernike wavefront sensor in a cascaded adaptive optics system
- Euclid Data Release 1 (DR1): Calibration strategy and observations
- The General Antiparticle Spectrometer (GAPS) Antarctic Balloon Payload
- Detection and excision of radio-frequency interference in radio arrays
- Unified matched-filter and excess-power pipeline for gravitational-wave detector characterization
- Ushering in sub-arcsecond angular resolution observatories in the far-infrared: The case for a double-Fourier interferometer