COSMIO: A Benchmark for Cross-Survey Modality Imputation

arXiv:2611.04139 · astro-ph.IM, cs.CV · Submitted 2026-11-03 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Astrophysics. The day in astrophysics: observations, models and what the data forces.

Vera: We're looking at the paper: "COSMIO: A Benchmark for Cross-Survey Modality Imputation".

Jocelyn: Here's the gist of the paper. Large-scale astronomical surveys define a heterogeneous, partially observed multisurvey learning setting: each object is observed by only a subset of surveys, while survey observations differ in resolutions, wavelength coverage, noise characteristics, and footprints Given observations of an object in some surveys, how to infer its appearance in other surveys The benchmark enables systematic evaluation of imputation across source-target survey combinations and missing-survey patterns.

Vera: The Problem Formalization The setting is formalized as the problem of cross-survey modality imputation, where the task is to infer the appearance of the same astrophysical source in a target survey from its available observations in one or more source surveys.

Jocelyn: This involves modeling the conditional distribution pθ x(T) x(O), where x(O) = y(k): k ∈ O and x(T) = y(k): k ∈ T.

Vera: The problem is particularly consequential during the initial observation phase of a new survey, when only a limited region of the sky has been covered.

Paper summary: Vera: So we're looking at this paper titled "COSMIO: A Benchmark for Cross-Survey Modality Imputation." Basically, they've set up this setup where you have multiple large astronomical surveys, and each object is only seen by a piece of those surveys. The core idea is figuring out how to guess what that object looks like in the surveys it hasn't been observed in yet.

Jocelyn: It’s about modeling this conditional distribution, which means taking what you see in some surveys and trying to infer the appearance of the same object in a target survey. They frame this as cross-survey modality imputation, and they want to see if models can do it well across different survey types and missing data patterns.

Subrahmanyan: It’s important because this is happening right now with big projects like LSST, Euclid, and Roman that are all observing the same sky but with different characteristics—different resolutions, wavelength coverage, noise levels. This whole problem is consequential when we're starting out with a new survey where only a small patch of the sky has been covered.

Vera: Exactly. The paper introduces COSMIO as this benchmark dataset. They’ve rendered one hundred fifteen thousand regular galaxy samples and fourteen thousand, eight hundred twenty five strong-lensing systems under realistic LSST, Euclid, and Roman survey configurations. This gives them a real test bed for the problem they are studying.

Jocelyn: They use this benchmark to ask three main questions: can models actually infer missing observations from what we have? Can models trained on common objects generalize when we look at scientifically important rare-event regimes? And finally, can these imputed images actually help us do downstream scientific tasks?

Subrahmanyan: That third question is critical. The authors found three main things. First, methods that work well for the most established imputation setting—multi-contrast brain MRI imputation—they transfer poorly to astronomical survey data and get beaten by simpler diffusion baselines, which shows a clear gap between the domains.

Vera: That’s a big finding because it suggests that just taking a model from one area and applying it to astronomy isn't going to work well. And second, they found that models trained on common galaxies actually degrade when you test them on rare galaxy-galaxy strong-lensing systems.

Jocelyn: And third, while the imputed observations can help with downstream tasks like detecting strong lenses, their performance still doesn't match what you'd see with real target survey observations, and it really depends on how good the reconstruction quality is.

Subrahmanyan: So what this means for the bigger picture is that they’ve established cross-survey imputation as a separate setting entirely. They highlight that we need dedicated methods instead of just reusing tools from other fields to solve these multi-survey problems.

Vera: It really does, and they emphasize that missing-modality rare-event imputation is a major challenge when we look at the early stages of new surveys. Strong gravitational lensing is used as a representative example of this rare event setting.

Paper summary: Jocelyn: And they point out that performance degrades under shifts in the rare-event distribution, which means if you miss a scientifically valuable phenomenon, your model struggles even more with it. This degradation then carries over to the downstream lens detection task.

Subrahmanyan: That connects back to structure formation and cosmology because if we can't reliably detect these lenses from imputed data, we lose information about the large-scale structure of the universe that those lenses help us map out.

Vera: The authors also mentioned their limitation, which is that the dataset itself is constructed through forward physical rendering. They admit this introduces a sim-toreal gap, and because scientifically important phenomena are rare, they feel simulation is necessary to test these concepts.

Jocelyn: That’s a fair point on the limitation. But they've made this benchmark interesting precisely because it forces us to confront that gap and test how well different imputation approaches handle it before we apply them to real data.

Subrahmanyan: The results also show that asynchronous diffusion has the lowest reported error measure, CRPS, in each target-survey group. This suggests that async diffusion and conditional diffusion consistently beat the medical imputation baselines they compared against, indicating architecture choices tailored for medical imaging don't translate well here.

Vera: It confirms what Jocelyn said about the domain gap. The methods developed for MRI don't generalize to this astronomical setting; you need something built specifically for this cross-survey modality problem.

Jocelyn: And they point out that the choice of statistical statistic matters a lot in their ranking, with a dedicated estimator having better statistical properties than just looking at image-level generative imputation through morphology and rare-event detection.

Subrahmanyan: That’s a good caution for anyone trying to use these results. You need to think about what you are actually measuring when you compare the performance of different methods.

Vera: And finally, for downstream utility, they measure things like AUROC and AUPRC for lens classification, which directly tests how useful the imputed images are for actual scientific tasks. They found that asynchronous diffusion is still the best performer in those metrics.

Jocelyn: So to wrap up this paper "COSMIO: A Benchmark for Cross-Survey Modality Imputation," it shows that cross-survey imputation is a distinct learning problem, different from traditional imputation domains like brain MRI. It highlights rare-event imputation as a key bottleneck, especially when we are studying early observations in new surveys.

Subrahmanyan: The implication is that we need to develop methods specifically for multi-survey settings, because off-the-shelf models don't work well here, and current models degrade when faced with scientifically important rare events like strong lensing.

Vera: The authors are looking ahead by planning to crossmatch Rubin LSST and Euclid observations with real data, which is a smart next step to see if their benchmark conclusions hold up when applied to actual telescope data.

Jocelyn: So the main point is that we need new tools for this, and the paper gives us a good idea of what those tools should look like by showing where current approaches fail and where they succeed.

Conclusion: Vera: So we're wrapping up this discussion on COSMIO, which is this benchmark for cross-survey modality imputation, and who wrote it—the authors are working on modeling how to guess what an object looks like in a survey it hasn't been observed in yet.

Jocelyn: Yeah, so essentially they built this test case using data from LSST and Euclid and Roman surveys to see if we can actually fill in the blanks when we only have partial data for an object.

Subrahmanyan: It’s about modeling that conditional distribution, ptheta x(T) given x(O), which is like taking what you see in survey one and trying to figure out what it looks like in survey two.

Vera: And the authors are showing that their setup is distinct from the typical imputation problems we usually see, so we need new tools for this kind of cross-survey scenario.

Jocelyn: They’re pointing out that methods developed for other things, like brain MRI imaging, just don't transfer well to astronomical survey data.

Subrahmanyan: That’s a big thing because it tells us that the architecture choices we use need to be tailored specifically for these multi-survey settings.

Vera: And they flag missing-modality rare-event imputation as a major hurdle, especially when we're looking at new surveys where rare but important things might only show up in older data.

Jocelyn: They used strong gravitational lensing as an example of a rare event setting that makes this problem particularly tricky.

Subrahmanyan: It means if we miss those scientifically valuable phenomena from prior surveys, our models will struggle even more with them later on, and that affects how well we can do the science.

Vera: And while they've built this benchmark through physical rendering, they admit it has a sim-toreal gap, so it’s important to remember that the results are based on a simulation.

Jocelyn: So the main point is that off-the-shelf methods don't work here and we need new approaches for these multi-survey problems.

Subrahmanyan: It really highlights that when we look at early observations of new surveys, we need to focus on figuring out how to handle those rare events correctly.

Vera: Next up, Jocelyn and I are going to look at what this means for real follow-up observations and planning future telescope time.

Dichang Zhang, Yixuan Shao, Jiaqi Deng, Jiali Cui, Haotian Yin, Yuanpeng Liu, Xiang Gao, Zhiqiang Lao

Department of Computer Science, Stony Brook University, USA · Department of Physics and Astronomy, Stony Brook University, USA · School of Computer Science, University of Technology Sydney Australia · Futurewei Technologies USA · Department of Computer Science, New Jersey Institute of Technology USA

astro-ph.IM, cs.CV

Submitted: 2026-11-03

Updated: 2026-11-03

Comments: Accepted at NeurIPS 2026. 34 pages, 8 figures

Code: https://github.com/fw-ic/COSMIO

License: http://creativecommons.org/licenses/by-sa/4.0/

Importance score: 90/100

The gist: Large-scale astronomical surveys define a heterogeneous, partially observed multisurvey learning setting: each object is observed by only a subset of surveys…

Key concepts

Cross-survey Modality Imputation
This is the task of guessing how a celestial object looks in one survey when you only have data from other, different surveys. The surveys differ in things like image quality or wavelength, and the goal is to fill in the missing information to create a complete picture.
COSMIO Benchmark
COSMIO is a test set containing realistic observations of galaxies rendered under different survey conditions (LSST, Euclid, Roman). It includes regular galaxies and strong-lensing systems, allowing researchers to systematically evaluate how well imputation methods work across various missing data patterns.
Rare Event Imputation
This refers to imputing data for scientifically valuable but infrequent phenomena, such as strong gravitational lensing. The paper finds that models struggle when the rare event distribution shifts, meaning they perform poorly on these important but underrepresented astronomical features.

Terminology

Summary

Large-scale astronomical surveys define a heterogeneous, partially observed multisurvey learning setting: each object is observed by only a subset of surveys, while survey observations differ in resolutions, wavelength coverage, noise characteristics, and footprints Given observations of an object in some surveys, how to infer its appearance in other surveys The benchmark enables systematic evaluation of imputation across source-target survey combinations and missing-survey patterns. p. 0

Where the paper says it

au Abstract Large-scale astronomical surveys define a heterogeneous, partially observed multisurvey learning setting: each object is observed by only a subset of surveys, while survey observations differ in resolutions, wavelength coverage, noise characteristics, and footprints. This setting gives rise to a natural machine learning problem: given observations of an object in some surveys, how to infer its appearance in other surveys.

The Problem Formalization

The setting is formalized as the problem of cross-survey modality imputation, where the task is to infer the appearance of the same astrophysical source in a target survey from its available observations in one or more source surveys. This involves modeling the conditional distribution pθ x(T) x(O), where x(O) = y(k): k ∈ O and x(T) = y(k): k ∈ T. p. 3

Where the paper says it

, K} denote the set of available modalities, and let x = x (k) k∈V be the collection of observations of the same underlying object under these acquisition functions. Given an observed subset O ⊂ V and a target subset T ⊆ (V \ O), the imputation task is to model the conditional distribution pθ x (T ) | x (O) , (2) where x (O) = {x (k) : k ∈ O} and x (T ) = {x (k) : k ∈ T }.

The problem is particularly consequential during the initial observation phase of a new survey, when only a limited region of the sky has been covered. p. 1
Where the paper says it

Such a capability would support more unified analysis across complementary surveys despite incomplete coverage. The problem is particularly consequential during the initial observation phase of a new survey, when only a limited region of the sky has been covered.

The Benchmark Construction

COSMIO is introduced as a benchmark of forward-rendered, aligned multi-survey observations. It contains 115k regular galaxy samples and 14,825 strong-lensing systems rendered under realistic LSST, Euclid, and Roman survey configurations. p. 2

Where the paper says it

Our contributions are as follows: (i) We formulate cross-survey modality imputation as a new learning problem setting arising from partially overlapping astronomical surveys, and define the associated scientific setting and machine learning tasks. (ii) We introduce COSMIO, a benchmark with 115k regular galaxy samples and 14,825 strong-lensing systems across LSST, Euclid, and Roman, enabling systematic evaluation under realistic cross-survey configurations.

The scenes are generated by applying a survey-specific observation operator x(k) = Ok(s) + ϵk, where k ∈

Key Findings on Model Transfer

The empirical study yields three main findings. First, methods developed for the most established missing-modality imputation setting, namely multi-contrast brain MRI imputation transfer poorly to astronomical survey data and are often outperformed by simple diffusion baselines, revealing a clear domain gap. p. 2

Where the paper says it

Our empirical study yields three main findings. First, methods developed for the most established missingmodality imputation setting, namely multi-contrast brain MRI imputation [10, 11, 12, 13, 14, 15], transfer poorly to astronomical survey data and are often outperformed by simple diffusion baselines, revealing a clear domain gap.

Second, models trained on common galaxies degrade when evaluated on rare galaxy–galaxy strong-lensing systems. p. 2
Where the paper says it

First, methods developed for the most established missingmodality imputation setting, namely multi-contrast brain MRI imputation [10, 11, 12, 13, 14, 15], transfer poorly to astronomical survey data and are often outperformed by simple diffusion baselines, revealing a clear domain gap. Second, models trained on common galaxies degrade when evaluated on rare galaxy–galaxy strong-lensing systems.

Third, imputed observations can support downstream strong-lens detection, but their performance remains below that of real target-survey observations and depends strongly on reconstruction quality. p. 2
Where the paper says it

Second, models trained on common galaxies degrade when evaluated on rare galaxy–galaxy strong-lensing systems. Third, imputed observations can support downstream strong-lens detection, but their performance remains below that of real target-survey observations and depends strongly on reconstruction quality.

Rare Event Challenges

Missing-modality rare-event imputation is identified as a central challenge arising in the early stages of new surveys when rare but scientifically valuable phenomena may be available only from prior surveys. Strong gravitational lensing is used as a representative rare-event setting. p. 0

Where the paper says it

This regime arises in the early stages of new surveys, when rare but scientifically valuable phenomena may be available only from prior surveys. We use strong gravitational lensing, where light from a background galaxy is gravitationally deflected by a foreground galaxy, as a representative rare-event setting.

Performance degrades under rare-event distribution shift, and this degradation further propagates to the downstream lens-detection task. p. 0
Where the paper says it

We evaluate model performance on zero-shot imputation of gravitational lenses and on downstream scientific tasks using the imputed lens observations. On our constructed evaluation set of galaxy–galaxy strong lenses, performance degrades under rare-event distribution shift, and this degradation further propagates to the downstream lens-detection task.

Methodological Contributions

The contributions include formulating cross-survey modality imputation as a new learning problem setting and introducing COSMIO, a benchmark with 115k regular galaxy samples and 14,825 strong-lensing systems across LSST, Euclid, and Roman. The work shows that methods developed for existing imputation tasks transfer poorly to this regime and identifies rare-event imputation as a key challenge which affects downstream scientific applications. p. 2

Where the paper says it

(iii) We show that methods developed for existing imputation tasks transfer poorly to this regime and identify rare-event imputation as a key challenge which affects downstream scientific applications. 2 Related Work 2.

Future Directions

Potential applications include generating candidate realizations before follow-up or Target-of-Opportunity (ToO) observations to support planning and prioritization. Future work plans to crossmatch Rubin/LSST and Euclid observations with real data to test whether conclusions from the controlled benchmark hold on real observations. p. 9

Where the paper says it

We will compare method rankings, relative performance gaps, and failure modes with those from the corresponding COSMIO LSST– Euclid tasks. This evaluation will test whether conclusions from the controlled benchmark hold on real observations, with training and testing performed within each domain.

Data and Code Availability

The dataset is available at https://www.kaggle.com/datasets/cosmidata/cosmio. p. 1

Where the paper says it

com/fw-ic/COSMIO, and the dataset is available at https://www. com/datasets/cosmidata/cosmio.

The repository is available at https://github. p. 1
Where the paper says it

repository: https://github. com/fw-ic/COSMIO, and the dataset is available at https://www.

com/fw-ic/COSMIO. p. 1
Where the paper says it

repository: https://github. com/fw-ic/COSMIO, and the dataset is available at https://www.

Limitations

The dataset is constructed through forward physical rendering, which inevitably introduces a sim-toreal gap. The rarity of scientifically important phenomena further necessitates simulation. p. 9

Where the paper says it

This simulation-first paradigm is standard in astronomy, where physically grounded forward models are widely used to approximate observational processes. Moreover, the rarity of scientifically important phenomena further necessitates simulation.

This limitation is one motivation for introducing cross-survey imputation as a complement to pure physical simulation. p. 9
Where the paper says it

This limits the reliability of method development and calibration when they are based solely on physical simulations. This limitation is one motivation for introducing cross-survey imputation as a complement to pure physical simulation.

Experimental Reproducibility

All experimental setups are in Section 4.2. All architecture details are in Appendix C. p. 4

Where the paper says it

Figure 2 shows some sample data from the dataset. Implementation details, including the software packages, cosmological parameters, survey bandpasses, selection cuts, and data format, are provided in Appendix B.

Release code and data also contains everything include config files for reproduction. p. 4
Where the paper says it

In the current dataset release, we use cutout sizes of 64×64 for Euclid, 32×32 for LSST, and 59×59 for Roman, yielding approximately matched angular coverage across the three surveys while retaining their native sampling characteristics. In total, COSMIO contains 115,000 common-galaxy scenes and 14,825 scenes containing strong lensing systems.

Results Summary

Asynchronous diffusion has the lowest reported CRPS in each target-survey group. Both Asynchronous diffusion and conditional diffusion consistently outperform medical imputation baselines. p. 7

Where the paper says it

Asynchronous diffusion has the lowest reported CRPS in each target-survey group. Both Asynchronous diffusion and conditional diffusion consistently outperform medical imputation baselines.

This suggests that cross-survey modality imputation differs fundamentally from MRI imputation, and that architecture choices tailored to medical imaging do not generalize well to the astronomical setting. p. 7
Where the paper says it

Both Asynchronous diffusion and conditional diffusion consistently outperform medical imputation baselines. This suggests that cross-survey modality imputation differs fundamentally from MRI imputation, and that architecture choices tailored to medical imaging do not generalize well to the astronomical setting.

Diagnostic Insights

The ranking depends on the scientific statistic, a dedicated estimator with well-understood statistical properties is preferable. The superior total-flux result of Sérsic-FM illustrates this distinction; it does not contradict the motivation for evaluating image-level generative imputation through morphology and rare-event detection. p. 8

Where the paper says it

By contrast, strong-lensing systems contain extended arc- or ring-like structures caused by lensing, which deviate from the dominant morphology learned from common galaxies. Probabilistic generative imputation models therefore tend to favor high-density target-modality structures when the rare-event evidence is ambiguous, causing the ring to fragment into compact galaxy-like point sources.

Downstream Utility

Downstream lens classification, measured by AUROC and AUPRC, directly evaluates the utility of the imputed images for our demonstrated scientific task. Asyn-Diff remains the best-performing method and RePaint remains the lowest-performing method at every reported prior. p. 8

Where the paper says it

95 AUROC GT Roman 0. 95 AUPRC GT Roman CRPS Task LSST+Euclid -> Roman LSST -> Roman Euclid -> Roman Method M2DN RePaint Palette asyn-diffusion Figure 5: Relationship between imputation quality and downstream classification performance.

Conclusion

In this work, we introduced COSMIO, a benchmark for cross-survey astronomical modality imputation. Our experiments show that this setting is distinct from conventional imputation domains, as off-the-shelf methods do not readily generalize to astronomical survey data. p. 9

Where the paper says it

5 Conclusion In this work, we introduced COSMIO, a benchmark for cross-survey astronomical modality imputation. Our experiments show that this setting is distinct from conventional imputation domains, as off-the-shelf methods do not readily generalize to astronomical survey data.

We identify rare-event imputation as a key bottleneck, with current models degrading on scientifically important but underrepresented phenomena. p. 9
Where the paper says it

Our experiments show that this setting is distinct from conventional imputation domains, as off-the-shelf methods do not readily generalize to astronomical survey data. We identify rare-event imputation as a key bottleneck, with current models degrading on scientifically important but underrepresented phenomena.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

1

No off-the-shelf imputation methods transfer well to astronomical survey data, suggesting that cross-survey imputation constitutes a distinct learning problem. This indicates a need for dedicated methods robust to distribution shift rather than direct extension of medical image imputation techniques.

2

The system can perform zero-shot imputation of rare events, such as galaxy–galaxy strong lensing, by training on common galaxies and evaluating on rare systems without lens-target supervision. This allows the model to test its ability to preserve scientifically important rare morphology.

3

The AI system can support downstream scientific tasks like lens detection by using imputed observations, though performance remains below real target-survey observations and is dependent on reconstruction quality. This helps in stress-testing early-stage image analysis pipelines.

Abstract

Large-scale astronomical surveys define a heterogeneous, partially observed multi-survey learning setting: each object is observed by only a subset of surveys, while survey observations differ in resolutions, wavelength coverage, noise characteristics, and footprints. This setting gives rise to a natural machine learning problem: given observations of an object in some surveys, how to infer its appearance in other surveys. We formalize this as the problem of cross-survey modality imputation and introduce COSMIO, a multi-survey benchmark of aligned observations obtained by forward-rendered shared astrophysical scenes across LSST, Euclid, and Roman under survey-specific instrument characteristics. The benchmark enables systematic evaluation of imputation across source-target survey combinations and missing-survey patterns. We evaluate a broad range of machine learning methods for imputation, which reveal a clear domain gap: methods developed for existing imputation settings transfer poorly to the survey setting. We identify rare-event imputation as a central challenge. This regime arises in the early stages of new surveys, when rare but scientifically valuable phenomena may be available only from prior surveys. We use strong gravitational lensing, where light from a background galaxy is gravitationally deflected by a foreground galaxy, as a representative rare-event setting. We evaluate model performance on zero-shot imputation of gravitational lenses and on downstream scientific tasks using the imputed lens observations. On our constructed evaluation set of galaxy-galaxy strong lenses, performance degrades under rare-event distribution shift, and this degradation further propagates to the downstream lens-detection task. Code: https://github.com/fw-ic/COSMIO.

Sources

Related papers