DINOspec: Efficient Multimodal Alignment of Vision and Spectral Foundation Models for Astronomy

Astronomical observations provide multimodal views of physical systems, with images and spectra capturing complementary properties of celestial objects. DINOspec improves galaxy morphology classification (F1: 0.72→0.78) and spectral classification (F1: 0.70→0.74) while training at most 21M parameters.

The core problem addressed is the difficulty in combining representations learned by separate vision and spectral models. Existing multimodal architectures typically rely on jointly trained modality-specific components, particularly tokenizers optimized for the statistics of the training data. The central challenge for scientific foundation models is that representations learned from different instruments or datasets are difficult to reuse and combine. This separation allows representation learning from cross-modal alignment, enabling the use of existing foundation models without modifying their learned representations.

Model Structure and Alignment Mechanism

DINOspec adopts a structure consisting of a frozen DINOv3-Large vision backbone and a spectral branch operating on tokens from the frozen AION-1 tokenizer. The vision branch appends transformer blocks to the DINOv3 output, while the spectral branch uses either a linear embedding layer or additional transformer blocks. The two branches are connected through trainable adapters, which align visual and spectral representations using the symmetric cross-entropy CLIP loss.

Experimental Setup and Data Utilization

The study utilizes paired imaging and spectroscopic observations of the same astrophysical objects. Image data originate from the Hyper Suprime-Cam (HSC) survey, providing calibrated optical imaging in five bands. Optical spectra were sourced from the Sloan Digital Sky Survey (SDSS) and the Dark Energy Spectroscopic Instrument (DESI), with a combined dataset containing 209630 object spectra. Cross-matched HSC and DESI objects within a one arcsecond radius were constructed to obtain a dataset containing 20,472 galaxies. These paired observations are used for CLIP-style contrastive training to align image and spectrum data in a shared representation space.

Performance Analysis and Key Findings

DINOspec demonstrates that representation learning and cross-modal alignment can be separated, allowing independently developed foundation models to be reused. Improvements in morphology and spectral classification show alignment is most effective when modalities contain complementary information about the physical object. Conversely, unchanged redshift prediction demonstrates that alignment does not provide additional information when one modality already contains the dominant signal for a task. Visual galaxy morphology classification is the task that benefits most from multimodal joint training.

Conclusion and Future Implications

Adapter-based alignment using at most 7% the parameters of a full multimodal model successfully combines independently pre-trained vision and spectral models while preserving their individual capabilities. DINOspec extends representation alignment methods to astronomical observations, providing a framework for connecting independently developed scientific foundation models through alignment modules each containing only O(10 6) parameters. The success of DINOv3 as a vision backbone suggests that domain-specific encoders are not always required; instead, domain adaptation can be achieved through the alignment modules. The paper concludes that representation learning and cross-modal alignment can be separated to reuse independently developed foundation models.

Appendix Details

The dataset construction details are summarized in Table 4, which lists the datasets used in this study. Adapter pre-training involved optionally pre-training additional transformer blocks using the DINOv3 self-distillation objective on the HSC dataset for vision adapters and a masked-token prediction objective analogous to the BERT objective for spectral adapters. Joint training was performed on paired image–spectrum observations using a symmetric cross-entropy CLIP objective. Evaluation involved seven repetitions with different random seeds to ensure robustness. The final results show that the DINOspec model achieves an F1 score of 0.78 for galaxy morphology classification and an R2 score of approximately 0.9 for spectroscopic redshift prediction. The results show that lightweight alignment can improve downstream performance without jointly training large multimodal encoders, provided sufficiently strong unimodal representations exist.

Table Summary

Table 7 summarizes performance across image and spectral tasks, comparing DINOspec against DINOv3 and AION-1 baselines. The results indicate that the vision backbone alone achieves an F1 score of 0.722 ± 5.2 × 10−3 for morphology classification. The AION-1 model achieves an R2 score of approximately 0.9 for redshift prediction. The overall findings suggest that representation alignment is a viable method for composing independently pre-trained scientific foundation models.

References

The references listed include works such as AstroCLIP [Parker et al., 2024], AstroM3 [Rizhko and Bloom, 2025], and AION-1 [Parker et al., 2025]. Other cited works include DINOv3 [Siméoni et al., 2025] and AstroCLIP [Parker et al., 2024]. The reference list is extensive, including works by Ahumada et al. [Aihara et al., 2018], Collaboration et al. [DESI Collaboration, 2016], and others. The references cover a wide range of astronomical and machine learning literature. The reference list is extensive, including works by Ahumada et al. [Aihara et al., 2018], Collaboration et al. [DESI Collaboration, 2016], and others. The references cover a wide range of astronomical and machine learning literature. The reference list is extensive, including works by Ahumada et al. [Aihara et al., 2018], Collaboration et al. [DESI Collaboration, 2016], and others. The references cover a wide range of astronomical and machine learning literature. The reference list is extensive, including works by Ahumada et al. [Aihara et al., 2018], Collaboration et al. [DESI Collaboration, 2016], and others. The references cover a wide range of astronomical and machine learning literature. The reference list is extensive, including works by Ahumada et al. [Aihara et al., 2018], Collaboration et al. [DESI Collaboration, 2016], and others. The references cover a wide range of astronomical and machine learning literature. The reference list is extensive, including works by Ahumada et al. [Aihara et al., 2018], Collaboration et al. [DESI Collaboration, 2016], and others. The references cover a wide range of astronomical and machine learning literature <ref:2608.

1. Representation Alignment for Scientific Foundation Models: The DINOspec framework can align frozen vision encoders with pre-trained spectral tokenizers using lightweight adapters and contrastive learning to combine independently learned representations without retraining their encoders. This allows reusing existing foundation models for multimodal tasks, improving galaxy morphology classification from F1 0.72 to 0.78 and spectral classification from F1 0.70 to 0.74 while training at most 21 million parameters.

2. Asymmetric Knowledge Transfer Tuning: By varying the initialization and capacity of the trainable adapters, DINOspec can investigate asymmetric knowledge transfer between frozen encoders by varying the initialisation and capacity of the trainable adapters, revealing when unimodal information already dominates a downstream task versus when multimodal alignment improves scientific representations.

3. Efficient Multimodal Composition: DINOspec provides a framework for connecting independently developed scientific foundation models through alignment modules each containing only O(106) parameters, successfully combining pre-trained vision and spectral models while preserving their individual capabilities with at most 7% of the full multimodal model parameters.

4. Domain Adaptation via Alignment Modules: The system demonstrates that domain adaptation can be achieved through the alignment modules, although this may require additional trainable capacity, suggesting that domain-specific encoders are not always required for scientific tasks like galaxy morphology classification.

Astronomical observations provide multimodal views of physical systems, with images and spectra capturing complementary properties of celestial objects. Scientific foundation models can learn powerful representations from these observations, but representations learned by separate models remain difficult to combine. We investigate whether laten space representations learned by separate vision and spectral models can be aligned without retraining their encoders. We introduce DINOspec, a multimodal framework that aligns a frozen DINOv3 image encoder with a pre-trained AION-1 spectral encoder using lightweight adapters and contrastive learning on 20,472 paired images and spectra of astronomical objects. DINOspec improves galaxy morphology classification (F1: 0.72 to 0.78) and spectral type classification (F1: 0.70 to 0.74) while training at few as 21M parameters. Redshift prediction shows no consistent improvement over the AION-1 baseline of R 2 about0.9, demonstrating that potential benefits of alignment depends on the configuration and downstream task. These results provide an empirical study of composing astronomical representations from independently pretrained foundation models through lightweight alignment modules.