DINOspec: Efficient Multimodal Alignment of Vision and Spectral Foundation Models for Astronomy

summary

Video file (mp4)

The gist

Astronomical observations provide multimodal views of physical systems, with images and spectra capturing complementary properties of celestial objects.

In short

DINOspec combines pre-trained vision and spectral models from separate sources by using lightweight adapters to align their representations. This method improves galaxy morphology classification (F1: 0.72->0.78) while reusing existing foundation models, showing alignment is most effective when modalities offer complementary information.

Key concepts

Representation Learning Separation
The core idea is to learn visual and spectral features independently first. Instead of training one large model, DINOspec allows vision and spectral encoders to be developed separately. This separation enables the reuse of these independently trained models for other tasks, which is crucial for scientific foundation models.
Adapter-Based Alignment
This mechanism connects the separate vision and spectral branches using small, trainable adapters. These adapters use a symmetric cross-entropy CLIP loss to force the visual and spectral representations to align in a shared space without retraining the entire large backbone. This keeps the model lightweight.
Cross-Modal Contrastive Training
The models are aligned using a CLIP-style contrastive training objective. This involves training on paired astronomical images and spectra from the same objects. The goal is to teach the encoders that visual and spectral data corresponding to the same galaxy should have similar representations, effectively bridging the gap between different data types.

Terminology used across episodes

This episode discusses

The paper

DINOspec: Efficient Multimodal Alignment of Vision and Spectral Foundation Models for Astronomy · Read on arXiv

University of Geneva

Astronomical observations provide multimodal views of physical systems, with images and spectra capturing complementary properties of celestial objects. Scientific foundation models can learn powerful representations from these observations, but representations learned by separate models remain difficult to combine. We investigate whether laten space representations learned by separate vision and spectral models can be aligned without retraining their encoders. We introduce DINOspec, a multimodal framework that aligns a frozen DINOv3 image encoder with a pre-trained AION-1 spectral encoder using lightweight adapters and contrastive learning on 20,472 paired images and spectra of astronomical objects. DINOspec improves galaxy morphology classification (F1: 0.72 to 0.78) and spectral type classification (F1: 0.70 to 0.74) while training at few as 21M parameters. Redshift prediction shows no consistent improvement over the AION-1 baseline of R 2 about0.9, demonstrating that potential benefits of alignment depends on the configuration and downstream task. These results provide an empirical study of composing astronomical representations from independently pretrained foundation models through lightweight alignment modules.

Transcript

Introduction to the show: ident: Astrophysics. The day in astrophysics: observations, models and what the data forces.

Vera: We're looking at the paper: "DINOspec: Efficient Multimodal Alignment of Vision and Spectral Foundation Models for Astronomy".

Jocelyn: Here's the gist of the paper. Astronomical observations provide multimodal views of physical systems, with images and spectra capturing complementary properties of celestial objects.

Vera: DINOspec improves galaxy morphology classification (F1: 0.72→0.78) and spectral classification (F1: 0.70→0.74) while training at most 21M parameters.

Jocelyn: The core problem addressed is the difficulty in combining representations learned by separate vision and spectral models.

Vera: Existing multimodal architectures typically rely on jointly trained modality-specific components, particularly tokenizers optimized for the statistics of the training data.

Paper summary: Vera: So, looking at "DINOspec: Efficient Multimodal Alignment of Vision and Spectral Foundation Models for Astronomy", the authors are showing how to align separate vision and spectral models using lightweight adapters trained on paired astronomical data without retraining the main encoders.

Jocelyn: The paper emphasizes that this method allows us to reuse existing foundation models, which is important because it means we don't have to spend enormous compute time rebuilding them just for a new data type.

Subrahmanyan: The authors suggest that representation learning and cross-modal alignment can be separated, providing a framework for connecting independently developed scientific foundation models through these small alignment modules, each containing only about ten six parameters.

Vera: It boils down to this: if you have strong unimodal representations already, you don't necessarily need to jointly train large multimodal encoders; you can achieve domain adaptation through these lightweight alignment modules.

Jocelyn: The success of using DINOv3 as the vision backbone suggests that we might not always need completely new, domain-specific encoders if the alignment mechanism is strong enough to bridge the gap.

Subrahmanyan: This gives us a concrete way to think about composing independently trained models for astronomy, which is really useful when we want to leverage knowledge from different instruments or datasets.

Conclusion: Vera: So, DINOspec is about taking two separate models, one for seeing images and one for looking at spectra, and figuring out how to make them talk to each other without retraining everything from scratch.

Jocelyn: It’s about this alignment mechanism using adapters so we can reuse those pre-trained components instead of building entirely new ones.

Subrahmanyan: The authors are focusing on separating the representation learning from the cross-modal alignment, which is a way to make these foundation models work together.

Vera: They use a frozen vision backbone, specifically DINOv3 Large, and connect it to a spectral branch using trainable adapters trained with CLIP loss.

Jocelyn: The paper shows that this lightweight adapter approach only uses about seven percent of the parameters of a full multimodal model to achieve this alignment.

Subrahmanyan: That’s significant because it means you can combine independently developed vision and spectral models without needing a massive, jointly trained multimodal encoder.

Vera: The results show improvements in galaxy morphology classification and spectral classification when the data from both modalities are complementary.

Jocelyn: But the paper also points out that if one modality already has the main signal for a task, like redshift prediction, alignment doesn't add much extra info there.

Subrahmanyan: So it seems the real win here is for tasks where seeing an object and seeing its light tells you different things but both pieces of data are needed.

Vera: The implication is that we don't always have to build one huge, monolithic multimodal model if we can use these small alignment modules to bridge the gap.

Jocelyn: It opens up a new path for connecting different scientific models, letting us leverage existing work from various instruments and surveys.

Subrahmanyan: This moves toward a framework where domain adaptation happens through these small modules rather than needing entirely new encoders for every single application.

Vera: And it sets the stage for seeing how this idea applies to other astronomical observations beyond just images and spectra, right?

More episodes

← Home

Watch the video for this episode