Self-Supervised Representation Learning for JWST Galaxy Images and Low Resolution Spectra

Introduction to the show: ident: Astrophysics Radio. Generated commentary on the latest astrophysics papers.

Vera: We're looking at the paper: "Self-Supervised Representation Learning for JWST Galaxy Images and Low Resolution Spectra".

Jocelyn: Here's the gist of the paper. The gist The study trains a self-supervised multimodal model to recover galaxy stellar masses from sparse JWST data by jointly learning representations from single-band F150W imaging and observed-frame 1–2 µm PRISM spectra, achieving an R2 of ∼ 0.9 at 1 < z ≤ 3 with cross-modal CLIP alignment further improving performance to R2 = 0.90; arXiv:2610.31078v1 Data Construction The researchers constructed a restricted dataset from multiple JWST archives to emulate the sparse band coverage of wide-field surveys, training on single-band F150W imaging and observed-frame 1–2 µm NIRSpec low-resolution spectra The imaging data utilized single-band JWST F150W observations from the OutThere pure-parallel survey (NIRISS) and the NIRCam surveys COSMOS-Web

16: , JADES

17: , and CEERS

18: Cutouts were extracted at 128 × 128 pixels (30 milliarcseconds per pixel, MJy sr−1 units) and centercropped to 100 × 100 after augmentation Spectra were taken from the DAWN JWST Archive (DJA)

19,20: , focusing only on the observed-frame 1–2 µm window (R ∼ 100, a fixed grid of 56 pixels), emulating both a sparse spectral setup and the wavelength coverage of the Roman grism

22: and the Euclid NISP red grisms

23: The matched image-spectrum pairs were grouped by unique DJA source identifier so that multiple cutouts of one source never leak across train/test splits during alignment and downstream tasks Model Architecture The framework employs two modality-specific encoders pre-trained independently in a self-supervised manner on the full imaging and spectroscopic samples The image encoder is a single-channel vision transformer

ViT; 24: pretrained via DINOv2

25: (upper-left panel of Figure 1), which combines the DINO loss, regularized by KoLeo, and the iBOT loss to align the

CLS: token between two networks The spectrum encoder is a transformer autoencoder with a masked-modeling objective

30: , where tokens are tokenized into overlapping flux patches (Appendix B.2), and the loss is weighted by reported uncertainties These modality-specific encoders are then aligned by a CLIP-style head using a contrastive alignment objective to map both modality embeddings into a shared 256-dimensional latent space Performance Evaluation Linear probes were used to benchmark each embedding setup by regressing physical parameters using a single linear transformation on the frozen embeddings Single encoder representations recovered physically meaningful structure, yielding a mean R2 = 0.75 (image encoder) and R2 = 0.87 (spectrum encoder) for catalog redshift over 1 < z ≤ 3 The stellar mass probe on the CLIP-aligned embedding reached R2 = 0.90 (σNMAD = 0.24 dex) at 1 < z ≤ 3, which is a gain of 0.15 over the photometric baseline alone Cross-modal CLIP alignment raised single-modality performance by ∆R2 = +0.038 ± 0.006 for imaging and +0.092 ± 0.01 for spectroscopy Key Findings and Limitations The joint image-spectrum representation achieved R2 = 0.9 in stellar mass, which is a gain of 0.15 over the photometric baseline alone CLIP alignment raised single-modality performance by ∆R2 = +0.04 (imaging) and +0.09 (spectroscopy), narrowing the gap to joint-modality inference The performance of the stellar mass probe depends strongly on redshift: stable below z ≃ 3.5, strongest at 1.5 3.5 to R2 = 0.61 and R2 = 0.40 in the last two bins The degradation at z > 3.5 occurs because the F150W band and the PRISM window sample the rest-frame ultraviolet, which correlates more with recent star formation rather than with assembled stellar mass Future work will use the full spectral coverage and improved labels to extend this method to higher redshifts Self-supervised models trained on JWST data recover physical information from a sparse setup of single-band F150W imaging and the observed-frame 1–2 µm PRISM window, reaching R2 ∼ 0.9 and σNMAD = 0.24 dex at 1 < z ≤ 3 CLIP alignment further raises single-modality performance by ∆R2 = +0.04 (imaging) and +0.09 (spectroscopy), narrowing the gap to joint-modality inference Self-supervised representations thus enable physical parameter estimation under sparse wavelength coverage and should extend to broader regimes, from fewer bands to heterogeneous photometric and spectroscopic combinations The data were obtained from the Mikulski Archive for Space Telescopes (MAST) at the Space Telescope Science Institute, which is operated by the Association of Universities for Research in Astronomy, Inc., under NASA contract NAS 5-03127 for JWST Software and Disclosure The software used includes astropy

41–43: , Python

44: , NumPy

45: , PyTorch

46: , PyTorch Lightning

47: , grizli

48: , msaexp

21: and Optuna

51: AI tools (GPT-5.6 Sol, GPT-5.5, GPT-5.4, GPT-5.3-codex, OpenAI, and Opus 4., Opus 5, Opus 5.5, Haiku 4.5) were used to assist with scripts for batch retrieval of survey data No AI tools were used to develop the model’s core components or to write any section of the manuscript All AI-assisted suggestions were reviewed, tested where applicable, and adopted by the authors References

1: S.

Vera: L.

Jocelyn: Finkelstein et al.

Vera: “CEERS Key Paper.

Paper summary: Vera: So, we're wrapping up this look at the paper "Self-Supervised Representation Learning for JWST Galaxy Images and Low Resolution Spectra." It’s basically about using a self-supervised model to figure out how to get stellar mass from very sparse JWST data.

Jocelyn: Right, so the main idea is training these two different encoders—one for the image and one for the spectrum—separately, just by letting them learn from massive amounts of data, before they start working together.

Subrahmanyan: And that joint representation then lets them predict physical properties like stellar mass even when you only have a single band of imaging and a low-resolution slice of the spectrum.

Vera: The authors achieved an R squared value around zero point nine zero at redshifts between one and three, which is pretty strong for this kind of limited input. They also showed that adding cross-modal alignment from CLIP really pushes that performance up by zero point one five compared to just using the photometric baseline alone for mass estimation.

Jocelyn: That’s interesting because it shows how combining different types of information—the picture and the spectrum—even in a simple self-supervised way, gives you a real boost in accuracy for things like stellar mass.

Subrahmanyan: From a theoretical side, this suggests that we can get meaningful structure information even when the wavelength coverage is incomplete, which is crucial for understanding how galaxies assemble over cosmic time.

Vera: But they did flag some limits. The performance starts dropping off significantly at redshifts higher than three point five because the F150W band and those one to two micro-meter windows start looking at different parts of the spectrum, which is more about recent star formation rather than the total stellar mass.

Jocelyn: So, what they aren't doing right now is using every single band available, which would definitely help push that accuracy even further for those high-redshift galaxies.

Subrahmanyan: That’s exactly where we need more data and better labels to test if this method can handle the full spectrum or if it really needs that sparse setup.

Conclusion: Vera: So, we're wrapping up this look at "Self-Supervised Representation Learning for JWST Galaxy Images and Low Resolution Spectra." It’s about using a self-supervised model to figure out how to get stellar mass from very sparse JWST data.

Jocelyn: Right, so the main idea is training these two different encoders—one for the image and one for the spectrum—separately, just by letting them learn from massive amounts of data before they start working together.

Subrahmanyan: And that joint representation then lets them predict physical properties like stellar mass even when you only have a single band of imaging and a low-resolution slice of the spectrum.

Vera: The authors achieved an R squared value around zero point nine zero at redshifts between one and three, which they found to be pretty strong for this kind of limited input. They also showed that adding cross-modal alignment from CLIP really pushes that performance up by zero point one five compared to just using the imaging data on its own.

Jocelyn: That suggests a way forward where we don't necessarily need every single band available to get meaningful physical information about galaxy structure and evolution in the early universe when using AI tools for representation learning.

Subrahmanyan: From a theoretical side, this suggests that we can get meaningful structure information even when the wavelength coverage is incomplete, which is crucial for understanding how galaxies assemble over cosmic time.

Vera: But they did flag some limits on their method. The performance starts dropping off significantly at redshifts higher than three and a half because the F150W band and those one to two micro-meter windows start looking at different parts of the spectrum, which is more about recent star formation rather than the total stellar mass.

Jocelyn: So, what they aren't doing right now is using every single band available, which would definitely help push that accuracy even further for those high-redshift galaxies.

Subrahmanyan: That’s exactly where we need more data and better labels to test if this method can handle the full spectrum or if it really needs that sparse setup.

Vera: It’s a really neat demonstration of how AI can extract physical meaning from messy, incomplete observational data. Next up, we're going to look at how this works in practice with their code and see what those results actually mean for the next round of JWST observations.