Identifying Scientifically Interesting Galaxies with Vision-Language Models
astro-ph.IM, astro-ph.GA
Submitted: 2026-10-27
Updated: 2026-10-27
Comments: Accepted to the Machine Learning and the Physical Sciences Workshop (ML4PS 2026)
License: http://creativecommons.org/licenses/by/4.0/
The gist: Anomaly searches in large astronomical image datasets aim to identify rare objects for follow-up investigation.
Abstract
Anomaly searches in large astronomical image datasets aim to identify rare objects for follow-up investigation. Yet the objects that are most statistically unusual are not necessarily those of greatest scientific relevance. We investigate whether pretrained vision-language models can directly assess scientific interest from natural-language instructions, enabling galaxy images to be ranked without labeled examples. We evaluate models on a 20,000-image Hubble Space Telescope sample that includes 167 reference anomalies previously identified through machine learning and expert inspection. By scoring images for scientific interest, Gemini 3.1 Flash-Lite places 98.2% of the reference anomalies within the highest-ranked 2,000 images, at a present-day inference cost of approximately 5.05 per 10,000 images. We recover known gravitational lenses, jellyfish-galaxy candidates, and galaxies with visible jets, and release 102 high-scoring candidates for further investigation. These results show that an off-the-shelf vision-language model, without any training on the target data, offers a practical way to rank large image datasets by scientific relevance rather than rarity. The project code is available at https://github.com/TheCheng2005/Galaxy-Anomaly-Search/
Related papers
- On-sky demonstration of a vector Zernike wavefront sensor in a cascaded adaptive optics system
- Euclid Data Release 1 (DR1): Calibration strategy and observations
- The General Antiparticle Spectrometer (GAPS) Antarctic Balloon Payload
- Detection and excision of radio-frequency interference in radio arrays
- Unified matched-filter and excess-power pipeline for gravitational-wave detector characterization
- Ushering in sub-arcsecond angular resolution observatories in the far-infrared: The case for a double-Fourier interferometer