Anomaly detection in Fink. I. Discovery, follow-up, and classification of unusual sources

summary

The gist

Anomaly detection in Fink presents the first-year performance of an anomaly-detection (AD) pipeline operating within the Fink broker on the Zwicky Transient Facility alert stream to identify…

In short

The Fink pipeline uses machine learning to detect unusual astronomical events from Zwicky Transient Facility data. It transforms light curve data into features and uses Isolation Forest models to rank alerts. This system successfully found rare objects like AM CVn systems and previously unreported supernovae, bridging large surveys with expert knowledge.

Key concepts

Isolation Forest Model
This is a machine learning technique used to identify outliers in data. The model works by randomly partitioning the data and isolating points that are few or different from the rest. It's effective for finding rare events like unusual transients without needing to know what 'normal' looks like beforehand.
Expert Feedback Loop
This is a process where domain experts review automated detections and provide feedback, such as marking an object as 'anomalous' or 'nominal'. This feedback is then used to retrain the anomaly detection model. It allows the system to learn what specific scientific anomalies mean within the context of astronomical research.
Active Anomaly Discovery Algorithm
This algorithm extends standard anomaly detection by incorporating expert feedback iteratively. Instead of just flagging outliers, it uses expert input to refine the decision boundaries of the model over time. This helps the system become more accurate at distinguishing true scientific discoveries from noise.
Broker-Level Anomaly Detection
This refers to applying anomaly detection directly within Fink's broker system that handles alerts from large surveys. By analyzing these alerts at this level, the pipeline can rapidly assess and prioritize potential scientific outliers before they are passed on for detailed study.

Terminology used across episodes

This episode discusses

The paper

Anomaly detection in Fink. I. Discovery, follow-up, and classification of unusual sources · Read on arXiv

M. V. Pruzhinskaya, M. V. Kornilov, A. V. Dodin, A. Baluta, T. A. Pshenichniy, A. M. Zubareva, E© E© O.– Ishida, J Peloton, I Beschastnov, I Ippolitov, A A Belinski, P Golysheva, N P Ikonnikova, V A Kiryukhina, V V Krushinsky O. A Tatarnikov S G Zheltoukhov D A Buckley, 12, 13, Kniazev 11, 14, S V Karpov A Möller Yusuke Tampo

Université Clermont Auvergne · Lomonosov Moscow State University · National Research University Higher School of Economics · Center for Astrophysics and Cosmology University of Nova Gorica · Institute of Astronomy Russian Academy of Sciences · Institute of Mechanics of Lomonosov Moscow State University · South African Astronomical Observatory South African Large Telescope Foundation Institute of Physics Czech Academy of Sciences Centre for Astrophysics and Supercomputing Swinburne University Technology

Modern wide-field time-domain surveys produce alert streams whose scientific potential is often concentrated in rare and unusual events. Efficient discovery therefore requires automated pipelines to be combined with rapid expert validation and follow-up. We present the first-year performance of the anomaly-detection (AD) pipeline operating within the Fink broker on the Zwicky Transient Facility alert stream, and assess its ability to identify scientifically valid outliers and enable discovery of rare phenomena. The pipeline transforms ZTF light curves into a compact set of features and ranks alerts using an Isolation Forest model trained on archival ZTF data. Each night, the 10 most anomalous candidates are distributed to experts via Slack/Telegram and exposed through an API. We also implement an expert-feedback loop using a public Telegram bot and retrain the model using the Active Anomaly Discovery algorithm. During the first year of operations (starting from 25 January 2023), the AD pipeline identified multiple high-interest sources and triggered dedicated photometric and spectroscopic follow-up. We report the discovery and multi-instrument (11-m SALT telescope, 2.5-m CMO telescope, 0.6-m ASA RC600, 0.25-m FRAM-ORM) follow-up of the rare AM CVn system Fink J062452.88+020818.3 of the WZ Sge type, UX Ori-type star Fink J222324.32+744222.0 and the unusual transient with precursor SN 2023mtp. In addition, the module triggered 33 supernovae, including 30 previously unreported ones, with candidates for superluminous and hostless events. Furthermore, nine new dwarf novae were discovered. These results show that broker-level anomaly detection, coupled with rapid dissemination, expert assessment, and follow-up observations, provide an effective bridge between large-scale survey streams and domain expertise, turning anomaly scores into astrophysical insights and concrete discoveries.

DOI: 10.1051/0004-6361/202660215

Transcript

Introduction to the show: ident: Astrophysics Radio. Generated commentary on the latest astrophysics papers.

Vera: We're looking at the paper: "Anomaly detection in Fink. I. Discovery, follow-up, and classification of unusual sources".

Jocelyn: Here's the gist of the paper. Anomaly detection in Fink presents the first-year performance of an anomaly-detection (AD) pipeline operating within the Fink broker on the Zwicky Transient Facility alert stream to identify scientifically valid outliers and enable discovery of rare phenomena The results show that broker-level anomaly detection, coupled with rapid dissemination, expert assessment, and follow-up observations, provide an effective bridge between large-scale survey streams and domain expertise Pipeline Architecture The pipeline transforms ZTF light curves into a compact set of features and ranks alerts using an Isolation Forest model trained on archival ZTF data The AD features module forms the first step of the Fink anomaly detection pipeline, whose purpose is to transform the photometric time series of alerts into a set of features suitable for machine-learning models This feature extraction uses 26 numerical features in total, which have demonstrated good performance in the analysis of transients and variable stars The model employs two separate IF models — one per passband — to handle cases where only a limited number of observations, or none at all, are available in a given passband Discovery of Rare Astrophysical Systems The AD pipeline identified multiple high-interest sources and triggered dedicated photometric and spectroscopic follow-up Notable individual anomalies include the rare AM CVn system Fink J062452.88+020818.3, which is a WZ Sge-type object The unusual transient SN 2023mtp was triggered, characterized by a luminous precursor and spectra that cannot be consistently matched by templates of a single supernova type across all epochs Furthermore, the module triggered 33 supernovae, including 30 previously unreported ones, with candidates for superluminous and hostless events Role of Expert Feedback The system implements an expert-feedback loop using a public Telegram bot and retrains the model using the Active Anomaly Discovery algorithm The incorporation of knowledge to the AD process can be done using the so-called active anomaly detection approach, which incorporates feedback from domain experts to iteratively refine the model Expert feedback is collected via a dedicated public Telegram bot where a “thumbs up” reaction indicates that an object was considered anomalous, while a “thumbs down” reaction classifies it as nominal The AAD algorithm extends the standard IF framework by incorporating expert feedback, allowing the model to adapt its decision boundaries iteratively Classification and Contaminants The results show that broker-level anomaly detection, coupled with rapid dissemination, expert assessment, and follow-up observations, provide an effective bridge between large-scale survey streams and domain expertise Misclassification among candidates in anomalies highlights that automated classification pipelines still benefit significantly from targeted expert validation Contaminants include instrumental issues such as CCD defects, optical system aberrations, poor weather conditions, or pipeline processing issues A significant fraction of detected anomalies were associated with moving objects, primarily known asteroids at early stages of the anomaly detection module The Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up This work demonstrates the value of incorporating expert feedback into the AD loop, shifting nominal classes toward lower anomaly ranks The next development step in the module involves the creation of personalized anomaly detection models tailored to the scientific interests of individual experts Finally, our results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The notion of an “anomaly” is expert-dependent, motivating the next development step toward personalized AD models tuned to the scientific interests of individual users The Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up The results show that the Fink AD pipeline provides an effective bridge between large-scale survey streams and domain experts by coupling machine-learning–based ranking with expert assessment and observational follow-up <ref:2603.

Paper summary: Vera: To recap this paper, "Anomaly detection in Fink. I. Discovery, follow-up, and classification of unusual sources," they describe a pipeline operating within the Fink broker on the Zwicky Transient Facility alert stream to find scientifically valid outliers and discover rare phenomena.

Jocelyn: The core claim is that by using machine learning tools to rank alerts based on features extracted from light curves, you can efficiently narrow down huge datasets to a manageable set of high-interest candidates for expert investigation.

Subrahmanyan: It matters because the authors show that this broker-level anomaly detection, when paired with rapid dissemination and expert assessment, provides an effective bridge between large-scale survey streams and domain expertise.

Vera: They detail the pipeline architecture where ZTF light curves are transformed into twenty-six numerical features using two Isolation Forest models, one for each passband to handle limited observation cases.

Jocelyn: The results show this approach successfully identified multiple high-interest sources, including specific examples like the AM CVn system Fink J062452 point 88+twenty thousand eight hundred eighteen point three and the unusual transient SN 2023mtp, which showed precursor characteristics that didn't match standard supernova templates.

Subrahmanyan: The study also highlights that this pipeline triggered thirty-three supernovae, including thirty previously unreported ones, suggesting its sensitivity to rare events like superluminous or hostless phenomena.

Vera: A key part they emphasize is the expert feedback loop, where domain experts use a public Telegram bot to provide reactions that retrain the model using an Active Anomaly Discovery algorithm.

Jocelyn: This feedback mechanism allows the model to adapt its decision boundaries iteratively by incorporating human knowledge about what constitutes an anomaly in that specific field of astronomy.

Subrahmanyan: They also discuss the limitations, pointing out that misclassification among candidates and contaminants like instrumental issues or moving objects, such as asteroids at early stages, still require targeted expert validation.

Vera: So the paper is demonstrating a working system that links data processing with human judgment to efficiently flag interesting astrophysical sources for follow-up observations.

Jocelyn: And they conclude by suggesting future work should focus on creating personalized anomaly detection models that are specifically tuned to the scientific interests of individual users.

Conclusion: Vera: So, looking at the title "Anomaly detection in Fink. I. Discovery, follow-up, and classification of unusual sources," it really highlights the three main actions this system performs: finding something new, deciding what to do with it through follow-up, and then figuring out what kind of source it actually is.

Jocelyn: The implication for the field is that you don't have to manually sift through everything; you can use automated ranking to find the outliers, and then leverage expert feedback to refine those rankings quickly.

Subrahmanyan: For cosmology and structure formation, this means we get a better way of prioritizing rare events in massive survey data before committing precious telescope time to detailed study.

Vera: It shows the value of coupling machine learning-based ranking with expert assessment and observational follow-up as a robust method for handling the sheer volume of data coming from surveys like ZTF.

Jocelyn: The real shift is in how we define what an "anomaly" means, since they suggest that this notion is dependent on the expert who is looking at it.

Subrahmanyan: That points toward personalized anomaly detection models tailored to specific scientific interests, meaning the definition of a rare event can become user-specific rather than universally defined.

More episodes

← Home