Euclid Data Release 1 (DR1). Near-infrared processing and data products
summary
The gist
The Euclid near-infrared processing function (NIR PF) converts raw observations into calibrated images and source-level products…
In short
The Euclid Near-Infrared Processing Function (NIR PF) converts raw near-infrared telescope data into calibrated images and source catalogs for DR1. Validation confirms the processing chain meets required astrometric accuracy of 10 mas RMS and photometric accuracy of 0.9% for bright sources, providing a robust foundation for scientific analysis.
Key concepts
- NIR PF
- This is the software pipeline that takes raw near-infrared spectrometer and photometer observations and transforms them into usable, calibrated images and source products. It involves detector corrections, calibration steps, and image processing to ensure the data is accurate for science.
- Astrometric Accuracy (RMS)
- This measures how precisely the positions of celestial objects are determined in the final images. The paper reports an average root-mean-square (RMS) of 10 milliarcseconds (mas), which is below 20 mas for 99% of observations across all three near-infrared bands, indicating good positional accuracy.
- Photometric Accuracy
- This refers to the precision with which the brightness or flux of astronomical sources is measured. For sources brighter than magnitude 19 in specific bands (YE, JE, HE), the relative flux calibration uncertainty achieved is 0.9%, meaning measurements are very reliable for these bright objects.
- Residual Effects
- These are small patterns or artifacts left in the final images after processing that cannot be fully explained by known instrumental issues. The analysis found faint striping and residual horizontal stripes due to detector readout noise, which need to be addressed in future data releases.
Terminology used across episodes
This episode discusses
The paper
Euclid Data Release 1 (DR1). Near-infrared processing and data products · Read on arXiv
G. Polenta, M. Frailis, A. Alavi, P. N. Appleton, P. Awad, A. Bonchi, L Bramante, G Calderone
European Space Agency (ESA) · INAF-Osservatorio Astronomico di Trieste · Caltech/IPAC · Leiden Observatory, Leiden University · Aerospace Logistics Technology Engineering Company · Centre de Recherche Astrophysique de Lyon, Université Claude Bernard Lyon 1 · University of Lyon 1, CNRS/IN2P3 Institut d’Astrophysique de Paris, Université Paris Cité · INAF-Osservatorio Astronomico di Roma · Kapteyn Astronomical Institute, University of Groningen · Aix-Marseille Université, CNRS/IN2P3 · Max Planck Institute für Astronomie, University of Oxford · California Institute of Technology · INAF-Osservatorio Astronomico di Padova · Department of Physics and Astronomy 'Augusto Righi' - Alma Mater Studiorum Università di Bologna · Institut de Astrofísica de Canarias, University of Tenerife · Institute for Astronomy, University of Edinburgh · Jet Propulsion Laboratory, California Institute of Technology · University of Helsinki Institute of Physics · Centre for Astrophysics and Cosmology, University of Nova Gorica · Department of Physics and Astronomy 'G. Galilei', Università di Padova · INAF-Sezione di Napoli, Università degli Studi di Napoli Federico II · INAF-Osservatorio Astrofisico di Torino · INAF-IASF Milano, INFN Sezione di Roma 2 · Department of Physics and Astronomy 'G. Galilei', Università di Padova · Institut de Recherche en Astrophysique et Planétologie, Université Toulouse, CNRS · University of Geneva · Mullard Space Science Laboratory, University College London · Department of Physics and Astronomy 'Augusto Righi' - Alma Mater Studiorum Università di Bologna · INAF-Osservatorio Astrofisico di Arcetri · Department of Physics and Astronomy 'G. Galilei', Università di Padova · INAF-Sezione di Roma 2 · Dipartimento di Fisica e Astronomia 'Augusto Righi' - Alma Mater Studiorum Università di Bologna
The Euclid near-infrared processing function (NIR PF) converts the raw observations acquired by the Near-Infrared Spectrometer and Photometer (NISP) into calibrated images and source-level products. Following the first implementation delivered with the Euclid Quick Data Release 1 (Q1), the NIR PF has been revised for the first major Euclid Data Release (DR1), with improvements to detector-level processing, calibration, image reconstruction, and quality control. The DR1 processing includes observations covering more than 1900, squared and has produced 71,659 calibrated near-infrared (NIR) images. We assess the performance of the processing chain through extensive astrometric and photometric validation. The astrometric solution reaches an average root-mean-square (RMS) of 10 mas, remaining below 20 mas for 99% of the dithers in all three NISP bands. Temporal variations in the instrumental throughput are typically corrected to a level of 2 mmag RMS, while the relative photometric accuracy reaches 0.9% for sources brighter than 19 mag in Y E, J E, and H E, and the residual spatial variations of the photometric calibration across the focal plane are at the level of approximately 10 mmag for the entire sample. Despite some remaining limitations, the DR1 NIR imaging products are consistent with the required astrometric and photometric performance over the large area covered by this release.
Transcript
Introduction to the show: ident: Astrophysics. The day in astrophysics: observations, models and what the data forces.
Vera: We're looking at the paper: "Euclid Data Release 1 (DR1). Near-infrared processing and data products".
Jocelyn: Here's the gist of the paper. The Euclid near-infrared processing function (NIR PF) converts raw observations into calibrated images and source-level products, and its performance validation on the first major Euclid Data Release (DR1) confirms that the processing chain meets required astrometric and photometric accuracy over a large area Processing Overview The NIR PF transforms raw Near-Infrared Spectrometer and Photometer (NISP) observations into calibrated images through detector-level corrections, calibration, and image-processing steps to produce calibrated images and associated source-level products The DR1 processing includes observations covering more than 1900 deg2 and has produced 71 659 calibrated near-infrared (NIR) images Instrument Stability The Near-Infrared Spectrometer and Photometer (NISP) has operated very consistently since launch, with no major evolution of instrumental properties except the slow buildup of water ice on NISP’s detector surfaces The physical number of unusable ‘bad’ pixels has not changed significantly since launch, although their characterisation and flagging has been updated Data Quality Control The master bad-pixel mask (BPM) is updated using performance verification (PV) flat-fields for ‘disconnected’ pixels and 500 PV photo darks for ‘hot’ pixels, while permanently ‘saturated’ pixels are flagged as INVALID New categories in the BPM include crossshaped ‘flower’ patterns identified using a Laplace filter on LED flat-field exposures A total of 239 878 INVALID bad pixels were identified, corresponding to approximately 0.4% of the focal plane Calibration and Modeling Non-linearity correction models the per-pixel response as a 4th-degree polynomial calibrated against LED flat-field exposures using in-flight data from PV-018 and F-008 Persistence masking was refactored for DR1, updating the model based on in-flight calibration data parameterized as IP(S, t) = α˜S (t − t˜0) −β˜ (1) Performance Metrics The astrometric solution reaches an average root-mean-square (RMS) of 10 mas, remaining below 20 mas for 99% of the dithers in all three NISP bands The relative photometric accuracy reaches 0.9% for sources brighter than 19 mag in YE, JE, and HE Residual Effects The analysis identified residual effects such as faint striping patterns within images attributed to 1/ f noise in the detector readout Residual spatial features are observed when restricting the analysis to bright sources only, which closely resemble persistence charge patterns The background estimation reveals residual horizontal stripes due to 1/ f noise in the detector readout, which will be addressed for DR2 Future Work Further increase in the number of usable pixels may be achieved by applying persistence corrections rather than relying on masking alone, and a smooth interpolation model will be used to describe throughput evolution in DR2 The exact optical surfaces causing arcs from bright stars and a model to describe them are under investigation for future releases Data Products The NIR PF produces calibrated single-epoch images, the stacked image, DpdNirStackedFrame, and its corresponding source catalogue The stacked image is the result of resampling and stacking 4 (76) single-epoch images for EWS observations The catalogue extracted from the stacked image is stored in the DpdNirStackedFrameCatalog DP Photometric Calibration The relative flux calibration uncertainty for sources brighter than 19 mag is at the level of 0.9% for JE, YE, and HE bands The absolute calibration is tied to a single-epoch observation of WDJ175318.65+644502.15 The median IS values of the other three WDs lie well within the 2σ scatter of the primary calibrator The DR1 NIR data products meet the required levels of astrometric and photometric accuracy and provide a robust basis for scientific exploitation, while residual effects may be relevant for other scientific applications The analysis has identified a number of residual effects and limitations in the DR1 processing Residual effects are not assigned a physical origin The results confirm that the DR1 NIR data products meet the required levels of astrometric and photometric accuracy The improvements being developed for DR2 are expected to further mitigate these effects and improve the quality and consistency of the NIR data products Acknowledgements The Euclid Consortium acknowledges the European Space Agency and a number of agencies and institutes that have supported the development of Euclid This work has made use of the Euclid Data Release 1 (DR1) from the European Space Agency (ESA) mission Euclid This work has made use of data from the European Space Agency (ESA) mission Gaia Funding for the DPAC has been provided by national institutions, in particular the institutions participating in the Gaia Multilateral Agreement The Euclid Collaboration acknowledges G.
Vera: Polenta⋆1, M.
Jocelyn: Frailis2, A.
Vera: Alavi3, P.
Title and authors: Vera: Now let’s talk about the title of this work, "Euclid Data Release one (DR1) <ref:2611.16119#pg1,Euclid Data Release 1 (DR1>. Near-infrared processing and data products," and who is behind it. It tells you exactly what we are looking at: the processing pipeline for the first major Euclid release.
Jocelyn: The authors include G. Polenta, M. Frailis, A. Alavi, P. N. Appleton, P Awad, A Bonchi, L Bramante and G Calderone among others >
Vera: It tells you this is a big collaborative effort from the Euclid team that’s been working hard on this data for years >
Jocelyn: And the implication is that they are presenting a fully revised version of their near-infrared processing function, moving beyond what they had in the first quick release >
Subrahmanyan: For theoretical work, it means we need to trust these final calibrated images because this paper validates the entire methodology used for deriving cosmological constraints from Euclid’s data >
Vera: It’s about showing that the instrument itself is stable enough to handle these complex corrections, even though there are still things they found they need to work on >
Jocelyn: So, in simple terms, this paper is the official quality check proving that the way Euclid handles its near-infrared data—from raw signal to final picture—is accurate enough for serious science >
Vera: Right. It’s about validating that the processing chain meets all the required accuracy levels across a massive dataset >
The paper's summary: Jocelyn: So, let’s look at what the paper actually summarizes in terms of its work. They detail how they transformed raw observations from the Near-Infrared Spectrometer and Photometer into usable images and source products >
Vera: Specifically, they cover detector-level corrections, calibration steps, image reconstruction methods, and quality control checks that went into this DR1 processing >
Jocelyn: And they show that the DR1 processing includes observations covering more than one thousand nine hundred square degrees and produced seventy-one thousand six hundred fifty-nine calibrated near-infrared images > <ref:2611.16119#pg1,the DR1 processing includes observations covering more than 1900>
Subrahmanyan: That large area covered is what makes it useful; we can’t just look at a small patch of sky, we need that wide coverage to build meaningful cosmological models >
Vera: And they detail the instrument stability too. They note that the Near-Infrared Spectrometer and Photometer has operated very consistently since launch, with no major evolution in instrumental properties except for the slow buildup of water ice on the detector surfaces >
Jocelyn: That’s a key point for us because it means we don't have to constantly worry about sudden changes in the detector itself, which simplifies things >
Subrahmanyan: But they also flag some issues. They found residual effects, like faint striping patterns within images that they attribute to one over f noise in the detector readout >
Vera: So the summary is this: they’ve produced calibrated data over a huge area, confirmed instrumental stability is good but not perfect, and identified specific noise artifacts that we need to keep an eye on >
The paper's improvements: Subrahmanyan: Moving onto the improvements section of the paper. The authors suggest ways to make the whole process better. They focus on refining how they handle non-linearity correction and persistence modeling >
Vera: They mention that they corrected the per-pixel response using a fourth-degree polynomial model, calibrated against LED flat-field exposures from in-flight data like PV018 and F008 >
Jocelyn: And they refactored the persistence masking for DR1, updating their model based on in-flight calibration data parameterized as IP of S equals alpha sub S times t minus tau naught minus beta one >
Subrahmanyan: That persistence correction method is crucial because it helps account for the signal from preceding exposures, which is necessary when we are looking at time-domain objects like variable stars >
Vera: They also updated the master bad-pixel mask by using performance verification flat fields for disconnected pixels and five hundred performance verification photo darks for hot pixels >
Jocelyn: That’s a concrete improvement in how they handle bad data; they're actively cleaning up the contamination based on specific diagnostic tools >
Subrahmanyan: The paper also discusses the residual effects, where they found that restricting the analysis to just bright sources shows residual spatial features that look like persistence charge patterns >
Vera: And finally, for future work, they suggest using a smooth interpolation model to describe how throughput evolves in DR2 instead of relying only on masking alone >
Conclusion: Jocelyn: So let’s wrap up with the conclusion. Basically, the authors state that the DR1 NIR data products meet the required levels of astrometric and photometric accuracy and they provide a robust basis for scientific exploitation >
Vera: But they also mention those residual effects we talked about—like faint striping patterns and horizontal stripes due to one over f noise—which are not assigned a physical origin yet >
Subrahmanyan: So, while the results confirm the DR1 NIR data products meet the required levels of astrometric and photometric accuracy, we have to remember those residual effects might be relevant for other scientific applications >
Vera: That’s right. The improvements they are developing for DR2 are expected to further mitigate these effects and improve the quality of these near-infrared data products >
Jocelyn: It’s a solid foundation, but it's clear there’s still work to do to make those residual noise patterns disappear before we rely on this data for the next big things >
More episodes
- 2607.17929-gevolution 2.0: GPU-accelerated relativistic N-body simulations for cosmology
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2509.02657-On the synergetic use of Ariel and JWST for exoplanet atmospheric science
- 2609.10042-Modelling Fourier filtering wavefront sensors for PSD-based methods: the AOPERA tool
- 2511.05639-Performance of the AstroPix Prototype Module for the Barrel Imaging Calorimeter at the ePIC Detector and in Space-Based Payloads
- 2610.19909-RPCA: a Resolved PSF reconstruction method for undersampled images
- 2512.03440-An Analysis of LIGO Glitches Using t-SNE During the First Part of the Fourth LIGO-Virgo-KAGRA Observing Run
- 2608.10284-Lunar Reflective Interferometry
- 2610.16633-Institutional broadening of the Japanese astronomy and astrophysics research community during 1996-2025