Euclid Data Release 1 (DR1). An overview of the Science Ground Segment architecture and operations
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Astrophysics. The day in astrophysics: observations, models and what the data forces.
Vera: I'm Vera.
Jocelyn: And I'm Jocelyn, here with Subrahmanyan.
Vera: We're looking at the paper: "Euclid Data Release 1 (DR1). An overview of the Science Ground Segment architecture and operations".
Jocelyn: Here's the gist of the paper. The Euclid Science Ground Segment (SGS) is a distributed system designed to transform raw telemetry into calibrated, validated, and scientifically exploitable data products for the Euclid mission, addressing fundamental questions on dark energy and dark matter.
Vera: The SGS comprises several components operating under a common architectural and procedural framework: the Euclid Science Operations Centre, Science Data Centres (SDCs), Instrument Operation Teams, and supporting Organisation Units (OUs).
Jocelyn: A central element is the Euclid Archive System (EAS), which functions as the metadata repository based on a common data model.
Vera: This system consists of a Distributed Storage System for scalable data file storage across SDCs, a Data Processing System used by analysis pipelines to register and access data files and their lineage, and a Science Archive that distributes validated data products.
Paper summary: Vera: So to wrap up what we just covered, this paper gives us an overview of the Science Ground Segment architecture and operations for Euclid Data Release one <ref:2611.16480#pg1,an overview of the Science Ground Segment architecture and operations>.
Jocelyn: The core idea they present is that this ground segment is a distributed system built on a common architectural and procedural framework designed to transform raw telemetry into calibrated, validated, and scientifically exploitable data products.
Subrahmanyan: They are laying out the structure comprising the Science Operations Centre, the Science Data Centres, Instrument Operation Teams, and supporting Organisation Units all working together within this shared setup.
Vera: And what makes it important is that they detail how this system was implemented specifically for DR1, describing its organisation and architecture in detail.
Jocelyn: The main focus is on the Euclid Archive System or EAS which serves as the central metadata repository built on a common data model.
Subrahmanyan: This EAS has three components: a Distributed Storage System for scalable storage, a Data Processing System for registering files and lineage, and a Science Archive that distributes those final validated products.
Vera: And they show how processing is orchestrated through the Infrastructure Abstraction Layer or IAL, which handles workflow management and resource abstraction across distributed compute clusters.
Jocelyn: Plus, they cover the development side with a Common Development Environment to support all the software integration and deployment needed for this complex setup.
Subrahmanyan: The paper shows that this infrastructure enabled the DR1 campaign, involving thirty-five million jobs across nine SDCs and six point seven million core-hours of computation time.
Vera: It demonstrates how robust and scalable this system is, showing they can handle that volume of work with only a very low operational error rate of just zero point seven percent.
Jocelyn: And they detail the specific processing chains, from LE1 PF through MER to PHZ and SHE, which shows exactly how the raw data gets turned into scientific results.
Subrahmanyan: It’s important that they outline these specific algorithms and units, like big O of n squared or n log n for scaling operations, so we know what kind of computational load we're dealing with.
Conclusion: Vera: So, looking at the authors, M. Ruffenach et al., they have really given us a complete picture of how they built this massive system for Euclid Data Release one <ref:2611.16480#pg1>.
Jocelyn: It’s about showing that handling the sheer volume of data from a wide-area survey while maintaining quality control is totally possible with this kind of distributed architecture.
Subrahmanyan: For those working on cosmology, it means we have a solid blueprint for how to manage the large-scale data processing pipelines needed for missions like Euclid.
Vera: The implication here is that if we can build this kind of robust ground segment, we can confidently expect the scientific output from the mission to be reliable and consistent.
Jocelyn: It means that when we look at those final results on dark energy and dark matter, we have a clear understanding of the data pipeline that produced them.
Subrahmanyan: And it also shows us exactly what kind of observational follow-up would be needed if our initial measurements don't align with the predictions from these pipelines.
Vera: It’s a blueprint for how to handle the complexity, showing that the engineering can keep up with the scientific ambition.
M. Ruffenach, M. Frailis, A. Zacchei, P. Casenove, M. S Holliman, H Aussel, B Altieri, G Angora
European Space Agency (ESA) · Euclid Consortium
astro-ph.IM, astro-ph.CO
Submitted: 2026-11-12
Updated: 2026-11-12
Comments: Paper intended for the A\&A Special Issue "Euclid First Data Release'', 25 pages, 12 figures
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 89/100
The gist: The Euclid Science Ground Segment (SGS) is a distributed system designed to transform raw telemetry into calibrated, validated…
Key concepts
- Euclid Archive System (EAS)
- This is the central metadata repository for the mission. It uses a common data model to store information about all scientific data. It includes storage for files across multiple centers, a system to track how files are processed (lineage), and an archive that distributes the final validated products.
- Infrastructure Abstraction Layer (IAL)
- The IAL is the framework used to coordinate processing workflows. It abstracts the underlying distributed compute clusters, allowing users to manage complex tasks easily. It handles resource allocation and workflow management, ensuring that data processing steps run correctly across different computing resources.
- Data Processing Pipelines
- These are the specific algorithms used to transform raw telemetry into scientific results. The DR1 campaign utilized several pipelines (like VIS, NIR, MER) sequentially to create stacked images and photometric catalogs. These pipelines are developed by specialized teams (OUs) to perform instrument-specific calibrations and analysis.
- Science Data Centres (SDCs)
- These are the nine distributed locations where data processing occurs. They run the data processing pipelines locally and store the generated products. This distribution is key to handling the massive scale of DR1, allowing for parallel execution and local storage of intermediate results.
Summary
The Euclid Science Ground Segment (SGS) is a distributed system designed to transform raw telemetry into calibrated, validated, and scientifically exploitable data products for the Euclid mission, addressing fundamental questions on dark energy and dark matter. p. 0 (Full author list and affiliations details can be found after the references)
November 13, 2026
ABSTRACT
The Euclid mission is designed to address fundamental questions on the nature of dark energy and dark matter by mapping the extragalactic sky
with a wide-area survey complemented by targeted deep observations. Achieving the required scientific performance places stringent demands on
the Euclid Science Ground Segment (SGS), responsible for transforming raw telemetry into calibrated, validated, and scientifically exploitable
data products. Achieving the required scientific performance places stringent demands on
the Euclid Science Ground Segment (SGS), responsible for transforming raw telemetry into calibrated, validated, and scientifically exploitable
data products. The SGS is a distributed system comprising the Euclid Science Operations Centre, the Science Data Centres, Instrument Operation
Teams, and supporting Organisation Units, operating within a common architectural and procedural framework.Where the paper says it
Where the paper says it
A central element is the Euclid Archive System (EAS), which functions as the metadata repository based on a common data model. p. 0 We outline the
main components of the SGS common infrastructure. A key element is the Euclid Archive System (EAS), which provides the central metadata
repository based on a common data model. A key element is the Euclid Archive System (EAS), which provides the central metadata
repository based on a common data model. It is composed of: a Distributed Storage System, providing scalable storage of data files across the
Science Data Centres; a Data Processing System used by the analysis pipelines to register and access data files and their lineage; and a Science
Archive, that distributes validated data products. It is composed of: a Distributed Storage System, providing scalable storage of data files across the
Science Data Centres; a Data Processing System used by the analysis pipelines to register and access data files and their lineage; and a Science
Archive, that distributes validated data products. Processing is coordinated through a common orchestration framework based on the Infrastructure
Abstraction Layer, which provides workflow management and resource abstraction of the distributed compute clusters. Processing is coordinated through a common orchestration framework based on the Infrastructure
Abstraction Layer, which provides workflow management and resource abstraction of the distributed compute clusters. Software development,
integration, and deployment are supported by the Common Development Environment.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The DR1 production campaign involved executing 35 million processing jobs across nine SDCs, consuming 6. p. 0 We describe daily processing operations, calibration
activities, and quality control, followed by the DR1 production campaign, including processing flows, validation strategy, and computational and
data volumes. This infrastructure enabled the production of Euclid DR1, involving the execution of 35 million processing jobs across nine Science
Data Centres and a total of 6. This infrastructure enabled the production of Euclid DR1, involving the execution of 35 million processing jobs across nine Science
Data Centres and a total of 6. 7 million core-hours, demonstrating the robustness and scalability of the Euclid SGS. The EC SGS has the important role of producing the official Euclid data releases and ingesting the data products into the
archive. The processing activity is spread across nine national
SDCs, which are computing centres responsible for implementing and running the data processing pipelines, as well as storing the locally generated data products. The processing activity is spread across nine national
SDCs, which are computing centres responsible for implementing and running the data processing pipelines, as well as storing the locally generated data products. The pipeline algorithms
are defined and developed by the EC Organisation Units (OUs):
VIS, Near-infrared (NIR), Spectroscopic reduction (SIR), External (EXT), Merge (MER), Spectroscopic redshift (SPE), Shear
(SHE), Photometric redshift (PHZ), Level 3 (LE3), and Simulations (SIM), described in Sect.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The processing chain commences with the LE1 PF, which receives Euclid satellite telemetry from the MOC and SOC, including VIS and NISP detector data, housekeeping telemetry (HKTM), and MOC auxiliary products. p. 3 Processing Functions overview
In the Euclid SGS, data processing is organised into a set of
self-contained PFs that interact through the central archive, as
illustrated in Fig. The processing chain begins with the LE1
PF, which receives Euclid satellite telemetry, acquired by the
MOC and transferred to the SOC, including VIS and NISP detector data, housekeeping telemetry (HKTM), and MOC auxiliary
products. The processing chain begins with the LE1
PF, which receives Euclid satellite telemetry, acquired by the
MOC and transferred to the SOC, including VIS and NISP detector data, housekeeping telemetry (HKTM), and MOC auxiliary
products. LE1 transforms these inputs into VIS and NISP rawframe products, along with HKTM products. These products are
aggregated Flexible Image Transport System (FITS) files containing decompressed detector data, on-board timestamps converted to estimated Coordinated Universal Time, and associated
pointing information, all of which serve the basis for the subsequent processing. From these raw data, the VIS and NIR PFs
independently perform instrument-specific calibrations. The stacking of PSF models from individual
frames is performed at source positions detected on each stacked
image, using either a source-by-source or an ensemble approach
depending on field density. The outputs of VIS, NIR, and EXT are subsequently combined by MER (Euclid Collaboration: Romelli et al. 2026;
Euclid Collaboration: Kümmel et al. 2026), which generates
stacked images (mosaics) and consolidated photometric source
catalogues. Source detection is performed with SExtractor++
independently on the VIS and NIR (three-band stacked) coadded images. Photometry is measured using complementary
methods: Kron aperture, PSF-matched aperture, template-fitting,
and 2D Sérsic profile fitting. Morphological measurements include the CAS criteria (concentration, asymmetry, smoothness)
and FLUX_RADIUS. These merged data products are then used by PHZ (Euclid
Collaboration: Tucci et al. 2026; Euclid Collaboration: Hartley
et al. 2026) to estimate photometric redshifts and derive source
properties, and by SHE (Euclid Collaboration: Congedo et al. 2026) to estimate photometric redshifts and derive source
properties, and by SHE (Euclid Collaboration: Congedo et al. 2024) to measure galaxy shapes from the photometric images
and quantify the distortions induced by the intervening mass
distribution, dominated by dark matter, through weak gravitational lensing. ,
noise maps, PSF characterisation, Galactic extinction, and zodiacal light) needed to compute limiting magnitudes and visibility
masks; (iv) depth maps estimated from rms noise; and (v) photometric visibility masks and random catalogues tailored to a given
source selection. Finally, the SGS includes a dedicated simulation pipeline,
SIM (Euclid Collaboration: Serrano et al. Finally, the SGS includes a dedicated simulation pipeline,
SIM (Euclid Collaboration: Serrano et al. 2024), designed to
reproduce realistic observations for validation and qualification
purposes.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The scale of the DR1 production campaign was substantial: 34. p. 0 Software development,
integration, and deployment are supported by the Common Development Environment. We describe daily processing operations, calibration
activities, and quality control, followed by the DR1 production campaign, including processing flows, validation strategy, and computational and
data volumes. This infrastructure enabled the production of Euclid DR1, involving the execution of 35 million processing jobs across nine Science
Data Centres and a total of 6. 7 million core-hours, demonstrating the robustness and scalability of the Euclid SGS. The LE1 production activity includes telemetry checking and assessment of both
the scientific and housekeeping telemetry and the production of
daily reports. The SOC is also responsible for designing, implementing
and operating the Euclid Science Archive, which contains the final validated Euclid data releases, and the Euclid Data Space,
which integrates the Science Archive with visualisation and
computational services to provide seamless scientific access to
the validated data products (Sect. The SGS is a distributed system comprising the Euclid Science Operations Centre, the Science Data Centres, Instrument Operation
Teams, and supporting Organisation Units, operating within a common architectural and procedural framework. This paper presents the SGS as
implemented for the Data Release 1 (DR1), describing its organisation, architecture, processing functions, and operational model. A Processing Plan specifies the
processing to be performed by defining a set of parameters, including the PF to be executed, the required calibration and input
data, the observation dates and sky area, and other relevant processing metadata. Based on these parameters, COORS invokes
a service provided by the DPS to expand the Processing Plan
into a set of PPOs, each defining an individual processing task. We describe daily processing operations, calibration
activities, and quality control, followed by the DR1 production campaign, including processing flows, validation strategy, and computational and
data volumes. This infrastructure enabled the production of Euclid DR1, involving the execution of 35 million processing jobs across nine Science
Data Centres and a total of 6. The EAS requires a distributed and scalable architecture,
with storage resources deployed on heterogeneous hardware and
file servers across nine data centres, together with distributed
computing capabilities spanning the same data centres and diverse hardware and software environments. It must also record
the inputs and outputs of the processing pipelines, including file
locations and copies through the DSS, as well as the parameter values used through the DPS, and keep track of all processing and calibration steps together with the corresponding software versions, thereby ensuring the reproducibility of individual data products. Pilots are long-running jobs submitted to
the Distributed Resource Manager that can execute multiple PF
tasks, thereby reducing queue waiting times. Both pilot generation and task scheduling rely on a scoring algorithm that considers compatibility between pilot specifications (defined at the
SDC level), task requirements, task priorities, fairness between
PFs, waiting times, and I/O intensity.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Automated data processing workflows using Argo Workflows can orchestrate containerised tasks as directed acyclic graphs to define end-to-end processing workflows for PFs, which
submit PPOs to the DPS
and monitor their execution status. -
The Infrastructure Abstraction Layer (IAL) provides an interface between the PFs and the EC SGS, enabling PF developers to define processing logic as a data-flow graph and supporting
data-parallel processing approaches on generated data.
-
The Meta-Scheduler manages bottlenecks by using multiple state machines, including one for DSS-Transfer and DSS-Retrieve, to manage file chunks locally on the SDC's local DSS instance before triggering the Pipeline Runner.
-
The Pipeline Runner employs a distributed microservice architecture with ZeroMQ communication, using publish–subscribe and request–reply patterns to execute PFs and perform continuous profiling of each task.
-
The system can provide near real-time monitoring by transmitting
telemetry and profiling data (for each PPO, pilot, node, and the IAL itself) to the Monitoring and Control subsystem,
which is visualized through Grafana dashboards. -
The Euclid Data Space integrates these services to support workflows spanning
data discovery, visualisation, and programmatic analysis,
allowing users to execute software and Jupyter notebooks close to the data. -
The system enables automated execution of PFs immediately after raw observations are ingested through OTF processing, which can be managed by COORS, thereby reducing operator workload and accelerating processing by
triggering re-execution on newly ingested sources.
Abstract
The Euclid mission is designed to address fundamental questions on the nature of dark energy and dark matter by mapping the extragalactic sky with a wide-area survey complemented by targeted deep observations. The required scientific performance places stringent demands on the Euclid Science Ground Segment (SGS), which transforms raw telemetry into calibrated, validated, and scientifically exploitable products. The SGS is a distributed system comprising the Euclid Science Operations Centre, the Science Data Centres, Instrument Operation Teams, and supporting Organisation Units, within a common architectural and procedural framework. This paper presents the SGS as implemented for Data Release 1 (DR1), describing its organisation, architecture, processing functions, and operational model. We outline the main components of the SGS common infrastructure. A key element is the Euclid Archive System (EAS), the central metadata repository based on a common data model. It comprises a Distributed Storage System, providing scalable storage of data files across the Science Data Centres; a Data Processing System used by the pipelines to register and access files and their lineage; and a Science Archive, distributing validated data products. Processing is coordinated by a common orchestration framework based on the Infrastructure Abstraction Layer, providing workflow management and resource abstraction of distributed compute clusters. Software development, integration, and deployment are supported by the Common Development Environment. We describe daily operations, calibration, and quality control, followed by the DR1 production campaign, including processing flows, validation strategy, and computational and data volumes. This infrastructure enabled Euclid DR1, involving 35 million jobs across nine Science Data Centres and 6.7 million core-hours, demonstrating robustness and scalability of the SGS.
Sources
- The Pan-STARRS1 Surveys
- Euclid: Quick Data Release (Q1) -- LensMC shear measurement catalogue for cluster lensing science
- Euclid Definition Study Report
Related papers
- On-sky demonstration of a vector Zernike wavefront sensor in a cascaded adaptive optics system
- Euclid Data Release 1 (DR1): Calibration strategy and observations
- The General Antiparticle Spectrometer (GAPS) Antarctic Balloon Payload
- Detection and excision of radio-frequency interference in radio arrays
- Unified matched-filter and excess-power pipeline for gravitational-wave detector characterization
- Ushering in sub-arcsecond angular resolution observatories in the far-infrared: The case for a double-Fourier interferometer