Operational Challenges in Efficient and Sustainable Operation of the Event Processing Nodes GPU Farm at the ALICE Experiment
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: High Energy Physics: Experiment. The day in high-energy experiment: detectors, searches and limits.
Kai: We're looking at the paper: "Operational Challenges in Efficient and Sustainable Operation of the Event Processing Nodes GPU Farm at the ALICE Experiment".
Mira: Here's the gist of the paper. ALICE’s Event Processing Nodes (EPN) farm is CERN’s largest GPU computing facility by capacity, comprising 350 GPU-based nodes and 2800 GPUs delivering approximately a total of 48 PFLOP/sIt has successfully enabled real-time processing of heavy-ion collisions at 50 kHz during LHC Run 3 System Architecture and Capacity The ALICE EPN farm is central to the Online–Offline (O2) framework, which unifies data acquisition and offline reconstruction through continuous readout The operational core of this infrastructure is the EPN farm, comprising approximately 350 nodes equipped with 2800 GPUs Nodes are interconnected through 100 Gb/s InfiniBand, complemented by 1 Gb/s Ethernet for management services Data Flow and Processing Pipeline The data flow consists of a continuous stream of raw TFs transferred to the EPN farm for online processing The EPN farm performs tracking, calibration, reconstruction, and compression synchronously with data taking The principal reconstruction workload is executed in the GPU barrel-processing chain The reconstructed data are finally passed to an entropy encoder, which produces CTFs Performance Metrics and Efficiency GPU acceleration substantially reduces the required server count and energy consumption The system comprises 350 nodes equipped with 2,800 AMD Instinct MI50 GPUs A single MI50 GPU replaces roughly 80 AMD Rome CPU cores in synchronous processing The ALICE EPN farm achieves a PUE of 1.05–1.15, with only 5–15% of the total power used for cooling and other infrastructure overheads Operational Sustainability The ALICE EPN farm is hosted in a modular, energy-efficient infrastructure using adiabatic air cooling and on-site reverse osmosis water treatment The modular container-based design enables incremental expansion and comprises IT and cooling containers, a dedicated power container with dual 2.5 MW transformers, a reverseosmosis plant, and spare-equipment facilities The EPN IT infrastructure life will be extended through Run 4 Run 3 Operational Success During Run 3, the ALICE Event Processing Node (EPN) farm demonstrated reliable synchronous processing under the demanding conditions of Pb–Pb data taking In 2024 and 2025, the instantaneous luminosity was successfully leveled at 50 kHz, confirming that the online computing infrastructure could sustain the expected interaction rates The successful operation of GPU-based synchronous reconstruction was essential to ALICE Run 3, enabling online calibration, reconstruction, and compression at the nominal 50 kHz Pb–Pb interaction rate and approximately 700 GB/s input to the reconstruction farm The use of an energy efficient air-to-air adiabatic cooling infrastructure and GPUs for online tracking was crucial to contain computing and environmental costs, reducing the number of servers by a factor of eight and halving the energy consumption compared to a CPU-only implementation The EPN farm processes the detector data stream during data taking and produces compressed data CTFs are archived in the Tier-0 and Tier-1 storage systems while AOD are made available on the GRID for physics analysis The EPN farm produces compressed Time Frames (CTFs), which are the only online data products retained for later processing CTFs and calibration data are transferred to the central 150 PB disk-storage system, which provides 360 GB/s throughput with approximately 25% redundancy CTFs are produced by an entropy encoder The raw TFs cannot be permanently stored Data Distribution System The EPN Data Distribution (DD) system is responsible for collecting detector sub-Time Frames (sTFs), transporting them to the EPN farm, and assembling complete Time Frames (TFs) At the First-Level Processing (FLP) servers, incoming detector raw data are aggregated into sub-Time Frames (sTFs) by the sTFBuilder which organizes the data blocks from the acquisition logic (CRU) over multiple LHC orbits (88.
Title and authors: Kai: So this paper is about the "Operational Challenges in Efficient and Sustainable Operation of the Event Processing Nodes GPU Farm at the ALICE Experiment". It sounds like they're focusing on making sure this massive computing setup actually runs reliably over time, not just if it works for one big run.
Mira: Right, and I think that’s a good starting point. The title suggests they aren't just talking about the science—they are talking about the engineering and how to keep this huge GPU farm going sustainably.
Lev: It sounds like the focus is on longevity and reducing overhead, which is always a big headache when you’re dealing with things that demanding.
Kai: Exactly. They talk about three hundred fifty nodes and two thousand eight hundred GPUs, which is a lot of hardware to manage continuously <ref:2611.06586#pg1>.
Mira: And it sets up the context for why they needed this new computing model in Run three especially given the high interaction rates they were hitting <ref:2611.06586#pg1>.
Lev: I wonder how much of that operational challenge comes from the hardware itself versus the software pipeline?
Kai: Well, it seems like a mix. They detail how GPU acceleration is central to everything for online tracking and calibration.
The paper's summary: Kai: So, basically, they summarize how they deployed this new computing model to handle the fifty kHz collision rates successfully during LHC Run three <ref:2611.06586#pg1>.
Mira: They emphasize that GPU acceleration is the key because it handles the parallel processing needed for online tracking while simultaneously reducing the number of physical nodes and cutting down on power usage.
Lev: I see why they would focus on that. If you're running synchronous reconstruction, a single MI50 GPU replaces roughly eighty AMD Rome CPU cores in that kind of workload.
Kai: That’s a big win for density and efficiency, right? They show how this approach allows them to do online calibration and reconstruction in real time at the nominal fifty kHz rate <ref:2611.06586#pg1>.
Mira: And they stress that this whole setup is designed for sustainability, not just speed. They talk about using modular, container-based infrastructure with adiabatic air cooling and on-site reverse osmosis water treatment.
Lev: That PUE of one point zero five to one point one five means only a small percentage of the total power goes to cooling and other overheads. That’s smart infrastructure planning.
The paper's improvements: Kai: The paper outlines several specific improvements they made, focusing on how they tackled those operational challenges we just touched on.
Mira: One major improvement is the deployment of a new computing model that performs online calibration, event reconstruction, and data compression in real time during data taking.
Lev: That’s where the AI comes in—the paper suggests an improved AI system can do this real-time work at rates like fifty kHz Pb–Pb collisions by using GPU acceleration for tracking <ref:2611.06586#pg1>.
Kai: They point out that this GPU acceleration is handling approximately ninety nine percent of the GPU workload, which means it's really taking on the heavy lifting.
Mira: Another big point they make is the physical scaling reduction; they show how this reduces the number of physical servers by a factor of eight compared to a CPU-only setup.
Lev: That reduction in server count directly impacts operational costs and maintenance complexity, which is crucial for long-term stability. They also mention extending the IT infrastructure life through Run four with this design <ref:2611.06586#pg2>.
Conclusion: Kai: So, to wrap up on "Operational Challenges in Efficient and Sustainable Operation of the Event Processing Nodes GPU Farm at the ALICE Experiment," they show how they managed high data rates using this highly parallel GPU approach.
Mira: It boils down to achieving real-time processing at fifty kHz while keeping the physical footprint and energy costs under control through smart hardware choices like GPUs and efficient cooling <ref:2611.06586#pg1>.
Lev: From a real hardware perspective, the overhead lands on how you manage that synchronous processing chain; it’s about making sure every component works in sync without introducing too much latency.
Kai: It also shows they kept the system modular so they could incrementally expand it, which is important for future needs beyond Run four <ref:2611.06586#pg2>.
Mira: The paper's main implication is demonstrating a viable path to sustain massive online processing for future runs by making the underlying infrastructure energy efficient and durable.
Lev: I think what matters most is that they laid out a clear roadmap for how these operational challenges will be tackled in the next generation of hardware, like supporting direct liquid cooling.
Federico Ronchetti, Giada Erba
CERN · Goethe University - Frankfurt am Main · Frankfurt Institute for Advanced Studies
hep-ex
Submitted: 2026-11-04
Updated: 2026-11-04
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
Importance score: 89/100
The gist: ALICE’s Event Processing Nodes (EPN) farm is CERN’s largest GPU computing facility by capacity, comprising 350 GPU-based nodes and 2800 GPUs delivering approximately a total of 48 PFLOP/sIt has…
Key concepts
- Event Processing Nodes (EPN) Farm
- This is the core GPU computing facility at CERN designed to handle the massive stream of raw data from heavy-ion collisions in real-time. It uses 350 GPU nodes to perform tracking, calibration, and reconstruction synchronously with data taking.
- GPU Acceleration
- The use of AMD Instinct MI50 GPUs is crucial for the system's performance. These GPUs handle approximately 99% of the GPU workload by performing essential tasks like tracking, allowing the farm to process high-rate collision data efficiently.
- Power Usage Effectiveness (PUE)
- PUE measures how efficiently a data center uses energy. The ALICE farm achieves a low PUE of 1.05–1.15 by employing adiabatic air cooling and modular design, meaning only 5–15% of total power is used for infrastructure overheads like cooling.
Terminology
Summary
ALICE’s Event Processing Nodes (EPN) farm is CERN’s largest GPU computing facility by capacity, comprising 350 GPU-based nodes and 2800 GPUs delivering approximately a total of 48 PFLOP/sIt has successfully enabled real-time processing of heavy-ion collisions at 50 kHz during LHC Run 3
System Architecture and Capacity
The ALICE EPN farm is central to the Online–Offline (O2) framework, which unifies data acquisition and offline reconstruction through continuous readout The operational core of this infrastructure is the EPN farm, comprising approximately 350 nodes equipped with 2800 GPUs Nodes are interconnected through 100 Gb/s InfiniBand, complemented by 1 Gb/s Ethernet for management services
Data Flow and Processing Pipeline
The data flow consists of a continuous stream of raw TFs transferred to the EPN farm for online processing The EPN farm performs tracking, calibration, reconstruction, and compression synchronously with data taking The principal reconstruction workload is executed in the GPU barrel-processing chain The reconstructed data are finally passed to an entropy encoder, which produces CTFs
Performance Metrics and Efficiency
GPU acceleration substantially reduces the required server count and energy consumption The system comprises 350 nodes equipped with 2,800 AMD Instinct MI50 GPUs A single MI50 GPU replaces roughly 80 AMD Rome CPU cores in synchronous processing The ALICE EPN farm achieves a PUE of 1.05–1.15, with only 5–15% of the total power used for cooling and other infrastructure overheads
Operational Sustainability
The ALICE EPN farm is hosted in a modular, energy-efficient infrastructure using adiabatic air cooling and on-site reverse osmosis water treatment The modular container-based design enables incremental expansion and comprises IT and cooling containers, a dedicated power container with dual 2.5 MW transformers, a reverseosmosis plant, and spare-equipment facilities The EPN IT infrastructure life will be extended through Run 4
Run 3 Operational Success
During Run 3, the ALICE Event Processing Node (EPN) farm demonstrated reliable synchronous processing under the demanding conditions of Pb–Pb data taking In 2024 and 2025, the instantaneous luminosity was successfully leveled at 50 kHz, confirming that the online computing infrastructure could sustain the expected interaction rates The successful operation of GPU-based synchronous reconstruction was essential to ALICE Run 3, enabling online calibration, reconstruction, and compression at the nominal 50 kHz Pb–Pb interaction rate and approximately 700 GB/s input to the reconstruction farm The use of an energy efficient air-to-air adiabatic cooling infrastructure and GPUs for online tracking was crucial to contain computing and environmental costs, reducing the number of servers by a factor of eight and halving the energy consumption compared to a CPU-only implementation The EPN farm processes the detector data stream during data taking and produces compressed data CTFs are archived in the Tier-0 and Tier-1 storage systems while AOD are made available on the GRID for physics analysis The EPN farm produces compressed Time Frames (CTFs), which are the only online data products retained for later processing CTFs and calibration data are transferred to the central 150 PB disk-storage system, which provides 360 GB/s throughput with approximately 25% redundancy CTFs are produced by an entropy encoder The raw TFs cannot be permanently stored
Data Distribution System
The EPN Data Distribution (DD) system is responsible for collecting detector sub-Time Frames (sTFs), transporting them to the EPN farm, and assembling complete Time Frames (TFs) At the First-Level Processing (FLP) servers, incoming detector raw data are aggregated into sub-Time Frames (sTFs) by the sTFBuilder which organizes the data blocks from the acquisition logic (CRU) over multiple LHC orbits (88.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improved AI system can perform real-time online calibration, event reconstruction, and data compression synchronously with data taking at high rates like 50 kHz Pb–Pb collisions. This is achieved by utilizing GPU acceleration for tracking, which accounts for
approximately 99% of the GPU workload.
-
The system can reduce the number of physical servers and associated power consumption significantly compared to CPU-only implementations. For synchronous processing,
a single MI50 GPU replaces roughly 80 AMD Rome CPU cores.
-
The AI system can operate on low-impact infrastructure by employing
adiabatic air cooling
and achieving a Power Usage Effectiveness (PUE) of 1.05–1.15, minimizing environmental impact while maintaining high availability. -
The data distribution system can efficiently transport detector raw data to the EPN farm using Remote Direct Memory Access (RDMA) over InfiniBand, aggregating detector sub-Time Frames into complete Time Frames (TFs).
-
The system can produce compressed Time Frames (CTFs) as only online data products retained for later processing, allowing the EPN farm to transform
the high-bandwidth detector stream into partially calibrated data.
-
Following synchronous processing, the AI system can perform asynchronous reprocessing to produce final reconstructed data in the form of Analysis Object Data (AOD), which is made available on a GRID for physics analysis.
Abstract
ALICE's Event Processing Nodes (EPN) farm is CERN's largest GPU computing facility by capacity, comprising 350 GPU-based nodes and 2800 GPUs delivering approximately a total of 48,PFLOP/s. It has successfully enabled real-time processing of heavy-ion collisions at 50,kHz during LHC Run 3. The facility was designed for sustainability, extended hardware lifetime, and reduced operational overhead. High availability is supported by a robust technical architecture and well-defined operational practices, ensuring reliable and continuous operation throughout data taking.
Sources
Related papers
- DrSai: an agentic AI system toward autonomous physics analysis workflows at BESIII
- CANTON- mu Proposal: A Next-Generation Muon g-2 Measurement at Sub-0.1 ppm Precision
- Search for dark photon dark matter near 1,meV using a superconducting mixer
- ANUBIS detector and its sensitivity to neutral long-lived particles
- How to scale your HEP ML models: A recipe for robust architecture comparisons at scale
- Search for the 16 O(ppp) to 13 C pi+ pi+ e+ Decay Mode in Super-Kamiokande Using Machine Learning Techniques