Open Science with the LHCb Ntupling Service

summary

The gist

The LHCb Ntupling Service presents an innovative methodology for providing open access to data collected in HEP experiments by allowing on-demand production of custom LHCb open data.

In short

The LHCb Ntupling Service is a web service enabling users to create and produce custom LHCb open data on demand. It allows for flexible exploration of physics selections without requiring experiment-specific software knowledge. The system streamlines data request tracking and provides a direct communication channel with the LHCb Open Data team, significantly lowering the barrier for new researchers to access vast datasets from Run 1 and Run 2 collisions.

Key concepts

LHCb Ntupling Service
This is a web service designed to let users create custom LHCb data on demand. Instead of needing complex software, users use a wizard to specify what kind of data they want, which the service then produces for them. It makes accessing experimental results much easier.
LHCb Ntuple Wizard
This is the user interface that guides researchers in selecting specific datasets and physics objects they are interested in. It helps users create configuration files (YAML) needed to generate their desired data, removing the need for users to learn complex production software like DaVinci.
GitLab Integration
The service uses GitLab to manage all requests. Each user request becomes an issue in a private repository, and its progress is tracked using labels like 'Awaiting LHCb review'. This provides a transparent workflow for managing data requests and audits.
FAIR Principles Promotion
This process allows users to formally promote their generated ntuples to permanent records on the CERN Open Data Portal. Once approved, these records receive a DOI, ensuring the data is discoverable and reusable according to FAIR principles.

Terminology used across episodes

This episode discusses

The paper

Open Science with the LHCb Ntupling Service · Read on arXiv

Christine A. Aidala, Michael Buchar, Chris Burr, Dillon S. Fitzgerald, Cameron Duncan McClymont, Adam Morris, Sebastian Neubert, Piet Nogga, Ryunosuke O’Neil, Eduardo Rodrigues, Daan Eduard Rosendal, Tibor Simko

University of Michigan, United States · CERN, Switzerland · Vilnius University, Lithuania · University of Bonn, Germany · Oliver Lodge Laboratory, University of Liverpool, United Kingdom · Lakera company

The data samples collected at the Large Hadron Collider at CERN are highly valuable, and the analysis of these data has led to numerous discoveries and the 2025 Breakthrough Prize in Fundamental Physics. Preserving and providing open access to these data for future analysts is imperative to maximize the scientific output of the collected data. The LHCb collaboration has developed a novel approach to this pursuit of open science, the LHCb Ntupling Service: a web service for on-demand production and publishing of custom LHCb open data, providing open access to both Run 1, and for the first time, Run 2 proton-proton collision data collected by the LHCb experiment, amounting to over 4 PB of data to explore.

Transcript

Introduction to the show: ident: High Energy Physics: Experiment. The day in high-energy experiment: detectors, searches and limits.

Kai: We're looking at the paper: "Open Science with the LHCb Ntupling Service".

Mira: Here's the gist of the paper. The LHCb Ntupling Service presents an innovative methodology for providing open access to data collected in HEP experiments by allowing on-demand production of custom LHCb open data.

Kai: This service is significant because it offers flexibility in exploring physics selections while reducing the barrier to entry for new users without the need for experiment-specific software.

Mira: The Problem of Data Access Contemporary research in high energy physics (HEP) necessitates vast data volumes stored in custom file formats to meet the needs of experimental programs.

Kai: Both the complexity and sheer volumes of these datasets make it challenging to preserve and provide open access to these data.

Title and authors: Kai: The paper title is "Open Science with the LHCb Ntupling Service", and it’s authored by Christine Aidala, Michael Buchar, Chris Burr, Dillon S. Fitzgerald, Cameron Duncan McClymont, Adam Morris, Sebastian Neubert, Piet Nogga, Ryunosuke O’Neil Eduardo Rodrigues Daan Eduard Rosendal Tibor Simko.

Mira: It really just frames the whole thing as a way to make the data more open and accessible for everyone interested in high energy physics. They are pushing for FAIR principles for this data output >

Lev: What I find interesting about the authors is that they’re coming from different areas—like experimentalists, theorists, and error correction—which suggests a broad view of what’s needed to make this work.

Kai: Right, it shows how different parts of the physics community need to collaborate on these kinds of infrastructure projects.

Mira: It sets up the idea that we should be aiming for consistent standards across all the LHC experiments, not just one silo >

The paper's summary: Kai: They summarize this as a novel approach: a web service for on-demand production and publishing of custom LHCb open data. It addresses the challenge of preserving and providing open access to vast data volumes in custom file formats.

Mira: So, the core idea is that instead of users having to deal with all those complicated parsing tools, they can use this service to generate exactly what they need >

Lev: The paper mentions that it provides a single entry point for users to submit ntuple production jobs, track their requests, communicate with the LHCb Open Data team, and download their resulting output files.

Kai: That tracking part is important; they model the lifecycle using GitLab issue labels like "Awaiting LHCb review" and "Running test production".

Mira: And they talk about the data storage: they use a CephFS file share for user profiles and comments, and EOS directories for those large ntuple output files >

Lev: The big thing mentioned is that requests can be promoted to permanent records on the CERN Open Data Portal where a DOI is attributed to the record, which helps with FAIR principles >

The paper's improvements: Kai: One improvement they highlight is the use of an AI system combined with the LHCb Ntuple Wizard. This wizard guides users through selecting datasets and physics objects of interest, then generates YAML files for a configuration client-side >

Mira: That’s key because it reduces the barrier to entry; it removes the need for a user to learn experiment-specific software just to start producing ntuples >

Lev: And security is a big point they make: since the configurations are captured in static YAML files, the backend itself generates the necessary Python production files from those YAML files >

Kai: That means no malicious code can be injected into the WLCG or the LHCb file system because what runs in Analysis Productions originates from that service >

Mira: They also mention a way to get even more interaction: providing a request-specific communication channel between the user and the LHCb Open Data team so users can raise questions directly about their specific request >

Conclusion: Kai: So, to wrap up, this paper describes how the LHCb Ntupling Service provides access to Run one and for the first time Run two pp data through an on-demand system <ref:2610.28296#pg1,Run 1 and for the first time>. It has this whole workflow from wizard selection to output retrieval >

Mira: The big implication is that it lets researchers explore unique data from LHCb without needing to learn all the experiment-specific software first, which lowers that barrier significantly >

Lev: From a real hardware standpoint, the fact that this relies on a backend generating files from static YAML suggests the overhead lands in managing those requests rather than in running complex analysis jobs directly on our systems >

Kai: It’s a significant step for open science because it makes exploring this massive dataset much more accessible to new users who might otherwise be intimidated by the file formats >

Mira: This work, "Open Science with the LHCb Ntupling Service", is a big milestone for both LHCb and general open science because it lets us explore that data without needing deep, experiment-specific knowledge first.

Lev: I think the service’s design shows a clear path forward for making these massive datasets usable in a more standardized, on-demand manner >

More episodes

← Home