Data-quality monitoring of luminosity blocks with ATLAS Open Data: an open benchmark and a learned within-run detector

summary

The gist

Data-quality monitoring at the LHC decides, luminosity block by luminosity block, which collision data enter physics analyses…

In short

The research creates a public benchmark using ATLAS data to test how detectors monitor data quality across different runs. It shows that training a detector on past, clean data fails because histograms change between runs. A learned classifier trained on these differences can detect injected defects with high accuracy, outperforming simple scanning methods.

Key concepts

Luminosity Block (LB)
A specific segment of collision data recorded during a single run. These blocks are used to monitor the quality of the collected physics data. The study uses 39,828 such blocks from ATLAS Open Data for testing.
Deployment-like Protocol
A method where a detector is trained and calibrated using data recorded *before* the specific data it is supposed to monitor. This simulates how a real system would be used in practice, allowing researchers to test detection capabilities against unseen data.
Residuals
The differences between the features calculated for one specific LB and the average or expected features from its own run. These 'residuals' are where injected defects become most visible, making them a key signal for the learned detector.
Classifier Ensemble
A machine learning model built by combining many decision trees, trained on reference runs and synthetic defects. This ensemble is used to classify whether an LB contains a defect based on the calculated features of that block.

Terminology used across episodes

This episode discusses

The paper

Data-quality monitoring of luminosity blocks with ATLAS Open Data: an open benchmark and a learned within-run detector · Read on arXiv

Renjie Wanga

Institute of High Energy Physics, Chinese Academy of Sciences

Data-quality monitoring at the LHC decides, luminosity block by luminosity block, which collision data enter physics analyses, yet its methods cannot be compared openly because the experiments' monitoring data are not public. We construct a public benchmark from the ATLAS Open Data release of the 2015--2016 proton--proton collisions at sqrt s =13 TeV: 1289-bin histograms of reconstructed jets, electrons, photons, muons and the missing transverse momentum computed from them, for 39,828 luminosity blocks of all 217 runs in the Good Run Lists; eight types of detector defects injected at event level at three strengths; real-data labels from the Good Run Lists and the liquid-argon calorimeter error flag; and a time-ordered protocol in which every detector is trained and calibrated on earlier runs only. Histogram shapes differ far more between runs than within a run (median χ 2/n dof of 5.9 against 1.1), so references and autoencoders built across runs fail, and all competitive detectors use residuals with respect to a robust reference of each run. A gradient-boosted classifier on position-invariant features of these residuals, trained on synthetic defects and calibrated by split-conformal prediction, detects 43% of the luminosity blocks with injected defects at a false-positive rate of 1.0%, against 25% for a matched-filter scan and 63% for a single-histogram test that knows the defect pattern. For defect types excluded from its training, its efficiency falls to 18%; adding pattern-agnostic synthetic detector failures to the training raises it to 34%, above the scan, at a cost of three percentage points on the known types.

Transcript

Introduction to the show: ident: High Energy Physics: Experiment. The day in high-energy experiment: detectors, searches and limits.

Kai: We're looking at the paper: "Data-quality monitoring of luminosity blocks with ATLAS Open Data".

Mira: Here's the gist of the paper. Data-quality monitoring at the LHC decides, luminosity block by luminosity block, which collision data enter physics analyses, yet its methods cannot be compared openly because the experiments’ monitoring data are not public The paper constructs a public benchmark from the ATLAS Open Data release of 2015–2016 proton–proton collisions to compare detectors under a deployment-like protocol Benchmark Construction The benchmark consists of 39,828 luminosity blocks of 240 consecutive GRL-certified LBs from 55,620 PHYSLITE files, featuring eight types of physics-motivated detector defects injected at event level in the same events The data includes 1289-bin histograms for reconstructed jets, electrons, photons, muons and the missing transverse momentum computed from them Events are cleaned with standard ATLAS event-cleaning criteria to remove events with error states for the LAr calorimeter or other subdetectors How the training signal is built The evaluation protocol involves a rolling split where a detector is trained and calibrated on data recorded before the data it monitors, using earlier runs as reference sets The alarm threshold at a false-positive rate of α = 1% is set by the split-conformal quantile of the scores of clean reference LBs in every fold In the learned detector, features are computed for every LB, including "mean squared pull, the largest positive and negative window sums at several widths" The classifier is a histogram-based gradient-boosted decision-tree ensemble trained on reference runs and synthetic defects Where the gains come from The paper demonstrates that references built across runs fail because "Histogram shapes differ far more between runs than within a run (median χ 2/ndof of 5.9 against 1.1)" The residuals with respect to each LB’s own run are the representation in which defects become visible, and a learned classifier on these residuals finds "43% of the LBs with injected defects at 1% false-positive rate, against 25% for a matched-filter scan" The temporal context configuration (CTX) achieves a detection efficiency of 43%, compared to 25% for the scan Limitations and Future Directions The study shows that "benign real-data anomalies inside certified runs, and defects whose signature is weaker than the statistical precision of one LB" limit all detectors at a 1% false-alarm rate Tile-module failures and weaker electron/photon scale shifts are below the information content of object-level histograms at 104 events per LB The generic detector-failure prior raises efficiency on unseen types from 17.9% to 33.

Paper summary: Kai: So, looking at the paper "Data-quality monitoring of luminosity blocks with ATLAS Open Data: an open benchmark and a learned within-run detector", what we see is they built this comprehensive public benchmark using ATLAS Open Data from two thousand fifteen to two thousand sixteen <ref:2611.07819#pg1,Data-quality monitoring of luminosity blocks with ATLAS Open Data: an open>.

Mira: The thesis here is that by creating this benchmark, they allow us to compare different detector monitoring methods under a deployment protocol without needing access to the raw experimental data itself.

Lev: Essentially, they are showing that training detectors on earlier runs only and using learned classifiers on residuals can actually perform better than relying on references built across separate runs.

Kai: The implication for us is that this approach gives us a concrete way to test detector monitoring algorithms against known defects and get a measurable false positive rate at one percent, even when dealing with data that isn't fully public.

Mira: It means we can start to compare how different systems handle those subtle, event level detector anomalies in a way that is statistically sound and reproducible.

Lev: It shifts the focus from just looking at the final output to understanding the actual quality control process happening block by block during data acquisition.

Kai: So, it's about using this benchmark to see how well our methods can catch things like scale shifts or muon sector losses before they contaminate our physics results.

Mira: And that comparison under a deployment protocol is what makes this work relevant for how these systems are actually deployed in the future.

Conclusion: Kai: So, this paper sets up a public way to test how we monitor data quality at the LHC using real ATLAS data from fifteen years ago.

Mira: They took nearly forty thousand blocks of collision data and injected eight different kinds of detector defects into them to see what happens.

Lev: The core idea is to build a benchmark so we can compare our monitoring methods without needing access to those sensitive experimental files directly.

Kai: The authors show that training detectors using only earlier runs doesn't work well because the histograms just look too different from one run to another.

Mira: But they introduce these residuals—the difference between an event and its own run—and a classifier trained on those differences can find defects much better.

Lev: They report that this learned classifier finds forty-three percent of the injected defects at a one percent false positive rate, which is better than what a simpler scan could achieve.

Kai: The paper points out that even with this method, we still can't catch weak signals or errors that are smaller than the statistical noise in just one block.

Mira: And they also note that if you look at real data inside a certified run, or defects that aren't strong enough to be seen by one block alone, the detector still has limits.

Lev: They also show how changing the type of detector failure we are looking for can boost efficiency from about seventeen point nine percent up to thirty three percent.

Kai: This whole setup is a big step toward creating standardized ways for detectors to self-check during data taking.

Mira: It moves us away from just checking final results and towards understanding the quality control process happening block by block.

Lev: So this benchmark gives us a concrete tool to test if our monitoring systems are robust enough for real, high-energy physics experiments now.

Kai: It opens up a way to rigorously stress-test the data pipelines before we rely on them for serious science.

Mira: This work really helps define what a successful deployment protocol should look like when you're dealing with massive datasets like these.

More episodes

← Home