Data-quality monitoring of luminosity blocks with ATLAS Open Data: an open benchmark and a learned within-run detector
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: High Energy Physics: Experiment. The day in high-energy experiment: detectors, searches and limits.
Kai: We're looking at the paper: "Data-quality monitoring of luminosity blocks with ATLAS Open Data".
Mira: Here's the gist of the paper. Data-quality monitoring at the LHC decides, luminosity block by luminosity block, which collision data enter physics analyses, yet its methods cannot be compared openly because the experiments’ monitoring data are not public The paper constructs a public benchmark from the ATLAS Open Data release of 2015–2016 proton–proton collisions to compare detectors under a deployment-like protocol Benchmark Construction The benchmark consists of 39,828 luminosity blocks of 240 consecutive GRL-certified LBs from 55,620 PHYSLITE files, featuring eight types of physics-motivated detector defects injected at event level in the same events The data includes 1289-bin histograms for reconstructed jets, electrons, photons, muons and the missing transverse momentum computed from them Events are cleaned with standard ATLAS event-cleaning criteria to remove events with error states for the LAr calorimeter or other subdetectors How the training signal is built The evaluation protocol involves a rolling split where a detector is trained and calibrated on data recorded before the data it monitors, using earlier runs as reference sets The alarm threshold at a false-positive rate of α = 1% is set by the split-conformal quantile of the scores of clean reference LBs in every fold In the learned detector, features are computed for every LB, including "mean squared pull, the largest positive and negative window sums at several widths" The classifier is a histogram-based gradient-boosted decision-tree ensemble trained on reference runs and synthetic defects Where the gains come from The paper demonstrates that references built across runs fail because "Histogram shapes differ far more between runs than within a run (median χ 2/ndof of 5.9 against 1.1)" The residuals with respect to each LB’s own run are the representation in which defects become visible, and a learned classifier on these residuals finds "43% of the LBs with injected defects at 1% false-positive rate, against 25% for a matched-filter scan" The temporal context configuration (CTX) achieves a detection efficiency of 43%, compared to 25% for the scan Limitations and Future Directions The study shows that "benign real-data anomalies inside certified runs, and defects whose signature is weaker than the statistical precision of one LB" limit all detectors at a 1% false-alarm rate Tile-module failures and weaker electron/photon scale shifts are below the information content of object-level histograms at 104 events per LB The generic detector-failure prior raises efficiency on unseen types from 17.9% to 33.
Paper summary: Kai: So, looking at the paper "Data-quality monitoring of luminosity blocks with ATLAS Open Data: an open benchmark and a learned within-run detector", what we see is they built this comprehensive public benchmark using ATLAS Open Data from two thousand fifteen to two thousand sixteen <ref:2611.07819#pg1,Data-quality monitoring of luminosity blocks with ATLAS Open Data: an open>.
Mira: The thesis here is that by creating this benchmark, they allow us to compare different detector monitoring methods under a deployment protocol without needing access to the raw experimental data itself.
Lev: Essentially, they are showing that training detectors on earlier runs only and using learned classifiers on residuals can actually perform better than relying on references built across separate runs.
Kai: The implication for us is that this approach gives us a concrete way to test detector monitoring algorithms against known defects and get a measurable false positive rate at one percent, even when dealing with data that isn't fully public.
Mira: It means we can start to compare how different systems handle those subtle, event level detector anomalies in a way that is statistically sound and reproducible.
Lev: It shifts the focus from just looking at the final output to understanding the actual quality control process happening block by block during data acquisition.
Kai: So, it's about using this benchmark to see how well our methods can catch things like scale shifts or muon sector losses before they contaminate our physics results.
Mira: And that comparison under a deployment protocol is what makes this work relevant for how these systems are actually deployed in the future.
Conclusion: Kai: So, this paper sets up a public way to test how we monitor data quality at the LHC using real ATLAS data from fifteen years ago.
Mira: They took nearly forty thousand blocks of collision data and injected eight different kinds of detector defects into them to see what happens.
Lev: The core idea is to build a benchmark so we can compare our monitoring methods without needing access to those sensitive experimental files directly.
Kai: The authors show that training detectors using only earlier runs doesn't work well because the histograms just look too different from one run to another.
Mira: But they introduce these residuals—the difference between an event and its own run—and a classifier trained on those differences can find defects much better.
Lev: They report that this learned classifier finds forty-three percent of the injected defects at a one percent false positive rate, which is better than what a simpler scan could achieve.
Kai: The paper points out that even with this method, we still can't catch weak signals or errors that are smaller than the statistical noise in just one block.
Mira: And they also note that if you look at real data inside a certified run, or defects that aren't strong enough to be seen by one block alone, the detector still has limits.
Lev: They also show how changing the type of detector failure we are looking for can boost efficiency from about seventeen point nine percent up to thirty three percent.
Kai: This whole setup is a big step toward creating standardized ways for detectors to self-check during data taking.
Mira: It moves us away from just checking final results and towards understanding the quality control process happening block by block.
Lev: So this benchmark gives us a concrete tool to test if our monitoring systems are robust enough for real, high-energy physics experiments now.
Kai: It opens up a way to rigorously stress-test the data pipelines before we rely on them for serious science.
Mira: This work really helps define what a successful deployment protocol should look like when you're dealing with massive datasets like these.
Renjie Wanga
Institute of High Energy Physics, Chinese Academy of Sciences
hep-ex
Submitted: 2026-11-05
Updated: 2026-11-05
Comments: 11 pages, 6 figures
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 91/100
The gist: Data-quality monitoring at the LHC decides, luminosity block by luminosity block, which collision data enter physics analyses…
Key concepts
- Luminosity Block (LB)
- A specific segment of collision data recorded during a single run. These blocks are used to monitor the quality of the collected physics data. The study uses 39,828 such blocks from ATLAS Open Data for testing.
- Deployment-like Protocol
- A method where a detector is trained and calibrated using data recorded *before* the specific data it is supposed to monitor. This simulates how a real system would be used in practice, allowing researchers to test detection capabilities against unseen data.
- Residuals
- The differences between the features calculated for one specific LB and the average or expected features from its own run. These 'residuals' are where injected defects become most visible, making them a key signal for the learned detector.
- Classifier Ensemble
- A machine learning model built by combining many decision trees, trained on reference runs and synthetic defects. This ensemble is used to classify whether an LB contains a defect based on the calculated features of that block.
Terminology
Summary
Data-quality monitoring at the LHC decides, luminosity block by luminosity block, which collision data enter physics analyses, yet its methods cannot be compared openly because the experiments’ monitoring data are not public The paper constructs a public benchmark from the ATLAS Open Data release of 2015–2016 proton–proton collisions to compare detectors under a deployment-like protocol
Benchmark Construction
The benchmark consists of 39,828 luminosity blocks of 240 consecutive GRL-certified LBs from 55,620 PHYSLITE files, featuring eight types of physics-motivated detector defects injected at event level in the same events The data includes 1289-bin histograms for reconstructed jets, electrons, photons, muons and the missing transverse momentum computed from them Events are cleaned with standard ATLAS event-cleaning criteria to remove events with error states for the LAr calorimeter or other subdetectors
How the training signal is built
The evaluation protocol involves a rolling split where a detector is trained and calibrated on data recorded before the data it monitors, using earlier runs as reference sets The alarm threshold at a false-positive rate of α = 1% is set by the split-conformal quantile of the scores of clean reference LBs in every fold In the learned detector, features are computed for every LB, including mean squared pull, the largest positive and negative window sums at several widths
The classifier is a histogram-based gradient-boosted decision-tree ensemble trained on reference runs and synthetic defects
Where the gains come from
The paper demonstrates that references built across runs fail because Histogram shapes differ far more between runs than within a run (median χ
2/ndof of 5. 9 against 1.Histogram shapes differ far more between runs than within a run (median χ 2/ndof of 5.9 against 1. p. 0
The residuals with respect to each LB’s own run are the representation in which defects become visible, and a learned classifier on these residuals finds Where the paper says it
43% of the LBs with injected defects at 1% false-positive rate, against 25% for a matched-filter scan
The temporal context configuration (CTX) achieves a detection efficiency of 43%, compared to 25% for the scan
Limitations and Future Directions
The study shows that benign real-data anomalies inside certified runs, and defects whose signature is weaker than the statistical precision of one LB
limit all detectors at a 1% false-alarm rate Tile-module failures and weaker electron/photon scale shifts are below the information content of object-level histograms at 104 events per LB The generic detector-failure prior raises efficiency on unseen types from 17.9% to 33.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Learned classifier on residuals can detect 43 percent of luminosity blocks with injected defects at a false-positive rate of 1.0 percent, surpassing methods like matched-filter scans which achieve only 25 percent detection efficiency under the same conditions.
-
The system can utilize temporal context (CTX) to improve performance, achieving a higher detection efficiency of 43 percent and lower false-positive rates of 0.8% when compared to single-LB models.
-
A safety net combining the learned score with the scan score allows the system to flag LBs based on either detector finding an anomaly, providing robustness against defects outside the training catalogue.
-
The system can be configured with a generic detector-failure prior that raises efficiency for unseen defect types from 18 percent to 34 percent by incorporating patterns not derived from the injected catalogue.
-
The system can be deployed using only the within-run scan as an initial deployment strategy, which achieves a detection efficiency of 25 percent with a conformal false-positive rate of 0.9 to 1.0 percent without requiring extensive historical data for each run.
Abstract
Data-quality monitoring at the LHC decides, luminosity block by luminosity block, which collision data enter physics analyses, yet its methods cannot be compared openly because the experiments' monitoring data are not public. We construct a public benchmark from the ATLAS Open Data release of the 2015--2016 proton--proton collisions at sqrt s =13 TeV: 1289-bin histograms of reconstructed jets, electrons, photons, muons and the missing transverse momentum computed from them, for 39,828 luminosity blocks of all 217 runs in the Good Run Lists; eight types of detector defects injected at event level at three strengths; real-data labels from the Good Run Lists and the liquid-argon calorimeter error flag; and a time-ordered protocol in which every detector is trained and calibrated on earlier runs only. Histogram shapes differ far more between runs than within a run (median χ 2/n dof of 5.9 against 1.1), so references and autoencoders built across runs fail, and all competitive detectors use residuals with respect to a robust reference of each run. A gradient-boosted classifier on position-invariant features of these residuals, trained on synthetic defects and calibrated by split-conformal prediction, detects 43% of the luminosity blocks with injected defects at a false-positive rate of 1.0%, against 25% for a matched-filter scan and 63% for a single-histogram test that knows the defect pattern. For defect types excluded from its training, its efficiency falls to 18%; adding pattern-agnostic synthetic detector failures to the training raises it to 34%, above the scan, at a cost of three percentage points on the known types.
Sources
- ATLAS data quality operations and performance for 2015-2018 data-taking
- Machine Learning for Anomaly Detection in Particle Physics
- Towards automation of data quality system for CERN CMS experiment
- Detector monitoring with artificial neural networks at the CMS experiment at the CERN Large Hadron Collider
- Autoencoder-based Anomaly Detection System for Online Data Quality Monitoring of the CMS Electromagnetic Calorimeter
- Anomaly Detection Based on Machine Learning for the CMS Electromagnetic Calorimeter Online Data Quality Monitoring
- Spatio-Temporal Anomaly Detection with Graph Networks for Data Quality Monitoring of the Hadron Calorimeter
- Anomaly Detection for Automated Data Quality Monitoring in the CMS Detector
- Fast kernel methods for Data Quality Monitoring as a goodness-of-fit test
- DINAMO: Dynamic and INterpretable Anomaly MOnitoring for Large-Scale Particle Physics Experiments
- MEDIC: a network for monitoring data quality in collider experiments
- The LHC Olympics 2020: A Community Challenge for Anomaly Detection in High Energy Physics
- The Dark Machines Anomaly Score Challenge: Benchmark Data and Model Independent Event Classification for the Large Hadron Collider
- LHC physics dataset for unsupervised New Physics detection at 40 MHz
- Software and computing for Run 3 of the ATLAS experiment at the LHC
- The anti-k_t jet clustering algorithm
- Jet reconstruction and performance using particle flow with the ATLAS Detector
- Monitoring and data quality assessment of the ATLAS liquid argon calorimeter
- A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification
- Operation of the ATLAS trigger system in Run 2
Related papers
- DrSai: an agentic AI system toward autonomous physics analysis workflows at BESIII
- CANTON- mu Proposal: A Next-Generation Muon g-2 Measurement at Sub-0.1 ppm Precision
- Search for dark photon dark matter near 1,meV using a superconducting mixer
- ANUBIS detector and its sensitivity to neutral long-lived particles
- How to scale your HEP ML models: A recipe for robust architecture comparisons at scale
- Search for the 16 O(ppp) to 13 C pi+ pi+ e+ Decay Mode in Super-Kamiokande Using Machine Learning Techniques