Benchmarking Machine-Learning Top Taggers from JetClass to a Full-Simulation Z'to t t bar Search: Transfer, Extrapolation, Sensitivity and Inference Cost

summary

The gist

Searches for heavy resonances decaying to tt¯ test physics beyond the Standard Model at the TeV scale…

In short

The study benchmarks machine learning top taggers trained on simulated jets (JetClass) against full CMS simulations for searching for heavy resonances decaying to ttbar at TeV scales. It finds that performance depends heavily on the specific analysis needs—like kinematic coverage and cost—rather than a single benchmark score. Transformers offer superior mass reach, especially at higher resonance masses, but tagger selection requires considering the working point of the experiment.

Key concepts

Top Tagging
This is the process of identifying jets originating from top quarks within a large-radius jet in particle collisions. It is crucial for separating these heavy particles from the much larger background of light-quark and gluon jets, which is essential for precision measurements and new physics searches.
JetClass Dataset
This is a public dataset used to train the taggers. It consists of simulated jets categorized into ten types, generated using high-fidelity simulation tools like MadGraph5 and Pythia 8.2. The training task involves classifying these jets as signal (TTBar) or background (ZJetsToNuNu).
Particle Transformers
These are a specific class of machine learning models that extend the mass reach of top tagging by approximately 400 GeV compared to other taggers. Their advantage grows with the resonance mass, making them particularly effective for searching for heavier particles.
Working Point Optimization
This refers to selecting the optimal operating point for a tagger based on specific analysis requirements, such as the desired discovery sensitivity and cost. The paper emphasizes that a tagger's performance must be judged by how well it performs at this specific working point, not just its overall benchmark score.

Terminology used across episodes

This episode discusses

The paper

Benchmarking Machine-Learning Top Taggers from JetClass to a Full-Simulation Z'to t t bar Search: Transfer, Extrapolation, Sensitivity and Inference Cost · Read on arXiv

Manisha Kumari, Amal Sarkara

School of Physical Sciences, Indian Institute of Technology Mandi

Transcript

Introduction to the show: ident: High Energy Physics: Experiment. The day in high-energy experiment: detectors, searches and limits.

Kai: I'm Kai.

Mira: And I'm Mira, here with Lev.

Kai: We're looking at the paper: "Benchmarking Machine-Learning Top Taggers from JetClass to a Full-Simulation Z'to t t bar Search".

Mira: Here's the gist of the paper. Searches for heavy resonances decaying to tt¯ test physics beyond the Standard Model at the TeV scale, and their reach depends on how well boosted top jets are identified The overall benchmark order of the taggers is preserved, but taggers with similar area under the ROC curve (AUC) differ by up to a factor of three in jet-level discovery significance The particle transformers extend the mass reach of the search by 400 GeV, and their lead grows with resonance mass A tagger must be chosen on its kinematic coverage, the working point of the analysis, its discovery sensitivity and its cost, and not on its benchmark score alone The top quark is the heaviest known elementary particle Separating top quarks from the much larger background of light-quark and gluon jets is therefore central to both precision measurements and new-physics searches In the boosted regime, all three decay products fall within a single large-radius jet Separating boosted top jets from QCD jets is known as top tagging How the training signal is built All taggers are trained on JetClass

23: , a public dataset of simulated jets in ten categories, generated with MadGraph5 aMC@NLO

24: and Pythia 8.2

25: The task is formulated as a binary classification problem, with TTBar jets as the signal and ZJetsToNuNu jets as the background The taggers are evaluated in a simulated Z' → tt¯ search using CMS Open Data simulation from the RunIISummer20UL16MiniAODv2 campaign The signal consists of the leptophobic topcolour Z' benchmark

1, 2: at six mass hypotheses: mZ' = 1200, 1400, 1600, 1800, 2000, and 2500 GeV The dominant background is QCD multijet production Where the gains come from The advantage of the transformers grows with resonance mass, and a single AUC value per tagger cannot describe this trend The transformers remain the leading group in the search, while UParT, the best transformer on JetClass, falls about 0.016–0.017 behind them The growing fraction of fully merged tops raises the AUC with mass, while each tagger becomes worse at recognizing a merged top as pT increases The transformers keep their discrimination best when extrapolating above the training pT range Transfer and Extrapolation The results show that a tagger must be chosen on its kinematic coverage, the working point of the analysis, its discovery sensitivity and its cost, and not on its benchmark score alone The performance measured inside that range need not represent a tagger across the full kinematic range of a search The transformer advantage is largest in a region outside the range used for the benchmark ranking The pT-binned comparison does not, however, single out extrapolation as the only cause Inference Cost and Robustness The BDT is the fastest tagger, with a latency about 140 times lower than that of transformers, but it also yields the weakest tagging The three transformers have nearly identical latency and model size, as expected from the similar network configurations of ParT and GloParT The overall ranking stays broadly the same, with GloParT highest at 1200 GeV and ParT leading at all higher masses The transformers also keep a smaller fraction of the background than the other taggers The results provide a quantitative basis for selecting a top tagger based on the kinematic regime, the required background rejection, and the computing limits of the intended analysis The overall AUC ranking is largely preserved from JetClass to full CMS simulation (Sections 5.1 and 5.2), but the benchmark metrics do not fully reflect the differences in search performance The transformers also keep their discrimination better as jet pT increases, including above the 500–1000 GeV training range The gap between the transformers and the other taggers is largest at high resonance mass, where the cross section is smallest Top taggers should therefore be evaluated within the physics analysis they are meant to serve The present study compares their discrimination, stability, and pileup robustness (Section 5.4), but several aspects remain open Systematic uncertainties such as those on the jet energy scale and resolution and on the mis-tag rate calibration are needed for a complete assessment Further work could extend the training pT range, match the constituent multiplicities, apply PUPPI pileup mitigation

54: , add more resonance mass points and fit the full tagger-score distribution REFERENCES

1: R.M.

Kai: Harris and S.

Mira: Jain, Cross Sections for Leptophobic Topcolor Z’ decaying to top-antitop, Eur.

Kai: Phys.

Title and authors: Kai: So, we're looking at this paper, "Benchmarking Machine-Learning Top Taggers from JetClass to a Full-Simulation Z'to t t bar Search: Transfer, Extrapolation, Sensitivity and Inference Cost". This whole study is about testing how well machine learning taggers work when we move from simple training samples to the messy reality of a full experiment.

Mira: Exactly. It’s not just about seeing which tagger scores highest on some test set. It's about how that performance holds up when you actually run a search for heavy resonances decaying to top quarks in a full simulation, including all the pileup effects from real data.

Lev: From my side, I'm thinking about what that means for hardware. If we use these taggers on real data, we need to know if the performance we see on paper translates to what our actual computing setup can handle in terms of latency and throughput.

Kai: Right. The paper focuses on seven different tagger architectures: a boosted decision tree, two image-based networks, ParticleNet, and three particle transformers like ParT, UParT, and GloParT. They train them all on the same JetClass dataset to establish a baseline ranking.

Mira: And what they immediately point out is that that initial benchmark ranking isn't the whole story. They find that taggers with similar performance on their training set can differ by up to a factor of three in how much discovery significance they add in a real search, depending on where you are looking one <ref:2611.15274#pg1>.

Lev: That sounds like a big deal for experimental design. It suggests that just picking the best-looking number isn't enough; you have to consider the actual physics analysis you plan to do.

Kai: And they push back on that idea by showing how mass dependence comes into play. They show that the advantage of those particle transformers grows with the resonance mass, and one single score for a tagger just can't capture that trend.

Mira: That makes sense theoretically because as the resonance gets heavier, the top quark's decay products become more collimated, which changes how easy it is to separate them from QCD background one <ref:2611.15274#pg1>. The paper shows that UParT, which was the best on their training set, falls about zero point sixteen to zero point seventeen behind the transformers at higher masses.

Lev: If we’re aiming for a high mass resonance, like above one thousand two hundred GeV, then that small difference in performance becomes much more significant when you're actually trying to make a discovery.

Title and authors: Kai: Then they introduce the idea of "particle transformers" as something that extends the mass reach of the search by four hundred GeV, and their lead over other taggers gets bigger as the resonance mass goes up one <ref:2611.15274#pg1>.

Mira: But they also caution that you can't just pick a tagger based on its benchmark score alone. You have to look at where it covers kinematically, what your analysis is actually working at—that's the working point—its discovery sensitivity, and obviously the cost of running it one <ref:2611.15274#pg1>.

Lev: So they are saying that you need to evaluate a tagger based on its kinematic coverage in the specific region you're interested in, not just some abstract number from a training set one <ref:2611.15274#pg1>.

Kai: They show that the performance measured inside that narrow range might not represent how good it is across the entire possible kinematic range of a search. The transformer advantage is biggest when you look outside of that specific benchmark ranking area one <ref:2611.15274#pg1>.

Mira: They use jet pT bins to show this because they look at comparisons in different reconstructed top quark transverse momentum bins, and that comparison doesn't just point to extrapolation as the only reason for the difference one <ref:2611.15274#pg1>.

Lev: That’s practical. It means we can actually test how far these models are reliable beyond where we trained them, which is important when searching for heavy things.

Kai: Then they look at inference cost and robustness. They mention that the boosted decision tree is the fastest tagger, with latency about one hundred forty times lower than those particle transformers one <ref:2611.15274#pg1>.

Mira: But that speed comes at a cost; it yields the weakest tagging performance overall. Conversely, those three particle transformers have nearly identical latency and model size, which makes sense given their network configurations are similar to ParT and GloParT one <ref:2611.15274#pg1>.

Lev: So you’re looking at a trade-off between raw speed and discrimination power here. For real hardware running large jet samples, that inference cost really matters for the overall analysis budget.

Kai: They note that the transformers also keep a smaller fraction of background compared to the other taggers tested one <ref:2611.15274#pg1>. That's important for reducing noise in a search.

Title and authors: Mira: The paper provides a quantitative basis for choosing which top tagger to use based on things like the kinematic regime, how much background rejection you need, and the computing limits of your intended analysis one <ref:2611.15274#pg1>.

Lev: So if I’m designing an experiment, I can now look at these models and think about how they actually fit into my real hardware constraints.

Kai: To wrap up on the performance side, they compare discrimination, stability, and pileup robustness across different conditions one <ref:2611.15274#pg1>. They point out that pileup seems to have only a minor effect on all the taggers when you look at AUC as a function of the number of reconstructed primary vertices one <ref:2611.15274#pg1>.

Mira: So even with pileup present in the full simulation, they suggest that for practical choices, you might not need explicit pileup mitigation algorithms because it doesn't hurt performance much one <ref:2611.15274#pg1>.

Lev: That simplifies things if we are building real detectors; less overhead to worry about if the effect is small across the board.

Kai: So overall, the benchmark ranking is mostly preserved from JetClass to a full CMS simulation, but those benchmark metrics don't fully capture the differences in how these taggers perform in a real search one <ref:2611.15274#pg1,from JetClass to a full>.

Mira: The particle transformers also maintain better discrimination as you increase the jet pT, even above that training range of five hundred to one thousand GeV one <ref:2611.15274#pg1>. The gap between the transformers and other taggers gets largest at high resonance mass, where the cross section is smallest one <ref:2611.15274#pg1>.

Lev: So for searching for those heavy resonances, especially at higher masses, those particle transformers seem like the most promising route based on this study.

Kai: The main thing to remember is that you have to evaluate a tagger within the specific physics analysis it's meant to serve one <ref:2611.15274#pg1>. It’s not just about some general score.

Mira: That's the core message of "Benchmarking Machine-Learning Top Taggers from JetClass to a Full-Simulation Z'to t t bar Search: Transfer, Extrapolation, Sensitivity and Inference Cost" one <ref:2611.15274#pg1,Benchmarking Machine-Learning Top Taggers from JetClass to a Full-Simulation Z>. They show us how to move beyond simple rankings to make real experimental decisions.

Lev: Yeah, it gives us a framework for when we select our tools for analyzing new physics signals.

Kai: We'll take another break now, and when we come back, we’re going to look at some work on searching for different decay modes in super-kamiokande five <ref:2611.15274#pg2>.

The paper's summary: Kai: So, we're looking at how researchers are actually testing these machine learning top taggers when they move from their simple training data to a full simulation of a search for heavy particles decaying into tops.

Mira: Exactly. The paper is basically taking these different tagging models—like those decision trees and the particle transformers—and seeing if the performance they get on one test set actually holds up when you run a real search, like looking for some Z prime boson, in a full simulation with all the messy background stuff.

Lev: From my side, I'm thinking about how much overhead this adds. If we take these models and run them on real hardware to look for these signals at the TeV scale, what does that latency actually look like?

Kai: Right. The researchers are saying that just looking at a single benchmark score doesn't tell you the whole story. They found that different taggers can be way different in their actual discovery power, depending on where you set your search parameters.

Mira: And they show how resonance mass matters a lot. As the mass of this heavy particle goes up, these particle transformers actually get better at separating those top jets from the background, and that advantage gets bigger for higher masses.

Lev: That’s interesting because it means we can actually predict where these models will perform best if we are looking for very heavy stuff at a high energy. But they also point out that you have to choose your working point carefully, not just rely on a general score from the training data.

Kai: They suggest that the performance measured in the narrow range they used for ranking might not be what happens when you push it to the full kinematic range of a search, especially at higher transverse momentum.

Mira: That’s because they look at comparisons in different top quark momentum bins, and that comparison shows that performance doesn't just depend on extrapolating outside the training set.

Lev: So if we are designing an experiment, this means we need to think about the specific kinematic region we are interested in and how much background rejection we actually need for our analysis.

Kai: They also look at the cost involved. The boosted decision tree is super fast, with latency maybe one hundred forty times lower than those transformers, but it gives weaker tagging performance overall.

Mira: But those particle transformers keep a smaller fraction of the background than others, which is important for reducing noise in the signal you’re looking for.

Lev: So they give us a way to choose—you can pick the fastest one if you need speed and can live with weaker discrimination, or pick something like GloParT if you prioritize discrimination and can handle the extra computational cost.

Kai: The overall picture is that while the benchmark rankings are mostly stable from their initial training set to a full simulation, those benchmark metrics don't fully capture how these taggers behave in a real search scenario.

Mira: And they add that when you push the mass higher, those transformers keep their edge over the competition as you look at higher top quark momentum.

Lev: That means for searching for heavy resonances, especially at high masses, those particle transformers seem to be the most promising route based on this study.

Kai: So we can stop just looking at a single number and start evaluating these tools in the context of our actual physics analysis and our hardware limits.

Mira: It gives us a framework for making practical decisions about which tagger architecture is right for a given search goal.

Lev: Next up, we’re going to look at some work on searching for different decay modes in super-kamiokande.

The paper's improvements: Kai: So, we're talking about how this paper suggests we can actually improve these tagging methods going forward. It’s not just about testing what works now, but figuring out how to make them better for real experiments down the line.

Mira: Right. The authors point out a few things they think would help move these tools closer to being useful in a real search scenario, specifically around how we use the data and how we handle those physical effects.

Lev: I'm thinking about what that means for hardware again—how do we actually build the system to implement these suggestions? Are they asking us to change the architecture or just change our training process?

Kai: They suggest extending the training transverse momentum range further, beyond that five hundred to one thousand GeV mark, which helps us see how stable the discrimination is when you look at higher energy jets.

Mira: That makes sense theoretically because we need to understand if those models stay reliable when they are extrapolating outside of their original training conditions. They also suggest matching the constituent multiplicities in the training data to better reflect what we’ll actually measure in a full experiment.

Lev: If we have to match multiplicities, that adds a lot of complexity to the simulation side, which I know is already heavy. It means more time spent on generating those synthetic jets before we can even start running searches on them.

Kai: And they also mention applying something called PUPPI pileup mitigation, which is an AI technique designed to reduce the effect of background noise from multiple proton collisions in real data.

Mira: That’s a big point because pileup effects are notoriously tricky in experiments, and if we can mitigate that noise without losing our tagging power, it makes the analysis much cleaner.

Lev: It's good that they flag pileup as something worth addressing, but they also mention that for all the taggers tested, the effect of pileup seems to be relatively minor when you look at how many primary vertices are reconstructed.

Kai: So they’re suggesting we don't necessarily need a massive dedicated pileup mitigation algorithm if we choose our tagger carefully and look at the overall performance across different numbers of reconstructed vertices.

Mira: And they also suggest adding more resonance mass points to their study, so they can get a better picture of how the performance gap changes as the search moves into even higher energy regimes.

Lev: More data points for mass means more simulation work, which is always a challenge when you’re trying to keep your error bars tight on real physics results.

Kai: It all comes back to this—the paper isn't just a performance report; it’s a roadmap showing where the next steps in improving these machine learning top taggers need to happen.

Mira: It gives us concrete ideas for how we should design future simulations and how we should think about the necessary software tools for real experimental searches.

Lev: So, moving forward, the focus shifts from just finding a good score to building robust systems that can handle the complexity of high-energy physics data.

Conclusion: Kai: So we’ve gone through how they test these machine learning top taggers on real simulation data, looking at transfer and extrapolation, sensitivity, and cost in their paper "Benchmarking Machine Learning Top Taggers from JetClass to a Full Simulation Z prime to t t bar Search".

Mira: Exactly. The main point is that the simple benchmark scores aren't enough; you need to look at how performance changes based on the specific physics analysis you're doing and the mass of the particle you are searching for.

Lev: From an engineering standpoint, it means we have a much clearer picture of where these tools will actually be useful in a real search pipeline. We know when we can trust their performance and when we need to spend more time on them.

Kai: They show that the particle transformers keep their discrimination better as the jet transverse momentum increases, even above that training range of five hundred to one thousand GeV.

Mira: That’s important because it suggests these models are robust for searching for heavier particles, which is exactly what we want when looking for new physics at the TeV scale.

Lev: If we can rely on that extrapolation, it reduces the need for us to run even more complex and time-consuming simulations just to see if a signal exists.

Kai: They also found that pileup has only a minor effect on all taggers when you look at how many primary vertices are reconstructed, so we might not need massive dedicated pileup mitigation algorithms if we choose our tagger carefully.

Mira: That’s reassuring because it simplifies the pipeline; less noise handling means more time spent looking for the actual physics signal.

Lev: It’s a practical point, because I'm constantly thinking about how much overhead that mitigation adds to the total run time and computational budget for real hardware.

Kai: Overall, this study provides a quantitative basis for choosing which tagger to use based on the kinematic regime, background rejection needs, and our computing limits.

Mira: So when you take that paper "Benchmarking Machine Learning Top Taggers from JetClass to a Full Simulation Z prime to t t bar Search", you see it’s less about finding the single best score and more about fitting the tool to the job.

Lev: It gives us a framework for when we select our tools for analyzing new physics signals, which is what matters most when building hardware.

Kai: It's a solid study that shows how to bridge the gap between training data and real experiment.

Mira: Agreed. And it opens up a lot of doors for future simulations and experimental design in this area.

Lev: Next time, we'll be looking at some work on searching for different decay modes in super-kamiokande to see how machine learning handles those specific topologies.

More episodes

← Home