SURFing to the Fundamental Limit of Jet Tagging

arXiv:2511.15779 · hep-ph, cs.LG, hep-ex, physics.data-an · Submitted 2025-11-19 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: High Energy Physics: Phenomenology. The day in particle phenomenology: models, signatures and constraints.

Kai: I'm Kai.

Mira: And I'm Mira, here with Lev.

Kai: We're looking at the paper: "SURFing to the Fundamental Limit of Jet Tagging".

Paper summary: Mira: To recap what we've covered, this paper introduces the SURF method to validate generative models by using one model with tractable likelihoods, like EPiCFM, as a surrogate reference for another model like GPT.

Lev: The core idea is training the target GPT model on samples from that tractable surrogate EPiCFM, and then comparing the resulting likelihood ratios against those calculated directly from the surrogate.

Kai: What they found is that this comparison shows the NP optimal ROC curve derived from the GPT likelihoods actually far exceeds what's achievable by the EPiCFM surrogate itself.

Mira: That discrepancy demonstrates unambiguously that these GPT models are artificially inflating their apparent separation between top and QCD jets beyond the physical limit represented by EPiCFM.

Lev: They also pointed out that this overfitting happens right away, which makes those likelihood ratios inherently hard for actual classifiers to reproduce accurately.

Kai: The paper essentially argues against the idea that we're hitting some fundamental wall just because a new generative model looks better on paper.

Mira: Instead, they conclude that the inflated optimal behavior comes from artifacts in the GPT likelihood landscape, not from deficiencies in classifier performance as previously suggested.

Lev: So, for those of us building error correction systems for real hardware, this is important because it tells us what kind of noise we’re actually dealing with when we try to model these processes.

Kai: It shifts the focus from just chasing better classifiers to understanding the statistical properties of the generative models themselves.

Conclusion: Mira: So, looking at "SURFing to the Fundamental Limit of Jet Tagging," it seems like this work is more about validating tools than finding a new physical limit for jet tagging.

Lev: The authors are focused on using their SURF method to check the validity of generative models by comparing them against a tractable surrogate reference model, EPiCFM.

Kai: Ultimately, the conclusion they draw is that there's no evidence of an inflated fundamental limit for jet tagging itself based on this comparison.

Mira: The picture they present is that the real limit isn't far away from what we already have in state-of-the-art classifiers, considering how different ROC curves align.

Lev: They suggest that the consistency between classifiers trained on JetClass and EPiCFM samples points to this conclusion about where we stand right now.

Kai: So, to put it plainly, the results show that the performance limitations aren't where people thought they were, but rather in how these generative models are producing their likelihood ratios.

Ian Pang, *, Darius A. Faroughy, †David Shih, ↓Ranit Das, Gregor Kasieczka

Rutgers University · Institute for Theoretical Physics, University of Heidelberg, Germany · Institut für Experimentalphysik, University of Hamburg

hep-ph, cs.LG, hep-ex, physics.data-an

Submitted: 2025-11-19

Updated: 2026-11-16

Code: https://github.com/dfaroughy/JetSequences

Importance score: 91/100

The gist: The paper introduces the Surrogate Reference (SURF) method to validate generative models by comparing them against a tractable surrogate reference model to determine if modern jet taggers have…

Key concepts

Surrogate Reference (SURF) Method
This method uses one tractable generative model (EPiC-FM) as a simple reference to test another complex one (GPT). It allows for exact statistical tests by training the target model on samples from the surrogate, helping to determine if the complex model is overstating its separation capabilities.
Generative Models (GPT vs. EPiC-FM)
The paper compares two models: GPT, which views jets as ordered sequences of tokens, and EPiC-FM, which treats jets as unordered particle clouds with kinematic features. The comparison shows that GPT's likelihood ratios inaccurately suggest a better separation than the simpler EPiC-FM model allows.
Fundamental Limit
This refers to the absolute maximum separation power achievable when distinguishing top jets from QCD jets. The study concludes that this limit is not far away from current state-of-the-art classifiers, suggesting that observed inflation in separation power comes from modeling artifacts, not classifier deficiencies.

Terminology

Summary

The paper introduces the Surrogate Reference (SURF) method to validate generative models by comparing them against a tractable surrogate reference model to determine if modern jet taggers have reached their fundamental statistical limit The GPT model unphysically exaggerates top vs. p. 0 QCD separation power encoded in the surrogate reference, implying that it is giving a misleading picture of the fundamental limit. p. 0

Revisiting Generative Model Interpretations

The authors reevaluate the conclusion from previous work regarding autoregressive GPTstyle generative models probing the fundamental limit for top vs. QCD jets. They found that while GPT-based likelihoods produce receiveroperator-characteristic (ROC) curves that appear to substantially exceed those of all existing classifiers trained on GPT samples, this cannot explain the inflated separation of the GPT jets because the discretized top and QCD jets are if anything less separable than their continuous counterparts. p. 0 They show that GPT-generated jets are perfectly separable from continuous jets, while being close to the discretized jets that the GPT models were trained on. p. 0

Introducing the SURF Method

The SURF method is introduced as a new approach to validating generative models by using one generative model with tractable likelihoods (EPiC-FM) to evaluate another generative model with tractable likelihoods (GPT). The key idea is to use one generative model with tractable likelihoods (EPiC-FM) to evaluate the other generative model with tractable likelihoods (GPT), where the first one serves as the surrogate reference model for the second. p. 1 This framework enables exact Neyman–Pearson tests by training the target model on samples from another tractable surrogate, which is itself trained on real data. p. 0

Evaluating Model Fidelity and Artifacts

Within the SURF framework, they find that the NP ROC curve derived from the GPT likelihoods far exceeds that of the EPiC-FM surrogate itself. This demonstrates unambiguously that the GPT model artificially inflates the apparent separation between top and QCD jets beyond the physical limit represented by the surrogate reference. p. 1 They find evidence that GPT models exhibit signs of overfitting to their training data, which produces likelihood ratios that are inherently difficult for classifiers to reproduce, leading to artificially inflated ROC curves. p. 1

Conclusion on the Fundamental Limit

In the end, they find no evidence of an inflated “fundamental limit” of jet tagging. Instead, the overall picture suggests that the fundamental limit is not far away from state-of-the-art classifiers, as the concurrence of top vs. p. 1 QCD ROC curves from OmniLearn trained on JetClass and on EPiC-FM together with the exact likelihoods of EPiC-FM suggests this. p. 1 The results suggest that the inflated NPoptimal behavior originates from artifacts in the GPT likelihood landscape, not from deficiencies in classifier performance as was claimed in Ref. p. 5 [13].

Investigating Sources of ROC Inflation

The paper investigates two potential sources for the inflated GPT top vs. QCD curve: bin-smearing and overfitting. p. 6 They show that bin-smearing reduces separability, as the binning of the jets, despite being a major deviation from the continuous jets, cannot be the cause of the inflated top vs. p. 7 QCD ROC curve. p. 0 Furthermore, they find evidence that GPT models exhibit overfitting almost immediately (increasing gap between train vs. p. 7 val losses), and this overfitting amplifies the apparent separation when training sets are small. p. 7

Future Directions for Probing the Limit

The authors suggest avenues for further probing the fundamental limit of jet tagging, including improving the surrogate reference model to be more expressive, such as replacing EPiC with a transformer. They also suggest repeating this study with a more faithful target generative model that does not inflate the true ROC curve of the surrogate reference. p. 9 Finally, they propose bringing the empirical, data-driven approach into contact with the theory-driven approach to elucidate further the true fundamental limit of jet tagging. p. 9

Model Descriptions and Evaluations

The paper details two generative models: GPT, which views jets as pT-ordered sequences of tokenized particles, and EPiC-FM, which represents jets as unordered particle clouds with kinematic features x = (pT, ∆η, ∆ϕ)i. The GPT model uses voxel tokenization where each three-dimensional voxel in feature space is mapped to a unique token, resulting in a vocabulary of size V = 36,000. p. 1 The EPiC-FM architecture uses conditional flow-matching (CFM) to train a continuous normalizing flow equipped with a timedependent EPiC architecture. p. 3 The GPT model is trained by maximizing the autoregressive likelihood log pθ(tx) = N sum n=1 log pθ t n given the preceding tokens. p. 3

Baseline Performance Comparisons

Baseline evaluations show that an Omnilearn classifier separates both datasets almost perfectly (AUC = 0.9949 for QCD and 0. p. 29734 for tops) when distinguishing JetClass data from GPT-generated jets. p. 2 When comparing EPiC-FM generated jets to JetClass jets, they find AUCs of 0. p. 36729 for QCD and 0. p. 37703 for tops, indicating that EPiC-FM captures the main features of the simulation with moderate levels of mismodeling. p. 3

Validation of Likelihood Computation

The authors verify the robustness of FM-based log-likelihood ratios to errors accumulated in the backward update by assessing consistency between forward and backward Euler integration. They also confirm that in a toy 10D Gaussian model, the ROC curve obtained from FM likelihood ratios coincides quite well with the analytic NP ROC. p. 5 The consistency of these checks supports the validity of their approach before applying it to more complex jet data. p. 0

SURF Method Application and Results

The SURF method allows exact NP tests even when the real data likelihood is intractable by training the target GPT model on samples from a tractable EPiC-FM surrogate. In Table II, they show that for continuous jets, the NP-optimal classifier AUC derived from EPiC-FM is close to state-of-the-art classifier performance. p. 0 For bin-smeared jets, they observe the same behavior previously identified for JetClass: bin-smeared EPiC-FM jets are nearly perfectly separable from their continuous counterparts. p. 4 The SURF method confirms that the GPT likelihood ratios dramatically overstate the true top vs. p. 8 QCD separation encoded in the EPiC-FM surrogate. p. 0

Final Summary of Findings

The consistent picture between JetClass and the EPiC-FM surrogate reference, particularly the concurrence of three ROC curves—classifiers trained on samples from JetClass and EPiC-FM samples, and the NPoptimal likelihood ratio from EPiC-FM—points to the fundamental limit of top tagging being not far away from state-of-the-art classifiers. The GPT likelihood ratios reflect artifacts of the generative process rather than performance limitations of the jet taggers. p. 9 Such highfrequency modes can be exploited by exact NP tests but are difficult for neural-network classifiers to learn, given their well-known low-frequency spectral bias. p. 9 The consistent picture suggests that the fundamental limit is not far away from state-of-the-art classifiers. p. 1

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Surrogate Reference (SURF) Method Implementation: The system can perform exact Neyman-Pearson tests by training a target model on samples from another tractable surrogate, allowing for exact Neyman–Pearson tests even when the true data likelihood is unknown.

  2. GPT Model Artifact Detection: The system can identify unphysical artifacts in generative models, specifically showing that GPT models exhibit signs of overfitting to their training data, which produces likelihood ratios that are inherently difficult for classifiers to reproduce.

  3. Discriminative Power Analysis: The improved system can differentiate between physical separation and artificial inflation by comparing the ROC curves derived from GPT likelihoods against those from the EPiC-FM surrogate reference, showing that the NP ROC curve derived from the GPT likelihoods far exceeds that of the EPiC-FM surrogate itself.

  4. Overfitting Mitigation: The system can detect overfitting in generative models by monitoring loss curves, noting that GPT jets overfit almost immediately (increasing gap between train vs. val losses), suggesting that models reducing to a sum of (smeared out) delta functions are the source of inflated separation.

  5. Discretization Fidelity Assessment: The system can quantify how discretization affects fidelity by showing that the GPT generated jets are faithful only to their discretized references, while being nearly perfectly separable from the original continuous jets, highlighting that discretization can introduce significant offmanifold effects.

Sources

Related papers