Defining a Minimum Resolution for Unbinned Analyses
summary
The gist
Collider analyses combine rigorous statistical techniques with state-of-the-art Machine Learning models to estimate signal parameters from unbinned data The Minimum Resolution Likelihood (MRL) method…
In short
The Minimum Resolution Likelihood (MRL) method improves signal estimation in unbinned analyses by defining a Fiducial Signal Region. This technique turns systematic model uncertainties into statistical uncertainties by finding a minimum likelihood threshold where the background model is no longer trusted, leading to less biased results despite conservative cuts.
Key concepts
- Model Misspecification
- This occurs when the mathematical model used to describe the background events is incorrect or incomplete. If the model underestimates event density in areas where signal and background overlap, it can lead to an overestimation of how many signal events are present in the data, causing biased estimates.
- Minimum Resolution Likelihood (Lmin)
- Lmin is a critical likelihood value determined by finding the point where the difference between expected event fractions from toys (simulations) and those from the true background model is maximized. This point marks a 'crossing' where the background model shifts from under-predicting to over-predicting likelihoods.
- Fiducial Signal Region
- This is a specific region in feature space defined by requiring that the probability density of observing an event, given the background model, must be greater than or equal to Lmin. Events falling outside this region are excluded because the model's reliability in those areas is deemed too low.
Terminology used across episodes
This episode discusses
- Defining a Minimum Resolution for Unbinned Analyses · Paper Radio
- Search for pair production of Higgs bosons in the b b barb b bar final state using proton-proton collisions at sqrt s = 13 TeV with the ATLAS detector
- Search for ZZ and ZH production in the b b barb b bar final state using proton-proton collisions at sqrt s = 13 TeV
- Improved results on Higgs boson pair production in the 4b final state
- Classifying Anomalies THrough Outer Density Estimation (CATHODE)
- CURTAINs for your Sliding Window: Constructing Unobserved Regions by Transforming Adjacent Intervals
- FETA: Flow-Enhanced Transportation for Anomaly Detection
- Resonant anomaly detection without background sculpting
- CURTAINs Flows For Flows: Constructing Unobserved Regions with Maximum Likelihood Estimation
- Residual ANODE
- Handling uncertainties in background shapes: the discrete profiling method
- Uncertainty in Physics and AI: Taxonomy, Quantification, and Validation
- Constraining Effective Field Theories with Machine Learning
- A Guide to Constraining Effective Field Theories with Machine Learning
- Unbinned multivariate observables for global SMEFT analyses from machine learning
- Constraining CP-violation in the Higgs-top-quark interaction using machine-learning-based inference
- Simulation-based inference in the search for CP violation in leptonic WH production
- Refinable modeling for unbinned SMEFT analyses
- From Optimal Observables to Machine Learning: an Effective-Field-Theory Analysis of e+e- to W+W- at Future Lepton Colliders
- Constraining the Higgs Potential with Neural Simulation-based Inference for Di-Higgs Production
- Unbinned inclusive cross-section measurements with machine-learned systematic uncertainties
The paper
Defining a Minimum Resolution for Unbinned Analyses · Read on arXiv
International Center for Advanced Studies (ICAS) · ICIFI-CONICET · UNSAM
Collider analyses combine rigorous statistical techniques with state-of-the-art Machine Learning models. However, when the latter are used directly to estimate the likelihood function of the background, hard to quantify systematic effects may bias the estimation of the relevant signal parameters. To address this problem, we present the Minimum Resolution Likelihood (MRL) method, which defines a Fiducial Signal Region that aims to turn systematic effects into statistical uncertainties. We show with examples that the resulting signal strength estimation has a reduced bias, with the effect depending on the sample size and type of signal considered. We consider both toy examples and a realistic application based on the HI-SIGMA technique applied to di-Higgs searches.
Transcript
Introduction to the show: ident: High Energy Physics: Phenomenology. The day in particle phenomenology: models, signatures and constraints.
Kai: I'm Kai.
Mira: And I'm Mira, here with Lev.
Kai: We're looking at the paper: "Defining a Minimum Resolution for Unbinned Analyses".
Mira: Here's the gist of the paper. Collider analyses combine rigorous statistical techniques with state-of-the-art Machine Learning models to estimate signal parameters from unbinned data The Minimum Resolution Likelihood (MRL) method defines a Fiducial Signal Region that effectively turns systematic effects into statistical uncertainties The Problem of Model Misspecification Traditional data-driven background estimation techniques are confined to few dimensions in feature space, which restricts the power of the analysis The risk of bias and uncertainty underestimation increases with high-dimensional data models used in unbinned analyses One possible failure mode is the overestimation of the signal-to-background likelihood ratio in regions where the signal is present, which can lead to a consequent overestimation of the number of signal events in the sample This occurs when a background model underestimates the probability density in regions where signal and background overlap due to training artifacts or architectural biases How the Training Signal is Built The MRL method involves defining a Calibration Region where there is only the background, and using this to define a minimum resolution in a smartly chosen feature below which the model is not trusted This leads to defining a Fiducial Signal Region given by pb(xθ) ≥ Lmin The extended likelihood is then modified to include efficiencies estimated at the Calibration Region, additional signal simulations, and using toys Where the Gains Come From The method effectively turns systematic uncertainties over the background model into statistical uncertainties due to the more limited sample size of the data sample The definition of Lmin is based on finding a critical point where the difference between toys and Calibration Region expectation values is maximal, defined as Lmin ≡ arg max Lc ∆(Lc) This choice of Lmin is such that on average the two likelihoods match, corresponding to the “crossing” region where we go from under- to over-prediction Application in Realistic Searches The MRL method was applied to a realistic di-Higgs study based on the HI-SIGMA technique introduced in Ref.
26: In this application, the results show that the inference with no masking shows a similar distribution of fraction differences as the toy examples, but the resulting cut is too conservative and actually captures the inverse crossing than intended The performance of MRL with and without masking shows how masking, which was introduced original in Ref.
26: to reduce the impact of mismodelling, is actually a source of bias Outlook The method as presented here is conservative by definition, and thus shows areas for improvement One could be to avoid the hard cuts of a Fiducial Signal Region definition and instead improve on the masking introduced in HI-SIGMA by combining it with the estimation of the remaining likelihood volume below the threshold MRL is envisioned to work generally, provided the set of conditions enumerated in this work are met The method can also be used as a diagnostic when background emulators are considered, such as ML-models for Simulation Based Inference analyses MRL can be useful as a diagnostic given its malleability and compatibility with the treatment of other systematic uncertainties The scaling of Lmin with training dataset size can help characterize the scaling laws of the relevant generative models Page 1 --- Collider analyses combine rigorous statistical techniques with state-of-the-art Machine Learning models to estimate signal parameters from unbinned data The Minimum Resolution Likelihood (MRL) method defines a Fiducial Signal Region that effectively turns systematic effects into statistical uncertainties Outlook The method as presented here is conservative by definition, and thus shows areas for improvement One could be to avoid the hard cuts of a Fiducial Signal Region definition and instead improve on the masking introduced in HI-SIGMA by combining it with the estimation of the remaining likelihood volume below the threshold MRL is envisioned to work generally, provided the set of conditions enumerated in this work are met The method can also be used as a diagnostic when background emulators are considered, such as ML-models for Simulation Based Inference analyses MRL can be useful as a diagnostic given its malleability and compatibility with the treatment of other systematic uncertainties Page 3 --- As a starting point, we define the usual extended likelihood considered in most collider analyses
27: for N total observed events in the Signal Region, with observed features The parameter of interest is µs, λs and λb are the expected signal and background rates and all others are nuisance parameters, for which we add a constraint term g that stands for any additional measurements or theoretical systematic uncertainties that constrain said nuisance parameters We then define the usual test statistic derived from the profiled log-likelihood, t(µs) = −2 ln L(D, µs, ˆµˆb, ˆˆθs, ˆˆθb) The Problem of Model Misspecification If our likelihood is incorrect, the test loses statistical power and worse, we will obtain biased estimates of µˆs and confidence intervals (µ min s, µ max s) without proper coverage We are particularly interested in the case where the bias in the background model leads to the overestimation of the signal-to-background likelihood ratio in the regions where the signal is present, with a consequent overestimation of the number of signal events in the sample How The Minimum Likelihood is Determined We can compute the expected fraction of events above a certain likelihood value, f(Lc model) = Z Lmax Lc dL p(L model) = Ex∼p(xmodel)
Θ(p(xθb) − Lc): We define a minimum resolution likelihood below which we no longer trust our model Lmin ≡ arg max Lc f(Lc θb) − f(Lc true b) ≡ arg max Lc ∆(Lc) This definition encodes the crossing between the regions where the background model underestimates and overestimates the likelihood surfaces The Fiducial Signal Region The definition of Lmin allows us to define a Fiducial Signal Region (FSR) given by pb(xθ) ≥ Lmin If the signal does not lie completely below this likelihood cut, then the model will find the signal with reduced significance but with less bias Toy Examples To illustrate the method, we explore two toy examples, chosen for their simplicity and their ability to highlight the power and limitations of the method In these examples, shown in sections 3.1 and 3.2, we define the true signal and background distributions and a background model in terms of parametric functions Page 4 --- The simplest case is when the signal and background are both gaussians, and we mismodel the standard deviation of the background To capture the range of applications of the method, we chose the parameters so that the signal-to-background is overestimated for values of x where the signal is present but also the difference is small enough that the definition of the fiducial region does not remove all the signal The Lmin estimation, shown in Fig.
Kai: 3, highlights how there is a clear global maximum in the deviation between the calibration and toy region, that it corresponds to the crossing between background over and underestimation and that its value depends on the quality of the background model Toy Examples In each pseudo-experiment, a new fit with the background model is performed after using the estimated Lmin to define a Fiducial Signal Region The size of the Calibration Region and the toy dataset is five times larger than the Signal Region, to avoid statistical fluctuations and disentangle the problems of estimating the Fiducial Signal Region from its impact on the fit itself Where The Gains Come From The MRL method greatly improves the results, yielding less biased estimations at the expense of slightly larger errors This is further reinforced by looking at the coverage in the right plot, where we observe how the coverage becomes much closer to the desired value, although some undercoverage remains Toy Examples Although illustrative, a single pseudo-experiment is not sufficient to quantify the effectiveness of the method We show the results in Fig.
Title and authors: Kai: So, we're looking at this paper called "Defining a Minimum Resolution for Unbinned Analyses". It’s about how to handle those tricky systematic uncertainties when we use Machine Learning models in collider analyses to find signals in unbinned data.
Mira: Right, it tackles the problem where using these models directly to estimate the background likelihood function can actually introduce bias into our signal parameter estimates if we aren't careful. It’s not just about getting better numbers, it’s about making sure those systematic effects translate into statistical uncertainties instead of skewing our results.
Lev: From my side, it sounds like a way to tame some of the heavy model dependence that comes with using complex background emulators for real hardware. If the model is wrong, you get biased estimates and bad confidence intervals, which is something we really have to avoid when we’re trying to build systems for real physics.
Kai: Exactly. The paper introduces this Minimum Resolution Likelihood method which basically defines a specific Fiducial Signal Region based on a resolution limit derived from the model's performance itself. This is designed to turn those systematic issues into something more manageable—statistical uncertainties dictated by how much data we actually have.
Mira: And what’s really clever about the core idea is how they figure out this minimum likelihood value, which they call Lmin. It isn't just picking some arbitrary cut; it’s based on finding a specific point where the difference between what their toy examples predict and what the true background predicts is at its maximum.
Lev: That sounds like they are looking for that sweet spot where the model stops being trustworthy, which makes sense because we need a clear boundary for when we should stop trusting an emulator. But I wonder how robust that maximum point is to changes in the underlying physics of the background itself?
Kai: Well, they use this Lmin to define a Fiducial Signal Region where the probability density of observing something above that likelihood value is greater than or equal to Lmin. If your signal doesn't fall entirely below that line, you still keep some signal events but with less bias from the model errors.
Title and authors: Mira: They also mention modifying the extended likelihood function by including efficiencies calculated in a calibration region, plus their own signal simulations and toys, and then using this Lmin cut to define that modified efficiency term for those toys. It’s an integrated approach to incorporating all those different pieces of information.
Lev: So they are essentially building a new way to calculate the expected fraction of events above a certain likelihood value by comparing the toy distributions against what they expect from the true background, using this Lmin criterion as the key threshold between under- and over-prediction regions. That sounds like it would be very computationally intensive, though.
Kai: It is, but they stress that this process effectively turns systematic uncertainties over the background model into statistical uncertainties because you’re operating with a more limited sample size of the data itself when you define that Fiducial Signal Region. They show examples using Gaussian cases and then move on to two-dimensional exponential and Gaussian distributions.
Mira: The toy examples they use, especially the 2D exponential and Gaussian one, are important because they illustrate how this method works in practice when the signal-to-background ratio is overestimated for values where the signal actually is present <ref:2606.28498#pg1>. It shows how Lmin captures that crossing region where the model transitions from underestimating to overestimating.
Lev: I saw them looking at a benchmark with three different tests, including one with an unlikely signal, and it showed that if you set the Fiducial Signal Region definition too restrictively based on Lmin, the method can become over-conservative and actually capture the inverse crossing than they intended. That suggests we need to be careful about how strict we make that initial cut.
Kai: It’s a caution, but it also shows where there is room for improvement in their approach. The paper itself admits that its definition of a Fiducial Signal Region can be conservative by definition, which points toward future work needing to avoid those hard cuts entirely.
Mira: So the suggested improvement is not to just change the cut, but maybe to refine the masking introduced in related techniques, like HI-SIGMA, by combining it with estimating the remaining likelihood volume below that threshold Lmin. It’s about smoothing out those sharp boundaries they set up.
Title and authors: Lev: Combining a mask with an estimation of the remaining likelihood volume sounds like a more nuanced way to handle things than just drawing a hard line on the graph. It’s about making it less of an arbitrary decision and more grounded in the actual data distribution near that threshold.
Kai: Beyond just this specific method, they suggest that this approach can be used as a diagnostic tool when you are considering background emulators, for instance, if you’re using Machine Learning models for simulation based inference analyses. It can help you probe how well those emulators are performing and where their limitations lie.
Mira: And they also noted that the scaling of that minimum likelihood value Lmin with the size of the training dataset can be used to characterize the scaling laws of those relevant generative models, which is really interesting for understanding how these complex models behave as they get more data to learn from.
Lev: So, to wrap up on what we've heard about this paper, it’s a method that uses a minimum resolution likelihood Lmin derived from comparing toy data against true background expectations to define a Fiducial Signal Region and turn model systematic errors into statistical uncertainties. It improves the results by reducing bias but they admit the definition itself can be conservative, suggesting future work might focus on refining those cuts or combining masking with likelihood volume estimation.
Kai: That’s it for this paper, "Defining a Minimum Resolution for Unbinned Analyses". It really shows how we can use these statistical techniques and machine learning models to get more reliable parameter estimates in complex searches where the background modeling is hard.
Mira: It’s a solid piece of work because it moves beyond just using models as black boxes and gives us a way to quantify the model's trustworthiness directly within the analysis framework.
Lev: For real hardware, this means we have a more rigorous way to assess how much error we are actually introducing by relying on an emulator versus trying to model everything from scratch.
Kai: We’ll be looking at other papers next, but for now, this method of using Lmin to define the Signal Region is a clear tool for turning those systematic uncertainties into something we can quantify as statistical noise.
The paper's summary: Kai: So, this paper is all about taking those tricky model dependencies in unbinned analyses—the ones where your background emulator might be slightly off—and turning them into statistical uncertainties instead.
Mira: Exactly. They introduce this Minimum Resolution Likelihood method, which essentially sets a boundary based on how much the background model disagrees with what their toy simulations predict.
Kai: So, instead of just throwing out data points that look weird, they define a specific region in the data space—a Fiducial Signal Region—where the model is actually reliable.
Mira: And they figure out this critical value called Lmin by finding where the difference between those toy predictions and the true background predictions is at its maximum. It’s that sweet spot where the model switches from underestimating to overestimating things.
Kai: That sounds like a really clever way to use a likelihood ratio test, but you have to be careful about setting that Lmin cut, because they admit their definition can be conservative by default.
Mira: Right, they do acknowledge that the method as written is conservative. They’re suggesting that instead of just drawing those hard cuts on the Fiducial Signal Region, we could improve on the masking used in related techniques by combining it with estimating the remaining likelihood volume below that Lmin threshold.
Kai: So they’re hinting at a way to make that boundary smoother instead of making it a sharp line. It’s about integrating that masking idea with knowing how much likelihood space is left beneath your threshold.
Mira: And they see this not just as a fix for one specific analysis, but also as something useful when you're using other background emulators, like those machine learning models for simulation based inference analyses. It’s a diagnostic tool in itself.
Kai: That makes sense because if you can use it to check the emulator, you can see where its limitations are without needing a full-blown fit first.
Mira: Plus, they look at how Lmin scales with the size of the training dataset, which could help us figure out how those generative models behave as they get more data to learn from. It gives us insight into their scaling laws.
Kai: So, it’s this method that turns model uncertainty into statistical noise by defining a region where the model is trustworthy based on its own performance metrics derived from toy experiments.
Mira: And while it improves results by reducing bias, they are cautious because the definition itself is conservative, meaning you still have to be mindful of those hard cuts they impose.
Kai: It gives us a framework for making better decisions about where we trust our model in these complex unbinned searches.
The paper's improvements: Kai: So, this paper lays out some ways to get better than just using those hard cuts in the Fiducial Signal Region definition for this method.
Mira: Right, they’re suggesting you stop relying solely on that fixed Lmin value and instead improve the masking technique from related work, like HI-SIGMA.
Kai: So instead of a hard cut, maybe you combine that original masking with estimating the remaining likelihood volume below that threshold Lmin.
Mira: That sounds like a way to smooth out those sharp boundaries they introduced, making the transition between regions less abrupt. It’s about blending the model assessment with the actual data distribution near that limit.
Kai: So it’s not just changing one parameter in the method, but rethinking how you define that boundary entirely by blending different estimation strategies together.
Mira: And they also mention using this whole setup as a diagnostic tool when you are working with background emulators, especially those machine learning models for simulation based inference analyses.
Kai: If you can use it to check the emulator performance, that means you can probe where its limitations are directly within your analysis framework.
Mira: Plus, they also noted that the way Lmin scales with the size of the training dataset could help us characterize how those generative models actually behave as they learn more data. It gives a way to understand their scaling laws.
Kai: So this paper suggests two main paths forward: either refine those hard cuts, or adopt this hybrid masking and volume estimation approach for a more nuanced boundary definition.
Mira: The implication is that we might move away from relying on single-point definitions of model trust and toward a more continuous assessment of the likelihood surface around that critical point.
Kai: It means we're looking at how to keep the statistical power up while still being honest about the systematic uncertainties introduced by our background models.
Conclusion: Tom: So, to wrap up, this paper on "Defining a Minimum Resolution for Unbinned Analyses" is about using the minimum resolution likelihood method to turn systematic model errors into statistical uncertainties in collider analyses.
Kai: Yeah, it’s about creating that fiducial signal region based on how well the background model predicts things compared to what the toy examples show.
Mira: It gives us a concrete way to define trust in our background emulators by finding that critical point where their predictions start crossing over.
Lev: From an error correction standpoint, this means we have a more rigorous way to assess how much uncertainty we are actually introducing by relying on an emulator versus trying to model everything from scratch.
Kai: So it’s about making sure the analysis isn't biased because the background model was slightly wrong in its assumptions.
Mira: Exactly. And they show that this method can be used as a diagnostic tool for those machine learning models, checking where they fall short without needing a full fit first.
Lev: I think that diagnostic capability is what makes it useful for real hardware runs, because you don't want to waste time building an emulator if you already know it’s fundamentally flawed in certain regions.
Kai: It really shows how we can use these statistical techniques and machine learning models to get more reliable parameter estimates in complex searches where the background modeling is tough.
Mira: It moves us toward a way of quantifying model trustworthiness directly within the analysis framework instead of treating the background emulator as a perfect black box.
Lev: For real hardware, this means we have a more rigorous way to assess how much error we are actually introducing by relying on an emulator versus trying to model everything from scratch.
Kai: So, it’s this method that turns model uncertainty into statistical noise by defining a region where the model is trustworthy based on its own performance metrics derived from toy experiments.
Mira: It moves us toward a way of quantifying model trustworthiness directly within the analysis framework instead of treating the background emulator as a perfect black box.
Lev: I think that diagnostic capability is what makes it useful for real hardware runs, because you don't want to waste time building an emulator if you already know it’s fundamentally flawed in certain regions.
Kai: So, to wrap up on this paper, "Defining a Minimum Resolution for Unbinned Analyses," we see a method that turns model errors into statistical noise by defining a region where the model is trustworthy based on its own performance metrics derived from toy experiments.
Mira: It gives us a concrete way to define trust in our background emulators by finding that critical point where their predictions start crossing over.
Lev: I think that diagnostic capability is what makes it useful for real hardware runs, because you don't want to waste time building an emulator if you already know it’s fundamentally flawed in certain regions.
Kai: It really shows how we can use these statistical techniques and machine learning models to get more reliable parameter estimates in complex searches where the background modeling is tough.
More episodes
- 2610.38463-Gauge-parameter cancellation and finite form factors in one-loop quark scattering in the refined Gribov-Zwanziger framework
- 2610.37198-Searches for new particle physics by methods of classical and neutrino astronomy
- 2511.15779-SURFing to the Fundamental Limit of Jet Tagging
- 2505.01492-Photo- and Hadrodisintegration constraints on massive relics decaying into neutrinos
- 2505.20389-Dispersion relation of the neutrino plasma: Unifying fast, slow, and collisional instabilities
- 2511.10731-Keeping It Renormalizable: Minimal Baryogenesis induced Asymmetric Dark Matter
- 2404.04021-Vacuum correlations of the stress-energy-momentum tensor with constituent quarks
- 2503.02931-Relativistic Navier-Stokes description of the quark-gluon plasma radial flow
- 2606.06105-Bulk viscosity of a binary mixture: the role of the intra-species interaction
- 2607.03343-First look at the evaluation of two-loop Feynman integrals for radiative return processes