Correlated Systematic Uncertainties and Errors-on-Errors in Measurement Combinations: Methodology and Application to the 7-8 TeV ATLAS-CMS Top Quark Mass Combination

arXiv:2407.05322 · hep-ex · Submitted 2024-07-07 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: High Energy Physics: Experiment. The day in high-energy experiment: detectors, searches and limits.

Kai: We're looking at the paper: "Correlated Systematic Uncertainties and Errors-on-Errors in Measurement Combinations".

Mira: Here's the gist of the paper. The Gamma Variance Model (GVM) provides a statistical framework to handle uncertainties in systematic errors—informally called errors-on-errors—which is particularly useful when combining results from several measurements, such as the 7–8 TeV ATLAS-CMS top quark mass combination.

Kai: The paper introduces the Gamma Variance Model (GVM) as a statistical model incorporating uncertainties in the assignment of systematic errors.

Mira: This model allows for the treatment of both types of errors-on-errors, including those arising from ambiguities in defining a "1-sigma" error and those stemming from the estimation procedures themselves.

Kai: The GVM extends the likelihood function by treating the systematic variance as an adjustable parameter, denoted as vs, which is modeled as a gamma-distributed random variable.

Title and authors: Kai: So we're looking at this paper today, "Correlated Systematic Uncertainties and Errors-on-Errors in Measurement Combinations: Methodology and Application to the seven-eight TeV ATLAS-CMS Top Quark Mass Combination" <ref:2407.05322#pg1,Application to the 7-8 TeV ATLAS-CMS Top Quark Mass Combination>. Mira, what's the main story here?

Mira: It’s essentially about how we handle those tricky errors that aren't just simple plus or minus one. It tackles how we combine measurements when those systematic uncertainties are actually correlated in a more complicated way.

Lev: So if I had to put it simply, the paper is building a statistical framework, the Gamma Variance Model, to deal with uncertainties in assigning those systematic errors.

Kai: Right, so it’s about making sure that when we combine results from different places—like ATLAS and CMS—we don't just treat all those systematic errors as perfectly known constants.

Mira: Exactly. It introduces this model where the variance itself becomes a random variable, modeled as a gamma distribution, which lets us adjust how much we trust each systematic assignment.

The paper's summary: Kai: So what does the paper actually do? It sets up this Gamma Variance Model to treat those errors-on-errors, which are the uncertainties in error estimation itself.

Mira: They show a general relationship between two ways of doing things—the Best Linear Unbiased Estimator method and what they call the nuisance-parameter method.

Lev: And they generalize that nuisance-parameter approach so it can handle these errors-on-errors, which is crucial because standard methods often get stuck when correlations are non-trivial.

Kai: They then derive analytical formulas for figuring out the central values of things, the confidence intervals, and even the goodness-of-fit under these new conditions.

Mira: The application they use is to this specific seven–eight TeV ATLAS-CMS top quark mass combination, looking at how those uncertain systematics affect the final result <ref:2407.05322#pg1,7–8 TeV ATLAS-CMS top quark mass combination>.

Lev: And they find something interesting about that combination: it shows that the central value is pretty robust to having those systematic errors be uncertain one at a time.

The paper's improvements: Kai: So what are the specific improvements they suggest over how we usually do these kinds of combinations?

Mira: The big improvement is incorporating those errors-on-errors directly into the model, which allows the central value to become less sensitive to discrepancies between different input data values.

Lev: They also show that when you add a fictitious outlier measurement, within the Gamma Variance Model framework, that outlier gets assigned lesser weight as the error-on-error parameter increases.

Kai: That means if we have a weird data point, it doesn't totally wreck our result; it just gets down-weighted based on how uncertain we are about our own error assignments.

Mira: And they also show that the size of the confidence interval itself becomes sensitive to the internal consistency of your input data, meaning it inflates when measurements don't line up perfectly.

Conclusion: Kai: So wrapping this up, what's the main point about this work on "Correlated Systematic Uncertainties and Errors-on-Errors in Measurement Combinations: Methodology and Application to the seven-eight TeV ATLAS-CMS Top Quark Mass Combination" <ref:2407.05322#pg1,Application to the 7-8 TeV ATLAS-CMS Top Quark Mass Combination>?

Mira: The core message is that for combinations like this top quark mass combination, treating those systematic uncertainties as having errors within their errors makes the central value very stable.

Lev: It’s about quantifying how much uncertainty comes from the input data not being fully compatible, which we see reflected in a larger confidence interval when those discrepancies are present.

Kai: So essentially, this framework is a general procedure for figuring out which systematic uncertainties you actually need to worry about when they have associated errors-on-errors.

Mira: The study concludes that the ATLAS-CMS mt combination is robust against these types of uncertainties, and it’s a useful tool for future combinations where those correlations are non-trivial.

Lev: For me, it shows us how to model the overhead of running this on real hardware because we can see how much uncertainty is added by these complex error structures.

Kai: That covers what we have here. We'll keep an eye on how this Gamma Variance Model helps us build better measurements for the next generation of experiments.

Enzo Canonero, Glen Cowan

Royal Holloway, University of London

hep-ex

Submitted: 2024-07-07

Updated: 2025-07-03

Comments: 19 pages, 5 figures

Journal ref: Eur. Phys. J. C 85, 156 (2025)

DOI: 10.1140/epjc/s10052-025-13884-w

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 74/100

The gist: The Gamma Variance Model (GVM) provides a statistical framework to handle uncertainties in systematic errors—informally called errors-on-errors—which is particularly useful when combining results…

Key concepts

Gamma Variance Model (GVM)
A statistical framework that models uncertainties in systematic errors by treating the systematic variance as a random variable following a gamma distribution. This allows researchers to account for 'errors-on-errors,' which are uncertainties in how systematic errors themselves are assigned, rather than just fixed values.
Best Linear Unbiased Estimator (BLUE) vs. Nuisance Parameter Method
The paper establishes a general link between two methods for estimating results when systematic uncertainties are correlated. It shows how to extend the nuisance parameter method to include errors-on-errors, providing analytical formulas for calculating central values and confidence intervals under complex correlation conditions.
Errors-on-Errors
These are uncertainties arising from errors in assigning systematic error values, often related to ambiguities in defining a '1-sigma' error or estimation procedures. The GVM specifically models these as random variables, allowing the analysis to account for how uncertainty in the error assignment affects the final result.
Robustness and Outlier Sensitivity
The GVM demonstrates that while the central value of a measurement combination is stable against uncertain systematic errors, its confidence interval size is sensitive to internal data consistency. When an outlier is added, its influence on the combination decreases as the error-on-error parameter increases.

Terminology

Summary

The Gamma Variance Model (GVM) provides a statistical framework to handle uncertainties in systematic errors—informally called errors-on-errors—which is particularly useful when combining results from several measurements, such as the 7–8 TeV ATLAS-CMS top quark mass combination. p. 0

Where the paper says it

Ha Top quarks Keywords: Gamma Variance Model, errors-on-errors, combining measurements, correlated systematics, nuisance parameters, Bartlett correction, ATLAS-CMS top-mass combination 1 Introduction In Particle Physics analyses such as those at the Large Hadron Collider (LHC), many results are already dominated by systematic uncertainties or will be shortly. In this context, uncertainties in the assigned values of systematic errors themselves – informally called errors-on-errors – become increasingly important.

Methodology of Error Modeling

The paper introduces the Gamma Variance Model (GVM) as a statistical model incorporating uncertainties in the assignment of systematic errors. This model allows for the treatment of both types of errors-on-errors, including those arising from ambiguities in defining a 1-sigma error and those stemming from the estimation procedures themselves. The GVM extends the likelihood function by treating the systematic variance as an adjustable parameter, denoted as vs, which is modeled as a gamma-distributed random variable. p. 3

Where the paper says it

(13) This model extends Eq. (3) by adding, in the last line, terms associated with treating σ 2 us as an adjustable parameter of the model and its estimate vs as gamma distributed.

Equivalence of Estimation Methods

The paper establishes a general relation between the Best Linear Unbiased Estimator (BLUE) method and the nuisance-parameter method when correlations induced by systematic uncertainties are non-trivial. It demonstrates how to generalize the nuisance-parameter approach to include errors-on-errors and derives analytical formulas for estimating central values, confidence intervals, and goodness-of-fit under these conditions. p. 0

Where the paper says it

, not ±1 or 0), and we then generalise the nuisance-parameter approach to include errors-on-errors. We then present analytical formulas for estimating central values, confidence intervals, and goodness-of-fit when errors-on-errors are incorporated into the statistical model.

The equivalence between the two methods is shown when auxiliary measurements are set to zero. p. 2
Where the paper says it

(3) is used instead. In fact, the two methods are equivalent when the auxiliary measurements are set to zero, us = 0.

Application to Top Quark Mass Combination

The GVM is applied to the 7–8 TeV ATLAS-CMS top quark mass combination. The study investigates the impact on the combination’s central value and confidence interval when the largest sources of systematic uncertainties are treated as uncertain, one at a time. p. 0

Where the paper says it

Specifically, we investigate the impact on the combination’s central value and confidence interval when the largest sources of systematic uncertainties are treated as uncertain. We present the results by considering the largest systematic uncertainties as uncertain, one at a time, and we vary their associated error-on-error parameters.

The results show that the central value of the combination is quite robust to the presence of uncertain systematic errors. p. 1
Where the paper says it

This example illustrates the behavior of the GVM in scenarios where the inputs of a combination are not entirely internally compatible. Unlike a standard combination, incorporating errors-on-errors makes the central value of the combination, here taken as the maximum likelihood estimator for the parameter of interest, more robust to discrepancies between input data values.

Sensitivity to Internal Consistency and Outliers

The GVM exhibits a key feature where its sensitivity to internal consistency of the input data becomes relevant when a fictitious measurement is added as an outlier. The central value of the combination becomes less sensitive to outliers, while the presence of discrepancies results in an increase in the confidence interval. p. 1

Where the paper says it

Specifically, the central value becomes less sensitive to outliers. At the same time, the presence of discrepancies results in an increase in the confidence interval for the parameter of interest.

This demonstrates that the size of the confidence interval is sensitive to internal consistency of the input data. p. 0
Where the paper says it

To illustrate the properties of the Gamma Variance Model, we apply it to the 7–8 TeV ATLAS-CMS top quark mass combination. We also explore a hypothetical scenario by artificially adding a fictitious measurement as an outlier to the combination, illustrating a key feature of the Gamma Variance Model — its sensitivity to the internal consistency of the input data — which could become relevant for future combinations.

Analytical Formulas for Implementation

The paper provides a set of analytical formulas designed to simplify the practical implementation of uncertain systematic errors. These formulas are useful for reducing the computational burden in complex analyses. p. 5

Where the paper says it

(15) and (18) prevent exact analytical maximization of the GVM loglikelihoods with respect to µ and θ, as well as the computation of confidence intervals for µ and the evaluation of goodness-of-fit for the observed data. Although these tasks can always be performed using numerical techniques, analytical formulas are particularly useful for reducing the computational burden that can arise in complex analyses.

The perturbative analytical methods used to derive results proved to be highly precise, even when the conditions for applying perturbative techniques were not fully met.

Conclusions on Robustness

The study concludes that the ATLAS-CMS mt combination is robust, with the central value remaining very stable across a broad spectrum of assumed values for errors-on-errors. The confidence interval remains stable as well, though it exhibits up to a 10% increase when the b-JES systematic uncertainty is considered uncertain. p. 1

Where the paper says it

At the same time, the presence of discrepancies results in an increase in the confidence interval for the parameter of interest. This increase reflects the uncertainty stemming from the fact that the measurements in the dataset are not entirely compatible.

The GVM serves as a general procedure to identify the systematic uncertainties to which a combination is sensitive when they have associated errors-on-errors. p. 0
Where the paper says it

We present the results by considering the largest systematic uncertainties as uncertain, one at a time, and we vary their associated error-on-error parameters. This approach serves as a general method to identify the systematic uncertainties to which a combination is sensitive when errors-on-errors are considered.

Summary of Key Findings

The Gamma Variance Model (GVM) provides a statistical framework to handle uncertainties in the assignment of systematic errors, informally called errors-on-errors. The model is of particular use in analyses that combine the results of several measurements. p. 0

Where the paper says it

The Gamma Variance Model is a statistical model that incorporates uncertainties in the assignment of systematic errors (informally called errors-on-errors). The model is of particular use in analyses that combine the results of several measurements.

The authors obtain a general relation between the BLUE and nuisance-parameter methods when correlations induced by systematic uncertainties are non-trivial. p. 0
Where the paper says it

In the past, combinations have been carried out using two alternative approaches: the Best Linear Unbiased Estimator (BLUE) method or what we will call the nuisance-parameter method. In this paper, we obtain a general relation between the BLUE and nuisanceparameter methods when the correlations induced by systematic uncertainties are non-trivial (i.

They present analytical formulas for estimating central values, confidence intervals, and goodness-of-fit when errors-on-errors are incorporated into the statistical model. p. 0
Where the paper says it

, not ±1 or 0), and we then generalise the nuisance-parameter approach to include errors-on-errors. We then present analytical formulas for estimating central values, confidence intervals, and goodness-of-fit when errors-on-errors are incorporated into the statistical model.

The application to the 7–8 TeV ATLAS-CMS top quark mass combination shows that the central value is robust to uncertain systematic errors. p. 0
Where the paper says it

This framework is then applied to the 7-8 TeV ATLAS-CMS top quark mass combination [7]. Specifically, we investigate the impact on the combination’s central value and confidence interval when the largest sources of systematic uncertainties are treated as uncertain.

Furthermore, the GVM illustrates that internal inconsistency in input data leads to an increase in the confidence interval. p. 1
Where the paper says it

This example illustrates the behavior of the GVM in scenarios where the inputs of a combination are not entirely internally compatible. Unlike a standard combination, incorporating errors-on-errors makes the central value of the combination, here taken as the maximum likelihood estimator for the parameter of interest, more robust to discrepancies between input data values.

The framework is shown to be effective even when dealing with non-trivial correlations and incorporates a mechanism where the outlier is assigned lesser weight within the combination as the error-on-error parameter increases. The overall conclusion is that the ATLAS-CMS mt combination is robust against these types of uncertainties. p. 0
Where the paper says it

05322v4 [hep-ex] 3 Jul 2025 EPJ manuscript No. (will be inserted by the editor) Correlated Systematic Uncertainties and Errors-on-Errors in Measurement Combinations with an Application to the 7 - 8 TeV ATLAS-CMS Top Quark Mass Combination Enzo Canonero and Glen Cowan Physics Department, Royal Holloway, University of London, U.

Pruning redundant reasoning tokens

The authors argue that the assumption of independent gamma distributions for estimates of variances is a plausible model that arises, e.g., if nuisance parameters are estimated through an average of Gaussian-distributed measurements. p. 3

Where the paper says it

Some studies carried out in connection with [1] of the log-normal distribution or a gamma distribution for the estimated standard deviations led to qualitatively similar results, but were not pursued further when the following reasons to use gammadistributed v became apparent. First, suppose that us itself is found as an average of some number of other Gaussian-distributed values us,i, i = 1, .

This assumption does not necessarily hold but the model should nevertheless provide more realistic inference than if one were to treat assigned variances as exactly known. p. 3
Where the paper says it

In the GVM, the value that would typically be assigned to σ 2 us in a standard combination is treated as an estimate of this parameter and is denoted as vs. One should keep in mind that the choice of independent gamma distributions for the estimates v is not expected to produce a perfect model, rather an approximation that represents a potentially significant improvement over the assumption of exactly specified variances.

How the training signal is built

The profile likelihood is maximized with respect to the NPs using a recursive approach, expressing profiled values as ˆˆθs = ˆˆθ(0)s + ε 2s ˆˆθ(1)s + ε 4s ˆˆθ(2)s + · · ·. The first term in this series is given by ˆˆθ(0)s = X M p=1 C(0)-1 sp [X N i,j=1 Γ p j V-1 ij (y j - µ) + u p v p]. p. 5

Where the paper says it

(21) and equating terms of equal powers of ε 2 s , we obtain a recursive set of equations for the coefficients ˆˆθ (n) s . The first term is given by ˆˆθ (0) s = X M p=1 C (0)−1 sp   X N i,j=1 Γ p i V −1 ij (yj − µ) + up vp   , (23) where the matrix C (0) sp is defined as C (0) sp = X N i,j=1 Γ s i V −1 ij Γ p j + δsp vs (24) and δsp is the Kronecker delta.

Where the gains come from

The Bartlett correction is used to adjust the likelihood ratio by a scaling factor so that its distribution more closely matches a χ squared distribution, reducing error terms to order ε 4s, while also ensuring that p-values remain accurate up to terms of the same order. This correction is applied to the GOF statistic and confidence intervals in the GVM likelihoods. p. 6

Where the paper says it

However, the GVM log-likelihoods in Eqs. (13) and (16) contain non-quadratic terms, causing deviations from the χ 2 distribution by terms of order ε 2 s , which affect the precision of confidence intervals.

The perturbative expansion is highly effective when the condition 2ε 2s vs X N i,j=1 u i s - ˆˆθf(s,i) ρ(s)−1 ij u j s - ˆˆθf(s,j) < 1 is satisfied. p. 7
Where the paper says it

These ε 2 s approximations are highly effective when the condition specified in Eq. (29) is satisfied.

The GVM framework is shown to be applicable even when the condition for applying perturbative techniques is not fully met by using numerical maximization instead. p. 6
Where the paper says it

However, when this occurs, it does not mean that the GVM is inapplicable; rather, it suggests that numerical maximization should be used instead. Using the perturbative solutions for ˆˆθs, the log-likelihood can be optimized with respect to µ to determine its best estimate.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improved Robust Parameter Estimation: The system can estimate central values that are more robust to discrepancies between input data values by incorporating errors-on-errors through the Gamma Variance Model, which makes the combination less sensitive to outliers.

  2. Non-Trivial Correlation Handling: The AI system can perform measurements combinations where systematic uncertainties induce correlations that are not simply plus or minus one, by using an extended nuisance parameter approach that introduces distinct NPs for each systematic source.

  3. Uncertainty Quantification via Scaling: The system can quantify the impact of uncertain systematic errors on a combination by treating the variance sigma squared us as an adjustable parameter and its estimate vs as gamma distributed, allowing it to model how the systematic error induced by source s in measurement yi scales.

  4. Outlier Sensitivity Mitigation: When an outlier is added to the dataset, the system can demonstrate that within the GVM framework as the error-on-error parameter increases, the outlier is assigned lesser weight within the combination, causing a shift back towards original results.

  5. Confidence Interval Inflation Modeling: The AI can predict that the size of the confidence interval is sensitive to the internal consistency of the input data, specifically showing how it inflates when errors-on-errors are considered and data are less compatible.

Abstract

The Gamma Variance Model (GVM) is a statistical model that incorporates uncertainties in the assignment of systematic errors (informally called errors-on-errors). The model is of particular use in analyses that combine the results of several measurements. In the past, combinations have been carried out using two alternative approaches: the Best Linear Unbiased Estimator (BLUE) method or what we will call the nuisance-parameter method. In this paper we derive useful relations that allow one to connect the BLUE and nuisance-parameter methods when the correlations induced by systematic uncertainties are non-trivial (1, -1 or 0), and we generalise the nuisance-parameter approach to include errors-on-errors. We then illustrate some of the properties of the GVM by applying it to the 7-8 TeV ATLAS-CMS top quark mass combination. We present results by considering the largest systematic uncertainties as uncertain, one at a time, and we vary their associated error-on-error parameters. This procedure is useful for identifying the systematic uncertainties to which a combination is sensitive when they are themselves uncertain. We also explore the hypothetical scenario of including an outlier in the combination, which could become relevant for future combinations, by artificially adding a fictitious measurement to it. This example highlights a key feature of the GVM: its sensitivity to the internal consistency of the input data.

Sources

Related papers