On Focusing Statistical Power for Searches and Measurements in Particle Physics

summary

The gist

Particle physics experiments rely on the (generalised) likelihood ratio test (LRT) for searches and measurements…

In short

The paper introduces a Focused Test Statistic (FTS) to improve statistical power in particle physics searches and measurements. Instead of using the standard likelihood ratio test, the FTS allows analysts to focus statistical power on specific, physics-motivated regions of parameter space. This technique increases sensitivity in key measurements without compromising the validity of confidence intervals.

Key concepts

Generalized Likelihood Ratio Test (LRT)
The standard test used in particle physics searches and measurements to test hypotheses about a parameter ($\mu$). While optimal for simple tests, it may not provide the best power against all possible alternatives in complex scenarios.
Focused Test Statistic (FTS)
A new statistic defined as $T_f(D; \mu_0) = -2 \log p(D; \mu_0) / R_{\Theta} p(D; \mu) f(\mu)$. This allows the data analyst to direct the test's sensitivity toward specific regions of parameter space defined by a focus function $f(\mu).
Focus Function ($f(\mu)$)
A weighting function used in the FTS that defines which parts of the parameter space are prioritized for increased statistical power. A narrow focus leads to better gains around its center but can reduce power if the true parameter value is far from that center.
Coverage and Bias Trade-off
The FTS deliberately introduces a small technical bias in regions where the focus function has low weight to boost power. Crucially, this deliberate bias does not compromise the nominal coverage of confidence intervals across all parameter values.

Terminology used across episodes

This episode discusses

The paper

On Focusing Statistical Power for Searches and Measurements in Particle Physics · Read on arXiv

James Carzon, Aishik Ghosh, ¡ Rafael Izbicki, Ann B. Lee, Luca Masserano, Daniel Whiteson

Department of Statistics and Data Science, Carnegie Mellon University · Department of Physics and Astronomy, University of California, Irvine Department of Physics Division, Lawrence Berkeley National Laboratory School of Physics, Georgia Institute of Technology Federal University of Sao Carlos

Particle physics experiments rely on the (generalised) likelihood ratio test (LRT) for searches and measurements, which consist of composite hypothesis tests. However, this test is not guaranteed to be optimal, as the Neyman-Pearson lemma pertains only to simple hypothesis tests. Any choice of test statistic thus implicitly determines how statistical power varies across the parameter space. An improvement in the core statistical testing methodology for general settings with composite tests would have widespread ramifications across experiments. We discuss an alternate test statistic that provides the data analyzer an ability to focus the power of the test on physics-motivated regions of the parameter space. We demonstrate the improvement from this technique compared to the LRT on a Higgs toττ dataset simulated by the ATLAS experiment and a dark matter dataset inspired by the LZ experiment. We also employ machine learning to efficiently calibrate critical values for a family of tests, which are then inverted to obtain statistically valid confidence intervals.

DOI: 10.1140/epjc/s10052-026-16023-1

Transcript

Introduction to the show: ident: High Energy Physics: Phenomenology. The day in particle phenomenology: models, signatures and constraints.

Kai: We're looking at the paper: "On Focusing Statistical Power for Searches and Measurements in Particle Physics".

Mira: Here's the gist of the paper. Particle physics experiments rely on the (generalised) likelihood ratio test (LRT) for searches and measurements, which consist of composite hypothesis tests The authors present an alternate test statistic that provides the data analyzer an ability to focus the power of the test on physics-motivated regions of the parameter space This technique allows for focusing statistical power on specific regions of the parameter space without assuming simple hypotheses or large sample sizes while maintaining validity of confidence intervals The Statistical Framework Searches and measurements in particle physics are typically performed as hypothesis tests on the parameter µ of the likelihood p(D; µ) of datum D implied by the physics processes under study for some parameter space Θ The generalized likelihood ratio test (LRT) is optimal for simple-vssimple hypothesis tests but is not guaranteed to be optimal for more general settings with high power against all alternatives The authors propose a focused test statistic (FTS) of the form Tf (D; µ0) = −2 log p(D; µ0) R Θ p(D; µ)f(µ)dµ This statistic allows the data analyst to focus the power of a test or confidence procedure to specific regions of the parameter space, as defined by the focus or weight function f How Gains Are Achieved The FTS is optimal for average constraining power under the focus function, minimizing the expected interval length when data are averaged according to the focused marginal distribution pf (D) = Z Θ p(D; µ) f(µ) dµ The theorem shows that among all confidence sets that achieve at least 1 − α coverage for every µ ∈ Θ, the FTS construction is optimal for average constraining power under the focus function The FTS framework deliberately introduces a small amount of bias in parameter regions where the focus function has small weight to increase power while retaining nominal coverage for every µ ∈ Θ Experimental Applications The paper demonstrates the improvement of FTS over LRT in two case studies: 1.

Kai: A Higgs measurement of the coupling of Higgs bosons to tau leptons, where FTS with a narrow focus function yields 22% smaller 68% confidence intervals than LRS when µ∗ is at the SM point 2.

Mira: A search for weakly interacting massive particles (WIMPs) inspired by the LZ experiment, where FTS with a narrow focus function provides 65% smaller intervals when there is no WIMP signal The focus test statistic (FTS; Equation 3) can be applied to any existing particle physics analysis and holds the potential to improve sensitivity in key measurements across experiments The FTS framework gives the scientist control over the regions of maximum sensitivity, whether chosen after optimization studies on independent simulations or informed by independent previous measurements without affecting the validity of the confidence intervals The improved power of the FTS, enabled by the rapid building of confidence intervals, can enhance discovery potential and physics sensitivities across many experiments The proposed method provides a convenient ML tool that builds confidence intervals by inverting hypothesis tests reliably and efficiently Page 9 James Carzon et al.: On Focusing Statistical Power for Searches and Measurements in Particle Physics 9 The end result of the chosen focus function is raising the lower bound for searches for small values of µ This is because FTS maintains an advantage over LRS when µ∗ ≠ 0, yielding higher lower bounds when there is signal The focus function allows the analyst to leverage prior knowledge about reasonable parameter regions within a fully frequentist analysis A narrow focus leads to larger gains in power around the center of mass of the focus function but pays a price in terms of decreased power if the focus center of mass differs from the true value of the parameter µ∗ The CLs method in particle physics also introduces a bias in order to adjust the power of the test in certain regions Page 10 James Carzon et al.: On Focusing Statistical Power for Searches and Measurements in Particle Physics 10 In Figure 6(A), the width of the LRS confidence belt remains constant for this example, indicating constant expected interval size with respect to x, whereas the FTS confidence belt is narrower for x values close to the marginal distribution of X This result is as expected because “optimal power” or “minimum average size” is defined with respect to the induced marginal distribution pX; see Appendix D for theoretical proof Page 12 James Carzon et al.: On Focusing Statistical Power for Searches and Measurements in Particle Physics 12 In Figure 9(center; µ∗ = 1.0) gives a case where FTS’ power curve is nowhere lower than LRS’ and higher for some µ This is because µ∗ is close to the center of the focus function When the power curve for a test drops below the test level, then we say that that test is biased However, this does not mean that the coverage of the confidence intervals is compromised The end result of the chosen focus function is raising the lower bound for searches for small values of µ Page 14 James Carzon et al.: On Focusing Statistical Power for Searches and Measurements in Particle Physics 14 In every case, FTS provides improved lower and upper bounds as seen in Figure 4 The distributions of the upper and the lower bounds of 68% confidence intervals for the LZ-inspired study, shown as box plots — the median values are represented by circles (or stars) with the 25th and 75th percentiles of each distribution connected with a line Both the median upper bound and the median lower bound are closer to the bisector for FTS than LRS, which is consistent with tighter parameter constraints Page 4 James Carzon et al.: On Focusing Statistical Power for Searches and Measurements in Particle Physics 4 The practice of using the Bayes factor as a frequentist test statistic to increase power was originally proposed by I.J.

Kai: Good who referred to the practice as “the Bayes/non-Bayes comprise” The introduction of this type of technical bias does not hinder a fully frequentist interpretation of the results and should not be mistaken for a constraint term sometimes introduced into the likelihood in particle physics Page 6 James Carzon et al.: On Focusing Statistical Power for Searches and Measurements in Particle Physics 6 The CLs method in particle physics also introduces a bias in order to adjust the power of the test in certain regions The CLs method was developed to address this issue, and

32: provides another approach to avoiding empty confidence intervals FTS could similarly be adapted to avoid empty confidence intervals, although such an adaptation may not preserve the optimality properties proven in Appendix D The CLs method in particle physics also introduces a bias in order to adjust the power of the test in certain regions Page 8 James Carzon et al.: On Focusing Statistical Power for Searches and Measurements in Particle Physics 8 FTS could similarly be adapted to avoid empty confidence intervals, although such an adaptation may not preserve the optimality properties proven in Appendix D Alternatively, one could report a singleton point estimate (e.g., the MLE, the focal point, or 0) when the interval would otherwise be empty The improved power of the FTS, enabled by the rapid building of confidence intervals can enhance scientific discovery power of flagship measurements Page 9 James Carzon et al.: On Focusing Statistical Power for Searches and Measurements in Particle Physics 9 The general rule is that a narrow focus leads to larger gains in power around the center of mass of the focus function, but one then pays a price (as compared to a wider focus) in terms of decreased power if the focus center of mass differs from the true value of the parameter µ∗ While FTS is biased in the technical sense, it is a deliberate choice to enhance the power of the test in physics-motivated regions without having to compromise on the coverage of the confidence intervals As an improvement to the core statistical methodology of the field, FTS can enhance discovery potential and physics sensitivities across many experiments The improved power of the FTS, enabled by the rapid building of confidence intervals can enhance scientific discovery power of flagship measurements Finally, our proposed method, which is here illustrated for histogram-based analyses, extends to unbinned, high-dimensional analyses with analytic expressions for the likelihood The improved power of the FTS, enabled by the rapid building of confidence intervals can enhance scientific discovery power of flagship measurements Finally, our proposed method, which is here illustrated for histogram-based analyses, extends to unbinned, high-dimensional analyses with analytic expressions for the likelihood

9: as well to as any other parametric

1: or nonparametric likelihood estimators [40, 2507.17831> Page 8 James Carzon et al.: On Focusing Statistical Power for Searches and Measurements in Particle Physics 8 The code used to perform analysis and generate figures for this article can be found publicly available on GitHub at https://github.com/lee-group-cmu/focus-paper The improved power of the FTS, enabled by the rapid building of confidence intervals can enhance scientific discovery power of flagship measurements Page 9 James Carzon et al.: On Focusing Statistical Power for Searches and Measurements in Particle Physics 9 The code used to perform analysis and generate figures for this article can be found publicly available on GitHub at https://github.com/lee-group-cmu/focus-paper The improved power of the FTS, enabled by the rapid building of confidence intervals can enhance scientific discovery power of flagship measurements Page 9 James Carzon et al.: On Focusing Statistical Power for Searches and Measurements in Particle Physics 9 The code used to perform analysis and generate figures for this article can be found publicly available on GitHub at https://github.

Paper summary: Kai: So, to recap, the core thesis of this paper is that we can improve searches and measurements by moving away from just the standard likelihood ratio test.

Mira: They introduce a focused test statistic, which they call Tf, that lets analysts concentrate the statistical power of their test onto specific areas in the parameter space.

Kai: This focused test statistic is defined as two log of the value under a specific hypothesis compared to the value under a general model, weighted by some focus function.

Mira: The authors show that this FTS outperforms the standard LRT statistic when applied to examples like Higgs boson measurements and dark matter searches for WIMPs.

Kai: The paper claims this focused test statistic can give significant gains in power, even in regimes where sample sizes are small or the signal fraction is low.

Mira: They also demonstrate that this method can maintain validity of confidence intervals even when you place the focus away from the true parameter value, which is a key advantage over some other methods.

Lev: From my side, it’s interesting that they mention that this statistic has Bayesian roots because it relates to the Bayes factor, but they are applying it in a purely frequentist framework with guarantees on coverage and power.

Kai: That's right. They are showing that using this concept as a frequentist test statistic is not just an academic exercise; it has the potential to improve discovery potential for many experiments.

Mira: The overall claim is that this FTS framework can be applied to any existing particle physics analysis and enhance sensitivity in key measurements across experiments.

Conclusion: Kai: Thinking about the authors, James Carzon and his team, they’ve developed a method that gives scientists explicit control over where their statistical tests are most sensitive.

Mira: It’s a tool that lets you decide whether you want to focus on the center of the distribution or push it towards a specific region based on prior knowledge.

Kai: The implication is that for future searches, physicists won't just run one test blindly; they can tailor their statistical procedure to maximize sensitivity where they need it most.

Mira: It shifts how we approach setting limits and making measurements by giving the analyst a way to leverage prior knowledge about reasonable parameter regions within a frequentist analysis.

Lev: For hardware development, this means that when you’re designing the system, you can perhaps design it to be more sensitive in those specific areas where your focus function dictates.

Kai: So, it’s not just about getting better numbers; it’s about giving the scientist a more flexible and powerful statistical framework for exploring their data.

More episodes

← Home