Inference after selection on randomized p-values

summary

The gist

Inference after selection on randomized p-values provides a procedure to decompose a p-value into components for inference and selection that are independent under the null hypothesis pg1.

In short

The paper develops statistical tests based on selective inference after selecting data using randomized p-values. It aims to decompose a p-value into parts for both inference and selection that are independent under the null hypothesis. The new tests are motivated by techniques like data thinning, fission, and carving to maintain control over Type 1 error rates when making selections.

Key concepts

Inference after selection on randomized p-values
This is a procedure used to break down a single p-value into separate components. These components allow researchers to perform both statistical inference (drawing conclusions about the data) and selection (deciding which data points are relevant) independently, ensuring that the error rates remain controlled under specific conditions.
Randomized p-values
This technique involves assigning random values to p-values. This randomization is crucial because it allows for a formal procedure to ensure that the components of the p-value related to selection and inference are statistically independent when the null hypothesis is true.
Selective inference approaches (data thinning, fission, carving)
These are methods used in statistics to control error rates when you intentionally select a subset of data. Data thinning involves randomly removing data points, while fission and carving refer to other ways of partitioning or isolating parts of the dataset for specific analyses.

Terminology used across episodes

This episode discusses

The paper

Inference after selection on randomized p-values · Read on arXiv

University of Washington

We consider inference on parameters and hypotheses selected using p-values. We provide a simple procedure to decompose a p-value into two components, one for inference and one for selection, that are independent under the null hypothesis. To control the Type 1 error rate conditional on selection of the null hypothesis from randomized p-values, we develop tests motivated by the selective inference approaches of data thinning, fission, and carving. Unlike the typical application of these approaches, however, our tests only make use of the p-values rather than the full data. Furthermore, we construct confidence intervals that attain the nominal conditional coverage and corresponding estimators. In the context of the file drawer problem, the winner's curse, and sequential testing, we show that our proposals offer both theoretical and empirical advantages over existing randomized and non-randomized approaches. Lastly, we apply our methods for selective inference on genetic associations using publicly-available p-values from the UK Biobank.

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: We're looking at the paper: "Inference after selection on randomized p-values".

Jane: Here's the gist of the paper. Inference after selection on randomized p-values provides a procedure to decompose a p-value into components for inference and selection that are independent under the null hypothesis pg1.

Tom: This work develops tests motivated by selective inference approaches like data thinning, fission, and carving to control the Type 1 error rate conditional on selection from randomized p-values pg2.

Paper summary: Tom: So to recap, this paper introduces a procedure to split a p-value into two parts: one for inference and one for selection, which are independent under the null hypothesis >

Jane: The core claim is that by developing tests inspired by selective inference ideas like data thinning or carving, we can control the Type one error rate when we select hypotheses using randomized p-values > <ref:2611.07459#pg1,control the Type 1 error rate>

Tom: It’s not just a theoretical exercise; they show that their specific tests, formulated in terms of the p-values, exactly match tests you would write using the full data under certain conditions >

Lu: They are showing that this works for things like Example one and Example two where selection happens either by thresholding p-values or by picking the hypothesis with the smallest p-value > <ref:2611.07459#pg1>

Jane: That really moves it from theory into application because it tackles situations like the winner's curse where you pick the best result based on its p-value >

Meng: From an engineering view, if we can make this work using only p-values instead of needing to re-run experiments or store massive datasets for every possible selection path, that would reduce our computational burden significantly >

Lalam: The paper focuses on what happens when you select a parameter based on the smallest p-value, p j equals arg min1≤k≤m pk, and the goal is to infer the parameter theta j given that p j is the smallest p-value >

Tom: And they introduce a threshold, xi, such that you want to ensure that if your selected p-value p j is less than or equal to that threshold, the probability of making a Type one error on your inference about theta j stays controlled at alpha > <ref:2611.07459#pg1>

Jane: That threshold sounds like the key mechanism they use to bridge the gap between what you select and what you can actually claim about it >

Conclusion: Tom: So, looking at the whole paper, this work by Perry, Witten, and their team is essentially providing a formal way to make sense of inference when your choice of which hypothesis to focus on is itself determined by a p-value >

Jane: It’s about making sure that the procedure you use to choose your best result doesn't mess up the math you use to draw conclusions about that result >

Lu: The implication is that for problems involving sequential testing or choosing hypotheses based on minimizing p-values, there are now valid ways to perform inference after selection without needing to rely on the full raw data set for every step >

Meng: For us in production, it suggests we can design more robust decision-making pipelines where we account for the randomness of hypothesis selection upfront rather than just hoping our initial data is good enough >

Lalam: I think it's about decoupling the selection mechanism from the inference mechanism so they don't interfere with each other when things are uncertain >

More episodes

← Home