Inference after selection on randomized p-values
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: We're looking at the paper: "Inference after selection on randomized p-values".
Jane: Here's the gist of the paper. Inference after selection on randomized p-values provides a procedure to decompose a p-value into components for inference and selection that are independent under the null hypothesis pg1.
Tom: This work develops tests motivated by selective inference approaches like data thinning, fission, and carving to control the Type 1 error rate conditional on selection from randomized p-values pg2.
Paper summary: Tom: So to recap, this paper introduces a procedure to split a p-value into two parts: one for inference and one for selection, which are independent under the null hypothesis >
Jane: The core claim is that by developing tests inspired by selective inference ideas like data thinning or carving, we can control the Type one error rate when we select hypotheses using randomized p-values > <ref:2611.07459#pg1,control the Type 1 error rate>
Tom: It’s not just a theoretical exercise; they show that their specific tests, formulated in terms of the p-values, exactly match tests you would write using the full data under certain conditions >
Lu: They are showing that this works for things like Example one and Example two where selection happens either by thresholding p-values or by picking the hypothesis with the smallest p-value > <ref:2611.07459#pg1>
Jane: That really moves it from theory into application because it tackles situations like the winner's curse where you pick the best result based on its p-value >
Meng: From an engineering view, if we can make this work using only p-values instead of needing to re-run experiments or store massive datasets for every possible selection path, that would reduce our computational burden significantly >
Lalam: The paper focuses on what happens when you select a parameter based on the smallest p-value, p j equals arg min1≤k≤m pk, and the goal is to infer the parameter theta j given that p j is the smallest p-value >
Tom: And they introduce a threshold, xi, such that you want to ensure that if your selected p-value p j is less than or equal to that threshold, the probability of making a Type one error on your inference about theta j stays controlled at alpha > <ref:2611.07459#pg1>
Jane: That threshold sounds like the key mechanism they use to bridge the gap between what you select and what you can actually claim about it >
Conclusion: Tom: So, looking at the whole paper, this work by Perry, Witten, and their team is essentially providing a formal way to make sense of inference when your choice of which hypothesis to focus on is itself determined by a p-value >
Jane: It’s about making sure that the procedure you use to choose your best result doesn't mess up the math you use to draw conclusions about that result >
Lu: The implication is that for problems involving sequential testing or choosing hypotheses based on minimizing p-values, there are now valid ways to perform inference after selection without needing to rely on the full raw data set for every step >
Meng: For us in production, it suggests we can design more robust decision-making pipelines where we account for the randomness of hypothesis selection upfront rather than just hoping our initial data is good enough >
Lalam: I think it's about decoupling the selection mechanism from the inference mechanism so they don't interfere with each other when things are uncertain >
University of Washington
stat.ME
Submitted: 2026-11-05
Updated: 2026-11-05
Code: https://github.com/rflperry/pthin
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 98/100
The gist: Inference after selection on randomized p-values provides a procedure to decompose a p-value into components for inference and selection that are independent under the null hypothesis pg1.
Key concepts
- Inference after selection on randomized p-values
- This is a procedure used to break down a single p-value into separate components. These components allow researchers to perform both statistical inference (drawing conclusions about the data) and selection (deciding which data points are relevant) independently, ensuring that the error rates remain controlled under specific conditions.
- Randomized p-values
- This technique involves assigning random values to p-values. This randomization is crucial because it allows for a formal procedure to ensure that the components of the p-value related to selection and inference are statistically independent when the null hypothesis is true.
- Selective inference approaches (data thinning, fission, carving)
- These are methods used in statistics to control error rates when you intentionally select a subset of data. Data thinning involves randomly removing data points, while fission and carving refer to other ways of partitioning or isolating parts of the dataset for specific analyses.
Terminology
Summary
Inference after selection on randomized p-values provides a procedure to decompose a p-value into components for inference and selection that are independent under the null hypothesis pg1. p. 0 Inference after selection on randomized p-values
Ronan Perry1∗
, Daniela Witten1,2
1 University of Washington, Department of Statistics
2 University of Washington, Department of Biostatistics
November 6, 2026
Abstract
We consider inference on parameters and hypotheses selected using p-values. We provide a simple procedure to decompose a p-value into two components, one
for inference and one for selection, that are independent under the null hypothesis. We provide a simple procedure to decompose a p-value into two components, one
for inference and one for selection, that are independent under the null hypothesis. To control the Type 1 error rate conditional on selection of the null hypothesis
from randomized p-values, we develop tests motivated by the selective inference
approaches of data thinning, fission, and carving.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Adaptive hypothesis testing can be implemented by using Algorithm 2 to iteratively reveal more finely masked p-values until a hypothesis is selected, allowing for conditional inference on that selected parameter. This procedure allows an analyst to avoid selecting likely null hypotheses while ensuring that the stopping index is not too low, as noted in the discussion of Algorithm 2.
-
Confidence intervals and point estimators can be constructed using the p-carving CDF from Lemma 6, which attains nominal conditional coverage at level one minus alpha. This allows for valid conditional confidence intervals even when raw data access is restricted to only p-values.
-
The system can perform hypothesis testing on sequential testing problems by applying Corollary 11, which provides a test that controls the conditional Type 1 error rate under a randomized stopping rule based on available p-values. This handles scenarios where more data is collected and new p-values are computed until a threshold is met.
-
For genetic association studies involving multiple tests, the system can select the set of hypotheses using an optimized parameter epsilon to minimize the number of other hypotheses selected while maintaining high power, as suggested by Proposition 16. This choice of epsilon ensures that
no hypotheses are 'lost during selection' due to the randomization in p-thinning.
-
The system can compare different selective inference methods empirically in settings like the file drawer problem or winner's curse by analyzing their mean absolute deviation (MAD) for estimators, showing that p-carving and data carving perform the best among tested methods. This comparison allows for a data-driven choice between selection strategies and inference techniques.
Abstract
We consider inference on parameters and hypotheses selected using p-values. We provide a simple procedure to decompose a p-value into two components, one for inference and one for selection, that are independent under the null hypothesis. To control the Type 1 error rate conditional on selection of the null hypothesis from randomized p-values, we develop tests motivated by the selective inference approaches of data thinning, fission, and carving. Unlike the typical application of these approaches, however, our tests only make use of the p-values rather than the full data. Furthermore, we construct confidence intervals that attain the nominal conditional coverage and corresponding estimators. In the context of the file drawer problem, the winner's curse, and sequential testing, we show that our proposals offer both theoretical and empirical advantages over existing randomized and non-randomized approaches. Lastly, we apply our methods for selective inference on genetic associations using publicly-available p-values from the UK Biobank.
Sources
- Optimal Inference After Model Selection
- Inference conditional on selection: a review
- Selective inference is easier with p-values
Related papers
- Learning Robust Treatment Rules for Censored Data
- Neural Estimation of Summary Statistics for Simulation-Based Inference with Non-Asymptotic Guarantees
- Spectral divide-and-conquer MCMC for long stationary time series
- Improving the Efficiency of Subgroup Analysis in Randomized Controlled Trials with TMLE
- Personalized Federated Vector Autoregression with Personalization Diversity
- Doubly Robust and Efficient Calibration of Prediction Sets for Right-Censored Time-to-Event Outcomes