A Family-Level Graphical Gatekeeping Procedure with Iterative Retesting
summary
The gist
In this paper, a novel Bonferroni based multiple testing procedure with retesting option is introduced for testing hierarchically ordered families of hypotheses in complex clinical trials…
In short
This paper introduces a novel Bonferroni-based gatekeeping procedure with retesting for testing hierarchically ordered hypotheses in complex clinical trials. The method sequentially tests families, iteratively updating local critical values based on previous rejections to strongly control the global familywise error rate under arbitrary dependence. It is extended to handle complex two-layer structures while maintaining strong error control.
Key concepts
- Gatekeeping Strategy
- A general approach used in clinical trials to test multiple research objectives sequentially, where each objective acts as a 'gatekeeper' for the ones following it. This ensures that testing proceeds in a logical, ordered manner based on the hierarchical relationship between the hypotheses being tested.
- Bonferroni-based Procedure with Retesting
- This is the core method where each hypothesis family is retested multiple times using a Bonferroni procedure. The critical value for each test is iteratively updated based on results from higher-ranked families, allowing for repeated checks to maintain strict error control.
- Global Familywise Error Rate (FWER)
- This measures the probability of making at least one false rejection across all tested hypotheses in the entire study. The procedure aims to strongly control this rate at a specified level $\alpha$, meaning there is a low chance of falsely concluding that any hypothesis is significant.
Terminology used across episodes
This episode discusses
The paper
A Family-Level Graphical Gatekeeping Procedure with Iterative Retesting · Read on arXiv
Sanofi · New Jersey Institute of Technology · Temple University
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: We're looking at the paper: "A Family-Level Graphical Gatekeeping Procedure with Iterative Retesting".
Jane: Here's the gist of the paper. In this paper, a novel Bonferroni based multiple testing procedure with retesting option is introduced for testing hierarchically ordered families of hypotheses in complex clinical trials, which controls the global familywise error rate strongly under arbitrary dependence Complex clinical trials often involve multiple research objectives grouped into sets based on their inherent hierarchical relationships, leading to hypotheses formulated into ordered families that must be tested sequentially The gatekeeping strategy is a general approach developed specifically to test preordered families of hypotheses in a sequential manner with each family working as a gatekeeper for the ones following it Several types of gatekeeping strategies exist, including serial gatekeeping strategy, parallel gatekeeping strategy, and tree gatekeeping strategy More powerful and flexible multiple testing methods have been developed based on these strategies Description of the Proposed Procedure The proposed procedure is a Bonferroni based gatekeeping procedure with retesting option where families are retested repeatedly using Bonferroni procedures in a sequential manner with different critical values at each repetition for a family Each family is assigned a pre-determined fraction of the overall level α for its initial critical value, which is then updated iteratively based on local critical values passed down from higher ranked families and initial levels assigned to lower ranked families The procedure exhibits several desirable features, including strongly controlling the global familywise error rate (FWER) under arbitrary dependence It strictly follows the hierarchical sequential scheme where higher rank families have more chances to be retested than lower rank families Mechanism of Level Updating The level for the local critical value for a family at each test depends on certain amount of the levels associated with the local critical values passed down from higher ranked families and the initial levels assigned to lower ranked families The distribution of the amount of critical value transferred among families can be pre-fixed by an m × m transition matrix G, where gij is defined as the proportion of the critical value that can be transferred from Fi to Fj In the two-family case, Algorithm 1 defines how local levels are updated based on rejected nulls in previous stages Theoretical Guarantees and Control Theorem 1 presents that the Bonferroni-based gatekeeping procedure with retesting option described in Algorithm 2 strongly controls the global FWER at level α under arbitrary dependence The proof relies on showing that for any stage k, the probability of rejecting at least one true null hypothesis is bounded by α, leading to a global FWER control of FWER = lim k→∞ Pr(Vk ≥ 1) ≤ α Clinical Trial Illustrations Two clinical trial examples are used to illustrate the application of the proposed procedures, comparing them with existing methods such as Guilbaud’s generalized Bonferroni parallel gatekeeping method and superchain procedure The Retest procedure was shown to reject more null hypotheses in one example compared to No-retest or BR procedures, demonstrating power improvement The two-layer structure extension is also developed with Algorithm 3, which strongly controls the global FWER at level α under arbitrary dependence The proposed procedure enjoys simplicity when compared with the superchain procedure because it sticks to the simple Bonferroni method for testing each family throughout the whole procedure and proceeds in a sequential manner The sequential testing strategy seems more natural than the superchain procedure which tests all families simultaneously, making it easier to implement and intuitive in a clinical sense The computational complexity is also noted as being less severe compared to the superchain procedure due to its simpler updating rules of the local critical values Extension to Complex Structures An extension considers a complex two-layer hierarchical structure with multiple families within each layer, where families between layers are tested sequentially while those within each layer are tested simultaneously Algorithm 3 is developed for this structure, allowing all m families of hypotheses to be tested more than once while strongly controlling the global FWER at α under arbitrary dependence The theorem for this two-layer procedure also confirms strong global FWER control at level α Final Assessment of Comparison Both Guibaud’s generalized Bonferroni parallel gatekeeping procedure and the proposed procedure are based on simple Bonferroni method, but the proposed procedure has some better, desirable features compared to existing methods While superchain procedures offer power improvement, the proposed procedure has a simpler implementation and is more natural than the superchain procedure which tests all families simultaneously The use of the Bonferroni procedure as the basic procedure for testing each family makes our proposed procedure slightly conservative compared to Dimitrienko et al.’s α-exhaustive multistage gatekeeping procedure and superchain procedure in some cases The proposed procedure controls the global FWER without any assumption of dependence structure among the underlying test statistics, which is a significant advantage References
1: Alosh M., Bretz F.
Tom: and Huque M.
Jane: (2014).
Tom: Advanced multiplicity adjustment methods in clinical trials.
Title and authors: Tom: The paper's title, "A Family-Level Graphical Gatekeeping Procedure with Iterative Retesting," tells us exactly what it’s about: a method for testing ordered families of hypotheses where you get to retest things more than once.
Jane: It’s about taking the existing gatekeeping ideas and adding this specific retesting option, which is crucial when dealing with those complex relationships in clinical trials or other multi-stage research.
Lu: They frame it as a Bonferroni based procedure, which is simple because it sticks to that basic error control mechanism for each family but adds the iteration on top.
Meng: I’m thinking, if you have to retest things repeatedly, the computational cost has to be significant; how much extra time are we looking at just for these retests?
Lalam: The paper suggests that the way you update those local critical values is managed by a pre-determined transition matrix G, which dictates how much significance level gets transferred between families.
The paper's summary: Tom: Basically, the summary says this procedure allows families to be sequentially tested more than once using different critical values at each repetition, and those values get updated based on what happened in the stages before.
Jane: It’s an iterative mechanism where you start with an initial fraction of your alpha level for a family, then you adjust that level down or up based on the results from higher-ranked families.
Lu: They prove that this approach strongly controls the global familywise error rate under arbitrary dependence, which is a big deal because it doesn't assume any specific structure on the underlying test statistics.
Meng: Arbitrary dependence is a strong claim; what if our real data has some hidden structure that this procedure misses, and we run into problems in production?
Lalam: The control mechanism relies on showing that at every stage k, the probability of rejecting at least one true null hypothesis is bounded by alpha.
The paper's improvements: Tom: One major improvement they highlight is how this approach handles the retesting option, allowing families to be retested sequentially with different critical values based on prior rejections.
Jane: They also show an extension to a complex two-layer hierarchical structure, which means you can have families within a layer being tested simultaneously while still testing the layers in sequence.
Lu: Algorithm three is developed for that two-layer structure, and it allows all m families in each layer to be tested more than once while maintaining strong global familywise error rate control at alpha under arbitrary dependence.
Meng: That two-layer structure sounds computationally heavier; we're talking about testing multiple families within a layer simultaneously, which means the complexity could spike quickly if the number of families in those layers gets large.
Lalam: The transition matrix G mentioned earlier is what allows you to pre-fix how that critical value transfer happens between different family levels, giving you flexibility in control.
Conclusion: Tom: So, to wrap up, the paper introduces this Family-Level Graphical Gatekeeping Procedure with Iterative Retesting as a simple Bonferroni based method that sequentially retests families with updated local critical values.
Jane: The main implication is that this procedure strongly controls the global familywise error rate under arbitrary dependence, which is a significant result when you can't assume the test statistics follow any specific distribution.
Lu: It’s an elegant way to manage those ordered hypotheses by using the structure itself to guide how much testing power you allocate at each step.
Meng: From an engineering standpoint, while it sounds powerful theoretically, I still need to see how efficiently this runs when we have hundreds of families in those complex two-layer structures on our production hardware.
Lalam: The final assessment is that while it uses the simple Bonferroni basis for each test, this procedure has better features than some existing methods because it controls the global familywise error rate without needing any assumptions about the dependence structure among your test statistics.
More episodes
- Daily Summary for 2026-11-19
- 2310.07973-Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning
- 2408.09155-Learning Robust Treatment Rules for Censored Data
- Daily Summary for 2026-11-19
- 2603.23397-Kinetic Langevin Splitting Schemes for Constrained Sampling
- 2606.09906-An information-geometric framework for mapping maximum potential biodiversity
- 2603.27718-Induced replication and the assessment of models
- 1307.0898-Entropy of a Zipfian Distributed Lexicon
- 2406.19974-Generalizing self-normalized importance sampling with couplings
- 2412.10213-Collaborative Optimal Design of Controlled Experiments with Subject Covariates