A Family-Level Graphical Gatekeeping Procedure with Iterative Retesting
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: We're looking at the paper: "A Family-Level Graphical Gatekeeping Procedure with Iterative Retesting".
Jane: Here's the gist of the paper. In this paper, a novel Bonferroni based multiple testing procedure with retesting option is introduced for testing hierarchically ordered families of hypotheses in complex clinical trials, which controls the global familywise error rate strongly under arbitrary dependence Complex clinical trials often involve multiple research objectives grouped into sets based on their inherent hierarchical relationships, leading to hypotheses formulated into ordered families that must be tested sequentially The gatekeeping strategy is a general approach developed specifically to test preordered families of hypotheses in a sequential manner with each family working as a gatekeeper for the ones following it Several types of gatekeeping strategies exist, including serial gatekeeping strategy, parallel gatekeeping strategy, and tree gatekeeping strategy More powerful and flexible multiple testing methods have been developed based on these strategies Description of the Proposed Procedure The proposed procedure is a Bonferroni based gatekeeping procedure with retesting option where families are retested repeatedly using Bonferroni procedures in a sequential manner with different critical values at each repetition for a family Each family is assigned a pre-determined fraction of the overall level α for its initial critical value, which is then updated iteratively based on local critical values passed down from higher ranked families and initial levels assigned to lower ranked families The procedure exhibits several desirable features, including strongly controlling the global familywise error rate (FWER) under arbitrary dependence It strictly follows the hierarchical sequential scheme where higher rank families have more chances to be retested than lower rank families Mechanism of Level Updating The level for the local critical value for a family at each test depends on certain amount of the levels associated with the local critical values passed down from higher ranked families and the initial levels assigned to lower ranked families The distribution of the amount of critical value transferred among families can be pre-fixed by an m × m transition matrix G, where gij is defined as the proportion of the critical value that can be transferred from Fi to Fj In the two-family case, Algorithm 1 defines how local levels are updated based on rejected nulls in previous stages Theoretical Guarantees and Control Theorem 1 presents that the Bonferroni-based gatekeeping procedure with retesting option described in Algorithm 2 strongly controls the global FWER at level α under arbitrary dependence The proof relies on showing that for any stage k, the probability of rejecting at least one true null hypothesis is bounded by α, leading to a global FWER control of FWER = lim k→∞ Pr(Vk ≥ 1) ≤ α Clinical Trial Illustrations Two clinical trial examples are used to illustrate the application of the proposed procedures, comparing them with existing methods such as Guilbaud’s generalized Bonferroni parallel gatekeeping method and superchain procedure The Retest procedure was shown to reject more null hypotheses in one example compared to No-retest or BR procedures, demonstrating power improvement The two-layer structure extension is also developed with Algorithm 3, which strongly controls the global FWER at level α under arbitrary dependence The proposed procedure enjoys simplicity when compared with the superchain procedure because it sticks to the simple Bonferroni method for testing each family throughout the whole procedure and proceeds in a sequential manner The sequential testing strategy seems more natural than the superchain procedure which tests all families simultaneously, making it easier to implement and intuitive in a clinical sense The computational complexity is also noted as being less severe compared to the superchain procedure due to its simpler updating rules of the local critical values Extension to Complex Structures An extension considers a complex two-layer hierarchical structure with multiple families within each layer, where families between layers are tested sequentially while those within each layer are tested simultaneously Algorithm 3 is developed for this structure, allowing all m families of hypotheses to be tested more than once while strongly controlling the global FWER at α under arbitrary dependence The theorem for this two-layer procedure also confirms strong global FWER control at level α Final Assessment of Comparison Both Guibaud’s generalized Bonferroni parallel gatekeeping procedure and the proposed procedure are based on simple Bonferroni method, but the proposed procedure has some better, desirable features compared to existing methods While superchain procedures offer power improvement, the proposed procedure has a simpler implementation and is more natural than the superchain procedure which tests all families simultaneously The use of the Bonferroni procedure as the basic procedure for testing each family makes our proposed procedure slightly conservative compared to Dimitrienko et al.’s α-exhaustive multistage gatekeeping procedure and superchain procedure in some cases The proposed procedure controls the global FWER without any assumption of dependence structure among the underlying test statistics, which is a significant advantage References
1: Alosh M., Bretz F.
Tom: and Huque M.
Jane: (2014).
Tom: Advanced multiplicity adjustment methods in clinical trials.
Title and authors: Tom: The paper's title, "A Family-Level Graphical Gatekeeping Procedure with Iterative Retesting," tells us exactly what it’s about: a method for testing ordered families of hypotheses where you get to retest things more than once.
Jane: It’s about taking the existing gatekeeping ideas and adding this specific retesting option, which is crucial when dealing with those complex relationships in clinical trials or other multi-stage research.
Lu: They frame it as a Bonferroni based procedure, which is simple because it sticks to that basic error control mechanism for each family but adds the iteration on top.
Meng: I’m thinking, if you have to retest things repeatedly, the computational cost has to be significant; how much extra time are we looking at just for these retests?
Lalam: The paper suggests that the way you update those local critical values is managed by a pre-determined transition matrix G, which dictates how much significance level gets transferred between families.
The paper's summary: Tom: Basically, the summary says this procedure allows families to be sequentially tested more than once using different critical values at each repetition, and those values get updated based on what happened in the stages before.
Jane: It’s an iterative mechanism where you start with an initial fraction of your alpha level for a family, then you adjust that level down or up based on the results from higher-ranked families.
Lu: They prove that this approach strongly controls the global familywise error rate under arbitrary dependence, which is a big deal because it doesn't assume any specific structure on the underlying test statistics.
Meng: Arbitrary dependence is a strong claim; what if our real data has some hidden structure that this procedure misses, and we run into problems in production?
Lalam: The control mechanism relies on showing that at every stage k, the probability of rejecting at least one true null hypothesis is bounded by alpha.
The paper's improvements: Tom: One major improvement they highlight is how this approach handles the retesting option, allowing families to be retested sequentially with different critical values based on prior rejections.
Jane: They also show an extension to a complex two-layer hierarchical structure, which means you can have families within a layer being tested simultaneously while still testing the layers in sequence.
Lu: Algorithm three is developed for that two-layer structure, and it allows all m families in each layer to be tested more than once while maintaining strong global familywise error rate control at alpha under arbitrary dependence.
Meng: That two-layer structure sounds computationally heavier; we're talking about testing multiple families within a layer simultaneously, which means the complexity could spike quickly if the number of families in those layers gets large.
Lalam: The transition matrix G mentioned earlier is what allows you to pre-fix how that critical value transfer happens between different family levels, giving you flexibility in control.
Conclusion: Tom: So, to wrap up, the paper introduces this Family-Level Graphical Gatekeeping Procedure with Iterative Retesting as a simple Bonferroni based method that sequentially retests families with updated local critical values.
Jane: The main implication is that this procedure strongly controls the global familywise error rate under arbitrary dependence, which is a significant result when you can't assume the test statistics follow any specific distribution.
Lu: It’s an elegant way to manage those ordered hypotheses by using the structure itself to guide how much testing power you allocate at each step.
Meng: From an engineering standpoint, while it sounds powerful theoretically, I still need to see how efficiently this runs when we have hundreds of families in those complex two-layer structures on our production hardware.
Lalam: The final assessment is that while it uses the simple Bonferroni basis for each test, this procedure has better features than some existing methods because it controls the global familywise error rate without needing any assumptions about the dependence structure among your test statistics.
Sanofi · New Jersey Institute of Technology · Temple University
stat.ME, stat.AP
Submitted: 2016-11-10
Updated: 2026-11-11
Comments: 42 pages, 7 figures, 6 tables. Includes proofs in the appendix
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 81/100
The gist: In this paper, a novel Bonferroni based multiple testing procedure with retesting option is introduced for testing hierarchically ordered families of hypotheses in complex clinical trials…
Key concepts
- Gatekeeping Strategy
- A general approach used in clinical trials to test multiple research objectives sequentially, where each objective acts as a 'gatekeeper' for the ones following it. This ensures that testing proceeds in a logical, ordered manner based on the hierarchical relationship between the hypotheses being tested.
- Bonferroni-based Procedure with Retesting
- This is the core method where each hypothesis family is retested multiple times using a Bonferroni procedure. The critical value for each test is iteratively updated based on results from higher-ranked families, allowing for repeated checks to maintain strict error control.
- Global Familywise Error Rate (FWER)
- This measures the probability of making at least one false rejection across all tested hypotheses in the entire study. The procedure aims to strongly control this rate at a specified level $\alpha$, meaning there is a low chance of falsely concluding that any hypothesis is significant.
Terminology
Summary
In this paper, a novel Bonferroni based multiple testing procedure with retesting option is introduced for testing hierarchically ordered families of hypotheses in complex clinical trials, which controls the global familywise error rate strongly under arbitrary dependence
Introduction to Gatekeeping Strategies
Complex clinical trials often involve multiple research objectives grouped into sets based on their inherent hierarchical relationships, leading to hypotheses formulated into ordered families that must be tested sequentially The gatekeeping strategy is a general approach developed specifically to test preordered families of hypotheses in a sequential manner with each family working as a gatekeeper for the ones following it Several types of gatekeeping strategies exist, including serial gatekeeping strategy, parallel gatekeeping strategy, and tree gatekeeping strategy More powerful and flexible multiple testing methods have been developed based on these strategies
Description of the Proposed Procedure
The proposed procedure is a Bonferroni based gatekeeping procedure with retesting option where families are retested repeatedly using Bonferroni procedures in a sequential manner with different critical values at each repetition for a family Each family is assigned a pre-determined fraction of the overall level α for its initial critical value, which is then updated iteratively based on local critical values passed down from higher ranked families and initial levels assigned to lower ranked families The procedure exhibits several desirable features, including strongly controlling the global familywise error rate (FWER) under arbitrary dependence It strictly follows the hierarchical sequential scheme where higher rank families have more chances to be retested than lower rank families
Mechanism of Level Updating
The level for the local critical value for a family at each test depends on certain amount of the levels associated with the local critical values passed down from higher ranked families and the initial levels assigned to lower ranked families The distribution of the amount of critical value transferred among families can be pre-fixed by an m × m transition matrix G, where gij is defined as the proportion of the critical value that can be transferred from Fi to Fj In the two-family case, Algorithm 1 defines how local levels are updated based on rejected nulls in previous stages
Theoretical Guarantees and Control
Theorem 1 presents that the Bonferroni-based gatekeeping procedure with retesting option described in Algorithm 2 strongly controls the global FWER at level α under arbitrary dependence The proof relies on showing that for any stage k, the probability of rejecting at least one true null hypothesis is bounded by α, leading to a global FWER control of FWER = lim k→∞ Pr(Vk ≥ 1) ≤ α
Clinical Trial Illustrations
Two clinical trial examples are used to illustrate the application of the proposed procedures, comparing them with existing methods such as Guilbaud’s generalized Bonferroni parallel gatekeeping method and superchain procedure The Retest procedure was shown to reject more null hypotheses in one example compared to No-retest or BR procedures, demonstrating power improvement The two-layer structure extension is also developed with Algorithm 3, which strongly controls the global FWER at level α under arbitrary dependence
Conclusion on Procedure Advantages
The proposed procedure enjoys simplicity when compared with the superchain procedure because it sticks to the simple Bonferroni method for testing each family throughout the whole procedure and proceeds in a sequential manner The sequential testing strategy seems more natural than the superchain procedure which tests all families simultaneously, making it easier to implement and intuitive in a clinical sense The computational complexity is also noted as being less severe compared to the superchain procedure due to its simpler updating rules of the local critical values
Extension to Complex Structures
An extension considers a complex two-layer hierarchical structure with multiple families within each layer, where families between layers are tested sequentially while those within each layer are tested simultaneously Algorithm 3 is developed for this structure, allowing all m families of hypotheses to be tested more than once while strongly controlling the global FWER at α under arbitrary dependence The theorem for this two-layer procedure also confirms strong global FWER control at level α
Final Assessment of Comparison
Both Guibaud’s generalized Bonferroni parallel gatekeeping procedure and the proposed procedure are based on simple Bonferroni method, but the proposed procedure has some better, desirable features compared to existing methods While superchain procedures offer power improvement, the proposed procedure has a simpler implementation and is more natural than the superchain procedure which tests all families simultaneously The use of the Bonferroni procedure as the basic procedure for testing each family makes our proposed procedure slightly conservative compared to Dimitrienko et al.’s α-exhaustive multistage gatekeeping procedure and superchain procedure in some cases The proposed procedure controls the global FWER without any assumption of dependence structure among the underlying test statistics, which is a significant advantage
References
[1] Alosh M., Bretz F. and Huque M. p. 1 p. 1 (2014). Advanced multiplicity adjustment methods in clinical trials. p. 0 Statistics in Medicine 33, 693–713. p. 0
[2] Bauer P. p. 1, Rohmel J., Maurer W. p. 1 p. 1 and Hothorn L. p. 1 (1998). Testing strategies in multidose experiments including active control. p. 3 Statistics in Medicine 17, 2133–2146. p. 0
[3] Bretz F. p. 1, Maurer W., Brannath W. and Posch M. (2009). A graphical approach to sequentially rejective multiple test procedures. p. 1 Statistics in Medicine 28, 586–604. p. 0
[4] Burman C. p. 1 F., Sonesson C. p. 1 and Guilbaud O. p. 1 (2009). A recycling framework for the construction of Bonferroni-based multiple tests. p. 4 Statistics in Medicine 28, 739–761. p. 0
[5] Chen X. p. 1, Luo X. and Capizzi T. p. 1 (2005). The application of enhanced parallel gatekeeping strategies. p. 1 Statistics in Medicine 24, 1385–1397. p. 0
[6] Dmitrienko A. p. 1, D’Agostino R. p. 1 B. and Huque M. F. (2013). Key multiplicity issues in clinical drug development. p. 0 Statistics in Medicine 32, 1079–1111. p. 0
[7] Dmitrienko A. p. 1, Kordzakhia G. p. 1 and Tamhane A. p. 1 p. 1 p. 1 C. (2011). Multistage and mixture gatekeeping procedures in clinical trials. p. 2 Journal of Biopharmaceutical Statistics 21, 726–747. p. 0
[8] Dmitrienko A. p. 1, Millen B. A., Brechenmacher T. and Paux G. (2011). Development of gatekeeping strategies in confirmatory clinical trials. p. 0 Biometrical Journal 53, 875–893.
[9] Dmitrienko A. p. 1, Offen W. and Westfall P. p. 1 H. (2003). Gatekeeping strategies for clinical trials that do not require all primary effects to be significant. p. 0 Statistics in Medicine 22, 2387–2400. p. 0
[10] Dmitrienko A. p. 1 and Tamhane A. C. (2011). Mixtures of multiple testing procedures for gatekeeping applications in clinical trials. p. 1 Statistics in Medicine 30, 1473–1488. p. 0
[11] Dmitrienko A. p. 1, Tamhane A. p. 1 p. 1 p. 1 p. 1 C. and Bretz F. p. 1 (2009). Gatekeeping procedures in clinical trials. p. 0 In Multiple Testing Problems in Pharmaceutical Statistics, Chapter 5. p. 0 Chapman and Hall/CRC Press: New York, 2009. p. 1
[12] Dmitrienko A. p. 1, Tamhane A. C., Liu L. and Wiens B. p. 1 p. 1 L. (2008). A note on tree gatekeeping procedures in clinical trials. p. 0 Statistics in Medicine 27, 3446–3451. p. 0
[13] Dmitrienko A. p. 1, Tamhane A. C., Wang X. and Chen X. p. 1 (2006). Stepwise gatekeeping procedures in clinical trial applications. p. 3 Biometrical Journal 48, 984–991.
[14] Dmitrienko A. p. 1, Tamhane A. C. and Wiens B. L. (2008). General multistage gatekeeping procedures. p. 1 Biometrical Journal 50, 667–677.
[15] Dmitrienko A. p. 1, Wiens B. L. and Tamhane A. C. (2007). Tree–structured gatekeeping tests in clinical trials with hierarchically ordered multiple objectives. p. 1 Statistics in Medicine 26, 2465–2478. p. 0
[16] Guilbaud O. p. 1 (2007). Bonferroni parallel gatekeeping - transparent generalizations, adjusted p-values, and short direct proofs. p. 9 Biometrical Journal 49, 917–927.
[17] Kim H. p. 1, Entsuah A. p. 1 R. and Shults J. p. 1 (2011). The union closure method for testing a fixed sequence of families of hypotheses. p. 1 Biometrika 98, 391–401.
[18] Kordzakhia G. p. 1 and Dmitrienko A. p. 1 p. 1 (2013). Superchain procedures in clinical trials with multiple objectives. p. 0 Statistics in Medicine 32, 486–508. p. 0
[19] Liu Y. p. 1 and Hsu J. (2009). Testing for efficacy in primary and secondary endpoints by partitioning decision paths. Journal of the American Statistical Association 104, 1661–1670. p. 1
[20] Maurer W. p. 1, Hothorn L. p. 1 and Lehmacher W. p. 1 (1995). Multiple comparisons in drug clinical trials and preclinical assays: a-priori ordered hypotheses. p. 0 In Biometrie in der Chemisch-pharmazeutischen Industrie, Vollmar J(ed.). Fischer Verlag: Stuttgart, 6, 3–18.
[21] Pitt B., Remme W., Zannad F., Neaton J., Martinez F., Roniker B., Bittman R., Hurley S., Kleiman J. and Gatlin M. (2003). Eplerenone, a selective aldosterone blocker, in patients with left ventricular dysfunction after myocardial infarction. New England Journal of Medicine 348, 1309–1321. p. 0
[22] Westfall P. p. 1 H. and Krishen A. p. 1 (2001). Optimally weighted, fixed-sequence, and gatekeeping multiple testing procedures. p. 1 Journal of Statistical Planning and Inference 99, 25–40. p. 1
[23] Wiens B. p. 1 L. (2003). A fixed-sequence Bonferroni procedure for testing multiple endpoints. p. 0 Pharmaceutical Statistics 2, 211–215. p. 0
[24] Wiens B. p. 1 L. and Dmitrienko A. (2005). The fallback procedure for evaluating a single family of hypotheses. p. 5 Journal of Biopharmaceutical Statistics 15, 929–942. p. 0
[25] Appendix Proof of Theorem 1
[26] Lemma 1 Consider a function f defined by f(j) = X j i=1 Ti ni + 1 − Ti ni Xm l=j+1 gil! n α∗ i(k) + Xm l=j+1
Tl nl + 1 − Tl nl Xm i=j+1 gli! n αl on the set, the desired result follows. p. 4
[27] By (6), in order to show FWERk ≤ α, it is sufficient to show 1 − Pr Dk ≤ α. p. 2
[28] The first inequality follows from Bonferroni’s inequality and the second follows from the assumption that true null p-values follow U(0, 1). p. 3
[29] FWER = lim k→∞ Pr(Vk ≥ 1) ≤ α, the desired result. p. 3
The procedure is extended to a complex two-layer hierarchical structure with multiple families within each layer, where families between layers are tested sequentially while those within each layer are tested simultaneously Algorithm 3 is developed for this structure, allowing all m families of hypotheses to be tested more than once while strongly controlling the global FWER at α under arbitrary dependence The theorem for this two-layer procedure also confirms strong global FWER control at level α
The computational complexity is even more severe when the number of families is large because of the updating rules of the local critical values and truncation parameters The use of the Bonferroni procedure as the basic procedure for testing each family makes our proposed procedure slightly conservative compared to Dimitrienko et al.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Develop an AI system capable of implementing a Bonferroni based gatekeeping procedure with retesting option by taking into account hierarchical relationships between research objectives. This system can sequentially test families of hypotheses and iteratively update local critical values based on previously rejected null hypotheses, as described in Algorithm 2.
-
The improved system will be able to strongly control the global familywise error rate under arbitrary dependence, as proven by Theorem 1, ensuring that the probability of falsely rejecting at least one true null hypothesis across all families remains below a specified level alpha.
-
The AI system can generalize its testing capability to handle complex two-layer hierarchical structures by implementing Algorithm 3, allowing families within each layer to be tested simultaneously while maintaining sequential testing across layers with updated local critical values.
-
The system can be configured based on a pre-defined transition matrix G to determine the proportion of critical value transfer between families, enabling flexible control over how significance levels are distributed across the ordered hypothesis families.
Related papers
- Neural Estimation of Summary Statistics for Simulation-Based Inference with Non-Asymptotic Guarantees
- Spectral divide-and-conquer MCMC for long stationary time series
- Improving the Efficiency of Subgroup Analysis in Randomized Controlled Trials with TMLE
- Personalized Federated Vector Autoregression with Personalization Diversity
- Doubly Robust and Efficient Calibration of Prediction Sets for Right-Censored Time-to-Event Outcomes
- Covariance-Calibrated Prediction-Powered Inference