Simulation Studies For Goodness-of-Fit and Two-Sample Methods For Univariate Data
summary
The gist
Simulation studies regarding the power of various goodness-of-fit as well as nonparametric two-sample tests for univariate data are presented…
In short
Simulation studies tested various goodness-of-fit and nonparametric two-sample tests for univariate data across continuous and discrete types. The results showed no single method is universally powerful; performance varies significantly depending on the specific test, data type, and null hypothesis. The paper proposes a small set of recommended methods for each scenario to ensure at least one test has good power.
Key concepts
- Goodness-of-Fit (GoF) Tests
- These tests are used to check if observed data matches a specific theoretical distribution. They are applied to univariate data, which can be continuous or discrete. The study compares different GoF methods like Wilson's test and chi-square tests with varying numbers of bins.
- Two-Sample Tests
- These statistical tests compare two independent samples to see if they come from the same underlying distribution. The study examines these tests for both continuous and discrete data, testing methods such as Kuiper's test and Anderson-Darling to assess their effectiveness.
- Power of a Test
- Power refers to a test's ability to correctly reject a false null hypothesis. Simulation results showed that no single method provides consistently good power across all scenarios. Some tests perform very well in certain cases, while others fail badly in others, meaning performance is highly dependent on the specific data and hypothesis.
- Optimal Bin Selection
- For chi-square goodness-of-fit tests, the study recommends using 10 bins as a reasonable compromise for both continuous and discrete data. This number is adjusted based on the number of parameters being estimated, aiming to find a balance between test sensitivity and accuracy.
Terminology used across episodes
This episode discusses
The paper
Simulation Studies For Goodness-of-Fit and Two-Sample Methods For Univariate Data · Read on arXiv
Wolfgang Rolke
University of Puerto Rico - Mayaguez
We present the results of a large number of simulation studies regarding the power of various goodness-of-fit as well as nonparametric two-sample tests for univariate data. This includes both continuous and discrete data. In general no single method can be relied upon to provide good power, any one method may be quite good for some combination of null hypothesis and alternative and may fail badly for another. Based on the results of these studies we propose a fairly small number of methods chosen such that for any of the case studies included here at least one of the methods has good power. The studies were carried out using the R packages R2sample and Rgof, available from CRAN.
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Simulation Studies For Goodness-of-Fit and Two-Sample Methods For Univariate Data".
Jane: Here's the gist of the paper. Simulation studies regarding the power of various goodness-of-fit as well as nonparametric two-sample tests for univariate data are presented, showing that no single method can be relied upon to provide good power Based on the results of these studies we propose a fairly small number of methods chosen such that for any of the case studies included here at least one of the methods has good power.
Tom: Scope and Methodology The study investigates both goodness-of-fit (gof) and nonparametric two-sample problems for univariate data, encompassing both continuous and discrete data.
Jane: The methods are implemented using R packages R2sample and Rgof, which support various features such as handling weights, random sample sizes assumed from a Poisson distribution, and large-sample formulas for the two-sample problem.
Tom: Discrete data includes histogram (aka discretized or binned) data, and the methods are implemented using both Rcpp and parallel programming.
Paper summary: Jane: We just talked about how the authors ran simulations to see which goodness-of-fit and two-sample tests worked best, and they concluded that no single method is universally good for everything. The core thesis of "Simulation Studies For Goodness-of-Fit and Two-Sample Methods For Univariate Data" is that you need a small, carefully chosen set of methods so that for any case study they included, at least one method will have good power.
Tom: Exactly. They're not claiming one test is the best; they’re saying if you use this curated list of tests—like Wilson’s test or Anderson-Darling—you have a better chance than just picking some random statistical tool.
Lu: The methodology involved using R packages R2sample and Rgof to run a large number of simulation studies covering both continuous and discrete data problems.
Meng: They explicitly included tests for discrete data because there weren't many implementations in R at the time, which made it possible to test those scenarios directly.
Jane: And they also addressed the issue of combining tests; they found that you can combine several of these individual tests so that one rejects the null hypothesis if any of them do so, and they could even find a correct p value for this combination test via simulation.
Tom: That’s a neat feature, because it means you don't have to commit to just one test and hope for the best; you can use a portfolio approach.
Lu: They also found that continuous data problems tend to have good power in the corresponding discrete histogram problem, which is interesting for cross-domain analysis.
Meng: But they did flag that chi square tests with a large number of bins are weak, and they gave specific advice on how many bins to use for a reasonable result.
Jane: So, it’s really about giving practitioners actionable advice on what to use when they have a continuous or discrete dataset, moving beyond just the abstract statistical theory.
Tom: It boils down to being practical: if you're dealing with continuous data, you pick from their list of four recommended methods.
Lu: And if you are dealing with discrete data, you pick from another list of three recommended methods.
Meng: This gives us a clear decision tree for deployment based on the data structure we encounter in production environments.
Jane: So the main point is that power isn't guaranteed by any one method alone; it depends on the specific problem setup.
Tom: And they confirm that within simulation error, all methods maintain the five percent type one error rate, which keeps them statistically honest.
Conclusion: Jane: To wrap up this discussion on "Simulation Studies For Goodness-of-Fit and Two-Sample Methods For Univariate Data," the authors have provided a very practical recommendation for choosing a robust statistical method.
Tom: They’ve basically shown that instead of chasing the single highest power number, you should choose one from their curated lists based on whether your data is continuous or discrete.
Lu: The implication is that for anyone using these tests, the choice of test should be guided by whether your data is continuous or discrete and which specific test family you are examining.
Meng: For deployment at scale, this means we can't just rely on a single generic goodness-of-fit test; we have to select one from their proposed set tailored to our specific data structure.
Jane: It’s a practical guide for picking the right tool for the job when you're trying to validate statistical assumptions on your data.
Tom: So, even if you use a combination of tests, it still comes down to having at least one method that performs well in your specific scenario.
Lu: The paper confirms that no single test is universally good across all scenarios; power varies dramatically depending on the null hypothesis and alternative being tested.
Meng: And while they do provide specific methods, they also warn you about the limitations, like how chi square tests with a large number of bins are generally not powerful enough.
Jane: So the main takeaway is that when you're selecting a test for your univariate data, look at what the paper says about continuous versus discrete and use their suggested lists to pick from there.
More episodes
- Daily Summary for 2026-11-19
- 2310.07973-Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning
- 2408.09155-Learning Robust Treatment Rules for Censored Data
- Daily Summary for 2026-11-19
- 2603.23397-Kinetic Langevin Splitting Schemes for Constrained Sampling
- 2606.09906-An information-geometric framework for mapping maximum potential biodiversity
- 2603.27718-Induced replication and the assessment of models
- 1307.0898-Entropy of a Zipfian Distributed Lexicon
- 2406.19974-Generalizing self-normalized importance sampling with couplings
- 2412.10213-Collaborative Optimal Design of Controlled Experiments with Subject Covariates