Simulation Studies For Goodness-of-Fit and Two-Sample Methods For Univariate Data
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Simulation Studies For Goodness-of-Fit and Two-Sample Methods For Univariate Data".
Jane: Here's the gist of the paper. Simulation studies regarding the power of various goodness-of-fit as well as nonparametric two-sample tests for univariate data are presented, showing that no single method can be relied upon to provide good power Based on the results of these studies we propose a fairly small number of methods chosen such that for any of the case studies included here at least one of the methods has good power.
Tom: Scope and Methodology The study investigates both goodness-of-fit (gof) and nonparametric two-sample problems for univariate data, encompassing both continuous and discrete data.
Jane: The methods are implemented using R packages R2sample and Rgof, which support various features such as handling weights, random sample sizes assumed from a Poisson distribution, and large-sample formulas for the two-sample problem.
Tom: Discrete data includes histogram (aka discretized or binned) data, and the methods are implemented using both Rcpp and parallel programming.
Paper summary: Jane: We just talked about how the authors ran simulations to see which goodness-of-fit and two-sample tests worked best, and they concluded that no single method is universally good for everything. The core thesis of "Simulation Studies For Goodness-of-Fit and Two-Sample Methods For Univariate Data" is that you need a small, carefully chosen set of methods so that for any case study they included, at least one method will have good power.
Tom: Exactly. They're not claiming one test is the best; they’re saying if you use this curated list of tests—like Wilson’s test or Anderson-Darling—you have a better chance than just picking some random statistical tool.
Lu: The methodology involved using R packages R2sample and Rgof to run a large number of simulation studies covering both continuous and discrete data problems.
Meng: They explicitly included tests for discrete data because there weren't many implementations in R at the time, which made it possible to test those scenarios directly.
Jane: And they also addressed the issue of combining tests; they found that you can combine several of these individual tests so that one rejects the null hypothesis if any of them do so, and they could even find a correct p value for this combination test via simulation.
Tom: That’s a neat feature, because it means you don't have to commit to just one test and hope for the best; you can use a portfolio approach.
Lu: They also found that continuous data problems tend to have good power in the corresponding discrete histogram problem, which is interesting for cross-domain analysis.
Meng: But they did flag that chi square tests with a large number of bins are weak, and they gave specific advice on how many bins to use for a reasonable result.
Jane: So, it’s really about giving practitioners actionable advice on what to use when they have a continuous or discrete dataset, moving beyond just the abstract statistical theory.
Tom: It boils down to being practical: if you're dealing with continuous data, you pick from their list of four recommended methods.
Lu: And if you are dealing with discrete data, you pick from another list of three recommended methods.
Meng: This gives us a clear decision tree for deployment based on the data structure we encounter in production environments.
Jane: So the main point is that power isn't guaranteed by any one method alone; it depends on the specific problem setup.
Tom: And they confirm that within simulation error, all methods maintain the five percent type one error rate, which keeps them statistically honest.
Conclusion: Jane: To wrap up this discussion on "Simulation Studies For Goodness-of-Fit and Two-Sample Methods For Univariate Data," the authors have provided a very practical recommendation for choosing a robust statistical method.
Tom: They’ve basically shown that instead of chasing the single highest power number, you should choose one from their curated lists based on whether your data is continuous or discrete.
Lu: The implication is that for anyone using these tests, the choice of test should be guided by whether your data is continuous or discrete and which specific test family you are examining.
Meng: For deployment at scale, this means we can't just rely on a single generic goodness-of-fit test; we have to select one from their proposed set tailored to our specific data structure.
Jane: It’s a practical guide for picking the right tool for the job when you're trying to validate statistical assumptions on your data.
Tom: So, even if you use a combination of tests, it still comes down to having at least one method that performs well in your specific scenario.
Lu: The paper confirms that no single test is universally good across all scenarios; power varies dramatically depending on the null hypothesis and alternative being tested.
Meng: And while they do provide specific methods, they also warn you about the limitations, like how chi square tests with a large number of bins are generally not powerful enough.
Jane: So the main takeaway is that when you're selecting a test for your univariate data, look at what the paper says about continuous versus discrete and use their suggested lists to pick from there.
Wolfgang Rolke
University of Puerto Rico - Mayaguez
stat.ME, hep-ex
Submitted: 2024-11-06
Updated: 2026-11-09
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 73/100
The gist: Simulation studies regarding the power of various goodness-of-fit as well as nonparametric two-sample tests for univariate data are presented…
Key concepts
- Goodness-of-Fit (GoF) Tests
- These tests are used to check if observed data matches a specific theoretical distribution. They are applied to univariate data, which can be continuous or discrete. The study compares different GoF methods like Wilson's test and chi-square tests with varying numbers of bins.
- Two-Sample Tests
- These statistical tests compare two independent samples to see if they come from the same underlying distribution. The study examines these tests for both continuous and discrete data, testing methods such as Kuiper's test and Anderson-Darling to assess their effectiveness.
- Power of a Test
- Power refers to a test's ability to correctly reject a false null hypothesis. Simulation results showed that no single method provides consistently good power across all scenarios. Some tests perform very well in certain cases, while others fail badly in others, meaning performance is highly dependent on the specific data and hypothesis.
- Optimal Bin Selection
- For chi-square goodness-of-fit tests, the study recommends using 10 bins as a reasonable compromise for both continuous and discrete data. This number is adjusted based on the number of parameters being estimated, aiming to find a balance between test sensitivity and accuracy.
Terminology
Summary
Simulation studies regarding the power of various goodness-of-fit as well as nonparametric two-sample tests for univariate data are presented, showing that no single method can be relied upon to provide good power Based on the results of these studies we propose a fairly small number of methods chosen such that for any of the case studies included here at least one of the methods has good power. p. 0 In general no single method can be relied upon to provide good power, any one method may
be quite good for some combination of null hypothesis and alternative and may fail badly for another. Based on the results of these studies we propose a fairly small number of methods chosen such that for
any of the case studies included here at least one of the methods has good power.Where the paper says it
Scope and Methodology
The study investigates both goodness-of-fit (gof) and nonparametric two-sample problems for univariate data, encompassing both continuous and discrete data. The methods are implemented using R packages R2sample and Rgof, which support various features such as handling weights, random sample sizes assumed from a Poisson distribution, and large-sample formulas for the two-sample problem. p. 0 • some of the methods allow for data with weights. • the routines allow for a random sample size, assumed to come from a Poisson distribution. Discrete data includes the
case of histogram (aka discretized or binned) data. • the methods are implemented using both Rcpp [Eddelbuettel et al. p. 0 Discrete data includes the
case of histogram (aka discretized or binned) data. • the methods are implemented using both Rcpp [Eddelbuettel et al.Where the paper says it
Where the paper says it
Where the paper says it
Key Findings on Method Performance
The simulation results indicate that no single method can be relied upon to provide good power, as any one method may be quite good for some combination of null hypothesis and alternative and may fail badly for another. Furthermore, all the methods included in the packages achieve the desired type I error rate. p. 1 Quick links: goodness-of-fit, twosample. All the methods included in the packages achieve the desired type I error rate. Chi-square tests with a large number of bins generally have poor power. Care needs to be taken so that
the expected counts in the goodness-of fit problem and the combined observed counts in the twosample
problem are at least 5.Where the paper says it
Where the paper says it
Proposed Method Selection
Based on these simulation results we propose to use a fairly small number of methods chosen such that for any of the case studies included here at least one of the methods has good power. The recommended methods are specified as follows:
-
Goodness-of-fit problem, continuous data: Wilson’s test, Zhang’s ZC, Anderson-Darling and a chi-square test with a small number of equal-size bins. p. 2
Where the paper says it
Based on the results of these studies we propose to use the following methods: Quick links: goodness-of-fit, twosample twosample. (a) Goodness-of-fit problem, continuous data: Wilson’s test, Zhang’s ZC, Anderson-Darling and a chi-square test with a small number of equal-size bins.
-
Goodness-of-fit problem, discrete data: Wilson’s test, Anderson-Darling and a chi-square test with a small number of bins. p. 2
p. 2Where the paper says it
(a) Goodness-of-fit problem, continuous data: Wilson’s test, Zhang’s ZC, Anderson-Darling and a chi-square test with a small number of equal-size bins. (b) Goodness-of-fit problem, discrete data: Wilson’s test, Anderson-Darling and a chi-square test with a small number of bins.
Where the paper says it
(a) Goodness-of-fit problem, continuous data: Wilson’s test, Zhang’s ZC, Anderson-Darling and a chi-square test with a small number of equal-size bins. (b) Goodness-of-fit problem, discrete data: Wilson’s test, Anderson-Darling and a chi-square test with a small number of bins.
-
Two-sample problem, continuous data: Kuiper’s test, Zhang’s ZA and ZK methods, the Wasserstein test as well as a chi-square test with a small number of equal spaced bins. p. 2
p. 2Where the paper says it
(b) Goodness-of-fit problem, discrete data: Wilson’s test, Anderson-Darling and a chi-square test with a small number of bins. (c) Two-sample problem, continuous data: Kuiper’s test, Zhang’s ZA and ZK methods, the Wasserstein test as well as a chi-square test with a small number of equal spaced bins.
Where the paper says it
(b) Goodness-of-fit problem, discrete data: Wilson’s test, Anderson-Darling and a chi-square test with a small number of bins. (c) Two-sample problem, continuous data: Kuiper’s test, Zhang’s ZA and ZK methods, the Wasserstein test as well as a chi-square test with a small number of equal spaced bins.
-
Two-sample problem, discrete data: Kuiper’s test, Anderson-Darling, Zhang’s ZA test as well as a chi-square test with a small number of equal spaced bins. p. 2
p. 2Where the paper says it
(b) Goodness-of-fit problem, discrete data: Wilson’s test, Anderson-Darling and a chi-square test with a small number of bins. (c) Two-sample problem, continuous data: Kuiper’s test, Zhang’s ZA and ZK methods, the Wasserstein test as well as a chi-square test with a small number of equal spaced bins.
Where the paper says it
(b) Goodness-of-fit problem, discrete data: Wilson’s test, Anderson-Darling and a chi-square test with a small number of bins. (c) Two-sample problem, continuous data: Kuiper’s test, Zhang’s ZA and ZK methods, the Wasserstein test as well as a chi-square test with a small number of equal spaced bins.
Test Characteristics and Combinations
Several tests can be combined such that one rejects the null hypothesis if any of the individual tests does so, and it is possible to find a correct p-value for this combination test via simulation. For continuous data, four methods are needed to always have one of them near the best, namely W, ZC, AD and a chi-square test with a small number of bins. p. 2 Based on the results of these studies we propose to use the following methods:
Quick links: goodness-of-fit, twosample twosample. (a) Goodness-of-fit problem, continuous data: Wilson’s test, Zhang’s ZC, Anderson-Darling and a
chi-square test with a small number of equal-size bins. (b) Goodness-of-fit problem, discrete data: Wilson’s test, Anderson-Darling and a chi-square test
with a small number of bins. (c) Two-sample problem, continuous data: Kuiper’s test, Zhang’s ZA and ZK methods, the Wasserstein test as well as a chi-square test with a small number of equal spaced bins.Where the paper says it
Where the paper says it
Data Type Correlation
The performance of a method depends on whether the data is continuous or discrete, but tests that work for both cases are highly correlated. For instance, Kolmogorov-Smirnov has an 86% power for continuous data and a 95% power for discrete data. p. 0 The power studies in this article were carried out using R programs in the packages R2sample and
Rgof, available from the CRAN website. Some tests such as the Kolmogorov-Smirnov test are already
implemented for both problems in base R. These have
edf’s Fb, Gb and Hb, respectively. 1 Chi-Square Tests
In the case of continuous data the routines include eight chi-square tests, with either equal size (ES) or equal
probability (EP) bins, either a large (l=50) or a small (s=10) number of bins and with either the Pearson
(P) or the log-likelihood (L) formula.Where the paper says it
Where the paper says it
Optimal Bin Selection
To determine the optimal number of bins for chi-square goodness-of-fit tests, it is recommended to run them with 10 bins, adjusted for the number of estimated parameters. This recommendation is based on results from rerunning the case studies where the highest power is achieved for a fairly small number of bins. p. 0 In general no single method can be relied upon to provide good power, any one method may
be quite good for some combination of null hypothesis and alternative and may fail badly for another. Based on the results of these studies we propose a fairly small number of methods chosen such that for
any of the case studies included here at least one of the methods has good power. In general no single method can be relied upon to provide good power, any one method may
be quite good for some combination of null hypothesis and alternative and may fail badly for another. Based on the results of these studies we propose a fairly small number of methods chosen such that for
any of the case studies included here at least one of the methods has good power. (a) Goodness-of-fit problem, continuous data: Wilson’s test, Zhang’s ZC, Anderson-Darling and a
chi-square test with a small number of equal-size bins. (b) Goodness-of-fit problem, discrete data: Wilson’s test, Anderson-Darling and a chi-square test
with a small number of bins. (a) Goodness-of-fit problem, continuous data: Wilson’s test, Zhang’s ZC, Anderson-Darling and a
chi-square test with a small number of equal-size bins. (b) Goodness-of-fit problem, discrete data: Wilson’s test, Anderson-Darling and a chi-square test
with a small number of bins.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Two-Sample Test Selection
For the two-sample problem, continuous data suggests that Kuiper’s test, Zhang’s ZA and ZK method, the Wasserstein test as well as a chi-square test with a small number of equal spaced bins should be used. In the discrete case, the selection includes Kuiper’s test, Anderson-Darling, Zhang’s ZA as well as a chi-square test with a small number of equal spaced bins. p. 2 (b) Goodness-of-fit problem, discrete data: Wilson’s test, Anderson-Darling and a chi-square test
with a small number of bins. (c) Two-sample problem, continuous data: Kuiper’s test, Zhang’s ZA and ZK methods, the Wasserstein test as well as a chi-square test with a small number of equal spaced bins. • in the two-sample problem some methods make use of large-sample formulas, therefore allowing for
very large data sets. • there are routines that allow the user to combine several tests and find a corrected p value.Where the paper says it
Where the paper says it
Conclusion on Power
In general, the powers of all methods differ widely, with no clear pattern of which methods are best. In most cases where the best method has a power exceeding 80%, the worst method has a power less than 30%. p. 0 This includes both continuous and
discrete data. In general no single method can be relied upon to provide good power, any one method may
be quite good for some combination of null hypothesis and alternative and may fail badly for another. Quick links: goodness-of-fit, twosample. All the methods included in the packages achieve the desired type I error rate. 1 Case Study 1: Uniform [0,1] - Linear Models
The density of the linear model is given by f(x) = 2sx + 1 − s; 0 < x < 1, so the parameter is the slope of
the line. Figure 1: Uniform vs Linear Models
Figure 1: Power Curves for Uniform vs Linear Models
10 If parameters have to be estimated, this is done via the user-provided routine phat. As long as the method
of estimation used is consistent and efficient and the expected counts are large enough the chi-square statistic
will have a chi-square distribution, as shown by (Fisher 1922) and (Fisher 1924). Based on the results of these studies we propose to use the following methods:
Quick links: goodness-of-fit, twosample twosample. (a) Goodness-of-fit problem, continuous data: Wilson’s test, Zhang’s ZC, Anderson-Darling and a
chi-square test with a small number of equal-size bins. (a) Goodness-of-fit problem, continuous data: Wilson’s test, Zhang’s ZC, Anderson-Darling and a
chi-square test with a small number of equal-size bins. (b) Goodness-of-fit problem, discrete data: Wilson’s test, Anderson-Darling and a chi-square test
with a small number of bins. • the routines can also use any other user-defined tests. • the packages include routines to easily carry out power studies and draw power graphs. In general no single method can be relied upon to provide good power, any one method may
be quite good for some combination of null hypothesis and alternative and may fail badly for another. Based on the results of these studies we propose a fairly small number of methods chosen such that for
any of the case studies included here at least one of the methods has good power. Several tests can be combined as follows: one rejects the null hypothesis if any of the individual tests
does so. It is possible to find a correct p-value for this combination test via simulation. • Goodness-of-Fit Problem - Discrete Data: We have a set of values vals and a vector of counts x. F is a discrete probability distribution, which may depend on unknown parameters.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Two-Sample Test Power
The power studies for the two-sample problem show that different methods exhibit varying levels of performance across various case studies. For example, in Case Study 27: Beta(2,2) - Beta(2,a), the Kuiper’s test has a power of 5. p. 0 Simulation Studies For Goodness-of-Fit and
Two-Sample Methods For Univariate Data
Wolfgang Rolke
University of Puerto Rico - Mayaguez
wolfgang. edu
November 13, 2024
Abstract
We present the results of a large number of simulation studies regarding the power of various goodnessof-fit as well as nonparametric two-sample tests for univariate data. 1 Case Study 1: Uniform [0,1] - Linear Models
The density of the linear model is given by f(x) = 2sx + 1 − s; 0 < x < 1, so the parameter is the slope of
the line. Figure 1: Uniform vs Linear Models
Figure 1: Power Curves for Uniform vs Linear Models
10 (b) Goodness-of-fit problem, discrete data: Wilson’s test, Anderson-Darling and a chi-square test
with a small number of bins. (c) Two-sample problem, continuous data: Kuiper’s test, Zhang’s ZA and ZK methods, the Wasserstein test as well as a chi-square test with a small number of equal spaced bins. Chi-square tests with a large number of bins generally have poor power. Care needs to be taken so that
the expected counts in the goodness-of fit problem and the combined observed counts in the twosample
problem are at least 5.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Final Recommendations
The final recommendations for the methods are summarized as follows:
-
Goodness-of-fit problem, continuous data: Wilson’s test, Zhang’s ZC, Anderson-Darling and a chisquare test with a small number of equal-size bins. p. 2
Where the paper says it
Based on the results of these studies we propose to use the following methods: Quick links: goodness-of-fit, twosample twosample. (a) Goodness-of-fit problem, continuous data: Wilson’s test, Zhang’s ZC, Anderson-Darling and a chi-square test with a small number of equal-size bins.
-
Goodness-of-fit problem, discrete data: Wilson’s test, Anderson-Darling and a chi-square test with a small number of bins.
-
Two-sample problem, continuous data: Kuiper’s test, Zhang’s ZA and ZK methods, the Wasserstein test as well as a chi-square test with a small number of equal spaced bins.
-
Two-sample problem, discrete data: Kuiper’s test, Anderson-Darling, Zhang’s ZA test as well as a chi-square test with a small number of equal spaced bins. The results show that all methods achieve the correct type I error rate of 5% within simulation error. p. 1
Where the paper says it
Quick links: goodness-of-fit, twosample. All the methods included in the packages achieve the desired type I error rate.
References
[Anderson, 1962] Anderson, T. W. (1962). On the distribution of the two-sample Cramer-von Mises criterion. p. 5 The GoF version is discussed in [Cramer, 1928] and
[von Mises, 1928]. The two-sample version was proposed in [Anderson, 1962]. Detailed discussions can be found
in [D’Agostini and Stephens, 1986], [Thas, 2010], [Raynor et al. p. 0 The literature on both of these problems is vast and steadily growing. Detailed discussions can be found
in [D’Agostini and Stephens, 1986], [Thas, 2010], [Raynor et al.Where the paper says it
Where the paper says it
Where the paper says it
[Anderson and Darling, 1952] Anderson, T. p. 5 5 Anderson-Darling (AD)
This test is similar to the Cramer-vonMises test but with an integrand that emphasizes the tails. It was first
proposed in [Anderson and Darling, 1952]. Some tests such as the Kolmogorov-Smirnov test are already
implemented for both problems in base R. Many others can be run through various packages, for example
the Anderson-Darling goodness-of-fit test is available in the R package ADGofTest [Bellosta, 2011]. In general a method that has good power for the continuous data problem will also have good power
for the corresponding discrete (aka histogram) problem) Quick links: goodness-of-fit. Based on the results of these studies we propose to use the following methods:
Quick links: goodness-of-fit, twosample twosample. Detailed discussions can be found
in [D’Agostini and Stephens, 1986], [Thas, 2010], [Raynor et al. For an introduction to Statistics and
hypothesis testing in general see [Casella and Berger, 2002] or [Bickel and Doksum, 2015].Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[Bellosta, 2011] Bellosta, C. p. 0 Some tests such as the Kolmogorov-Smirnov test are already
implemented for both problems in base R. Many others can be run through various packages, for example
the Anderson-Darling goodness-of-fit test is available in the R package ADGofTest [Bellosta, 2011]. Based on the results of these studies we propose to use the following methods:
Quick links: goodness-of-fit, twosample twosample. (a) Goodness-of-fit problem, continuous data: Wilson’s test, Zhang’s ZC, Anderson-Darling and a
chi-square test with a small number of equal-size bins.Where the paper says it
Where the paper says it
[Berkson, 1980] Berkson, J. p. 3 This
method has the desirable feature that if the null hypothesis is rejected for this set of values, it will always
be rejected for any other as well. For a discussion of this estimation method see [Berkson, 1980]. Alternatively we can use the argument ChiUsePhat=FALSE. In that case the value provided by phat is
used as a starting point but the parameters are estimated via the method of minimum chi-square. 1
arXiv:2411. 05839v2 [stat.Where the paper says it
Where the paper says it
Where the paper says it
[Bickel and Doksum, 2015] Bickel, P. p. 0 Detailed discussions can be found
in [D’Agostini and Stephens, 1986], [Thas, 2010], [Raynor et al. For an introduction to Statistics and
hypothesis testing in general see [Casella and Berger, 2002] or [Bickel and Doksum, 2015]. Detailed discussions can be found
in [D’Agostini and Stephens, 1986], [Thas, 2010], [Raynor et al. For an introduction to Statistics and
hypothesis testing in general see [Casella and Berger, 2002] or [Bickel and Doksum, 2015]. Detailed discussions can be found
in [D’Agostini and Stephens, 1986], [Thas, 2010], [Raynor et al. For an introduction to Statistics and
hypothesis testing in general see [Casella and Berger, 2002] or [Bickel and Doksum, 2015].Where the paper says it
Where the paper says it
Where the paper says it
[Casella and Berger, 2002] Casella, G. p. 0 Detailed discussions can be found
in [D’Agostini and Stephens, 1986], [Thas, 2010], [Raynor et al. For an introduction to Statistics and
hypothesis testing in general see [Casella and Berger, 2002] or [Bickel and Doksum, 2015]. Detailed discussions can be found
in [D’Agostini and Stephens, 1986], [Thas, 2010], [Raynor et al. For an introduction to Statistics and
hypothesis testing in general see [Casella and Berger, 2002] or [Bickel and Doksum, 2015].Where the paper says it
Where the paper says it
[Cramer, 1928] Cramer, H. p. 5 4 Cramer-vonMises (CvM)
This test is based on the integrated squared differences. The GoF version is discussed in [Cramer, 1928] and
[von Mises, 1928].Where the paper says it
[D’Agostini and Stephens, 1986] D’Agostini, R. p. 0 The literature on both of these problems is vast and steadily growing. Detailed discussions can be found
in [D’Agostini and Stephens, 1986], [Thas, 2010], [Raynor et al. The literature on both of these problems is vast and steadily growing. Detailed discussions can be found
in [D’Agostini and Stephens, 1986], [Thas, 2010], [Raynor et al. Simulation Studies For Goodness-of-Fit and
Two-Sample Methods For Univariate Data
Wolfgang Rolke
University of Puerto Rico - Mayaguez
wolfgang. edu
November 13, 2024
Abstract
We present the results of a large number of simulation studies regarding the power of various goodnessof-fit as well as nonparametric two-sample tests for univariate data.Where the paper says it
Where the paper says it
Where the paper says it
[Dowd, 2022] Dowd, C. p. 0 Many others can be run through various packages, for example
the Anderson-Darling goodness-of-fit test is available in the R package ADGofTest [Bellosta, 2011]. There
are also packages that allow the user to run several tests, for example the twosamples [Dowd, 2022] package. In this situation the theoretical distribution is continuous
but the data is discrete. For the tests in the two-sample problem p-values are found via the permutation method.Where the paper says it
Where the paper says it
[Eddelbuettel et al., 2024] Eddelbuettel, D. p. 0 • the methods are implemented using both Rcpp [Eddelbuettel et al. , 2024] and parallel programming. Discrete data includes the
case of histogram (aka discretized or binned) data. • the methods are implemented using both Rcpp [Eddelbuettel et al.Where the paper says it
Where the paper says it
[Kolmogorov, 1933] Kolmogorov, A. p. 3 2 Kolmogorov-Smirnov (KS)
This test is based on the largest absolute distance between F and Fb in the goodness-of-fit problem and between Fb and Gb in the two-sample problem. The tests were first proposed in [Kolmogorov, 1933], [Smirnov, 1939]
and are among the most widely used tests today.Where the paper says it
[Kuiper, 1960] Kuiper, N. p. 4 3 Kuiper (K)
This test is closely related to Kolmogorov-Smirnov, but it uses the sum of the largest positive and negative
differences as a test statistic. It was first proposed in [Kuiper, 1960]. • some of the methods allow for data with weights. • the routines allow for a random sample size, assumed to come from a Poisson distribution.Where the paper says it
Where the paper says it
[Lehmann, 1951] Lehmann, E. p. 8 9 Lehmann-Rosenblatt (LR), Two-sample Problem
Let ri and si be the ranks of x and y in the combined sample, then the test statistic is given by
1
nm(n + m)
"
n
Xn
i=1
(ri − 1)2 + m
Xm
i=1
(si − 1)2
#
For details see [Lehmann, 1951] and [Rosenblatt, 1952]. Simulation Studies For Goodness-of-Fit and
Two-Sample Methods For Univariate Data
Wolfgang Rolke
University of Puerto Rico - Mayaguez
wolfgang. edu
November 13, 2024
Abstract
We present the results of a large number of simulation studies regarding the power of various goodnessof-fit as well as nonparametric two-sample tests for univariate data.Where the paper says it
Where the paper says it
[Pettitt, 1976] Pettitt, A. p. 5 It was first
proposed in [Anderson and Darling, 1952]. The two-sample version is discussed in [Pettitt, 1976]. The reason is that they are built on the concept of
ranks, and in the discrete data case there are simply to many ties. 2 Two-sample Problem
We define Rx
j
to be the rank of the j
th order statistic of the x data set in the pooled data set, and R
y
j
accordingly.Where the paper says it
Where the paper says it
[Raynor et al., 2012] Raynor, J. p. 0 The literature on both of these problems is vast and steadily growing. Detailed discussions can be found
in [D’Agostini and Stephens, 1986], [Thas, 2010], [Raynor et al. Simulation Studies For Goodness-of-Fit and
Two-Sample Methods For Univariate Data
Wolfgang Rolke
University of Puerto Rico - Mayaguez
wolfgang. edu
November 13, 2024
Abstract
We present the results of a large number of simulation studies regarding the power of various goodnessof-fit as well as nonparametric two-sample tests for univariate data.Where the paper says it
Where the paper says it
[Rosenblatt, 1952] Rosenblatt, M. p. 8 9 Lehmann-Rosenblatt (LR), Two-sample Problem
Let ri and si be the ranks of x and y in the combined sample, then the test statistic is given by
1
nm(n + m)
"
n
Xn
i=1
(ri − 1)2 + m
Xm
i=1
(si − 1)2
#
For details see [Lehmann, 1951] and [Rosenblatt, 1952]. 4 Cramer-vonMises (CvM)
This test is based on the integrated squared differences. The GoF version is discussed in [Cramer, 1928] and
[von Mises, 1928].Where the paper says it
Where the paper says it
[Smirnov, 1939] Smirnov, N. p. 3 2 Kolmogorov-Smirnov (KS)
This test is based on the largest absolute distance between F and Fb in the goodness-of-fit problem and between Fb and Gb in the two-sample problem. The tests were first proposed in [Kolmogorov, 1933], [Smirnov, 1939]
and are among the most widely used tests today. For their details see the references. In the following we use the following notations:
Goodness-of-fit problem, continuous data: we denote the cumulative distribution functions (cdf) by F,
its empirical distribution function (edf) by Fb.Where the paper says it
Where the paper says it
[Thas, 2010] Thas, O. p. 0 The literature on both of these problems is vast and steadily growing. Detailed discussions can be found
in [D’Agostini and Stephens, 1986], [Thas, 2010], [Raynor et al. We want to test F = G. • Two-sample Problem - Discrete Data: We have a set of values vals and vectors of counts x and
y, drawn from some unknown discrete probability distributions F and G.Where the paper says it
Where the paper says it
[Vaserstein, 1969] Vaserstein, L. p. 6 In the goodness-of-fit problem
for continuous data the user has to supply a function that calculates the inverse of the cdf under the null
hypothesis. For a discussion of the Wasserstein distance see [Vaserstein, 1969].Where the paper says it
[von Mises, 1928] von Mises, R. p. 5 4 Cramer-vonMises (CvM)
This test is based on the integrated squared differences. The GoF version is discussed in [Cramer, 1928] and
[von Mises, 1928].Where the paper says it
[Watson, 1961] Watson, G. p. 7 It adjust that tests statistic via a squared difference
of the mean of Fb(xi) and 0. It was proposed in [Watson, 1961]. Simulation Studies For Goodness-of-Fit and
Two-Sample Methods For Univariate Data
Wolfgang Rolke
University of Puerto Rico - Mayaguez
wolfgang. edu
November 13, 2024
Abstract
We present the results of a large number of simulation studies regarding the power of various goodnessof-fit as well as nonparametric two-sample tests for univariate data.Where the paper says it
Where the paper says it
[Zhang, 2002] Zhang, J. p. 6 For a discussion of the Wasserstein distance see [Vaserstein, 1969]. 7 Zhang’s tests (ZA, ZK and ZC)
These tests were proposed in [Zhang, 2002] and [Zhang, 2006]. 7 Zhang’s tests (ZA, ZK and ZC)
These tests were proposed in [Zhang, 2002] and [Zhang, 2006]. They are variations of test statistics based
on the likelihood ratio.Where the paper says it
Where the paper says it
[Zhang, 2006] Zhang, J. p. 6 For a discussion of the Wasserstein distance see [Vaserstein, 1969]. 7 Zhang’s tests (ZA, ZK and ZC)
These tests were proposed in [Zhang, 2002] and [Zhang, 2006]. 7 Zhang’s tests (ZA, ZK and ZC)
These tests were proposed in [Zhang, 2002] and [Zhang, 2006]. They are variations of test statistics based
on the likelihood ratio.Where the paper says it
Where the paper says it
The optimal number of bins for chi-square goodness-of-fit tests is recommended to be 10, adjusted for the number of estimated parameters. This recommendation is based on results from rerunning the case studies where the highest power is achieved for a fairly small number of bins. The optimal number of bins varies depending on whether the data is continuous or discrete, with 10 bins being a reasonable compromise.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improved hypothesis testing capabilities through method selection: The paper proposes a specific set of methods for each problem type, such as
Goodness-of-fit problem, continuous data: Wilson’s test, Zhang’s ZC, Anderson-Darling and a chi-square test with a small number of equal-size bins.
This allows AI systems to select the most appropriate statistical tool based on the data type (continuous vs. discrete) and problem structure (goodness-of-fit vs. two-sample). -
Enhanced power analysis for model comparison: By providing power curves for various distribution comparisons, AI systems can assess the probability of correctly rejecting a null hypothesis across different scenarios like
Uniform - Linear
orNormal - Stretch,
helping to determine if a specific testing approach is robust under alternative hypotheses. -
Robust handling of discrete data: The paper highlights that
Discrete data includes the case of histogram (aka discretized or binned) data,
and suggests using tests like the chi-square test with a small number of bins for discrete goodness-of-fit, enabling AI systems to perform accurate statistical testing on count data. -
Optimized binning strategy: The recommendation to use
10 bins
as a reasonable compromise for chi-square tests in both continuous and discrete data suggests an optimized approach for discretization, reducing the need for arbitrary bin selection when the true distribution is unknown. -
Combination of tests with adjusted p-values: AI systems can be implemented with routines like
gof test adjusted pvalue
which generate a number of data sets under the null hypothesis to find the smallest p value across selected tests, addressing the issue of simultaneous inference by adjusting for dependence.
Abstract
We present the results of a large number of simulation studies regarding the power of various goodness-of-fit as well as nonparametric two-sample tests for univariate data. This includes both continuous and discrete data. In general no single method can be relied upon to provide good power, any one method may be quite good for some combination of null hypothesis and alternative and may fail badly for another. Based on the results of these studies we propose a fairly small number of methods chosen such that for any of the case studies included here at least one of the methods has good power. The studies were carried out using the R packages R2sample and Rgof, available from CRAN.
Related papers
- Learning Robust Treatment Rules for Censored Data
- Neural Estimation of Summary Statistics for Simulation-Based Inference with Non-Asymptotic Guarantees
- Spectral divide-and-conquer MCMC for long stationary time series
- Improving the Efficiency of Subgroup Analysis in Randomized Controlled Trials with TMLE
- Personalized Federated Vector Autoregression with Personalization Diversity
- Doubly Robust and Efficient Calibration of Prediction Sets for Right-Censored Time-to-Event Outcomes