Simulation Studies For Goodness-of-Fit and Two-Sample Methods For Univariate Data

arXiv:2411.05839 · stat.ME, hep-ex · Submitted 2024-11-06 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Simulation Studies For Goodness-of-Fit and Two-Sample Methods For Univariate Data".

Jane: Here's the gist of the paper. Simulation studies regarding the power of various goodness-of-fit as well as nonparametric two-sample tests for univariate data are presented, showing that no single method can be relied upon to provide good power Based on the results of these studies we propose a fairly small number of methods chosen such that for any of the case studies included here at least one of the methods has good power.

Tom: Scope and Methodology The study investigates both goodness-of-fit (gof) and nonparametric two-sample problems for univariate data, encompassing both continuous and discrete data.

Jane: The methods are implemented using R packages R2sample and Rgof, which support various features such as handling weights, random sample sizes assumed from a Poisson distribution, and large-sample formulas for the two-sample problem.

Tom: Discrete data includes histogram (aka discretized or binned) data, and the methods are implemented using both Rcpp and parallel programming.

Paper summary: Jane: We just talked about how the authors ran simulations to see which goodness-of-fit and two-sample tests worked best, and they concluded that no single method is universally good for everything. The core thesis of "Simulation Studies For Goodness-of-Fit and Two-Sample Methods For Univariate Data" is that you need a small, carefully chosen set of methods so that for any case study they included, at least one method will have good power.

Tom: Exactly. They're not claiming one test is the best; they’re saying if you use this curated list of tests—like Wilson’s test or Anderson-Darling—you have a better chance than just picking some random statistical tool.

Lu: The methodology involved using R packages R2sample and Rgof to run a large number of simulation studies covering both continuous and discrete data problems.

Meng: They explicitly included tests for discrete data because there weren't many implementations in R at the time, which made it possible to test those scenarios directly.

Jane: And they also addressed the issue of combining tests; they found that you can combine several of these individual tests so that one rejects the null hypothesis if any of them do so, and they could even find a correct p value for this combination test via simulation.

Tom: That’s a neat feature, because it means you don't have to commit to just one test and hope for the best; you can use a portfolio approach.

Lu: They also found that continuous data problems tend to have good power in the corresponding discrete histogram problem, which is interesting for cross-domain analysis.

Meng: But they did flag that chi square tests with a large number of bins are weak, and they gave specific advice on how many bins to use for a reasonable result.

Jane: So, it’s really about giving practitioners actionable advice on what to use when they have a continuous or discrete dataset, moving beyond just the abstract statistical theory.

Tom: It boils down to being practical: if you're dealing with continuous data, you pick from their list of four recommended methods.

Lu: And if you are dealing with discrete data, you pick from another list of three recommended methods.

Meng: This gives us a clear decision tree for deployment based on the data structure we encounter in production environments.

Jane: So the main point is that power isn't guaranteed by any one method alone; it depends on the specific problem setup.

Tom: And they confirm that within simulation error, all methods maintain the five percent type one error rate, which keeps them statistically honest.

Conclusion: Jane: To wrap up this discussion on "Simulation Studies For Goodness-of-Fit and Two-Sample Methods For Univariate Data," the authors have provided a very practical recommendation for choosing a robust statistical method.

Tom: They’ve basically shown that instead of chasing the single highest power number, you should choose one from their curated lists based on whether your data is continuous or discrete.

Lu: The implication is that for anyone using these tests, the choice of test should be guided by whether your data is continuous or discrete and which specific test family you are examining.

Meng: For deployment at scale, this means we can't just rely on a single generic goodness-of-fit test; we have to select one from their proposed set tailored to our specific data structure.

Jane: It’s a practical guide for picking the right tool for the job when you're trying to validate statistical assumptions on your data.

Tom: So, even if you use a combination of tests, it still comes down to having at least one method that performs well in your specific scenario.

Lu: The paper confirms that no single test is universally good across all scenarios; power varies dramatically depending on the null hypothesis and alternative being tested.

Meng: And while they do provide specific methods, they also warn you about the limitations, like how chi square tests with a large number of bins are generally not powerful enough.

Jane: So the main takeaway is that when you're selecting a test for your univariate data, look at what the paper says about continuous versus discrete and use their suggested lists to pick from there.

Wolfgang Rolke

University of Puerto Rico - Mayaguez

stat.ME, hep-ex

Submitted: 2024-11-06

Updated: 2026-11-09

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 73/100

The gist: Simulation studies regarding the power of various goodness-of-fit as well as nonparametric two-sample tests for univariate data are presented…

Key concepts

Goodness-of-Fit (GoF) Tests
These tests are used to check if observed data matches a specific theoretical distribution. They are applied to univariate data, which can be continuous or discrete. The study compares different GoF methods like Wilson's test and chi-square tests with varying numbers of bins.
Two-Sample Tests
These statistical tests compare two independent samples to see if they come from the same underlying distribution. The study examines these tests for both continuous and discrete data, testing methods such as Kuiper's test and Anderson-Darling to assess their effectiveness.
Power of a Test
Power refers to a test's ability to correctly reject a false null hypothesis. Simulation results showed that no single method provides consistently good power across all scenarios. Some tests perform very well in certain cases, while others fail badly in others, meaning performance is highly dependent on the specific data and hypothesis.
Optimal Bin Selection
For chi-square goodness-of-fit tests, the study recommends using 10 bins as a reasonable compromise for both continuous and discrete data. This number is adjusted based on the number of parameters being estimated, aiming to find a balance between test sensitivity and accuracy.

Terminology

Summary

Simulation studies regarding the power of various goodness-of-fit as well as nonparametric two-sample tests for univariate data are presented, showing that no single method can be relied upon to provide good power Based on the results of these studies we propose a fairly small number of methods chosen such that for any of the case studies included here at least one of the methods has good power. p. 0

Where the paper says it

In general no single method can be relied upon to provide good power, any one method may be quite good for some combination of null hypothesis and alternative and may fail badly for another. Based on the results of these studies we propose a fairly small number of methods chosen such that for any of the case studies included here at least one of the methods has good power.

Scope and Methodology

The study investigates both goodness-of-fit (gof) and nonparametric two-sample problems for univariate data, encompassing both continuous and discrete data. The methods are implemented using R packages R2sample and Rgof, which support various features such as handling weights, random sample sizes assumed from a Poisson distribution, and large-sample formulas for the two-sample problem. p. 0

Where the paper says it

• some of the methods allow for data with weights. • the routines allow for a random sample size, assumed to come from a Poisson distribution.

Discrete data includes histogram (aka discretized or binned) data, and the methods are implemented using both Rcpp and parallel programming. p. 0
Where the paper says it

Discrete data includes the case of histogram (aka discretized or binned) data. • the methods are implemented using both Rcpp [Eddelbuettel et al. p. 0

Where the paper says it

Discrete data includes the case of histogram (aka discretized or binned) data. • the methods are implemented using both Rcpp [Eddelbuettel et al.

Key Findings on Method Performance

The simulation results indicate that no single method can be relied upon to provide good power, as any one method may be quite good for some combination of null hypothesis and alternative and may fail badly for another. Furthermore, all the methods included in the packages achieve the desired type I error rate. p. 1

Where the paper says it

Quick links: goodness-of-fit, twosample. All the methods included in the packages achieve the desired type I error rate.

A key finding is that chi-square tests with a large number of bins generally have poor power, and care needs to be taken so that the expected counts in the goodness-of-fit problem and the combined observed counts in the twosample problem are at least 5. p. 2
Where the paper says it

Chi-square tests with a large number of bins generally have poor power. Care needs to be taken so that the expected counts in the goodness-of fit problem and the combined observed counts in the twosample problem are at least 5.

Proposed Method Selection

Based on these simulation results we propose to use a fairly small number of methods chosen such that for any of the case studies included here at least one of the methods has good power. The recommended methods are specified as follows:

  1. Goodness-of-fit problem, continuous data: Wilson’s test, Zhang’s ZC, Anderson-Darling and a chi-square test with a small number of equal-size bins. p. 2

    Where the paper says it

    Based on the results of these studies we propose to use the following methods: Quick links: goodness-of-fit, twosample twosample. (a) Goodness-of-fit problem, continuous data: Wilson’s test, Zhang’s ZC, Anderson-Darling and a chi-square test with a small number of equal-size bins.

  2. Goodness-of-fit problem, discrete data: Wilson’s test, Anderson-Darling and a chi-square test with a small number of bins. p. 2

    Where the paper says it

    (a) Goodness-of-fit problem, continuous data: Wilson’s test, Zhang’s ZC, Anderson-Darling and a chi-square test with a small number of equal-size bins. (b) Goodness-of-fit problem, discrete data: Wilson’s test, Anderson-Darling and a chi-square test with a small number of bins.

    p. 2
    Where the paper says it

    (a) Goodness-of-fit problem, continuous data: Wilson’s test, Zhang’s ZC, Anderson-Darling and a chi-square test with a small number of equal-size bins. (b) Goodness-of-fit problem, discrete data: Wilson’s test, Anderson-Darling and a chi-square test with a small number of bins.

  3. Two-sample problem, continuous data: Kuiper’s test, Zhang’s ZA and ZK methods, the Wasserstein test as well as a chi-square test with a small number of equal spaced bins. p. 2

    Where the paper says it

    (b) Goodness-of-fit problem, discrete data: Wilson’s test, Anderson-Darling and a chi-square test with a small number of bins. (c) Two-sample problem, continuous data: Kuiper’s test, Zhang’s ZA and ZK methods, the Wasserstein test as well as a chi-square test with a small number of equal spaced bins.

    p. 2
    Where the paper says it

    (b) Goodness-of-fit problem, discrete data: Wilson’s test, Anderson-Darling and a chi-square test with a small number of bins. (c) Two-sample problem, continuous data: Kuiper’s test, Zhang’s ZA and ZK methods, the Wasserstein test as well as a chi-square test with a small number of equal spaced bins.

  4. Two-sample problem, discrete data: Kuiper’s test, Anderson-Darling, Zhang’s ZA test as well as a chi-square test with a small number of equal spaced bins. p. 2

    Where the paper says it

    (b) Goodness-of-fit problem, discrete data: Wilson’s test, Anderson-Darling and a chi-square test with a small number of bins. (c) Two-sample problem, continuous data: Kuiper’s test, Zhang’s ZA and ZK methods, the Wasserstein test as well as a chi-square test with a small number of equal spaced bins.

    p. 2
    Where the paper says it

    (b) Goodness-of-fit problem, discrete data: Wilson’s test, Anderson-Darling and a chi-square test with a small number of bins. (c) Two-sample problem, continuous data: Kuiper’s test, Zhang’s ZA and ZK methods, the Wasserstein test as well as a chi-square test with a small number of equal spaced bins.

Test Characteristics and Combinations

Several tests can be combined such that one rejects the null hypothesis if any of the individual tests does so, and it is possible to find a correct p-value for this combination test via simulation. For continuous data, four methods are needed to always have one of them near the best, namely W, ZC, AD and a chi-square test with a small number of bins. p. 2

Where the paper says it

Based on the results of these studies we propose to use the following methods: Quick links: goodness-of-fit, twosample twosample. (a) Goodness-of-fit problem, continuous data: Wilson’s test, Zhang’s ZC, Anderson-Darling and a chi-square test with a small number of equal-size bins.

In the discrete case, three methods suffice: W, AD and a chi-square test with a small number of bins. p. 2
Where the paper says it

(b) Goodness-of-fit problem, discrete data: Wilson’s test, Anderson-Darling and a chi-square test with a small number of bins. (c) Two-sample problem, continuous data: Kuiper’s test, Zhang’s ZA and ZK methods, the Wasserstein test as well as a chi-square test with a small number of equal spaced bins.

Data Type Correlation

The performance of a method depends on whether the data is continuous or discrete, but tests that work for both cases are highly correlated. For instance, Kolmogorov-Smirnov has an 86% power for continuous data and a 95% power for discrete data. p. 0

Where the paper says it

The power studies in this article were carried out using R programs in the packages R2sample and Rgof, available from the CRAN website. Some tests such as the Kolmogorov-Smirnov test are already implemented for both problems in base R.

The chi-square tests with either a large or a small number of bins are also highly correlated between the continuous and discrete versions. p. 2
Where the paper says it

These have edf’s Fb, Gb and Hb, respectively. 1 Chi-Square Tests In the case of continuous data the routines include eight chi-square tests, with either equal size (ES) or equal probability (EP) bins, either a large (l=50) or a small (s=10) number of bins and with either the Pearson (P) or the log-likelihood (L) formula.

Optimal Bin Selection

To determine the optimal number of bins for chi-square goodness-of-fit tests, it is recommended to run them with 10 bins, adjusted for the number of estimated parameters. This recommendation is based on results from rerunning the case studies where the highest power is achieved for a fairly small number of bins. p. 0

Where the paper says it

In general no single method can be relied upon to provide good power, any one method may be quite good for some combination of null hypothesis and alternative and may fail badly for another. Based on the results of these studies we propose a fairly small number of methods chosen such that for any of the case studies included here at least one of the methods has good power.

p. 0
Where the paper says it

In general no single method can be relied upon to provide good power, any one method may be quite good for some combination of null hypothesis and alternative and may fail badly for another. Based on the results of these studies we propose a fairly small number of methods chosen such that for any of the case studies included here at least one of the methods has good power.

The optimal number of bins varies depending on whether the data is continuous or discrete, with 10 bins being a reasonable compromise. p. 2
Where the paper says it

(a) Goodness-of-fit problem, continuous data: Wilson’s test, Zhang’s ZC, Anderson-Darling and a chi-square test with a small number of equal-size bins. (b) Goodness-of-fit problem, discrete data: Wilson’s test, Anderson-Darling and a chi-square test with a small number of bins.

p. 2
Where the paper says it

(a) Goodness-of-fit problem, continuous data: Wilson’s test, Zhang’s ZC, Anderson-Darling and a chi-square test with a small number of equal-size bins. (b) Goodness-of-fit problem, discrete data: Wilson’s test, Anderson-Darling and a chi-square test with a small number of bins.

Two-Sample Test Selection

For the two-sample problem, continuous data suggests that Kuiper’s test, Zhang’s ZA and ZK method, the Wasserstein test as well as a chi-square test with a small number of equal spaced bins should be used. In the discrete case, the selection includes Kuiper’s test, Anderson-Darling, Zhang’s ZA as well as a chi-square test with a small number of equal spaced bins. p. 2

Where the paper says it

(b) Goodness-of-fit problem, discrete data: Wilson’s test, Anderson-Darling and a chi-square test with a small number of bins. (c) Two-sample problem, continuous data: Kuiper’s test, Zhang’s ZA and ZK methods, the Wasserstein test as well as a chi-square test with a small number of equal spaced bins.

The routine twosample test adjusted pvalues can be used to combine several tests. p. 1
Where the paper says it

• in the two-sample problem some methods make use of large-sample formulas, therefore allowing for very large data sets. • there are routines that allow the user to combine several tests and find a corrected p value.

Conclusion on Power

In general, the powers of all methods differ widely, with no clear pattern of which methods are best. In most cases where the best method has a power exceeding 80%, the worst method has a power less than 30%. p. 0

Where the paper says it

This includes both continuous and discrete data. In general no single method can be relied upon to provide good power, any one method may be quite good for some combination of null hypothesis and alternative and may fail badly for another.

The simulation studies confirm that all methods achieve the correct type I error rate of 5% within simulation error. p. 1
Where the paper says it

Quick links: goodness-of-fit, twosample. All the methods included in the packages achieve the desired type I error rate.

For example, in continuous data, the KS test achieves a power of 74 for Uniform - Linear models. p. 9
Where the paper says it

1 Case Study 1: Uniform [0,1] - Linear Models The density of the linear model is given by f(x) = 2sx + 1 − s; 0 < x < 1, so the parameter is the slope of the line. Figure 1: Uniform vs Linear Models Figure 1: Power Curves for Uniform vs Linear Models 10

The power comparison shows that the method with the highest power is shown in blue and the method with the lowest power is shown in red. p. 2
Where the paper says it

If parameters have to be estimated, this is done via the user-provided routine phat. As long as the method of estimation used is consistent and efficient and the expected counts are large enough the chi-square statistic will have a chi-square distribution, as shown by (Fisher 1922) and (Fisher 1924).

The best default selection for continuous data includes W, ZC, AD and a chi-square test with a small number of bins. p. 2
Where the paper says it

Based on the results of these studies we propose to use the following methods: Quick links: goodness-of-fit, twosample twosample. (a) Goodness-of-fit problem, continuous data: Wilson’s test, Zhang’s ZC, Anderson-Darling and a chi-square test with a small number of equal-size bins.

The best default selection for discrete data includes W, AD and a chi-square test with a small number of bins. p. 2
Where the paper says it

(a) Goodness-of-fit problem, continuous data: Wilson’s test, Zhang’s ZC, Anderson-Darling and a chi-square test with a small number of equal-size bins. (b) Goodness-of-fit problem, discrete data: Wilson’s test, Anderson-Darling and a chi-square test with a small number of bins.

The power comparison shows that the power graphs above the curves never cross. p. 1
Where the paper says it

• the routines can also use any other user-defined tests. • the packages include routines to easily carry out power studies and draw power graphs.

Any one method can perform very well in one case and very poorly in another. p. 0
Where the paper says it

In general no single method can be relied upon to provide good power, any one method may be quite good for some combination of null hypothesis and alternative and may fail badly for another. Based on the results of these studies we propose a fairly small number of methods chosen such that for any of the case studies included here at least one of the methods has good power.

The procedure for combining several tests requires generating about 1000 data sets under the null hypothesis to find the smallest p-value. p. 2
Where the paper says it

Several tests can be combined as follows: one rejects the null hypothesis if any of the individual tests does so. It is possible to find a correct p-value for this combination test via simulation.

The transformed p values are then applied using the Probability Integral Transform to achieve a uniform distribution. p. 1
Where the paper says it

• Goodness-of-Fit Problem - Discrete Data: We have a set of values vals and a vector of counts x. F is a discrete probability distribution, which may depend on unknown parameters.

Two-Sample Test Power

The power studies for the two-sample problem show that different methods exhibit varying levels of performance across various case studies. For example, in Case Study 27: Beta(2,2) - Beta(2,a), the Kuiper’s test has a power of 5. p. 0

Where the paper says it

Simulation Studies For Goodness-of-Fit and Two-Sample Methods For Univariate Data Wolfgang Rolke University of Puerto Rico - Mayaguez wolfgang. edu November 13, 2024 Abstract We present the results of a large number of simulation studies regarding the power of various goodnessof-fit as well as nonparametric two-sample tests for univariate data.

p. 9
Where the paper says it

1 Case Study 1: Uniform [0,1] - Linear Models The density of the linear model is given by f(x) = 2sx + 1 − s; 0 < x < 1, so the parameter is the slope of the line. Figure 1: Uniform vs Linear Models Figure 1: Power Curves for Uniform vs Linear Models 10

0. The Wasserstein test shows a power of 4. p. 2
Where the paper says it

(b) Goodness-of-fit problem, discrete data: Wilson’s test, Anderson-Darling and a chi-square test with a small number of bins. (c) Two-sample problem, continuous data: Kuiper’s test, Zhang’s ZA and ZK methods, the Wasserstein test as well as a chi-square test with a small number of equal spaced bins.

8. The chi-square - Noncentral Chisq. p. 2
Where the paper says it

Chi-square tests with a large number of bins generally have poor power. Care needs to be taken so that the expected counts in the goodness-of fit problem and the combined observed counts in the twosample problem are at least 5.

has a power of 5.2.

Final Recommendations

The final recommendations for the methods are summarized as follows:

  1. Goodness-of-fit problem, continuous data: Wilson’s test, Zhang’s ZC, Anderson-Darling and a chisquare test with a small number of equal-size bins. p. 2

    Where the paper says it

    Based on the results of these studies we propose to use the following methods: Quick links: goodness-of-fit, twosample twosample. (a) Goodness-of-fit problem, continuous data: Wilson’s test, Zhang’s ZC, Anderson-Darling and a chi-square test with a small number of equal-size bins.

  2. Goodness-of-fit problem, discrete data: Wilson’s test, Anderson-Darling and a chi-square test with a small number of bins.

  3. Two-sample problem, continuous data: Kuiper’s test, Zhang’s ZA and ZK methods, the Wasserstein test as well as a chi-square test with a small number of equal spaced bins.

  4. Two-sample problem, discrete data: Kuiper’s test, Anderson-Darling, Zhang’s ZA test as well as a chi-square test with a small number of equal spaced bins. The results show that all methods achieve the correct type I error rate of 5% within simulation error. p. 1

    Where the paper says it

    Quick links: goodness-of-fit, twosample. All the methods included in the packages achieve the desired type I error rate.

References

[Anderson, 1962] Anderson, T. W. (1962). On the distribution of the two-sample Cramer-von Mises criterion. p. 5

Where the paper says it

The GoF version is discussed in [Cramer, 1928] and [von Mises, 1928]. The two-sample version was proposed in [Anderson, 1962].

Annals of Mathematical Statistics, 33(3):1148–1159. p. 0
Where the paper says it

Detailed discussions can be found in [D’Agostini and Stephens, 1986], [Thas, 2010], [Raynor et al. p. 0

Where the paper says it

The literature on both of these problems is vast and steadily growing. Detailed discussions can be found in [D’Agostini and Stephens, 1986], [Thas, 2010], [Raynor et al.

For an introduction to Statistics and hypothesis testing in general see [Casella and Berger, 2002] or [Bickel and Doksum, 2015].

[Anderson and Darling, 1952] Anderson, T. p. 5

Where the paper says it

5 Anderson-Darling (AD) This test is similar to the Cramer-vonMises test but with an integrand that emphasizes the tails. It was first proposed in [Anderson and Darling, 1952].

W. and Darling, D. p. 0
Where the paper says it

Some tests such as the Kolmogorov-Smirnov test are already implemented for both problems in base R. Many others can be run through various packages, for example the Anderson-Darling goodness-of-fit test is available in the R package ADGofTest [Bellosta, 2011].

A. (1952). Asymptotic theory of certain goodness-of-fit criteria based on stochastic processes. p. 2
Where the paper says it

In general a method that has good power for the continuous data problem will also have good power for the corresponding discrete (aka histogram) problem) Quick links: goodness-of-fit. Based on the results of these studies we propose to use the following methods: Quick links: goodness-of-fit, twosample twosample.

Annals of Mathematical Statistics, 23:193–212. p. 0
Where the paper says it

Detailed discussions can be found in [D’Agostini and Stephens, 1986], [Thas, 2010], [Raynor et al. For an introduction to Statistics and hypothesis testing in general see [Casella and Berger, 2002] or [Bickel and Doksum, 2015].

[Bellosta, 2011] Bellosta, C. p. 0

Where the paper says it

Some tests such as the Kolmogorov-Smirnov test are already implemented for both problems in base R. Many others can be run through various packages, for example the Anderson-Darling goodness-of-fit test is available in the R package ADGofTest [Bellosta, 2011].

G. (2011). Anderson-Darling GoF test with p-value calculation based on Marsaglia’s 2004 paper Evaluating the Anderson-Darling Distribution. p. 2
Where the paper says it

Based on the results of these studies we propose to use the following methods: Quick links: goodness-of-fit, twosample twosample. (a) Goodness-of-fit problem, continuous data: Wilson’s test, Zhang’s ZC, Anderson-Darling and a chi-square test with a small number of equal-size bins.

[Berkson, 1980] Berkson, J. p. 3

Where the paper says it

This method has the desirable feature that if the null hypothesis is rejected for this set of values, it will always be rejected for any other as well. For a discussion of this estimation method see [Berkson, 1980].

(1980). Minimum chi-square, not maximum likelihood. p. 3
Where the paper says it

Alternatively we can use the argument ChiUsePhat=FALSE. In that case the value provided by phat is used as a starting point but the parameters are estimated via the method of minimum chi-square.

Ann. Math. Stat, 8(3):457–487. p. 0
Where the paper says it

1 arXiv:2411. 05839v2 [stat.

[Bickel and Doksum, 2015] Bickel, P. p. 0

Where the paper says it

Detailed discussions can be found in [D’Agostini and Stephens, 1986], [Thas, 2010], [Raynor et al. For an introduction to Statistics and hypothesis testing in general see [Casella and Berger, 2002] or [Bickel and Doksum, 2015].

J. and Doksum, K. p. 0
Where the paper says it

Detailed discussions can be found in [D’Agostini and Stephens, 1986], [Thas, 2010], [Raynor et al. For an introduction to Statistics and hypothesis testing in general see [Casella and Berger, 2002] or [Bickel and Doksum, 2015].

A., Mathematical Statistics Vol 1 and 2. p. 0
Where the paper says it

Detailed discussions can be found in [D’Agostini and Stephens, 1986], [Thas, 2010], [Raynor et al. For an introduction to Statistics and hypothesis testing in general see [Casella and Berger, 2002] or [Bickel and Doksum, 2015].

[Casella and Berger, 2002] Casella, G. p. 0

Where the paper says it

Detailed discussions can be found in [D’Agostini and Stephens, 1986], [Thas, 2010], [Raynor et al. For an introduction to Statistics and hypothesis testing in general see [Casella and Berger, 2002] or [Bickel and Doksum, 2015].

and Berger, R. p. 0
Where the paper says it

Detailed discussions can be found in [D’Agostini and Stephens, 1986], [Thas, 2010], [Raynor et al. For an introduction to Statistics and hypothesis testing in general see [Casella and Berger, 2002] or [Bickel and Doksum, 2015].

, Statistical Inference.

[Cramer, 1928] Cramer, H. p. 5

Where the paper says it

4 Cramer-vonMises (CvM) This test is based on the integrated squared differences. The GoF version is discussed in [Cramer, 1928] and [von Mises, 1928].

(1928). On the composition of elementary errors.

[D’Agostini and Stephens, 1986] D’Agostini, R. p. 0

Where the paper says it

The literature on both of these problems is vast and steadily growing. Detailed discussions can be found in [D’Agostini and Stephens, 1986], [Thas, 2010], [Raynor et al.

B. and Stephens, M. p. 0
Where the paper says it

The literature on both of these problems is vast and steadily growing. Detailed discussions can be found in [D’Agostini and Stephens, 1986], [Thas, 2010], [Raynor et al.

A., Goodness-of-Fit Techniques. p. 0
Where the paper says it

Simulation Studies For Goodness-of-Fit and Two-Sample Methods For Univariate Data Wolfgang Rolke University of Puerto Rico - Mayaguez wolfgang. edu November 13, 2024 Abstract We present the results of a large number of simulation studies regarding the power of various goodnessof-fit as well as nonparametric two-sample tests for univariate data.

[Dowd, 2022] Dowd, C. p. 0

Where the paper says it

Many others can be run through various packages, for example the Anderson-Darling goodness-of-fit test is available in the R package ADGofTest [Bellosta, 2011]. There are also packages that allow the user to run several tests, for example the twosamples [Dowd, 2022] package.

, twosamples: Fast Permutation Based Two Sample Tests. p. 1
Where the paper says it

In this situation the theoretical distribution is continuous but the data is discrete. For the tests in the two-sample problem p-values are found via the permutation method.

[Eddelbuettel et al., 2024] Eddelbuettel, D. p. 0

Where the paper says it

• the methods are implemented using both Rcpp [Eddelbuettel et al. , 2024] and parallel programming.

, Francois, R., Allaire, J., Ushey, K., Kou, Q., Russell, N., Ucar, I., Bates, D. and Chambers, J. (2024). Rcpp: Seamless R and C++ Integration. p. 0
Where the paper says it

Discrete data includes the case of histogram (aka discretized or binned) data. • the methods are implemented using both Rcpp [Eddelbuettel et al.

[Kolmogorov, 1933] Kolmogorov, A. p. 3

Where the paper says it

2 Kolmogorov-Smirnov (KS) This test is based on the largest absolute distance between F and Fb in the goodness-of-fit problem and between Fb and Gb in the two-sample problem. The tests were first proposed in [Kolmogorov, 1933], [Smirnov, 1939] and are among the most widely used tests today.

(1933). Sulla determinazione empirica di una legge di distribuzione.

[Kuiper, 1960] Kuiper, N. p. 4

Where the paper says it

3 Kuiper (K) This test is closely related to Kolmogorov-Smirnov, but it uses the sum of the largest positive and negative differences as a test statistic. It was first proposed in [Kuiper, 1960].

H. (1960). Tests concerning random points on a circle. p. 0
Where the paper says it

• some of the methods allow for data with weights. • the routines allow for a random sample size, assumed to come from a Poisson distribution.

[Lehmann, 1951] Lehmann, E. p. 8

Where the paper says it

9 Lehmann-Rosenblatt (LR), Two-sample Problem Let ri and si be the ranks of x and y in the combined sample, then the test statistic is given by 1 nm(n + m) " n Xn i=1 (ri − 1)2 + m Xm i=1 (si − 1)2 # For details see [Lehmann, 1951] and [Rosenblatt, 1952].

, Consistency and unbiasedness of certain nonparametric tests. p. 0
Where the paper says it

Simulation Studies For Goodness-of-Fit and Two-Sample Methods For Univariate Data Wolfgang Rolke University of Puerto Rico - Mayaguez wolfgang. edu November 13, 2024 Abstract We present the results of a large number of simulation studies regarding the power of various goodnessof-fit as well as nonparametric two-sample tests for univariate data.

[Pettitt, 1976] Pettitt, A. p. 5

Where the paper says it

It was first proposed in [Anderson and Darling, 1952]. The two-sample version is discussed in [Pettitt, 1976].

, A two-sample anderson-darling rank statistic. p. 7
Where the paper says it

The reason is that they are built on the concept of ranks, and in the discrete data case there are simply to many ties. 2 Two-sample Problem We define Rx j to be the rank of the j th order statistic of the x data set in the pooled data set, and R y j accordingly.

[Raynor et al., 2012] Raynor, J. p. 0

Where the paper says it

The literature on both of these problems is vast and steadily growing. Detailed discussions can be found in [D’Agostini and Stephens, 1986], [Thas, 2010], [Raynor et al.

C., Thas, O., and Best, D. J., Smooth Tests of Goodness of Fit. p. 0
Where the paper says it

Simulation Studies For Goodness-of-Fit and Two-Sample Methods For Univariate Data Wolfgang Rolke University of Puerto Rico - Mayaguez wolfgang. edu November 13, 2024 Abstract We present the results of a large number of simulation studies regarding the power of various goodnessof-fit as well as nonparametric two-sample tests for univariate data.

[Rosenblatt, 1952] Rosenblatt, M. p. 8

Where the paper says it

9 Lehmann-Rosenblatt (LR), Two-sample Problem Let ri and si be the ranks of x and y in the combined sample, then the test statistic is given by 1 nm(n + m) " n Xn i=1 (ri − 1)2 + m Xm i=1 (si − 1)2 # For details see [Lehmann, 1951] and [Rosenblatt, 1952].

, Limit theorems associated with variants of the von mises statistic. p. 5
Where the paper says it

4 Cramer-vonMises (CvM) This test is based on the integrated squared differences. The GoF version is discussed in [Cramer, 1928] and [von Mises, 1928].

[Smirnov, 1939] Smirnov, N. p. 3

Where the paper says it

2 Kolmogorov-Smirnov (KS) This test is based on the largest absolute distance between F and Fb in the goodness-of-fit problem and between Fb and Gb in the two-sample problem. The tests were first proposed in [Kolmogorov, 1933], [Smirnov, 1939] and are among the most widely used tests today.

, Estimate of deviation between empirical distribution functions in two independent samples. p. 2
Where the paper says it

For their details see the references. In the following we use the following notations: Goodness-of-fit problem, continuous data: we denote the cumulative distribution functions (cdf) by F, its empirical distribution function (edf) by Fb.

[Thas, 2010] Thas, O. p. 0

Where the paper says it

The literature on both of these problems is vast and steadily growing. Detailed discussions can be found in [D’Agostini and Stephens, 1986], [Thas, 2010], [Raynor et al.

, Continuous Distributions. p. 1
Where the paper says it

We want to test F = G. • Two-sample Problem - Discrete Data: We have a set of values vals and vectors of counts x and y, drawn from some unknown discrete probability distributions F and G.

[Vaserstein, 1969] Vaserstein, L. p. 6

Where the paper says it

In the goodness-of-fit problem for continuous data the user has to supply a function that calculates the inverse of the cdf under the null hypothesis. For a discussion of the Wasserstein distance see [Vaserstein, 1969].

N., Markov processes over denumerable products of spaces, describing large systems of automata.

[von Mises, 1928] von Mises, R. p. 5

Where the paper says it

4 Cramer-vonMises (CvM) This test is based on the integrated squared differences. The GoF version is discussed in [Cramer, 1928] and [von Mises, 1928].

E., Wahrscheinlichkeit, Statistik und Wahrheit.

[Watson, 1961] Watson, G. p. 7

Where the paper says it

It adjust that tests statistic via a squared difference of the mean of Fb(xi) and 0. It was proposed in [Watson, 1961].

S., Goodness-of-fit tests on a circle. p. 0
Where the paper says it

Simulation Studies For Goodness-of-Fit and Two-Sample Methods For Univariate Data Wolfgang Rolke University of Puerto Rico - Mayaguez wolfgang. edu November 13, 2024 Abstract We present the results of a large number of simulation studies regarding the power of various goodnessof-fit as well as nonparametric two-sample tests for univariate data.

[Zhang, 2002] Zhang, J. p. 6

Where the paper says it

For a discussion of the Wasserstein distance see [Vaserstein, 1969]. 7 Zhang’s tests (ZA, ZK and ZC) These tests were proposed in [Zhang, 2002] and [Zhang, 2006].

, Powerful goodness-of-fit tests based on likelihood ratio. p. 6
Where the paper says it

7 Zhang’s tests (ZA, ZK and ZC) These tests were proposed in [Zhang, 2002] and [Zhang, 2006]. They are variations of test statistics based on the likelihood ratio.

[Zhang, 2006] Zhang, J. p. 6

Where the paper says it

For a discussion of the Wasserstein distance see [Vaserstein, 1969]. 7 Zhang’s tests (ZA, ZK and ZC) These tests were proposed in [Zhang, 2002] and [Zhang, 2006].

, Powerful two-sample tests based on the likelihood ratio. p. 6
Where the paper says it

7 Zhang’s tests (ZA, ZK and ZC) These tests were proposed in [Zhang, 2002] and [Zhang, 2006]. They are variations of test statistics based on the likelihood ratio.

The optimal number of bins for chi-square goodness-of-fit tests is recommended to be 10, adjusted for the number of estimated parameters. This recommendation is based on results from rerunning the case studies where the highest power is achieved for a fairly small number of bins. The optimal number of bins varies depending on whether the data is continuous or discrete, with 10 bins being a reasonable compromise.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improved hypothesis testing capabilities through method selection: The paper proposes a specific set of methods for each problem type, such as Goodness-of-fit problem, continuous data: Wilson’s test, Zhang’s ZC, Anderson-Darling and a chi-square test with a small number of equal-size bins. This allows AI systems to select the most appropriate statistical tool based on the data type (continuous vs. discrete) and problem structure (goodness-of-fit vs. two-sample).

  2. Enhanced power analysis for model comparison: By providing power curves for various distribution comparisons, AI systems can assess the probability of correctly rejecting a null hypothesis across different scenarios like Uniform - Linear or Normal - Stretch, helping to determine if a specific testing approach is robust under alternative hypotheses.

  3. Robust handling of discrete data: The paper highlights that Discrete data includes the case of histogram (aka discretized or binned) data, and suggests using tests like the chi-square test with a small number of bins for discrete goodness-of-fit, enabling AI systems to perform accurate statistical testing on count data.

  4. Optimized binning strategy: The recommendation to use 10 bins as a reasonable compromise for chi-square tests in both continuous and discrete data suggests an optimized approach for discretization, reducing the need for arbitrary bin selection when the true distribution is unknown.

  5. Combination of tests with adjusted p-values: AI systems can be implemented with routines like gof test adjusted pvalue which generate a number of data sets under the null hypothesis to find the smallest p value across selected tests, addressing the issue of simultaneous inference by adjusting for dependence.

Abstract

We present the results of a large number of simulation studies regarding the power of various goodness-of-fit as well as nonparametric two-sample tests for univariate data. This includes both continuous and discrete data. In general no single method can be relied upon to provide good power, any one method may be quite good for some combination of null hypothesis and alternative and may fail badly for another. Based on the results of these studies we propose a fairly small number of methods chosen such that for any of the case studies included here at least one of the methods has good power. The studies were carried out using the R packages R2sample and Rgof, available from CRAN.

Related papers