The Earth Moves, But So Does the Bias: Systematic Upward Bias of the Wasserstein (Earth Mover's) Distance and Permutation-Based Null Calibration

arXiv:2608.09863 · stat.ME, stat.AP · Submitted 2026-08-10 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: We're looking at the paper: "The Earth Moves, But So Does the Bias".

Jane: Here's the gist of the paper. The Earth Mover’s Distance (EMD) is gaining increasing interest among political scientists for assessing similarity in preference distributions, but there remains a risk of finite-sample upward bias induced by sampling variation in empirical probability measures <ref:2608.09863#pg2> This paper proposes a permutation-based null calibration framework for more robust hypothesis testing <ref:2608.09863#pg5> As a non-parametric approach, it frees researchers from making directional or distributional shape assumptions <ref:2608.09863#pg6> The Problem with Standard EMD and Bootstrapping The Earth Mover’s Distance (EMD) suffers from an upward bias due to finite sampling error, where the expected empirical distance is generally positive even when the true population distance is zero <ref:2608.09863#pg7> This problem is common in naive plug-in estimators of non-negative distance metrics, where random noise accumulates rather than cancels out <ref:2608.09863#pg8> Standard bootstrapping only captures the conditional sampling variability of the empirical statistic and does not identify the finite-sample null baseline against which the observed distance should be evaluated <ref:2608.09863#pg10> The bias-corrected estimate can become negative if it falls outside the non-negative parameter space of the EMD metric The Permutation Calibration Framework The proposed framework estimates how large the EMD would be expected to be under the null hypothesis of no group difference This involves estimating the expected magnitude of the EMD under the null hypothesis of no group difference The primary inferential object is the permutation distribution of the EMD statistic under the null hypothesis of exchangeable group labels The permutation-based p-value estimates the probability of observing an EMD at least as large as the empirical EMD if the two groups were in fact drawn from the same underlying distribution Simulation Results and Robustness Checks Four sets of Monte Carlo simulations demonstrate the utility of this framework across continuous and sparse discrete settings The simulations show that the expected EMD increases as the conjoint support becomes more complex and fragmented, illustrating how sparsity exacerbates upward shift The permutation-calibrated tests maintain Type I error rates closest to the nominal 0.

Paper summary: Tom: So wrapping up the "The Earth Moves, But So Does the Bias" paper by Ho Ting Hung and Bosco. The main point is that standard methods for estimating EMD in real data have a known upward bias from sampling, especially in tricky settings like sparse distributions. Jane And they propose this permutation-based null calibration framework as a way to fix that testing problem without having to make restrictive assumptions about the shape of the distribution.

Lu: Essentially, they are giving researchers a non-parametric way to calibrate their tests by estimating what the EMD would be under the null hypothesis of no group difference through shuffling labels. Meng This moves away from relying on standard bootstrap uncertainty because it gives you a specific baseline against which to measure your empirical distance.

Jane: The authors show that this calibration works well in simulations across continuous and sparse discrete settings, and their permutation-calibrated tests maintain type one error rates very close to what they should be. Tom It’s about making the hypothesis test for preference similarity much more rigorous by accounting for this specific sampling bias in a way that doesn't require guessing the distribution shape.

Lu: The implication is that we can use EMD with greater confidence when comparing preference distributions, provided we adopt this calibration framework instead of just using standard bootstrap methods on their own. Jane It gives people a concrete tool to handle the uncertainty inherent in empirical measures without forcing them into rigid directional assumptions about the underlying data structure.

Tom: So, for anyone working on preference comparison, this paper suggests moving toward permutation-based null calibration because it provides a more robust way to test for group differences against that known upward bias. Jane It’s a solid piece of work that gives you a principled method to handle the noise in your measurements when using EMD.

Conclusion: Tom: So we’ve been talking about how standard ways of measuring preference similarity, like Earth Mover’s Distance, have this built-in problem where the numbers are just systematically too high because of random sampling noise.

Jane: Right, so we’re looking at a paper titled "The Earth Moves, But So Does the Bias: Systematic Upward Bias of the Wasserstein (Earth Mover's) Distance and Permutation-Based Null Calibration."

Lu: That paper tackles this bias head-on by introducing a framework that uses shuffling labels to figure out what the expected distance should be when there’s actually no difference between groups.

Meng: So, instead of just trusting our empirical distance measurement, they’re creating a null baseline using permutations. What does that actually mean for someone trying to compare two datasets?

Tom: It means you can use this calibration to get a much more honest p-value when you test if two groups are different. It lets you account for that inherent upward shift in the data without having to make big assumptions about the true shape of your underlying preference distribution.

Jane: I think the authors show that this permutation method works well whether you're looking at continuous preferences or something more fragmented, like sparse discrete choices. They found their tests keep their error rates right where they should be—close to one percent error when you expect zero difference.

Lu: The main point is that this isn't just a fancy statistical trick; it’s a way to make the comparison of preference distributions much more robust against the noise we get from just taking a sample. It moves us away from those naive plug-in estimators you see everywhere.

Meng: For deployment, that robustness matters. If we’re building an AI system that relies on these distance metrics, knowing how to calibrate that baseline properly prevents us from flagging real differences as just sampling error.

Tom: Exactly. It gives us a principled method for testing group differences in preference similarity, making sure our conclusions are based on the actual signal and not just random fluctuations in the data collection process.

Jane: So, it’s about taking a common metric that has this known flaw and giving researchers a better tool to measure it accurately. Next up, we’re going to look at how this kind of calibration shows up in real-world preference studies involving human behavior.

University of Oxford

stat.ME, stat.AP

Submitted: 2026-08-10

Updated: 2026-11-05

Comments: This version was submitted to Political Analysis (R&R) on October 12, 2026. The first submission was made on May 21, 2026. It was accepted on November 4, 2026

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 98/100

The gist: The Earth Mover’s Distance (EMD) is gaining increasing interest among political scientists for assessing similarity in preference distributions…

Key concepts

Earth Mover’s Distance (EMD)
A metric used to measure the similarity between two probability distributions by calculating the minimum cost required to transform one distribution into the other. It quantifies how much 'work' is needed to move mass from one set of points to another.
Upward Bias
A systematic error where empirical estimates of a non-negative distance metric consistently yield values higher than the true population value, even when there is no real difference between the groups being compared. This bias arises from random sampling noise.
Permutation Calibration Framework
A statistical method that uses resampling (permutations) to estimate what the EMD would look like if there were truly no difference between groups. This calibration allows researchers to establish a proper null baseline for their tests, making them more reliable.

Terminology

Summary

The Earth Mover’s Distance (EMD) is gaining increasing interest among political scientists for assessing similarity in preference distributions, but there remains a risk of finite-sample upward bias induced by sampling variation in empirical probability measures <ref:2608. p. 009863#pg2> This paper proposes a permutation-based null calibration framework for more robust hypothesis testing <ref:2608. p. 009863#pg5> As a non-parametric approach, it frees researchers from making directional or distributional shape assumptions <ref:2608. p. 009863#pg6>

The Problem with Standard EMD and Bootstrapping

The Earth Mover’s Distance (EMD) suffers from an upward bias due to finite sampling error, where the expected empirical distance is generally positive even when the true population distance is zero <ref:2608.09863#pg7> This problem is common in naive plug-in estimators of non-negative distance metrics, where random noise accumulates rather than cancels out <ref:2608. p. 309863#pg8> Standard bootstrapping only captures the conditional sampling variability of the empirical statistic and does not identify the finite-sample null baseline against which the observed distance should be evaluated <ref:2608. p. 309863#pg10> The bias-corrected estimate can become negative if it falls outside the non-negative parameter space of the EMD metric

The Permutation Calibration Framework

The proposed framework estimates how large the EMD would be expected to be under the null hypothesis of no group difference This involves estimating the expected magnitude of the EMD under the null hypothesis of no group difference The primary inferential object is the permutation distribution of the EMD statistic under the null hypothesis of exchangeable group labels The permutation-based p-value estimates the probability of observing an EMD at least as large as the empirical EMD if the two groups were in fact drawn from the same underlying distribution

Simulation Results and Robustness Checks

Four sets of Monte Carlo simulations demonstrate the utility of this framework across continuous and sparse discrete settings The simulations show that the expected EMD increases as the conjoint support becomes more complex and fragmented, illustrating how sparsity exacerbates upward shift The permutation-calibrated tests maintain Type I error rates closest to the nominal 0.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Robust hypothesis testing for preference distributions can be achieved by using a permutation-based null calibration framework instead of standard bootstrap uncertainty bounds, as this approach allows researchers to distinguish true distributional divergence from the positive EMD noise under the null.

  2. The improved system can perform more reliable inference on empirical Earth Mover's Distance estimates by estimating how large the EMD would be expected to be under the null hypothesis of no group difference, which establishes a finite-sample null baseline for interpretation.

  3. The system can better handle sparse data structures, such as those found in conjoint experiments, by using the permutation distribution of EMD to create a calibrated p-value that accounts for finite-sample fluctuation, sample-size imbalance, and sparsity in the observed support.

  4. The AI system can provide more accurate statistical power assessments across different scenarios by using permutation-based tests, as these tests are shown to control the Type I error rate at level alpha while retaining sensitivity to sufficiently pronounced differences.

  5. The system can offer a more methodologically defensible foundation for empirical evaluations of congruence and preference distributions by treating the permutation distribution as a foundational inferential object that maps and calibrates against the inherent finite-sample upward bias.

  6. For distance metrics like the 2-Wasserstein distance, the system can calculate an excess EMD statistic using a formula like Wr,excess(p, q) = max (0, Wr(p, q) − Enull[Wr(p˜, q˜)]) to quantify how far observed distances exceed what is expected from finite-sample noise.

Sources

Related papers