Fair Box ordinate transform for forecasts following a multivariate Gaussian law

arXiv:2506.22601 · stat.ME, stat.AP · Submitted 2025-06-27 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: We're looking at the paper: "Fair Box ordinate transform for forecasts following a multivariate Gaussian law".

Jane: Here's the gist of the paper. Monte Carlo techniques are the method of choice for making probabilistic predictions of an outcome in several disciplines.

Tom: The fair Gaussian sample BOT is capable of correctly identifying miscalibration when the sample size is moderate in contrast to the alternative BOT versions.

Jane: This approach addresses the need to reliably assess calibration even when the sample size n is moderate, which can be a limitation for other methods.

Tom: Derivation of the Fair Transform The fair version of the Gaussian sample BOT, denoted as uF n m, S; x0, is derived by considering the squared Mahalanobis distance D 2:= x0 - m T S-1 x0 - m.

Title and authors: Tom: This paper introduces this Fair Box ordinate transform, and the authors are Sándor Baran and Martin Leutbecher from the Faculty of Informatics at the University of Debrecen in Hungary.

Jane: So, what does this mean for us practically? It suggests a way to reliably test calibration even when we can't afford massive sample sizes for our Monte Carlo simulations.

Lu: The core idea is that they derived a formula for this Fair Box ordinate transform based on the squared Mahalanobis distance, which is related to how far off our forecast is from the true mean and covariance structure >

Meng: So, if we can use this formula to check calibration with a smaller sample size n, that lowers the barrier for using these checks in our real-time operational pipeline.

Lalam: It addresses the need to reliably assess calibration even when the sample size n is moderate, which can be a limitation for other methods >

The paper's summary: Tom: The paper summarizes that they found this Fair Box ordinate transform is capable of correctly identifying miscalibration when the sample size n is moderate, which contrasts with alternative versions of the Box ordinate transform.

Jane: So, even if your ensemble size isn't huge, this method can still tell you if your forecasts are calibrated correctly or not >

Lu: For calibrated forecasts, they found that both their Fair Box ordinate transform and the theoretical Box ordinate transform result in completely flat histograms with p-values of zero point three six zero two and zero point one one nine five >

Meng: A flat histogram means the forecast is behaving as expected under a Gaussian distribution, which is a clear sign of good calibration for us to look for >

Lalam: It confirms that when forecasts are calibrated, the Fair Box ordinate transform gives us that flat histogram regardless of the ensemble size n >

The paper's improvements: Tom: The authors point out several improvements, and one major one is how they handle different types of misspecification, like variance underforecast or overforecast.

Jane: They show that for Type one miscalibration, the distances D values for this Fair Box ordinate transform are around zero point two for the underforecast variance and around zero point one three for the overforecast variance case >

Lu: And they also show that even in cases of severe misspecification, like when the size to dimension ratio is very low, this Fair Box ordinate transform still performs well even when other versions fail >

Meng: That’s interesting because it suggests robustness against some kinds of structural errors in our forecast models, not just sample size limitations >

Lalam: They also suggest that the shape of these Fair Box ordinate transform histograms mimics the forms of their theoretical counterparts in misspecified cases, which gives us qualitative insight into forecast quality even when classical sample Box ordinate transform versions are biased >

Conclusion: Tom: So to wrap up, the introduction of this Fair Box transform suggests that a reliable forecast of the multivariate normal distribution will result in a flat histogram of Fair Box ordinate transform values regardless of ensemble size.

Jane: The authors argue that the shape of these Fair Box ordinate transform histograms is qualitatively similar to that of the theoretical Box ordinate transform for several unreliable scenarios, which isn't true for classical sample Box ordinate transform versions >

Lu: This leads them to suggest combining this Fair Box transform histogram with univariate rank histograms and lower-dimensional subspace Fair Box ordinates to reduce the likelihood of falsely concluding reliability >

Meng: So it’s not a magic fix, but rather a way to use this new tool alongside other checks to make sure we don't get fooled by false confidence in our models >

Lalam: Ultimately, the introduction of the Fair Box transform allows us to use a reliable forecast of the multivariate normal distribution to get a flat histogram of Fair Box ordinate transform values regardless of ensemble size >

Faculty of Informatics, University of Debrecen · European Centre for Medium-Range Weather Forecasts

stat.ME, stat.AP

Submitted: 2025-06-27

Updated: 2025-06-27

Comments: 37 pages, 24 figures

Journal ref: Quarterly Journal of the Royal Meteorological Society (2026), paper e70351

DOI: 10.1002/qj.70351

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 76/100

The gist: Monte Carlo techniques are the method of choice for making probabilistic predictions of an outcome in several disciplines.

Key concepts

Fair Sample BOT (uF_n m, S; x0)
This is a specific transformation derived from the squared Mahalanobis distance used to create a fair version of the Gaussian sample BOT. It is designed so that if a forecast truly follows a multivariate Gaussian law, its distribution will be a standard uniform distribution for any ensemble size (n), allowing for reliable calibration checking.
Calibration Misspecification
This refers to errors where the model's predicted probabilities do not match the actual observed frequency of outcomes. The paper tests how the fair BOT handles different types of these errors, such as underforecasting variance or overforecasting variance, showing its robustness against these issues.
Multivariate Gaussian Law
This is the assumption that a set of related variables (like weather quantities) follows a specific probability distribution where all possible combinations are described by a single normal distribution. The paper uses this ideal law to establish what a perfectly calibrated forecast should look like when analyzed by the fair BOT.
Flat Histogram
When analyzing the results using the fair BOT, if the forecast is perfectly calibrated and follows a multivariate Gaussian law, the resulting histogram of values will be completely flat. This flatness is a key indicator that no specific bias or systematic error exists in how often certain outcomes occur.

Terminology

Summary

Monte Carlo techniques are the method of choice for making probabilistic predictions of an outcome in several disciplines. p. 0 The fair Gaussian sample BOT is capable of correctly identifying miscalibration when the sample size is moderate in contrast to the alternative BOT versions. p. 0 This approach addresses the need to reliably assess calibration even when the sample size n is moderate, which can be a limitation for other methods. p. 0

Derivation of the Fair Transform

The fair version of the Gaussian sample BOT, denoted as uF n m, S; x0, is derived by considering the squared Mahalanobis distance D 2:= x0 - m T S-1 x0 - m. This leads to the formula uF n m, S; x0 = 1 − Fp,n-p n(n − p) p(n squared − 1) x0 - m T S-1 x0 - m. p. 4 This formula ensures that for calibrated forecasts, the distribution of u is standard uniform for any ensemble size n. p. 3

Simulation Study on Misspecification

The simulation study explores the behavior of the fair sample BOT (2.5) across various dimensions p, ensemble sizes n, and types of calibration misspecification. p. 4 The results confirm that the fair sample BOT correctly identifies miscalibration when the sample size is moderate in contrast to alternative versions. p. 0 Specifically, for calibrated forecasts, the fair sample BOT and theoretical BOT result in completely flat histograms with p-values of 0. p. 83602 and 0. p. 81195.

Performance Under Different Scenarios

The performance of the fair sample BOT is tested under various conditions, including different types of covariance structure misspecification and bias in the mean vector. For instance, when considering Type 1 miscalibration (variance underforecast), the D values for the fair sample BOT are around 0. p. 82 for the underforecast variance and around 0. p. 813 for the overforecast variance case. p. 5 Furthermore, in cases of severe misspecification, such as a very low size to dimension ratio, the fair sample BOT still performs well even when other versions fail. p. 2

Operational Forecast Verification

The paper verifies the properties of the fair sample BOT using operational ECMWF ensemble forecasts for various weather quantities and lead times. For perfectly reliable situations, the fair sample BOT histograms are almost perfectly flat for all investigated ensemble sizes. p. 0 When verified against analyses, the fair sample BOT histograms correctly reflect the situation when the forecast is nearly calibrated and samples a Gaussian law. p. 2

Conclusion on Robustness

The introduction of the fair BOT suggests that a reliable forecast of the multivariate normal distribution will result in a flat histogram of fair BOT values regardless of ensemble size. The shape of the fair sample BOT histograms is qualitatively similar to that of the theoretical BOT histograms for several unreliable scenarios, which is not true for classical sample BOT versions. p. 5 This suggests combining the fair BOT histogram with univariate rank histograms and lower-dimensional subspace fair BOT histograms to reduce the likelihood of falsely concluding reliability. p. 8 The introduced fair BOT can also be utilized to verify post-processed multivariate forecasts. p. 2

References

Allen, S., Ziegel, J. and Ginsbourger, D. (2024) Assessing the calibration of multivariate probabilistic forecasts. p. 1

Baran, S., Hemri, S. p. 1 p. 1 p. 1 and El Ayari, M. (2019) Statistical postprocessing of water level forecasts using bayesian model averaging with doubly truncated normal components. p. 1

Ben Bouallegue, Z. (2025) Casting light on dependency structures in ensemble forecasts with the 2-d rank histogram. p. 1

Berrocal, V. p. 2 J., Raftery, A. p. 1 E., Gneiting, T. p. 1 and Steed, R. C. (2010) Probabilistic weather forecasting for winter road maintenance. p. 2

Buizza, R. (2018a) Introduction to the special issue on “25 years of ensemble forecasting”. p. 1

Buizza, R. (2018b) Ensemble forecasting and the need for calibration. p. 1

Chen, J., Janke, T., Steinke, F. and Lerch, S. (2024) Generative machine learning methods for multivariate ensemble postprocessing. p. 1

Dai, Y. and Hemri, S. p. 1 (2021) Spatially coherent postprocessing of cloud cover ensemble forecasts. p. 1

Dietz, S. J., Kneringer, P., Mayr, G. J. and Zeileis, A. (2019) Low-visibility forecasts for different flight planning horizons using tree-based boosting models. p. 1

Duan, Q., Ajami, N. K., Gao, X. and Sorooshian, S. (2007) Multi-model ensemble hydrologic prediction using bayesian model averaging. p. 1

Good, I. J. (1952) Rational decisions. p. 1

Hamill, T. M. (2001) Interpretation of rank histograms for verifying ensemble forecasts. p. 2

H¨ardle, W. p. 3 K. and Simar, L. p. 3 (2019) Applied Multivariate Statistical Analysis. p. 2

Hemri, S., Fundel, F. and Zappa, M. (2013) Simultaneous calibration of ensemble river flow predictions over an entire range of lead times. p. 1

Henze, N. and Zirkler, B. (1990) A class of invariant consistent tests for multivariate normality. p. 2

Lakatos, M., Lerch, S., Hemri, S. and Baran, S. p. 0 p. 0 (2023) Comparison of multivariate postprocessing methods using global ECMWF ensemble forecasts. p. 0

Leutbecher, M. p. 1 p. 1 and Baran, S. (2025) Ensemble size dependence of the logarithmic score for forecasts issued as multivariate normal distributions. p. 2

Leutbecher, M. and Palmer, T. p. 1 N. (2008) Ensemble forecasting. p. 1

Lewis, J. M. (2005) Roots of ensemble forecasting. p. 1

Price, I., Sanchez-Gonzalez, A., Alet, F., Andersson, T. R., El-Kadi, A., Masters, D., Ewalds, T., Stott, J., Mohamed, S., Battaglia, P., Lam, R. and Willson, M. (2025) Probabilistic weather forecasting with machine learning. p. 1

Richardson, C. p. 4 W. (1981) Stochastic simulation of daily precipitation, temperature, and solar radiation. p. 4

Schefzik, R. p. 2 (2016) Combining parametric low-dimensional ensemble postprocessing with reordering methods. p. 1

Siegert, S. p. 1, Ferro, C. p. 1 A. T., Stephenson, D.. B. and Leutbecher, M. p. 1 (2019) The ensembleadjusted ignorance score for forecasts issued as normal distributions. p. 1

Thorarinsdottir, T. p. 2 L., Scheuerer, M. and Heinz, C. (2506.226018pg3>.

Vannitsem, S. p. 1, Bremnes, J. B., Demaeyer, J., Evans, G. R., Flowerdew, J., Hemri, S., Lerch, S., Roberts, N., Theis, S., Atencia, A., Ben Bouallegue, Z. B., Bhend, J., Dabernig, M., Cruz, L. D., Hieta, L.), 2021) Statistical postprocessing for weather forecasts – review. p. 1

Wilks, D. S. (2017) On assessing calibration of multivariate ensemble forecasts. p. 1

Wilks, D. S. (2019) Statistical Methods in the Atmospheric Sciences. p. 1

--- Page 34 ---

D=0.1423 D=0.1462 D=0.1804 D=0.1429 Fair Sample BOT Theoretical BOT Naive Sample BOT Adjusted Sample BOT

--- Page 35 ---

D=0.1148 D=0.1207 D=0.1540 D=0.1168 Fair Sample BOT Theoretical BOT Naive Sample BOT Adjusted Sample BOT

--- Page 36 ---

D=2389 D=3684 9395 2487 Fair Sample BOT Theoretical BOT Naive Sample BOT Adjusted Sample BOT

--- Page 37 ---

D=1108 D=1462 3215 1288 Fair Sample BOT Theoretical BOT Naive Sample BOT Adjusted Sample BOT

--- Page 30 ---

D=0.0095 D=0.0108 D=0.2112

--- Page 36 ---

D=5131 D=7046 9753 438.

--- Page 37 ---

D=0.1429 D=0.1462 D=0.1804 Fair Sample BOT Theoretical BOT Naive Sample BOT Adjusted Sample BOT

--- Page 37 ---

D=0.1578 D=0.2070 D=3705 1706 Fair Sample BOT Theoretical BOT Naive Sample BOT Adjusted Sample BOT

--- Page 36 ---

D=1288 D=1423 D=1462 1804 Fair Sample BOT Theoretical BOT Naive Sample BOT Adjusted Sample BOT

--- Page 35 ---

D=0.0053 D=0.0076 D=0.2119 Fair Sample BOT Theoretical BOT Naive Sample BOT Adjusted Sample BOT

--- Page 27 ---

D=1148 D=1207 D=1548 1288 Fair Sample BOT Theoretical BOT Naive Sample BOT Adjusted Sample BOT

--- Page 30 ---

D=0.0966 D=0.1231 D=0.3044 Fair Sample BOT Theoretical BOT Naive Sample BOT Adjusted Sample BOT

--- Page 27 ---

D=1122 D=1203 D=1588 1129 Fair Sample BOT Theoretical BOT Naive Sample BOT Adjusted Sample BOT

--- Page 36 ---

D=0.0896 D=0.1207 D=0.3008 Fair Sample BOT Theoretical BOT Naive Sample BOT Adjusted Sample BOT

--- Page 27 ---

D=1147 D=1213 D=1538 1136 Fair Sample BOT Theoretical BOT Naive Sample BOT Adjusted Sample BOT

--- Page 29 ---

D=0.0872 D=0.1185 D=0.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Fair Box ordinate transform for forecasts following a multivariate Gaussian law can be used to assess calibration even when sample size n is moderate, which addresses a limitation of alternative BOT versions noted in Section 1.

  2. The fair Gaussian sample BOT is capable of correctly identifying miscalibration when the sample size is moderate in contrast to the alternative BOT versions, confirming its reliability for iterative model development where cost limits sample size.

  3. A reliable forecast of the multivariate normal distribution will result in a flat histogram of fair BOT values regardless of ensemble size, which provides a robust calibration indicator across different ensemble sizes and dimensions.

  4. The fair sample BOT histograms mimic the forms of their theoretical counterparts in misspecified cases, suggesting that for certain miscalibrations, it provides qualitative insight into forecast quality even when classical sample BOT versions are biased.

  5. The introduced fair BOT can be utilized to verify post-processed multivariate forecasts by combining it with univariate rank histograms for marginal distributions and fair BOT histograms for lower-dimensional subspaces, reducing the likelihood of falsely concluding reliability.

Sources

Related papers