A generalisation of the signal-to-noise ratio using proper scoring rules

arXiv:2510.26809 · stat.AP, physics.ao-ph · Submitted 2025-10-23 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: We're looking at the paper: "A generalisation of the signal-to-noise ratio using proper scoring rules".

Jane: Here's the gist of the paper. A generalised concept of the signal-to-noise ratio is provided, based on proper scoring rules rather than merely linear correlations This definition is the natural generalisation of the classical RPC, yet it allows one to define and analyse the signal-to-noise properties of any type of forecast that is amenable to scoring, thus drastically widening the applicability of these concepts Generalization Framework The paper proposes a generalisation of the Ratio of Predictable Components (RPC) by considering instead the ratio of two skill scores, namely the skill score of the forecast against a verification drawn from the forecast, versus the skill score of the perfectly recalibrated forecast against the actual verifications This proposed ratio is denoted as RSSs and can be applied to any type of forecast that is amenable to scoring.

Tom: The authors introduce proper scoring rules, which provide a coherent framework for forecast evaluation.

Jane: Classical Definition Comparison The classical definition of the signal-to-noise ratio refers to the ratio of predictable components or RPC, and the signal-to-noise ratio is defined as anomalous if RPC > 1.

Tom: The authors show that this generalized definition agrees with the classical definition under specific circumstances, such as when using the mean-squared error of the forecast mean and when the forecast means are in a linear relationship.

Title and authors: Tom: So we're looking at this paper called "A generalisation of the signal-to-noise ratio using proper scoring rules". Jane, what's the main idea here in plain English?

Jane: Well, Tom, it takes that old idea of the signal-to-noise ratio and makes it much broader. Instead of just looking at one specific way to measure how good a forecast is—like just comparing the average prediction to reality—this paper suggests using proper scoring rules to look at the signal in any type of forecast that can be scored.

Lu: That's right. It takes the classical ratio of predictable components, which we call RPC, and generalizes it into something called RSSs. The goal is to define a measure of signal quality that works for any forecasting setup you can score against verification data <ref:2510.26809#pg1>.

Meng: So it's not just about one formula anymore? It's adaptable to different kinds of forecasts, right? That sounds useful for what we actually do when we try to deploy models.

Tom: Exactly. The classical definition has limitations because it ties the signal-to-noise ratio too closely to linear relationships between means, but this new concept doesn't have those same restrictions <ref:2510.26809#pg1>.

Jane: It’s about defining a coherent framework for evaluation, using these proper scoring rules, which give us a consistent way to judge quality across different forecasting tasks.

The paper's summary: Tom: So what does the paper actually show in terms of results? We're talking about how this generalized concept compares to the old RPC definition.

Lu: They show that this RSS ratio is basically the natural extension of the classical RPC, meaning it agrees with it under certain conditions, like when you're using the mean-squared error of the forecast mean <ref:2510.26809#pg1>.

Jane: But there’s a key finding here: Theorem one proves that if the RSS ratio is greater than one, then the RPC ratio is also greater than one, depending on which scoring rule you pick <ref:2510.26809#pg1>.

Meng: Okay, so it's not just a theoretical agreement; it’s a practical tool for diagnosis. If we can measure this RSS and see it’s high, we know the forecast is anomalous, regardless of the specific scoring rule we chose <ref:2510.26809#pg3>.

Tom: Right. And they test this on both synthetic data and real seasonal hindcasts for things like the North Atlantic Oscillation index <ref:2510.26809#pg3>.

Jane: The numerical tests are pretty telling. For synthetic data with a normal signal-to-noise ratio, where RPC equals one, the ratios of skill scores using both the continuous ranked probability score and the binary logarithmic score don't give a strong signal that things are anomalous <ref:2510.26809#pg3>.

Lu: However, when they use synthetic data with an anomalous signal-to-noise ratio, where RPC is one point five two, the skill score ratios for both scoring rules are typically larger than one, which gives a strong indication of that anomalous SNR <ref:2510.26809#pg3>.

The paper's improvements: Tom: So what are the actual suggested improvements or the future work they point toward in this paper? We need to know what's next for this concept.

Lu: They conclude that the RSS basically delivers the same message as the classical RPC, which validates its use as a generalisation, but they also found that different statistics have very different variances <ref:2510.26809#pg3>.

Jane: That’s important because it means you can't just rely on one statistic to tell you if the signal is normal or anomalous; the statistical properties of these quantities are quite sensitive <ref:2510.26809#pg3>.

Meng: From a deployment standpoint, if we use different scoring rules, like CRPS versus the logarithmic score, we see these different statistical variances, which means we have to be careful about which metric we're using for production monitoring <ref:2510.26809#pg3>.

Tom: They also mention that a future area is analyzing the sampling properties of these scoring rule based RSSs, and another is finding better ways to estimate pi, which represents the optimally recalibrated forecast <ref:2510.26809#pg3>.

Jane: And they suggest we need more powerful methods for estimating that optimal recalibrated forecast, because that seems to be a key missing piece in fully understanding the signal quality here.

Conclusion: Tom: So to wrap things up, what’s the big picture takeaway from this whole paper on "A generalisation of the signal-to-noise ratio using proper scoring rules"?

Jane: Basically, it gives us a way to define the signal-to-noise ratio based on proper scoring rules instead of just simple linear correlations. It shows that this definition is a natural generalization of the classical RPC, meaning it can be applied to almost any forecast you can score against verification data <ref:2510.26809#pg1>.

Lu: The core message is that the RSS statistic basically delivers the same diagnostic information as the classical RPC, which proves its suitability as a generalisation for wider application <ref:2510.26809#pg3>.

Meng: But we have to keep in mind that just because the statistics give you an indication of anomaly, it’s not enough; you have to understand the variance differences between them <ref:2510.26809#pg3>.

Lalam: I am confident this paper is solid because it connects different scoring rules and shows how they behave under different conditions, which helps us diagnose anomalous signal-to-noise ratios reliably <ref:2510.26809#pg3>.

Tom: Yeah, we've seen how this new RSS statistic works by looking at the synthetic data and the NAO hindcasts in this paper. We'll keep an eye on those future directions, especially figuring out better ways to estimate pi <ref:2510.26809#pg3>.

Jane: That’s what we’re going to do next, looking at how these RSS statistics behave under different sampling properties, which is a really interesting angle for us to explore.

Department of Meteorology and Department of Mathematics and Statistics, University of Reading · Centre for the Mathematics of Planet Earth, University of Reading · National Centre for Earth Observation

stat.AP, physics.ao-ph

Submitted: 2025-10-23

Updated: 2026-03-28

Comments: 19 pages, 2 figures, 3 tables

Journal ref: Meteorological Applications, 33(4), 2026, e70223

DOI: 10.1002/met.70223

Code: https://github.com/eviatarbach/snp

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 82/100

The gist: A generalised concept of the signal-to-noise ratio is provided, based on proper scoring rules rather than merely linear correlations This definition is the natural generalisation of the classical…

Key concepts

Ratio of Predictable Components (RPC)
The classical definition where the signal-to-noise ratio is determined by comparing the predictable components of a forecast. It is considered anomalous if RPC > 1, suggesting the forecast has more predictable structure than expected.
Ratio of Skill Scores (RSS)
A generalized measure comparing two skill scores: one based on a verification drawn from the forecast, and another based on a perfectly recalibrated forecast against actual verifications. This ratio is applied to any score-based forecast, making it more versatile than the classical RPC.
Proper Scoring Rules
A coherent framework for evaluating forecasts where the score is determined by comparing the forecast to an actual outcome. These rules provide a structured way to assess forecast quality beyond simple error metrics.
Anomalous Signal-to-Noise Ratio
A condition identified when the RSS is greater than 1. This indicates that the forecast exhibits signal properties beyond what is expected from random noise or normal predictability, suggesting unusual forecasting behavior.

Terminology

Summary

A generalised concept of the signal-to-noise ratio is provided, based on proper scoring rules rather than merely linear correlations This definition is the natural generalisation of the classical RPC, yet it allows one to define and analyse the signal-to-noise properties of any type of forecast that is amenable to scoring, thus drastically widening the applicability of these concepts

Generalization Framework

The paper proposes a generalisation of the Ratio of Predictable Components (RPC) by considering instead the ratio of two skill scores, namely the skill score of the forecast against a verification drawn from the forecast, versus the skill score of the perfectly recalibrated forecast against the actual verifications This proposed ratio is denoted as RSSs and can be applied to any type of forecast that is amenable to scoring. The authors introduce proper scoring rules, which provide a coherent framework for forecast evaluation. p. 1

Where the paper says it

In common with previous authors, skill score in this paper refers to the score of a forecast, relative to the score of a reference forecast, typically a climatological forecast. To score forecasts we use proper scoring rules (Bröcker and Smith, 2007a), as these provide a coherent framework for forecast evaluation.

Classical Definition Comparison

The classical definition of the signal-to-noise ratio refers to the ratio of predictable components or RPC, and the signal-to-noise ratio is defined as anomalous if RPC > 1. The authors show that this generalized definition agrees with the classical definition under specific circumstances, such as when using the mean-squared error of the forecast mean and when the forecast means are in a linear relationship. p. 5

Where the paper says it

We will show that the generalised and the classical definitions agree under the following circumstances (to be made precise below): 1. For the scoring rule in Definition 1 we use the mean-squared error of the forecast mean.

Theorem 1 proves that RPC ≥ 1 if and only if RSSσ ≥ 1, where RSSσ is understood with respect to the specific scoring rule σ (f, y):= (y − m(f))2. p. 6
Where the paper says it

m(π) = a + b m(f ), with a, b determined so as to minimise Å( (Y − m(π) )2 ). Then RPC ≥ 1 if and only if RSSσ ≥ 1, where RSSσ is understood with respect to the specific scoring rule σ (f , y ) := (y − m(f ) )2 .

Numerical Validation

Numerical experiments are carried out using both synthetic data with prescribed signal-to-noise ratios as well as seasonal ensemble hindcasts of the North Atlantic Oscillation (NAO) index. The results show that for synthetic data with a normal signal-to-noise ratio (RPC = 1), the ratios of skill scores for both the CRPS and the binary logarithmic score fail to give a strong indication of an anomalous SNR, with RSS values within the 95% bootstrap confidence intervals. p. 0

Where the paper says it

The methodology is illustrated through numerical examples of ensemble forecasts, scored using the continuous ranked probability score (CRPS), and of probability forecasts of a binary event, scored using the logarithmic score. Numerical examples are carried out using both synthetic data with prescribed signal-to-noise ratios as well as seasonal ensemble hindcasts of the North Atlantic Oscillation (NAO) index.

Conversely, synthetic data with an anomalous signal-to-noise ratio (RPC = 1. p. 0
Where the paper says it

Numerical examples are carried out using both synthetic data with prescribed signal-to-noise ratios as well as seasonal ensemble hindcasts of the North Atlantic Oscillation (NAO) index. The latter have previously been interpreted as having a signal-to-noise “paradox”, or anomalous signalto-noise ratio, using the RPC statistic.

52) show that the ratios of skill scores are typically larger than 1 for both scoring rules, providing strong indication of an anomalous signal-to-noise ratio. p. 7
Where the paper says it

This is also evident from the characterisation (17) of the classical RPC in the proof of Theorem 1. We can conclude that since the definition of an anomalous signal-to-noise ratio refers to the skill score rather than the score itself, anomalous signal-to-noise ratios might occur because either the score of the recalibrated forecast against actual observations is better than expected, or because the climatology of the actual observations has a larger entropy than expected, or a combination of both; see Equation (12).

Implications and Future Work

The authors conclude that the RSS basically delivers the same message as the classical RPC, demonstrating its suitability as a generalisation. However, experiments indicate that the variance of different statistics can be very different; for instance, in seasonal hindcast data, RSS based on CRPS exhibits a variance which is a mere 10% of the classical RPC’s variance. p. 0

Where the paper says it

For the synthetic data, the RPC statistic as well as the scoring rule–based ones agree regarding which data sets exhibit anomalous signal-to-noise ratios, but exhibit different variance, indicating different statistical properties. For the NAO data, on the other hand, the different statistics are more equivocal on whether the signal-to-noise ratio is anomalous.

A future area of work will therefore be the analysis of the sampling properties of scoring rule–based RSSs. p. 2
Where the paper says it

We compare the RSSs with the classical RPC in the context of synthetic data sets as well as data from an operational seasonal forecasting system. Section 5 gives concluding remarks and areas for future work.

Another area is to consider more powerful methods for estimating π, the optimally recalibrated forecast. p. 9
Where the paper says it

Furthermore, we provide details as to how we approximate the optimally calibrated forecast π. 1 | Ensemble forecasts and the CRPS We consider verifications {yn ; n = 1, .

Conclusion

The paper presents a generalised concept of the signal-to-noise ratio based on proper scoring rules rather than merely linear correlations. It is shown that this definition is the natural generalisation of the classical ratio of predictable components (RPC), but it can be applied to any type of forecast that is amenable to scoring, not only real-valued ensemble forecasts and verifications. p. 0

Where the paper says it

com Funding information A generalised concept of the signal-to-noise ratio (or equivalently the ratio of predictable components, or RPC) is provided, based on proper scoring rules. This definition is the natural generalisation of the classical RPC, yet it allows one to define and analyse the signal-to-noise properties of any type of forecast that is amenable to scoring, thus drastically widening the applicability of these concepts.

The numerical examples show that the new ratio of skill scores (RSS) basically delivers the same message as the classical RPC, thus demonstrating that it can indeed be considered a suitable generalisation of the classical RPC. p. 0
Where the paper says it

This definition is the natural generalisation of the classical RPC, yet it allows one to define and analyse the signal-to-noise properties of any type of forecast that is amenable to scoring, thus drastically widening the applicability of these concepts. The methodology is illustrated through numerical examples of ensemble forecasts, scored using the continuous ranked probability score (CRPS), and of probability forecasts of a binary event, scored using the logarithmic score.

At the same time, however, the experiments already indicate that the variance of different statistics can be very different. p. 0
Where the paper says it

For the synthetic data, the RPC statistic as well as the scoring rule–based ones agree regarding which data sets exhibit anomalous signal-to-noise ratios, but exhibit different variance, indicating different statistical properties. For the NAO data, on the other hand, the different statistics are more equivocal on whether the signal-to-noise ratio is anomalous.

The ability of the RSS or RPC statistics to distinguish between cases of normal and anomalous signal-to-noise ratios is not just a matter of the variance of those statistics (under either one or both scenarios). p. 0
Where the paper says it

For the synthetic data, the RPC statistic as well as the scoring rule–based ones agree regarding which data sets exhibit anomalous signal-to-noise ratios, but exhibit different variance, indicating different statistical properties. For the NAO data, on the other hand, the different statistics are more equivocal on whether the signal-to-noise ratio is anomalous.

The statistical properties of these quantities need to be investigated under both scenarios. p. 0
Where the paper says it

The latter have previously been interpreted as having a signal-to-noise “paradox”, or anomalous signalto-noise ratio, using the RPC statistic. For the synthetic data, the RPC statistic as well as the scoring rule–based ones agree regarding which data sets exhibit anomalous signal-to-noise ratios, but exhibit different variance, indicating different statistical properties.

The paper is 2510. p. 0
Where the paper says it

For the NAO data, on the other hand, the different statistics are more equivocal on whether the signal-to-noise ratio is anomalous. Keywords — signal-to-noise ratio, signal-to-noise paradox, scoring rules, probabilistic forecasting 1 arXiv:2510.

26809.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

1

Improve ensemble forecast evaluation by using proper scoring rules to define a generalized signal-to-noise ratio (RSS) based on the ratio of two skill scores, namely the skill score of the forecast against a verification drawn from the forecast, versus the skill score of the perfectly recalibrated forecast against the actual verifications.

2

Enable analysis for any type of forecast amenable to scoring, not just real-valued ensemble forecasts. This allows systems to diagnose anomalous signal-to-noise ratios using different scoring rules, such as continuous ranked probability score (CRPS) or logarithmic score for binary events.

3

Develop a method to test whether a given estimator of the ratio of predictable components (RPC) is significantly larger than one by utilizing the new RSS statistic. This addresses the limitation that a proper testing methodology would have to be able to answer that question.

4

Allow for comparison between classical RPC and RSS across different scoring rules, such as CRPS and logarithmic score, which can reveal how different statistics exhibit different statistical properties and different levels of variation.

5

Provide a framework for interpreting anomalous signal-to-noise ratios by understanding the components of the ratio: the entropy of pi, the resolution, and the reliability of f, which quantifies how much a forecast fails to match its optimal recalibration.

Abstract

A generalised concept of the signal-to-noise ratio (or equivalently the ratio of predictable components, or RPC) is provided, based on proper scoring rules. This definition is the natural generalisation of the classical RPC, yet it allows one to define and analyse the signal-to-noise properties of any type of forecast that is amenable to scoring, thus drastically widening the applicability of these concepts. The methodology is illustrated through numerical examples of ensemble forecasts, scored using the continuous ranked probability score (CRPS), and of probability forecasts of a binary event, scored using the logarithmic score. Numerical examples are carried out using both synthetic data with prescribed signal-to-noise ratios as well as seasonal ensemble hindcasts of the North Atlantic Oscillation (NAO) index. The latter have previously been interpreted as having a signal-to-noise "paradox", or anomalous signal-to-noise ratio, using the RPC statistic. For the synthetic data, the RPC statistic as well as the scoring rule-based ones agree regarding which data sets exhibit anomalous signal-to-noise ratios, but exhibit different variance, indicating different statistical properties. For the NAO data, on the other hand, the different statistics are more equivocal on whether the signal-to-noise ratio is anomalous.

Sources

Related papers