A generalisation of the signal-to-noise ratio using proper scoring rules
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: We're looking at the paper: "A generalisation of the signal-to-noise ratio using proper scoring rules".
Jane: Here's the gist of the paper. A generalised concept of the signal-to-noise ratio is provided, based on proper scoring rules rather than merely linear correlations This definition is the natural generalisation of the classical RPC, yet it allows one to define and analyse the signal-to-noise properties of any type of forecast that is amenable to scoring, thus drastically widening the applicability of these concepts Generalization Framework The paper proposes a generalisation of the Ratio of Predictable Components (RPC) by considering instead the ratio of two skill scores, namely the skill score of the forecast against a verification drawn from the forecast, versus the skill score of the perfectly recalibrated forecast against the actual verifications This proposed ratio is denoted as RSSs and can be applied to any type of forecast that is amenable to scoring.
Tom: The authors introduce proper scoring rules, which provide a coherent framework for forecast evaluation.
Jane: Classical Definition Comparison The classical definition of the signal-to-noise ratio refers to the ratio of predictable components or RPC, and the signal-to-noise ratio is defined as anomalous if RPC > 1.
Tom: The authors show that this generalized definition agrees with the classical definition under specific circumstances, such as when using the mean-squared error of the forecast mean and when the forecast means are in a linear relationship.
Title and authors: Tom: So we're looking at this paper called "A generalisation of the signal-to-noise ratio using proper scoring rules". Jane, what's the main idea here in plain English?
Jane: Well, Tom, it takes that old idea of the signal-to-noise ratio and makes it much broader. Instead of just looking at one specific way to measure how good a forecast is—like just comparing the average prediction to reality—this paper suggests using proper scoring rules to look at the signal in any type of forecast that can be scored.
Lu: That's right. It takes the classical ratio of predictable components, which we call RPC, and generalizes it into something called RSSs. The goal is to define a measure of signal quality that works for any forecasting setup you can score against verification data <ref:2510.26809#pg1>.
Meng: So it's not just about one formula anymore? It's adaptable to different kinds of forecasts, right? That sounds useful for what we actually do when we try to deploy models.
Tom: Exactly. The classical definition has limitations because it ties the signal-to-noise ratio too closely to linear relationships between means, but this new concept doesn't have those same restrictions <ref:2510.26809#pg1>.
Jane: It’s about defining a coherent framework for evaluation, using these proper scoring rules, which give us a consistent way to judge quality across different forecasting tasks.
The paper's summary: Tom: So what does the paper actually show in terms of results? We're talking about how this generalized concept compares to the old RPC definition.
Lu: They show that this RSS ratio is basically the natural extension of the classical RPC, meaning it agrees with it under certain conditions, like when you're using the mean-squared error of the forecast mean <ref:2510.26809#pg1>.
Jane: But there’s a key finding here: Theorem one proves that if the RSS ratio is greater than one, then the RPC ratio is also greater than one, depending on which scoring rule you pick <ref:2510.26809#pg1>.
Meng: Okay, so it's not just a theoretical agreement; it’s a practical tool for diagnosis. If we can measure this RSS and see it’s high, we know the forecast is anomalous, regardless of the specific scoring rule we chose <ref:2510.26809#pg3>.
Tom: Right. And they test this on both synthetic data and real seasonal hindcasts for things like the North Atlantic Oscillation index <ref:2510.26809#pg3>.
Jane: The numerical tests are pretty telling. For synthetic data with a normal signal-to-noise ratio, where RPC equals one, the ratios of skill scores using both the continuous ranked probability score and the binary logarithmic score don't give a strong signal that things are anomalous <ref:2510.26809#pg3>.
Lu: However, when they use synthetic data with an anomalous signal-to-noise ratio, where RPC is one point five two, the skill score ratios for both scoring rules are typically larger than one, which gives a strong indication of that anomalous SNR <ref:2510.26809#pg3>.
The paper's improvements: Tom: So what are the actual suggested improvements or the future work they point toward in this paper? We need to know what's next for this concept.
Lu: They conclude that the RSS basically delivers the same message as the classical RPC, which validates its use as a generalisation, but they also found that different statistics have very different variances <ref:2510.26809#pg3>.
Jane: That’s important because it means you can't just rely on one statistic to tell you if the signal is normal or anomalous; the statistical properties of these quantities are quite sensitive <ref:2510.26809#pg3>.
Meng: From a deployment standpoint, if we use different scoring rules, like CRPS versus the logarithmic score, we see these different statistical variances, which means we have to be careful about which metric we're using for production monitoring <ref:2510.26809#pg3>.
Tom: They also mention that a future area is analyzing the sampling properties of these scoring rule based RSSs, and another is finding better ways to estimate pi, which represents the optimally recalibrated forecast <ref:2510.26809#pg3>.
Jane: And they suggest we need more powerful methods for estimating that optimal recalibrated forecast, because that seems to be a key missing piece in fully understanding the signal quality here.
Conclusion: Tom: So to wrap things up, what’s the big picture takeaway from this whole paper on "A generalisation of the signal-to-noise ratio using proper scoring rules"?
Jane: Basically, it gives us a way to define the signal-to-noise ratio based on proper scoring rules instead of just simple linear correlations. It shows that this definition is a natural generalization of the classical RPC, meaning it can be applied to almost any forecast you can score against verification data <ref:2510.26809#pg1>.
Lu: The core message is that the RSS statistic basically delivers the same diagnostic information as the classical RPC, which proves its suitability as a generalisation for wider application <ref:2510.26809#pg3>.
Meng: But we have to keep in mind that just because the statistics give you an indication of anomaly, it’s not enough; you have to understand the variance differences between them <ref:2510.26809#pg3>.
Lalam: I am confident this paper is solid because it connects different scoring rules and shows how they behave under different conditions, which helps us diagnose anomalous signal-to-noise ratios reliably <ref:2510.26809#pg3>.
Tom: Yeah, we've seen how this new RSS statistic works by looking at the synthetic data and the NAO hindcasts in this paper. We'll keep an eye on those future directions, especially figuring out better ways to estimate pi <ref:2510.26809#pg3>.
Jane: That’s what we’re going to do next, looking at how these RSS statistics behave under different sampling properties, which is a really interesting angle for us to explore.
Department of Meteorology and Department of Mathematics and Statistics, University of Reading · Centre for the Mathematics of Planet Earth, University of Reading · National Centre for Earth Observation
stat.AP, physics.ao-ph
Submitted: 2025-10-23
Updated: 2026-03-28
Comments: 19 pages, 2 figures, 3 tables
Journal ref: Meteorological Applications, 33(4), 2026, e70223
DOI: 10.1002/met.70223
Code: https://github.com/eviatarbach/snp
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 82/100
The gist: A generalised concept of the signal-to-noise ratio is provided, based on proper scoring rules rather than merely linear correlations This definition is the natural generalisation of the classical…
Key concepts
- Ratio of Predictable Components (RPC)
- The classical definition where the signal-to-noise ratio is determined by comparing the predictable components of a forecast. It is considered anomalous if RPC > 1, suggesting the forecast has more predictable structure than expected.
- Ratio of Skill Scores (RSS)
- A generalized measure comparing two skill scores: one based on a verification drawn from the forecast, and another based on a perfectly recalibrated forecast against actual verifications. This ratio is applied to any score-based forecast, making it more versatile than the classical RPC.
- Proper Scoring Rules
- A coherent framework for evaluating forecasts where the score is determined by comparing the forecast to an actual outcome. These rules provide a structured way to assess forecast quality beyond simple error metrics.
- Anomalous Signal-to-Noise Ratio
- A condition identified when the RSS is greater than 1. This indicates that the forecast exhibits signal properties beyond what is expected from random noise or normal predictability, suggesting unusual forecasting behavior.
Terminology
Summary
A generalised concept of the signal-to-noise ratio is provided, based on proper scoring rules rather than merely linear correlations This definition is the natural generalisation of the classical RPC, yet it allows one to define and analyse the signal-to-noise properties of any type of forecast that is amenable to scoring, thus drastically widening the applicability of these concepts
Generalization Framework
The paper proposes a generalisation of the Ratio of Predictable Components (RPC) by considering instead the ratio of two skill scores, namely the skill score of the forecast against a verification drawn from the forecast, versus the skill score of the perfectly recalibrated forecast against the actual verifications This proposed ratio is denoted as RSSs and can be applied to any type of forecast that is amenable to scoring. The authors introduce proper scoring rules, which provide a coherent framework for forecast evaluation. p. 1 In common with previous authors, skill score in this paper refers to the score of a
forecast, relative to the score of a reference forecast, typically a climatological forecast. To score forecasts we use
proper scoring rules (Bröcker and Smith, 2007a), as these provide a coherent framework for forecast evaluation.Where the paper says it
Classical Definition Comparison
The classical definition of the signal-to-noise ratio refers to the ratio of predictable components or RPC, and the signal-to-noise ratio is defined as anomalous if RPC > 1. The authors show that this generalized definition agrees with the classical definition under specific circumstances, such as when using the mean-squared error of the forecast mean and when the forecast means are in a linear relationship. p. 5 We will show that
the generalised and the classical definitions agree under the following circumstances (to be made precise below):
1. For the scoring rule in Definition 1 we use the mean-squared error of the forecast mean. m(π) = a + b m(f ), with a, b determined so as to minimise Å( (Y − m(π) )2
). Then RPC ≥ 1 if and only if RSSσ ≥ 1, where RSSσ is understood with respect to the specific scoring rule σ (f , y ) :=
(y − m(f ) )2
.Where the paper says it
Where the paper says it
Numerical Validation
Numerical experiments are carried out using both synthetic data with prescribed signal-to-noise ratios as well as seasonal ensemble hindcasts of the North Atlantic Oscillation (NAO) index. The results show that for synthetic data with a normal signal-to-noise ratio (RPC = 1), the ratios of skill scores for both the CRPS and the binary logarithmic score fail to give a strong indication of an anomalous SNR, with RSS values within the 95% bootstrap confidence intervals. p. 0 The methodology is illustrated through numerical examples of ensemble forecasts, scored using the continuous ranked probability score (CRPS), and of probability forecasts of a binary event, scored using the logarithmic score. Numerical examples are carried out using both synthetic
data with prescribed signal-to-noise ratios as well as seasonal ensemble hindcasts of the North Atlantic Oscillation
(NAO) index. Numerical examples are carried out using both synthetic
data with prescribed signal-to-noise ratios as well as seasonal ensemble hindcasts of the North Atlantic Oscillation
(NAO) index. The latter have previously been interpreted
as having a signal-to-noise “paradox”, or anomalous signalto-noise ratio, using the RPC statistic. This is also evident from the characterisation (17) of the
classical RPC in the proof of Theorem 1. We can conclude that since the definition of an anomalous signal-to-noise ratio refers to the skill score rather than
the score itself, anomalous signal-to-noise ratios might occur because either the score of the recalibrated forecast
against actual observations is better than expected, or because the climatology of the actual observations has a
larger entropy than expected, or a combination of both; see Equation (12).Where the paper says it
Where the paper says it
Where the paper says it
Implications and Future Work
The authors conclude that the RSS basically delivers the same message as the classical RPC, demonstrating its suitability as a generalisation. However, experiments indicate that the variance of different statistics can be very different; for instance, in seasonal hindcast data, RSS based on CRPS exhibits a variance which is a mere 10% of the classical RPC’s variance. p. 0 For the synthetic data, the RPC statistic as well as the
scoring rule–based ones agree regarding which data sets
exhibit anomalous signal-to-noise ratios, but exhibit different variance, indicating different statistical properties. For
the NAO data, on the other hand, the different statistics
are more equivocal on whether the signal-to-noise ratio is
anomalous. We compare the RSSs with the classical RPC in the context of synthetic
data sets as well as data from an operational seasonal forecasting system. Section 5 gives concluding remarks and
areas for future work. Furthermore, we
provide details as to how we approximate the optimally calibrated forecast π. 1 | Ensemble forecasts and the CRPS
We consider verifications {yn ; n = 1, .Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
The paper presents a generalised concept of the signal-to-noise ratio based on proper scoring rules rather than merely linear correlations. It is shown that this definition is the natural generalisation of the classical ratio of predictable components (RPC), but it can be applied to any type of forecast that is amenable to scoring, not only real-valued ensemble forecasts and verifications. p. 0 com
Funding information
A generalised concept of the signal-to-noise ratio (or equivalently the ratio of predictable components, or RPC) is provided, based on proper scoring rules. This definition is the
natural generalisation of the classical RPC, yet it allows one
to define and analyse the signal-to-noise properties of any
type of forecast that is amenable to scoring, thus drastically
widening the applicability of these concepts. This definition is the
natural generalisation of the classical RPC, yet it allows one
to define and analyse the signal-to-noise properties of any
type of forecast that is amenable to scoring, thus drastically
widening the applicability of these concepts. The methodology is illustrated through numerical examples of ensemble forecasts, scored using the continuous ranked probability score (CRPS), and of probability forecasts of a binary event, scored using the logarithmic score. For the synthetic data, the RPC statistic as well as the
scoring rule–based ones agree regarding which data sets
exhibit anomalous signal-to-noise ratios, but exhibit different variance, indicating different statistical properties. For
the NAO data, on the other hand, the different statistics
are more equivocal on whether the signal-to-noise ratio is
anomalous. For the synthetic data, the RPC statistic as well as the
scoring rule–based ones agree regarding which data sets
exhibit anomalous signal-to-noise ratios, but exhibit different variance, indicating different statistical properties. For
the NAO data, on the other hand, the different statistics
are more equivocal on whether the signal-to-noise ratio is
anomalous. The latter have previously been interpreted
as having a signal-to-noise “paradox”, or anomalous signalto-noise ratio, using the RPC statistic. For the synthetic data, the RPC statistic as well as the
scoring rule–based ones agree regarding which data sets
exhibit anomalous signal-to-noise ratios, but exhibit different variance, indicating different statistical properties. For
the NAO data, on the other hand, the different statistics
are more equivocal on whether the signal-to-noise ratio is
anomalous. Keywords — signal-to-noise ratio, signal-to-noise paradox, scoring rules, probabilistic forecasting
1
arXiv:2510.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
1
Improve ensemble forecast evaluation by using proper scoring rules to define a generalized signal-to-noise ratio (RSS) based on the ratio of two skill scores, namely the skill score of the forecast against a verification drawn from the forecast, versus the skill score of the perfectly recalibrated forecast against the actual verifications.
2
Enable analysis for any type of forecast amenable to scoring, not just real-valued ensemble forecasts. This allows systems to diagnose anomalous signal-to-noise ratios using different scoring rules, such as continuous ranked probability score (CRPS) or logarithmic score for binary events.
3
Develop a method to test whether a given estimator of the ratio of predictable components (RPC) is significantly
larger than one by utilizing the new RSS statistic. This addresses the limitation that a proper testing methodology would have to be able to answer that question.
4
Allow for comparison between classical RPC and RSS across different scoring rules, such as CRPS and logarithmic score, which can reveal how different statistics exhibit different statistical properties
and different levels of variation.
5
Provide a framework for interpreting anomalous signal-to-noise ratios by understanding the components of the ratio: the entropy of pi, the resolution, and the reliability of f,
which quantifies how much a forecast fails to match its optimal recalibration.
Abstract
A generalised concept of the signal-to-noise ratio (or equivalently the ratio of predictable components, or RPC) is provided, based on proper scoring rules. This definition is the natural generalisation of the classical RPC, yet it allows one to define and analyse the signal-to-noise properties of any type of forecast that is amenable to scoring, thus drastically widening the applicability of these concepts. The methodology is illustrated through numerical examples of ensemble forecasts, scored using the continuous ranked probability score (CRPS), and of probability forecasts of a binary event, scored using the logarithmic score. Numerical examples are carried out using both synthetic data with prescribed signal-to-noise ratios as well as seasonal ensemble hindcasts of the North Atlantic Oscillation (NAO) index. The latter have previously been interpreted as having a signal-to-noise "paradox", or anomalous signal-to-noise ratio, using the RPC statistic. For the synthetic data, the RPC statistic as well as the scoring rule-based ones agree regarding which data sets exhibit anomalous signal-to-noise ratios, but exhibit different variance, indicating different statistical properties. For the NAO data, on the other hand, the different statistics are more equivocal on whether the signal-to-noise ratio is anomalous.
Sources
- Machine Learning for Inverse Problems and Data Assimilation
- Ensemble reliability and the signal-to-noise paradox in ECMWF subseasonal forecasts
Related papers
- Unraveling time-varying causal effects of multiple exposures: integrating Functional Data Analysis with Multivariable Mendelian Randomization
- Strategic Causal Policy Learning with Manipulable Eligibility
- BayesJudge: Uncertainty-Aware Bayesian Meta-Evaluation of Human and LLM Judgments
- Automating structural reliability analysis with a multi-agent large language model framework
- Improving Minority Population Sampling with BISG Probabilities: Evidence from a Survey of Jewish Americans
- When Do Sports Leagues Stop Being Competitive? Bayesian Detection of Competitive-Balance Eras from Match-Level Data