Improved rainfall forecasts in daily use over East Africa

summary

The gist

Ensemble forecasting has proven to be a vital tool for predicting extreme, life-threatening or only partially predictable weather events The paper tests the performance of a forecast system…

In short

The study tested a system using a conditional Generative Adversarial Network (cGAN) to correct global weather forecasts towards satellite rainfall data. This method, compared against other techniques like IDR, showed that both cGAN and IDR improve forecast skill for rainfall models. The cGAN proved practical due to low computational cost, offering competitive performance in East Africa.

Key concepts

Ensemble Forecasting
This involves running multiple versions of a weather model with slightly different starting conditions. By comparing these varied outcomes, forecasters can better assess the uncertainty and potential range of future weather events. It is a vital tool for predicting extreme or unpredictable weather.
cGAN (Conditional Generative Adversarial Network)
A machine learning model used here to adjust raw forecast outputs. The cGAN learns to generate realistic rainfall scenarios by comparing its predictions against actual satellite observations, effectively correcting the initial ensemble forecast towards more accurate rainfall estimates.
CRPS (Continuous Rank Probability Score)
This is a statistical measure used to evaluate how well a weather forecast predicts the actual observed values. A lower CRPS score indicates better performance, meaning the model's predicted probabilities are closer to reality over time.
IDR (Isotonic Distributional Regression)
A statistical method used to estimate the probability distribution of rainfall forecasts. It helps correct biases in these distributions by assuming a certain order (isotonicity) for the data, which can yield better estimates than simply using ensemble members.

Terminology used across episodes

This episode discusses

The paper

Improved rainfall forecasts in daily use over East Africa · Read on arXiv

Department of Physics, University of Oxford · National Centre for Atmospheric Science, Oxford · IGAD Climate Prediction and Applications Centre, IGAD · Kenya Meteorological Service Authority, Government of Kenya · Ethiopian Meteorological Institute, Government of Ethiopia · European Centre for Medium-Range Weather Forecasts

Transcript

Introduction to the show: ident: Physics. The day in general physics: instruments, methods and results across the field.

Kai: We're looking at the paper: "Improved rainfall forecasts in daily use over East Africa".

Mira: Here's the gist of the paper. Ensemble forecasting has proven to be a vital tool for predicting extreme, life-threatening or only partially predictable weather events The paper tests the performance of a forecast system comprising the global ECMWF ensemble forecast, post-processed using cGAN, a neural network model, and forecasts calibrated using IDR against probabilistic climatology The study evaluates a system that uses the conditional Generative Adversarial Network (cGAN) to correct the ECMWF ensemble forecast towards IMERG1 blended satellite rainfall data This system is compared against quantile mapping, isotonic distributional regression (IDR), and post-processed FuXi and GraphCast models The cGAN model is computationally inexpensive, producing 1000 spatially correlated ensemble members on a standard desktop computer The training data consists of 50 members of the IFS operational ensemble forecast, linearly interpolated to the IMERG v7 final observational data grid Model Comparison and Skill Gains The research compares different post-processing techniques to see which brings physics-based and machine-learning models to comparable levels of forecast skill Both cGAN and IDR bring substantial improvements by applying empirical corrections to physics-based or pure machine-learning models The performance is assessed using the Continuous Rank Probability Score (CRPS) over a one-year test period The results show that both post-processing methods, cGAN and IDR, bring the IFS model and the FuXi and GraphCast models to comparable levels of forecast skill Performance Analysis by Lead Time The analysis examines how lead time affects forecast skill across different regions For 24h rainfall accumulations, the IFS forecast has a better (lower) CRPS than climatology in parts of Kenya, Uganda, and Tanzania at 30 hours lead time after initialization However, over large areas the IFS forecast CRPS is higher (worse), including the entire vicinity of the Congo rainforest and mountainous regions The skill assessment shows that IFS forecasts become worse than climatology after less than 1 day for 6h accumulated rainfall and after 2 to 3 days for 24h accumulations Distributional Corrections and Trade-offs The paper details the application of quantile mapping (QM) and IDR to correct forecast distributions Quantile mapping corrects the forecast rainfall distribution of each ensemble member to match that of the measurements Isotonic distributional regression (IDR) estimates these distributions, bypassing ensemble production and potentially yielding superior estimators when assuming isotonicity However, using IDR to obtain a forecast rainfall distribution at each grid point removes all spatial correlation information, retaining only the local 1D distribution Furthermore, applying quantile mapping to cGAN outputs increases the CRPS compared to IFS+cGAN alone Regional Performance and Robustness The quality of the forecast depends on location, with results averaged over select countries showing that the relative order of the models is quite robust For example, in Ethiopia and Kenya, IFS+cGAN achieves the best CRPS among models considered The study also investigates ROC and AUC scores to assess forecast quality for different seasons and test periods across various East African nations The analysis suggests that optimizing the overall area mean score might not be the most useful model, indicating a trade-off where different models are best applied only to specialist regions The conditional Generative Adversarial Network (cGAN) is competitive due to its low computational cost, producing 1000 ensemble members in less than 30 minutes on a standard desktop computer without requiring a GPU The paper concludes that cGAN is practical for producing multiple probabilistic forecasts per day using hardware readily available in the region The authors speculate that this approach, alongside the increasing difficulty of improving CRPS further, points to a law of diminishing returns regarding forecast skill improvement A way forward might be to use weighted scoring rules that emphasize the tails of the distribution for better performance on extreme events Data and Future Work The authors plan to make use of 20-year hindcasts of the latest operational IFS retained at ECMWF for more detailed analysis They also suggest that improving observational data, such as additional rain gauges or improved quality control of IMERG, can further enhance forecast skill The paper notes that cGAN underestimates rainfall in the high rainfall part of the distribution above approximately 3.5 mm/h Further research should explore using weighted scoring rules that emphasize the tails to address precision versus sharpness trade-offs The results are sensitive to training data, and more years of test data would allow for a more detailed analysis of locally and seasonally varying skill The IMERG v7 final data product is noted as a great product for these purposes despite some uncertainties in certain measurements Figure Interpretation Figure 3 illustrates the 6h and 24h rainfall accumulation CRPS values averaged over the East African domain and one-year test period Figure S5 shows the difference between the one-year mean CPRS of the 6h rainfall accumulation cGAN forecasts and IMERG climatological distribution for various lead times Figure S7 compares the CRPS of IFS+cGAN against other models, showing that IFS+cGAN-all is better over populated regions Figure S8 shows the performance comparison over a later test period, indicating robustness across different rainfall amounts The ROC curves in Figure S17 and S18 provide AUC scores derived from ROC curves for each season and the whole year for various countries The histograms in Figure 5 show that cGAN applied to IFS under quantile mapping corrects the rainfall distribution well, but naive application increases the CRPS The maps in Figure 2 illustrate how forecast skill varies over the course of a day The results are robust across two different test years despite differences in absolute CRPS values due to differing amounts of rainfall The spatial patterns of rainfall have similarities between the two test periods, which is reflected in the similarity of relative improvements across models The AUC scores derived from ROC curves show that IFS+cGAN has a higher AUC score for Ethiopia compared to IMERG The results are robust across different countries and seasons when averaged, suggesting a consistent pattern of skill improvement with post-processing The paper demonstrates useful gains in rainfall forecast skill by applying empirical corrections to physics-based forecast models or pure machine-learning models The model tested generates 1000 ensemble forecasts in a single forecast cycle on a standard desktop computer Both post-processing methods (cGAN and IDR) bring the physics-based IFS model and the machine-learning FuXi and GraphCast models to comparable levels of forecast skill The cGAN is cheap to train/run and requires no additional post-processing, making it suitable for Meteorological Services with limited computational facilities The ability of cGAN to produce individual realisations of future rainfall was important when assessed in an operational context by the National Meteorological and Hydrological Services of Kenya and Ethiopia The paper details the performance of models that have been developed for daily use at national meteorological centres and tested in operational forecast offices Author Contributions F.C.C.

Kai: performed the research and wrote the paper A.T.T.M adapted the GAN code from the model developed in

1: F.C.C., A.W., T.P., and J.M guided the direction of the paper A.T.T.M., S.N., B.A., Z.M., B.T., and B.K provided data download and processing and valuable technical insight F.P.

Mira: and M.C organised data access and compute resources All authors provided valuable comments, suggestions, and support for both production of the manuscript and direction of the research Acknowledgements The authors acknowledge funding from Google.org via the United Nations World Food Programme A.

Kai: Weisheimer acknowledges funding from the National Centre for Atmospheric Science, UK The authors declare no competing interest The acknowledgements section is present on page 5 of the paper.

Paper summary: Kai: So we're looking at this paper here, "Improved rainfall forecasts in daily use over East Africa." Basically it's about taking the existing ensemble forecasts and making them much better for predicting rain in that region.

Mira: Right, so the core idea is they're testing a system that uses an ensemble forecast from ECMWF and then they post-process it using a conditional generative adversarial network, or cGAN. They also compare this against other correction methods like quantile mapping and isotonic distributional regression, or IDR.

Lev: From an engineering standpoint, the cGAN approach is interesting because the paper says it's computationally inexpensive; it can produce one thousand spatially correlated ensemble members on a standard desktop computer without needing a graphics card.

Kai: That low computational cost is significant for operational use, Lev. So what are they trying to achieve with this comparison? What’s the main claim here?

Mira: The main thing they're doing is comparing different post-processing techniques to see which brings the physics-based model, like the Integrated Forecasting System or IFS, and these pure machine learning models like FuXi and GraphCast to a comparable level of forecast skill.

Lev: They are using the continuous rank probability score, or CRPS, over a one year test period to measure that skill comparison. It sounds like they're trying to see if adding these empirical corrections actually helps the forecast more than just running the base model alone.

Kai: And where does this all apply? The paper focuses specifically on rainfall forecasts for daily use over East Africa, which is a region known for its complex weather patterns, right?

Mira: Exactly. They look at how lead time affects skill across different areas. For twenty-four hour rainfall accumulations, they found that the IFS forecast has a better, lower CRPS than climatology in parts of Kenya, Uganda, and Tanzania at thirty hours lead time after initialization.

Lev: But there's a catch; over larger areas, like the entire vicinity of the Congo rainforest and mountainous regions, the IFS forecast CRPS actually gets worse than climatology after less than one day for six hour accumulations and after two to three days for twenty-four hour accumulations.

Paper summary: Kai: That variation is important. So what’s this about distribution corrections like quantile mapping or IDR? Mira, what’s the trade-off there?

Mira: Quantile mapping corrects the forecast rainfall distribution of each ensemble member to match that of the measurements. But when they apply it to the cGAN outputs, they see that it actually increases the CRPS compared to using just IFS plus cGAN.

Lev: That suggests there's a trade-off between getting a better distribution and maintaining predictive skill; you might be fixing one aspect but hurting another. IDR estimates those distributions by bypassing ensemble production, but using IDR to get that distribution at each grid point removes all the spatial correlation information, leaving only the local one dimensional distribution.

Kai: So it seems like the choice between these correction methods depends entirely on what you prioritize: improving the overall distribution shape or keeping that spatial relationship intact.

Mira: That's right. And in terms of regional performance, they found that while results are averaged across select countries, the relative order of the models is quite robust. For example, in Ethiopia and Kenya, IFS plus cGAN achieves the best CRPS among all the models considered.

Lev: That makes sense when you think about where this work is applied; it's not about finding one perfect model for everything across a whole continent, but maybe optimizing for specific regions.

Kai: What does that mean practically for someone who only listens to the show? If you're running a national meteorological service, what should you be looking at based on this paper?

Mira: You should focus on the idea that the quality of the forecast depends heavily on location, so optimizing just one area mean score might not be the most useful model for everything. It points toward applying different models only to specialist regions.

Lev: And regarding practicality, Kai mentioned that the cGAN is competitive because it’s cheap; it produces one thousand ensemble members in less than thirty minutes on a standard desktop computer without requiring a graphics card. That’s a big win for operational settings with limited hardware.

Paper summary: Kai: So, despite the limitations we just discussed regarding skill at certain lead times, the authors feel this approach is practical for producing multiple probabilistic forecasts per day using hardware that's readily available in the region.

Mira: They also point out a limitation regarding rainfall estimation; the cGAN underestimates rainfall in the high rainfall part of its distribution above approximately three point five millimeters per hour. That’s a concrete bias you have to account for when using it.

Lev: And looking ahead, they suggest that improving observational data, like adding more rain gauges or better quality control of the IMERG product, could further enhance forecast skill and make this system even more robust over time.

Kai: So where does this leave us? If we look at the whole picture of "Improved rainfall forecasts in daily use over East Africa," what's the final thought on its importance for weather forecasting?

Mira: The paper shows that applying empirical corrections to physics-based or pure machine learning models can bring them to comparable levels of forecast skill. It highlights that while there are trade-offs with distribution corrections, techniques like the cGAN offer a practical way to get multiple probabilistic forecasts daily using accessible hardware.

Lev: And the authors suggest a way forward involves using weighted scoring rules that emphasize the tails of the distribution for better performance on extreme events, which is where those ensemble forecasts really matter.

Kai: So, in summary, this work demonstrates useful gains in rainfall forecast skill by applying empirical corrections to physics-based forecast models or pure machine learning models. It shows that the cGAN is cheap to train and run and requires no additional post-processing for its basic use.

Mira: And it underscores the importance of location; the results are robust across different countries and seasons when averaged, suggesting a consistent pattern of skill improvement with post-processing.

Lev: The paper's limitation is that they note that cGAN underestimates rainfall in the high rainfall part of the distribution above approximately three point five millimeters per hour. That’s a bias you need to watch out for.

Kai: It seems like this work is really about making complex, physically-based or machine learning forecasts more accessible and competitive in daily operations across East Africa by using relatively straightforward post-processing steps.

Conclusion: Mira: So, we've looked at how they take an existing ensemble forecast and try to make it better for predicting rain in East Africa using things like a conditional generative adversarial network, or cGAN.

Kai: It's really about taking these complex models and applying some empirical corrections to bring them up to the standard needed for daily operational use.

Mira: The authors are testing this system against other correction methods, like quantile mapping and isotonic distributional regression, to see which actually improves the forecast skill using continuous rank probability score over a year.

Lev: From my side, what's interesting is that they found a way to get these models—the IFS system and those machine learning models like FuXi—to be comparable in accuracy.

Kai: That's the main point here, right? It’s not about inventing a brand new forecasting model from scratch, but improving how we use what we already have.

Mira: Exactly. The implication is that if you can get a method to correct these errors reliably, you can get better probabilistic rainfall predictions for regions like East Africa.

Lev: But they did flag some things; the cGAN underestimates rainfall in the highest parts of the distribution above about three point five millimeters per hour. That’s a bias we have to consider when we talk about precision versus sharpness.

Kai: So, while it's practical for getting multiple forecasts daily with standard hardware, we still have those specific biases and trade-offs to manage.

Mira: Right. It shows that regional skill matters a lot; the performance isn't uniform across all of East Africa.

Lev: That means any system you build needs to know exactly where it's running because what works in Ethiopia might not work the same way near the Congo rainforest.

Kai: So, we see this as a way to make high-quality ensemble forecasting more accessible and useful for daily operations in these areas.

Mira: And that leads us next into how much those specific biases and regional differences actually matter when you're trying to run a real forecast system.

More episodes

← Home