GraphCast Skill and Systematic Biases in Indian Summer Monsoon Forecasts: Evaluation Against ERA5 and IMERG
summary
The gist
The study presents a systematic climatological evaluation of Google’s GraphCast AI weather prediction system against ERA-5 reanalysis and IMERG observational precipitation datasets for the Indian…
In short
The study systematically evaluated Google's GraphCast AI weather prediction system against ERA-5 and IMERG data for the Indian Summer Monsoon. Findings show GraphCast reproduces broad patterns but suffers from a significant domain-averaged wet bias and severely suppresses rainfall variance across all timescales. This deterministic smoothing signature points to deficiencies in capturing rapid thermodynamic transitions.
Key concepts
- Domain-Averaged Wet Bias
- GraphCast consistently predicts more rain on average over the core monsoon zone compared to observations. This is a broad offset rather than a localized error, suggesting the model systematically overestimates total rainfall across the entire monsoon region.
- Suppression of Rainfall Variance
- The model dramatically reduces the natural variability of rainfall, quantified by a variance ratio of approximately 0.14 relative to IMERG. This means GraphCast struggles to capture both light and extremely heavy rain events accurately.
- Deterministic Smoothing Signature
- This refers to a pattern where GraphCast's output is overly smooth, consistently biasing the rainfall distribution toward moderate values. It suggests the model favors a single, predictable outcome over the true range of possible weather scenarios.
Terminology used across episodes
This episode discusses
- GraphCast Skill and Systematic Biases in Indian Summer Monsoon Forecasts: Evaluation Against ERA5 and IMERG · Paper Radio
- FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators
The paper
GraphCast Skill and Systematic Biases in Indian Summer Monsoon Forecasts: Evaluation Against ERA5 and IMERG · Read on arXiv
Somnath Luitel, Parthasarathi Mukhopadhyay, Manmeet Singh, Sandeep Juneja, Saptarishi Dhanuka
Department of Earth, Environmental, and Atmospheric Sciences, Western Kentucky University · Ashoka University
The Indian Summer Monsoon (ISM) is one of the most dynamically complex phenomena in the global climate system, yet its accurate prediction remains challenging for both physics-based and data-driven models. This study presents a systematic climatological evaluation of Google's GraphCast, a machine-learning global weather prediction system, against ERA-5 reanalysis and IMERG satellite precipitation for the boreal summer season (June-September, JJAS) over 2021-2024. We analyze deterministic GraphCast forecasts initialized at 00 UTC across 6-hourly steps to assess +24h, +48h, and +72h performance over the full ISM domain. The evaluation covers the climatological mean state (spatial rainfall, surface temperature, low-level winds), rainfall intensity distributions, thermodynamic drivers (tropospheric temperature gradient, DTT; apparent heat source and moisture sink, Q1 and Q2), monsoon dynamics (vertical wind shear, meridional structure), and variability across intraseasonal, synoptic, and spectral scales. GraphCast reproduces the spatial pattern and annual cycle of monsoon rainfall with high fidelity at short leads, showing a near-zero domain-averaged rainfall bias over the core monsoon zone (-0.7 to-1.2 mm/day) and high daily temporal correlation with IMERG (0.89-0.92). However, temporal variance is moderately suppressed across timescales (power-spectrum variance ratio 0.63), and the intensity distribution systematically underrepresents extreme events and the 95th percentile. GraphCast closely captures the vertical structure and seasonal evolution of Q1 and Q2, but degrades in northward-propagating intraseasonal variability. Overall, GraphCast shows a moderate underrepresentation of rainfall intensity and variance rather than a wet bias or variance collapse, providing benchmark diagnostics for AI weather models.
Transcript
Introduction to the show: ident: Physics. The day in general physics: instruments, methods and results across the field.
Kai: We're looking at the paper: "GraphCast Skill and Systematic Biases in Indian Summer Monsoon Forecasts".
Mira: Here's the gist of the paper. The study presents a systematic climatological evaluation of Google’s GraphCast AI weather prediction system against ERA-5 reanalysis and IMERG observational precipitation datasets for the Indian Summer Monsoon, revealing that while GraphCast reproduces broad spatial patterns, it suffers from a substantial domain-averaged wet bias and a dramatic suppression of rainfall variance across nearly all timescales Systematic Biases in Rainfall Output The results demonstrate that GraphCast exhibits a "domain-averaged wet bias over the core monsoon zone alongside a pronounced suppression of rainfall variance across nearly all timescales".
Kai: This suppression is quantified by a "regional power-spectrum variance ratio ≈0.14 relative to IMERG".
Mira: Furthermore, the distribution of rainfall is "right-shifted and compressed relative to observations," meaning that "moderate-heavy rainfall (the 95th percentile) is biased wet while the most extreme events are systematically underrepresented".
Kai: This pattern suggests a "deterministic-smoothing signature in GraphCast’s precipitation field that is consistent across independent diagnostics".
Paper summary: Mira: So looking at the title "GraphCast Skill and Systematic Biases in Indian Summer Monsoon Forecasts: Evaluation Against ERA5 and IMERG", it really tells you the focus here is on finding the systematic errors, not just checking if it's accurate overall.
Lev: Yeah, the authors are essentially saying that while GraphCast is competitive in some areas, its performance in simulating tropical monsoon climatology has these specific structural weaknesses (ref: <ref:2607.11905#pg3>).
Kai: So what’s the real implication? It seems like the biggest finding is that this model is incentivized toward a smoothed, moderate-amplitude output rather than committing to the true, occasionally extreme rainfall realization because of that deterministic smoothing (ref: <ref:2607.11905#pg3>).
Mira: That's right. The convergence of three independent diagnostics—intensity distribution, spectral variance, and spatial variability amplitude—all point to this single deterministic-smoothing interpretation for the rainfall biases (ref: <ref:2607.11905#pg2>).
Lev: So the most actionable finding they've got is that improving AI monsoon prediction should prioritize addressing those northward propagating BSISO dynamics (ref: <ref:2607.11905#pg3>).
Kai: And because of that, making analysis code available upon reasonable request to the corresponding author supports reproducibility in AI weather model development (ref: <ref:2607.11905#pg3>).
Mira: So ultimately, this paper provides benchmark diagnostics for next-generation AI weather models by mapping GraphCast biases onto physically interpretable metrics so we can guide their development (ref: <ref:2607.11905#pg3>).
Conclusion: Kai: So, we've looked at this paper about GraphCast and the Indian Summer Monsoon forecasts against ERA5 and IMERG data, focusing on its skill and these systematic biases.
Mira: Yeah, the authors are showing that while GraphCast gets the broad spatial patterns right, it struggles with two big things—a wet bias across the core monsoon zone and a massive suppression of rainfall variance at almost every timescale.
Lev: The paper points to a deterministic smoothing signature in that precipitation field, which is consistent across different checks. It’s not just one glitch; it seems built into how the model processes tropical convection.
Kai: That's what really stuck with me—that convergence on a single smoothing interpretation for the rainfall biases. It suggests that when you train an AI to minimize errors under genuine day-to-day uncertainty in tropical convection, it ends up producing a smoother output than what we expect from the true, occasionally extreme rainfall realization.
Mira: Exactly. The most actionable finding they pinpoint is this convergence of three different diagnostics—intensity distribution, spectral variance, and spatial variability amplitude—all leading to that same smoothing interpretation. It tells us *why* the model isn't capturing those extremes.
Lev: And because of that, the authors say we need to prioritize improving the northward propagating BSISO dynamics in future work if we want real gains for monsoon prediction.
Kai: So, it boils down to this: GraphCast is good at structure but loses the "fuzziness" or variance in the rainfall, and that loss is tied to how it handles those north-south movements of monsoon weather systems.
Mira: Right. And they’re giving us these benchmark diagnostics now—mapping those biases onto physical things like Q1 heating profiles—to give developers a better way to see what the model is actually missing physically.
Lev: It sets a clear path for where we need to put our next error-correction efforts if we want to move beyond just making the mean state look good.
Kai: So, it’s not just about getting better numbers; it’s about understanding the physical mechanism causing that deterministic smoothing signature in precipitation.
More episodes
- 2611.14082-Two atoms with strong light-induced dipolar interactions
- 2611.14040-Characterization of BNLs 20 and 30 micrometer thick LGADs for the Electron Ion Collider
- 2611.13399-Lift-element-based dynamic network of vortex gust-airfoil interactions
- 2611.13476-Hopfion Superlattices in 3D Twisted Moir'e Light Fields
- 2611.13642-Investigation of the timing performance of double-sided p-type 3D silicon sensors produced by CNM
- 2611.13830-SiPM optical readout of a PICOSEC Micromegas precise timing detector
- 2611.14152-Where are the shortest-path nodes in spatial networks?
- 2611.14219-Sub-Ice: semi-automated mapping of basal channels from ice shelf surface elevation data
- 2611.16341-SUMNO: Spatially Unrolled Multigrid Neural Operator for Fluid Flow Surrogates
- 2611.16090-Multi-UTE Myelin Fraction Mapping: Repeatability and Cross-Field-Strength Reproducibility at 1.5T and 3T