GraphCast Skill and Systematic Biases in Indian Summer Monsoon Forecasts: Evaluation Against ERA5 and IMERG
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Physics. The day in general physics: instruments, methods and results across the field.
Kai: We're looking at the paper: "GraphCast Skill and Systematic Biases in Indian Summer Monsoon Forecasts".
Mira: Here's the gist of the paper. The study presents a systematic climatological evaluation of Google’s GraphCast AI weather prediction system against ERA-5 reanalysis and IMERG observational precipitation datasets for the Indian Summer Monsoon, revealing that while GraphCast reproduces broad spatial patterns, it suffers from a substantial domain-averaged wet bias and a dramatic suppression of rainfall variance across nearly all timescales Systematic Biases in Rainfall Output The results demonstrate that GraphCast exhibits a "domain-averaged wet bias over the core monsoon zone alongside a pronounced suppression of rainfall variance across nearly all timescales".
Kai: This suppression is quantified by a "regional power-spectrum variance ratio ≈0.14 relative to IMERG".
Mira: Furthermore, the distribution of rainfall is "right-shifted and compressed relative to observations," meaning that "moderate-heavy rainfall (the 95th percentile) is biased wet while the most extreme events are systematically underrepresented".
Kai: This pattern suggests a "deterministic-smoothing signature in GraphCast’s precipitation field that is consistent across independent diagnostics".
Paper summary: Mira: So looking at the title "GraphCast Skill and Systematic Biases in Indian Summer Monsoon Forecasts: Evaluation Against ERA5 and IMERG", it really tells you the focus here is on finding the systematic errors, not just checking if it's accurate overall.
Lev: Yeah, the authors are essentially saying that while GraphCast is competitive in some areas, its performance in simulating tropical monsoon climatology has these specific structural weaknesses (ref: <ref:2607.11905#pg3>).
Kai: So what’s the real implication? It seems like the biggest finding is that this model is incentivized toward a smoothed, moderate-amplitude output rather than committing to the true, occasionally extreme rainfall realization because of that deterministic smoothing (ref: <ref:2607.11905#pg3>).
Mira: That's right. The convergence of three independent diagnostics—intensity distribution, spectral variance, and spatial variability amplitude—all point to this single deterministic-smoothing interpretation for the rainfall biases (ref: <ref:2607.11905#pg2>).
Lev: So the most actionable finding they've got is that improving AI monsoon prediction should prioritize addressing those northward propagating BSISO dynamics (ref: <ref:2607.11905#pg3>).
Kai: And because of that, making analysis code available upon reasonable request to the corresponding author supports reproducibility in AI weather model development (ref: <ref:2607.11905#pg3>).
Mira: So ultimately, this paper provides benchmark diagnostics for next-generation AI weather models by mapping GraphCast biases onto physically interpretable metrics so we can guide their development (ref: <ref:2607.11905#pg3>).
Conclusion: Kai: So, we've looked at this paper about GraphCast and the Indian Summer Monsoon forecasts against ERA5 and IMERG data, focusing on its skill and these systematic biases.
Mira: Yeah, the authors are showing that while GraphCast gets the broad spatial patterns right, it struggles with two big things—a wet bias across the core monsoon zone and a massive suppression of rainfall variance at almost every timescale.
Lev: The paper points to a deterministic smoothing signature in that precipitation field, which is consistent across different checks. It’s not just one glitch; it seems built into how the model processes tropical convection.
Kai: That's what really stuck with me—that convergence on a single smoothing interpretation for the rainfall biases. It suggests that when you train an AI to minimize errors under genuine day-to-day uncertainty in tropical convection, it ends up producing a smoother output than what we expect from the true, occasionally extreme rainfall realization.
Mira: Exactly. The most actionable finding they pinpoint is this convergence of three different diagnostics—intensity distribution, spectral variance, and spatial variability amplitude—all leading to that same smoothing interpretation. It tells us *why* the model isn't capturing those extremes.
Lev: And because of that, the authors say we need to prioritize improving the northward propagating BSISO dynamics in future work if we want real gains for monsoon prediction.
Kai: So, it boils down to this: GraphCast is good at structure but loses the "fuzziness" or variance in the rainfall, and that loss is tied to how it handles those north-south movements of monsoon weather systems.
Mira: Right. And they’re giving us these benchmark diagnostics now—mapping those biases onto physical things like Q1 heating profiles—to give developers a better way to see what the model is actually missing physically.
Lev: It sets a clear path for where we need to put our next error-correction efforts if we want to move beyond just making the mean state look good.
Kai: So, it’s not just about getting better numbers; it’s about understanding the physical mechanism causing that deterministic smoothing signature in precipitation.
Somnath Luitel, Parthasarathi Mukhopadhyay, Manmeet Singh, Sandeep Juneja, Saptarishi Dhanuka
Department of Earth, Environmental, and Atmospheric Sciences, Western Kentucky University · Ashoka University
physics.ao-ph
Submitted: 2026-06-30
Updated: 2026-10-30
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 83/100
The gist: The study presents a systematic climatological evaluation of Google’s GraphCast AI weather prediction system against ERA-5 reanalysis and IMERG observational precipitation datasets for the Indian…
Key concepts
- Domain-Averaged Wet Bias
- GraphCast consistently predicts more rain on average over the core monsoon zone compared to observations. This is a broad offset rather than a localized error, suggesting the model systematically overestimates total rainfall across the entire monsoon region.
- Suppression of Rainfall Variance
- The model dramatically reduces the natural variability of rainfall, quantified by a variance ratio of approximately 0.14 relative to IMERG. This means GraphCast struggles to capture both light and extremely heavy rain events accurately.
- Deterministic Smoothing Signature
- This refers to a pattern where GraphCast's output is overly smooth, consistently biasing the rainfall distribution toward moderate values. It suggests the model favors a single, predictable outcome over the true range of possible weather scenarios.
Terminology
Summary
The study presents a systematic climatological evaluation of Google’s GraphCast AI weather prediction system against ERA-5 reanalysis and IMERG observational precipitation datasets for the Indian Summer Monsoon, revealing that while GraphCast reproduces broad spatial patterns, it suffers from a substantial domain-averaged wet bias and a dramatic suppression of rainfall variance across nearly all timescales
Systematic Biases in Rainfall Output
The results demonstrate that GraphCast exhibits a domain-averaged wet bias over the core monsoon zone alongside a pronounced suppression of rainfall variance across nearly all timescales
. This suppression is quantified by a regional power-spectrum variance ratio ≈0.14 relative to IMERG
. Furthermore, the distribution of rainfall is right-shifted and compressed relative to observations,
meaning that moderate-heavy rainfall (the 95th percentile) is biased wet while the most extreme events are systematically underrepresented
. This pattern suggests a deterministic-smoothing signature in GraphCast’s precipitation field that is consistent across independent diagnostics
.
Thermodynamic and Dynamic Deficiencies
The analysis of thermodynamic drivers reveals a deficient lower-tropospheric Q1 profile
and a degraded representation of northward-propagating intraseasonal variability
. Specifically, GraphCast shows a pronounced deficit in lower-tropospheric Q1 (below 700 hPa)
which is physically consistent with insufficient shallow convection over orographic slopes and boundary-layer heating
. The model also shows a slow bias in the DTT annual cycle,
suggesting difficulty in reproducing rapid thermodynamic transitions associated with monsoon onset and withdrawal.
Variability and Propagation Asymmetry
The study highlights an • The AI model captures the large-scale thermodynamic drivers (DTT, Q1/Q2 vertical structure) but underestimates lower-tropospheric diabatic heating, consistent
with deficient orographic precipitation over the Western Ghats. • Intraseasonal variability and northward propagation of the BSISO are degraded
at +72h lead, highlighting the current limits of purely data-driven forecasting for
tropical subseasonal dynamics.asymmetric degradation of subseasonal propagation that is more severe in the northward (BSISO) than the eastward (MJO/Kelvin-wave-like) component
. The northward propagation signal is substantially weaker and less coherent in GraphCast than in IMERG, particularly at +48h/+72h,
while the eastward envelope is preserved with markedly higher fidelity at all three lead times
. This asymmetry is attributed to the absence of air–sea coupling preventing positive feedback for BSISO propagation and inadequate lower-tropospheric Q1 heating. p. 1Where the paper says it
Reconciliation of Spatial Biases
The paper addresses the apparent contradiction between a large positive domain-mean rainfall bias
and a localized orographic-scale dry bias
by suggesting they operate at different spatial scales. The domain-mean wet bias is consistent with a broad, roughly uniform additive offset of a different character than the deterministic-smoothing signature emphasized elsewhere in this Discussion,
while the localized dry bias is linked to a terrain-locked smoothing mechanism
tied to orographic precipitation maxima.
Wind Field Fidelity
In contrast to rainfall, the dynamical wind field is well represented, as the 10-m wind speed field shows a The evaluation encompasses the climatological mean state (spatial distribution of rainfall, surface temperature, and low-level winds), the distribution
of rainfall by intensity class, thermodynamic drivers (tropospheric temperature
gradient, DTT; apparent heat source and moisture sink, Q1 and Q2), monsoon dynamics (vertical wind shear, meridional structure), and rainfall variability
across timescales (intraseasonal, synoptic, and spectral power). Results demonstrate that GraphCast reproduces the broad spatial pattern of monsoon rainfall
and the timing of the annual cycle with considerable fidelity at short leads, but
exhibits a domain-averaged wet bias over the core monsoon zone alongside a
pronounced suppression of rainfall variance across nearly all timescales (regional
power-spectrum variance ratio ≈0.near-observational spectral fidelity
with a Total Var Ratio of 0.98–1.01 and a near-zero bias. p. 0Where the paper says it
clear dichotomy between GraphCast’s strong dynamical and weak precipitation-variance skill
.
Conclusion
The convergence of three independent diagnostics on a single deterministic-smoothing interpretation for the rainfall biases
is the most actionable finding, suggesting that future improvements should prioritize addressing northward-propagating BSISO dynamics. The authors conclude that the convergence of three independent diagnostics (intensity distribution, spectral variance, and spatial ISV amplitude) on a single deterministic-smoothing interpretation for the rainfall biases
is the most actionable finding.
Pruning redundant reasoning tokens
The paper systematically evaluates GraphCast against ERA-5 reanalysis and IMERG precipitation to identify systematic errors in Indian Summer Monsoon forecasts. The evaluation focuses on deterministic hindcasts initialized at 00 UTC over the period 2021–2024 using four lead times of +24h, +48h, and +72h. p. 0 This study presents a systematic climatological evaluation
of Google’s GraphCast, a machine-learning-based global weather prediction system, against ERA-5 reanalysis and the Integrated Multi-satellitE Retrievals for
GPM (IMERG) observational precipitation dataset for the boreal summer season
(June–September, JJAS) over the period 2021–2024. We analyze deterministic
GraphCast forecasts initialized at 00 UTC using four 6-hourly lead times, composited to assess +24h, +48h, and +72h forecast performance across the full ISM
domain.Where the paper says it
How the training signal is built
GraphCast is a deterministic, autoregressive global weather prediction system based on a graph neural network architecture
. It was trained on four decades of ERA-5 reanalysis data
and produces forecasts at a 6-hourly temporal resolution for up to 10 days.
Where the gains come from
The study suggests that improving AI monsoon prediction will require explicit coupling with an ocean model, stochastic perturbation schemes, or new training approaches
specifically targeting northward-propagating BSISO dynamics. The availability of GraphCast probabilistic ensemble products is noted as an opportunity to examine whether ensemble spread can compensate for the variance suppression identified here
.
Limitations
The evaluation period covers only four monsoon seasons, and the comparison against ERA-5 is not entirely independent since GraphCast was trained on ERA-5 data. The PSD-based spectral metrics characterize coherent, domain-scale variability rather than a spatial average of point-wise spectral behavior
.
This study presents the first systematic climatological evaluation of Google’s GraphCast AI weather prediction system for the Indian Summer Monsoon. The principal findings are that GraphCast reproduces broad spatial organization but suffers from a substantial domain-averaged wet bias and a dramatic suppression of rainfall variance across nearly all timescales
.
The analysis is organized around three complementary diagnostic frameworks: (i) the climatological mean state and annual cycle, (ii) the thermodynamic and dynamic drivers of monsoon variability as captured by Q1/Q2 heating profiles and the Differential Tropospheric Temperature (DTT) index, and (iii) rainfall variability across synoptic to intraseasonal timescales.
The study provides benchmark diagnostics for next-generation AI weather models
by mapping GraphCast biases onto physically interpretable metrics. These results aim to provide actionable benchmarks for the ongoing development of AI weather prediction systems
.
The study provides benchmark diagnostics for next-generation AI weather models by mapping GraphCast biases onto physically interpretable metrics. The results provide actionable benchmarks for the ongoing development of AI weather prediction systems
.
The analysis involves computing four parallel diagnostic modules—mean-state, thermodynamic, dynamic, and variability metrics—independently at each forecast lead time and evaluating them against the reference datasets using the verification metrics described.
The results are synthesized in Section 6 to attribute systematic error sources to specific physical processes
by linking mean-state, thermodynamic, dynamic, and variability biases. This synthesis is intended to provide a physically grounded basis for interpreting model biases in the thermodynamic budget
.
The results are synthesized in Section 6 to attribute systematic error sources to specific physical processes by linking mean-state, thermodynamic, dynamic, and variability biases. This synthesis is intended to provide a physically grounded basis for interpreting model biases in the thermodynamic budget
.
The study uses standard verification metrics including spatial correlation coefficient (SCC), root-mean-square error (RMSE), and mean bias for all evaluated fields. These are computed for both nested domains: a Core Monsoon Zone and the full ISM Domain. p. 5 Spatial verification statistics (area-weighted Pearson pattern correlation, bias, and
RMSE on the JJAS-mean climatological field) are reported alongside temporal verification statistics (bias, correlation, and RMSE on the area-weighted domain-mean
daily time series, matched by exact calendar-date intersection) for two nested domains:
a Core Monsoon Zone (15°–25°N, 75°–90°E) and the full ISM Domain (5°–35°N, 60°–
100°E). These two halves of the verification statistics answer distinct questions: the
spatial statistics assess climatological pattern fidelity (which is largely lead-invariant
by construction, since it is computed on a multi-year mean field), while the temporal
6Where the paper says it
The verification metrics allow researchers to assess Temporal (day-to-day) correlation against IMERG is
markedly lower than the spatial pattern correlation (0. 91 spatially)
and degrades slightly with lead time, indicating that GraphCast’s skill at reproducing the climatological rainfall pattern substantially exceeds its skill at reproducing
the correct rainfall amount on any given day — a distinction that matters for any
application requiring day-specific rather than climatological guidance. Temporal (day-to-day) correlation against IMERG is
markedly lower than the spatial pattern correlation (0. 91 spatially)
and degrades slightly with lead time, indicating that GraphCast’s skill at reproducing the climatological rainfall pattern substantially exceeds its skill at reproducing
the correct rainfall amount on any given day — a distinction that matters for any
application requiring day-specific rather than climatological guidance.climatological pattern fidelity
in the spatial statistics and day-to-day forecast skill
in the temporal statistics. This distinction is crucial for understanding how GraphCast performs at different scales. p. 8Where the paper says it
Where the paper says it
The verification metrics allow researchers to assess climatological pattern fidelity
in the spatial statistics and day-to-day forecast skill
in the temporal statistics. This distinction is crucial for understanding how GraphCast performs at different scales.
The study uses a two-tailed Student’s t-test at the 5% level applied to the paired daily series, with a reduced effective sample size Neff to account for strong day-to-day persistence.
The use of this test is necessary because Where differences in domain-mean daily time series are tested for significance, we use
a two-tailed Student’s t-test at the 5% level applied to the paired daily series, but
with the sample size reduced to an effective sample size Neff = N (1 − r1)/(1 + r1),
where r1 is the lag-1 autocorrelation of the series, to account for the strong day-to-day
persistence of monsoon rainfall and circulation; this deflation is necessary because the
nominal daily sample size otherwise overstates the true number of independent degrees
of freedom and inflates significance. For spatially aggregated fields, the corresponding
reduction in spatial degrees of freedom due to spatial autocorrelation is acknowledged, and we therefore emphasize effect sizes (bias magnitudes and correlation values)
alongside, rather than in place of, formal significance. Where differences in domain-mean daily time series are tested for significance, we use
a two-tailed Student’s t-test at the 5% level applied to the paired daily series, but
with the sample size reduced to an effective sample size Neff = N (1 − r1)/(1 + r1),
where r1 is the lag-1 autocorrelation of the series, to account for the strong day-to-day
persistence of monsoon rainfall and circulation; this deflation is necessary because the
nominal daily sample size otherwise overstates the true number of independent degrees
of freedom and inflates significance. For spatially aggregated fields, the corresponding
reduction in spatial degrees of freedom due to spatial autocorrelation is acknowledged, and we therefore emphasize effect sizes (bias magnitudes and correlation values)
alongside, rather than in place of, formal significance.the nominal daily sample size otherwise overstates the true number of independent degrees of freedom and inflates significance
. This methodological rigor ensures that formal significance is not mistaken for true skill. p. 6Where the paper says it
Where the paper says it
The use of this test is necessary because the nominal daily sample size otherwise overstates the true number of independent degrees of freedom and inflates significance
. This methodological rigor ensures that formal significance is not mistaken for true skill.
The spectral analysis shows that GraphCast’s rainfall spectrum sits below IMERG’s at virtually every frequency, not only within a specific band
. This indicates that GraphCast’s rainfall time series retains only about 14% of IMERG’s total rainfall temporal variance regardless of lead
.
This spectral suppression is a robust property, indicating that The distribution of rainfall amount by intensity class (area-weighted
contribution to total JJAS rainfall per 2 mm day−1 bin) is used to assess whether
GraphCast reproduces the contribution of high-intensity events to total seasonal
rainfall, independent of its mean-state or temporal-variance performance. Spatial verification statistics (area-weighted Pearson pattern correlation, bias, and
RMSE on the JJAS-mean climatological field) are reported alongside temporal verification statistics (bias, correlation, and RMSE on the area-weighted domain-mean
daily time series, matched by exact calendar-date intersection) for two nested domains:
a Core Monsoon Zone (15°–25°N, 75°–90°E) and the full ISM Domain (5°–35°N, 60°–
100°E). The distribution of rainfall amount by intensity class (area-weighted
contribution to total JJAS rainfall per 2 mm day−1 bin) is used to assess whether
GraphCast reproduces the contribution of high-intensity events to total seasonal
rainfall, independent of its mean-state or temporal-variance performance. Spatial verification statistics (area-weighted Pearson pattern correlation, bias, and
RMSE on the JJAS-mean climatological field) are reported alongside temporal verification statistics (bias, correlation, and RMSE on the area-weighted domain-mean
daily time series, matched by exact calendar-date intersection) for two nested domains:
a Core Monsoon Zone (15°–25°N, 75°–90°E) and the full ISM Domain (5°–35°N, 60°–
100°E). The evaluation encompasses the climatological mean state (spatial distribution of rainfall, surface temperature, and low-level winds), the distribution
of rainfall by intensity class, thermodynamic drivers (tropospheric temperature
gradient, DTT; apparent heat source and moisture sink, Q1 and Q2), monsoon dynamics (vertical wind shear, meridional structure), and rainfall variability
across timescales (intraseasonal, synoptic, and spectral power). Results demonstrate that GraphCast reproduces the broad spatial pattern of monsoon rainfall
and the timing of the annual cycle with considerable fidelity at short leads, but
exhibits a domain-averaged wet bias over the core monsoon zone alongside a
pronounced suppression of rainfall variance across nearly all timescales (regional
power-spectrum variance ratio ≈0. The evaluation encompasses the climatological mean state (spatial distribution of rainfall, surface temperature, and low-level winds), the distribution
of rainfall by intensity class, thermodynamic drivers (tropospheric temperature
gradient, DTT; apparent heat source and moisture sink, Q1 and Q2), monsoon dynamics (vertical wind shear, meridional structure), and rainfall variability
across timescales (intraseasonal, synoptic, and spectral power). Results demonstrate that GraphCast reproduces the broad spatial pattern of monsoon rainfall
and the timing of the annual cycle with considerable fidelity at short leads, but
exhibits a domain-averaged wet bias over the core monsoon zone alongside a
pronounced suppression of rainfall variance across nearly all timescales (regional
power-spectrum variance ratio ≈0.the deterministic-model smoothing signature
is a significant limitation for subseasonal prediction where the BSISO provides the primary source of predictability. The high fidelity of the 10-m wind field, with its Total Var Ratio of 0. p. 5Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
This spectral suppression is a robust property, indicating that the deterministic-model smoothing signature
is a significant limitation for subseasonal prediction where the BSISO provides the primary source of predictability". The high fidelity of the 10-m wind field, with its Total Var Ratio of 0.98–1.01, contrasts sharply with this rainfall suppression.
The study concludes that the convergence of three independent diagnostics (intensity distribution, spectral variance, and spatial ISV amplitude) on a single deterministic-smoothing interpretation for the rainfall biases
is the most actionable finding. This convergence suggests that "a model trained to minimize expected error under genuine day-to-day forecast uncertainty in tropical convection is incentivized toward a smoothed, moderate-amplitude output rather than committing to the (uncertain, occasionally extreme) true realization".
This convergence suggests that "a model trained to minimize expected error under genuine day-to-day forecast uncertainty in tropical convection is incentivized toward a smoothed, moderate-amplitude output rather than committing to the (uncertain, occasionally extreme) true realization. This interpretation is consistent with the
markedly higher spectral and intensity fidelity of the 10-m wind field".
The convergence of three independent diagnostics on a single deterministic-smoothing interpretation for the rainfall biases
is the most actionable finding. This convergence suggests that "a model trained to minimize expected error under genuine day-to-day forecast uncertainty in tropical convection is incentivized toward a smoothed, moderate-amplitude output rather than committing to the (uncertain, occasionally extreme) true realization".
The authors note that the comparison of GraphCast against ERA-5 atmospheric variables is not entirely independent since GraphCast is trained on ERA-5. This means biases in upper-level fields may reflect genuine model error or, alternatively, the model’s divergence from the reanalysis under real initial conditions
.
The use of IMERG as an independent precipitation benchmark partially mitigates this concern for the rainfall evaluation
. This allows for a more robust assessment of GraphCast's performance on precipitation specifically.
The use of IMERG as an independent precipitation benchmark partially mitigates this concern for the rainfall evaluation
. This allows for a more robust assessment of GraphCast's performance on precipitation specifically.
The study is confined to the medium range, focusing on leads of +24h to +72h
. The present evaluation is confined to the medium range (+24h to +72h)
.
This focus allows for a detailed assessment of GraphCast's utility for Our analysis is organized around three complementary diagnostic frameworks: (i) the climatological mean state and annual cycle,
(ii) the thermodynamic and dynamic drivers of monsoon variability as captured by
Q1/Q2 heating profiles and the Differential Tropospheric Temperature (DTT) index,
and (iii) rainfall variability across synoptic to intraseasonal timescales. By mapping
GraphCast biases onto physically interpretable metrics, we aim to provide actionable
benchmarks for the ongoing development of AI weather prediction systems. Our analysis is organized around three complementary diagnostic frameworks: (i) the climatological mean state and annual cycle,
(ii) the thermodynamic and dynamic drivers of monsoon variability as captured by
Q1/Q2 heating profiles and the Differential Tropospheric Temperature (DTT) index,
and (iii) rainfall variability across synoptic to intraseasonal timescales. By mapping
GraphCast biases onto physically interpretable metrics, we aim to provide actionable
benchmarks for the ongoing development of AI weather prediction systems.tropical medium-range forecasting
. This specificity is important for guiding the development of AI weather models. p. 2Where the paper says it
Where the paper says it
This focus allows for a detailed assessment of GraphCast's utility for tropical medium-range forecasting
. This specificity is important for guiding the development of AI weather models.
The authors note that the availability of GraphCast probabilistic ensemble products from 2025 onward offers an opportunity to examine whether ensemble spread can compensate for the variance suppression identified here
. This suggests that probabilistic skill scores (CRPS, reliability diagrams) reveal additional sources of forecast uncertainty over the monsoon domain
.
Examining ensemble products will help determine if ensemble spread can compensate for the variance suppression identified here
. This future work is motivated by the need to assess whether ensemble spread can compensate for the variance suppression identified here
.
Examining ensemble products will help determine if ensemble spread can compensate for the variance suppression identified here
. This future work is motivated by the need to assess whether ensemble spread can compensate for the variance suppression identified here
.
The authors state that their analysis code is available upon reasonable request to the corresponding author. This ensures transparency in the research process. p. 2 GraphCast has demonstrated competitive
or superior performance relative to the ECMWF operational analysis on standard
verification metrics such as the anomaly correlation coefficient (ACC) and root-meansquare error (RMSE) for mid-latitude geopotential height and wind fields (Lam et al,
2023). Standardized benchmarks such as WeatherBench 2 (Rasp et al, 2024) have since
accelerated headline-metric intercomparison of these systems, but such global-mean
skill scores can mask process-level regional biases; in particular, recent work shows
that current data-driven models tend to produce overly smooth forecasts with suppressed small-scale spectral energy that worsens with forecast lead time (Bonavita,
2024).Where the paper says it
Making analysis code available Our analysis is organized around three complementary diagnostic frameworks: (i) the climatological mean state and annual cycle,
(ii) the thermodynamic and dynamic drivers of monsoon variability as captured by
Q1/Q2 heating profiles and the Differential Tropospheric Temperature (DTT) index,
and (iii) rainfall variability across synoptic to intraseasonal timescales. By mapping
GraphCast biases onto physically interpretable metrics, we aim to provide actionable
benchmarks for the ongoing development of AI weather prediction systems. Our analysis is organized around three complementary diagnostic frameworks: (i) the climatological mean state and annual cycle,
(ii) the thermodynamic and dynamic drivers of monsoon variability as captured by
Q1/Q2 heating profiles and the Differential Tropospheric Temperature (DTT) index,
and (iii) rainfall variability across synoptic to intraseasonal timescales. By mapping
GraphCast biases onto physically interpretable metrics, we aim to provide actionable
benchmarks for the ongoing development of AI weather prediction systems.upon reasonable request to the corresponding author
ensures transparency in the research process. This practice supports reproducibility in AI weather model development. p. 2Where the paper says it
Where the paper says it
Making analysis code available upon reasonable request to the corresponding author
ensures transparency in the research process. This practice supports reproducibility in AI weather model development.
The authors declare no conflicts of interest and state that ethics approval and consent to participate are not applicable. This confirms the study's independence from external funding or ethical review processes.
Declaring no conflicts of interest and stating that Despite decades
of advancement in numerical weather prediction (NWP), the ISM continues to pose
fundamental challenges to physics-based models, arising from complex interactions
among the large-scale monsoonal circulation, mesoscale orographic systems (particularly the Western Ghats and the Himalayan foothills), organized convection, and
ocean–atmosphere coupling on intraseasonal timescales (Sperber et al, 2013; Goswami,
2005). The recent emergence of machine-learning-based global weather prediction systems
has generated considerable scientific interest and sparked debate about the prospects
2 Despite decades
of advancement in numerical weather prediction (NWP), the ISM continues to pose
fundamental challenges to physics-based models, arising from complex interactions
among the large-scale monsoonal circulation, mesoscale orographic systems (particularly the Western Ghats and the Himalayan foothills), organized convection, and
ocean–atmosphere coupling on intraseasonal timescales (Sperber et al, 2013; Goswami,
2005). The recent emergence of machine-learning-based global weather prediction systems
has generated considerable scientific interest and sparked debate about the prospects
2ethics approval and consent to participate are not applicable
confirms the study's independence from external funding or ethical review processes. This supports the integrity of the scientific findings. p. 1Where the paper says it
Where the paper says it
Declaring no conflicts of interest and stating that ethics approval and consent to participate are not applicable
confirms the study's independence from external funding or ethical review processes. This supports the integrity of the scientific findings.
The paper is organized as follows, describing Section 2 (data and methods), Section 3 (climatological mean-state evaluation), Section 4 (thermodynamic and dynamic structure), Section 5 (rainfall variability and propagation), Section 6 (physical mechanisms underlying identified biases and their implications), and Section 7 (summarizes conclusions).
This organization allows readers to follow the logical flow from data ingestion through four parallel diagnostic modules to the final physical synthesis of lead-time-dependent biases. This structure facilitates a comprehensive understanding of the st… p. 2 Third, the intraseasonal variability of the monsoon, particularly the Boreal Summer Intraseasonal Oscillation (BSISO; Lee et al, 2013) and its
northward-propagating counterpart, constitutes the primary source of predictability
on 2–4 week timescales—precisely the range where AI models may offer the greatest
advantage if their biases are well characterized. This study for the first time addresses these gaps by conducting a comprehensive
climatological evaluation of GraphCast deterministic hindcasts initialized at 00 UTC
over 2021–2024 against ERA-5 reanalysis (Hersbach et al, 2020) and IMERG (Huffman
et al, 2020) precipitation retrievals.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Training models to explicitly account for deterministic smoothing signatures during rainfall prediction will improve accuracy by addressing how GraphCast produces a
deterministic-smoothing signature in GraphCast’s precipitation field that is consistent across independent diagnostics.
This will result in better representation of high-intensity rainfall events, which are currently underrepresented. -
Incorporate mechanisms to better resolve lower-tropospheric heating by improving the representation of Q1 profiles below 700 hPa, as GraphCast exhibits a
pronounced deficit in lower-tropospheric Q1 (below 700 hPa) is apparent in GraphCast.
This will enhance the model's ability to simulate orographic precipitation enhancement over features like the Western Ghats. -
Implement techniques to improve subseasonal propagation fidelity by explicitly modeling air-sea coupling, as
the absence of air–sea coupling in GraphCast prevents the positive feedback between warm SST anomalies and enhanced surface evaporation north of active convective cells.
This will lead to more coherent northward propagation of the BSISO signal.
Abstract
The Indian Summer Monsoon (ISM) is one of the most dynamically complex phenomena in the global climate system, yet its accurate prediction remains challenging for both physics-based and data-driven models. This study presents a systematic climatological evaluation of Google's GraphCast, a machine-learning global weather prediction system, against ERA-5 reanalysis and IMERG satellite precipitation for the boreal summer season (June-September, JJAS) over 2021-2024. We analyze deterministic GraphCast forecasts initialized at 00 UTC across 6-hourly steps to assess +24h, +48h, and +72h performance over the full ISM domain. The evaluation covers the climatological mean state (spatial rainfall, surface temperature, low-level winds), rainfall intensity distributions, thermodynamic drivers (tropospheric temperature gradient, DTT; apparent heat source and moisture sink, Q1 and Q2), monsoon dynamics (vertical wind shear, meridional structure), and variability across intraseasonal, synoptic, and spectral scales. GraphCast reproduces the spatial pattern and annual cycle of monsoon rainfall with high fidelity at short leads, showing a near-zero domain-averaged rainfall bias over the core monsoon zone (-0.7 to-1.2 mm/day) and high daily temporal correlation with IMERG (0.89-0.92). However, temporal variance is moderately suppressed across timescales (power-spectrum variance ratio 0.63), and the intensity distribution systematically underrepresents extreme events and the 95th percentile. GraphCast closely captures the vertical structure and seasonal evolution of Q1 and Q2, but degrades in northward-propagating intraseasonal variability. Overall, GraphCast shows a moderate underrepresentation of rainfall intensity and variance rather than a wet bias or variance collapse, providing benchmark diagnostics for AI weather models.
Sources
Related papers
- Anthropogenic Heat in Urban Climate Systems: Forcing, Sensitivity, and Feedback
- NORi: An ML-Augmented Ocean Boundary Layer Parameterization
- A Checklist to assess the energy and carbon impacts of ML/AI applications in Earth System Modeling
- Skillful Thunderstorm Forecasts Learned Directly from Observations
- Wind vector retrieval from miniaturized wave-enabled sea-surface drifters
- Stochastic coupling of climate variables and ice volume over the Late Pleistocene glacial cycles