Do AI Forecast Ensembles Sample the Correct Conditional Distribution?

The paper demonstrates that a trained probabilistic AI forecast model can have positive marginal skill while simultaneously failing to capture the correct joint conditional distribution. Specifically, a denoising diffusion probabilistic model (DDPM) trained to forecast subseasonal coastal sea surface height (SSH) anomalies at eight US East Coast tide gauge stations achieves positive skill at every station and lead time marginally, while joint spatial structure is worse than climatological draws. The authors state: "We train a diffusion model for probabilistic subseasonal coastal sea level forecasts at eight US East Coast tide gauge stations, with sea level derived from reanalysis, and find that marginal and joint forecast quality decouple: positive skill at every station and lead time marginally, while joint spatial structure is worse than climatological draws."

The evaluation uses three metrics: continuous ranked probability score (CRPS) for marginal skill, energy score (ES) and variogram score (VS) for joint skill. The results show: "The DDPM achieves statistically significant positive CRPS skill at all four lead times... Energy Score skill is positive for short leads and slightly negative for longer leads... In contrast, Variogram Score skill is negative at all lead times, indicating that the joint spatial structure of the DDPM ensemble is worse than a climatological draw despite the positive marginal skill. A shuffle-based permutation decomposition reveals that this failure is invisible to the energy score but detected by the variogram score. The authors explain: ES is dominated by marginal contributions and is largely insensitive to the multivariate structure of the forecast, while VS clearly detects a structural failure of the distribution represented by the ensemble."

The pairwise analysis shows the failure is spatially heterogeneous: "Most station pairs have negative VS skill, indicating the DDPM ensemble is worse than climatology for nearly all pairs. The skill deficit is strongly heterogeneous: the Atlantic City-Battery pair is an extreme outlier, with the highest observed correlation (r=0.98) and the largest VS deficit, consistent with the DDPM over-correlating this already highly correlated pair."

To isolate the role of training data volume, the authors conduct Lorenz-96 experiments across training volumes equivalent to 0.7-170 years of ERA5 data. They find: "At a lead of 5 days (1 TU, approximately 1.7 Lyapunov times), both CRPS and VS skill increase with training volume, but CRPS skill is consistently higher than VS skill and a statistically significant gap between CRPS and VS skill is present at all training volumes tested... At a lead of 10 days (2 TU, approximately 3.3 Lyapunov times), CRPS skill continues to grow with training volume while VS skill remains near zero across all training set sizes, demonstrating a near-complete decoupling of marginal and joint forecast improvement. The gap does not close with increasing training data volume out to an equivalent of 2X the ERA5 record, suggesting the failure is due to structural inadequacy of the distribution represented by the models output rather than insufficient training data. A linear multivariate normal baseline exhibits the same qualitative pattern at every lead and training volume: CRPS skill tracks closely with the DDPM while VS skill remains near zero or below, indicating the CRPS-VS decoupling observed in the SSH forecast therefore should not be dismissed as a consequence of insufficient training data."

The authors also compare against a dynamical ensemble and a deterministic emulator. The dynamical ensemble does not replicate the failure mode while a deterministic emulator reproduces the failure mode. Specifically: "In the quasi-linear dynamical regime (σic = 0.01), the VS/CRPS ratio is approximately 1.0 (ratio = 1.02), confirming that chaotic divergence alone does not produce a joint calibration deficit. When σic is tuned to match the DDPM ensemble spread (σic = 0.158), the dynamical ratio falls modestly to 0.91... The deterministic emulator performs worst at both noise levels (VS/CRPS 0.42-0.58), consistent with a deterministic forward map collapsing ensemble diversity in a way that disproportionately harms joint relative to marginal calibration. Both AI-based methods show a substantially lower VS/CRPS ratio than the dynamical ensemble at matched spread, suggesting the joint calibration gap is a property of data-driven emulation rather than a consequence of ensemble spread or nonlinear divergence alone."

The paper concludes with four key findings: "1) probabilistic generative AI forecast tools may sample from a distribution that does not properly represent the joint conditional structure of the true uncertainty, 2) additional training data cannot be assumed to remedy this failure mode, 3) CRPS and ES used as evaluation or training metrics will not reliably identify or mitigate this failure mode, and 4) the failure mode is reproduced for a deterministic AI emulator but not a dynamical model. The authors also note consistency with recent work on AI data assimilation showing covariance errors as a fundamental limitation and AI emulators producing ensembles that are miscalibrated in a multivariate sense even as they compare favorably on standard skill metrics. The final recommendation is that AI forecast ensembles may appear skillful while simultaneously representing the multivariate distribution incorrectly with no guarantee this failure will improve with increased training data quantity. This strongly supports the continued use of dynamical and physics-based models to complement AI forecasts."

Improvements to AI Systems:

1. Add multivariate calibration metrics to training objectives. Modify the loss function of generative forecast models (e.g., DDPMs) to include a differentiable approximation of the variogram score (VS) alongside CRPS. This forces the model to optimize joint spatial correlations, not just marginal distributions, preventing the over-correlation failure seen in the SSH case.

2. Implement a post-hoc covariance correction module. After sampling an ensemble from a generative model, apply a learned or analytical transformation (e.g., a Cholesky-based re-scaling of the ensemble covariance) that matches the climatological or dynamical covariance structure. This directly fixes the over-correlated pairs (e.g., Atlantic City–Battery) without retraining.

3. Add a joint-distribution discriminator during training. Train an auxiliary network that distinguishes between forecast ensemble samples and climatological samples based on their multivariate spatial structure (e.g., using a graph neural network over station pairs). Use its output as an adversarial loss term to penalize ensembles with unrealistic joint patterns, even when marginal skill is high.

4. Implement an ensemble diversity regularizer. For deterministic emulators, add a term that penalizes collapse of ensemble members in high-dimensional output space (e.g., minimizing the trace of the ensemble covariance or maximizing the minimum pairwise distance). This addresses the deterministic forward map collapsing ensemble diversity failure mode.

5. Create a hybrid AI-dynamical blending system. Instead of relying solely on AI forecasts, train a meta-model that combines AI ensemble outputs with a dynamical model's ensemble (e.g., via a weighted average or a learned gating mechanism). The meta-model can detect when the AI's joint structure is unreliable (e.g., via VS) and dynamically increase the dynamical model's weight, ensuring robust multivariate calibration.

6. Add a training-data-volume diagnostic to model selection. Before deployment, evaluate the model across multiple training set sizes (as in the Lorenz-96 experiments). If the CRPS-VS gap does not close with increased data (as shown), flag the model as structurally inadequate and require architectural changes (e.g., different noise schedules, latent space constraints) rather than simply adding more data.

7. Implement a spatial-correlation-aware sampling strategy. During inference, modify the diffusion sampling process to include a constraint that the generated ensemble's pairwise correlations fall within a climatological range (e.g., using a rejection sampling or guided diffusion step that penalizes samples with extreme correlations like r=0.98). This prevents over-correlation without altering the trained model.

8. Add a joint-calibration validation layer to AI forecast pipelines. Build a monitoring system that continuously computes VS and ES on held-out data. If VS skill drops below a threshold (e.g., negative while CRPS is positive), automatically trigger a retraining with a multivariate-focused loss or switch to a dynamical ensemble fallback.

What the improved AI system can do:

• Produce probabilistic forecasts that are simultaneously accurate at individual locations (marginal skill) and correctly represent spatial dependencies (joint skill), avoiding the decoupling failure.

• Detect and correct its own multivariate miscalibration in real-time, even when standard metrics (CRPS, ES) appear positive.

• Maintain robust ensemble diversity and realistic spatial correlations across all lead times and training data volumes.

• Provide reliable uncertainty estimates for spatially correlated variables (e.g., coastal sea level, weather fields) that can be safely used for risk assessment and decision-making.

• Seamlessly integrate with dynamical models to ensure physical consistency, especially in regimes where AI emulators are structurally inadequate.

Ensemble forecasting aims to sample the conditional distribution of outcomes; whether AI forecast ensembles do this correctly in a joint sense remains largely untested. We train a diffusion model for probabilistic subseasonal coastal sea level forecasts at eight US East Coast tide gauge stations, with sea level derived from reanalysis, and find that marginal and joint forecast quality decouple: positive skill at every station and lead time marginally, while joint spatial structure is worse than climatological draws. A shuffle-based permutation decomposition reveals this failure is invisible to the energy score but detected by the variogram score. Lorenz-96 experiments across 0.7-170 equivalent years show the gap persists regardless of training volume and is reproduced by a linear baseline, indicating structural inadequacy of the learned distribution. A dynamical ensemble does not replicate the failure while a deterministic emulator does, suggesting it is specific to learned emulators rather than ensemble forecasting generally.