Timestep-Conditioned Transformers for Global Weather Forecasting
Sam Levang, Fran Bartolic, Ty Dickinson, Chase Dwelle, Paulius Rauba, Viktor Cikojevic
cs.LG, cs.OS
Submitted: 2026-08-06
License: http://creativecommons.org/licenses/by/4.0/
The gist: Existing machine-learning weather forecasting models rely on predetermined and fixed autoregressive timesteps.
Terminology
Abstract
Existing machine-learning weather forecasting models rely on predetermined and fixed autoregressive timesteps. The choice of model timestep involves a fundamental trade-off: shorter timesteps (e.g. 1 to 6 hours) finely resolve atmospheric dynamics within the diurnal cycle but increase error accumulation for a given forecast horizon, while longer timesteps (e.g. 24 hours) reduce error accumulation but limit the usability of short-range forecasts where sub-daily predictability is high. In this work, we present GEM-3, a probabilistic global weather model that addresses this trade-off through explicit multi-timestep inference. With a single set of trained weights, the model timestep can be configured at inference time to balance predictability and usability across a broad forecast horizon. Additionally, we find that mixed-timestep training consistently improves rollout stability relative to timestep-specialist models. Under the hood, GEM-3 is a lightweight neighborhood-attention transformer with 134M parameters on an equirectangular grid with a number of architectural advancements beyond its predecessor GEM-2. The result is a practical forecasting system that couples near-SOTA medium-range probabilistic skill, stable extended-range rollouts, efficient training and inference, and decision-relevant diagnostics.
Sources
- Skillful joint probabilistic weather forecasting from marginals
- GraphDOP: Towards skilful data-driven medium-range weather forecasts learnt and initialised directly from observations
- FourCastNet 3: A geometric approach to probabilistic machine-learning weather forecasting at scale
- ArchesWeather & ArchesWeatherGen: a deterministic and generative model for efficient ML weather forecasting
- Forecasting Global Weather with Graph Neural Networks
- Demystifying Data-Driven Probabilistic Medium-Range Weather Forecasting
- AIFS -- ECMWF's data-driven forecasting system
- Can AI Weather Models Predict Beyond Two Weeks? A Quantitative Benchmark and Analysis of Long Rollouts
- Muon is Scalable for LLM Training
- OmniCast: A Masked Latent Diffusion Model for Weather Forecasting Across Time Scales
- FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators
- GenCast: Diffusion-based ensemble forecasting for medium-range weather
- Probabilistic Transformers for Joint Modeling of Global Weather Dynamics and Decision-Centric Variables
- Swift: An Autoregressive Consistency Model for Efficient Weather Forecasting
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks