Temporal Bridges for Spatial Resolution: Enhancing Climate Data Super-Resolution with Bidirectional Alignment

The paper proposes a novel framework for climate data super-resolution (SR), also termed climate data downscaling, which transforms low-resolution climate data into higher-resolution formats. The authors state that "High-resolution climate data is crucial for meteorological predictions and for informing decision support across diverse domains. However, the acquisition of such high-resolution climate information is often prohibitively costly, necessitating the development of data-driven meteorological prediction models. They identify a key research gap: recent advancements in deep learning for climate data SR have primarily focused on leveraging single-frame spatial information, largely neglecting the temporal correlations between different time frames that could enhance SR outcomes. Furthermore, climate data are inherently stochastic and noisy, rendering widely used temporal alignment methods, such as optical flow models, ineffective in this context."

To address this, the authors propose "a novel Temporal-Enhanced framework with bidirectional temporal alignment. In essence, our framework establishes a temporal bridge to enhance spatial resolution in climate data SR through bidirectional alignment, leading to improved SR performance. The framework has three components: Paired Latent Mapping achieves spatial alignment and noise reduction by unifying latent spaces. Then a Bidirectional Temporal Alignment captures temporal correlations by training forward and backward networks on consecutive latent frames. Temporal Enhanced Super-resolution then optimizes the entire framework for climate data SR. Experiments on large-scale real-world datasets demonstrated the superior performance of our framework."

The paper emphasizes that high-resolution climate information is essential for decision support in many sectors such as urban management, transportation optimization, and disaster prevention and that deploying sufficient sensors to collect such high-resolution climate information is often impractical due to cost and infrastructure barriers. While deep learning models like ForecastNet, Pangu-Weather, and GraphCast demonstrate the potential of training foundational weather prediction models, the authors note that existing data-driven models... typically predict at coarse resolutions (e.g., 30 × 30 km), which are insufficient for the decision-making needs of sectors such as urban management and disaster prevention.

The authors' central insight is that the inherent temporal correlations within climate data across different time steps have been largely overlooked. Existing models such as CNNs, GANs, and the ClimaX foundation model have primarily focused on exploiting spatial information within individual time frames, neglecting inter-time correlations. They hypothesize that incorporating these temporal relationships, in conjunction with spatial information, can enhance high-resolution outputs and improve SR performance.

The paper explains why video super-resolution (VSR) temporal techniques cannot be directly applied: "The general idea of VSR for leveraging temporal information involves utilizing optical flow models to estimate motion between consecutive frames. However, a fundamental assumption of these optical flow models is that pixel brightness remains relatively consistent over time... this assumption is invalid for climate data which exhibit inherently stochastic and prone to sudden changes. Additionally, observational limitations and instrumental inaccuracies often introduce substantial spatial noise into climate data... which can significantly hinder the performance of super-resolution tasks."

The contributions are summarized as: (1) a carefully designed Temporal Enhanced SR framework tailored for the climate data SR task which aims to significantly boost SR performance; (2) "a novel temporal alignment technique that can adapt to the stochastic nature of climate data and effectively leverages temporal information to enhance spatial details, establishing Temporal Bridges for Spatial Resolution in climate data SR; and (3) thorough evaluations on CMIP6 and ERA5 datasets, which demonstrates the superior performance of our framework in real-world datasets."

The objective is formalized as transforming low-resolution (LR) input frames, denoted as I LR ∈ R(T×V×H×W), into high-resolution (HR) output frames, denoted as I HR ∈ R(T×V×sH×sW), where "T denotes the size of the temporal window, V denotes the number of climate variables (e.g. temperature), s denotes the upsampling factor, and H and W denote the horizontal and vertical dimensions of the Earth's grid format respectively."

To address the noise issue, the authors develop a Paired Latent Mapping module that maps both high and low-resolution data into a unified and discrete latent space using Vector Quantised-Variational Autoencoder (VQ-VAE). Both low and high-resolution VQ-VAE models share the same codebook. This paired design ensures uniform feature representation across different spatial resolution. The module is formally defined as:

• Z LR(t) = q(ϕ LR(I LR(t)))

• Z HR(t) = q(ϕ HQ(I HR(t)))

• Î LR(t) = ω LR(Z LR(t))

• Î HR(t) = ω HR(Z HR(t))

where Z LR(t) and Z HR(t) denote the latent representations for low and high-resolution data inputs at time t, respectively, q(·) is the vector quantization operation, and ϕ and ω are encoder and decoder functions. Training starts with the high-resolution VQ-VAE (encoder with six layers of residual convolutional networks and four Transformer network layers; decoder with upsampling layers that mirror the encoder's structure), then replicates for low-resolution data using the same codebook trained from the high-resolution data, with the codebook's parameters frozen to ensure feature space consistency across different resolutions. Both Mean Squared Error (MSE) and commitment loss are used.

This module leverages temporal correlations: the temporal correlation between consecutive low-resolution time frames, Z LR(t − 1) and Z LR(t), shares common features with the correlation in high-resolution frames Z HR(t − 1) and Z HR(t). The forward and backward alignment networks are defined as:

• Ž HR(t) = ψ F(Z LR(t − 1), Z HR(t − 1), Z LR(t))

• Ž HR(t) = ψ B(Z LR(t + 1), Z HR(t + 1), Z LR(t))

Each network "initially concatenates inputs... processed through two Transformer blocks, followed by a convolutional block to refine the features, and then through two additional Transformer layers to predict Z HR(t). The network is trained using Mean Squared Error (MSE)."

Training occurs in two steps. Step 1 focuses on latent space: the spatial-temporal SR network ψ ST transforms Z LR(t) into Z HR(t) as Ẑ HR(t) = ψ ST(Z LR(t)). The temporal alignment networks are integrated as regularization: the loss function is

L = (1/T) Σ [MSE(ψ ST(Z LR(t)), Z HR(t)) + λ·MSE(ψ F(Z LR(t−1), Z LR(t), ψ ST(Z LR(t−1))), Z HR(t)) + λ·MSE(ψ B(Z LR(t+1), Z LR(t), ψ ST(Z LR(t+1))), Z HR(t))]

The authors note "the alignment network in Equation 4 is different from Equation 2. For instance, the high resolution input Z HR(t − 1) in ψ F is replaced by ψ ST(Z LR(t − 1)). This is to comply with the specifications of super-resolution tasks, which exclusively use low-resolution data as inputs. The ψ ST applies patch embedding and positional encoding to Z LR(t − 1), Z LR(t), and Z LR(t + 1)... processed through two Transformer modules and a 3D transposed convolution layer, producing the predicted values Ẑ HR(t − 1), Ẑ HR(t), and Ẑ HR(t + 1)."

Step 2 unfreezes the encoder and decoder, retains the alignment regularization, and fine-tunes using Latitude-weighted Mean Squared Error (LMSE) to account for the varying area sizes at different latitudes on a global grid:

L LMSE = Σ L(h) (Î HR − I HR)2 / (T × V × sH × sW), with latitude weighting factor L(h) = cos(lat(h)) / [(1/H) Σ cos(lat(h′))].

During inference: Z LR(t) = q(ϕ LR(I LR(t))); Ẑ HR(t) = ψ ST(Z LR(t)); Î HR(t) = ω HR(Z HR(t)), using the trained low-resolution encoder, high-resolution decoder, and SR network.

Following ClimaX, the dataset is constructed from CMIP6 (low-resolution input at 5.625° resolution, with six-hour interval, experiment ID 'historical', table ID '6hrPlevPt', variant label 'r1i1p1f1', regridded via xesmf bilinear interpolation) and ERA5 (high-resolution target at 1.40625° from WeatherBench, downsampled to six-hour intervals). The dataset spans 37 years (1979–2015): the training data is from 1979 to 2010, the validation data is in 2011 and 2012, and the test data is from 2013 and 2015. Five key variables are evaluated: geopotential at 500 mb (Z500), temperature at 850 mb (T850), 2-meter temperature (T2m), 10-meter u-component of wind (U10), and 10-meter v-component of wind (V10).

Baselines include ClimaX (climate SR model), VRT (video restoration transformer with optical flow), and SwinIR (adapted by replacing the internal Swin Transformer with a Video Swin Transformer). The metric is Latitude-weighted RMSE. Implementation uses learning rate 4.5e−5 for VQ-VAE training (batch size 24, 50 epochs, Adam), 4.5e−5 for temporal alignment (batch size 5, 50 epochs), and 1e−6 for SR training (batch size 1, 50 epochs), on four NVIDIA V100 GPUs.

Table 1 results (mean RMSE over five random runs, lower is better):

| Method | Z500 | T850 | T2m | U10 | V10 |

|---|---|---|---|---|---|

VRT 1098.95(9.94) 5.58(0.12) 6.33(0.18) 4.24(0.01) 4.24(0.01)

SwinIR 1099.07(3.41) 5.54(0.04) 6.22(0.12) 4.23(0.01) 4.24(0.02)

ClimaX 1088.42(2.00) 5.51(0.01) 6.11(0.02) 4.23(0.01) 4.24(0.01)

Our Method 1077.73(1.02) 5.41(0.01) 6.02(0.01) 4.22(0.01) 4.23(0.01)

The authors report our framework achieves the lowest RMSE across all variables... significantly outperforming all baselines. Notably, "VRT and SwinIR are quite close, despite VRT's use of optical flow models... This similarity indicates that optical flow models, despite their utility in video data for handling visual continuity and motion, may not effectively address the spatial and temporal variability in climate data. Our method even outperforms ClimaX despite ClimaX using more climate variables as input, with notably reducing the RMSE of Z500 from 1088.42 (2.00) to 1077.73 (1.02) and T2m from 6.11 (0.02) to 6.02 (0.01)."

The ablation removes the forward alignment network (w/o ψ F), backward alignment network (w/o ψ B), or all temporal alignment networks (w/o align). Results show "the bidirectional temporal alignment we designed significantly contributes to the super-resolution (SR) results. When ψ F and ψ B are removed, there is a substantial decline in model performance across all five variables. Additionally, the contributions of alignment in different directions vary among the variables. Compared to U10 and V10, the backward alignment appears to be more critical for Z500, T850, and T2m, as the removal of ψ B leads to greater drops in SR performances of these variables."

Table 2 comparison:

Model Training Time (h/epoch) Inference Time (s/instance) Model Size (MBytes)

|---|---|---|---|

| VRT | 0.598 | 0.058 | 30.7 |

| SwinIR | 0.130 | 0.016 | 25.1 |

| ClimaX | 0.423 | 0.081 | 110 |

| Our Model | 0.355 | 0.05 | 180 |

"Despite its larger size of 180 MBytes compared to VRT's 30.7 MBytes and ClimaX's 110 MBytes, our model maintains superior efficiency. It achieves a faster training time of 0.355 hours per epoch, outperforming VRT at 0.598 hours and ClimaX at 0.423 hours. Furthermore, it provides faster inference, taking 0.05 seconds per global instance... which is much quicker than 0.081 seconds of ClimaX and 0.058 seconds of VRT. The authors conclude this reveals our model's ability to achieve a balance between efficiency and high performance, demonstrating stronger practicality in real-world applications compared to others."

The paper concludes: "we develop a novel Temporal-Enhanced framework that utilizes temporal information to enhance SR performance. The core of the framework involves training two temporal alignment networks for SR task to achieve temporal alignment. This design effectively addresses the challenge of stochastic variations in climate data, which hinder the use of conventional optical flow methods in VSR. Experiments on CMIP6 and ERA5 datasets confirm the superior performance of our framework. The authors note limitations: We followed the practice in optical flow models, which mainly model the mapping relationships between adjacent time frames. As a result, our framework does not consider the temporal correlation between longer time frames. Future research could explore ways to integrate these longer temporal correlations to investigate whether they enhance the performance in climate data SR, and the framework could potentially be extended to similar climate datasets and the super-resolution of spatial-temporal data in other fields."

Based on this paper, here are specific improvements to AI systems and the resulting capabilities:

1. Add a bidirectional temporal alignment module to super-resolution models for stochastic, non-optical-flow data. Instead of relying on optical flow assumptions (constant brightness), train forward and backward alignment networks on consecutive latent frames. This captures temporal correlations in noisy, abruptly changing data. The improved AI system can super-resolve climate data, weather radar, atmospheric chemistry, or any geophysical field where optical flow fails, producing sharper, more accurate high-resolution outputs.

2. Use a shared codebook VQ-VAE paired latent mapping to unify low- and high-resolution representations. Train a high-resolution VQ-VAE, then train a low-resolution encoder against the frozen high-resolution codebook. The improved AI system can map noisy, low-resolution inputs into a clean, resolution-invariant latent space before temporal alignment and SR, reducing the impact of sensor noise and instrumental inaccuracies.

3. Regularize SR training with bidirectional temporal alignment networks that operate on predicted super-resolved frames, not ground-truth high-resolution frames. Replace the ground-truth high-res frame in the alignment network with the SR network's own prediction from the previous/next low-res frame. The improved AI system learns to produce temporally consistent high-resolution sequences even when only low-resolution data is available at inference, improving stability and reducing flicker or temporal artifacts.

4. Apply latitude-weighted loss (LMSE) when training on global grid data. Weight the reconstruction error by the cosine of latitude to account for varying grid-cell areas. The improved AI system can generate globally unbiased high-resolution fields, avoiding overfitting to polar regions and improving performance at mid-latitudes where most human decision-making occurs.

5. Use a two-stage training pipeline: first optimize the SR network in latent space with alignment regularization, then fine-tune the full encoder-decoder system end-to-end. The improved AI system benefits from stable, efficient convergence in the discrete latent space, followed by fine-tuning that restores fine pixel-level details in the final high-resolution output.

6. Replace optical-flow-based video SR backbones (e.g., VRT) with a lighter architecture using patch embedding, positional encoding, Transformer modules, and 3D transposed convolutions, regularized by bidirectional temporal alignment. The improved AI system achieves lower RMSE than ClimaX, VRT, and SwinIR while having faster training time than VRT and ClimaX, and faster inference than ClimaX and VRT—despite a larger model size—making it practical for operational weather forecasting.

• Downscale coarse climate model outputs (e.g., 5.625° → 1.40625°) with lower error on key variables including geopotential height (Z500), temperature (T850), 2-meter temperature, and 10-meter wind components, outperforming existing climate SR and video SR models.

• Produce temporally coherent high-resolution climate sequences from consecutive low-resolution frames, using both past and future frames via bidirectional alignment, without requiring optical flow or brightness-constancy assumptions.

• Handle noisy, stochastic climate observations by first mapping them into a shared, denoised discrete latent space, then performing alignment and super-resolution in that robust representation.

• Generate globally accurate high-resolution maps with latitude-aware loss, avoiding polar bias and improving decision support for urban management, transportation, and disaster prevention.

• Run efficiently in operational settings—about 0.05 seconds per global inference instance—enabling near-real-time downscaling of climate and weather forecasts for early warning systems.

• Transfer to other spatiotemporal domains with irregular, noisy, or non-rigid dynamics (e.g., ocean currents, air quality, satellite remote sensing) where temporal correlations exist but optical flow methods are ineffective.