Sensitivity Fidelity Reveals Physical Dependencies Beyond Forecast Accuracy in Deep-Learning Earth-System Models

arXiv:2610.15540 · cs.LG, physics.ao-ph, physics.geo-ph · Submitted 2026-10-11 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Sensitivity Fidelity Reveals Physical Dependencies Beyond Forecast Accuracy in Deep-Learning Earth-System Models".

Jane: The gist: Sensitivity fidelity reveals physical dependencies beyond forecast accuracy in deep-learning Earth-system models by examining model responses to input perturbations against established physical knowledge.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper, "Sensitivity Fidelity Reveals Physical Dependencies Beyond Forecast Accuracy in Deep-Learning Earth-System Models," and what they're arguing is that just having a good forecast number doesn't mean the model actually learned how the real world works.

Jane: Right, so if a deep learning model makes an accurate prediction but it’s based on some weird, unphysical internal connections, that accuracy doesn't tell us anything about the actual physics of Earth systems.

Lu: The authors introduce sensitivity fidelity as a new check alongside the usual prediction error to see if those learned connections line up with established physical knowledge or what we know from simulations.

Meng: That sounds like a necessary step when you start deploying these complex models, because if they’re just memorizing patterns without understanding the underlying dynamics, that's dangerous at scale.

Lalam: They set up this framework where you take a trained deep learning model and you backpropagate an objective function to get these input sensitivities, S X = d J / d X (<ref:2610.15540#pg2>).

Tom: So they take the output we want to predict,, and they calculate this sensitivity field, which tells us how much each input variable X affects that specific output objective J (<ref:2610.15540#pg2>).

Jane: And then the main idea is to use this sensitivity field to directly compare it against what we know from established physical knowledge or a physics-based adjoint model (<ref:2610.15540#pg3>).

Lu: They test this in a controlled benchmark, specifically Rayleigh–Bénard convection, where they train four different surrogate architectures—FNO, AFNO, Swin Transformer, and IFactFormer-RB—on identical trajectories (<ref:2610.15540#pg2>).

Tom: And the key finding there was that IFactFormer-RB reproduced the adjoint reference evolution most faithfully (<ref:2610.15540#pg3>).

Meng: They gave us some concrete numbers on that fidelity, saying IFactFormer-RB had a mean relative L2 error below one for all but one trajectory, with a mean of approximately zero point six (<ref:2610.15540#pg2>).

Jane: That’s pretty strong evidence for matching the known physics in that specific setup, because it shows how well the model's internal responses align with the physics we already have modeled for that convection problem (<ref:2610.15540#pg3>).

Lalam: And they showed that the averaged cosine similarity was about zero point eight, which means it had better agreement with the adjoint pattern (<ref:2610.15540#pg2>).

Tom: So, to wrap up this paper, "Sensitivity Fidelity Reveals Physical Dependencies Beyond Forecast Accuracy in Deep-Learning Earth-System Models," they're arguing that prediction skill and sensitivity fidelity are two different things for a model (<ref:2610.15540#pg1>).

Jane: They’re showing that just having an accurate deterministic forecast doesn't guarantee the model is correctly propagating uncertainty from the initial conditions (<ref:2610.15540#pg2>).

Lu: The implication is that if the sensitivities are wrong, then even a highly accurate prediction based on those wrong sensitivities won't be physically meaningful (<ref:2610.15540#pg3>).

Meng: For deploying these models in production, this means we can’t just trust the output; we have to validate the underlying gradients against what physics expects (<ref:2610.15540#pg3>).

Lalam: The authors suggest that derivative-informed and sensitivity-constrained training might improve this fidelity without hurting predictive skill if you have that adjoint information available (<ref:2610.15540#pg3>).

Tom: So they provide a way to ground these black-box models in physical reality using gradient checks, which is a practical test of whether the learned gradients are actually meaningful (<ref:2610.15540#pg3>).

Jane: It’s about making sure that when we use these AI models for things like weather forecasting, we aren't just getting a statistically good answer, but an answer that actually respects the laws of physics (<ref:2610.15540#pg3>).

Conclusion: Tom: So we've been talking about how these deep learning models can be accurate but still totally disconnected from what actually happens in nature, and this paper calls that sensitivity fidelity.

Jane: Right, so they are looking at the title itself: "Sensitivity Fidelity Reveals Physical Dependencies Beyond Forecast Accuracy in Deep-Learning Earth-System Models." That's the core idea, isn't it?

Lu: Exactly. It suggests that just getting a good prediction number isn't enough to prove the model has learned the actual physical rules of Earth systems.

Meng: I mean, if we’re shipping these things into a real weather system, I need more than just low forecast error; I need to know *why* it predicts what it does so I can debug when things go sideways.

Lalam: The authors are showing that the learned gradients—those sensitivities—are actually telling them how the model responds to changes in the input data, and they compare those responses against established physics.

Tom: And they use two big tests for this: first, a controlled setup like Rayleigh–Bénard convection where they can compare against a known physics-based adjoint model.

Jane: So, what’s the payoff for us? It means we can start using this as a tool to check if our black-box models are learning something sensible about the underlying dynamics.

Lu: The implication is that if the sensitivities don't match reality, then even a highly accurate deterministic forecast won't reliably tell us how uncertain initial conditions will propagate through the system.

Meng: That’s important for deployment because it shifts our focus from just chasing low error rates to making sure the model isn't learning some fake shortcut.

Tom: And they hint that training the models with sensitivity constraints might help improve that fidelity without ruining the predictive skill we’ve worked hard to get.

Jane: It gives us a new way to evaluate these models, moving beyond just looking at a single prediction number and actually checking how the model *thinks* about its inputs.

Qi Chen, Peng Zhan, Hongxing Cui, Wenqing Cui, Ibrahim Hoteit

King Abdullah University of Science and Technology · Southern University of Science and Technology

cs.LG, physics.ao-ph, physics.geo-ph

Submitted: 2026-10-11

Updated: 2026-10-11

Code: https://github.com/QiChen-7-c/Sensitivity-Fidelity-Test

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 86/100

The gist: The gist: Sensitivity fidelity reveals physical dependencies beyond forecast accuracy in deep-learning Earth-system models by examining model responses to input perturbations against established

Key concepts

Sensitivity Fidelity: This is a new evaluation criterion that checks if a deep learning model's internal responses to small changes in its input match known physical laws. It goes beyond just checking if the final forecast number is correct; it verifies that the model understands *why* it made that prediction by looking at how sensitive its output is to tiny input perturbations.
Sensitivity Calculation (SX): This involves calculating a field where each value shows how much the model's objective function changes when a single input variable is slightly altered. This calculation uses backpropagation through the trained model to map these local, first-order responses, revealing which parts of the input drive the forecast most strongly.
Adjoint Reference: An adjoint reference is a physics-based simulation run alongside the deep learning model that follows known physical equations exactly. This reference provides a 'ground truth' for what the sensitivities *should* look like under ideal conditions, allowing researchers to directly compare the model's learned sensitivities against this trusted physical pattern.

Terminology

Accuracy — 1× in this paper · explained in 1 episode
Fidelity — 1× in this paper · explained in 2 episodes

Summary

The gist: Sensitivity fidelity reveals physical dependencies beyond forecast accuracy in deep-learning Earth-system models by examining model responses to input perturbations against established physical knowledge.

Introduction and Motivation

Deep learning models are increasingly rivaling numerical simulation in Earth-system forecasting, yet forecast error alone cannot determine whether their learned input–output dependencies are physically consistent. This paper introduces sensitivity fidelity as an additional criterion alongside prediction error, requiring model responses to input perturbations to be examined against a trusted reference or established physical knowledge. While optimizing for prediction error yields high accuracy, it provides insufficient evidence that a model has learned the governing physical mechanisms. Strong predictive skill alone does not resolve the black-box nature of DL.

Sensitivity Fidelity Framework

The paper establishes a sensitivity-fidelity framework for evaluating differentiable forecast models through both predictive accuracy and the physical faithfulness of their gradients. This framework generates forecasts with a trained differentiable model and backpropagates a scalar forecast objective to obtain sensitivities to input features. The workflow involves three main components:

Forward Process: A trained DL model maps an input state or input sequence X to a forecast Yˆ, from which a scalar objective J(Yˆ) is selected.
Gradient-based Sensitivity: Backpropagation through the trained model yields the input sensitivity SX = ∂J(Yˆ)/∂X, which characterizes the learned local, first-order response of the objective to input perturbations.
Direct Fidelity Validation: In a controlled benchmark like Rayleigh–Bénard convection, neural sensitivities are directly compared with sensitivities from a matched physics-based adjoint for fidelity validation.

Benchmark Results: Rayleigh–Bénard Convection

In the idealized two-dimensional Rayleigh–Bénard (RB) convection benchmark, four surrogate architectures—FNO [38], AFNO [39, 40], Swin Transformer [41], and IFactFormer-RB [42, 43]—were trained on identical trajectories. IFactFormer-RB reproduced the adjoint reference evolution most faithfully, including displaced early-time temperature sensitivity and organized velocity sensitivities. Quantitative metrics confirmed these differences: IFactFormer-RB had a mean relative L2 error below one for all but one trajectory with a mean of approximately 0.6. Furthermore, the averaged cosine similarity showed that IFactFormer-RB had a mean similarity of approximately 0.8, indicating better agreement with the adjoint pattern.

Evaluation in Realistic Settings: ENSO Prediction

In the El Niño–Southern Oscillation (ENSO) application, where no adjoint reference exists, learned sensitivities were assessed through three complementary diagnostics. Statistical robustness was tested by averaging event gradients across 11 events during 1980–2021. Quantitative agreement was confirmed because simulated Niño 3.4 responses closely matched first-order estimates obtained from corresponding gradients under finite-amplitude perturbations. Finally, spatiotemporal evolution was consistent with established ENSO dynamics, showing positive SST sensitivity aligning with the onset of the Bjerknes positive feedback.

Conclusion and Future Directions

The study demonstrates that predictive accuracy and sensitivity fidelity are distinct properties of a differentiable surrogate. The results indicate that accurate deterministic forecasts do not guarantee faithful propagation of initial-condition uncertainty because inaccurate sensitivities lead to inaccurate responses to uncertain initial states. Future work should focus on determining whether derivative-informed and sensitivity-constrained training can improve sensitivity fidelity without compromising predictive skill when matched adjoint information is available. The framework provides a physically grounded criterion for model evaluation that complements conventional accuracy-based metrics.

Methods Summary

The sensitivity calculation is specified by the scalar objective J(Yˆ), where X denotes the input state or sequence, and the corresponding input sensitivity field is SX = ∂J(Yˆ)/∂X. This field provides a local, first-order estimate of how perturbations to each input variable affect the objective. In the RB benchmark, the objective JRB is defined as the mean temperature within the target region at forecast lead step 20. Sensitivities are computed with respect both to the initial temperature and velocity fields and to the model state at each intermediate lead step by backpropagating through the remainder of the rollout. For ENSO prediction, following Chen et al. [44], the objective is defined as the CTEFNet-predicted Niño 3.4 index for each target month associated with an El Niño event during 1980–2021.

Improvements for AI systems

Bold header: Sensitivity fidelity as a criterion for physical consistency

This improvement involves evaluating models alongside prediction error by examining model responses to input perturbations against a trusted reference or established physical knowledge, which provides a discriminating complement to forecast-error metrics.

Bold header: Gradient-based sensitivity evaluation in controlled benchmarks

For idealized systems like Rayleigh–Bénard convection, the system can be evaluated by comparing neural sensitivities with sensitivities from a matched physics-based adjoint to test whether forecast accuracy is accompanied by physically consistent gradients.

Bold header: Robustness and consistency checks for unreferenced applications

In applications lacking an adjoint reference, the system can be validated through statistical robustness across events, quantitative agreement with the model’s finite-perturbation responses and consistency with established physical mechanisms.