Do Deep Ensembles Actually Capture Uncertainty in Graph Neural Networks?

arXiv:2605.22593 · cs.LG · Submitted 2026-05-21 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Do Deep Ensembles Actually Capture Uncertainty in Graph Neural Networks?".

Jane: The gist The results demonstrate that deep ensembles largely do not capture uncertainty in graph neural networks,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, this paper, "Do Deep Ensembles Actually Capture Uncertainty in Graph Neural Networks?", it looks at deep ensembles—which are usually the go-to for uncertainty quantification in deep learning—and checks if they work well when you apply them to graph neural networks. Jane Basically, they assume that because they worked so well for things like computer vision, they should work fine on graphs too. Tom But this study looks at standard deep ensembles specifically on message-passing graph neural networks and finds that the improvements are actually pretty small compared to using just a single model. Jane They claim the marginal gains they see usually come from just stabilizing the noise in point predictions rather than getting better uncertainty estimates for what's going on in the data. Tom The big finding here is something called epistemic collapse, which means independently trained networks end up predicting nearly identical things to each other. Jane And since disagreement between models is how ensembles usually capture that model uncertainty, this lack of diversity basically cancels out their main advantage. Tom They even show that this collapse is so bad in classification tasks that the epistemic uncertainty becomes almost zero. Jane This suggests that what you get from random initializations isn't actually giving you different models, just slightly different versions of the same thing. Tom So, it’s a sober look at whether deep ensembles transfer their success seamlessly from classic deep learning settings to graphs.

Lu: From my side, I think this is interesting because it points toward how message passing might induce some form of convexity in the functional space, which restricts that functional diversity even in shallow networks nineteen.

Meng: From an engineering standpoint, if the disagreement is collapsing so fast, it means we can't rely on running many random initializations to get a diverse set of models. We need a different way to inject meaningful variation into the training process if we want these ensembles to actually do their job sixteen.

Lalam: If I could summarize this for you, it’s like having twenty identical robots trying different ways to build a bridge, but they all end up building the exact same bridge because the instructions are too restrictive <ref:2605.22593#pg5>.

Conclusion: Tom: So wrapping up this paper, "Do Deep Ensembles Actually Capture Uncertainty in Graph Neural Networks?", the authors Pedro C. Vieira, Pedro Ribeiro, and Viacheslav Borovitskiy found that deep ensembles mostly don't improve uncertainty quantification for graph neural networks. Jane They argue that the marginal gains they see are really just about reducing optimization noise in point predictions instead of getting better estimates for uncertainty itself. Tom The core issue they identify is epistemic collapse, where the independently trained models converge to overly similar predictions because there's no real disagreement left to capture model uncertainty. Jane This means that the mechanism deep ensembles use—having different models disagree—is neutralized because the individual graph neural networks don't end up being very different in their outputs. Tom It really suggests that for high-quality uncertainty quantification in graph learning, we need to look beyond just ensembling message-passing architectures. Jane A single "lucky" model can often do just as well as the whole ensemble, which is a big thing for practical application.

Lu: I see this as a strong push toward developing methods that deliberately encourage functional diversity in the GNNs themselves, rather than just relying on random initialization noise nineteen.

Meng: For me, it means our focus should shift from simple model aggregation to finding ways to make the underlying graph learning process itself more varied or robust against local optimization traps sixteen.

Lalam: If I were to try and improve culture based on this, I would say that we need more creative ways for the AI to explore its solution space, not just training it many times in the same way <ref:2605.22593#pg5>.

University of Edinburgh · DCC/FCUP, University of Porto

cs.LG

Submitted: 2026-05-21

Updated: 2026-10-13

Code: https://github.com/PedrV/gnn-uq-inspector

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 68/100

The gist: The gist The results demonstrate that deep ensembles largely do not capture uncertainty in graph neural networks, as they suffer from epistemic collapse where independently trained models converge to

Key concepts

Deep Ensembles (DEs): DEs involve training multiple GNN models from different random starting points and then averaging their predictions. The idea is that disagreement between these models reveals uncertainty. However, the paper shows this mechanism breaks down in GNNs because the models converge too closely together.
Epistemic Collapse: This occurs when independently trained GNNs consistently produce nearly identical predictions. Because deep ensembles rely on different models disagreeing to capture model uncertainty, this lack of diversity means the ensemble loses its primary advantage in estimating knowledge-based uncertainty.
Aleatoric vs. Epistemic Uncertainty: Aleatoric uncertainty reflects inherent noise in the data itself, which is irreducible. Epistemic uncertainty reflects a model's lack of knowledge about the data due to insufficient training or diversity. The study found that GNN ensembles are overwhelmingly dominated by aleatoric uncertainty, with very little useful epistemic component.

Terminology

Benchmarking — 1× in this paper · explained in 4 episodes
Epistemic Uncertainty — 1× in this paper · explained in 11 episodes
Uncertainty Quantification — 1× in this paper · explained in 9 episodes

Summary

The gist The results demonstrate that deep ensembles largely do not capture uncertainty in graph neural networks, as they suffer from epistemic collapse where independently trained models converge to overly similar predictions, meaning their marginal gains stem primarily from stabilizing optimization noise rather than yielding meaningfully better uncertainty estimates <ref:2605.22593#pg2>.

Background

Graph neural networks (GNNs) are defined by a message-passing mechanism where node representations are built by iteratively exchanging and processing information within local neighborhoods, as shown in equation (1) <ref:2605.22593#pg5>. Uncertainty quantification is conceptually divided into aleatoric uncertainty, which reflects irreducible noise in the data, and epistemic uncertainty, which reflects the model’s lack of knowledge <ref:2605.22593#pg4>. Deep ensembles (DEs) provide a straightforward approach to this by aggregating predictions from M copies of the same architecture trained from different random weight initializations <ref:2605.22593#pg5>. For regression, the total predictive uncertainty decomposes into aleatoric variance and the variance of the means, elegantly written in equation (2) <ref:2605.22593#pg4>.

Motivating Example: The PEMS Benchmark

The PEMS road network benchmark showed a striking failure of deep ensembles on GNNs, producing highly overconfident and uninformative uncertainty estimates compared to the geometrically grounded Graph GP <ref:2605.22593#pg5>. While DEs improved point estimation (RMSE) over the Trivial baseline, their uncertainty quantification was catastrophically poor, yielding a significantly worse NLL score <ref:2605.22593#pg5>. The Bayesian GCN utilized the exact same architecture yet achieved an NLL that comfortably beat the Trivial baseline <ref:2605.22593#pg5>.

Benchmarking Ensembles of Graph Neural Networks

The study systematically benchmarked standard deep ensembles of GNNs across seven diverse datasets spanning node classification, node regression, and graph regression <ref:2605.22593#pg5>. The evaluation included datasets such as Cora (CORA), Citeseer (CTSR), Tolokers2 (TLK2), Artnetviews (ARTNV), Chameleon (CHAM), and QM9-5% <ref:2605.22593#pg5>. In contrast to classic computer vision benchmarks, graph ensembles yielded improvements ranging from merely 0.1% to 1% in NLL for classification tasks <ref:2605.22593#pg5>.

Deconstructing the NLL Gains

The analysis revealed that the modest gains in NLL are driven almost entirely by variance reduction and predictive accuracy rather than better uncertainty estimates <ref:2605.22593#pg5>. For regression, all models utilizing the DE mean (DE, DE-R, DE-A) achieved nearly identical NLL improvements over the single-model baseline <ref:2605.22593#pg5>. For classification, Jensen’s gap between individual and ensemble NLL was remarkably small across all tested graph classification datasets, implying that the ensemble provides no fundamental enhancement to the predictive capabilities of the model <ref:2605.22593#pg5>.

Diversity and Epistemic Collapse

The core mechanism deep ensembles rely on—disagreement among constituent models—is neutralized because individual GNNs fail to meaningfully disagree, leading to epistemic collapse <ref:2605.22593#pg5>. Figure 4 demonstrated that the predictive uncertainty of GNN ensembles is overwhelmingly aleatoric-dominant, with the epistemic entropy close to zero in classification tasks <ref:2605.22593#pg5>. This systemic failure confirms that independent random initializations lead to models converging to very similar predictions, leaving the epistemic component too weak to be useful <ref:2605.22593#pg5>.

Discussion: GNNs and Convexity

The observed epistemic collapse suggests that GNNs operate in a regime inducing convexity in the functional space, where independently trained models find different sets of weights that ultimately implement the same function <ref:2605.22593#pg5>. This contrasts with classic deep learning settings where ensembles capture critical epistemic uncertainty <ref:2605.22593#pg5>. The weight space analysis showed that the model soup generally underperformed a single model, confirming that the weight space itself remains highly non-convex <ref:2605.22593#pg5>. Therefore, GNNs exhibit functional collapse rather than weight-space convexity <ref:2605.22593#pg5>.

Conclusion

Deep ensembles of GNNs typically outperform the Trivial baseline but consistently fail to deliver meaningful improvements in uncertainty quantification because they lack functional diversity and merely stabilize optimization noise <ref:2605.22593#pg5>. Consequently, a single “lucky” model can often match or even exceed the NLL performance of the full ensemble, suggesting that high-quality uncertainty quantification in graph learning requires moving beyond naive ensembling of message-passing architectures <ref:2605.22593#pg5>.

How it works

Deep ensembles (DEs) are constructed by repeatedly training the same architecture from different initializations <ref:2605.22593#pg5>.
For regression, total predictive uncertainty decomposes into aleatoric variance and the variance of the means, as shown in equation (2) <ref:2605.22593#pg4>.
For classification, total uncertainty is measured by predictive entropy H, which decomposes into expected data uncertainty and mutual information <ref:2605.22593#pg4>.
Epistemic collapse occurs when independently trained GNNs converge to highly similar functions, neutralizing the mechanism deep ensembles rely on to capture epistemic uncertainty <ref:2605.22593#pg5>.
The minor NLL improvements are primarily attributable to moderately improved prediction accuracy rather than better uncertainty calibration <ref:2605.22593#pg5>.
The lack of functional diversity means the ensemble merely acts as a variance reduction technique over optimization noise, stabilizing performance rather than fundamentally improving it <ref:2605.22593#pg5>.
The hypothesis suggests that message-passing architecture induces a form of convexity in the functional space, restricting functional diversity even in shallow networks <ref:2605.22593#pg5>.
The weight space remains non-convex, as the model soup fails to match even the single-model baseline, arguing against weight-space convexity <ref:2605.22593#pg5>.
For OOD settings, deep ensembles provide little gains over a single model for uncertainty quantification, with relative improvements being less than 1% for classification and less than 3% for regression <ref:2605.22593#pg5>.
The paper concludes that high-quality uncertainty quantification in graph learning requires moving beyond naive ensembling of message-passing architectures <ref:2605.22593#pg5>.

References

[1] T. Abe, E. K. Buchanan, G. Pleiss, R. Zemel, and J. P. Cunningham <ref:2605.22593#pg5>.

[7] V. Borovitskiy PeMS Regression: A Benchmark Suite for Node Regression with Uncertainty <ref:2605.22593#pg5>.

[11] S. Depeweg, J.-M. Hernandez-Lobato, F. Doshi-Velez, and S. Udluft Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning <ref:2605.22593#pg4>.

[47] T. Vinchurkar Kitchin Uncertainty quantification in graph neural networks with shallow ensembles <ref:2605.22593#pg5>.

[16] S. Fort H. Hu, and B. Lakshminarayanan Deep Ensembles: A Loss Landscape Perspective <ref:2605.22593#pg5>.

[18] R. Garnett Bayesian optimization <ref:2605.22593#pg5>.

[38] Y. Ovadia E Fertig J Ren Z Nado D Sculley S Nowozin J Dillon B Lakshminarayanan and J Snoek Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift <ref:2605.22593#pg5>.

[48] F. Wang Y Liu K Liu Y Wang S Medya and P Yu Uncertainty in Graph Neural Networks: A Survey <ref:2605.22593#pg5>.

[17] V. Fung J Zhang E Juarez and B G Sumpter Benchmarking graph neural networks for materials chemistry <ref:2605.22593#pg5>.

[19] J. Gilmer S Schoenholz P Riley O Vinyals and G E Dahl Neural message passing for Quantum chemistry <ref:2605.22593#pg5>.

[31] C. Louizos M Welling Multiplicative normalizing flows for variational Bayesian neural networks <ref:2605.

Improvements for AI systems

A single best-performing model can be selected to drastically reduce compute and memory overhead during inference, as because individual GNNs converge to highly similar functions, they neutralize the exact mechanism deep ensembles rely on to improve uncertainty quantification.
The system can be trained using a single GNN initialization instead of an ensemble, achieving performance comparable to those of the full ensemble while avoiding the computational cost.
For tasks where epistemic uncertainty is not critical, models can rely on a single GNN since the ensemble provides little uncertainty improvement over a much cheaper single-model baseline, even though ensemble averaging yields moderate gains in prediction accuracy.

Abstract

While deep ensembles are widely considered to be the default method for uncertainty quantification in deep learning, their effectiveness for graph-structured data is often simply assumed based on successes in domains like computer vision. We investigate standard deep ensembles specifically for message-passing graph neural networks. Benchmarking across eight datasets representing varied tasks and complexities, we reveal that ensembles provide only marginal improvement over a single model. These marginal gains stem primarily from averaging out optimization noise in point predictions rather than from meaningfully better uncertainty estimates. Through an aleatoric-epistemic decomposition, we identify epistemic collapse: independently trained networks consistently converge to overly similar predictions. Because disagreement is the fundamental mechanism through which ensembles capture epistemic uncertainty, this lack of diversity neutralizes their key advantage. Towards explaining this phenomenon, we derive a bound on epistemic variance in terms of the graph topology and Dirichlet energy budget, linking epistemic collapse to the smoothing bias of message passing. Our results suggest that deep ensemble success does not seamlessly transfer to graph machine learning.