Multi-fidelity Gaussian process regression for noisy outputs and non-nested experimental designs: decoupled versus joint likelihood optimization

arXiv:2511.20183 · stat.AP · Submitted 2025-11-25 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Multi-fidelity Gaussian process regression for noisy outputs and non-nested experimental designs".

Jane: Here's the gist of the paper. Multi-fidelity Gaussian process regression for noisy outputs and non-nested experimental designs: a comparison between the recursive and non-recursive formulations investigates a decoupled optimization strategy based on an expectation–maximization algorithm to compare recursive and non-recursive formulations in the challenging setting of noisy and non-nested highand low-fidelity data Model Formulation The paper introduces a recursive formulation of auto-regressive multi-fidelity Gaussian process regression in the challenging setting of noisy and non-nested highand low-fidelity data The proposed strategy involves a decoupled optimization based on the expectation–maximization algorithm, which exploits the structure of the recursive model This approach derives closed-form update formulas when the scaling factor is modeled as a parametric linear predictor The LF parameters are optimized by MLE as it is done classically in GP regression, and the HF parameters are optimized with the EM algorithm Optimization Strategies Comparison The study compares the proposed recursive strategy with the fully coupled likelihood maximization of the classical non-recursive formulation introduced by Kennedy and O’Hagan The results demonstrate that the proposed recursive strategy significantly reduces training time, especially when large low-fidelity datasets are available, while maintaining competitive predictive accuracy and uncertainty estimation The fully coupled likelihood optimization has a time complexity of O((NL+NH)3), which is higher than O(N3L+N3H) for the decoupled EM-based optimization Performance Metrics and Applications Several performance metrics are presented to evaluate the surrogate model when the target function y is known, including the predictivity coefficient Q2, which assesses the performance of the mean prediction The paper shows that for increasing complexity, such as an analytical 4D test function and a real-world sea surface temperature dataset, both MFGP formulations yield almost identical results for every UQ metric However, the recursive model is notably faster than the non-recursive model for large low-fidelity datasets The decoupled strategy achieves similar predictions compared to the non-recursive model but with faster training times for important amounts of LF data The MFGP models yield the smallest error values when only few HF data points are available, suggesting it is the most useful model in realistic situations when the computational budget is limited The research suggests that a sequential design of experiments strategy could improve performance when the budget allows it Acknowledgments This research was supported by the CEA (French Alternative Energies and Atomic Energy Commission) and the SEISM Institute The code and data needed to reproduce the results are available on demand and will be available on Github in the future References

1: Carl Edward Rasmussen and Christopher K.

Tom: I.

Jane: Williams.

Tom: Gaussian Processes for Machine Learning

2: Robert B.

Paper summary: Jane: So, wrapping up the discussion on "Multi-fidelity Gaussian process regression for noisy outputs and non-nested experimental designs: decoupled versus joint likelihood optimization". What does this paper really leave us with?

Tom: It leaves us with a clear choice in how we approach these complex modeling tasks. The authors compared the recursive and non-recursive formulations, and they showed that the decoupled strategy is faster for important amounts of low fidelity data while keeping accuracy competitive.

Lu: Essentially, this means you can use a sequential design of experiments strategy to improve performance if your computational budget allows it, but the immediate benefit is gaining speed when you have large datasets from lower fidelity simulations.

Meng: For us shipping models to production, that means we can deploy faster training cycles and still get reliable uncertainty estimates without running into the prohibitive time costs of those joint likelihood methods.

Jane: And they specifically highlighted that when only a few high fidelity data points are available, this recursive model tends to give the smallest error values, which is crucial for realistic scenarios where your budget is limited.

Tom: So if you're dealing with noisy outputs and non-nested experimental designs, this paper suggests leaning towards the decoupled optimization strategy because it balances speed and predictive quality very well.

Lu: I think what’s important is that they are providing a framework where the expectation–maximization algorithm can be used for hyperparameter optimization when direct closed forms aren't available, which is a helpful extension of earlier work.

Jane: It's about making the complex machinery of multi-fidelity Gaussian process regression more accessible by showing how to break it down into steps that are easier to manage computationally.

Tom: It’s a comparison between two major optimization philosophies, and the evidence suggests this decoupled path is much more practical for getting results in a reasonable timeframe.

Conclusion: Tom: So we're wrapping up on this paper about multi-fidelity Gaussian process regression, looking at how to handle messy data and experimental designs that don't fit together well.

Jane: The title itself is pretty long, isn't it? "Multi-fidelity Gaussian process regression for noisy outputs and non-nested experimental designs: decoupled versus joint likelihood optimization." What does that actually mean in plain English?

Tom: It means they’re comparing two ways to run this modeling. one way is recursive, and the other is non-recursive, and they’re seeing which one gives you a better result when your data is noisy and you're mixing high fidelity stuff with low fidelity stuff that isn't neatly stacked.

Lu: The core idea they’re pushing here is decoupling the optimization. instead of trying to solve the whole thing at once, they use an expectation–maximization algorithm to do things in separate steps, which lets them find closed-form update formulas for some parts of the model.

Meng: From a deployment standpoint, that decoupling is huge because it means faster training time. if you have a ton of low fidelity data available, this approach cuts down on the processing load compared to what those joint likelihood methods require.

Jane: So if I'm just listening and I don't read the math, what’s the big picture here for me? How does this change how we think about building these predictive models?

Tom: It changes the trade-off. you get almost identical predictions compared to the older non-recursive method, but you train much faster when you have a lot of that cheap low fidelity data. it’s like getting a good result without waiting forever for the whole system to settle down.

Lu: And they show that when you only have a small amount of high fidelity data, this recursive model actually gives the smallest error values on those high fidelity points, which is pretty telling about its stability in limited settings.

Jane: That sounds like a practical win for researchers who can't afford massive computational runs. So what’s the real implication for the world outside of just faster training?

Tom: It suggests that if you are working with expensive simulations or experiments, using a sequential design strategy might actually be better than trying to do everything at once and coupled, if your budget is tight.

Lu: It points toward a more efficient way to use multi-fidelity data where you can strategically choose what data to gather next based on what the model needs most.

Jane: So basically, this paper gives us a roadmap for making these complex multi-fidelity models usable in real science with realistic time and data constraints. Next up, we’ll look at the specific performance numbers they used to prove this difference between the two optimization methods.

Nils Baillie, Baptiste Kerleguer, Cyril Feau, Josselin Garnier

Université Paris-Saclay · CEA · Service d’Etudes Mécaniques et Thermiques · CMAP · CNRS · Ecole Polytechnique · Institut Polytechnique de Paris

stat.AP

Submitted: 2025-11-25

Updated: 2026-11-06

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 79/100

The gist: Multi-fidelity Gaussian process regression for noisy outputs and non-nested experimental designs: a comparison between the recursive and non-recursive formulations investigates a decoupled…

Key concepts

Recursive Formulation
This is an auto-regressive way of modeling multi-fidelity Gaussian process regression. It exploits the structure of the model to derive closed-form update formulas, allowing for a decoupled optimization strategy where high-fidelity (HF) and low-fidelity (LF) parameters are optimized separately using different algorithms.
Non-recursive Formulation
This is the classical approach where all parameters are optimized simultaneously by maximizing the joint likelihood of all data points. This method has a high computational complexity, making it slower to train, especially with large datasets.
Expectation-Maximization (EM) Algorithm
An optimization technique used here to solve the decoupled problem. It iteratively estimates missing or latent parameters by first calculating an 'expectation' over the data and then maximizing that expectation. This allows for a more manageable training process compared to joint maximization.
Decoupled Optimization Strategy
This refers to optimizing the high-fidelity (HF) and low-fidelity (LF) parameters independently rather than jointly. The paper shows this strategy leads to faster training times, particularly when large amounts of low-fidelity data are available, without sacrificing predictive performance.

Terminology

Summary

Multi-fidelity Gaussian process regression for noisy outputs and non-nested experimental designs: a comparison between the recursive and non-recursive formulations investigates a decoupled optimization strategy based on an expectation–maximization algorithm to compare recursive and non-recursive formulations in the challenging setting of noisy and non-nested highand low-fidelity data

Model Formulation

The paper introduces a recursive formulation of auto-regressive multi-fidelity Gaussian process regression in the challenging setting of noisy and non-nested highand low-fidelity data The proposed strategy involves a decoupled optimization based on the expectation–maximization algorithm, which exploits the structure of the recursive model This approach derives closed-form update formulas when the scaling factor is modeled as a parametric linear predictor The LF parameters are optimized by MLE as it is done classically in GP regression, and the HF parameters are optimized with the EM algorithm

Optimization Strategies Comparison

The study compares the proposed recursive strategy with the fully coupled likelihood maximization of the classical non-recursive formulation introduced by Kennedy and O’Hagan The results demonstrate that the proposed recursive strategy significantly reduces training time, especially when large low-fidelity datasets are available, while maintaining competitive predictive accuracy and uncertainty estimation The fully coupled likelihood optimization has a time complexity of O((NL+NH)3), which is higher than O(N3L+N3H) for the decoupled EM-based optimization

Performance Metrics and Applications

Several performance metrics are presented to evaluate the surrogate model when the target function y is known, including the predictivity coefficient Q2, which assesses the performance of the mean prediction The paper shows that for increasing complexity, such as an analytical 4D test function and a real-world sea surface temperature dataset, both MFGP formulations yield almost identical results for every UQ metric However, the recursive model is notably faster than the non-recursive model for large low-fidelity datasets

Conclusion and Implications

The decoupled strategy achieves similar predictions compared to the non-recursive model but with faster training times for important amounts of LF data The MFGP models yield the smallest error values when only few HF data points are available, suggesting it is the most useful model in realistic situations when the computational budget is limited The research suggests that a sequential design of experiments strategy could improve performance when the budget allows it

Acknowledgments

This research was supported by the CEA (French Alternative Energies and Atomic Energy Commission) and the SEISM Institute The code and data needed to reproduce the results are available on demand and will be available on Github in the future

References

[1] Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning

[2] Robert B. Gramacy. Surrogates: Gaussian Process Modeling, Design and Optimization for the Applied Sciences

[3] B. Efron. Bootstrap Methods: Another Look at the Jackknife

[4] Anastasios N Angelopoulos, Stephen Bates, et al. Conformal prediction: A gentle introduction

[5] Marc Kennedy and Anthony O’Hagan. Predicting the output from a complex computer code when fast approximations are available

[6] Alexander I.J Forrester, Andr´as S´obester, and Andy J Keane. p. 1

Where the paper says it

This construction can be generalized for more than two fidelity levels. Forrester et al [6] give more details regarding the maximum likelihood estimation (MLE) of the hyperparameters.

Multi-fidelity optimization via surrogate modelling

[7] Loic Le Gratiet. Bayesian analysis of hierarchical multifidelity codes

[8] Loic Le Gratiet and Josselin Garnier. Recursive co-kriging model for design of computer experiments with multiple levels of fidelity

[9] Loic Le Gratiet and Claire Cannamela. Cokriging-based sequential design strategies using fast cross-validation techniques for multi-fidelity computer codes

[10] Hossein Mohammadi and Peter Challenor. Sequential adaptive design for emulating costly computer codes

[11] Pulong Ma. Objective Bayesian analysis of a cokriging model for hierarchical multifidelity codes

[12] Rui Paulo. Default priors for Gaussian processes

[13] Loic Le Gratiet. Multi-fidelity Gaussian process regression for computer experiments

[14] A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EM algorithm

[15] Federico Zertuche. Assessment of uncertainty in computer experiments when working with multifidelity simulators

[16] Christophette Blanchet-Scalliet, Bruno Demory, Thierry Gonon, and C´eline Helbert. Gaussian process regression on nested spaces

[17] H. Babaee, C. Bastidas, M. DeFilippo, C. Chryssostomidis, and G. E. Karniadakis. A multifidelity framework and uncertainty quantification for sea surface temperature in the Massachusetts and Cape Cod bays

[18] Giulio Gori, Olivier Le Maˆıtre, and Pietro M. Congedo. Debiased multifidelity approach to surrogate modeling in aerospace applications

[19] P. Perdikaris, M. Raissi, A. Damianou, N. D. Lawrence, and G. E. Karniadakis. Nonlinear information fusion algorithms for data-efficient multi-fidelity modelling

[20] Katerina Giannoukou, Stefano Marelli, and Bruno Sudret. Uncertainty-aware multifidelity surrogate modeling with noisy data

[21] Dennis D Cox, Jeong-Soo Park, and Clifford E Singer. A statistical method for tuning a computer code to a data base

[22] Shifeng Xiong, Peter Z. G. Qian, and C. F. Jeff Wu. Sequential design and analysis of high-accuracy and low-accuracy computer codes

[23] Xuhui Meng and George Em Karniadakis. A composite neural network that learns from multi-fidelity data: Application to function approximation and inverse PDE problems

[24] Kurt Cutajar, Mark Pullin, Andreas Damianou, Neil Lawrence, and Javier Gonz´alez. Deep Gaussian processes for multi-fidelity modeling

[25] Yiming Yang, Deyu Ming, and Serge Guillas. Distribution of deep Gaussian process gradients and sequential design for simulators with sharp variations

[26] Baptiste Kerleguer, Claire Cannamela, and Josselin Garnier. A Bayesian neural network approach to multi-fidelity surrogate modeling

[27] Mengyang Gu, Xiaojing Wang, and James Berger. Robust Gaussian stochastic process emulation

[28] Fran¸cois Bachoc. Cross validation and maximum likelihood estimations of hyperparameters of Gaussian processes with model misspecification

[29] Ciyou Zhu, Richard H. Byrd, Peihuang Lu, and Jorge Nocedal. Algorithm 778: LBFGS-B: Fortran subroutines for large-scale bound-constrained optimization

[30] Lee W. Schruben. A coverage function for interval estimators of simulation response

[31] Charles Demay, Bertrand Iooss, Loic Le Gratiet, and Amandine Marrel. Model selection based on validation criteria for Gaussian process regression: An application with highlights on the predictive variance

[32] Amandine Marrel and Bertrand Iooss. Probabilistic surrogate modeling by Gaussian process: A review on recent insights in estimation and validation

[33] M. D. McKay, R. J. Beckman, and W. J. Conover. Comparison of three methods for selecting values of input variables in the analysis of output from a computer code

[34] Brian Andrews, Wayne Baldwin, Daniel Sampson, and William Schwab. Continuous bathymetry and elevation models of the Massachusetts coastal zone and continental shelf

[35] Roger A. Horn and Charles R. Johnson.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Decoupled optimization strategy for multi-fidelity Gaussian process regression enables significant reduction in training time, especially when large low-fidelity datasets are available. This speedup is achieved by optimizing LF parameters via MLE and HF parameters via the expectation–maximization algorithm separately, rather than fully coupled likelihood maximization.

  2. The proposed recursive formulation of the auto-regressive AR(1) model allows for predictions to be made for all fidelity levels, improving overall interpretability and reducing computation cost in terms of matrix inversions compared to fully coupled formulations.

  3. The system can provide competitive predictive accuracy and uncertainty estimation by exploiting the structure of the recursive model, demonstrating performance comparable to non-recursive methods across various benchmark functions and real-world datasets like sea surface temperature measures.

  4. For scenarios with limited high-fidelity data, the multi-fidelity GP models yield smaller error values compared to single-fidelity GP models when the computational budget is limited, suggesting they are more useful in realistic situations.

  5. The model can be adapted for sequential design of experiments strategies that are not limited to nested experimental design input sets, allowing for improved performance when the computational budget permits.

Abstract

This paper investigates a recursive formulation of auto-regressive multi-fidelity Gaussian process regression in the challenging setting of noisy and non-nested high- and low-fidelity data. We propose a decoupled optimization strategy based on the expectation-maximization algorithm, which exploits the structure of the recursive model. In particular, we derive closed-form update formulas when the scaling factor is modeled as a parametric linear predictor. This approach is compared with the fully coupled likelihood maximization of the classical, non-recursive formulation. A series of benchmark experiments, covering applications of increasing complexity, highlights the performance of both approaches. The results demonstrate that the proposed recursive strategy substantially reduces training time when large low-fidelity datasets are available, while maintaining competitive predictive accuracy and uncertainty estimation.

Related papers