Review of Multi-fidelity Models

1. Motivation

In science and engineering, high-fidelity models (HFMs) commonly refer to complex high-dimensional systems that can make highly accurate predictions. However, the cost of developing and utilizing such models is often prohibitively high, which limits their practicality for many applications. Alternatively, low-fidelity models (LFMs), their simpler and cheaper counterparts, offer a more affordable alternative. LFMs are less accurate due to dimensionality reduction, linearization, use of simpler physics models, coarser domains, or partially converged results, as depicted schematically in Figure 1. If a model is considered to have low or high fidelity can only be determined relative to another. A fully three-dimensional simulation can be considered expensive compared to an analytical function evaluation but cheap compared to actual experiments. In the early 2000s, multi-fidelity models (MFMs), which combine multiple fidelities in a single model, gained significant attention due to their potential to achieve the desired level of accuracy at a lower cost.

MFMs typically involve the construction of surrogate models (SMs) in order to reduce the computational burden associated with a large number of expensive simulations required for tasks such as optimization (e.g., [58,183]) and uncertainty quantification (UQ) (e.g., [136]). SMs are approximations created to model the behavior of the underlying system. The construction of SM architectures trained using data from different levels of fidelity is referred to as multi-fidelity surrogate modeling (MFSM), which is the primary focus of this review. SMs can also be constructed to reduce the computational cost of individual models. While it is assumed that the reader is familiar with the concept of surrogate modeling, Appendix B provides a brief overview of the most commonly used surrogate models in the context of MFMs. It should be noted that the choice of the most appropriate SM will depend heavily on the specific characteristics of the problem at hand.

In the framework of computational simulations, the process of fitting an SM to high-fidelity data may be impeded by the computational resources required to obtain just enough data for an accurate approximation. In this scenario, a possible approach to circumvent this issue is to rely on lower-cost, lower-fidelity simulations previously employed to analyze similar problems when computers were less powerful. Alternatively, SMs may be constructed to approximate LFMs, although they may be sufficiently inexpensive to warrant direct usage in some instances. A case in point is illustrated by Nguyen et al., 2013 [138].

The construction of MFSMs by integrating various fidelity levels is not mandatory for MFMs, as exemplified by Choi et al., 2008 [36]. In their paper, they employed different fidelity types proficiently through adaptive sampling without constructing an MFSM. This alternative MFM technique is known as MFHM. Figure 2 illustrates the options for constructing an MFM. The MFM, where a surrogate model is constructed to combine the fidelities, is termed an MFSM. In contrast, if no surrogate model is built, and the fidelities are combined hierarchically, the resulting technique is known as MFHM. In both techniques, HFMs and LFMs or their surrogate models are utilized.

Most of the articles reviewed in this paper have limited the construction of MFMs to two fidelities. However, it is worth noting that MFMs can be developed using more than two fidelities, as demonstrated in several studies, including Huang et al., 2006 [76], Forrester et al., 2007 [58], Qian et al., 2008 [147], Le Gratiet, 2013 [109] and Goh et al., 2013 [66].

Another survey on multi-fidelity modeling is the work of Peherstorfer et al., 2016a [145]. They focus primarily on methodologies for using MFMs in outer-loop applications, such as optimization, uncertainty propagation, and inference. In contrast, this survey focuses on the penalty and savings associated with combining multiple physics-based models, particularly when these models are fused to create MFSMs. It is worth noting that in this survey, the term multi-fidelity is used only for methods combining at least two physical models. Therefore, articles that consider a physical model and its SM as different fidelities are not seen as MFMs in this review. Methods such as multi-level methods, which replace the HFM with an LFM (with occasional accuracy checks), are not considered as MFMs in this review either. These model reduction methods speed up processes such as optimization at the cost of reduced accuracy. In contrast, MFMs aim to balance accuracy and affordability.

The development of MFM techniques offers the potential to reduce computational costs while maintaining accuracy in various scientific fields. However, it has been noted that the process of developing MFMs requires a significant investment of time and effort by the user and that the point at which the payoff justifies it remains unclear. Hence, one of the goals of this study aims to quantify the efficacy of MFM techniques in maintaining modeling accuracy while reducing its computational cost. This study also aims to comprehensively explore the use of MFMs in scientific research. MFMs will first be introduced, providing information on their field of application, simulation models, year of publication, and fidelity types gathered from the analysis of over 150 papers. The different fidelities utilized in MFMs across scientific fields are categorized. The available techniques for combining fidelities through surrogate models in MFMs are reviewed and evaluated. Sampling strategies used in constructing MFMs are also investigated. Successful combinations of fidelities that significantly improve accuracy and cost through MFMs are identified. Finally, a standard and effective reporting method is suggested to provide readers with a clear understanding of the benefits of MFMs. Papers that have reported such information to facilitate a better understanding of the potential benefits of MFMs are discussed. Overall, this study seeks to contribute to a deeper understanding of the advantages and shortcomings of MFMs in scientific research.

2. Overview

In this study, a comprehensive analysis of various MFM implementations has been carried out, and a classification scheme based on six key attributes has been constructed. The classification categories include application, fidelity type, method for constructing the MFSM (i.e., deterministic method and non-deterministic method), year of publication, paper field, and SM used. Figure 3 presents an overview of the categories in the reviewed literature.

The reviewed literature focuses on using MFMs in three main applications: optimization (including inference and inverse problems), uncertainty quantification or UQ, and optimization under uncertainty. In cases where papers describe a generic procedure without a specific application, such as analytic functions, they are classified as none. The nature of the fidelities in the model, referred to as the types of fidelity, can be classified into four categories. The first category is physics, which accounts for differences in the assumptions and considerations in the physical model. The second category is numerical solution accuracy, which accounts for different levels of discretization in space or time and partially converged solutions. The third category is numerical models, which refers to instances where the same physical model and assumptions are used, but the method of computing the results varies. Finally, the fourth category is sim+exp, which refers to the combination of simulations, usually as LFM, and experiments, usually as HFM, in constructing an MFM.

The criterion used to fit the data in the MFSM construction is referred to as the method, which can be classified into two categories: deterministic or DM and non-deterministic or NDM. Papers that use MFHMs where no MFSM is constructed are classified as none. The paper’s year of publication is referred to as the year published. The area of application of the problem addressed in the paper is referred to as the field. The most common fields observed in the reviewed literature were fluid mechanics and solid mechanics. Finally, the type of SM used to construct the MFSM is referred to as the surrogate model. Papers that use MFHMs without constructing an MFSM are classified as none.

Figure 3a shows that the most common application found for MFMs is optimization, followed by UQ and optimization under uncertainty. These applications are introduced as outer-loop applications by Peherstorfer et al., 2016a [145] and extensively discussed in sections 5, 6, and 7 of their work. The fact that optimization is the main application is understandable because UQ and optimization under uncertainty are relatively new subjects. However, it is expected that more publications will appear in these applications. Figure 3b shows the distribution of papers by the type of fidelities used; these are discussed in Section 3.

Figure 3c shows that the proportion of papers that use DMs or NDMs to construct an MFSM is similar. The category none refers to papers that present MFMs without constructing an MFSM, called MFHMs. This approach is frequently observed in fields such as optimization, where LFMs are initially utilized to narrow down the domain of interest, followed by the implementation of HFMs to achieve more accurate information about the optimum location, see Rodriguez et al., 2001 [161] and Peherstorfer et al., 2016b [144]. The most common methods used to combine fidelities in the MFM context are presented in Section 4. Section 4.3 presents a further study of the time distribution of DMs and NDMs.

Figure 3d shows that the use of MFMs seems to be expanding since its beginning in the late ’80s. Finally, the distribution of MFMs applications across various fields is presented in Figure 3e. It indicates that most of the reviewed papers implement MFMs in fluid mechanics and solid mechanics. However, other fields, such as electronics, aeroelasticity, and thermodynamics, also feature in the reviewed literature. Furthermore, some papers do not have any specific application, but rather use mathematical functions like Hartman or Rosenbrock to test the methods.

Figure 3f displays the distribution of papers by surrogate type used for constructing MFSMs. The study found that basis function regression and Kriging surrogates are the two most commonly used types for constructing an MFSM. The category others includes artificial neural networks, moving least squares, support vector machines, radial basis interpolation, and proper orthogonal decomposition, each with a usage of less than 1%. In addition, the research reveals that some MFMs do not require the construction of an MFSM, referred to as MFHM, which is included in the category none. For instance, Choi et al., 2008 [36] proposes a hierarchical MFM approach for optimization, utilizing HFMs selectively to rectify inadequacies of LFMs. As mentioned, this technique does not involve constructing MFSMs or explicitly integrating fidelities. Other examples are Kalivarapu and Winer, 2008 [84], where an MFM is used for interactive modeling of advective and diffusive contaminant transport with no MFSM construction. Giunta et al., 1995 [64], and Zahir et al., 2013 [199] are other examples of such cases.

Section 5 provides a detailed analysis of the computational cost and accuracy associated with using MFMs. Additionally, it offers guidelines for authors on how to present cost savings and accuracy improvements. Sampling methods used for MFSM construction are discussed in Appendix A, while Appendix B introduces the most commonly used surrogate methods in MFSM research.

3. Types of Fidelity

The current study aimed to review the literature and identify different types of fidelities commonly associated with three principal categories: model, accuracy and source (Figure 4). Model involves simplifying the mathematical representation of the physical phenomenon, typically by simplifying the differential equations being solved or the numerical model. Accuracy refers to changes on the discretization of the model, such as using smaller grid elements or shorter time steps for HFMs. Source is associated with the incorporation of experimental results in addition to simulations, which are regarded as having the highest level of fidelity.

Authors may explicitly state the superiority of one fidelity over another, as in the scenario involving a refined grid as opposed to a coarser grid if the same model is used. Nevertheless, in cases such as comparing a one-dimensional model with a refined grid against a three-dimensional model with a coarser grid, the comparative superiority of fidelities may not be apparent. Peherstorfer et al., 2016a [145] classified LFMs in three categories: simplified models, data-fit models, and projection-based models. Their simplified models’ category includes the differences between fidelities discussed in this study. In contrast, data-fit models and projection-based models are included in this review as surrogate modeling techniques in Appendix B.

This survey identified two main fields where MFMs are used, fluid mechanics and solid mechanics. In fluid mechanics, the primary fidelity types were analytical expressions, empirical relations, numerical linear approximations, potential flow, numerical non-linear non-viscous approximations (Euler), numerical non-linear viscous approximations (RANS), coarse vs. refined analysis, and simulations vs. experiments. A detailed listing of the papers that employed these types of fidelities as LFMs and HFMs is provided in Table 1.

Other types of fidelities, that were not included in the table, were found in the literature. These included simplifying physics found in Castro et al., 2006 [29], where an earth penetrator problem is simplified by assuming a rigid penetrator and in Goldfeld et al., 2005 [67], where the physics are simplified by assuming constant instead of variable material properties; using different geometries in Forrester et al., 2010 [56], where the LFM is a RANS simulation with simplified geometry and the HFM is a RANS simulation with full geometry; and using different numbers of Monte Carlo samples in Keane, 2012 [88]. In fluid mechanics, additional categories were also found including dimensionality (e.g., 2D/3D), coarse vs. refined analysis, simulations vs. experiments, transient vs. steady, and semiconverged vs. converged solutions (Table 2).

Several types of fidelities were identified in the literature but not included in Table 2. For instance, Castro et al., 2006 [29] simplified a physics problem by assuming a rigid penetrator, while Goldfeld et al., 2005 [67] simplified the physics by assuming constant material properties instead of variable ones. Forrester et al., 2010 [56] used RANS simulations with simplified and complete geometries for the LFM and HFM, respectively. Additionally, Keane, 2012 [88] varied the number of Monte Carlo samples used. In the field of fluid mechanics, various additional fidelity categories were also found, such as dimensionality, level of analysis refinement, simulation versus experimentation, transient versus steady states, and semiconverged versus fully converged solutions. Table 2 presents a detailed summary of these categories.

Table 3 presents the common types of fidelities employed in the domain of solid mechanics. These types comprise of mesh density, material models, and temperature. However, the investigation revealed other fidelity types that were not included in Table 3. For example, Kim et al., 2007 [91] employed isothermal and non-isothermal analyses for LFM and HFM, respectively. In addition to these, several more categories of fidelities were discovered in solid mechanics, such as dimensionality (e.g., 2D/3D), coarse vs. refined, simulations vs. experiments, and boundary condition simplification (e.g., infinite plate vs. finite plate), and are presented in Table 4.

In addition, papers from other fields such as electronics and robotics, and papers based on analytical functions were reviewed. In electronics, the most common method utilized was coarse vs. refined analysis, although some papers, such as Absi and Mahadevan, 2016 [1], employed steady vs. transient models. In robotics, Winner et al., 2000 [192], determined the fidelities based on the complexity of the robot’s resources. Several academic papers have utilized mathematical functions to examine the effectiveness of different methods, including analytical function versus analytical approximations of the function. These studies include the works of Robinson et al., 2006 [157][159], Zimmermann and Han, 2010 [206], Ng et al., 2012 [136], Le Gratiet, 2013 [110], Raissi and Seshaiyer, 2013 [149], Raissi and Seshaiyer, 2014 [150], and Goh et al., 2013 [66]. Lastly, the category of methods for uncertainty analyses with no particular application, includes Burton and Hajela, 2003 [28], Eldred, 2009 [50], Perdikaris et al., 2015 [146], Peherstorfer et al., 2016b [144], and Chaudhuri and Willcox, 2016 [32]. In Burton and Hajela, 2003, Eldred, 2009, and Perdikaris et al., 2015, the types of fidelity pertained to less and more accurate uncertainty analysis. Peherstorfer et al., 2016b [144] utilized LF models to aid in the construction of the biasing distribution for importance sampling and a small number of HF samples to obtain an unbiased estimate. Chaudhuri and Willcox, 2016 [32], employed an iterative method that used low-fidelity surrogate models (LFSMs) to estimate coupling variables and adaptively sampled the HF system to enhance the SM, while retaining a comparable level of precision in uncertainty analysis as the fully coupled HF multidisciplinary system.

4. Methods for Combining Fidelities

This section discusses different methods for combining fidelities. Fidelity refers to the level of detail and accuracy of the model or simulation, where a higher fidelity model provides more accurate results but requires more computational resources. The goal of combining fidelities is to take advantage of the strengths of each model and create an accurate, computationally efficient model. The focus will be on two main categories of methods: MFSMs and MFHMs. MFSMs use an algebraic surrogate to correct LFMs using HFM predictions, while MFHMs use different fidelities based on some criterion (see Figure 2). This survey also examines the four principal correction techniques employed in MFSMs: multiplicative correction, additive correction, comprehensive correction, and space mapping. Furthermore, this article encompasses various illustrations and applications of these techniques.

4.1. Multi-Fidelity Surrogate Models vs. Multi-Fidelity Hierarchical Models

This article provides a comparative analysis between MFSMs and MFHMs based on the literature reviewed. Out of the total of 157 papers surveyed, 67% focused on MFSMs, while the remaining 33% utilized MFHMs (see Figure 5). The latter approach combines different fidelities using a criterion to optimize a process, instead of constructing a surrogate model architecture where information from both is considered. Examples of such techniques include Burton and Hajela, 2003 [28], Choi et al., 2005 [34], and Singh and Grandhi, 2010 [168]. For instance, Christen and Fox, 2005 [37] used predictions from the LFM for Markov Chain Monte Carlo sampling and only employed the HFM when the acceptance criterion was met, whereas Rethore et al., 2011 [155] utilized simpler/faster cost functions with coarse discretization and increased the resolution of the domain and complexity of models where needed. Other examples include the work by Narayan et al., 2014 [134] which employed a stochastic collocation approach to select data points for HFM construction based on LFMs, and Peherstorfer et al., 2016b [144] which used an importance sampling method based on an LFM to choose sampling points for HFSM construction.

In their 2016 survey, Peherstorfer et al. 2016 classified techniques for integrating multiple fidelities into three categories: adaptation, fusion, and filtering. The adaptation approach improves the LFM by incorporating information from the HFM during computation, with the SM being adjusted in each iteration. Fusion methods combine information from both LFMs and HFMs by evaluating them separately, as in the case of the co-Kriging method [55]. Filtering methods, on the other hand, employ the HFM after the LFM has been evaluated by a filter, using the HFM only when the LFM is inaccurate or when the candidate point satisfies some criterion based on the LFM assessment. Peherstorfer et al., 2016 classified MFHMs under adaptation and filtering management methods. This survey’s MFSMs category, on the other hand, was included in their fusion methods category. For more information, interested readers can refer to their work [145].

4.2. Multi-Fidelity Surrogate Models

This article focuses on MFSMs that incorporate fidelities into a single SM. MFSMs utilize an algebraic SM to improve the accuracy of LFMs with the aid of HFMs. Four main correction techniques are utilized in MFSMs, namely multiplicative correction, additive correction, comprehensive correction, and space mapping. In some cases, the parameters of the LFM differ from those of the HFM, requiring problem-specific conversion from LF to HF parameters. Examples are provided in Robinson et al., 2008 [158] and Koziel et al., 2009 [101]. In conventional mathematical techniques, convergence can be ensured if the LFM and HFM meet first-order consistency (i.e., their derivatives are equal). For example, studies by Alexandrov et al., 2000 [3] and Alexandrov et al., 2001 [5] fulfill first-order consistency. If second-order consistency is obtained, convergence rates can be increased, as shown by Eldred et al., 2004 [49]. However, meta-heuristic optimization, which is not mathematically rigorous and typically has lower accuracy and slower convergence rates, is favored in global optimization, as described in Kaveh and Talatahari, 2010 [87].

4.2.1. Additive and Multiplicative Corrections

One possible approach to enhance the accuracy of LFMs is to construct a SM of the discrepancy or ratio between HFMs and LFMs. These surrogate models are known as additive or multiplicative corrections, respectively.

The MFSM, which estimates the HFM using an additive correction, can be expressed as:

$${\widehat{y}}_{HF}=y_{LF}\left(\mathbf{x}\right)+\delta \left(\mathbf{x}\right),$$

(1)

where $y_{LF}$ denotes the LFM, which may be substituted with an LFSM if the cost is prohibitive. Additionally, $\delta \left(\mathbf{x}\right)$ refers to an additive correction or discrepancy function, which is an SM that accounts for the difference between the HFM and the LFM.

Alternatively, the MFSM using a multiplicative correction can be expressed as:

$${\widehat{y}}_{HF}=\rho \left(\mathbf{x}\right)·y_{LF}\left(\mathbf{x}\right),$$

(2)

where $\rho \left(\mathbf{x}\right)$ is the multiplicative correction, which is an SM constructed using the ratio between the HFM and LFM. Note that if the LFM is not cheap enough, it can be replaced by an LFSM.

MFSMs have been widely used in aerodynamic optimization problems to reduce computational costs. Various correction techniques have been proposed to construct MFSMs, including additive and multiplicative corrections. Alexandrov et al., 2001 utilized MFSM with multiplicative corrections for aerodynamic optimization problems. Balabanov et al., 1998 compared the performance of MFSMs constructed using additive and multiplicative corrections for a similar optimization problem. Forrester et al., 2006 used an additive correction based on fully converged results to correct partially converged results. The literature provides more references to MFSMs constructed using additive and multiplicative corrections, which are presented in Table 5 and Table 6 in Section 4.3.

To illustrate the concepts of additive and multiplicative corrections, a simple algebraic one-dimensional problem is presented. Consider two analytical functions, $y_{LF}$ and $y_{HF}$, given by Equations (3) and (4), respectively:

$$y_{LF}=5x+sin\left(11x\right)+12$$

(3)

and

$$y_{HF}=5x+sin(10x+2)+exp\left(x\right)+10$$

(4)

where $x\in [1,2]$, $y_{HF}$ is the HFM and $y_{LF}$ represents the LFM. In practice, LFM and HFM may be unavailable, and surrogate models are used instead. For this toy problem, is assumed the functions are accesible, but due to cost constraints, only three samples from the HFM can be afforded. Evenly distributed samples are chosen, specifically $[x_1,x_2,x_3]=[1.1,1.5,1.9]$, from two models, and the objective is to estimate the output of the HFM while minimizing expenses by utilizing LFM for prediction. To achieve this, additive and multiplicative corrections are introduced, as shown in Figure 6. The ratio $y_{HF}$/$y_{LF}$ or the difference $y_{HF}$-$y_{LF}$ at the sampling points $x_i$ are used to fit the multiplicative or additive corrections, respectively. In simpler terms, the sampling points $[y_{HF}\left(x_i\right)/y_{LF}\left(x_i\right)]$ for $i\in 1,2,3$ are used to fit the multiplicative correction $\rho \left(x\right)$, whereas the sampling points $[y_{HF}\left(x_i\right)-y_{LF}\left(x_i\right)]$ for $i\in 1,2,3$ are used to fit the additive correction $\delta$. The functions $\rho \left(x\right)$ and $\delta \left(x\right)$ (Eqs. (1) and (2), respectively) are obtained using basis function regression by fitting a linear combination of second-order polynomial basis functions where the polynomial coefficients are optimized to minimize the residuals ([52,202]). Figure 6 displays the original HFM and LFM within the interval [1,2], alongside the additive and multiplicative corrections and the sample locations. Although the performance appears to be similar in this particular case, it is important to note that for more complex systems, there can be significant variations in performance that may favor one approach over the other. Additional information regarding the methodology employed to generate Figure 6 may interest readers and can be found in the supplementary material.

To further illustrate the use of correction factors, consider a scenario in which only 20 HFM and 200 LFM data points can be performed to build an MFSM. Step 1 is to build an SM to approximate the discrepancy or ratio between the LFM and HFM based on the 20 shared data points. Two options are available for Step 2: (a) building an LFSM using the 200 LFM analyses and summing or multiplying it with the surrogate from the first step (as done in the analytic example from the previous section), or (b) using the SM from the first step to estimate the discrepancy or ratio at the 180 points where only LFM data is available. Then we calculate the predicted HFM results at these 180 data points by summing or multiplying to the corresponding 180 LFM results. We now have HFM data at 20 sampling points, and we have estimated HFM data at 180 points. We treat them equally and fit an SM to the 200 points using this surrogate as an MFSM.

It is worth noting that the choice between these two options can lead to noticeable differences when using regression rather than interpolation. Specifically, the first option may result in larger differences between the MFSM predictions and the HFM data at points where HFM data is available, whereas the second option may result in smaller differences, which can be further reduced by employing a weighted least squares approach with higher weights for the HFM data.

4.2.2. Comprehensive Corrections

A comprehensive correction involves the use of both additive and multiplicative corrections in the same MFSM. One widely used comprehensive correction is defined as follows:

$${\widehat{y}}_{HF}=\rho \left(\mathbf{x}\right)·y_{LF}\left(\mathbf{x}\right)+\delta \left(\mathbf{x}\right)$$

(5)

where $\rho \left(\mathbf{x}\right)$ represents the multiplicative correction surrogate and $\delta \left(\mathbf{x}\right)$ represents the additive correction surrogate. The literature review indicates that the most common approach is to set the multiplicative factor $\rho$ as a constant and to use an SM to approximate the additive correction, as has been demonstrated in previous studies such as Keane, 2012[88] and Perdikaris et al., 2015[146] and [202]. However, Qian et al., 2008 [147] proposed a comprehensive correction method with a non-constant $\rho \left(x\right)$.

The literature also provides a comprehensive correction technique known as the hybrid method, which was devised by Gano et al. in 2005 [59]. This method can be described as follows:

$${\widehat{y}}_{HF}=w\left(\mathbf{x}\right)·\rho \left(\mathbf{x}\right)·y_{LF}\left(\mathbf{x}\right)+(1-w\left(\mathbf{x}\right))[y_{LF}\left(\mathbf{x}\right)+\delta \left(\mathbf{x}\right)],$$

(6)

where $w\left(\mathbf{x}\right)$ is a weight function. A weight function is employed to assign greater importance or preference to specific data points while training an SM. This technique has been widely adopted in numerous research studies, including those conducted by Zheng et al., 2013 [203] and Fischer et al., 2017 [54].

The Space Mapping technique is a comprehensive correction method that offers an alternative approach to directly modifying the output of the LFM. Instead, it involves establishing a connection between the original input variables and a new set of input variables as shown in (7). This new set of variables, when evaluated in the LFM, approximates the response of the HFM.

This approach has been shown to be a viable alternative for achieving the desired corrections. Bandler et al. [15][14] introduced Space Mapping for the first time in early ’90s. The fundamental concept behind this technique is to generate an appropriate transformation of the vector of HF input parameters, ${\mathbf{x}}_{HF}$, to the vector of LF input parameters, ${\mathbf{x}}_{LF}$, such that:

$${\mathbf{x}}_{LF}=\mathbf{F}\left({\mathbf{x}}_{HF}\right).$$

(7)

The purpose of this iterative process is to allow the vectors ${\mathbf{x}}_{HF}$ and ${\mathbf{x}}_{LF}$ to have varying dimensions. Although it is not necessary, it is desirable for $\mathbf{F}$ to be invertible. The objective is to ensure that the response of the HFM, ${\mathbf{y}}_{HF}\left({\mathbf{x}}_{HF}\right)$, and the response of the LFM, ${\mathbf{y}}_{LF}\left({\mathbf{x}}_{LF}\right)$, fulfill the following condition:

$${\mathbf{y}}_{HF}\left({\mathbf{x}}_{HF}\right)-{\mathbf{y}}_{LF}\left({\mathbf{x}}_{LF}\right)\le ϵ$$

(8)

within some local region, where · is a suitable norm and $ϵ$ is a tolerance setting. Space Mapping has been only been found in publications using DMs.

The first review paper of the Space Mapping method was published ten years after its implementation [17], and the second one was published two decades later [152]. The concept of Space Mapping has been expanded to include other techniques, such as Aggressive Space Mapping [16], Trust Regions [10], Artificial Neural Networks [11], Implicit Space Mapping [18], Neural-Based Space Mapping [200][201], Inverse Problems[153], Corrected Space Mapping [158], and Tuning Space Mapping [101]. Further literature regarding the utilization of Space Mapping to construct MFSMs can be found in Table 5 of Section 4.3.

In Table 5 and Table 6 presented in Section 4.3, the interested reader can locate additional references which illustrate the construction of MFSMs utilizing comprehensive corrections.

4.3. Deterministic Methods vs. Non-Deterministic Methods

The present study categorizes MFSMs into DMs and NDMs, based on the model used to estimate the MFSM parameters. DMs assume a set of basis functions and determine their coefficients by minimizing the discrepancy between the data and the function [65,186], whereas NDMs assume either the function or the coefficients to be uncertain and utilize samples to reduce the uncertainty [111]. Although DMs do not require an uncertainty structure and can be applied to any surrogate, prior research has revealed that NDMs demonstrate superior accuracy over DMs [88,142]. The schematic diagram in Figure 7 shows the differences between DMs and NDMs, depending on the assumptions made about the unknown parameters.

In the context of outer-loop applications, specifically uncertainty quantification, NDMs need a method of statistical inference in order to treat parameter uncertainties, thus avoiding the costly standard Monte Carlo simulations [156] [71]. A prevalent non-deterministic methodology in statistical inference is the Bayesian framework, which employs Bayes’ theorem to derive the posterior distribution for the model’s unknown parameters. This distribution is conditioned on both the prior distribution for the parameters and the likelihood of the observed data. Gaussian processes have become a popular and widely used class of SMs due to their flexibility and convenience in incorporating prior knowledge about the data [90]. However, alternative methods to the Gaussian process have also been proposed, such as the approach presented by Koutsourelakis, 2009 [96], where non-Gaussian distributions are used to model the uncertainties. While Gaussian process-based models have been successful in low-dimensional contexts, they are not regarded as suitable for tackling high-dimensional problems or large datasets.

In contrast, for DMs, the parameters are estimated by minimizing the difference between the predictions of the LFSM (${\widehat{y}}_{LF}$) and the HFSM (${\widehat{y}}_{HF}$) at the high-fidelity data points. On the other hand, NDMs such as Bayesian discrepancy or co-Kriging estimate parameters that simplify the discrepancy function $\delta$ as much as possible, even if this leads to an increase in its magnitude. By simplifying $\delta$, the accuracy of the discrepancy surrogate can be improved beyond that achieved by simply minimizing the discrepancy [143]. This approach offers an alternative method for handling parameter uncertainties in outer-loop applications.

Calibration is a widely-used technique in engineering to enhance simulation predictions by adjusting physical parameters to attain better agreement with experiments. Numerous studies in the engineering community, such as Kosonen and Shemeikka, 1997 [95], Owen et al., 1998 [139], Lee et al., 2008 [117], McFarland et al., 2008 [126], Coppe et al., 2012 [40], and Yoo and Choi, 2013 [195], have employed calibration to improve simulation predictions. Although pure calibration is not considered an MFM, Kennedy and O’Hagan, 2001 [90] introduced a popular Bayesian calibration method that views calibration from a different perspective. Bayesian calibration involves treating calibration parameters in the same way as other non-physical hyper-parameters. It is worth noting that the calibrated physical parameters obtained through this approach may deviate from their true values. This technique, known as calibration along with comprehensive correction is considered an MFM. Several studies, including Qian et al., 2008 [147] and Biehler et al., 2015 [21], have investigated the potential applications of this methodology.

Figure 8 indicates that 55% of the studies that construct an MFSM utilize deterministic methods (DMs), while 45% employ non-deterministic methods (NDMs). Additionally, the figure provides insight into the distribution of each method employed for MFSM construction, as outlined in Section 4.2. An inference that can be made from the data is that multiplicative methods are predominantly utilized in DMs, while comprehensive corrections are the prevailing approach in NDMs.

The 20th-century publications were predominantly characterized by the use of DMs. However, there has been a notable surge in interest towards NDMs within the statistical community in the early 21st century. Several examples of NDMs that have been widely embraced include Kriging [93], which entails spatial dependence estimation in geostatistics, co-Kriging models [123], an expansion of Kriging to multiple-fidelity datasets, and related MFSMs such as Kennedy and O’Hagan’s Bayesian-based calibration models [90]. Figure 9 displays histograms illustrating the distribution of published papers related to DMs and NDMs over different yearly intervals.

In the context of MFSMs, the uncertainty prediction in Kriging surrogates can be modeled using techniques such as Generalized Least Squares or Gaussian Process (GP). A GP can be defined as a collection of random variables having the property that the joint distribution of any finite subset follows a Gaussian distribution. Kriging surrogates constructed using a GP kernel have gained popularity in recent years, as evidenced by their application in Kennedy and O’Hagan, 2000 [89], as well as in LeGratiet, 2012 [111], LeGratiet, 2013 [110], and LeGratiet, 2014 [113].

Numerous popular MFSMs, such as co-Kriging and Bayesian-based comprehensive corrections, employ GP to model each fidelity response and its corresponding prediction uncertainty. However, MFSMs are typically based on certain assumptions, and if these assumptions are not met, the accuracy of the models could be impacted. For instance, the most prevalent assumption in GP-based techniques is the independence of the predictive uncertainty among fidelities.

Table 5 and Table 6 have been designed to systematically arrange the reviewed literature that employs DMs and NDMs, respectively, within the four categories of combining methods discussed in Section 4. The four categories are additive correction, multiplicative correction, comprehensive correction and space mapping.

5. Accuracy and Cost Reporting for Multi-Fidelity Papers

5.1. Reporting Assessment

The primary objectives of employing MFMs are cost and time savings. Thus, a comprehensive account of the savings achieved is critical. Despite this, the current body of literature that reports on cost versus accuracy analyses of MFMs is somewhat limited. An exemplary instance of proficient reporting can be observed in the work conducted by Park et al., 2017 [142] on the Hartmann 6 function example. Their research findings showcase a potential for achieving cost savings while maintaining high levels of accuracy. Specifically, the study reveals that maximum cost savings of up to 86% can be achieved while ensuring the same level of accuracy. Furthermore, it was demonstrated that maximum accuracy improvements of up to 51% can be attained while maintaining the same level of cost.

However, the savings offered by MFMs are highly dependent on the problem at hand. Thus, unless one is dealing with a class of problems of similar structure, the savings that a researcher reports for one problem may differ significantly from those obtained for other problems, even when the same methodology is employed. This issue becomes even more severe when the savings are attributed not solely to the SM construction but to the entire optimization process. For example, convergence results attained with first-order corrections can ensure global convergence of some algorithms, implying that the algorithm will converge to a local optimum of a problem, regardless of the initial guess. Nonetheless, the convergence rate will depend on the relative properties of the LFM and HFM. A more detailed examination of these issues can be found in Eldred et al. [49] and Peherstorfer et al. [145].

The accuracy of SMs is a critical factor in their selection for design optimization tasks. Cross-validation error (CVE) is a popular and effective measure for SM selection, which can be applied to both DMs and NDMs. In a study by Park et al., 2017 [142], the authors evaluated the efficacy of different MFSMs for a given design of experiments using CVE and compared the results with their actual rank based on the root mean squared error (RMSE). The study investigated eleven cases of six different MFSMs, including co-Kriging and NDM calibration with and without a discrepancy function, with various options. The results of the study showed that while CVE was not a reliable measure for selecting a good MFSM candidate, it was useful in identifying the worst candidate. Another measure for SM selection is the model likelihood, which gives different weights based on the MFSM uncertainty estimation. However, this measure is only valid for an MFSM that uses NDMs, such as co-Kriging.

It would be also informative to include the accuracy of LFMs, HFMs, and MFMs obtained at equivalent computational cost, as well as the cost of HFMs and MFMs obtained for the same level of accuracy, if available. Such an analysis has been carried out by Peherstrofer et al., 2016 [144], who account for accuracy in the calculation of the quantity of interest by presenting a plot of the RMSE as a function of the number of samples used. This plot provides an answer to the question of the accuracy of the MFM relative to the LFMs and HFMs at the same computational cost. Additionally, a second plot is included in the analysis, which reports time savings for multiple sample options. This plot provides an answer to the question of the savings associated with the implementation of the MFMs relative to the HFMs for the same level of accuracy.

Notwithstanding the challenges encountered, Table 7 summarizes the cost of MFMs versus HFMs based on reported data to give readers a general idea of the savings that MFMs may offer. Furthermore, the table is divided based on the application field, enabling MFM users to identify where MFMs have been more successful.

The expectation is that computational efficiencies are enhanced when low-fidelity models LFMs incur significantly lower costs than their high-fidelity counterparts HFMs. To validate this hypothesis, an empirical investigation was carried out by analyzing data obtained from prior studies that utilized MFSMs for optimization tasks. Figure 10 presents data extracted from 18 out of 120 reviewed papers that conducted optimization procedures, wherein both the LFM/HFM cost ratio for a single analysis (LFA/HFA cost ratio) and the MFM/HFM cost ratio for a complete optimization process (MFO/HFO cost ratio) were explicitly reported. The dashed line in the figure represents the threshold, points below which do not benefit from speed ups from using MFMs.

The results depicted in Figure 10 indicate that there is no noticeable correlation between the LFA/HFA and the MFO/HFO cost ratios. This outcome may be attributed to the problem-dependent nature of savings, as previously noted, as well as to the relationship between the cost and accuracy of the LFM. Specifically, less expensive models may exhibit lower levels of precision, delaying the optimization convergence. Additionally, the complexity of the MFM model may also increase the cost of optimization.

In summary, the utilization of MFMs has the potential to enhance computational efficiency. However, the extent of benefits attained highly relies on the cost and accuracy of the LFM employed and the particularities of the optimization problem at hand. As a result, the choice of model should be made carefully, considering the trade-off between cost and accuracy and the specific attributes of the optimization problem under investigation.

5.2. Reporting Recommendations

The present study suggests that authors who employ MFMs in their research should provide a comprehensive account of cost, savings, and accuracy comparisons between said models and other models utilized in their investigations. While the inclusion of such information is occasionally present in the manuscripts, its presentation is often hidden, posing difficulties for the readers in assessing the benefits and shortcomings of implementing multi-fidelity models. As a solution to this matter, this work advised authors to consider presenting the pertinent information in a tabular format that encompasses the following constituents, when applicable:

1.

Basic information

(a)

Differentiation between low-fidelity and high-fidelity models.

(b)

Details of surrogate models constructed, if any.

(c)

Description of the process or method employed to combine fidelities.

2.

Cost

(a)

Cost comparison between low-fidelity and high-fidelity models.

(b)

Cost comparison between multi-fidelity and high-fidelity models.

(c)

Accuracy comparison between low-fidelity, high-fidelity, and multi-fidelity models at equivalent costs.

(d)

Cost comparison between multi-fidelity and high-fidelity models for the same level of accuracy.

3.

Cost-benefit analysis

(a)

Cost comparison between low-fidelity and high-fidelity surrogate models (if surrogates are constructed).

(b)

Cost comparison between high-fidelity surrogate models and high-fidelity models (if surrogates are constructed).

(c)

Time and resources invested in constructing the multi-fidelity model.

5.3. Example of Good Reporting

The implementation of MFMs has the potential to provide benefits in terms of both time savings and accuracy improvements. However, the usefulness of such implementations is often difficult to ascertain from research papers. To address this issue, Section 5.2 includes a guidance for authors on how to report of the cost, savings, and accuracy associated with the use of MFMs in their research. Padrón et al. [140] provides a thorough investigation into the optimization of an airfoil, serving as an exemplary illustration. The study employs computational fluid dynamics (CFD) RANS simulations as HFM and Eulerian CFD simulations as LFM, and the use of a stochastic polynomial chaos expansion as SM combined through additive correction. Future authors are encouraged to follow Padrón et al.’s approach and include the information presented in Table 8, which provides an overview of the essential elements that must be included in any study related to MFM.

6. Conclusions

The versatility of the multi-fidelity modeling approach makes it a promising method open to improvement and development, as evidenced by its diverse applications in various fields. This study provides an exhaustive review of multi-fidelity models and their remarkable characteristics, with optimization being the most prevalent application in 70% of the reviewed publications. Fidelity-management strategies in multi-fidelity models consist of two approaches: creating a surrogate model that blends the different fidelities or utilizing the various fidelity models in a hierarchical manner according to a specific criterion. The former method is more commonly used, as evidenced by 67% of the examined literature. In the late 1990s, deterministic methods were favored; however, non-deterministic methods are currently preferred due to their ability to provide uncertainty estimates. Gaussian Process-like surrogates have replaced basis function regression as the most common surrogate model for constructing multi-fidelity models. Despite the potential for cost and time savings, there was no clear correlation between the cost ratio of low-fidelity and high-fidelity analysis and the cost ratio of multi-fidelity and high-fidelity optimization. This lack of correlation may be attributed to the problem and model-specific characteristics of the considered analysis. Reporting of cost and time savings metrics was often absent or unclear in the reviewed literature, so this paper recommends standardizing reporting to enable better understanding of multi-fidelity model efficacy in various fields.