Towards the Validation of Dynamical Models in Regions where there is no Data

The creation of computer models is often driven by the need to make predictions in regions where there is no data (i.e. extrapolations). This makes validation challenging as it is difficult to ensure that a model will be suitable when it is applied in a region where there are no observations of the system of interest. The current paper proposes a method that can reveal flaws in a model which may be difficult to identify using traditional approaches for model calibration and validation. The method specifically targets the situation where one is attempting to model a dynamical system that is believed to possess time-invariant calibration parameters. The proposed approach allows these parameters to vary with time, even though it is believed that they are time-invariant. The of such an analysis is to identify key discrepancies indications that a model has inherent flaws and, as a result, should not be used to influence decisions in regions where there is no data. The proposed method isn’t necessarily a predictor of extrapolation performance, rather, it is a stringent test that, the authors believe, should be applied before extrapolation is attempted. The approach could therefore form a useful part of wider validation frameworks in the future.


Introduction
Computational models are crucial to many fields of science and engineering.Before being used to influence decision-making, such models must be validated; it must be established that the model can represent the real world with a level of accuracy sufficient for its intended purpose.Model validation is, as a result, an active research area.
In a broad sense, the current paper considers the situation where: 1.There may be several competing model structures that can, potentially, be employed by a user to simulate a quantity of interest.
2. Each model structure features a set of parameters that will need to be calibrated during the validation process.
3. A set of observations, from the system of interest, are available to the user (such that this data can be used to help infer parameter estimates and/or facilitate model selection).
While such scenarios are considered by much of the literature, the current paper differs from most in that it explicitly considers the issues associated with extrapolation.Here, 'extrapolation' refers to a situation where a model will be used to make predictions in a region that, according to a form of distance metric, is 'very far' from current experimental observations.Using the terminology given in [1], the current work considers the situation where a model must be applied outside of its 'domain of applicability'.
Validating models that will be used to extrapolate is often confounded by the fact that, in the regions where measurements are available, calibration can mask model error.In other words, a model's parameters can often be tuned such that the model appears to be sufficiently valid compared to the available data, even if the model will be a poor extrapolator (this is illustrated in Section 4.2 of the current paper).The current work presents an approach that can, in such a scenario, illustrate the flaws in a model; providing a warning that the model should not be used for extrapolative predictions.
The proposed approach focuses on situations where it is believed that the model of interest's parameters are time-invariant.A calibration procedure is then employed that deliberately treats the model's parameters as being time-varying, even though this is not believed to be the case.It is by observing that a model's parameters must vary with time if it is to replicate a set of measurements with a high level of fidelity that one can identify key discrepancies in the model that may be hidden by other calibration procedures.The approach is demonstrated on applications from the field of structural dynamics.

Literature Review
The idea that statistical models can be used to emulate model discrepancy (see [2] and [3] for example) has influenced a large body of work.Such an approach accepts that there will always be a discrepancy between a model that is formulated on the understanding of the system of interest (a model that is based on physical laws, for example) and the system's true response.A purely data-based model is then trained and used to realise predictions of this discrepancy.Subsequent to calibration, then, predictions are made using a combination of the model that is based on an understanding of the system and the data-based discrepancy model 1 .
One of the best-known implementations of this approach is that of Kennedy and O'Hagan [2], where Gaussian Processes (GPs) were used to create statistical models of model discrepancy (as well as emulators, for the cases where simulations of the system of interest are computationally expensive).This idea has contributed to many works in the field of model validation, including generalised validation frameworks [4] [5] [6], model validation metrics that consider model discrepancy [1], metrics to establish when sufficient experimental data has been gathered in the model validation process [7] and approaches for resource allocation (between code development and experimental testing [8] or the development of substructures within a model [9]).
While undoubtedly a useful contribution, employing data-based models of discrepancy cannot help improve extrapolations that are far from the training data (GPs, for example, will typically return very quickly to predictions based only on prior knowledge of the discrepancy if extrapolated to a region where there is no data).This issue is highlighted by a number of authors.In [10] it is stated that 'the primary use of engineering models is to extrapolate to a system with new nominals when there are no new field data' but that extrapolation, using the approach described in [2], can only be performed if there is a 'modest' change in the inputs.The authors of [11] note that 'such a statistically adjusted engineering model lacks predictive power in the sense that it can be used only in the experimental conditions identical or very similar to those under which the statistical model was fitted' and in [12] it is stated that '...the discrepancy representation ... is highly dependent on calibration against observables and, hence, should not be used in situations in which it cannot be trained and tested'.
It is important to note that the above discussion isn't a direct criticism of [2] -the authors of [2] did not claim that the method will provide reliable extrapolations.A discussion of the Kennedy and O'Hagan approach is given by BrynjarsdÃşttir and O'Hagan in [13], which considers how the use of GP emulators of model discrepancy can (1) aid the identification of a system's 'true' parameter values and (2) aid extrapolations.It is concluded that true parameter values can only be uncovered if realistic prior information about the model discrepancy is available and that, even if realistic prior information is available, extrapolation is not advisable.[13] highlights these two issues using a simple case study.
The current paper considers the situation where there is no data in the region one wishes to make predictions.This separates it from [1] and [7] which (respectively) describe a 'predictive maturity index' and a 'forecasting metric', as these two approaches utilise hold-out experiments (experimental observations, in the application domain, that are deliberately removed from the training data so that they can be used to aid validation at a later stage).The proposed method is designed to reveal additional engineering knowledge about a particular system and, as a result, form part of the frameworks described in [11] and [12], both of which specifically consider extrapolation.

General Framework
The current work specifically concerns the analysis of dynamical systems.Using x i to represent a system's state at time i∆t then, generally, it is assumed that the system's state evolves according to where ∆t is a fixed time increment, u i is a system input, f is a model that predicts the next system state given (x i−1 , u i ) and v i−1 is a noise term that reflects the uncertainties involved in predicting the system's next state (arising because f is an approximation of the true system).Equation ( 1) is typically referred to as the 'prediction equation'.
Likewise, using z i to represent an observation of some aspect of the system at time i∆t, it is also supposed that measurements are made according to where h is a (potentially nonlinear) function of the system's state and n i is used to capture uncertainty in the observation process (measurement noise).Equation ( 2) is typically referred to as the 'observation equation'.It is noted that, in the literature, the prediction and observation equations are often written as potentially nonlinear functions of their noise terms (v i−1 and n i respectively) -this has not been done here for the sake of notational simplicity later in the paper.

Extended Kalman Filter
The extended Kalman filter is one of a collection of methods that target situations where the aim is to probabilistically 'track' how the state of the system of interest changes, as more observations arrive over time.If, in the prediction and observation equations, f and h are linear functions of x i−1 and x i respectively then it is possible to realise closed-form solutions for this problem.These solutions form the well-known Kalman filter.If f and h are nonlinear then, generally, such closed-form solutions are unavailable.One way to address this is to approximate f and h using linear functions (via first-order Taylor series expansions), before applying the 'standard' Kalman filter equations to the new, linearised relationships.
This linearised approach is called the extended Kalman filter (EKF) 2 .
2 There are many other ways in which this problem could be tackled.One of the most well-known is the particle filter -a numerical method that is fundamentally based on importance sampling (see [14] for a tutorial).Here, however, it was found that an extended Kalman filter performed acceptably well and that the prediction and measurement equations were not sufficiently nonlinear to warrant use of a particle filter.
The following text gives a brief description of an extended Kalman filter, to establish notation used later in the paper.Considering the situation where (such that the probability density function (PDF) of x i−1 given z 1:i−1 is Gaussian with mean m i−1|i−1 and covariance matrix P i−1|i−1 ) then, with an EKF, the linearised prediction equation is used to predict the next system state.This yields the PDF Once a new observation arrives (z i ) then, through the observation equation, the current estimate of the system's state can be updated to obtain: where the parameters of the PDFs in equations ( 3), ( 4) and ( 5) are as follows: and the following definitions have been employed: 3.3.Application to the 4th order Runge-Kutta integration scheme The following examples consider dynamical systems that may be subjected to a time history of known excitations: δ 1 , δ 2 , .... Throughout the paper, the state of these systems is calculated recursively using the 4th order Runge-Kutta numerical integration scheme.To help relate this scheme back to the general prediction equation (equation ( 1)), some additional notation is required.Specifically, the following definitions are employed: where ∆t is the time-step of the integration scheme and δ int = 1 2 (δ i−1 + δ i ).Using this notation, the function f that maps x i−1 to x i can be written as To find an expression for F x (equation ( 12)) it is noted that Evaluation of terms 1-4 in equation (21) essentially requires the application of some straightforward vector calculus.For the sake of completeness, the required derivatives are shown in detail in Appendix Appendix B. Ultimately, it can be shown that where F (x, δ) represents the Jacobian of ẋ, with respect to x i−1 , evaluated at the points x and δ.

Description
Example 1 considers modelling a single degree of freedom nonlinear dynamic system whose equation of motion is In equation ( 26) x is displacement, c is a linear damping coefficient, k is a linear stiffness coefficient and k * is a nonlinear stiffness coefficient that controls the strength of the system's hardening stiffness nonlinearity.
δ(t) is a random excitation that, in this case, consists of samples drawn from a zero-mean unit-variance Gaussian.
To generate a set of observation data, the nonlinear system was simulated using a time step of ∆t = 0.01s.The following considers the case where one is attempting to fit a linear model to data that is obtained from the nonlinear system.It is also considered that, after calibration, the model will be used to predict the response of the system to a much greater amplitude excitation than that used to generate the data in Figure 1.It is noted that the excitation used to generate the observation data is relatively low, such that the nonlinearity is difficult to detect.

Highlighting the Problem
This section illustrates that, by tuning its parameters, it is possible for the linear model to represent observations of the nonlinear system fairly accurately in the low amplitude regime where there is data, potentially giving false confidence in the model's ability to extrapolate.Specifically, a common Bayesian approach (that assumed time-invariant parameters) is adapted to the calibration of the model.The approach begins with Bayes' theorem: where θ is a vector of parameters to be calibrated and D represents training data (in this case D is a time history of the excitation and the observed displacement response of the system).The likelihood, p(D| θ), is defined based on the assumption that each observation is equal to the response of the model, corrupted by the addition of Gaussian noise.This noise is assumed to be zero-mean, have variance R and be independent of the noise that corrupted previous observations.In the following, R is treated as a time-invariant parameter that also requires calibration such that θ = (k, c, R) T .
Defining the likelihood in this way treats any differences between observations and model response (i.e.errors that arise because of both measurement noise and model discrepancy) as independent Gaussian noise 3 .Moreover, by allowing R to be calibrated, the variance of this noise can be tuned at the same time as the model's stiffness and damping parameters.This framework means that the maximum likelihood set of model parameters are those that make the discrepancy between the model and the observations most closely approximate a Gaussian distribution with variance R.This way of treating uncertainty in the calibration procedure has been applied by many (including the author's of the current manuscript) and is referred to in [13] as a 'traditional approach'.
For the academic example discussed in the current section, prior distributions over the calibration parameters were chosen to be such that the prior over R is a Gamma distribution whose shape and rate parameters are equal to 1 and 100, respectively.Using the first 50 seconds of data for training, the Markov chain Monte Carlo (MCMC) algorithm detailed in [17] was used to generate 1000 samples from the posterior, p(θ |D).
Histograms of the resulting samples are shown 'close up' in Figure 3 while, in Figure 4, histograms are plotted alongside the true parameter values.These figures highlight two observations.Firstly the posterior distribution over k is concentrated over values that are larger than the true stiffness.This illustrates how k has been calibrated to overcome the model discrepancy (the lack of the nonlinear term).Secondly, the distribution over R is also concentrated on values that are larger than the true measurement noise variance, highlighting that this particular approach has attempted to describe the model discrepancy present in the problem as additional noise on the observations (in other words, the errors present in the model have increased the most probable value of R, given the available training data).An illustration of the problem that motivates the current paper comes when the MCMC samples are used to propagate these parameter uncertainties into future model predictions.Recalling that there is only training data in a relatively low amplitude regime, Figure 5 shows the model's predictions relative to (1) the training data and (2) data that was 'held back' for validation purposes.Figure 6 shows a close-up of the model's ability to replicate the validation data.These results illustrate that, for low amplitude excitations, the model is able to closely replicate data that was not used in training; the model has been tuned to work effectively in the region where we have data, despite lacking the nonlinear stiffness term.Subsequently, the model was used to predict the response of the system to a higher amplitude excitation, whereby δ(t) consisted of samples drawn from a zero-mean Gaussian whose standard deviation was equal to 2. Figure 7 illustrates that, for this higher amplitude excitation, the model is not able to accurately replicate the response of the system (or accurately quantify the uncertainties involved in the predictions).This numerical example highlights how, without the data from the high amplitude case, it can be difficult to infer that the model would be a poor extrapolator.The calibration parameters of M 1 and M 2 are now written as θ 1 and θ 2 respectively, such that Here, for illustrative purposes, Bayes factor is used as a model selection indicator. 4Evaluating Bayes factor involves computing the relative probability of the two model structures, given the available measurement data: where In this case, equal prior probability mass was assigned to each model such that Pr(M 1 ) = Pr(M 2 ).
The integrals shown in equation (31) can be estimated using a variety of numerical methods ( [19][20] [21] for example).Here, Bayes factor was estimated using Laplace's method, which employs Gaussian approximations of the posterior parameter PDFs of the two models (see appendix Appendix A for details).It was found that: Consequently, according to this particular model selection criterion, the incorrect linear model is viewed as preferable relative to the nonlinear model.This is despite the fact that the nonlinear model is actually a perfect representation of the system of interest.The results reflect that, in this low amplitude region where the effects of the nonlinearity is relatively small, the chosen model selection criterion views the nonlinear model as overly complex (indeed, in the region where data is available, the linear model does appear to perform well). 4The authors acknowledge that many different model selection criteria exist and that the question of how model selection criteria should be defined is still widely debated (as illustrated in [18], for example).The authors do not intend to critique these criteria here, except to say that the majority of model selection criteria (1) penalise overly complex models (as these are more prone to overfitting) and (2) define 'model complexity' as a function of the number of tunable parameters that are in a model.Bayes' factor is an example of such a criterion that is used relatively frequently in the literature.
The next section (Section 4.3) illustrates the application of the proposed approach to the same situation, using the same set of observation data.

Proposed Approach
In the proposed approach, to treat a model's parameters as time-variant, the parameters are included in the definition of the system's state.This allows them to be 'tracked', alongside the system's displacement and velocity, using the EKF (see [22] for a similar approach that utilised particle filtering techniques).Here it is illustrated how the approach can be implemented for M 2 (the model that includes the nonlinear stiffness term), where numerical integration is conducted using the RK4 integration scheme.Implementation of the approach to M 1 is very similar and, as such, is not discussed in detail here.
The state of M 2 at time i∆t is now defined as In order to implement the EKF, one must hypothesise how the model's parameters vary through time i.e. a definition for f in equation ( 1) must be realised.As with [22], the chosen prediction model assumes that each system parameter is equal to its preceding value, such that Consequently, the Jacobian F is given by One must also quantify the uncertainty in this predictive model (in other words, the covariance matrix 1) must be defined).Here, Q i−1 is defined as follows where G i−1 is a diagonal matrix that, for model M 2 , is of size 3 × 3. The approach assumes that there is zero uncertainty associated in the prediction of displacement and velocity, and that all of the uncertainty is manifested in the forward prediction of the model's time-varying parameters.This is deliberate -the intention is that, by tracking the parameter estimates as they vary with time, it will be possible to infer indications of model error.Following on from [22], the diagonal elements of G were chosen such that the coefficient of variation in the drift of each parameter is fixed.For the numerical case study shown here the coefficient of variation was set equal to 0.1 % (note that the influence of this term is investigated more thoroughly in Section 5).  Figure 8 shows the results that were obtained when the parameters of the linear model were treated as being time-variant, and tracked using the EKF.As one would expect, this method allows the displacement and velocity to be tracked very accurately -a comparison between the observed and predicted is not shown here because the two are essentially indistinguishable.Figure 8 shows that, relatively speaking, the system's damping value converges while the linear stiffness varies with time.Bounds on parameter estimates are 3 standard deviations from the mean.
To aid comparison this analysis was repeated for the nonlinear model.These results are shown in Figure 9 while Figure 10 shows a comparison between the stiffness values that were identified, as a function of time, for both the linear and nonlinear models.To achieve an excellent fit to the observation data it can be see that the stiffness of the linear model had to vary significantly more than the stiffness of the nonlinear model.
Recalling that the analysis is (deliberately) applied to systems whose parameters, it is believed, should not be varying with time the important question is: can the results in Figure 10 be used to infer that the linear model would extrapolate poorly if used at high amplitudes?The authors concede that the answer is not straightforward and it is clear that these results would require a significant amount of interpretation, however, it is also argued that allowing the parameters to vary with time provides a different type of information relative to the more traditional approach shown in Section 4.2.It is certainly conceivable that, by studying the results in the current section, a practitioner may lead to a different set of conclusions about the suitability of the linear model for extrapolations compared to if they had studied the results in Section 4.2 (i.e. the more traditional approach).
A more in depth discussion of how such results could be utilised in a general setting is postponed until Section 6.First, in Section 5, it is illustrated how the approach may be applied to an experimental case study.

Description
This section analyses the dynamics of the two storey aluminium structure shown in Figure 11.Testing was conducted whereby lateral vibrations were induced by a shaker at the base of the structure.Measurements were obtained from 3 triaxial accelerometers that were placed at the base, first floor and second floor of the structure.The structure was excited sinusoidally at a frequency close to its first linear natural frequency, at a variety of amplitudes.The full set of resulting observation data is shown in Figure 12.In the following, measurements of the system's dynamic response are used to infer a model of the structure.Again, the situation where the model must be used to extrapolate far beyond available data is considered (for example, the model may be needed to predict the response of the structure to excitation levels far higher than those shown in Figure 12).
A 2DOF shear-storey model of the building was developed.Defining x 1 and x 2 as the displacement of floors 1 and 2 respectively, the dynamic model is based on the following: where x base is the displacement of the base, k 1 and c 1 are the stiffness and damping between the base and floor 1, k 2 and c 2 are the stiffness and damping between floor 2 and floor 1 and m i is the mass of the ith floor.Measurements of base displacement and velocity were obtained via numerical integration of the observed acceleration time histories (after the appropriate signals had been passed through a low-pass butterworth filter).The two masses, m 1 and m 2 , were measured directly (with, it was assumed, a sufficient degree of accuracy) thus leaving the parameters k 1 , k 2 , c 1 and c 2 to be calibrated.
In the next two sections two different approaches to the calibration and validation of this model are illustrated.The first shows the results of a Bayesian analysis where it is assumed that the parameters do not vary with time (this is the same 'traditional approach' that was demonstrated in Section 4.2 of the numerical case study).The second is the proposed approach, whereby the calibration parameters are treated as being time-variant.It is important to emphasise that, in the subsequent analyses, the aim is not necessarily to identify a model which could be relied upon for extrapolations.Rather, the intention is to illustrate how the proposed approach can reveal useful information that, after interpretation, helps to establish a model's validity outside of the region where data is available.

Time-invariant parameters
Here, the first 20 seconds of the data shown in Figure 12 were used for training.The observations of base excitations were treated as an input to the model while acceleration measurements from the two floors were used to infer parameter estimates.Fundamentally, the analysis proceeded in the same manner as described in Section 4.2 (the 'traditional approach').To formulate the likelihood it was assumed that, at each floor, the acceleration observations were equal to the response of the model plus the addition of independent Gaussian noise.The variance of the noise that corrupts observations for floors 1 and 2 was defined as R 1 and R 2 respectively.In the following, the masses of the floors are assumed known while the remaining stiffness and damping terms were treated as calibration parameters alongside R 1 and R 2 .Prior distributions were as follows: (noting that, to help form p(k 1 ) and p(k 2 ), initial estimates of the inter-storey stiffnesses were realised by modelling each column as a cantilever beam).Once again, MCMC was used to generate 1000 samples from the posterior parameter distribution.Histograms of these samples are shown in Figure 13.Using the samples to propagate parameter uncertainty into model predictions, the ability of the model to replicate data that was not used for training is shown in Figure 14.It can be seen that the model appears to perform relatively well and that the observed response stays within three standard deviations of the mean prediction.Black represents previously unseen data and red represents model predictions.Bounds on model predictions are 3 standard deviations from the mean.

Time-variant parameters
This section describes the application of the proposed approach to the experimental case study.To track how the model's parameters vary with time, the system's state was defined as while the derivative of the state with respect to time was ẋ1 ẋ2 ẍ1 ẍ2 0 0 0 0 where ẍ1 and ẍ2 are defined by equations ( 36) and (37) respectively.The matrices F and H that correspond to this system are shown in Appendix Appendix C.
The matrix Q was defined as described previously, such that the coefficient of variation in the drift of each parameter was set equal to 0.1%.Defining z 1 and z 2 as the observed acceleration of floors 1 and 2 respectively, the full set of parameter tracking results are shown in Figure 15.It was observed that, once again, the acceleration predictions made by the EKF were almost indistinguishable from the observed.Note that, in the following figures, uncertainty bounds are omitted for the sake of clarity.Figure 16 shows a region where the system transitions from a relatively high amplitude response to a relatively low amplitude response.When in steady state it is observed that, for all four parameters, a periodic oscillation is required for the model to fit the training data very precisely (see results between 150 and 152 seconds, for example).This type of behaviour was observed for all steady state portions of the data.During a transitional period (see between 152 and 154 seconds, for example), as well as relatively large changes in stiffness, the magnitude of both damping terms approximately doubles.The reasons for this could be many (for example, from experience, it is known that the response of this system is very sensitive to the torque used when bolting the columns to the two floors -a possible indication that a linear damping model may be insufficient).For now this is left as speculation.The authors wish to emphasise that the current paper is not contributing an overall framework for validation but, instead, is providing a tool that could form part of such a framework in the future.It is also argued that, relative to the more traditional approach, this example highlights how treating the parameters as time-varying can lead to a richer source of information for future model development.At the very least, the results indicate that the model performs worse in transitional regions and that, before being used for extrapolation, more efforts should perhaps be diverted towards understanding the damping mechanisms present in the structure.Such conclusions would have been more difficult to reach using the results presented in Section 5.2.

Discussion and future work
One of the key parameters in the proposed approach is the 'drift term', which essentially controls the magnitude of the diagonal elements of the matrix G in equation ( 35).Selection of the drift term requires some thought.If the drift term is large then large variations in parameters become more probable.This can lead to over fitting, whereby the calibration parameters are varied such that the model begins to fit the measurement noise in the data.Likewise, if the drift term is small then the parameters cannot vary quickly enough for the model to fit the observation data.
To illustrate how the drift term may affect results, the experimental case study was revisited using coefficient of variations in the drift of each parameter set equal to 0.001%, 0.01% and 0.1% respectively.
Figure 19 shows the resulting parameter estimates for these three cases, (specifically focusing on the transitional region shown in Figure 16).As one would expect, larger drift terms leads to larger variations in the parameters.Figure 19 shows the ability of the model to replicate the observed acceleration data for these three scenarios.While the closeness of the overlays makes it difficult to see the measurement data, it can be observed that the red and green lines are almost exactly the same while the blue line, which represents the lowest drift term, deviates slightly from the others.Figure 19   In the simulated case study, model error was deliberately introduced to a well-defined problem.This was to illustrate how the proposed methodology could behave in such a scenario.For the experimental case study, where results indicate that the model should be improved before being relied upon for extrapolations, a similar procedure could be used to aid model development.Specifically, in a similar manner to Section 4.3, numerical experiments could be conducted whereby models with deliberately introduced discrepancies could be calibrated against synthetic data until results similar to those obtained from the experiment are observed (parameters varying periodically with time, large increases in damping in transitional regions etc.) To this end, the authors revisited the numerical case study in Section 4.3 using a sinusoidal excitation instead of a random excitation.Interestingly this did not lead to results similar to the experimental case study -the parameters did not have to vary periodically for an excellent fit to the training data to be achieved.This, perhaps, indicates that the type of model error being simulated (missing stiffness nonlinearity) is not what was witnessed in the experimental case study.Future work may investigate other types of model error (for example, the errors induced when reducing the system to a 2 degree-of-freedom model).
Whether or not such model error can be 'reverse engineered' from such results is an interesting topic for future work.
Further work will also seek to combine the proposed approach with a sensitivity analysis.This would help to establish, in a more rigorous manner, whether the witnessed variations in parameter values are truly significant.Combining the approach with a sensitivity analysis of the model in the region where it is to be extrapolated could be an important part of an overall validation framework.In particular, a detailed study concerning the affect of 'extrapolation severity' (in other words, the possible consequences of extrapolating further beyond the available data) should be conducted.Ultimately, the intention is to embed the proposed approach into validation frameworks that specifically consider extrapolation (see [11] and [12] for example).

Conclusions
This paper focuses on the situation where a model of a dynamical system is required to perform extrapolations (predictions in regions where there is no data).It considers how such a model could be validated, given that observation data is only available in a region far from the model's application domain.Such situations are often complicated by the fact that, through tuning a model's calibration parameters, it is often possible to gain false confidence in a model that will, in reality, perform poorly when used for extrapolation.
Such a scenario is demonstrated using a numerical case study.
For the situation where is believed that a model's calibration parameters should be time-invariant, a procedure that allows these parameters to vary with time (even though it is believed that, physically, this should not be the case) is proposed.Results suggest that, by following such an approach, it is possible to identify important flaws in a model that would otherwise be hidden by more traditional calibration procedures.This is illustrated by a numerical and an experimental case study.Noting that acceleration measurements are begin utilised and that acceleration is related to the system's state by equations ( 36) and (37), it can be show that ).
were created by taking the resulting displacement time history and corrupting it with zeromean Gaussian noise (variance equal to 1 × 10 −5 ).The model's parameters were k = 10 N/m, c = 0.1 Ns/m and k * = 50 N/m 3 .Observation data was generated over a time period of 200 seconds.The full set of observation data is shown in Figure 1 while a zoomed-in portion is shown in Figure 2.

Figure 1 : 7 PreprintsFigure 2 :
Figure 1: 200 seconds of displacement time history observations from the numerical case study.Black shows the true response of the system while red shows noisy observations.

Figure 3 :Figure 4 :
Figure 3: Histograms of samples from the posterior parameter distribution of the linear model in the numerical case study.All parameters in SI units.

Figure 5 :Figure 6 :
Figure 5: Propagating parameter uncertainties of the linear model into predictions (numerical case study).Blue represents training data, black represents previously unseen data and red represents the average model predictions.Bounds on model predictions are 3 standard deviations from the mean.

Figure 7 :
Figure 7: Propagating parameter uncertainties of the linear model into predictions of previously unseen, 'high amplitude' data (numerical case study).Bounds on model predictions are 3 standard deviations from the mean.

Figure 8 :
Figure 8: Tracking time varying parameters for the linear model (numerical case study).Black lines show true parameter values, red lines show estimated parameter values.Bounds on parameter estimates are 3 standard deviations from the mean.

Figure 9 :
Figure 9: Numerical case study.Tracking time varying parameters for the nonlinear model.Black lines show true parameter values, red lines show estimated parameter values.Bounds on parameter estimates are 3 standard deviations from the mean.Blue axes show a 'close up' view of results relating to k, the linear stiffness.

Figure 10 :
Figure 10: Numerical case study.Tracking the stiffness for the nonlinear model (red) and linear model (blue).

Figure 11 :Floor 2 Figure 12 :
Figure 11: Test rig for the experimental case study.

Figure 13 : 18 Preprints 2 )Figure 14 :
Figure 13: Histograms of samples from the posterior parameter distribution (experimental case study).All parameters in SI units.

2 Figure 15 :
Figure 15: Experimental case study.Panels 1 and 2 show the observed acceleration of floors 1 and 2 respectively.Remaining panels show the tracked values of the model's parameters (mean values only).Red, blue and green portions are shown close-up in Figures 16, 17 and 18 respectively.SI units used throughout.

2 Figure 16 :
Figure 16: Close-up view of the red portion of results in Figure 15 (experimental case study).

Figures 17 and 18 2 Figure 17 : 21 Preprints 2 Figure 18 :
Figures 17 and 18  show the results obtained when the system is at steady state, for a relatively low and high amplitude response respectively.It is interesting to note that, regardless of amplitude, near-periodic oscillations in all four parameters are required for the model to accurately fit the training data.

Figure 20 :
Figure 20: Experimental case study.Tracking the acceleration of floors 1 and 2 (mean values only) using different drift coefficients of variation.Acceleration in SI units.

m1 1 m1
fact, these are just the 3rd and 4th rows of F

preprints.org) | NOT PEER-REVIEWED | Posted: 19 September 2018 doi:10.20944/preprints201809.0389.v1 are
now considered: model 1 (M 1 ) is the linear model that was used in the previous analysis while model 2 (M 2 ) includes the nonlinear stiffness term.Model 2 has the same structure as equation (26) and therefore represents a scenario where there is no model error.The following analysis illustrates how, even if a model whose structure was exactly correct is included in the analysis, well-known model selection criteria, based on the available data, may still lead to the selection of the incorrect model.
To further illustrate the problem, model selection is now considered.Specifically, two potential models 11 Preprints (www. therefore illustrates how, if the drift term is too low, the model loses fidelity with respect to the measurement data.Figure 19: Experimental case study.Tracking the values of the model's parameters (mean values only) using different drift coefficients of variation.All parameters in SI units.