Where Fault Detection and Diagnosis Meets MPC Performance Assessment: Review and Case Study of an Integrated Framework

1. Introduction

Model Predictive Control (MPC) is a prominent control strategy in industry due to its ability to handle system constraints, anticipate process behavior, and be implemented in multivariable systems. The usage of models gives more robustness regarding uncertainties for the control system [1,2].

However, this generation of controllers is susceptible to performance issues, just like any other kind of controller. The main challenge in monitoring control loops is distinguishing between control-related causes of poor performance and those associated with disturbances, instrumentation limitations, or the process itself. Among the control problems, poor tuning settings are one of the main problems in control loops [3,4]. In the MPC formulation, although there is an expectation of how each tuning parameter is meant to influence in the predictive controller formulation, it is not easy to find the combination of parameter values that achieves a desired controller performance [5].

Moreover, a successful implementation of MPC, since it uses models to predict process behavior, also depends on the accuracy of process models [4,5,6]. When a model predictive controller starts to perform poorly due to model-plant mismatch, it may be difficult to determine which submodel(s) need to be updated. Often, a whole model identification and updating is unfeasible, since it is one of the most time-consuming activities during the implementation of predictive controllers, and it may cause unnecessary disturbances to the process. Most industrial MPC packages include model identification tools that can help the user to develop models needed for MPC based on plant data. However, identification software can suffer from poor workflow, requiring many steps, which increases opportunities for user mistakes [5,7].

To help operators and engineers identify whether a control loop is performing poorly, the field of Control Performance Assessment (CPA) has been developed to study metrics and methodologies for monitoring control loops. Different metrics have been proposed over the years, aiming to monitor different types of controllers. For linear MPC, one of the main contributions was the Harris index, which compares the variance of the process with the one expected by the Minimum Variance Control theory [8]. However, there are some limitations regarding that metric. According to Bauer et al. [3], minimum variance-based indices, such as the Harris index, do not have the same impact in the industrial community as they have in the academic control community, due to the presence of non-stationary signals present in industrial data.

One of the main conclusions of a survey conducted by Bauer et al. [3] is that the simpler the method, the greater the number of respondents who find it useful. This means that CPA techniques need to be as simple as possible to guarantee good acceptance. Furthermore, even though MPC is becoming popular in the industry, there is still some resistance by operators to use it, due to its complexity. Operators need to have confidence to decide whether the MPC is underperforming [5]. The existence of an automated system that detects and diagnoses the controller’s performance may increase the acceptance of this kind of controller.

For this reason, Machine Learning (ML) techniques become interesting in the CPA context. ML can both assess and diagnose the control performance at once [9]. Moreover, the size and complexity of industrial data increased significantly over the years, especially considering variable interactions in a multivariable process. Some studies have already been conducted about the use of ML techniques applied to CPA. However, the number of publications remains underwhelming given the development of machine learning, especially in the context of MPC.

This article aims to elucidate the different methodologies available for MPC performance assessment. The classical breaking point for MPC performance monitoring, together with statistical methods used both for Fault Detection and Diagnosis (FDD) and CPA, is explained. Since model mismatch identification is one of the main challenges in MPC applications, methodologies focused on this issue are also elucidated. Finally, studies conducted using ML algorithms for MPC performance monitoring and assessment are presented. Moreover, a control-aware FDD methodology using ML models to detect and diagnose the closed-loop health of model predictive controllers is proposed. The specific objectives include evaluating the capability of the methodology to classify precisely whether the closed-loop is under healthy conditions, with instrumentation and equipment issues, control poor tuning, or plant-model mismatch.

This article is organized as follows: Section 2 gives an overview of a qualitative bibliometric analysis when considering the development of CPA techniques, especially regarding the ML tools, and also a review of different methodologies developed over the years for control performance assessment and its integration with FDD; Section 3 gives the methodology followed for the results presented in Section 4. Finally, Section 5 gives the main conclusion of this study.

2. Methodologies for Control Performance Assessment: Review

2.1. Bibliometric Analysis

This analysis aims to identify the thematic relevance over the years in the literature, based on the number of publications and their distribution per year. It is essential to note that the results presented in this section are for qualitative analysis only. The database used was the Web of Science, and different queries were used in the advanced search to plot the results. The choice of this database is due to its selectivity and data quality, since the Web of Science makes use of metrics to filter out articles with low impact [10]. Thus, since the focus is on the qualitative analysis, the macro tendencies are expected to be the same for different databases. The analysis presented in this section refers to the values found on March 27th, 2026.

To analyze the interest in control performance monitoring and assessment, for all kinds of controllers, the query ALL=("control* performance monitoring" OR "control* performance assessment" OR "control loop performance assessment") was used, returning 495 results and the distribution presented in Figure 1. The results indicate an increase in publication activity from the 1990s to the 2010s, with the latter representing the period of highest average output. This reflects the increase in monitoring and control performance. In the current decade, the average number of publications remains higher than in the early 2000s, but it is lower than the peak observed in the 2010s. However, the current decade is still incomplete.

By adding the query AND ("model predictive control*" OR "MPC" OR "NMPC") to the previous one to restrict the results only to model predictive controllers, the results drop to 79 publications, as can be observed in Figure 2. A similar decade-wise trend is observed when restricting the analysis to MPC-based approaches, with an increase in publication activity from the 2000s to the 2010s. However, the overall number of publications is significantly lower, reflecting the more specific scope of MPC within the broader field. Thus, there is still room to conduct further assessment studies on the performance assessment of this kind of controller, which may reflect the challenge in finding methodologies for monitoring model predictive controllers.

To analyze the presence of machine learning (ML) techniques in research about control performance monitoring, the query ALL=(("machine learning" OR "artificial intelligence" OR "neural network") AND ("control* performance monitoring" OR "control* performance assessment" OR "control loop performance assessment")) was used. Just 33 publications were found, with the distribution presented in Figure 3. From those publications found, only 4 include "model predictive control*" OR "advanced control*" OR "MPC" OR "NMPC" in their text.

However, when the number of publications for machine learning applied to fault detection and diagnosis is analyzed, by using the query ALL=(("machine learning" OR "artificial intelligence“ OR “neural network”) AND ("fault detection and diagnosis" OR "fault detection" OR "fault diagnosis")), the results showed a total of 25,717 publications. Figure 4 shows that there is also a crescent tendency in the number of publications in this field, also reflecting the popularity of the field and the interest in using ML tools to detect and diagnose faults in different applications.

The integration of control performance assessment and FDD is also uncommon in the literature. By using the query ALL = (("fault detection and diagnosis" OR "fault detection" OR "fault diagnosis") AND ("control* performance monitoring" OR "control* performance assessment")), only 38 articles were found, with the distribution presented in Figure 5. From those results, only 10 include "predictive control*" in their text. This shows that there is potential to explore FDD methodologies for MPC performance assessment, which is still underexplored.

The results presented so far show that, even though there is an increased interest in studying predictive controllers and ML applied to FDD, there are still few studies applied to the performance assessment of predictive controllers, especially using ML algorithms. According to Özdemir and Yıldırım [11], many ML algorithms have been used for decades without referring to ML explicitly. Venkatasubramanian et al. [12], for instance, discussed the use of artificial neural networks for fault detection and diagnosis without referring to it as a machine learning methodology.

Hence, variations in the terminology used to describe machine learning methods could potentially influence the results of the bibliometric analysis. However, when a more wide query was implemented (ALL=(("machine learning" OR "artificial intelligence" OR "neural network" OR "support vector machine" OR "random forest" OR "boosting") AND ("control* performance monitoring" OR "control* loop performance monitoring" OR "closed*loop performance monitoring" OR "control* performance assessment" OR "control* loop performance assessment" OR "closed*loop performance assessment"))), the number of publications increased just from 33 to 36.

This evidences lack of investigation into the technical feasibility of using ML algorithms applied to the performance assessment of predictive controllers. There may be some reasons why there are so few studies about using ML algorithms to monitor and assess predictive controllers. Firstly, there is the challenge of labeling and distinguishing the "normal" dataset from the "abnormal" one. When there is an instrumentation problem, it is more evident and easier to take corrective actions than when there is an inappropriate tuning setting in an MPC, due to its complexity.

Moreover, when the lack of performance in an MPC is detected, it may be more interesting for the industry to correct it right away instead of identifying the issue. This is reflected in the number of publications mentioning adaptive control, which was more than 220 results only in 2025 (ALL=("model predictive control*" AND "adaptive control*")).

Even though there is a high interest in adaptive controllers, it is still important to evaluate the technical feasibility of using ML algorithms applied to the performance assessment of predictive controllers. For this reason, this work was elaborated to fill some of those gaps presented in the literature, by both exploring MPC performance degradation as a type of fault and by using ML models for the detection and diagnosis of root causes. In the next sections, a more detailed analysis was made of the methodologies available for monitoring and assessing predictive controllers.

2.2. Methodologies for MPC Monitoring and Detecting Performance Degradation

The model predictive controller, since its creation, has also been a subject of concern regarding its performance. Throughout the years, some methodologies have been developed to detect and diagnose the underperformance of this kind of controller. Many methods of control performance monitoring are from Fault Detection and Diagnosis (FDD), and most data-driven control performance monitoring methods rely on the use of normal process data to build normal-case process models.

2.2.1. Minimum Variance Control Based Metrics

The Harris Index, firstly mentioned by Harris [8] and then improved by Desborough and Harris [13], was the breaking point for MPC performance monitoring. It is based on the Minimum Variance Control (MVC) benchmark. Åström [14] proposed the MVC as a control strategy and benchmark that should be achieved. Its principle is based on the fact that, once the best achievable performance is reached, the variance of the controlled variables cannot be reduced by changing the tuning or the type of control strategy. Harris Index used the MVC theory as a lower-bound benchmark for assessing the performance of control loops. The index is defined according to Equation 1, in which ${\sigma }_{MVC}^2$ is the theoretical minimum variance of the process, ${\sigma }_{Process}^2$ is the actual variance observed in the process, and ${\mu }_{Process}$ is the mean absolute error from the setpoint.

$$I_{Harris}=1-{\displaystyle \frac{{\sigma }_{MVC}^2}{{\sigma }_{Process}^2+{\mu }_{Process}}}$$

(1)

To estimate the theoretical minimum variance of the process ${\sigma }_{MVC}^2$ from process data, Harris [8] used an auto-regressive moving average (ARMA) model [15]. The reason for using the ARMA model is to separate the predictable from the unpredictable variance in data, without the need for a physical model of the plant. Under minimum variance control, the controlled variable y can be written based on its past values and on its moving average time series model, according to Equation 2. The parameters $\phi$ and $\theta$ are the auto-regressive (AR) and the moving average (MA) parameters of the model, respectively.

$$y\left(z\right)=\epsilon \left(z\right)+{\displaystyle \sum _{i=1}^p}{\phi }_{i}y(z-i)+{\displaystyle \sum _{i=1}^q}{\theta }_i\epsilon (z-i)$$

(2)

The variable $\epsilon$ represents the error between the ARMA model and the process data, considered to be the white noise from the process. The parameter p is the AR order of the model, while the parameter q, on the other hand, refers to the model order of the MA. Harris [8] suggests using the autocorrelation function of y to choose the order of the ARMA model. The parameter p should be chosen as the lag beyond which the autocorrelation function is zero. The order q, on the other hand, should be equal to $p-1$. The minimum variance of the process ${\sigma }_{MVC}^2$ is then defined according to Equation 3, in which ${\sigma }_{\epsilon }^2$ is the noise variance and d is the dead time of the system.

$${\sigma }_{MVC}^2={\sigma }_{\epsilon }^2\left(1+{\displaystyle \sum _{i=1}^{d-1}}{\phi }_i^2\right)$$

(3)

For the performance assessment of multivariable controllers, Harris et al. [16] extended the approach by proposing a global performance index, as well as individual indices for each controlled variable. This approach has to take into consideration the coupling between variables. Thus, once each controlled variable has different dead times regarding each manipulated variable, the interactor matrix is necessary to calculate these indices.

Even though the Harris Index was a breaking point for MPC monitoring, it still finds some resistance in the industry. According to Bauer et al. [3], it is not so relevant for real applications as it is for the academic community. The main criticism about the approach is that it is based on an unrealistic benchmark. MVC is often an aggressive controller and may not be economically feasible. Another disadvantage of the Harris Index is the consideration of only stochastic disturbances, which limit its applicability for this scenario. Also, the methodology considers a stationary data trend, and it is meaningless if the data contains non-stationary signals [3].

Although the Harris Index for a MIMO system is not as practical as its form for a SISO system, the idea of considering variance as a metric to be monitored is implemented by commercial controller software such as Aspen APC and the EcoStruxure Control Expert. Those controllers include metrics such as the standard deviation of each variable as information about the closed-loop performance, especially for regulatory control.

Furthermore, the studies conducted to develop the indices based on MVC served as a basis for the emergence of other metrics. Numerous metrics have been developed and proposed in the literature. Huang et al. [17] proposed the method based on filtering and correlation (FCOR) for MIMO systems, also described by Huang and Shah [18]. It is also based on the MVC theory and requires a previous knowledge of the interactor matrix for MIMO systems, as it was done by Harris et al. [16].

2.2.2. Other Benchmarks for MPC Performance Monitoring

Other metrics were also developed, considering different benchmarks. Tyler and Morari [19] used Generalized Likelihood Ratio (GLR) tests instead of a single value based on a benchmark for comparison. This is a hypothesis testing problem to verify whether the control system is performing satisfactorily or whether the controller has deteriorated. The approach is more realistic than those based on a benchmark such as MVC. The index is calculated based on a ratio between the Maximum Likelihood of the model with constraints in the impulse response coefficients and of the model without the constraints. However, this methodology is also more computationally expensive, since it requires an optimization procedure, and may not lead to a precise diagnosis.

Huang and Shah [18] proposed using a Linear Quadratic Gaussian (LQG), instead of an MVC, as a benchmark. It takes into consideration the control efforts ($var\left(u\right)$ or variance of u) and the objective is to minimize both $var\left(y\right)$ and $var\left(u\right)$, considering the trade-off curve that represents the variability of each group of variables. This approach is more realistic than the MVC benchmark and may give some diagnostic insights. However, it requires the process model and is more computationally expensive than MVC for online applications.

Uduehi et al. [20] conducted a study on using Generalized Predictive Control as a benchmark for monitoring the performance of controllers in a SISO system, which is a benchmark less aggressive and more robust than MVC and includes the control efforts in its equations. However, the index based on GPC also requires more computational efforts than the MVC theory, since the calculation is not as simple and requires a successive approximation algorithm.

2.2.3. Statistical Techniques

Qin [21] reviewed the use of data-driven methods for process monitoring. The study includes control performance monitoring and assessment as a type of process monitoring, and the degradation of control performance due to changes in process or disturbance characteristics. Among the reviewed methods, Principal Component Analysis (PCA) and Partial Least Squares (PLS) are the main methodologies employed. The main purpose is to characterize normal variations and detect abnormal changes [22].

PCA is a statistical method based on singular value decomposition (SVD), as described in Equation 4, in which X represents the data matrix, T is a score matrix (principal components, PC), P is a loading matrix (the contribution of each variable to each PC), and $\tilde{X}$ is the residual matrix.

$$X=T{P}^T+\tilde{X}$$

(4)

PLS, as well as PCA, can be used for fault detection and diagnosis. Unlike PCA, PLS is a supervised methodology to define whether the system is under normal conditions. It takes into consideration not just the process variables X, but also the process output Y, which might be quality variables. PLS is modelled according to Equation 5, where l is the number of latent variables, $T=[t_1,...,t_l]$ are the latent score vectors, $P=[p_1,...,p_l]$ and $Q=[q_1,...,q_l]$ are the loadings, while E and F are the residuals for X and Y, respectively.

$$\left\{\begin{array}{cc}\hfill & X={\displaystyle {\sum }_{i=1}^l}{t}_i{p}_i^T+E=T{P}^T+E\hfill \\ \hfill & Y={\displaystyle {\sum }_{i=1}^l}{t}_i{q}_i^T+F=T{Q}^T+F\hfill \end{array}\right.$$

(5)

For the detecting procedure, Hotelling’s $T^2$ statistic and the Q-statistic, also known as the squared prediction error (SPE), can be used. Hotelling’s $T^2$ statistic is used to measure the variation inside the model and to detect unusual and statistically unlikely events in the dataset. It is calculated according to Equation 6, where x represents the sample vector from X and $\Lambda$ is a diagonal matrix with the eigenvalues of the covariance matrix of the data. The detection procedure is performed through the definition of a confidence limit (usually 95% or 99%). Faulty data samples, according to Hotelling’s $T^2$ statistic, indicate that the correlation structure is preserved, but the variables’ values have drifted too far from the average.

$$T^2=x^{T}P{\Lambda }^{-1}{P}^{T}x={\displaystyle \sum _{i=1}^k}{\displaystyle \frac{t_i^2}{{\lambda }_i}}$$

(6)

SPE, on the other hand, deals with the variation in the residual subspace. It is calculated according to Equation 7, where $\widehat{x}$ is the fitted data in the model. Since SPE relies on residuals, it evaluates how well the data is fitted in the model. Just like Hotelling’s $T^2$ statistic, SPE also relies on a confidence limit for detection. Due to the complementary nature of these two indices (SPE and Hotelling’s $T^2$), combined indices are recommended for fault detection and diagnosis [21].

$$SPE=\left|\right|x-\widehat{x}{\left|\right|}^2=\left|\right|(I-P{P}^T)x{\left|\right|}^2$$

(7)

Once a fault has been detected, the diagnosis is possible through the contribution plots. As Qin [21] has mentioned, those techniques can also be used for performance monitoring of controllers. ZHANG and LI [23] already used $T^2$ statistics and PCA for an MPC monitoring. However, diagnosing the process variables that lead to the quality problems remains a difficult task, since both PCA and PLS have limitations. Both models have the assumption of a linear relationship between variables, which does not represent the reality in many chemical processes. They are also static, ignoring autocorrelation in lagged data. In addition, there is a lack of causality during the diagnosis procedure, due to the high correlation between variables, and the inability to give a precise diagnosis, since different faults may lead to the same contribution plot [21].

2.3. Methodologies for Identification and Diagnosis

Although the methodologies described so far help determine whether the control system is healthy or underperforming, they usually do not provide the root cause of performance degradation. For this reason, some metrics were developed to provide a diagnosis of the process conditions. The main issue with diagnosis is the lack of a unique characteristic that determines whether the control system is under unmeasured and unpredictable disturbance, with a process fault, an instrumentation fault, a plant-model mismatch (PMM), or inappropriate tuning settings.

Even though instrumentation problems also cause MPC degradation, the key challenge in this type of controller is identifying issues in its internal configuration. The model is the crucial aspect of a predictive controller and affects its prediction accuracy. Due to the sake of computational efficiency, linear models are preferred in industrial applications, even though chemical processes are inherently nonlinear. However, changes in operational regions or process dynamics may lead to loss of model representability and, consequently, to degradation of MPC performance.

Once the model is degraded, it is required to update the model by performing a plant re-identification. However, this procedure requires intrusive plant tests, which have economic repercussions. Therefore, it is desired to identify only the part or the subsystem of the plant where a significant mismatch occurs. In addition, it is preferred to perform the re-identification when no other procedure is possible to improve MPC performance, since other solutions may limit the intervention required in the process. The most relevant issue in model assessment is determining whether the bad performance comes from a PMM or an unmeasured disturbance. Both cause similar effects in the process output. Nevertheless, the corrective action for an unmeasured disturbance might be less invasive than for a PMM. Thus, methodologies focused on identifying PMM were developed.

Badwe et al. [24] propose a methodology based on the analysis of partial correlations between the model residuals and manipulated variables for the detection and isolation of PMM. The main purpose of this approach is to evaluate the correlation between the manipulated variables and the prediction error, isolating the effect of the closed-loop. In an open-loop scenario, the manipulated variables (MVs) are not affected by unmeasured disturbances, so they should not be correlated. A correlation between them when closed-loop effects are removed means that the prediction error is caused by the MVs, and, consequently, there is a PMM. Equation 8 shows the relationship between the prediction error e and the uncorrelated manipulated variable u, where y and $\widehat{y}$ are the plant and the model controlled variables, respectively, while $\Delta u\left(k\right)$ and v are the PMM and the Gaussian disturbances acting on the process.

$$e\left(k\right)=y\left(k\right)-\widehat{y}\left(k\right)=\Delta u\left(k\right)+v\left(k\right)$$

(8)

However, even though the Badwe et al. [24] methodology uses only closed-loop operating data for the analysis and does not require intrusive tests on the plant, the system data must be excited. In the absence of target changes, it is not possible to determine the extent of mismatch, which is the main limitation of this methodology.

Botelho et al. [25] developed another methodology to distinguish PMM from unmeasured disturbances. It is a statistical approach based on a nominal output estimation. Equation 9 represents how the nominal controlled variable $y_0$ is estimated from the controlled variable y, where $S_0$ is the nominal output sensitivity transfer matrix and $y_{sim}$ is the simulated controlled variable of the nominal model.

$$y_0=S_0(y_{sim}-y)+y$$

(9)

The main purpose of Equation 9 is also to decouple PMM from controller configuration, since the effect of the PMM in the controlled system depends on the controller tuning. The relation between the nominal outputs and the nominal prediction errors $e_0$ is then evaluated through the Pearson correlation between their statistical distribution coefficients Z in a moving window, according to Equation 10. The statistical distribution coefficients considered are the skewness and kurtosis coefficients.

$$c{o}_Z={\displaystyle \frac{cov(Z_{e_0},Z_{y_0})}{\sqrt{var\left(Z_{e_0}\right)\times var\left(Z_{y_0}\right)}}}$$

(10)

A high correlation between both variables indicates the presence of PMM. Otherwise, unmeasured disturbances are the cause of prediction errors. The main challenge in implementing this methodology is its dependency on the nominal output sensitivity transfer matrix $S_0$, which industrial MPCs do not give explicitly.

Giraldo et al. [26] have made an analysis of the Botelho et al. [25] and the Badwe et al. [24] methodologies. The critic relies on the limitations of those methods for real-time applications, given their relatively high computational cost. Furthermore, the assumptions underlying those methods, which rely on the feasibility of achieving an ideal model, limit their applicability in practical scenarios. Giraldo et al. [26] have proposed an algorithm to distinguish PMM from unmeasured disturbances, trying to mitigate the limitations of the previous methods. However, the method focused on the Filtered Smith Predictor and is also based on the sensitivity transfer function $S_0$. Moreover, the methodologies for identifying PMM are focused on distinguishing it from an unmeasured disturbance, requiring other methods to discard other sources of performance loss.

2.4. Industrial Methodologies

Even though the methods discussed so far are very important, since they give evidence of the possibility of detecting and diagnosing poor performance in MPC, they are still not being used in industries. In this section, the methods that Aspen Technology, an industrial software company, implements to monitor advanced controllers are explored.

Aspen Watch Performance Monitor is Aspen Tech’s software for monitoring control loops. Its metrics are grouped into libraries. In version 14.2, the Basic library has 57 metrics, while the Extended library has 86. There are also a library focused on crude oil distillation and a library for quality analysis.

The Extended library has three main groups of metrics: the General’s, the Manipulated Variables’, and the Controlled Variables’ metrics. The General’s metrics evaluate the control system as a whole. This group includes metrics such as the percentage of time the control is on, the percentage of active MVs and CVs, and the percentage of CV constraint violations. The groups of MV and CV metrics include metrics about target deviation, the variable’s standard deviation, constraint violations, oscillation indices, the percentage each variable is on, prediction error, which is the difference between the model prediction and the variable’s measurement, and so on.

Many of the metrics implemented have a literature background. Regarding the variable oscillation indication, Aspen documentation does not supply this metric equation, only the information that the more oscillating the variable is, the closer the metric is to 1. In the literature, some metrics have been developed to detect oscillation in closed-loop systems. Thornhill and Hägglund [27] have proposed an oscillation detector based on the IAE of the system, and it is also in a range $[0,1]$. Thus, it is likely that Aspen Watch follows a similar approach. The variable’s standard deviation, on the other hand, especially regarding the dependent variable, is aligned with the Harris Index. Even though the software does not use the Harris Index directly to monitor the control-loop, nor use the MVC theory as a benchmark, it does use the variability of the variables as metrics to be evaluated.

A single metric rarely provides a precise diagnosis. Consequently, Aspen Watch has an extensive range of metrics to draw information from the process data. While these metrics offer clues regarding the potential causes of diverging values, the number of possibilities they present means they do not eliminate the need for a deep investigation into the control loop.

2.5. Machine Learning Methodologies

Many researchers have already applied ML algorithms in the chemical engineering context, not only on abnormality detection, but also on signal processing and process modeling [11]. Most applications in the chemical engineering context already focus on modeling, optimization, control, and monitoring [28]. As mentioned before, control performance monitoring may also be part of the FDD spectrum. Thus, ML techniques already have a solid base in the academic context. As more of this kind of methodology is studied, it becomes more reliable and feasible for use in industrial applications.

Early detection and diagnosis of process faults while the plant is still operating in a controllable region can help avoid abnormal event progression and reduce productivity loss. Venkatasubramanian et al. [12,29,30], in their review about FDD, already considered neural networks as a non-statistical method for feature extraction.

One of the conclusions of the survey conducted by Bauer et al. [3] is that, for control-loop monitoring, the simpler the method, the more respondents find it useful. The respondents also asked for better guidance on corrective actions to be taken. de Campos et al. [31] affirms that classic metrics used by Petrobras are not good to diagnose the root cause of the poor performance. It is necessary to use multiple techniques combined, together with a specialized team, to evaluate and correct control performance issues, and still, there are a lot of challenges to evaluate and diagnose effectively the multivariable model predictive controller performance.

Under this scenario, ML techniques have a great potential to diagnose control performance. As it was shown in Section 2.1, research about ML has become popular in the last years. Different methods and models have been developed to address different types of applications, especially in chemical engineering [11,28,32]. This evidence shows that ML models have great potential and are already under development in the PSE area. However, the use of ML tools restrictively for control-loops performance assessment is still under-evaluated. As shown in Section 2.1, only few studies have been conducted to apply the use of ML tools in control performance monitoring, despite its potential. By using the classification task approach, it is possible to assess and diagnose control performance at once, instead of using combined methodologies such as the ones shown in the previous sections.

Most studies on the use of ML techniques to monitor controller performance have considered classic and SISO controllers. Pillay and Govender [33] analyzed the use of a Multi-Class Support Vector Machine (SVM) classifier to detect and diagnose tuning issues in a proportional-integral (PI) controller. Grelewicz et al. [34,35] focused on developing an ML model that can distinguish between acceptable and poor PID performance, independent of the process, and without any additional learning stage.

In studies aimed at MPC, the key object of this study, neural networks have been used to identify the performance index [36], based on the Minimum Variance Control benchmark. The approach was to estimate the minimum variance of the process for the Harris index. The models evaluated were a radial basis function network and a Laguerre network instead of using the ARMA model proposed by the original methodology [8]. Even though this approach showed a great capacity to estimate the Harris Index of closed-loops, it did not explore the NN capacity to identify the degradation of the MPC performance.

Wang et al. [37] also used neural networks, but directly as a classification problem. The approach was to identify MPC performance degradation due to noise variance change, model mismatch, control variables constraint saturation, and manipulated variables constraint saturation. The process evaluated was a two-tank liquid level, and the neural network used was an MLP, being compared to a SVM model. The inputs used were the manipulated and controlled variables, the control effort, and the control errors. Xu et al. [38] evaluated the use of SVM as well, but with a Mahalanobis distance performance index as input. The proposal was to eliminate the correlation between process variables before the classification procedure. The evaluated systems were the Wood-Berry distillation column process and the two-tank liquid level process, with the same classification task performed by Wang et al. [37]. Both studies explored the capacity of MLP to identify the loss of performance of an MPC, but the classes used, especially regarding constraint saturation, could be easily extracted from process data, without the need for a ML model. Furthermore, the classes explored do not give a precise diagnosis of the closed-loop condition, except for the model mismatch.

Loquasto and Seborg [39] conducted a study that might be one of the most relevant in this thematic. They also employed the Wood-Berry process as a case study of a methodology using neural networks to monitor the performance of MPC systems. In this approach, MLP independent classifiers were used in cascade to detect and diagnose the MPC condition. The first MLP is responsible for classifying whether the process is in a normal or abnormal condition. If an abnormality is detected, the second MLP is responsible for detecting the presence of an abnormal disturbance. Finally, the third model is used for diagnosing PMM. The single four-class classifier (normal, disturbance, plant change, and both disturbance and plant change) methodology was also evaluated. Once a model-plant mismatch is detected, another MLP classifier is used to diagnose the specific sub-models that are inaccurate, also considering binary classifications. The MLP inputs are the manipulated and controlled variables, the control effort, the control errors, and the one-step prediction errors. The results presented showed that, in general, all models performed well. The disadvantage of using the multi-class classification is that each sample can belong to only one of the classes determined, while the multi-binary models can be arranged to allow the classification of two problems simultaneously. This study explored different arrangements of ML models and diagnoses focused on PMM. However, they used only one type of ML model and did not explore the possibility of the abnormality being from process instrumentation, which may cause similar effects in the closed-loop, or from the MPC tuning, which might require retuning, especially after operational changes. Furthermore, the process studied - the Wood-Berry process - is linear and has relatively few variables.

Some applications are more specific to each MPC degradation reason. Lu et al. [40] have focused their study only on PMM. The ML tool evaluated was the SVM to discriminate PMM from noise model change. The problem was a binary classification, and the case study was a SISO system. The classification is based on the Finite Impulse Response (FIR) coefficients, which requires an identification procedure before the classification task.

Dambros et al. [41], on the other hand, focused on oscillation detection. The ML algorithm evaluated was a deep feedforward network, and data in the frequency domain was used as input. The ML tool was trained and validated with artificial data before being tested with industrial data. Rabba et al. [42] also proposed the use of an ML algorithm to detect oscillation in a closed-loop process. The ML model used was a XGBoost, and the process was the Tennessee Eastman. A previous step of feature extraction is performed to select the features with a high degree of importance and correlation to be used as inputs. Still focusing on oscillation detection, Akavalappil et al. [43] proposed the use of a CNN using the process variables (PV) and the controller output (OP) signals as input.

The main contribution of all those studies together is their indication that ML tools have the potential to be applied to predictive controller performance monitoring. However, there is a lack of studies that evaluate whether ML can be applied to diagnose the closed-loop condition as a whole, considering different possibilities of the source of fault. Also, there is a lack of studies that expand the evaluation to nonlinear systems or explore their use in complex systems for more direct diagnosis, including process and controller source of degradation. Furthermore, other gaps include the lack of complex processes as case studies, the application of commercial controllers, and the evaluation of deep neural networks.

3. Methodology

3.1. Models in Cascade

Models in cascade have already been used for FDD issues. Vasudevan et al. [44] used a sequence of binary classification models for the detection of intrusion failure in the context of network attacks. Krishna and Kumar [45] employed Decision Tree, Naive Bayes, and KNN algorithms to detect and diagnose faults in a microgrid. Loquasto and Seborg [39] used independent neural networks to monitor and diagnose performance issues in MPC in the Wood-Berry process. However, this methodology has not been widely explored, especially for more complex processes and comparing different machine learning models. Unlike what was previously done, this study compared different ML models, such as Random Forest, MLP, and RNNs, in sequence to detect and diagnose the MPC closed-loop conditions. Also, expanding what was done by Loquasto and Seborg [39], more fault classes were used in order to identify how precise the diagnoses of ML models can be.

In this work, the general scheme presented in Figure 6 is adopted. The AI Monitoring System assumes a sequence of binary classification ML models, organized in cascade, named here Cascade Machine Learning Models (CMLM), used for detecting and diagnosing predictive controllers’ performance issues. This methodology is compared with a Single Machine Learning Model (SMLM) for a multiclass classification, in which detection and diagnosis occur with the same model. The multiclass approach is used in other studies for MPC status classification [37,38].

Figure 7 illustrates the general arrangement of the CMLM and its equivalent in the SMLM approach. In general, the CMLM was designed to discriminate between the control configuration problem and instrumentation and process problems. The first ML model is responsible for detecting whether the closed-loop performance is abnormal. Once abnormalities are detected, the process data goes to the second ML model, which is responsible for diagnosing the first type of fault (Fault 1). If Fault 1 is not detected, the data flows to the next ML model in order to identify the second type of fault (Fault 2). This logic continues until all possible faults are covered by the cascade. The specific adaptation of the CMLM from the general arrangement presented in the figure is based on the physical knowledge of the process and on the main typical source of degradation. Process particularities must be taken into consideration to design the cascade arrangement.

The main advantage of using CMLM is the structure based on process knowledge that mimics the decision-making logic of an operator. Moreover, this approach has the facility for maintenance and adaptations. Chemical processes with a large number of variables and controllers may suffer from a huge number of possible issues in the closed-loop. In case there is a necessity to update the diagnosis or include a new one, for the SMLM approach, the whole model should be retrained and rearranged, while the CMLM allows the possibility to update just one of the structures used or include a new model for a new diagnosis. Besides, CMLM allows the use of different ML algorithms combined, as one technique can be better for one specific diagnosis than others. Furthermore, the technical team can also contribute to the precise diagnosis, removing models from the cascade. For example, if it is certain that the process is not under normal conditions, the operators and engineers can use only the models regarding the diagnosis, skipping the detection model. Moreover, for each diagnosis, different features are expected to be affected. For the SMLM, even though it requires only one hyperparameter optimization and training procedure, all features that might contribute must be used, resulting in a complex ML model. For the CMLM, even though more models should be optimized and trained, different features can be used for each model in the cascade, allowing the construction of simpler models.

Firstly, simulations were run to generate a dataset used for training and testing the ML models. The presence of measurement noise in the dataset was considered. Secondly, ML models were trained following the SMLM approach and the CMLM approach to compare the best type of arrangements. Finally, each of those ML algorithms was compared for each class of diagnosis to elucidate the best ML algorithm for each class. The comparison was carried out by evaluating metric values and the implementation of those ML models.

Each ML model was trained with the same inputs:

• The process MVs, with lagged data with up to 2 time-units for non-recurrent models.
• The Integral Time-Weighted Absolute Error (ITAE), calculated considering the last 5 samples, of each CV.
• The prediction error of each CV.

CVs and MVs are considered to capture information regarding the closed-loop. Lagged MVs are used to capture process dynamics and improve ML predictions. ITAE is used to capture setpoint deviation persistent over time. The prediction error, on the other hand, is used to help ML models identify possible model mismatch. To evaluate the performance of each architecture, the van de Vusse reactor was used as a case study, due to its non-linearity. Details about the simulation are presented in Section 3.5.

The following sections elucidate the ML architectures built. All ML algorithms were implemented in Python through the libraries Scikit-learn and TensorFlow. For each ML algorithm, as part of the preprocessing data procedure, the dataset was scaled by using the StandardScaler function from Scikit-learn. The reason to use this scaler is due to its sensitivity to outliers, which may happen due to the presence of faults. Also, according to LeCun et al. [46], the gradients of back-propagation learning usually converge faster when inputs have a zero mean.

Two metrics were used to compare ML algorithms: Accuracy (Equation 11) and F1-score (Equation 12). For F1-score calculation, the macro average was taken into account, since all classes are equally important. To confirm the results obtained, training was executed 10 times, removing seeds from the split of data and parameters initialization. This execution aims to evaluate the model’s dependency on the data used for training and testing, and also the training initialization.

$$Accuracy={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(11)

$$\mathit{F}\mathit{1}-\mathit{Score}={\displaystyle \frac{2\times TP}{2\times TP+FP+FN}}$$

(12)

For each optimized architecture, to confirm the results obtained, the training procedure was executed 10 times, removing seeds from the split of data and parameters initialization, and the metrics were calculated for each one of those independent training runs. This execution aims to evaluate the model’s dependency on the data used for training and testing, and also the training initialization.

3.2. Random Forest

The algorithm was implemented in Python through the library Scikit-learn, using the function RandomForestClassifier. The dataset was split into training, validation, and test sets, using shuffle to choose samples randomly. The proportions chosen were 80% for training and 20% for testing. Optuna was used to optimize Random Forest hyperparameters [47]. The parameters considered for optimization for both case studies were:

• Number of estimators between 10 and 100
• Maximum depth between 5 and 15
• Maximum features among "None", "Sqrt", and "log2"
• Minimum samples at each leaf between 5 and 10
• Criterion among "gini", "entropy", and "log_loss"

The objective of the optimization was to maximize the mean accuracy of the cross-validation, using the stratified K-Fold with 5 folds. To turn the optimization less computationally expensive, MedianPruner was used to stop the trial if the best intermediate result is worse than the intermediate results of previous trials at the same step. A total of 100 trials were performed to find the best hyperparameters.

3.3. Multilayer Perceptron - MLP Architecture

For all MLPs, a batch size of 64 was considered, since it is an intermediate value for a batch size. A batch size too high leads to unstable learning, while a size too small increases learning time significantly. The dataset was split into training, validation, and test sets, using shuffle to choose samples randomly. The proportions chosen were 64% for training, 16% for validation, and 20% for testing.

The encoding used was One-Hot Encoding for multiclass classification, since, for this type of architecture and problem, it assures orthogonality among classes. For the CMLM, label encoding was used, as each model in the cascade corresponds to a binary classification problem. This choice is due to its simplicity when compared to One-Hot Encoding.

The architecture considered for MLP models used Optuna for hyperparameter optimization [47]. The objective of the optimization was to maximize the validation accuracy at the last epoch (considering a maximum of 50 epochs during optimization). MedianPruner was also used to reduce computational cost. The hyperparameters optimized and the boundaries used for research were:

• Number of layers between 2 and 4
• Number of neurons at each layer between 8 and 64
• The activation function among ReLU, LeakyReLU, and ELU
• Learning rate between ${10}^{-4}$ and ${10}^{-2}$
• Batch normalization between each layer: yes or no
• Dropout rate between 0.1 and 0.3

The batch size and the $L_2$ values were kept constant and equal to 64 and 0.0001, to avoid a too large search space for the optimization. The learning rate depends mainly on the batch size [48], as well as the $L_2$ regularization, which directly affects the weight values and, consequently, the number of parameters in the model during optimization.

3.4. Recurrent Neural Network

Since the process data is a time series, GRU was evaluated in diagnosing the controller’s conditions. A previous study was conducted to evaluate the three types of RNN models: Simple RNN, for being the simplest RNN algorithm; LSTM, for having superior performance for long memory; and GRU, for having a lower computational cost than LSTM. The study has shown that GRU has a good performance and a lower computational cost [49]. The dataset was split into training, validation, and test sets, using shuffle to choose samples randomly. The proportions chosen were 64% for training, 16% for validation, and 20% for testing.

The data length to be analyzed by GRU was half the settling time, which is 20 samples for the Van de Vusse reactor, equivalent to 3 minutes of simulation. Based on the window size defined, the data was reshaped into a 3D tensor. The features used were the same for MLP and RF: the manipulated variables, the ITAE from the last 5 points, and the prediction error of each controlled variable. Since this type of algorithm already captures process dynamics, lagged data was not considered.

For the Optuna optimization HyperbandPruner was used for pruning, since it is more efficient to reduce computational cost. The following hyperparameters were considered for optimization:

• Number of layers between 1 and 2
• Number of neurons at each layer between 8 and 64
• The activation function between ReLU, and tanh
• Learning rate between ${10}^{-5}$ and ${10}^{-2}$
• Layer normalization between each layer: yes or no
• Dropout rate, between 0.1 and 0.5

3.5. Case Study: van de Vusse Reactor Controlled by NMPC

The system under study is the van de Vusse reactor, presented in Figure 8, proposed firstly by van de Vusse [50], and has been used as a benchmark for nonlinear control problems. The reactor modeling and design have been developed throughout the years, especially by Kantor [51] and Engell and Klatt [52,53]. It is a reaction to produce cyclopentenol (B) from cyclopentadiene (A), with the generation of cyclopentanediol (C) and dicyclopentathiene (D) as byproducts. The reaction scheme is given by [50]:

$$\begin{array}{cc}\hfill & A\stackrel{K_1}{\to }B\stackrel{K_2}{\to }C\hfill \\ \hfill & 2A\stackrel{K_3}{\to }D\hfill \end{array}$$

(13)

The feed flow, with a concentration of $C_{Ain}$ of reagent A, gets into a CSTR reactor with a temperature $T_{in}$ and a flow rate of F. The control objective is to control the production of B ($C_B$) and, consequently, control the reaction temperature (T) by using the heat exchanger duty $\dot{Q}$ and the flow and reactor volume ratio ($u_1={\displaystyle \frac{F}{V}}$) as manipulated variables. The equations that describe the van de Vusse reactor are given by:

$$\begin{array}{cc}\hfill {\displaystyle \frac{d{C}_A}{dt}}=& {\displaystyle \frac{F}{V}}(C_{A,in}-C_A)-K_1\left(T\right)C_A-K_3\left(T\right)C_A^2\hfill \\ \hfill {\displaystyle \frac{d{C}_B}{dt}}=& {\displaystyle \frac{F}{V}}{C}_B+K_1\left(T\right)C_A-K_2\left(T\right)C_B\hfill \\ \hfill {\displaystyle \frac{d{T}_k}{dt}}=& {\displaystyle \frac{\dot{Q}+K_w{A}_r}{m_k{C}_{p,k}}}(T-T_k)\hfill \\ \hfill {\displaystyle \frac{dT}{dt}}=& {\displaystyle \frac{K_1\left(T\right)C_A(-\Delta H_{AB})+K_2\left(T\right)C_B(-\Delta H_{BC})+K_3\left(T\right)C_A^2(-\Delta H_{AD})}{\rho C_P}}+\hfill \\ \hfill & {\displaystyle \frac{F}{V}}(T_{in}-T)+{\displaystyle \frac{K_w{A}_r}{\rho C_{P}V}}(T_k-T)\hfill \end{array}$$

(14)

in which $T_k$, $m_k$, $C_{p,k}$ are the outlet temperature, the mass, and the heat capacity of the coolant; $A_r$ and $K_w$ are the heat exchange area and the heat transfer coefficient; $\rho$ and $C_P$ are the reaction mixture’s specific mass and heat capacity. $\Delta H_i$ and $K_i\left(T\right)$ are the molar enthalpy of reaction and the reaction rate constant of i. This last one depends on the reactor’s temperature according to Arrhenius Law (Equation 15):

$$K_i\left(T\right)=k_{0,i}exp\left(-{\displaystyle \frac{E_{A,i}}{R(T+273.15)}}\right)$$

(15)

The values for each parameter considered for the simulation were the same explored by Engell and Klatt [52]. The hypotheses considered for the process equation are:

• Perfect mixing;
• Constant volume;
• Constant heat capacity;
• Incompressible fluid;
• Elementary kinetics for all reactions;
• Reaction rate constants described by the Arrhenius Law;
• No inlet flow of B, C and D.

3.5.1. Control System

The reactor is controlled by a nonlinear MPC (NMPC) following the implementation of Lima et al. [54], with the presence of a Constrained Extended Kalman Filter (CEKF). All simulations were run considering the same steady state as the initial process condition. The simulations were carried out using the same structure adopted by Lima et al. [54], considering the measurement vector $z={[C_A,C_B,T,T_K]}^{⊺}$, the input vector $u={[F/V,\dot{Q}/\left(K_w{A}_r\right)]}^{⊺}$, $C_{A,in}$ as an unmeasured disturbance, and $y={[C_B,T]}^{⊺}$ as controlled variable. The upper and lower bounds of $\tilde{x}\left(k\right)={[C_A,C_B,T,T_K,C_{A,in}]}^{⊺}$ to compose the NMPC and the CEKF are given by Equation 16:

$$\left[\begin{array}{c}0\\ 0\\ 50\\ 50\\ 0.1\end{array}\right]\le \tilde{x}\left(k\right)\le \left[\begin{array}{c}4\\ 3\\ 200\\ 200\\ 6.1\end{array}\right]$$

(16)

The manipulation variability constraints assumed are given by Equation 17:

$$\left[\begin{array}{c}-50\\ -0.15\end{array}\right]\le \left[\begin{array}{c}\Delta (F/V)\\ \Delta (\dot{Q}/\left(K_w{A}_r\right))\end{array}\right]\le \left[\begin{array}{c}50\\ 0.15\end{array}\right]$$

(17)

A timestep of 2.5 × ${10}^{-3}$ h was employed. The weights and horizons considered for NMPC are given in Table 1. The initial conditions used for each scenario are given by Table 2.

The CEKF was implemented as a state estimator, for which the parameter values are presented in Equations 18-20. The CEKF estimates $\tilde{x}\left(k\right)$ together with $k_{0,1}$ and $C_P$ by considering the measurements of $\tilde{x}\left(k\right)$. Lima et al. [54] employed the CEKF to evaluate its performance under some scenarios, including PMM in the $k_{0,1}$ and in the $C_P$. In this work, the implementation was kept to filter out noise and estimate unmeasured states, as well as being able to estimate some parameters to update the model.

$$\mathrm{Initial}\mathrm{Covariance}=[{10}^3,{10}^3,{10}^3,{10}^3,{10}^3,{10}^3,{10}^3]$$

(18)

$$\mathrm{Model}\mathrm{Uncertainties}=[3\times {10}^{-6},5\times {10}^{-6},{10}^{-3},{10}^{-3},{10}^{-3},{10}^{-2},{10}^{-2}]$$

(19)

$$\mathrm{Measurement}\mathrm{Uncertainties}=[3\times {10}^{-2},5\times {10}^{-3},3\times {10}^{-1},8\times {10}^{-1}]$$

(20)

3.5.2. Scenarios Conditions

To evaluate the capability of ML tools to predict closed-loop issues, the following scenarios were simulated in the van de Vusse reactor: normal conditions, valve stiction, wrong tuning settings, and plant model mismatch.

The normal operating conditions of the process consist of data generated under typical regulatory control actuation, representing fault-free operation in which disturbances in $C_{A,in}$ happen. Disturbances in $C_{A,in}$ occur randomly every 0.2 hour (12 min) during the simulation, following a normal (Gaussian) distribution, as defined in Equation 21. A total of 10,000 samples were generated in this condition.

$$C_{A,in}=5.1\times [1+0.02\times \left(random(mean=0,st.dev.=1)\right)]$$

(21)

For a valve stiction simulation, a simple stiction model, employed by Hägglund [55] and presented in Equation 22, was considered. A dead band, presented by d, is the percentage of the valve operational range in which the valve does not change its position. $u_s$ corresponds to the $F/V$ value in the process, and u corresponds to the control effort found by the NMPC. In this model, the value of the inlet flow is stuck if the difference between the controller output and the variable value is within the dead band. When this difference is greater than the band d, the inlet flow slips to the value given by the controller. Table 3 shows the number of samples generated considering different dead band values. The total of samples under this condition is 10,000.

$$u_s\left(k\right)=\left\{\begin{array}{cc}{u}_s(k-1)\hfill & \mathrm{if}|u\left(k\right)-u_s(k-1)|<d\hfill \\ u\left(k\right)\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(22)

In the simulations of Incorrect Tuning Settings, the prediction and control horizons were reduced relative to those used in the normal scenario to intentionally degrade controller performance. Table 4 shows the different combinations used in the simulation to generate the dataset. For each combination, 1,000 samples were generated, to sum up a total of 10,000 samples under this condition.

For the scenario with PMM, constant changes in the parameters $C_P$ and the pre-exponential constant $k_{0,1}$ are considered. From Equation 14, it is possible to observe that $K_1$ has a direct effect on concentrations and temperatures, while $C_P$ seems to have an inverse effect on temperature and a more nonlinear effect on concentrations. Lima et al. [54] used a difference of 20% in the value of the specific heat of the mixture ($C_P$) and 5% in the value of the pre-exponential constant of $K_1$ ($k_{0,1}$) to evaluate how the CEKF handles PMM.

A previous analysis was made to evaluate the impact that different changes in those variables could have on the controller. The results showed that a model-mismatch with values up to 5% for $k_{0,1}$ and 20% for Cp are handled by the controller. Based on this analysis, different percentages of plant-model mismatches were used for each parameter individually. Percentages of 25%, 30%, 50%, 75%, 80%, 120%, 125%, 200%, 300%, and 400% of the $C_P$ nominal value were used. A similar approach was made to generate the dataset with plant-model mismatch in the parameter k$k_{0,1}$, in which the percentages of 50%, 60%, 70%, 90%, 95%, 105%, 110%, 150%, 175% and 200% of the k$k_{0,1}$ nominal value were used. For each condition, 1,000 samples were generated, to sum up a total of 10,000 samples under each parameter mismatch.

3.5.3. Monitoring Strategies: Proposed CMLM and SMLM Approaches

Figure 9 illustrates how the CMLM and the SMLM models are organized for the van de Vusse case study. The CMLM was designed to segregate the instrumentation and process problem from the control configuration problem. The first ML model is responsible for detecting whether the closed-loop performance is abnormal. Once abnormalities are detected, the process data goes to the second ML model, which is responsible for diagnosing whether the issue is due to the process actuator, such as valve stiction, or internal control configuration. Once an internal control configuration issue is detected, the third ML model is used for diagnosing poor control tuning or PMM. Finally, if a PMM is identified, the final model is responsible for diagnosing which of the two process model parameters should be evaluated.

3.6. Standard Methodology

One traditional methodology used in industry is to use the standard deviation of variables to capture variability changes. This analysis is based on the minimum variance control theory and the Harris Index [8] to monitor control performance. The evaluation of standard deviation changes over time was performed to explore the potential of ML models to detect and diagnose sources of performance degradation against traditional methods. For this analysis, the standard deviation time history of each MV ($F/V$ and $\dot{Q}$) and CV ($C_B$ and T) was plotted using a moving window of 100 samples (15 min of simulation).

4. Results and Discussion

The noisy data was generated to evaluate the ML models’ robustness under measurement uncertainty. The noise follows a Gaussian distribution and was implemented on the measurement of $C_B$ and T, with the percentages of the random values of 5% and 2%. The Gaussian distribution has a mean of 0 and a standard deviation of 1, computed as presented in Equation 23.

$$y_{mea{s}_{noise}}=y_{meas}\times [1+\mathrm{percentage}\times \left(random(mean=0,st.dev.=1)\right)]$$

(23)

For a fair evaluation of the ML models’ robustness, they were employed as if they were being implemented offline in a real process, a procedure called here the Realistic Implementation. It was executed by generating a new dataset in the same scenarios as the training and test datasets. However, since the disturbance and the noise are randomly applied to the process, this new set of data will evaluate the generalization capabilities of each ML model and arrangement.

4.1. Random Forest

Table 5 shows the results obtained for the test dataset for each model in the CMLM and for the SMLM. All models in the CMLM have better metric values than the SMLM. Model 1, responsible for detecting abnormalities, had the lowest performance among the models in CMLM.

The results from multiple training sessions (Table 6) show once again that the results are relatively stable and do not depend heavily on the split or the weights initialization. They also reinforce the conclusions about the performance of Random Forest for noisy data.

After the overall implementation, the normalized confusion matrices presented in Figure 10 are obtained. For the Random Forest technique, the CMLM approach seems to have a lower performance when compared to the single-model approach. The main low-performance model of the CMLM is the detection model (Model 1). It is noticeable that many samples are misclassified as under normal conditions, when actually they are under abnormal conditions. The main misleading class is the valve stiction. This might happen due to the use of Accuracy as an optimization function during ML model training. Since the use of models in CMLM creates an artificial imbalance in the dataset, especially in the first model, the high Accuracy might be due to the correct classification of the majority class.

4.2. Multilayer Perceptron

Table 7 shows the results obtained for MLP models. By comparing the CMLM models against the SMLM, Model 1 was the only model with a lower performance than the SMLM. It also has the lowest performance among all models in the CMLM, similar to what was observed by using Random Forest. This leads to the conclusion that MLP also struggles to detect abnormalities in the van de Vusse data with noise.

Table 8 shows the results after 10 independent training runs of each model. The highest variability relies on the SMLM. This reflects how stable the results are.

Figure 11 shows the results obtained for the MLPs. The diagnosis of normal data was the most affected one, with a lower performance for CMLM. In contrast, other diagnoses, especially with valve stiction, were mislabeled as normal for the SMLM. This suggests that, in this case, the SMLM approach yields false negatives in abnormality detection. The CMLM approach, on the other hand, seems to give more false positives, since some normal data was misclassified as abnormal. The valve stiction diagnosis was better for the CMLM, while the diagnosis of mismatch in the parameter $C_P$ was also slightly better for the CMLM approach. The diagnosis of a mismatch in the parameter $k_{0,1}$ was quite similar for both models.

4.3. Gated Recurrent Unit - GRU

Melo [49] have conducted an analysis, showing that the window size of 20 samples is enough to capture correlations in the dataset. Table 9 shows the results for the GRU models. Besides Model 2, responsible for diagnosing valve stiction, the remaining models have metric values higher than 0.95.

When the results of multiple independent training runs are evaluated (Table 10), the patterns observed remain the same. Model 4 of the cascade achieved 1.0000 for all 10 trainings. The remaining models also have stable results, especially Models 2 and 3.

Figure 12 shows the results for the overall implementation. It is noticeable that SMLM and CMLM have similar performance, except for valve stiction, where the SMLM approach seems higher. However, contrary to what was observed in RF and MLP algorithms, the valve stiction data were not misclassified as normal. The misclassification was a mix of other control performance problems, regarding tuning and model-plant mismatch. Since the recurrent models use a bigger window than the one given to MLP or RF, for this kind of model, the difference between normal and valve stiction classes is more evident. The underperformance of the CMLM might still be due to the imbalance of the dataset.

4.4. Implementation - Realistic

By executing the realistic implementation, the results obtained follow the pattern observed in the overall implementation: the CMLM approach seems to have a performance similar to the single-model approach, except for detecting whether the process is with abnormalities.

By comparing the SMLM with the CMLM approach, with results presented in Figure 13, considering first the RF model, the classification of normal was better for the CMLM, while the valve stiction and tuning issues classes were better for the SMLM approach. The model-plant mismatch diagnosis was slightly similar for both approaches. The same pattern of underperformance in the CMLM might be due to the imbalance in the dataset.

Evaluating the MLPs (Figure 14), the performance was similar for model-plant mismatch classes. The classification of valve stiction was better for the CMLM, while normal and tuning classifications were better for the SMLM approach. Again, the underperformance of the CMLM is due to Model 1, which is detecting the tuning-issues data incorrectly as normal, while normal data is being classified as abnormal. This might be happening due to the imbalance in the dataset.

Finally, considering the GRU algorithm, Figure 15 shows the results obtained. The first noticeable difference when compared to previous algorithms is that this type of ML model did not perform well for model-plant mismatch in $K_{0,1}$. As an advantage, GRU in CMLM was better for classifying data in normal conditions and with valve stiction, which was hardly done by the previous algorithms in CMLM. This suggests the possibility of using different ML algorithms for closed-loop diagnosis, once different models have different performance, depending on the issue.

The CMLM shows to be better at classifying model-plant mismatch in the $K_{0,1}$ than the SMLM, while normal and model-plant mismatch in the $C_P$ were quite similar. When compared to the other recurrent ML models in CMLM, this algorithm has better results for classifying mismatch in the $K_{0,1}$, even though the Accuracy is lower than that obtained from RF and MLP.

4.5. Standard Methodology

To evaluate the potential of CMLM against traditional methodologies, the evaluation of the standard deviation changes through time was performed. Figure 16 shows the standard deviation history for each MV and CV of the van de Vusse reactor. Visually, both the normal (blue area) and the valve stiction (green area) have the same range of standard deviation values, except for some samples equal to 0 for the $F/V$.

Poor NMPC tuning and mismatches in its internal model parameters cause the standard deviation range for some variables to be different than those observed in normal conditions, especially in the tuning changes. However, the standard deviation gives information regarding a specific variable and does not give evidence about the root cause of performance degradation. A more accurate diagnosis would require an exhaustive data investigation.

5. Conclusions

In this work, a review of the different methodologies available for predictive controllers’ performance monitoring is presented. Most of them rely on monitoring linear MPCs, and with assumptions that limit their application online in industry. Commercial controllers have their own metrics implemented to monitor the control system, with a lot of them based on the literature. However, they are inefficient in giving the root cause of degradation. An FDD methodology based on cascaded binary classification using machine learning models and incorporating predictive controller performance monitoring was proposed and investigated. The proposed methodology was compared to multiclass ML models for different types of ML models. To provide a solid validation case, the van de Vusse reactor, a classical and highly nonlinear benchmark system, was investigated. Although based on data-driven (ML-based) models, the methodology also incorporates knowledge-based reasoning through the proposed cascade architecture, which mimics the decision-making logic of an operator (i.e., “normal operation”, “instrumentation OK”, “process OK”, and “controller performance OK”). In this sense, the approach can be regarded as hybrid in nature, providing advantages such as more direct interpretability.

The main findings of this study include that there is no universal ML model that is better for controller performance monitoring. The best approach depends on the process and the evaluated diagnosis. Random Forest and MLP require smaller datasets for training and have relatively low computational cost. Random Forest is particularly effective when the diagnosis involves evaluating whether variables cross predefined thresholds. MLP, on the other hand, is well-suited to capturing nonlinear relationships between variables. However, when the diagnosis depends on a time-dependent characteristic, RNNs are better suited to make a diagnosis based on a window of data, even though they require a larger dataset for training. Models in cascade give the possibility of combining the best ML models for each case. Even though the SMLM approach outperformed the CMLM in many architectures, it limits the methodology to a single ML algorithm. As observed in the realistic implementation, the diagnosis of some root causes may be compromised by the SMLM approach. In the realistic implementation, it was possible to observe that the RNNs were better at distinguishing between normal conditions and valve stiction conditions, while the remaining diagnoses were best performed with RF and MLP.

For future work, it is recommended to further investigate the integrated framework using different ML models and more industrial applications as case studies, while assessing computational cost limitations in industrial applications. Additional efforts should consider multi-objective optimization (balancing accuracy and model complexity), cascade structures with heterogeneous ML techniques, and the combined use of single and cascade models.