Mean Adjusted Exponent (MADEX): A Novel Metric for the Evaluation of Blood Glucose Prediction Machine Learning Models

1. Introduction

Due to various reasons, Diabetes Mellitus (DM) worldwide prevalence has been on a constant and alarming rise during the past decades. This trend is expected to continue unhindered for the years to come. Recent data [1,2,3] show that almost one in ten adults (~537 million) are living with the disease, and this number is expected to increase to 643 and 783 million by the years 2030 and 2045, respectively. In addition, another 541 million adults are affected by Impaired Glucose Tolerance (IGT), being at high risk of developing diabetes in the following years.

The hypothesis that there is an excess risk link between diabetes and cancer is not new in the literature [4], for both Type 1 and Type 2 DM patients [5] and it remains an active field of scientific investigation [6]. Cancer is one of the most common mortality causes for Type 1 DM [7] and its death risk has been shown to be increased in the presence of DM comorbidity [8,9]. This highly probable link arguably adds grave importance to the enrichment of the arsenal available to tackle the challenges of DM prevention, timely diagnosis and management.

In response to this situation and mainly fueled by data and computational resources availability and the rising adoption of medical Internet-of-Things in the last years, there has been a momentous production in machine learning/deep learning research, aspiring to address several aspects of the disease (onset and adverse events prediction/prevention, blood glucose levels regulation, etc.) [10,11,12] .

Often overlooked, the common denominator of every relative research effort is the need for reliable metrics that facilitate the absolute and relative/comparative assessment of the models under consideration and play a crucial role in the algorithms' training-evaluation workflow. After all, the applicability and success of a trained model in real world conditions is directly analogous to the proper assessment of its accuracy during the development and validation phase.

In the relevant literature, one observes various combinations of mathematical/statistical and clinical evaluation criteria, and the lack of uniformity is apparent. This can be mainly attributed to the fact that widely used mathematical evaluation metrics, generic by nature, do not give due consideration to the clinical implications of the results. In contrast, clinical evaluation tools like error grid analyses and expert comparisons do not promote easy comparison between the examined methods.

2. State of the Art

Woldaregay et al. [13] conducted a comprehensive literature review regarding the machine learning modelling strategies/techniques and reported that the most commonly used prediction/modelling performance evaluation tools, accounting for 80% of the included studies were the Root Mean Squared Error (36%), the Clarke Error Grid Analysis (19%), the Correlation Coefficient (12%), the Temporal Gain (8%) and the Mean Absolute Error (5%) [13].

2.1. Mathematical/Statistical Tools

2.1.1. Root Mean Squared Error (RMSE)

The Root Mean Squared Error – RMSE (1.2) measures the average error performed by the model in predicting the outcome for an observation. Mathematically, the RMSE is the square root of the Mean Squared Error – MSE (Eq. 1.1), which is the average squared difference between the observed actual outcome values and the values predicted by the model. The lower the RMSE, the better the model.

$$MSE=\frac{1}{N}\sum _{i=1}^N{\left(y_i-\widehat{y}\right)}^2$$

(1.1)

$$\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}=\sqrt{\mathrm{M}\mathrm{S}\mathrm{E}}$$

(1.2)

2.1.2. Correlation Coefficient

The Correlation Coefficient (Pearson's r) measures the linear correlation (2) between the prediction ($\widehat{Y}$) and the actual values (Y) datasets (cov: covariation, σ: standard deviation).

$${\rho }_{Y,\widehat{Y}}=\frac{cov(Y,\widehat{Y})}{{\sigma }_Y{\sigma }_{\widehat{Y}}}$$

(2)

cov: covariation, σ: standard deviation

2.1.3. Mean Absolute Error

The Mean Absolute Error -MAE (3) is an arithmetic average of the absolute errors $\left|{\mathrm{e}}_{\mathrm{i}}\right|=\left|{\mathrm{y}}_{\mathrm{i}}-\widehat{y}\right|$ where $\widehat{\mathrm{y}}$ is the prediction and ${\mathrm{y}}_{\mathrm{i}}$ the true value.

$$\mathrm{M}\mathrm{A}\mathrm{E}=\frac{1}{\mathrm{N}}\sum _{\mathrm{i}=1}^{\mathrm{N}}{\left|{\mathrm{y}}_{\mathrm{i}}-\widehat{\mathrm{y}}\right|}^2$$

(3)

2.2. Clinical Tools

2.2.1. Clarke’s Error Grid Analysis (C-EGA)

The Clarke Error Grid Analysis (EGA) was developed in 1987 to quantify the clinical accuracy of patient estimates of their current blood glucose as compared to the blood glucose value obtained in their meter [14]. It was then used to quantify the clinical accuracy of blood glucose estimates generated by meters as compared to a reference value. A description of the EGA appeared in Diabetes Care in 1987. Eventually, the EGA became accepted as one of the “gold standards” for determining the accuracy of blood glucose meters.

The grid breaks down a scatterplot of a reference glucose meter and an evaluated glucose meter into five regions:

• Region A: points within 20% deviation from of the reference value.
• Region B: points outside 20% deviation but would not lead to inappropriate treatment.
• Region C: points leading to unnecessary treatment.
• Region D: points indicating a potentially dangerous failure to detect hypoglycemia or hyperglycemia.
• Region E: points that would confuse the treatment of hypoglycemia for hyperglycemia and vice versa.

2.2.2. Parkes’ Error Grid Analysis

The consensus error grid (also known as the Parkes error grid) was developed as a new tool for evaluating the accuracy of a blood glucose meter. In recent times, the consensus error grid has been used increasingly by blood glucose meter manufacturers in their clinical studies. It was published in August 2000 [15].

3. Methods

3.1. Derivation of the Metric Equation

Adding clinical significance to the mathematical representation of the error function can be achieved by applying different weights on the residuals based on the clinical importance of the error in different situations. To achieve this, we defined the desirable behaviour as follows:

• The penalizing power applied to the residuals should vary, getting bigger in proportion to the distance of the reference value from a certain point on the reference axis (the arithmetical center of the normal range). In this manner, the calculated error gets more prominent as the reference value moves away from the normal levels (euglycaemia).
• The penalizing factor should be specific and behave differently based on the direction of the error. More specifically it should penalize more heavily values that provide a distorted evaluation of the reference glycaemic status, that is high predicted values when the true value is in the hypoglycaemic range, as well as low values when the true value is in the hyperglycaemic range.

Based on these preconditions, finding a suitable mathematical function with such behaviour suggests a member of the “S” shaped function family [16]. After testing several candidate functions, using as criteria i) the mathematical behavior and ii) the convenience of factoring in relevant clinical parameters, we derived the following metric (4):

$$\begin{gathered} \mathrm{M}\mathrm{e}\mathrm{a}\mathrm{n}\mathrm{A}\mathrm{D}\mathrm{j}\mathrm{u}\mathrm{s}\mathrm{t}\mathrm{e}\mathrm{d}\mathrm{E}\mathrm{X}\mathrm{p}\mathrm{o}\mathrm{n}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{E}\mathrm{r}\mathrm{r}\mathrm{o}\mathrm{r}=\frac{1}{N}\sum _{i=1}^n{\left|\widehat{y_i}-y_i\right|}^{exp} \\ exp=2-\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left(\frac{{\mathrm{y}}_{\mathrm{i}}-\mathrm{a}}{\mathrm{b}}\right)\times \left(\frac{\widehat{{\mathrm{y}}_{\mathrm{i}}}-{\mathrm{y}}_{\mathrm{i}}}{\mathrm{c}}\right) \\ ,\mathrm{a}:\mathrm{ }\mathrm{c}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{r},\mathrm{ }\mathrm{b}:\mathrm{c}\mathrm{r}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{ }\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{g}\mathrm{e},\mathrm{ }\mathrm{c}:\mathrm{s}\mathrm{l}\mathrm{o}\mathrm{p}\mathrm{e} \end{gathered}$$

(4)

Figure 1 shows the behaviour of the equation in three characteristic circumstances. We observe that when the reference value lies in the normal range (e.g., 120 mg/dL) the error calculation for both sides of the predicted value difference is almost identical with the respective MSE value. However, in the abnormal ranges the asymmetry and the difference from the MSE calculation is apparent: “deceptive” predicted values (lower than reference in the hyperglycemic range and higher than reference in the hypoglycemic range) are highly penalized.

3.2. Proof-Of-Concept Marginal Test

Checking the usefulness of the proposed approach requires considering mathematically marginal but clinically plausible scenarios in which MADEX metric performs better than existing “golden standard” alternatives, in terms of assessing successfully both mathematically and clinically a set of measures or modelling predictions. Table 2 shows the scenarios used for our testing purposes. Both scenarios’ scores are identical in terms of the MSE and RMSE metrics, 4321.88 and 65.74 respectively. However, their clinical evaluation is critically different, as shown in Figure 2. All the values in scenario 1 fall within ranges A and B (considered not to lead to inappropriate treatment), whilst a considerable proportion of the second scenario’s values (3 out of 7, ~43%) falls into range D (potentially dangerous).

3.3. Preliminary Application on Real-World Data

In the context of author’s SP doctoral thesis [17] personalized blood glycose concentration prediction models were developed using real-world data from the OHIO-T1DM dataset [18], which contains data collected from six people with type 1 diabetes over eight weeks. All subjects were between 40 and 60 years old at the time of data collection. Two were men and four women.

For each combination of subject, time-window, and prediction horizon, a trial training takes place using a multitude of Regression Machine Learning algorithms and an ordered list is obtained on the axis of the performance dimension, as calculated based on a series of metrics. The PyCaret library [19], an open source library that facilitates machine learning workflows, manages the end-to-end process, significantly speeds up the experimental cycle, and ensures mistakes and/or omissions are avoided, was used to automate this process. In the table below (Table 3), there is a list of the Machine Learning algorithms, which are available in the model training and verification processes and were used in the experimental investigation and software development of the platform.

The main metric used for model assessment, to the extent that the desired estimate should take into account both the mathematical and clinical dimensions of the efficacy of the models under consideration, is that introduced in the context of this work, MADEX, and its square root (RMADEX).

4. Results

The values of the Proof-Of-Concept Test scenarios were explicitly chosen to depict a situation in which two models with the same performance based on the (R)MSE metrics could have very different clinical significance. Table 4 and Figure 3 show that in scenarios like this, the (R)MADEX metrics can capture the clinical context and penalize the model with the worse predictions clinical-wise accordingly.

As far as the application to real-world data is concerned, the following table (Table 5) gives an example of an “algorithm leaderboard” for one of the investigated patient cases:

The best and worst performing algorithms (LGBM and Dummy Regressors, respectively) in this example are the same for RMADEX and RMSE which is chosen as the “golden standard” comparison metric. However, there are evident differences in the ranking, as well as indicators of RMADEX’s clinical value. For example, the Linear Regression algorithm which is usually considered clinically underperforming, ranks 2^nd (25.6) in the RMSE list, while only 9^th (1860) in the RMADEX list.

5. Discussion

This approach, compared with the alternative of defining specific penalties to clinically defined regions of the reference-prediction value space, has apparent advantages, as follows:

• It is a continuous function assuring smooth value progression and avoiding stepping phenomena
• It is not dependent on clinical interpretations (a center of euglycemia is easier to agree upon)
• It is computationally more efficient compared to algorithms with conditional (if_then_else, switch case) arguments

One of MADEX's interesting features is that it could, in theory, apply to medical domains totally different from diabetes, as it could be parametrized to handle, instead of blood glucose concentration, any quantity (e.g., biomarkers or in-vitro biochemical tests) whose value lies in a continuum with a centre and a range of "normality".

The main limitation of this work is that choosing the scenario data conveniently to facilitate the proof-of-concept of the MADEX metric does not guarantee its comparable performance in a real-data setting; thus, extended experimentation with real-world data and application in actual machine learning research workflows is warranted in order for MADEX to become established as a valuable tool in the arsenal of researchers, data scientists and machine learning practitioners.

6. Conclusions

This paper proposes a mathematical formula (MADEX) that can be used to assess machine learning predictive models for the future concentration of glucose in a patient's blood. This formula combines numerical error/residual calculation with clinical context, a property which, to the best of our knowledge, has not yet been proposed in the relevant literature. This work holds promise for modelling and prediction efforts in the domain of diabetes, as well as other medical domains with quantities that share common characteristics with blood glucose concentration. However, extensive testing by application in real-world data is warranted if MADEX should be promoted from its current proof-of-concept status to an actual item in the practitioner's toolset.