A crucial aspect in developing machine learning algorithms (or any type of predictive models) is the comparison of different algorithmic candidates based on evaluation criteria that measure their accuracy and practical value in terms of successfully capturing the complexity of the underlying problem and generalizing in a wide range of real-world scenarios. In the domain of diabetes mellitus machine learning techniques are widely employed in the prediction of future values of glucose concentration in the blood in order to assist the patient in avoiding deviations from the normo-glycemic value range and the consequences of hyper- and hypoglycemia. In the relative literature there is an apparent lack of a uniformly adopted evaluation metric which could combine the clarity and direct comparative nature of statistical mathematical formulas dealing with prediction error residuals (e.g., the RMSE) with the clinical insights and the visual-qualitative approach of clinical evaluation tools (e.g., Clarke's EGA). Mean Adjusted Exponent (MADEX) error metric, proposed in this paper, attempts to address this need by providing a validation tool based on an easy to implement mathematical formula, that incorporates adjustable parameters of clinical significance.