When Error Metrics Contradict: Clarifying RMSE and MAE in Machine Learning Evaluation

1. Introduction and Motivation

Quantitative evaluation plays a central role in the development and comparison of machine learning and optimization algorithms. Performance metrics such as the root mean squared error (RMSE) and the mean absolute error (MAE) are routinely employed to assess predictive accuracy, guide hyperparameter optimization, and rank competing models across a wide range of regression-based applications. Although widely used, these metrics are not interchangeable and are governed by fundamental mathematical relationships that restrict their admissible values.

In a recent study published in the leading artificial intelligence journal Artificial Intelligence Review [1], benchmark results were reported in which RMSE values are numerically smaller than the corresponding MAE values across multiple datasets (Tables 4–8). Under the standard definitions of RMSE and MAE, such a relationship is mathematically impossible. Importantly, this error is not confined to a single article. The same pattern can be identified in a series of algorithmic studies by the same authors published over the past six years [2,3,4,5], which primarily develop hybrid machine-learning frameworks optimized by nature-inspired and swarm-intelligence-based metaheuristic algorithms and validate their performance through extensive numerical comparisons.

The recurrence of this issue across multiple publications is therefore a matter of technical concern. In these studies, RMSE and MAE are not merely auxiliary descriptors but function as primary evaluation criteria and, in some cases, as objective functions within the optimization process itself [5]. Consequently, violations of basic metric relationships call into question the internal consistency of the reported numerical results and, by extension, the reliability of subsequent comparisons, statistical analyses, and claims of algorithmic superiority derived from them.

This technical note focuses exclusively on the mathematical validity and theoretical relationship between RMSE and MAE. Because these metrics satisfy exact and easily verifiable inequalities, errors can be unambiguously identified without access to source code, raw data, or the need to execute complex algorithms. When such errors arise even at this fundamental level, they raise legitimate concerns regarding the numerical reliability and trustworthiness of other reported results in the same studies, particularly those derived from more complex optimization procedures or multi-stage learning frameworks that are not readily subject to simple analytical checks. Ensuring the correctness of basic performance metrics is therefore a necessary first step for establishing confidence in subsequent analyses and conclusions drawn from increasingly sophisticated machine learning algorithms.

2. Mathematical Properties of RMSE and MAE

Let ${\left\{y_i\right\}}_{i=1}^n$ denote observed values and ${\left\{{\widehat{y}}_i\right\}}_{i=1}^n$ corresponding model predictions. Define the prediction errors as

$$e_i=y_i-{\widehat{y}}_i.$$

The mean absolute error and root mean squared error are given by

$$\mathrm{MAE}={\displaystyle \frac{1}{n}}\sum _{i=1}^n\left|e_i\right|,\mathrm{RMSE}=\sqrt{{\displaystyle \frac{1}{n}}{\sum }_{i=1}^n{e}_i^2}.$$

(1)

These two metrics are related by the well-known inequality

$$\mathrm{MAE}\le \mathrm{RMSE}\le \sqrt{n}\mathrm{MAE},$$

(2)

which has been discussed extensively in the literature [6,7,8].

Under additional constraints, a tighter upper bound can be obtained. For least-squares fits, where the errors satisfy ${\sum }_{i=1}^n{e}_i=0$, Willmott et al. [8] showed that the root-mean-square error is bounded by

$$\mathrm{RMSE}\le \sqrt{{\displaystyle \frac{n}{2}}}\mathrm{MAE}.$$

(3)

This upper bound is attained when the total absolute error is concentrated in two countervailing errors of equal magnitude and opposite sign. Equation (3) therefore represents a tighter, but condition-dependent, bound that applies specifically to least-squares error structures and does not contradict the general inequality in Eq. (2).

2.1. Lower Bound

The lower bound in (2) follows from a standard inequality between the arithmetic mean and the quadratic mean applied to the absolute errors. Specifically, applying the Cauchy–Schwarz inequality to the vectors $\left(\right|e_1|,\dots ,|e_n\left|\right)$ and $(1,\dots ,1)$ yields

$${\left(\sum _{i=1}^n\left|e_i\right|\right)}^2\le \left(\sum _{i=1}^n{\left|e_i\right|}^2\right)\left(\sum _{i=1}^n{1}^2\right)=n\sum _{i=1}^n{e}_i^2.$$

Dividing both sides by $n^2$ gives

$${\left({\displaystyle \frac{1}{n}}\sum _{i=1}^n\left|e_i\right|\right)}^2\le {\displaystyle \frac{1}{n}}\sum _{i=1}^n{e}_i^2.$$

Taking square roots on both sides leads directly to

$$\mathrm{MAE}\le \mathrm{RMSE}.$$

Equality holds if and only if all absolute errors are identical, i.e., $|e_1|=\dots =|e_n|$.

2.2. Upper Bound

The upper bound reflects the fact that, for a fixed MAE, the sum of squared errors is maximized when the absolute error is maximally concentrated in a single observation. Let

$$S=\sum _{i=1}^n\left|e_i\right|=n\mathrm{MAE}.$$

Since $|e_i|\ge 0$, the quantity $\sum e_i^2$ is maximized under the constraint $\sum |e_i|=S$ when one error equals S and all remaining errors are zero. In this case,

$$\sum _{i=1}^n{e}_i^2\le S^2,$$

with equality attained if and only if $|e_j|=S$ for some j and $|e_i|=0$ for all $i\ne j$. Substituting this bound into the definition of RMSE yields

$$\mathrm{RMSE}=\sqrt{{\displaystyle \frac{1}{n}}{\sum }_{i=1}^n{e}_i^2}\le \sqrt{{\displaystyle \frac{1}{n}}{S}^2}=\sqrt{n}\mathrm{MAE}.$$

Together with the lower bound, Eq. (2) therefore provides inequalities that hold under the standard definitions of RMSE and MAE.

2.3. A Tighter Upper Bound Under Least-Squares Fit

For least-squares fits, the errors satisfy the additional constraint

$$\sum _{i=1}^n{e}_i=0,$$

(4)

which implies that the total positive and negative error magnitudes are equal. Let

$$S=\sum _{i=1}^n\left|e_i\right|=n\mathrm{MAE},S_{+}=\sum _{i:e_i>0}{e}_i,S_{-}=\sum _{i:e_i<0}(-e_i).$$

Then $S=S_{+}+S_{-}$ and (4) gives $S_{+}=S_{-}=S/2$. Since, for any nonnegative numbers with fixed sum T, the sum of squares is maximized when the mass is concentrated in a single entry, we have

$$\sum _{i:e_i>0}{e}_i^2\le S_{+}^2,\sum _{i:e_i<0}{e}_i^2\le S_{-}^2,$$

and therefore

$$\sum _{i=1}^n{e}_i^2=\sum _{i:e_i>0}{e}_i^2+\sum _{i:e_i<0}{e}_i^2\le S_{+}^2+S_{-}^2={\displaystyle \frac{S^2}{2}}.$$

Substituting into the definition of RMSE yields

$$\mathrm{RMSE}=\sqrt{{\displaystyle \frac{1}{n}}{\sum }_{i=1}^n{e}_i^2}\le \sqrt{{\displaystyle \frac{1}{n}}·{\displaystyle \frac{S^2}{2}}}=\sqrt{{\displaystyle \frac{n}{2}}}\mathrm{MAE},$$

which is the tighter upper bound reported by Willmott et al. [8]. Equality holds when the error mass is concentrated in two countervailing errors of equal magnitude and opposite sign, with all remaining errors equal to zero.

3. Identification of Metric Errors in Reported Results

In the benchmark results reported by [1], RMSE values are shown to be consistently smaller than MAE values across five test datasets (Tables 4–8). Under the inequality in Eq. (2), such a relationship cannot occur if RMSE and MAE are computed according to their standard/correct definitions.

Under the standard definitions of RMSE and MAE, the inequality $\mathrm{RMSE}\ge \mathrm{MAE}$ must always hold. The reported results therefore indicate that the values labeled as RMSE are numerically incorrect and are not consistent with the definition of root mean squared error. The issue is not one of model quality but of metric interpretation: when fundamental mathematical relationships are violated, reported performance measures lose their diagnostic meaning, and subsequent comparisons across models or algorithms become unreliable.

It is worth emphasizing that similar RMSE–MAE errors can also be identified in several other publications by the same authors over the past six years [2,3,4,5]. However, as these studies were published in different journals and lie outside the scope of the present submission, they are not examined in detail here. Their recurrence nonetheless underscores the broader importance of carefully verifying the correctness of basic evaluation metrics before drawing conclusions from more complex analyses.

4. Consistency with Literature Results

It is noteworthy that the other studies cited alongside the results in [1] correctly satisfy the inequality $\mathrm{RMSE}\ge \mathrm{MAE}$. This contrast demonstrates that the inequality $\mathrm{RMSE}\ge \mathrm{MAE}$ is consistently satisfied in the literature results when RMSE is computed according to its standard/correct definition, confirming that the reported violation cannot be attributed to dataset characteristics or benchmarking context. Indeed, under the standard definitions of RMSE and MAE, no dataset can intrinsically exhibit the property $\mathrm{RMSE}<\mathrm{MAE}$, as this would contradict a fundamental mathematical inequality. When viewed together with similar errors documented across multiple additional publications by the same authors, the evidence suggests that the issue is systematic rather than isolated and is related to the computation and reporting of RMSE and MAE. This observation further underscores the importance of verifying basic metric relationships as an integral component of sound benchmarking practice.

5. A Straightforward and Low-Complexity Baseline: Random Forest and XGBoost as Reference Models

Beyond metric consistency, model evaluation should also consider the principle of parsimony. Occam’s razor suggests that, all else being equal, simpler and more transparent models should be preferred over increasingly complex algorithmic frameworks, particularly when performance gains are marginal or inconsistent.

To examine this issue, we implemented two widely used and well-understood machine learning algorithms—Random Forest (RF) and Extreme Gradient Boosting (XGBoost)—as baseline reference models. Following the experimental protocol described in [1], all datasets were partitioned using the same data split: 63% for training, 27% for validation, and 10% for testing. No metaheuristic optimization or problem-specific tuning strategies were employed beyond standard practices commonly adopted in applied machine learning.

Table 1 summarizes the comparative results between the SAPSO-based framework reported by Truong and Chou [1] and our RF and XGBoost implementations. For completeness, MAE, RMSE, and MAPE (Mean Absolute Percentage Error) are reported for each dataset.

The results in Table 1 show that, despite their conceptual simplicity, Random Forest and XGBoost achieve comparable or superior performance in three out of the five datasets when evaluated using standard error metrics. In particular, both baseline models yield lower MAE and MAPE values than those reported in [1] for Dataset 1 and Dataset 4, without relying on a multi-stage stacking architecture or a sophisticated metaheuristic optimization scheme. RMSE values reported in [1] are not used for direct comparison here because they violate the fundamental inequality $\mathrm{RMSE}\ge \mathrm{MAE}$ and therefore cannot be interpreted as valid root mean squared errors under the standard/correct definition.

These findings do not invalidate the methodological contributions of the SAPSO-based framework. However, they raise a substantive question regarding the necessity and practical benefit of introducing a highly complex optimization pipeline when simpler, well-established machine learning models already provide competitive performance under the same data partitioning strategy. From a benchmarking perspective, such comparisons underscore the importance of including strong yet low-complexity baseline models when assessing the claimed advantages of newly proposed optimization algorithms.

6. Implications for Benchmarking and Model Evaluation

The results presented in this technical note have several broader implications for benchmarking and model evaluation in machine learning research. First, they demonstrate that even widely used and ostensibly simple error metrics such as RMSE and MAE require careful verification. When fundamental mathematical relationships between evaluation metrics are violated, the resulting performance measures lose their interpretability, rendering subsequent model comparisons and optimization claims unreliable.

Second, the additional analyses using Random Forest and XGBoost highlight the importance of including strong yet low-complexity baseline models in benchmarking studies. Under the same data partitioning strategy, these conventional machine learning methods achieve comparable or superior performance in a majority of the examined datasets, despite their substantially lower algorithmic complexity and absence of sophisticated optimization schemes. This finding underscores the need to contextualize reported performance gains by demonstrating that they cannot be readily attained by simpler and well-established models.

Taken together, these observations emphasize that rigorous metric validation and transparent baseline comparisons are not optional refinements but essential components of reproducible and credible empirical research in artificial intelligence.

7. Concluding Remarks

This technical note revisited the mathematical relationship between RMSE and MAE, identified systematic numerical errors in reported RMSE values that violate fundamental inequalities, and demonstrated the practical value of simple baseline models through additional empirical analysis. The purpose is not to challenge the scientific objectives or methodological creativity of prior studies, but to highlight that reliable metric computation and meaningful baseline selection form the foundation of trustworthy model evaluation.

As machine learning and optimization methods continue to grow in complexity, ensuring clarity, mathematical correctness, and interpretability in performance assessment remains a shared responsibility of the research community. Attention to these fundamentals is crucial for sustaining cumulative progress and maintaining confidence in reported advances.

Not applicable. This study does not involve human participants or animals.

Not applicable.

The code used to generate the results in the comparison table using Random Forest and XGBoost is available from the author upon reasonable request.