On the Mathematical Relationship Between RMSE and NSE Across Evaluation Scenarios

1. Introduction

The evaluation of predictive models is a central concern in hydrology and related environmental sciences. Accurate assessment of model performance is essential for tasks such as streamflow forecasting, rainfall–runoff simulation, and soil erosion prediction, where decision-making often depends directly on the reliability of hydrological models. Among the many performance metrics that have been proposed, the Nash–Sutcliffe efficiency (NSE) [1] remains the most widely adopted in the hydrology community. Defined as a normalized measure of squared error relative to the variance of observations, the NSE provides a convenient and interpretable scale: a value of 1 represents a perfect model, 0 corresponds to predictions no better than the observed mean, and negative values indicate performance worse than using the observed mean. Nevertheless, reliance on a single metric such as the NSE inevitably raises questions regarding the completeness and robustness of model evaluation.

More broadly, choosing appropriate statistical criteria for evaluating watershed models that simulate streamflow and sediment transport is not trivial. Despite their widespread use, different performance metrics emphasize distinct aspects of model behavior, and no single statistic can fully characterize model accuracy. Recognizing this limitation, Moriasi et al. [2] proposed a set of standardized evaluation guidelines based on three complementary statistics: the NSE, the percent bias (PBIAS), and the ratio of the root mean square error to the standard deviation of observed data (RSR).

Within this broader framework of multi-metric evaluation, the NSE is closely related to another widely used error metric: the Root Mean Squared Error (RMSE). Both metrics are functions of the same underlying quantity, the sum of squared errors (SSE), but they differ in normalization. The RMSE reports error in the same units as the observations, whereas the NSE expresses model skill relative to observed variability. This duality leads to subtle but practically important differences in behavior when comparing models, especially under dataset expansion or distributional shifts. Understanding these relationships is critical for fair model comparison and for interpreting results across different studies.

Beyond hydrology [3,4,5,6,7,8,9,10,11], the role of the NSE is growing in the machine learning and deep learning literature [12,13,14,15,16,17,18], where it is increasingly used as a performance measure for models dealing with continuous target variables. Machine learning practitioners are drawn to the NSE for its ability to contextualize model error against a baseline predictor, complementing traditional metrics such as the mean squared error (MSE) and the mean absolute error (MAE). As machine learning techniques are increasingly applied to hydrological problems, careful scrutiny of the relationship between the NSE and the RMSE becomes even more important. Recently, a study of surface water velocity prediction demonstrated that a model can achieve a higher NSE while simultaneously exhibiting a higher RMSE compared to another model, highlighting an inconsistency between these two metrics [19,20].

In hydrology, numerous studies have critiqued the NSE and related metrics, emphasizing their limitations and sensitivity to data characteristics. Early discussions raised concerns about the interpretability and practical usefulness of the NSE as a measure of hydrological model performance [21], and subsequent analyses documented the widespread misuse of popular performance metrics, including the NSE [22]. Gupta et al. [23] presented the well-known decomposition of the mean squared error (MSE) into correlation, bias, and variability components, and cautioned that the NSE alone can be misleading if these components are not examined. Subsequent studies have shown that the dependence of the NSE on local variability undermines cross-site comparisons: Williams [24] argued that both the NSE and the Kling–Gupta Efficiency (KGE) are unsuitable metrics, whereas Melsen [25] reviewed the widespread yet problematic reliance on the NSE in hydrological practice. Onyutha [26] further demonstrated that efficiency criteria can shift rankings under changing variability, bias, or outliers, underscoring that the choice of metric itself introduces calibration uncertainty. Methodological refinements, such as the probabilistic estimators proposed by Lamontagne et al. [27], aim to improve the NSE and the KGE performance, yet their structural sensitivities remain. Related paradoxes have also been observed for other error measures, with Willmott and his colleagues [28,29] arguing in favor of the MAE and Chai and Draxler [30] showing that the RMSE is preferable when the error distribution is Gaussian.

Despite extensive discussion and widespread application of the NSE and the RMSE across hydrology, meteorology, and machine learning, existing studies have focused primarily on empirical comparisons, heuristic interpretations, or decompositions of error components. To the author’s knowledge, no prior work has provided a formal, case-by-case mathematical characterization of the conditions under which RMSE- and NSE-based model rankings must coincide or can provably diverge. From this perspective, the present contribution differs in that it is analytical rather than empirical and therefore addresses questions that are not resolved by previous experiments or case studies involving the NSE and the RMSE. Instead, the analysis proceeds directly from the definitions of the RMSE and the NSE to derive necessary and sufficient conditions governing their comparative behavior under different evaluation scenarios.

The present study directly addresses this gap by systematically analyzing three scenarios in which the RMSE and the NSE can exhibit different comparative behaviors between models. We show that while both metrics are monotonic with respect to the SSE on a common dataset, discrepancies may arise when models are evaluated on different data subsets or when additional variability is introduced. By deriving precise mathematical conditions for these cases, we clarify the circumstances under which the NSE and the RMSE rankings agree or diverge. Beyond their theoretical significance, these insights also support sustainable water resources management and climate adaptation strategies, aligning with the United Nations Sustainable Development Goals, particularly SDG 6 (Clean Water and Sanitation) and SDG 13 (Climate Action).

2. Materials and Methods

This section outlines the analytical framework used to examine the relationship between the RMSE and the NSE. We first describe the predictive models under consideration and how their prediction errors are represented. Next, we define the datasets and how they are expanded across cases, followed by the formal expressions of the RMSE and the NSE that form the basis of our derivations. Together, these elements provide the foundation for the comparative analysis presented in the Results section.

2.1. Models

Two generic predictive models (hydrological models or machine learning models) are considered and denoted as Model A and Model B. The internal structures of the models are not specified, as they are irrelevant to the present analysis, which focuses solely on their error behavior under different evaluation scenarios on validation or test datasets (commonly referred to as the validation dataset for hydrological models and the test dataset for machine learning models). In the case of hydrological models, Models A and B are assumed to have been calibrated using a calibration dataset, whereas in the case of machine learning or deep learning models, Models A and B are assumed to have been fitted using a training dataset. The predictions of Model A and Model B for sample i are denoted by ${\widehat{y}}_{A,i}$ and ${\widehat{y}}_{B,i}$, respectively, and may be written simply as ${\widehat{y}}_i$ when distinguishing between Models A and B is unnecessary, while the observed value is denoted by $y_i$.

2.2. Datasets

Let X denote the original dataset consisting of n observations,

$$X=\{y_1,y_2,\dots ,y_n\}.$$

A second dataset block of equal size n is introduced,

$$Z=\{y_{n+1},y_{n+2},\dots ,y_{2n}\},$$

and the combined dataset is denoted by

$$Y=X\cup Z.$$

The mean of X is denoted by ${\overline{y}}_X$, and the mean of the combined dataset Y is denoted by ${\overline{y}}_Y$.

2.3. Evaluation Metrics

The analysis considers two standard metrics of model performance: the RMSE and the NSE. For a dataset with N observations, these are defined as

$$\begin{array}{cc}\hfill \mathrm{RMSE}& =\sqrt{{\displaystyle \frac{1}{N}}\sum _{i=1}^N{(y_i-{\widehat{y}}_i)}^2},\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill \mathrm{NSE}& =1-{\displaystyle \frac{{\sum }_{i=1}^N{(y_i-{\widehat{y}}_i)}^2}{{\sum }_{i=1}^N{(y_i-\overline{y})}^2}},\hfill \end{array}$$

(2)

where ${\widehat{y}}_i$ denotes the model prediction and $\overline{y}$ is the mean of the observed data. Both metrics are functions of the SSE, but the RMSE reports error in the same units as the observations, whereas the NSE normalizes error relative to the total variance of the observations.

2.4. Evaluation Scenarios

Three scenarios were investigated to examine the comparative behavior of the RMSE and the NSE:

1.

Case I: Same Test Dataset Evaluation. Both models were evaluated on the test dataset X, which represented the original test data.

2.

Case II: Expanded Test Dataset Evaluation. Both models remained fixed and were re-evaluated on the expanded test dataset Y, which included all samples from X along with additional test data.

3.

Case III: Unequal Test Dataset Evaluation. Model A was evaluated on X, while Model B was evaluated on Y; neither model is re-calibrated or retrained, ensuring that differences arose solely from the test datasets rather than model re-estimation.

These three scenarios are designed to clarify the mathematical relationship between the RMSE and the NSE, showing conditions where their rankings were consistent and conditions where contradictions may arise.

3. Results

The analysis in this section follows the three evaluation scenarios introduced in Section 2.4. Case I considers both models evaluated on the same dataset, Case II examines re-evaluation on an expanded dataset, and Case III addresses the unequal evaluation scenario in which one model is tested on the original dataset while the other is tested on the expanded dataset. The results highlight the specific conditions under which the RMSE and the NSE yield consistent model rankings and those where discrepancies emerge.

3.1. Case I: Same Dataset Evaluation

Definition 1. 

Let $X={\left\{(y_i,{\widehat{y}}_{A,i},{\widehat{y}}_{B,i})\right\}}_{i=1}^n$ be the common test dataset, with mean ${\overline{y}}_X={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n{y}_i$ and total sum of squares

$${\mathrm{SST}}_X=\sum _{i=1}^n{(y_i-{\overline{y}}_X)}^2.$$

(3)

Define the models’ sums of squared errors (SSE) on X as

$${\mathrm{SSE}}_A=\sum _{i=1}^n{(y_i-{\widehat{y}}_{A,i})}^2,{\mathrm{SSE}}_B=\sum _{i=1}^n{(y_i-{\widehat{y}}_{B,i})}^2.$$

(4)

Then

$${\mathrm{RMSE}}_M=\sqrt{{\displaystyle \frac{{\mathrm{SSE}}_M}{n}}},{\mathrm{NSE}}_M=1-{\displaystyle \frac{{\mathrm{SSE}}_M}{{\mathrm{SST}}_X}},$$

(5)

for $M\in \{A,B\}$, with the usual caveat that ${\mathrm{SST}}_X>0$ (i.e., the observed data series is not constant) for the NSE to be defined.

Proposition 1. 

If ${\mathrm{RMSE}}_A<{\mathrm{RMSE}}_B$ on the same dataset X, then ${\mathrm{NSE}}_A>{\mathrm{NSE}}_B$.

Proof. 

Because the square root is strictly increasing and n is identical for both models,

$$\begin{array}{cc}\hfill {\mathrm{RMSE}}_A<{\mathrm{RMSE}}_B& ⟺\sqrt{\frac{{\mathrm{SSE}}_A}{n}}<\sqrt{\frac{{\mathrm{SSE}}_B}{n}}\hfill \\ \hfill & ⟺\frac{{\mathrm{SSE}}_A}{n}<\frac{{\mathrm{SSE}}_B}{n}\hfill \\ \hfill & ⟺{\mathrm{SSE}}_A<{\mathrm{SSE}}_B.\hfill \end{array}$$

Now consider the NSE difference:

$${\mathrm{NSE}}_A-{\mathrm{NSE}}_B=\left(1-\frac{{\mathrm{SSE}}_A}{{\mathrm{SST}}_X}\right)-\left(1-\frac{{\mathrm{SSE}}_B}{{\mathrm{SST}}_X}\right)=\frac{{\mathrm{SSE}}_B-{\mathrm{SSE}}_A}{{\mathrm{SST}}_X}.$$

(6)

Since ${\mathrm{SST}}_X>0$, the sign of ${\mathrm{NSE}}_A-{\mathrm{NSE}}_B$ matches the sign of ${\mathrm{SSE}}_B-{\mathrm{SSE}}_A$. From ${\mathrm{SSE}}_A<{\mathrm{SSE}}_B$ we obtain ${\mathrm{SSE}}_B-{\mathrm{SSE}}_A>0$, hence

$${\mathrm{NSE}}_A-{\mathrm{NSE}}_B=\frac{{\mathrm{SSE}}_B-{\mathrm{SSE}}_A}{{\mathrm{SST}}_X}>0⟹{\mathrm{NSE}}_A>{\mathrm{NSE}}_B.$$

This proves the claim.    □

Proposition 2 

(Equivalent linear relation). On a fixed dataset X, the NSE is a strictly decreasing linear function of the ${\mathrm{RMSE}}^2$.

Proof. 

Since $\mathrm{MSE}={\mathrm{RMSE}}^2=\mathrm{SSE}/n$, one can write

$$\mathrm{NSE}=1-{\displaystyle \frac{n{\mathrm{RMSE}}^2}{{\mathrm{SST}}_X}}.$$

(7)

Therefore,

$${\mathrm{RMSE}}_A<{\mathrm{RMSE}}_B⟺{\mathrm{RMSE}}_A^2<{\mathrm{RMSE}}_B^2⟺{\mathrm{NSE}}_A>{\mathrm{NSE}}_B.$$

Hence, Equation (7) demonstrates that the $\mathrm{NSE}$ is a strictly decreasing linear function of the ${\mathrm{RMSE}}^2$, with slope $-n/{\mathrm{SST}}_X$ and intercept 1.    □

Interpretation 1. 

On the same dataset, the RMSE and the NSE always produced consistent rankings: whichever model achieved a lower RMSE necessarily achieved a higher NSE. Ties occured only if ${\mathrm{SSE}}_A={\mathrm{SSE}}_B$, while the NSE becomes undefined if ${\mathrm{SST}}_X=0$. This confirms that for hydrological model evaluation under fixed datasets, the RMSE and the NSE cannot yield contradictory conclusions.

3.2. Case II: Expanded Dataset with Both Models Re-Evaluated

Definition 2. 

Let the original dataset X contain n observations, with Model A and Model B errors measured by

$${\mathrm{SSE}}_{A1}=\sum _{i=1}^n{(y_i-{\widehat{y}}_{A,i})}^2,{\mathrm{SSE}}_{B1}=\sum _{i=1}^n{(y_i-{\widehat{y}}_{B,i})}^2.$$

(8)

Assume that ${\mathrm{SSE}}_{A1}<{\mathrm{SSE}}_{B1}$, so Model A initially has a lower RMSE and a higher NSE. Define the deficit of Model B as

$${\mathrm{\Delta }}_1={\mathrm{SSE}}_{B1}-{\mathrm{SSE}}_{A1}>0.$$

(9)

A new block of n observations, denoted Z, is added, producing the expanded dataset $Y=X\cup Z$ with $2n$ points. Model errors on Z are

$${\mathrm{SSE}}_{A2}=\sum _{i=n+1}^{2n}{(y_i-{\widehat{y}}_{A,i})}^2,{\mathrm{SSE}}_{B2}=\sum _{i=n+1}^{2n}{(y_i-{\widehat{y}}_{B,i})}^2.$$

(10)

Thus, on Y the total squared errors are

$${\mathrm{SSE}}_A={\mathrm{SSE}}_{A1}+{\mathrm{SSE}}_{A2},{\mathrm{SSE}}_B={\mathrm{SSE}}_{B1}+{\mathrm{SSE}}_{B2}.$$

(11)

Proposition 3 

(Consistency on expanded dataset). When both models are evaluated on the expanded dataset Y, the RMSE and the NSE always produce consistent rankings:

$${\mathrm{RMSE}}_A<{\mathrm{RMSE}}_B⟺{\mathrm{NSE}}_A>{\mathrm{NSE}}_B.$$

(12)

Proof. 

Both the RMSE and the NSE are monotonic functions of the $\mathrm{SSE}$ when evaluated on the same dataset. Therefore, comparing the ${\mathrm{RMSE}}_A$ and the ${\mathrm{RMSE}}_B$ is equivalent to comparing the ${\mathrm{SSE}}_A$ and the ${\mathrm{SSE}}_B$, which in turn determines the sign of ${\mathrm{NSE}}_A-{\mathrm{NSE}}_B$. Hence, contradictory outcomes such as ${\mathrm{RMSE}}_A<{\mathrm{RMSE}}_B$ together with ${\mathrm{NSE}}_A<{\mathrm{NSE}}_B$ are impossible.    □

Proposition 4 

(Condition for reversal). For Model B to surpass Model A on Y, the requirement is

$${\mathrm{SSE}}_B<{\mathrm{SSE}}_A.$$

(13)

Equivalently,

$${\mathrm{SSE}}_{B1}+{\mathrm{SSE}}_{B2}<{\mathrm{SSE}}_{A1}+{\mathrm{SSE}}_{A2}⟺{\mathrm{SSE}}_{A2}-{\mathrm{SSE}}_{B2}>{\mathrm{\Delta }}_1.$$

(14)

Proof. 

Substituting the blockwise decompositions ${\mathrm{SSE}}_A={\mathrm{SSE}}_{A1}+{\mathrm{SSE}}_{A2}$ and ${\mathrm{SSE}}_B={\mathrm{SSE}}_{B1}+{\mathrm{SSE}}_{B2}$, the inequality ${\mathrm{SSE}}_B<{\mathrm{SSE}}_A$ rearranges to (14), where ${\mathrm{\Delta }}_1={\mathrm{SSE}}_{B1}-{\mathrm{SSE}}_{A1}$. This condition means that Model B must outperform Model A on the added block Z by more than its initial deficit on X.    □

Interpretation 2. 

On the expanded dataset Y, the RMSE and the NSE remained consistent in their ranking of models. Model B could only overtake Model A if its relative improvement on Z outweighed its earlier disadvantage ${\mathrm{\Delta }}_1$ from X. Otherwise, Model A continues to be superior on both metrics. Thus, when both models were assessed on the same expanded dataset, the RMSE and the NSE could not yield conflicting conclusions.

3.3. Case III: Unequal Dataset Evaluation

Definition 3. 

Let the original dataset X contain n observations, with sums of squared errors ${\mathrm{SSE}}_{A1}$ and ${\mathrm{SSE}}_{B1}$ for Model A and Model B, respectively. Assume ${\mathrm{SSE}}_{A1}<{\mathrm{SSE}}_{B1}$ so that Model A initially outperforms Model B. Model A is evaluated only on X, while Model B is evaluated on the expanded dataset $Y=X\cup Z$, where Z is a new block of n observations. For Model A,

$${\mathrm{RMSE}}_A=\sqrt{\frac{{\mathrm{SSE}}_{A1}}{n}},{\mathrm{NSE}}_A=1-\frac{{\mathrm{SSE}}_{A1}}{{\mathrm{SST}}_X},{\mathrm{SST}}_X=\sum _{i=1}^n{(y_i-{\overline{y}}_X)}^2,$$

(15)

while for Model B on Y,

$${\mathrm{SSE}}_B={\mathrm{SSE}}_{B1}+{\mathrm{SSE}}_{B2},{\mathrm{RMSE}}_B^{\left(Y\right)}=\sqrt{\frac{{\mathrm{SSE}}_{B1}+{\mathrm{SSE}}_{B2}}{2n}},{\mathrm{NSE}}_B^{\left(Y\right)}=1-\frac{{\mathrm{SSE}}_{B1}+{\mathrm{SSE}}_{B2}}{{\mathrm{SST}}_Y},$$

(16)

where ${\mathrm{SST}}_Y$ is the total sum of squares of the combined dataset.

Proposition 5 

(RMSE condition). For Model B to have a larger RMSE on Y than Model A on X, it is necessary and sufficient that

$${\mathrm{SSE}}_{B1}+{\mathrm{SSE}}_{B2}>2{\mathrm{SSE}}_{A1}.$$

(17)

Proof. 

The inequality ${\mathrm{RMSE}}_B^{\left(Y\right)}>{\mathrm{RMSE}}_A$ expands to $\sqrt{\frac{{\mathrm{SSE}}_{B1}+{\mathrm{SSE}}_{B2}}{2n}}>\sqrt{\frac{{\mathrm{SSE}}_{A1}}{n}}$. Squaring both sides and multiplying through by $2n$ gives the stated condition.    □

Proposition 6 

(NSE condition). For Model B to have a larger NSE than Model A, it is necessary and sufficient that

$${\mathrm{SSE}}_{B1}+{\mathrm{SSE}}_{B2}<{\displaystyle \frac{{\mathrm{SST}}_Y}{{\mathrm{SST}}_X}}{\mathrm{SSE}}_{A1}.$$

(18)

Proof. 

The inequality ${\mathrm{NSE}}_B^{\left(Y\right)}>{\mathrm{NSE}}_A$ expands to $1-\frac{{\mathrm{SSE}}_{B1}+{\mathrm{SSE}}_{B2}}{{\mathrm{SST}}_Y}>1-\frac{{\mathrm{SSE}}_{A1}}{{\mathrm{SST}}_X}$. Rearranging yields the stated condition.    □

Proposition 7 

(Combined conditions). Both conditions hold simultaneously if and only if

$$2{\mathrm{SSE}}_{A1}<{\mathrm{SSE}}_{B1}+{\mathrm{SSE}}_{B2}<{\displaystyle \frac{{\mathrm{SST}}_Y}{{\mathrm{SST}}_X}}{\mathrm{SSE}}_{A1}.$$

(19)

Equivalently, the feasible interval for ${\mathrm{SSE}}_{B2}$ is

$$max\left(0,2{\mathrm{SSE}}_{A1}-{\mathrm{SSE}}_{B1}\right)<{\mathrm{SSE}}_{B2}<{\displaystyle \frac{{\mathrm{SST}}_Y}{{\mathrm{SST}}_X}}{\mathrm{SSE}}_{A1}-{\mathrm{SSE}}_{B1}.$$

(20)

Since ${\mathrm{SSE}}_{B2}$ is a sum of squared errors, it must satisfy ${\mathrm{SSE}}_{B2}\ge 0$.

Proof. 

Combining the RMSE and the NSE inequalities (17) and (18) gives the stated two-sided bound. The equivalent expression follows by solving for the ${\mathrm{SSE}}_{B2}$.

Two possible conditions can be distinguished. If $2{\mathrm{SSE}}_{A1}-{\mathrm{SSE}}_{B1}>0$, then $\frac{{\mathrm{SST}}_Y}{{\mathrm{SST}}_X}>2$. Conversely, if $2{\mathrm{SSE}}_{A1}-{\mathrm{SSE}}_{B1}\le 0$, then $\frac{{\mathrm{SST}}_Y}{{\mathrm{SST}}_X}>\frac{{\mathrm{SSE}}_{B1}}{{\mathrm{SSE}}_{A1}}$. Both conditions are derived in the Supplementary Material. In practical applications, the two models under comparison rarely differ to the extent that the sum of squared errors (SSE) of one model exceeds twice that of the other. Therefore, for the remainder of this paper, we assume that $2{\mathrm{SSE}}_{A1}-{\mathrm{SSE}}_{B1}>0$, ensuring that the lower bound in (20) is nonnegative. The complementary case, $2{\mathrm{SSE}}_{A1}-{\mathrm{SSE}}_{B1}<0$, which yields the condition $\frac{{\mathrm{SST}}_Y}{{\mathrm{SST}}_X}>\frac{{\mathrm{SSE}}_{B1}}{{\mathrm{SSE}}_{A1}}$, is left for future investigation.    □

Proposition 8 

(Variance decomposition). For equal block sizes, the variance ratio satisfies

$${\displaystyle \frac{{\mathrm{SST}}_Y}{{\mathrm{SST}}_X}}=1+{\displaystyle \frac{{\mathrm{SST}}_Z}{{\mathrm{SST}}_X}}+{\displaystyle \frac{n}{2{\mathrm{SST}}_X}}{({\overline{y}}_X-{\overline{y}}_Z)}^2,$$

(21)

where ${\mathrm{SST}}_Z={\sum }_{i=n+1}^{2n}{(y_i-{\overline{y}}_Z)}^2$ is the within-block variance of Z and ${\overline{y}}_Z$ is its mean. Hence, the feasible interval is nonempty if and only if $\frac{{\mathrm{SST}}_Y}{{\mathrm{SST}}_X}>2$ when $2{\mathrm{SSE}}_{A1}-{\mathrm{SSE}}_{B1}>0$. The proof of this inequality is provided in the Supplementary Material.

Proof. 

Let ${\overline{y}}_X=\frac{1}{n}{\sum }_{i=1}^n{y}_i$, ${\overline{y}}_Z=\frac{1}{n}{\sum }_{i=n+1}^{2n}{y}_i$, and ${\overline{y}}_Y=\frac{1}{2}({\overline{y}}_X+{\overline{y}}_Z)$. By definition,

$${\mathrm{SST}}_Y=\sum _{i=1}^{2n}{(y_i-{\overline{y}}_Y)}^2=\sum _{i=1}^n{(y_i-{\overline{y}}_Y)}^2+\sum _{i=n+1}^{2n}{(y_i-{\overline{y}}_Y)}^2.$$

For block X, write $y_i-{\overline{y}}_Y=(y_i-{\overline{y}}_X)+({\overline{y}}_X-{\overline{y}}_Y)$ and expand:

$$\sum _{i=1}^n{(y_i-{\overline{y}}_Y)}^2=\underset{{\mathrm{SST}}_X}{\underbrace{{\sum }_{i=1}^n{(y_i-{\overline{y}}_X)}^2}}+n{({\overline{y}}_X-{\overline{y}}_Y)}^2,$$

where the cross term vanishes because ${\sum }_{i=1}^n(y_i-{\overline{y}}_X)=0$. For block Z, similarly,

$$\sum _{i=n+1}^{2n}{(y_i-{\overline{y}}_Y)}^2={\mathrm{SST}}_Z+n{({\overline{y}}_Z-{\overline{y}}_Y)}^2.$$

Adding both parts gives

$${\mathrm{SST}}_Y={\mathrm{SST}}_X+{\mathrm{SST}}_Z+n{({\overline{y}}_X-{\overline{y}}_Y)}^2+n{({\overline{y}}_Z-{\overline{y}}_Y)}^2.$$

Since ${\overline{y}}_Y=\frac{1}{2}({\overline{y}}_X+{\overline{y}}_Z)$, we obtain

$$n{({\overline{y}}_X-{\overline{y}}_Y)}^2+n{({\overline{y}}_Z-{\overline{y}}_Y)}^2=\frac{n}{2}{({\overline{y}}_X-{\overline{y}}_Z)}^2.$$

Hence

$${\mathrm{SST}}_Y={\mathrm{SST}}_X+{\mathrm{SST}}_Z+\frac{n}{2}{({\overline{y}}_X-{\overline{y}}_Z)}^2,$$

which yields (21) after dividing by ${\mathrm{SST}}_X$.    □

Interpretation 3. 

In the unequal dataset case, contradictions between the RMSE and the NSE become possible. Specifically, Model B may have a larger RMSE yet simultaneously a larger NSE than Model A, provided that the variance of the combined dataset more than doubles relative to X. This inflation of ${\mathrm{SST}}_Y$ arises when Z exhibits high within-block variability, a substantial mean shift from X, or both. In such cases, the enlarged denominator in the NSE formula reduces the relative penalty for Model B’s higher errors, allowing the NSE to rank it above Model A even though the RMSE confirms that Model A remains superior in absolute error.

4. Discussion

The preceding mathematical analysis considered three distinct scenarios for comparing models under the RMSE and the NSE metrics. While the Results section established precise conditions under which the two metrics agree or diverge, it is equally important to interpret these findings in the broader context of hydrological model evaluation and machine learning practice. This section discusses the implications of each case, emphasizing when the metrics remain consistent and when apparent contradictions can arise.

4.1. Case I and Case II

When both models are evaluated on the same test dataset, either the original dataset X (Case I) or the expanded dataset Y (Case II), the ranking of models by the RMSE and the NSE is always consistent. Since both metrics are monotonic functions of the sum of squared errors (SSE) relative to the same variance baseline, we obtain the equivalence

$${\mathrm{RMSE}}_A<{\mathrm{RMSE}}_B⟺{\mathrm{NSE}}_A>{\mathrm{NSE}}_B.$$

(22)

Therefore, no contradictions can arise: whichever model achieves a lower RMSE necessarily achieves a higher NSE. This alignment ensures that, in practice, evaluations on the same dataset leave no ambiguity about which model is superior.

4.2. Case III

In contrast, when the two models are evaluated on unequal test datasets, it is possible for the rankings to diverge. Specifically, Model A may be evaluated only on X, while Model B is evaluated on the expanded dataset $Y=X\cup Z$. In this situation, Model B’s NSE is normalized by ${\mathrm{SST}}_Y$, the variance of the combined dataset. If ${\mathrm{SST}}_Y$ is much larger than ${\mathrm{SST}}_X$, Model B’s relative error ratio can shrink even if its absolute error (and thus its RMSE) remains larger. The necessary and sufficient condition for this outcome is

$$2{\mathrm{SSE}}_{A1}<{\mathrm{SSE}}_{B1}+{\mathrm{SSE}}_{B2}<{\displaystyle \frac{{\mathrm{SST}}_Y}{{\mathrm{SST}}_X}}{\mathrm{SSE}}_{A1},$$

(23)

with feasibility and the condition $2{\mathrm{SSE}}_{A1}-{\mathrm{SSE}}_{B1}>0$ requiring

$${\displaystyle \frac{{\mathrm{SST}}_Y}{{\mathrm{SST}}_X}}>2.$$

(24)

Using the pooled variance decomposition, we have

$${\displaystyle \frac{{\mathrm{SST}}_Y}{{\mathrm{SST}}_X}}=1+{\displaystyle \frac{{\mathrm{SST}}_Z}{{\mathrm{SST}}_X}}+{\displaystyle \frac{n}{2{\mathrm{SST}}_X}}{({\overline{y}}_X-{\overline{y}}_Z)}^2.$$

(25)

This condition can be satisfied if the added block Z has either very large internal variability (large ${\mathrm{SST}}_Z$), a substantial mean shift relative to X (large $|{\overline{y}}_X-{\overline{y}}_Z|$), or both. Under such conditions, it becomes possible for Model A to achieve a lower RMSE yet a lower NSE than Model B, thus producing a genuine contradiction between the two metrics.

4.3. Implications and Potential for Inflated NSE

The results highlight that contradictions between the RMSE and the NSE rankings only arise when models are tested on different datasets. In hydrological and machine learning practice, such situations can occur when trained models are tested on unequal time periods or spatial domains. In machine learning, in particular, it is common for models to be benchmarked on entirely different test datasets. In these settings, direct comparison of the NSE values across studies or regions is not meaningful: a higher NSE in one study does not necessarily indicate a better model than one with a lower NSE in another, because the underlying test datasets may have very different variance structures, mean levels, or event distributions. Without controlling for test dataset characteristics, the NSE alone is unsuitable for comparing models across independent experiments. Case III further emphasizes that the NSE can be artificially inflated by adding new test data with sufficiently high variance or shifted means. A weak model may therefore appear superior under the NSE simply because the denominator in the NSE formula grows faster than its errors.

Beyond variance inflation, other potential forms of the NSE manipulation include: (i) selectively extending the test dataset with extreme events that dominate the variance without proportionally increasing the error; (ii) rescaling the observed test series (e.g., through unit changes or aggregation choices) so that the variance baseline increases; and (iii) cherry-picking test evaluation periods with naturally high variability (such as wet seasons in rainfall–runoff studies). Each of these strategies can make a weak model appear competitive under the NSE, while its absolute accuracy as measured by the RMSE remains poor. Importantly, the RMSE is not subject to these distortions, since it directly reflects the magnitude of prediction errors without reference to the variance of the test observations. Unlike the NSE, the RMSE cannot be inflated by test dataset characteristics such as variance shifts or mean differences. As a result, the RMSE provides a more stable and accurate measure of absolute model performance, making it a valuable complement to the NSE when comparing models.

These findings suggest that the NSE should always be interpreted with caution, especially in comparative studies involving test datasets of different scales or variability. For hydrology and related sustainability fields, the broader implication is that rigorous model assessment requires transparency in test dataset selection and metric reporting, ensuring that apparent performance gains are not merely artifacts of test evaluation design.

5. Algorithmic Demonstration of NSE Inflation

The preceding analysis shows that contradictions between the RMSE and the NSE rankings arise only when models are evaluated on unequal test datasets. In particular, increasing the variance of an expanded test evaluation set $Y=X\cup Z$ can raise the NSE even when absolute errors remain large. This section provides an explicit constructive algorithm which, for a given test dataset $X={\left\{(y_i,{\widehat{y}}_i)\right\}}_{i=1}^n$ and desired margin $\delta >0$, produces a block Z such that

$${\mathrm{NSE}}_Y={\mathrm{NSE}}_X+\delta .$$

(26)

5.1. Algorithm Outline

Let

$$SS{T}_X=\sum _{i=1}^n{(y_i-{\overline{y}}_X)}^2,SS{E}_X=\sum _{i=1}^n{(y_i-{\widehat{y}}_i)}^2,{\mathrm{NSE}}_X=1-{\displaystyle \frac{SS{E}_X}{SS{T}_X}}.$$

We target (26), which is equivalent to the required squared error of the added block Z:

$$SS{E}_Z=\left({\displaystyle \frac{SS{E}_X}{SS{T}_X}}-\delta \right)SS{T}_Y-SS{E}_X.$$

(27)

Write the variance ratio $k_{\mathrm{var}}:=\frac{{\mathrm{SST}}_Y}{{\mathrm{SST}}_X}>1$ (the case $k_{\mathrm{var}}=1$ is trivial). Then (27) becomes

$$SS{E}_Z=\left[(1-{\mathrm{NSE}}_X-\delta )k_{\mathrm{var}}-(1-{\mathrm{NSE}}_X)\right]SS{T}_X.$$

(28)

To keep $SS{E}_Z\ge 0$, for the nontrivial case of ${\mathrm{SST}}_X\ne 0$, the minimal variance inflation factor must satisfy

$$k_{\mathrm{var}}\ge {\displaystyle \frac{1-{\mathrm{NSE}}_X}{1-{\mathrm{NSE}}_X-\delta }}.$$

(29)

For equal block sizes $\left|Z\right|=\left|X\right|=n$, the pooled-variance identity is

$${\displaystyle \frac{SS{T}_Y}{SS{T}_X}}=1+{\displaystyle \frac{SS{T}_Z}{SS{T}_X}}+{\displaystyle \frac{n}{2SS{T}_X}}{({\overline{y}}_X-{\overline{y}}_Z)}^2.$$

(30)

Hence, any desired $k_{\mathrm{var}}$ can be realized by choosing a within-block spread $SS{T}_Z$ and/or a mean shift $|{\overline{y}}_Z-{\overline{y}}_X|$. Two canonical constructions are provided below: (i) All Spread (no mean shift), and (ii) All Shift (minimal spread). An optional “Mixed” version is also discussed.

Proposition 9 

(Guarantee). Fix X, choose δ with $0<\delta \le 1-{\mathrm{NSE}}_X$, and select $k_{\mathrm{var}}$ for which equality holds in (29). If Z is constructed to realize $SS{T}_Y=k_{\mathrm{var}}SS{T}_X$ via (30), and its prediction errors satisfy (27), then ${\mathrm{NSE}}_Y={\mathrm{NSE}}_X+\delta$. Furthermore, if $SS{E}_Z$ is strictly smaller than (27), then ${\mathrm{NSE}}_Y>{\mathrm{NSE}}_X+\delta$.

5.2. Practical Notes and Constraints

• Valid range: $0<\delta \le 1-{\mathrm{NSE}}_X$; otherwise ${\mathrm{NSE}}_Y$ would exceed 1 or feasibility fails.
• Larger $\delta$ demands larger $k_{\mathrm{var}}$ per (29), achievable through $SS{T}_Z$ and/or mean shift $|{\overline{y}}_Z-{\overline{y}}_X|$ in (30).
• If the right-hand side of (27) is negative, then $k_{\mathrm{var}}$ cannot deliver the target $\delta$.
• Error assignment on Z: construct a vector $e\in {\mathbb{R}}^n$ with ${\parallel e\parallel }_2^2=SS{E}_Z$, then set ${\widehat{y}}_j=y_j-e_j$ for $j=n+1,\dots ,2n$. This can be done deterministically (equal-magnitude entries with alternating signs) or stochastically (i.i.d. random draws rescaled to the exact norm).

5.3. Minimal Worked Algorithms

All Spread (no mean shift)

This construction achieves the required variance inflation entirely by enlarging the within-block spread of Z while keeping its mean identical to that of X. It demonstrates how the NSE can be raised through variance inflation alone.

Algorithm 1 Variance-Only Construction (All Spread)

1: Input: $X={\left\{(y_i,{\widehat{y}}_i)\right\}}_{i=1}^n$, target margin $\delta \in (0,1-{\mathrm{NSE}}_X]$ 2: Compute $SS{T}_X$, $SS{E}_X$, ${\mathrm{NSE}}_X$. 3: Set $k_{\mathrm{var}}\leftarrow {\displaystyle \frac{1-{\mathrm{NSE}}_X}{1-{\mathrm{NSE}}_X-\delta }}$. 4: Set ${\overline{y}}_Z\leftarrow {\overline{y}}_X$.          ▹ No mean shift 5: Set $SS{T}_Z\leftarrow (k_{\mathrm{var}}-1)SS{T}_X$. 6: Construct $y_{n+1:2n}$ with mean ${\overline{y}}_X$ and variance $SS{T}_Z$. 7: Compute $SS{T}_Y\leftarrow k_{\mathrm{var}}SS{T}_X$. 8: Compute $SS{E}_Z$ from (27). 9: Build error vector e with ${\parallel e\parallel }_2^2=SS{E}_Z$; set ${\widehat{y}}_j\leftarrow y_j-e_j$. 10: Output: $Y=X\cup Z$ with ${\mathrm{NSE}}_Y={\mathrm{NSE}}_X+\delta$.

All Shift (Minimal Spread)

This construction achieves the required variance inflation almost entirely by shifting the mean of Z, while keeping its internal variance negligible. It illustrates that even a simple mean displacement can artificially inflate the NSE.

Algorithm 2 Mean-Shift-Only Construction (All Shift)

1: Input: $X={\left\{(y_i,{\widehat{y}}_i)\right\}}_{i=1}^n$, target margin $\delta \in (0,1-{\mathrm{NSE}}_X]$ 2: Compute $SS{T}_X$, $SS{E}_X$, ${\mathrm{NSE}}_X$. 3: Set $k_{\mathrm{var}}\leftarrow {\displaystyle \frac{1-{\mathrm{NSE}}_X}{1-{\mathrm{NSE}}_X-\delta }}$. 4: Set $SS{T}_Z\approx 0$.         ▹ Minimal variance 5: Set ${\mathrm{\Delta }}_{\mu }\leftarrow \sqrt{\frac{2SS{T}_X}{n}(k_{\mathrm{var}}-1)}$. 6: Set ${\overline{y}}_Z\leftarrow {\overline{y}}_X+{\mathrm{\Delta }}_{\mu }$; construct $y_{n+1:2n}$ with mean ${\overline{y}}_Z$ and negligible variance. 7: Compute $SS{T}_Y\leftarrow k_{\mathrm{var}}SS{T}_X$. 8: Compute $SS{E}_Z$ from (27); build e with ${\parallel e\parallel }_2^2=SS{E}_Z$. 9: Output: $Y=X\cup Z$ with ${\mathrm{NSE}}_Y={\mathrm{NSE}}_X+\delta$.

Mixed Spread+Shift (optional)

Choose any nonnegative pair $(SS{T}_Z,{\mathrm{\Delta }}_{\mu })$ satisfying

$$1+{\displaystyle \frac{SS{T}_Z}{SS{T}_X}}+{\displaystyle \frac{n}{2SS{T}_X}}{\mathrm{\Delta }}_{\mu }^2=k_{\mathrm{var}},$$

then apply (27). For example, fix ${\mathrm{\Delta }}_{\mu }$ to control the mean of Z and solve for $SS{T}_Z=(k_{\mathrm{var}}-1)SS{T}_X-\frac{n}{2}{\mathrm{\Delta }}_{\mu }^2$.

These constructions serve as feasibility demonstrations, highlighting the NSE’s sensitivity to test-dataset design under unequal evaluation bases. By contrast, in Cases I and II, where the dataset is fixed, the RMSE and the NSE rankings remained strictly consistent.

5.4. Ensuring Larger RMSE Together with Higher NSE

The constructions above guarantee that the NSE of the expanded dataset $Y=X\cup Z$ can be increased by any desired margin $\delta$, but they do not ensure that the ${\mathrm{RMSE}}_Y$ also exceeds the ${\mathrm{RMSE}}_X$. In the following, we show that it is possible to construct Z such that ${\mathrm{RMSE}}_Y>{\mathrm{RMSE}}_X$. At the beginning of the paper, two models (A and B) were introduced. Here, for simplicity, we focus on a single model, which can be regarded as Model B. The aim is to demonstrate that Model B, when evaluated on the expanded test dataset Y, can have both a larger $\mathrm{RMSE}$ and a larger $\mathrm{NSE}$ than when it is evaluated on X. In other words, the model does not improve in terms of RMSE, yet appears to improve according to the NSE. To see this, consider

$${\mathrm{RMSE}}_X=\sqrt{\frac{SS{E}_X}{n}},{\mathrm{RMSE}}_Y=\sqrt{\frac{SS{E}_X+SS{E}_Z}{2n}}.$$

We have ${\mathrm{RMSE}}_Y>{\mathrm{RMSE}}_X$ if and only if $SS{E}_Z>SS{E}_X$.

Substituting (28) and using $SS{E}_X=(1-{\mathrm{NSE}}_X)SS{T}_X$, this condition becomes

$$k_{\mathrm{var}}>{\displaystyle \frac{2(1-{\mathrm{NSE}}_X)}{1-{\mathrm{NSE}}_X-\delta }},$$

(31)

which is a stricter requirement than (29).

Interpretation 4. 

Inequality (29) establishes the minimum variance inflation needed to raise the NSE by $\delta$, but it does not constrain the RMSE. The stronger bound (31) guarantees that ${\mathrm{NSE}}_Y>{\mathrm{NSE}}_X$ and ${\mathrm{RMSE}}_Y>{\mathrm{RMSE}}_X$. Importantly, when $\delta =0$, (31) reduces to $\frac{SS{T}_Y}{SS{T}_X}>2$, exactly the threshold identified earlier for contradictions to be possible. Thus, the simple “greater than two” rule is revealed as a special case of this general framework.

This result demonstrates that it is possible to construct a dataset Z such that the combined dataset $Y=X\cup Z$ yields a higher NSE but also a higher RMSE compared with X. In other words, an inferior model with larger errors can nevertheless appear superior under the NSE once the test dataset is artificially expanded, underscoring the vulnerability of the NSE as a performance metric in unequal evaluation settings.

6. Numerical Demonstrations of RMSE–NSE Behavior

To illustrate the theoretical relationships and algorithms developed in the previous sections, this section presents two concise numerical examples based on real hydrological observations and model predictions reported in our previous studies [19,20]. In those studies, we presented a case where Model B, which was initially worse than Model A in terms of both the RMSE and the NSE, suddenly became better than Model A in terms of the NSE (but not the RMSE) when new test data from subsequent months became available. This section presents two artificially constructed example test datasets using the earlier algorithms. Both examples use the same underlying dataset from the Yufeng No. 2 torrential stream but differ in how the test dataset is expanded. The first example (all spread) shows that increasing the total variance of the observed data without changing its mean artificially inflates the NSE, whereas the second example (all shift) alters the mean level of the test dataset to produce a different type of divergence between the RMSE and the NSE.

6.1. Example 1: The “All Spread” Scenario

Figure 1 presents the results of the first numerical example. Panel (a) shows the original prediction–observation scatterplot for a deep learning model of flow velocity obtained from the Yufeng No. 2 dataset, yielding the $\mathrm{RMSE}=0.047$ m/s and the $\mathrm{NSE}=0.519$. Applying the “all spread” algorithm implemented in Python 3.11.13, we appended a new block of test data whose observations share the same mean as the original dataset but exhibit greater variability. As a result, the total variance (SST) increased, whereas the SSE and the RMSE remained unchanged, producing a higher NSE value. As shown in Panel (b), the resulting plot exhibits the same RMSE ($0.047$ m/s) but an increased $\mathrm{NSE}=0.619$, corresponding to an exact $+0.1$ increment relative to the original case, a target value preassigned for demonstration purposes. This example illustrates the mechanism of variance-driven inflation of the NSE: the efficiency score can increase solely by enlarging the variance of the test dataset, even though the absolute prediction errors remain unchanged. Such behavior exemplifies the analytical condition derived in the previous sections and highlights why comparisons of the NSE across test datasets with differing variability must be interpreted with caution.

In Figure 1(b), two nearly parallel lines can be observed running above and below the 45-degree reference line. These artificially introduced points represent the additional data generated by the “all spread” algorithm, which increase the total variance of the observed dataset while keeping the residuals, and thus the RMSE, unchanged. The lines appear somewhat artificial and obvious by design, as this example is meant purely for demonstration. In practice, one could selectively introduce less conspicuous data exhibiting similar spread characteristics to achieve the same effect of inflating the NSE without genuine model improvement. This figure therefore serves solely as an illustrative example of the algorithmic behavior rather than a realistic outcome.

6.2. Example 2: The “All Shift” Scenario

The second numerical example, termed the “all shift” scenario, uses the same base dataset as the previous section. As shown in Figure 2(a), the initial prediction–observation relationship yields the $\mathrm{RMSE}=0.047$ m/s and the $\mathrm{NSE}=0.519$, identical to the values in the earlier case. In this demonstration, the “all shift” algorithm appends an equal-sized block Z whose observations are tightly clustered around a shifted mean (mean shift only, negligible internal spread). This modification shifts the pooled mean and increases the total sum of squares (SST) through the between-mean term without introducing visible patterns; in our example, the absolute errors—and thus the RMSE—remain unchanged, whereas the NSE rises by exactly $0.1$. The added points are subtle and visually indistinguishable, demonstrating that a controlled mean shift, rather than additional spread, can inflate the NSE without altering absolute prediction errors.

Figure 2(b) shows the resulting scatterplot after the algorithmic adjustment, with the $\mathrm{RMSE}=0.047$ m/s and the $\mathrm{NSE}=0.619$. Unlike the previous example, the added data are visually indistinguishable from the original points. This subtlety highlights the strength of the algorithm, as it can effectively enhance the NSE without introducing clearly visible artificial patterns. In practical situations, one could achieve similar results by carefully reweighting or selectively including specific observations. This example thus demonstrates that apparent performance improvements in the NSE can occur without genuine error reduction, emphasizing the need for caution when interpreting the NSE-based comparisons across datasets or models.

7. Conclusions

This study examined the relationship between the NSE and the RMSE across three evaluation scenarios. The key findings can be summarized as follows:

1.

When models are evaluated on the same test dataset (Cases I and II), the rankings by the RMSE and the NSE are always consistent. A lower RMSE necessarily implies a higher NSE, and no contradictory outcomes are possible.

2.

When models are evaluated on unequal test datasets (Case III), contradictions may arise. In this setting, it is possible for one model to have a lower RMSE but simultaneously a lower NSE. The necessary and sufficient condition for this outcome, assuming that $2{\mathrm{SSE}}_{A1}-{\mathrm{SSE}}_{B1}>0$, is that the total variance of the expanded dataset more than doubles that of the original dataset, i.e.,

$${\displaystyle \frac{{\mathrm{SST}}_Y}{{\mathrm{SST}}_X}}>2.$$

This situation may occur if the new data block has very large variability, a substantial mean shift, or both.

3.

A strengthened bound was derived showing that, for a targeted increase of $\delta$ in the NSE, one requires

$$k_{\mathrm{var}}>{\displaystyle \frac{2(1-{\mathrm{NSE}}_X)}{1-{\mathrm{NSE}}_X-\delta }},$$

which guarantees that the combined RMSE is also larger than the original RMSE. This result generalizes the $>2$ rule: when $\delta =0$, the strengthened bound reduces exactly to $\frac{{\mathrm{SST}}_Y}{{\mathrm{SST}}_X}>2$. The implication is that an inferior model, already worse in the RMSE, can nevertheless appear superior under the NSE once the test dataset is artificially expanded. In other words, the model remains less accurate in absolute terms yet appears better in relative efficiency, exposing a structural vulnerability of the NSE.

In summary, the analysis demonstrates that the RMSE and the NSE provide fully consistent guidance when applied to a common test dataset, but contradictions emerge once models are compared across different evaluation bases. The inequalities derived here formalize the exact conditions under which such paradoxes occur, clarifying the mechanisms that drive apparent improvements in the NSE despite deteriorating the RMSE. For hydrological and machine learning applications, this emphasizes the critical importance of consistent evaluation datasets and cautions against overinterpreting the NSE values in cross-dataset comparisons, where a model may seem improved by the NSE even though its RMSE—and thus its absolute accuracy—is worse. By making these conditions explicit, the study contributes a rigorous theoretical foundation for interpreting efficiency metrics and highlights the need for transparency in evaluation design.