A Review of Four Information‐Theoretic Criteria for Selecting the Best Probability Distribution from a Set of Candidate Distributions Fitted to Observed Data

1. Introduction

Assume that a sample y= {y₁, y_2, .., y_n} containing n observations for a continuous variable Y is generated from an unknown true distribution g(y). We aim to select, based on a statistical criterion, the distribution from a set of candidate distributions that best fit the observed data. Distribution selection is a crucial task in applications across numerous scientific and engineering fields, such as finance [1], biology [2], measurement science [3,4], hydrology [5,6], and wind engineering [7,8,9].

Akaike [10] introduced a statistical criterion that later became known as the Akaike Information Criterion (AIC). AIC was developed based on the Kullback–Leibler divergence between a fitted model and the unknown true data-generating model. It provides a unified approach for selecting among statistical models in applications such as regression, factor analysis, and time-series modeling, and distribution selection. deLeeuw [11] discussed Akaike's 1973 paper. Cavanaugh and Neath [12] provided a rigorous derivation of AIC. AIC is arguably the most well-known and most widely used model selection criterion. Since the introduction of the original criterion, numerous AIC variants have been proposed and developed. These include the corrected AIC (AICc) [13], Bayesian Information Criterion (BIC) [14], Consistent AIC (CAIC) [15], Hannan–Quinn Criterion (HQC) [16], Generalized Information Criterion (GIC) [17], and Quasi-likelihood under the Independence model Criterion (QIC) [18]. For reviews of AIC and other model selection methods, the reader is referred to [19,20]. Additionally, Reba et al. [21] recently proposed an approach that integrates AIC and its four variants—BIC, AICc, CAIC, and Hannan–Quinn Criterion (HQC)—to construct a composite score for distribution selection. This approach is similar to the "estimator averaging" or "model averaging" methods employed in various application fields in recent years, such as causal effect estimation in high-dimensional settings [22], high-dimensional covariance matrix estimation with network structures [23], and meta-analysis [24].

AIC originates from the Kullback-Leibler (KL) divergence in information theory and statistics, yet its calculation relies on log-likelihood values. In fact, log-likelihood is in intimate relation to Shannon information as defined in information theory [25]. We find that as early as the 1930s, Ramsey [26] proposed a criterion for distribution selection based solely on Fisher’s "Maximum Likelihood Principle", which we term the Minimum Negative Log-Likelihood (NLL) Criterion. This criterion can be viewed as a simplified form of AIC and derived without recourse to the KL divergence, yet it has been consistently overlooked in the existing statistical literature.

Two information-theoretic criteria have recently been proposed to address the problem of distribution selection. The first is the Minimum Entropy Criterion introduced by Huang [27]. Developed specifically for measurement uncertainty analysis, this criterion aims to meet the need for determining probability distributions of influence quantities—particularly in the context of Monte Carlo simulations [4]. In such applications, available information—such as observed data or technical specifications—can be utilized to construct candidate distributions. Huang [27] examined the two most common scenarios in measurement uncertainty analysis: Scenarios A and B. In Scenario A, where available information consists of a series of observations, the candidate distributions considered include the scaled and shifted z-distribution (i.e., the normal distribution), the scaled and shifted t-distribution, and the Laplace distribution. In Scenario B, where available information consists of the maximum permissible error specified in manufacturer technical specifications, the candidate distributions considered include the rectangular, triangular, quadratic, raised-cosine, and half-cosine distributions. According to the Minimum Entropy Criterion, the scaled and shifted z-distribution (i.e., the normal distribution) is the best distribution for Scenario A, while the raised-cosine distribution is the best distribution for Scenario B.

The second newly proposed criterion is the Maximum Informity Criterion [28]. For continuous distributions characterized by a probability density function (PDF), "informity" is defined as the integral of the squared PDF. Informity is closely related to the second-order Rényi entropy [29] and extropy proposed by Lad et al. [30]. It measures the degree of certainty represented by a distribution (i.e., its “Yang” aspect); thus, its meaning is diametrically opposed to the meaning of Shannon entropy, which measures the degree of uncertainty (i.e., its “Yin” aspect). Consequently, informity serves as the complementary dual of entropy [31]. Accordingly, the Maximum Informity Criterion is regarded as the counterpart to the Minimum Entropy Criterion [28]. Huang [28] examined Scenario B discussed in Huang [27] and identified the raised cosine distribution as the best distribution under the Maximum Informity Criterion, which is consistent with the result derived from the Minimum Entropy Criterion.

This paper reviews AIC, the Minimum Negative Log-Likelihood (NLL) Criterion, the Minimum Entropy Criterion, and the Maximum Informity Criterion. The discussion of AIC is limited to the context of distribution selection and does not address any of its variants or extensions (such as AICc and BIC), as these methods do not differ significantly from AIC in terms of distribution selection. The paper outlines the principles of these four criteria and explores the intrinsic connections between them.

In the following sections, Section 2, Section 3 and Section 4, and 5 describe AIC, the Minimum Negative Log-Likelihood (NLL) Criterion, the Minimum Entropy Criterion, and the Maximum Informity Criterion, respectively. Section 6 provides discussion. Section 7 provides two illustrative examples. Section 8 presents conclusions.

2. The Akaike Information Criterion (AIC)

Let $p\left(y\right)$ be the probability density function (PDF) of the candidate distribution $p$. The Akaike Information Criterion (AIC) can be expressed as

$$\mathrm{b}\mathrm{e}\mathrm{s}\mathrm{t} \mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{b}\mathrm{u}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}={\displaystyle \genfrac{}{}{0pt}{}{\arg \mathrm{m}\mathrm{i}\mathrm{n}}{p\left(y\right)}}\left[\mathrm{A}\mathrm{I}\mathrm{C}\left(p\right)\right],$$

(1)

where $\mathrm{A}\mathrm{I}\mathrm{C}\left(p\right)$ is given by

$$\mathrm{A}\mathrm{I}\mathrm{C}\left(p\right)=-2\mathrm{l}\mathrm{n}L\left(\widehat{\theta }\right|y)+2k,$$

(2)

where $\mathrm{l}\mathrm{n}L\left(\widehat{\theta }\right|y)$ is the empirical log-likelihood value calculated at the Maximum Likelihood Estimation (MLE) of the parameter vector $\theta$ of the candidate distribution $p$, and $k$ is the number of estimated parameters.

The deviation of AIC begins with the Kullback-Leibler (KL) divergence, defined as the relative entropy between the true distribution $g$ and the candidate distribution $p$ with respect to $g$ [10,12]

$$D_{\mathrm{K}\mathrm{L}}\left(g\right|\left|p\right)={\mathbb{E}}_g\left[\ln {\displaystyle \frac{g\left(y\right)}{p\left(y\right)}}\right],$$

(3)

where ${\mathbb{E}}_g[.]$ denotes the expectation under $g\left(y\right)$.

Rather than presenting the derivation in detail, we outline only the fundamental logic: specifically, how the AIC formula—utilizing the empirical log-likelihood—is derived from the KL divergence. We consider this essential for understanding AIC as an information-theoretic criterion for distribution selection. For a rigorous derivation of AIC, the reader is referred to [12].

Equation (3) can be rewritten as

$$D_{\mathrm{K}\mathrm{L}}\left(g\right|\left|p\right)={\mathbb{E}}_g\left[\ln g\left(y\right)\right]-{\mathbb{E}}_g[\ln p\left(y\right)]=\mathrm{H}(g,p)-\mathrm{H}(g)\ge 0,$$

(4)

where $\mathrm{H}\left(g\right)$ is the entropy of the true distribution $g$, given by

$$\mathrm{H}\left(g\right)=-\int g\left(y\right)\ln g\left(y\right)dy={-\mathbb{E}}_g[\ln g\left(y\right)],$$

(5)

and $\mathrm{H}(g,p)$ is the cross-entropy between $g$ and $p$, given by

$$\mathrm{H}\left(g,p\right)=-\int g\left(y\right)\ln p\left(y\right)dy={-\mathbb{E}}_g[\ln p\left(y\right)].$$

(6)

The principle of AIC lies in minimizing the KL divergence $D_{\mathrm{K}\mathrm{L}}\left(g\right|\left|p\right)$, that is, finding the candidate distribution $p$ that is close to the true distribution $g$ in the KL divergence sense. Notably, $\mathrm{H}\left(g\right)$ is unknown because the true distribution $g$ is unknown. However, $\mathrm{H}\left(g\right)$ is a constant independent of the candidate distribution $p$. Because of this, minimizing $D_{\mathrm{K}\mathrm{L}}\left(g\right|\left|p\right)$ is equivalent to maximizing the expected log-density ${\mathbb{E}}_g[\ln p\left(y\right)]$ (i.e., the cross-entropy $\mathrm{H}\left(g,p\right)$ ). Since ${\mathbb{E}}_g[\ln p\left(y\right)]$ depends on the true distribution $g$ that is unknown, it is inaccessible. However, according to the Law of Large Numbers, ${\mathbb{E}}_g[\ln p\left(y\right)]$ can be estimated as [32]

$${\mathbb{E}}_g[\ln p\left(y\right)]\approx {\displaystyle \frac{1}{n}}\sum \mathrm{l}\mathrm{n}p\left(y_i|\theta \right).$$

(7)

Notably, the sum of the log-density is the log-likelihood

$$\sum \mathrm{l}\mathrm{n}p\left(y_i|\theta \right)=\mathrm{l}\mathrm{n}L\left(\theta \right|y).$$

(8)

However, the empirical log-likelihood value calculated at the MLE of the parameter vector $\theta$ is optimistically biased as an estimator of the sum of the log-density. Consequently,

$${\displaystyle \frac{1}{n}}\mathrm{l}\mathrm{n}L\left(\widehat{\theta }|y\right)>{\mathbb{E}}_g[\ln p\left(y\right)].$$

(9)

Akaike [10] demonstrated from an asymptotic perspective that this bias can be approximately corrected using k. Multiplying by $-2$ (a historical convention) yields the well-known paneity term (i.e., the bias correction term) 2k in the expression for AIC.

It is important to note that in many practical situations of distribution selection, the contribution of the penalty term 2k to the AIC value is negligible. This is because the log-likelihood is the sum of all observations, and therefore its magnitude grows approximately proportionally to the sample size n. That is, $-2\mathrm{l}\mathrm{n}L\left(\widehat{\theta }|y\right)=O\left(n\right).$ By contrast, the number of parameters does not increase with the sample size. That is, $2k=O\left(1\right).$ Therefore, as $n\to \infty$, ${\displaystyle \frac{2k}{-2\mathrm{l}\mathrm{n}L\left(\widehat{\theta }|y\right)}}\to 0.$Furthermore, ignoring the penalty term is especially true for distribution selection, as most candidate distributions have a similar number of parameters. For instance, the normal, lognormal, logistic, Weibull, and Gamma distributions all have two parameters (k=2). The penalty only affects the comparison when k differs among the candidate distributions. When the number of parameters is the same across all candidate distributions, maximizing the log-likelihood $\mathrm{l}\mathrm{n}L\left(\widehat{\theta }\right|y)$ and minimizing AIC are equivalent. Therefore, for many distribution-selection problems, AIC and the log-likelihood may yield the same ranking because either all candidate distributions have the same number of parameters or the log-likelihood differences are much larger than the penalty differences. The distinction between the two metrics becomes significant only when comparing models of different complexity or when the improvement in goodness-of-fit is marginal.

3. The Minimum Negative Log-Likelihood (NLL) Criterion

Although AIC is derived based on the Kullback-Leibler (KL) divergence, its calculation relies on likelihood. This raises the question: can we construct a criterion for distribution selection based solely on Fisher’s "Maximum Likelihood Principle" without recourse to the KL divergence? The answer is yes. In fact, as early as the 1930s, Ramsey ([26], p. 209) proposed such a criterion. He noted:

In choosing a system we have to compromise between two principles: subject always to the proviso that the system must not contradict any facts we know, we choose (other things being equal) the simplest system, and (other things being equal) we choose the system which gives the highest chance to the facts we have observed. This last is Fisher’s ‘Principle of Maximum Likelihood’, and gives the only method of verifying a system of chance.

That is, the Maximum Likelihood Principle can provide a method for ranking candidate distributions, identifying the one with the highest empirical likelihood as the best distribution. Notably, when the likelihood function reaches its maximum, the negative log-likelihood reaches its minimum. This is because: (1) the logarithm is a strictly monotonically increasing function, so taking the logarithm does not alter the location of extrema, and the highest likelihood corresponds to the highest log-likelihood; and (2) multiplying by a negative number reverses the optimization direction, converting a maximum into a minimum. This leads to the Minimum Negative Log-Likelihood (NLL) criterion expressed as follows

$$\mathrm{b}\mathrm{e}\mathrm{s}\mathrm{t} \mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{b}\mathrm{u}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}={\displaystyle \genfrac{}{}{0pt}{}{\arg \mathrm{m}\mathrm{i}\mathrm{n}}{p\left(y\right)}}\left[\mathrm{N}\mathrm{L}\mathrm{L}\left(p\right)\right],$$

(10)

where $\mathrm{N}\mathrm{L}\mathrm{L}\left(p\right)$ is

$$\mathrm{N}\mathrm{L}\mathrm{L}\left(p\right)=-\mathrm{l}\mathrm{n}L\left(\widehat{\theta }|y\right)$$

(11)

Since a lower NLL score implies a higher likelihood, the distribution selected based on the Minimum NLL Criterion has the highest probability of producing the observed data among the candidate distributions considered.

In addition, the relationship between $\mathrm{N}\mathrm{L}\mathrm{L}\left(p\right)$ and $\mathrm{A}\mathrm{I}\mathrm{C}\left(p\right)$ can be expressed as follows

$$\mathrm{N}\mathrm{L}\mathrm{L}\left(p\right)={\displaystyle \frac{1}{2}}\left[\mathrm{A}\mathrm{I}\mathrm{C}\left(p\right)-2k\right].$$

(12)

Thus, the Minimum NLL Criterion can be viewed as a simplified form of AIC.

4. The Minimum Entropy Criterion

Let $\mathrm{H}\left(p\right)$ be Shannon entropy (also known as the differential or continuous entropy) of the candidate distribution $p$, written as

$$\mathrm{H}\left(p\right)=-\int p\left(y\right)\ln \left[p\left(y\right)\right]dy={-\mathbb{E}}_p[\ln p\left(y\right)].$$

(13)

The Minimum Entropy Criterion is expressed as follows [27]

$$\mathrm{b}\mathrm{e}\mathrm{s}\mathrm{t} \mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{b}\mathrm{u}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}={\displaystyle \genfrac{}{}{0pt}{}{\arg \mathrm{m}\mathrm{i}\mathrm{n}}{p\left(y\right)}}\left[\mathrm{H}\left(p\right)\right].$$

(14)

The Minimum Entropy Criterion can be derived using the KL divergence defined as the relative entropy between the candidate distribution $p$ and the reference measure $m\left(y\right)$ with respect to $p$, expressed as

$$D_{\mathrm{K}\mathrm{L}}\left(p\right|\left|m\right)={\mathbb{E}}_p\left[\ln {\displaystyle \frac{p\left(y\right)}{m\left(y\right)}}\right],$$

(15)

which can be rewritten as

$$D_{\mathrm{K}\mathrm{L}}\left(p\right|\left|m\right)={\mathbb{E}}_p\left[\ln p\left(y\right)\right]-{\mathbb{E}}_p[\ln m\left(y\right)]=\mathrm{H}(p,m)-\mathrm{H}\left(p\right),$$

(16)

where $\mathrm{H}(p,m)$ is the cross-entropy between $p$ and $m$, given by

$$\mathrm{H}\left(p,m\right)=-\int p\left(y\right)\ln m\left(y\right)dy={-\mathbb{E}}_p[\ln m\left(y\right)].$$

(17)

Huang [27] assumed that the reference measure $m\left(y\right)$ to be a Gaussian PDF with a mean of 0 and a very large variance ${\omega }^2$. Consequently, the shape of $m\left(y\right)$ is nearly flat, effectively acting as a non-informative distribution. In this paper, we assume that $m\left(y\right)$ to be a uniform distribution with a density of $1/\mathsf{\Omega }$, where $\mathsf{\Omega }$ represents a very large length (or range) sufficient to cover the vast majority of the support of the candidate distribution $p$. The result obtained using this uniform reference distribution is identical to that obtained using the Gaussian reference distribution, yet the derivation is more concise and straightforward.

Substituting $m\left(y\right)=1/\mathsf{\Omega }$ into Eq. (17), $D_{\mathrm{K}\mathrm{L}}\left(p\right|\left|m\right)$ becomes

$$D_{\mathrm{K}\mathrm{L}}\left(p\right|\left|m\right)=-\int p\left(y\right)\mathrm{l}\mathrm{n}{\displaystyle \frac{1}{\mathsf{\Omega }}}dy-\mathrm{H}\left(p\right)=\mathrm{l}\mathrm{n}\mathsf{\Omega }-\mathrm{H}\left(p\right)>0.$$

(18)

In this content, the KL divergence $D_{\mathrm{K}\mathrm{L}}\left(p\right|\left|m\right)$ represents the reduction in Shannon entropy of the candidate distribution $p$ relative to the non-informative refence measure $m\left(y\right)$. Notably, a larger value of $D_{\mathrm{K}\mathrm{L}}\left(p\right|\left|m\right)$ implies a greater reduction in entropy. In information theory, entropy corresponds to the degree of lack of information regarding an event. It has been established in machine learning that the reduction in entropy resulting from a data split is equivalent to information gain, and that maximizing this reduction is often used as a criterion for determining the optimal split (e.g., [33]). Therefore, a higher value of $D_{\mathrm{K}\mathrm{L}}\left(p\right|\left|m\right)$ corresponds to a higher information gain of the candidate distribution $p$. Our objective is to identify, among a set of candidate distributions, the one that maximizes information gain. Since $\mathrm{l}\mathrm{n}\mathsf{\Omega }$ is a constant, $D_{\mathrm{K}\mathrm{L}}\left(p\right|\left|m\right)$ reaches its maximum when $\mathrm{H}\left(p\right)$ reaches its minimum. Thus, the best distribution is the one with the lowest entropy, leading to the Minimum Entropy Criterion.

5. The Maximum Informity Criterion

Huang [28] introduced the concept of informity in informity theory. In the continuous case, the informity (also known as continuous informity) of the candidate distribution $p$ is given by

$$\beta \left(p\right)=\int {p\left(y\right)}^{2}dy={\mathbb{E}}_p\left[p\right(y\left)\right]$$

(19)

The Maximum Informity Criterion is expressed as follows [28]

$$\mathrm{b}\mathrm{e}\mathrm{s}\mathrm{t} \mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{b}\mathrm{u}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}={\displaystyle \genfrac{}{}{0pt}{}{\arg \mathrm{m}\mathrm{a}\mathrm{x}}{p\left(y\right)}}\left[\beta \left(p\right)\right].$$

(20)

However, Huang [28] did not provide the derivation of the Maximum Informity Criterion. Instead, he noted that since informity is inversely proportional to entropy, the Maximum Informity Criterion can be viewed as the counterpart to the Minimum Entropy Criterion.

Here we derive the Maximum Informity Criterion by considering the informity divergence, defined as the squared Euclidean distance (i.e., the L₂ norm) between the candidate distribution $p$ and the refence measure $m\left(y\right)$

$$D_{\mathrm{I}}\left(p\left|\right|m\right)=\int [p\left(y\right)-m\left(y\right){]}^{2}dy=\beta \left(p\right)-2\beta \left(p,m\right)+\beta \left(m\right).$$

(21)

where $\beta (p,m)$ is the cross-informity between $p$ and $m$, given by

$$\beta \left(p,m\right)=-\int p\left(y\right)m\left(y\right)dy={\mathbb{E}}_p\left[m\left(y\right)\right]={\mathbb{E}}_m\left[p\left(y\right)\right],$$

(22)

and $\beta \left(m\right)$ is the informity of the reference measure $m\left(y\right)$

$$\beta \left(m\right)=\int {m\left(y\right)}^{2}dy={\mathbb{E}}_m\left[m\right(y\left)\right]$$

(23)

Again, we assume that $m\left(y\right)$ is a uniform distribution with a density of $1/\mathsf{\Omega }$. Substituting $m\left(y\right)=1/\mathsf{\Omega }$ into Eqs. (22) and (23), Eq. (21) becomes

$$D_{\mathrm{I}}\left(p\left|\right|m\right)=\beta \left(p\right)-2\int p\left(y\right){\displaystyle \frac{1}{\mathsf{\Omega }}}dy+\int {\displaystyle \frac{1}{{\mathsf{\Omega }}^2}}dy=\beta \left(p\right)-{\displaystyle \frac{1}{\mathsf{\Omega }}}.$$

(24)

The informity divergence $D_{\mathrm{I}}\left(p\left|\right|m\right)$ represents the informity gain of the candidate distribution $p$ relative to the non-informative refence measure $m\left(y\right)$. According to informity theory [28], informity gain is equivalent to information gain. Therefore, a higher value of $D_{\mathrm{I}}\left(p\left|\right|m\right)$ implies the candidate distribution $p$ possesses higher information gain. Again, our objective is to identify, among a set of candidate distributions, the one that maximizes information gain. Since $1/\mathsf{\Omega }$ is a constant approaching zero (due to the assumption that Ω is extremely large), $D_{\mathrm{I}}\left(p\left|\right|m\right)$ reaches its maximum when $\beta \left(p\right)$ reaches its maximum. Consequently, the best distribution is the one with the highest informity, leading to the Maximum Informity Criterion.

6. Discussion

6.1. Comparison Between AIC and the Minimum Entropy Criterion

As discussed in Section 2 and Section 4, both AIC and the Minimum Entropy Criterion originate from the Kullback-Leibler (KL) divergence. However, the definition of the KL divergence differs in deriving AIC and the Minimum Entropy Criterion, leading to differences in their underlying principles. In the derivation of AIC, the KL divergence (denoted as $D_{\mathrm{K}\mathrm{L}}\left(g\right|\left|p\right)$) is defined as the relative entropy between the true distribution $g$ and the candidate distribution $p$ with respect to $g$; the principle of AIC lies in minimizing $D_{\mathrm{K}\mathrm{L}}\left(g\right|\left|p\right)$, thereby minimizing the discrepancy between the candidate distribution and the true distribution. In contrast, when deriving the Minimum Entropy Criterion, the KL divergence (denoted as $D_{\mathrm{K}\mathrm{L}}\left(p\right|\left|m\right)$) is defined as the relative entropy between the candidate distribution $p$ and the non-informative reference measure $m\left(y\right)$ with respect $p$; the principle of the Minimum Entropy Criterion lies in maximizing $D_{\mathrm{K}\mathrm{L}}\left(p\right|\left|m\right)$, thereby maximizing the reduction in entropy of the candidate distribution relative to the non-informative reference measure $m\left(y\right)$. Clearly, these two criteria are based on different principles or perspectives. However, interestingly, they are essentially the same. To understand this, first consider the relationship between $\mathrm{A}\mathrm{I}\mathrm{C}\left(p\right)$ and $\mathrm{N}\mathrm{L}\mathrm{L}\left(p\right)$. As stated in Section 2, for distribution selection, the penalty term 2k is insignificant and can be ignored, resulting in the following relationship between $\mathrm{A}\mathrm{I}\mathrm{C}\left(p\right)$ and $\mathrm{N}\mathrm{L}\mathrm{L}\left(p\right)$

$$\mathrm{A}\mathrm{I}\mathrm{C}\left(p\right)\approx -2\mathrm{l}\mathrm{n}L\left(\widehat{\theta }|y\right)=2\mathrm{N}\mathrm{L}\mathrm{L}\left(p\right).$$

(25)

Next, the negative log-likelihood $\mathrm{N}\mathrm{L}\mathrm{L}\left(p\right)$ is related to entropy $\mathrm{H}\left(p\right)$ as follows

$$\mathrm{N}\mathrm{L}\mathrm{L}\left(p\right)=-\mathrm{l}\mathrm{n}L\left(\widehat{\theta }|y\right)=-{\displaystyle \frac{n}{n}}\sum \mathrm{l}\mathrm{n}p\left(y_i|\widehat{\theta }\right)={-n\mathbb{E}}_p[\ln p\left(y|\widehat{\theta }\right)]=n\mathrm{H}\left(p\right).$$

(26)

Therefore, minimizing $\mathrm{A}\mathrm{I}\mathrm{C}\left(p\right)$ is equivalent to minimizing $\mathrm{N}\mathrm{L}\mathrm{L}\left(p\right)$, which in turn is equivalent to minimizing $\mathrm{H}\left(p\right)$. Thus, when the distribution parameters are estimated using the MLE method, AIC, the Minimum NLL Criterion, and the Minimum Entropy Criterion will yield the same result for distribution selection. This is demonstrated by Example 1 in the next section.

However, there are two key differences between AIC, the Minimum NLL Criterion, and the Minimum Entropy Criterion. First, the former two rely on the MLE method for parameter estimation, whereas the latter does not. The Minimum Entropy Criterion can be used regardless of the parameter estimation method employed, as the distribution parameters required to calculate entropy can be obtained via any statistically valid method (such as method of moments or unbiased estimation). Second, while the former two rely on observed data, the Minimum Entropy Criterion can be applied without observed data, provided that empirical candidate distributions can be constructed from available information. This aspect is particularly important for some applications such as the Type B evaluation of measurement uncertainty due to calibration errors, where available information is often limited to the maximum permissible error and the central tendency of the error. Huang [27] applied the Minimum Entropy Criterion to examine five candidate distributions for calibration errors: the rectangular, triangular, quadratic, raised cosine, and half-cosine distributions. He identified that the raised-cosine distribution is the optimal choice under the Minimum Entropy Criterion. This selection aligns with the views expressed by Castrup [34] regarding the physical plausibility and credibility of the raised-cosine distribution.

6.2. Comparison Between the Minimum Entropy Criterion and the Maximum Informity Criterion

For the same dataset, the Minimum Entropy Criterion and the Maximum Informity Criterion may or may not select the same distributions. In some applications (such as Example 2 in the next section), they yield identical results, whereas in others (such as Example 1 in the next section), the results may differ. This is because entropy measures the uncertainty represented by a distribution, while informity measures the certainty it represents. Although the reduction in entropy of a candidate distribution relative to the non-informative measure m(y) corresponds to the increase in its informity relative to m(y), the two metrics operate on different scales. Specifically, informity is calculated in the probability density space, whereas entropy is calculated in the log-transformed probability density space. Due to this scale transformation, the ranking in the former space does not necessarily align with the ranking in the latter. Consequently, for the same dataset or information, a maximum informity distribution does not necessarily correspond to a minimum entropy distribution. Nevertheless, these two criteria are not competing alternatives but rather complementary tools.

Furthermore, similar to the Minimum Entropy Criterion, the Maximum Informity Criterion does not reply on the MLE method for parameter estimation, nor is it constrained by any specific estimation method; it can be applied even in the absence of observed data, provided that available information allows for the construction of empirical candidate distributions. Huang [28] applied the Maximum Informity Criterion to examine five candidate distributions for calibration errors, the same case considered by Huang [27] using the Minimum Entropy Criterion. The results indicate that the ranking of these distributions by informity aligns with their ranking by entropy; the raised cosine distribution emerged as the best choice under the Maximum Informity Criterion, consistent with the findings derived from the Minimum Entropy Criterion.

7. Illustrative Examples

7.1. Synthetic Data Generated from a Weibull Distribution

Xie [35] examined a sample of 200 observations generated from a Weibull distribution with a shape parameter of k=1.8 and a scale parameter of λ=1. He fitted the data using the exponential, Weibull, Gamma, and normal distributions and estimated the distribution parameters via the maximum likelihood estimation method. Table 1 lists the probability density functions (PDFs) of these four candidate distributions and the parameter estimates provided by Xie [35].

Xie [35] compared these four candidate distributions using AIC and the Pearson Q statistic, calculated by partitioning the observed values ​​into 10 approximately equal-sized groups. Huang [27,28] previously investigated this example using the Minimum Entropy Criterion and the Maximum Informity Criterion. In this study, we re-examined this example by incorporating the Minimum NLL Criterion and the estimated KL divergence values, thereby providing a more comprehensive comparison of these four candidate distributions.

The KL divergence of the candidate distribution $p$ from the true distribution $g$, Eq. (4), can be rewritten as

$$D_{\mathrm{K}\mathrm{L}}\left(g\right|\left|p\right)\approx -{\displaystyle \frac{1}{n}}\mathrm{l}\mathrm{n}L\left(\widehat{\theta }|y\right)-\mathrm{H}\left(g\right)={\displaystyle \frac{1}{n}}\mathrm{N}\mathrm{L}\mathrm{L}\left(p\right)-\mathrm{H}\left(g\right).$$

(27)

In this example, the true data-generating distribution is the Weibull distribution with the shape parameter $k=1.8$ and scale parameter $\lambda =1$. Its entropy can be calculated as follows

$$\mathrm{H}\left(g\right)=\gamma \left(1-{\displaystyle \frac{1}{k}}\right)+\ln \left({\displaystyle \frac{\lambda }{k}}\right)+1=0.57721\left(1-{\displaystyle \frac{1}{1.8}}\right)+\ln \left({\displaystyle \frac{1}{1.8}}\right)+1=0.6688nats.$$

(28)

Table 2 shows the performance of these four candidate distributions based on four metrics: AIC,

Q-statistic, entropy, and informity, as well as the estimated NLL and KL divergence values.

As shown in Table 2, the Weibull distribution ranks first under AIC, the Minimum Entropy Criterion, and the Maximum NLL Criterion, indicating that these criteria successfully identified the true data-generating distribution. The Gamma distribution ranks first under the Q-statistic and the Maximum Informity Criterion. However, Xie [35] noted that the ΔAIC value between the Weibull and Gamma distributions is 4.7; this relatively small difference implies that the Gamma distribution also receives significant support from the data.

In addition, as shown in Table 2, the estimated KL divergence value for the Weibull distribution is the lowest among the four candidate distributions. This confirms the principle of minimizing KL divergence underlying AIC. Moreover, the estimated NLL/n values ​​for the four distributions are 0.8457, 0.6815, 0.6933, and 0.7797, respectively; these values ​​align closely with the entropy values ​​of 0.8456, 0.6798, 0.6935, and 0.7797. This confirms the relationship between NLL and entropy, as illustrated in Eq. (26).

It is worth noting that, in this example, the normal distribution is clearly not an appropriate model and should not be considered as a candidate distribution, as its domain includes negative values, whereas the data lie entirely within the positive range. Xie [35] considered the normal distribution but demonstrated its unsuitability for this synthetic dataset. He further pointed out: “Although the information-theoretic criteria make an automatic model selection process possible, it should be noted that the best way to ensure selecting a ‘good’ or ‘useful’ model is to reduce the number of candidate models fit to the data by thoughtful a priori model formulation.”

7.2. Wind Speed Data from Three US Wind Farms and a Weather Station in Canada

Sohoni et al. [7] examined four sets of wind speed data (A, B, C, and D). Datasets A, B, and C consisted of 10-minute average wind speed data (measured at a height of 100 m) from the SCADA (Supervisory Control and Data Acquisition) system of an NREL (National Renewable Energy Laboratory) wind farm, whereas dataset D comprised hourly average wind speed data (measured at a height of 10 m) recorded at a weather station in Alberta, Canada. Each dataset covered a one-year period. They employed five probability distributions to fit the wind speed data: Weibull, Rayleigh, gamma, lognormal, and inverse Gaussian. Following a preliminary visual comparison of the fitted distributions against the histograms, they excluded the lognormal and inverse Gaussian distributions from further analysis. They used the method of moments to estimate the model parameters for the remaining three candidate distributions. Table 3 lists the parameter estimates provided by Sohoni et al. [7].

Sohoni et al. [7] evaluated the goodness-of-fit of these three candidate distributions using Root Mean Square Error (RMSE)—defined based on the difference between fitted and observed probabilities—and R-squared metrics. Based on the rankings derived from the RMSE and R-squared, they concluded that the Weibull distribution provided the best fit for datasets A and D, whereas the Gamma distribution performed best for datasets C and B. Notably, they did not employ AIC metric in their analysis because the model parameters were not estimated via Maximum Likelihood Estimation (MLE); the conventional theoretical justification for AIC does not apply when the model parameters are estimated by the method of moments rather than by maximum likelihood.

In this study, we evaluated the performance of these three candidate distributions using the entropy and informity metrics. Both metrics can be applied without relying on likelihood estimates. Table 4 lists the PDFs of the three candidate distributions, along with their corresponding expressions for entropy and informity. Table 5 presents the performance of these three candidate distributions based on the entropy and informity metrics.

As shown in Table 5, the Gamma distribution ranks first under both the Minimum Entropy and Maximum Informity criteria. While this result aligns with the conclusions drawn by Sohoni et al. [7] for datasets C and B based on the RMSE and R-squared metrics, it differs from their findings for datasets A and D using the same metrics. It is worth noting that the entropy and informity metrics can be theoretically interpreted in terms of "information gain," whereas the RMSE directly measures the discrepancy between the empirical density and the fitted density; high information gain does not necessarily correspond to the minimum discrepancy between the empirical and fitted densities.

8. Conclusions

For distribution selection, when the candidate distributions have the same number of parameters, minimizing AIC is equivalent to minimizing the negative log-likelihood (NLL), and minimizing the NLL is equivalent to minimizing entropy. Therefore, AIC, the Minimum NLL Criterion, and the Minimum Entropy Criterion are essentially based on the same principle: minimizing negative log-likelihood or maximizing likelihood. It follows that, for a given dataset, when distribution parameters are estimated using the Maximum Likelihood Estimation (MLE) method, AIC, the Minimum NLL Criterion, and the Minimum Entropy Criterion will yield the same result for distribution selection.

However, AIC and the Minimum NLL Criterion rely on maximum likelihood estimates of parameters, while the Minimum Entropy Criterion and the Maximum Informity Criterion are not restricted to specific parameter estimation methods; this is because the distribution parameters required to calculate entropy or informity can be obtained via any statistically valid method, such as the method of moments or unbiased estimation. Furthermore, while AIC and the Minimum NLL Criterion rely on observed data, the Minimum Entropy and Maximum Informity criteria can be applied even in the absence of observed data, provided that available information allows for the construction of empirical candidate distributions. This represents a significant advantage of the entropy and informity metrics over the AIC and NLL metrics when conducting measurement uncertainty analysis involving calibration error—typically defined as the maximum permissible error specified in the manufacturer's technical specifications.

AIC, the Minimum NLL Criterion, and the Minimum Entropy Criterion operate on the log-transformed probability density space, whereas the Maximum Informity Criterion operate on the original probability density space. Due to the scale transformation, the ranking in the former space does not necessarily align with the ranking in the latter. Consequently, for the same dataset or information, a maximum informity distribution does not necessarily correspond to a minimum AIC, or NLL, or entropy distribution. Nevertheless, these criteria are not competing alternatives but rather complementary tools.