Maximum Entropy-Minimum Residual Model: An Optimum Solution to Comprehensive Evaluation and Multiple Attribute Decision Making

1. Introduction

Comprehensive evaluation is to assess a subject based on multiple factors or dimensions. It often collaborates with the methods of multiple criteria decision making (MCDM) or multiple attribute decision making (MADM), quantifying and assigning weights to each criterion (or attribute), in order to determine the best decision or choice, providing a holistic judgment. In complex decision-making environments, it facilitates making the optimal decision. Comprehensive evaluation is widely used in emerging strategic fields such as the humanities, environmental protection, technology, diversity, and sustainable development. In the process of constructing the comprehensive evaluation index (CEI), how to reasonably assign weights to the corresponding factors is a key consideration. The weight of each evaluation factor indicates its role and reflects the value orientation of the evaluation subject, is the core to work out scientific, reasonable and fair results. Therefore, the weighting method is a key consideration in research on multi-factor comprehensive evaluation. In this regard, Greco et al. (2019) [1] systematically reviewed the problems associated with weighting, aggregation, and robustness in CEI construction. In the existing weight assignment methods for CEI, there is no recognized standard. For the same evaluation subject, adopting various comprehensive evaluation methods lead to different rank and judgment. There is no existing perfect weighting method since each approach has its advantages and limitations. It is an inevitable problem in the field of CEI to be solved.

The handbook published by OECD/EU-JRC in 2008 [2] introduces various weighting methods for CEI calculation, outlining the advantages and disadvantages of each. The lack of a unified weighting method has led some researchers (Sharpe, 2004 [3]; Stiglitz, 2009 [4]) to question the rationality and fairness of multi-factor comprehensive evaluation, suggesting that subjectivity is a central defect. In the handbook [2], the authors dedicate a considerable amount of space to the benefit of doubt (BoD) method proposed by Melyn and Moesen(1991) [5] and Cherchye et al. (2007 [20], 2008 [7]) with reference to the data envelopment analysis (DEA) model as a tool for handling weights in comprehensive evaluation analysis. However, BoD has some serious theoretical problems. For example, some components might be 0 in the weight calculation results, and the extreme situation in which the weight of one factor is not 0 while the weight of others is 0 might arise (i.e., the 0-1 weight problem). Moreover, BoD models have multiple solutions. In addition, since the weight vectors of each evaluation object in the original BoD model are different, and the sum might not be equal to 1, CEI calculated by BoD is theoretically not comparable.

Especially in China, with the development of the market economy, the existing comprehensive evaluation methods are becoming increasingly inadequate in providing decision-makers with accurate measurements to assess the real situation of economic and social development in various regions under the new circumstances. At the same time, for those evaluation subjects affected by low rankings, if we do not have an objective and standardized comprehensive evaluation system, the evaluated subjects may have reasons to question the fairness and rationality of the evaluation results, which could lead to a reduction in the authority of the comprehensive evaluation. With the rapid growth of available information, CEIs are becoming more widely used. Bandura (2011) [8], for example, determined more than 400 CEIs for ranking or evaluating countries based on economic, political, social, and environmental factors. Yang (2014) [9], meanwhile, recorded more than 100 CEIs of human progress in a report by the United Nations Development Program. Although these lists are still far from being exhaustive compared to the actual number of applications in use, they provide a good understanding of the growing popularity of CEIs and the increasing demand for data interpretation and integration. However, even if people choose a theoretical framework, invest human and financial resources to collect data, and strive to implement the comprehensive evaluation process flawlessly, there might still be some defects in the evaluation process since there is no “gold standard” for determining factor weight in CEI. Despite such problems, CEIs are widely used by international organizations to measure economic, environmental, and social phenomena, thus serving as highly relevant tools [2].

Comprehensive evaluation or multi attribute decision making is still in a stage of blind exploration. This study aims to address the estimation of factor weights in comprehensive evaluation or MADM by incorporating statistics and informatics. Specifically, it focuses on how to assign scientific, objective, fair, and reasonable weight coefficients to the factors involved in constructing the comprehensive evaluation. The goal is to make a contribution to the methodology of constructing comprehensive evaluation indices. The authors believe that the relationship between the CEI and each evaluation factor, is similar to the relationship between work and energy in physics. The CEI represents the performance of each evaluation factor in comprehensive evaluation, and each evaluation factor is the source of abilities possessed by the CEI. There exists an underlying quantitative functional relationship between the CEI and the individual evaluation factors. In our current study, we establish a specific equation for this underlying function, but we have not yet found a method to solve it. So far, all mathematical approaches to exploring the CEI have not attempted to quantitatively solve this function. Instead, they have explored the correlation between the CEI and the individual evaluation factors from other perspectives. As a result, these non-quantitative methods that deviate from the underlying quantitative function have been in vain.

To address this issue, the author proposes a nonlinear fitting method based on the maximum information entropy (Jaynes, 1957 [10]; Golan and Harte, 2022 [11]) and the least squares residual, which is similar to weighted regression analysis. This method quantitatively fits the relationship between the CEI and the individual evaluation factors, treating the weight coefficient estimation and indicator fusion as a whole. This approach aims to solve the weights of the individual factors and fundamentally resolve the issues in the modeling process of the CEI. Section 2 specifies the definitions of the proposed maximum entropy-minimum residual model for comprehensive evaluation, and the related statistical tests in the new proposed model. Section 3 provides the definition of min-max normalization that is mainly used as data preprocessing technique in this paper, and gives an illustrative example for elaborate the optimization results of the new model with DPS software. Section 4 addresses comparisons of the proposed model with some existing commonly used models in various cases.

2. Maximum Entropy-Minimum Residual Comprehensive Evaluation Index Model

2.1. Comprehensive Evaluation Index Model

Given n samples of p factors, let $x_{ij}$ denote the observation value of the i-th sample from the j-th factor, $i=1,2,\cdots ,n,j=1,2,\cdots ,p$. From the perspective of statistical models, constructing a multi-factor comprehensive evaluation index, or the so-called composite indicator, ${\left\{y_i\right\}}_{i=1}^n$, requires estimating the weight coefficients, $w_j$, of each factor in the following equation

$$y_i=x_{i1}{w}_1+x_{i2}{w}_2+\cdots +x_{ip}{w}_p,$$

(1)

where $w_1+w_2+\cdots +w_p=1$, and $w_j\ge 0$. Theoretically, the CEI $y_i$ can be considered as composed of multiple theoretical components $C_{ij}$, each defined as the theoretical CEI value of the j-th factor in the i-th sample of the CEI model. According to the weight coefficients $w_j$, the theoretical CEI $y_i$ can be decomposed into the individual factor indicators corresponding to the theoretical components $C_{ij}$, implying that $C_{ij}=y_i\times w_j$. Therefore, $y_i$ can also be expressed in the form of the sum of its theoretical components,

$$y_i=C_{i1}+C_{i2}+\cdots +C_{ip}.$$

(2)

Since the Equations (1) and (2) are equivalent from the left side, it is justified to assume that $x_{ij}{w}_j$ and $C_{ij}$ are in one-to-one correspondence. If we consider $x_{ij}{w}_j$ as the observed values of the theoretical components $C_{ij}$, then the closer the two values are, the more representative the CEI becomes.

We define the difference between the observed and the theoretical components as the residual, ${\sigma }_{ij}=x_{ij}{w}_j-C_{ij}$, which is considered as a fitted error of the i-th sample of the j-th factor. In a regression model, the parameter estimation is derived from minimizing the residual, or say the root mean square error of $y_i$, denoted by $\sigma$ in our case,

$$\sigma =\sqrt{{\displaystyle \frac{{\sum }_{i=1}^n{\sum }_{j=1}^p{(x_{ij}{w}_j-C_{ij})}^2}{(n-1)p}}}=\sqrt{{\displaystyle \frac{{\sum }_{i=1}^n{\sum }_{j=1}^p{\left[(x_{ij}-y_i)w_j\right]}^2}{(n-1)p}}}.$$

(3)

From Equation (3), the residual error ${\sigma }_{ij}$ can be expressed as a weighted difference between the observed value $x_{ij}$ and the theoretical CEI $y_i$. The larger the weight coefficient of a factor, the greater its impact on the residuals, indicating that the factor is "more important." This approach is essentially a weighted regression analysis process. It is consistent with the principle in multi-factor comprehensive analysis, where a larger weight coefficient for a factor means that the factor is more important. The fitted error term ${\sigma }_{ij}$ is assumed independent of other factors, since under ideal circumstances, the component of each individual factor can be considered independent of the other factors in order to measure a specific characteristic of the CEI. The root mean square error of Equation (3) is similar as the decomposition of the sum of squared errors in a regression model, and the corresponding degree of freedom in our case is $(n-1)p$. Equation (3) can be considered as an error function to be minimized when deriving the solution of weights. However, If the weight coefficient of the j-th factor in Equation (3) is equal to one and the weight coefficients of the rest factors are zeros, then $y_i$ is equal to the observed value $x_{ij}$ while the residual $y_i-x_{ij}$ is minimized to zero. Obviously, directly minimizing Equation (3) ends up to a trivial solution in which a certain weight is 1 and zero otherwise, making the corresponding factor as the CEI. Although $\sigma =0$ in this special case, the resulting CEI is pointless.

2.2. Maximum Entropy-Minimum Residual (MEMR) Model for Comprehensive Evaluation

Entropy is a key concept in information theory, as a unit of measurement when quantifying the total amount of information in a dataset. The maximum entropy (ME) principle in [10] provides a method for reasoning from incomplete information. For problems in science, the more information we have, the better the models and theories we can construct. However, for a complex system, such as a multi-factor CEI model, developers never have enough information to unambiguously predict the probability distribution of the weights of individual factors. Indeed, constructing a CEI for such a complex system faces an underdetermined reasoning problem. When the ME principle is adopted by specifying available information as constrains, a model with the least bias can be chosen from all possible models consistent with the information. In establishing a CEI model, since the dimensions of the various indicators are inconsistent, the common approach is to preprocess the data using min-max normalization introduced in Section 3 or other normalization methods, so that the normalized values of each factor fall between 0 and 1. We can treat these as ratio values between 0 and 1, representing the probability of a factor’s distribution within the evaluation range. ME inference is considered as the correct way to estimate an unknown probability distribution given limited information (in the form of moments of the distribution) when entropy defined by Shannon (1948) [12] is chosen to effectively measure information for the probability distribution of random events (Jaynes, 1982 [13]). Distributions with higher entropy implicate greater diversity and are more likely to be observed [13]. Thus, ME inference leads to the best probability estimate from the given information, with no need to assume any further knowledge beyond that in a set of constraints. According to ME inference, any other form will be based on nonoptimal inference, either using less information than that available or expressing unjustified bias.

The ME principle is thus adopted in the present study to determine weight coefficients of multiple factors in the construction of a CEI. If each weight coefficient follows a probability distribution for the importance of each factor, the ME form of the probability distribution can be achieved through maximization of Shannon’s information entropy, which can be expressed in the following formula containing weight coefficients $w_j$ of all concerned factors,

$$E=-\sum _{j=1}^p{w}_j\log \left(w_j\right),$$

(4)

in which information entropy is at the maximum when all weight coefficients are equal. When the weight coefficient is equal to 1 for one factor and 0 for each of other factors, information entropy is at the minimum. This seems to contradict the optimization of a CEI by maximizing information entropy and minimizing residual error according to the ME principle.

The idea of using information entropy for CEI modeling is intuitive, similar to compression regression methods such as LASSO regression, where the optimization function for the CEI model is established in the following form,

$$minf(\sigma ,E)=min(\sigma -\lambda E),$$

in which taking different values for $\lambda$ leads to different optimization results. An appropriate $\lambda$ may provide a model as required. However, the biggest problem with this approach is how to choose $\lambda$. At the same time, the objective function to be optimized is a hybrid, because the dimensions of the residuals and information entropy are different. Combining them in the form of a sum or difference as the objective function is inappropriate, since the CEI model essentially aims to optimize the residuals defined by Equation (3). Another possible way to combine residual and entropy is through multiplication or division as

$$minf(\sigma ,E)=min(\sigma /E),$$

that simply scales the residuals up or down by several times, which is a linear relationship. The result is the same as optimizing based on the residual $\sigma$ defined by Equation (3), and it does not yield the desired statistical model. Given that the previous two forms are not suitable for constructing the CEI model, a natural idea is to adopt an exponential function to establish the error function, using the residual $\sigma$ as the base and information entropy E as the exponent, defined as

$$minf(\sigma ,E)=min\left({\sigma }^E\right).$$

(5)

Since the dimensions of all concerned factors are not consistent, data must be properly normalized in advance, such min-max normalization method, in order to constructing a CEI or MADM model. The normalized dimension value of each factor is essentially a ratio between 0 and 1, and the resulting residual $\sigma$ must be less than 1 accordingly, such that the error function value in Equation (5) becomes smaller when the entropy is larger. Therefore, the property of the error function in Equation (5) can be interpreted as the process of minimizing the residual $\sigma$ within the space of information entropy E. For original data without normalization, each observed value can be divided by the maximum (absolute value) of all observed values to convert it to a ratio between 0 and 1 before the optimization begins. In summary, the maximum entropy-minimum residual (MEMR) comprehensive evaluation index model can be expressed as

$$mi{n}_{{\left\{w_j\right\}}_{j=1}^p}\left({\sigma }^E\right),s.t.\sum _{j=1}^p{w}_j=1,w_j\ge 0,$$

(6)

where $w_j,j=1,2,\cdots ,p$ with constraint is the optimization variable in this nonlinear regression equation model.

The nonlinear regression Equation (6) with constraints can be solved using a variety of nonlinear optimization methods with constraints. Since convergence speed varies with different optimization algorithms, it may take a long time to achieve the bootstrap sampling estimation of model weight parameters. Thus, it is worthy of searching for an algorithm with stable operation and fast speed. We try to use sequential unconstrained minimization technique (SUMT) for optimization and obtain optimal results (shown in Section 3). The SUMT optimization algorithm has been included in DPS (data processing system) software package (Tang and Zhang, 2013 [14]). In this paper, a CEI constructed with the proposed MEMR model is called MEMR index, and the corresponding weight coefficient is called MEMR weight. MEMR modeling actually uses least squares techniques to extract the common trends of various factors in the process of building a CEI model or conducting MADM, while taking into account all factors (information) as comprehensively as possible. This is highly consistent with the motivation behind conducting comprehensive evaluation or MADM.

2.3. Statistical Tests in Comprehensive Evaluation Index Models

In constructing a CEI, the simplest approach is the equal weighting (EW) model, where all variables are assigned the same weight. If there are p variables, the weight coefficients for all factors are equal to ${\displaystyle \frac{1}{p}}$. Clearly, in actual comprehensive evaluation and MADM processes, the importance of each evaluation factor is not the same. Here, we use the equal weighting model as a reference (null model) for the CEI model, treating it as a submodel of the MEMR model, and performing statistical tests using a nested model approach. This means we can use statistical testing methods to compare the MEMR model with the EW model, i.e., to test whether the MEMR model can be simplified into the EW model (null hypothesis). The statistical test can be conducted using the F-test. The sum of squared residuals for each model is shown in Table 1, in which

$$\begin{array}{cc}\hfill RSS\left(w_{EW}\right)& =\sum _{i=1}^n\sum _{j=1}^p{\left({\displaystyle \frac{x_{ij}-y_i}{p}}\right)}^2,y_i=\sum _{j=1}^p{\displaystyle \frac{x_{ij}}{p}},\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill RSS\left(w_{MEMR}\right)& =\sum _{i=1}^n\sum _{j=1}^p{[(x_{ij}-y_i)·w_j]}^2,y_i=\sum _{j=1}^p{x}_{ij}{w}_j.\hfill \end{array}$$

(8)

The F-test statistic is expressed in terms of the sum of squared residuals and serves as a measure of the distance between the EW model and the MEMR model. A larger F-statistic indicates that the two models differ significantly, suggesting that the EW model does not describe the data well due to its equal weight coefficients for all factors. Therefore, the more general model, MEMR, is more suitable for comprehensive evaluation and MADM. The F-test statistic is a ratio of the last two mean square terms in Table 1, where the numerator is the difference in the variance terms between the two models, divided by the difference in the number of parameters between the models. The denominator is the mean square error of the MEMR model (the full model). The statistical test formula is as follows:

$$F={\displaystyle \frac{\frac{RSS\left({\widehat{w}}_{EW}\right)-RSS\left({\widehat{w}}_{MEMR}\right)}{p-1}}{\frac{RSS\left({\widehat{w}}_{MEMR}\right)}{(n-1)p}}}.$$

(9)

The F-test statistic is defined the same way as in linear models. However, unlike in linear models, the F-distribution only holds approximately (it becomes closer as the sample size increases). Based on the F-value and its degrees of freedom, it is not difficult to obtain the significance probability, or p-value. From the F-statistic test, we can conclude that if the statistical test is not significant (for example, using a 0.05 significance level), we can accept the EW model as the CEI model. However, generally speaking, the MEMR model can always improve the effectiveness of comprehensive evaluation or MADM to some extent. If $p<0.05$, then the EW model may not be suitable as the CEI model, and the MEMR model should be adopted. This is because, in the CEI model, at least one of the weight coefficients is different from the others, making the optimization process meaningful. The weight coefficients generated through optimization significantly improve the closeness of the composite indicator to the observed values compared to the equal weighting coefficients.

In addition, we define the model optimization index R as the percentage decrease of the difference in the sum of squared residuals between the EW model and the MEMR model, compared to the sum of squared residuals of the EW model,

$$R={\displaystyle \frac{RSS\left({\widehat{w}}_{EW}\right)-RSS\left({\widehat{w}}_{MEMR}\right)}{RSS\left({\widehat{w}}_{EW}\right)}}\times 100\%.$$

(10)

The meaning of R is that, after optimization, it measures the extent to which the CEI model has been improved compared to the null hypothesis, where each factor is assigned an equal weight. This is reflected by the percentage decrease in the sum of squared residuals, or it can be interpreted as the degree to which the goodness of fit has been improved.

Traditional parameter inference mainly relies on the Central Limit Theorem, which states that under large sample conditions, the sampling distribution follows a normal distribution. However, for certain statistics where the sampling distribution is unknown or difficult to compute, the Bootstrap method is very useful. Since the MEMR model is a nonlinear model, it is difficult to accurately estimate the standard errors of the weight coefficients. Here, we apply the Bootstrap method for estimation. For the multi-factor MEMR CEI model, a set of weight coefficients can be calculated for each Bootstrap sample. If we draw $n_B=1000$ Bootstrap samples, we can obtain 1000 sets of weight coefficients.

1. 95% Confidence Interval of Weight Coefficients

Based on these 1000 sets of weight coefficients, the $2.5th$ percentile and the $97.5th$ percentile are calculated to obtain the 95% confidence intervals for each weight coefficient. If the interval does not include ${\displaystyle \frac{1}{p}}$, it can be considered statistically significant when compared to the equal weight. In this case, it is not appropriate to use the equal weight to calculate the composite indicator in a multi-factor comprehensive evaluation.

2. 95% Standard Error of Weight Coefficients and Coefficient of Variation

Based on certain samples drawn using Bootstrap sampling, each having the same sample size as the initial one, and for each sample, the mean and variance of the sample statistics are calculated. For instance, the sample means are denoted by ${\overline{x}}_1,{\overline{x}}_2,\cdots ,{\overline{x}}_{n_B}$, and the mean estimation is given by $\widehat{\mu }={\sum }_{i=1}^{n_B}{\displaystyle \frac{{\overline{x}}_i}{n_B}}$. According to the formula, ${\sigma }^2={\sum }_i^{n_B}{\displaystyle \frac{{({\overline{x}}_i-\widehat{\mu })}^2}{n_B-1}}$, the coefficient of variation for each factor is given by $CV=\widehat{\sigma }/\widehat{\mu }$.

The DPS system calculates the average coefficient of variation for each factor to examine the fluctuation range of the weight coefficients of each factor in the CEI model, as well as the stability of the weight coefficients during the comprehensive evaluation process. Based on the size of the coefficient of variation, models with smaller coefficients of variation are selected from several models as application references.

3. An Example for Optimization of MEMR Model

3.1. Min-Max Normalization

When constructing CEI, many evaluation factors are measured with different units. Sometimes, the factors are considered to be ’good’, if a higher value indicates better performance. Otherwise, the factors are considered ‘bad’, such as ‘long-term unemployment’ and ‘obstruction of daily activities owing to chronic illness’ for which higher values associate with worse performance. To put all factors on a common dimension, data must be normalized using a rescaling technique. According to a factor is considered ‘good’ or ‘bad’, the following normalization technique is called min-max normalization that is popularly used for preprocessing data in comprehensive evaluation,

$$x_{ij}^{\prime }=\left\{\begin{array}{cc}{\displaystyle \frac{x_{ij}-x_{minj}}{x_{maxj}-x_{minj}}}\hfill & x_{ij}\mathrm{is}\mathrm{a}\text{’}\mathrm{good}\text{’}\mathrm{factor},\hfill \\ {\displaystyle \frac{x_{maxj}-x_{ij}}{x_{maxj}-x_{minj}}}\hfill & x_{ij}\mathrm{is}\mathrm{a}\text{’}\mathrm{bad}\text{’}\mathrm{factor}.\hfill \end{array}\right.$$

(11)

All normalized factors are represented by ratios between 0 and 1, which correspond to the ‘worst’ and the ‘best’ performance, respectively.

3.2. An Illustrative Example

For example, the min-max normalization is adopted when conducting a comprehensive evaluation analysis of the average per capita living consumption expenditure level of rural households in different regions of China in 2007 with the original data listed in the following Table 2, downloaded from https://download.csdn.net/download/weixin_42119432/12617871 (accessed on 8 January 2025).

After data normalization using Equation (11), the MEMR model is optimized with sequential unconstrained minimization technique (SUMT) conducted through DPS software. The system first outputs the information entropy of the nonlinear MEMR model with constraints with the value of 2.9806, while the logarithm of the objective function is equal to $-13.2752$ and the residual error $\sigma =0.0116$, as listed in Table 3. The second part of results is about the ANOVA table where the null hypothesis $H_0$ is that the weight coefficient of the j-th factor is ${\displaystyle \frac{1}{p}}$, and the alternative hypothesis is that at least one factor j has a weight coefficient unequal to ${\displaystyle \frac{1}{p}}$. This part of results are interpreted from statistical perspective in order to test whether there exists a difference among the weight coefficients of factors. If there is no significant difference (do not reject $H_0$), then a EW model can be used to construct the composite indicator. If the p-value is less than 0.05 (reject $H_0$), then the composite indicator needs to be calculated by the MEMR weights. In Table 3, p-value is 0.0002, implying that the CEI in this case should be calculated by the estimated weights of the factors instead of equal weights.

The third part of the output represents the weight coefficients of various factors, which serve as the core and focus of the comprehensive evaluation analysis model. These coefficients reflect both the contribution of each factor to the formation of the CEI and the importance of each factor in the CEI. They are the result of the interaction between individual factors and the CEI, derived under the condition of maximum entropy, minimizing fitting errors while accounting for central tendency, dispersion, and the interrelationships among multiple factors. The second column of the third part of Table 3 is the estimated weight coefficients, where a greater weight indicates that the corresponding factor is more important in comprehensive evaluation. In this case, based on weights, the two factors ’Transportation and Communication’ and ’Household Appliance’ have the greatest influence on the CEI. The second important factors are ’Others’, ’Housing’ and ’Food’, while the factors such as ’Entertainment’ and ’Healthcare’ have mild influence. The sample mean and standard deviation columns are estimated values calculated based on the results of Bootstrap sampling (1,000 iterations in this case). The standard deviation here represents the standard error of the weight coefficients under the normality assumption. Using the standard error, the 95% confidence interval can be estimated. The "Median" and "95% Confidence Interval" are estimated based on the percentiles of Bootstrap sampling. Generally, when the 95% confidence interval exceeds ${\displaystyle \frac{1}{p}}$, the weight coefficient is considered significantly larger; otherwise, the weight coefficient is considered significantly smaller. For example, in this case, the confidence intervals for the weights of the ’Household Appliance’ and ’Transportation and Communication’ factors are [0.1263, 0.1884] and [0.1350, 0.1826], respectively, both greater than ${\displaystyle \frac{1}{p}}=0.125$. Therefore, these two factors can be considered significantly larger in a statistical sense.

Bootstrap sampling estimation serves two purposes: statistical testing and assessing the sensitivity of each weight coefficient. The smaller the range of the 95% confidence interval based on Bootstrap percentiles, the more robust the weight coefficient of the factor is in evaluating the composite indicator or making multi-attribute decisions. Conversely, a wider confidence interval suggests that the weight coefficient should be treated with more caution when estimating the composite indicator and making decisions. Using Bootstrap simulation sampling, we estimate the standard error of each factor’s weight coefficient. To examine the estimation accuracy, we can calculate the mean standard error for all factors, as well as the mean weight coefficient for each factor (equal to ${\displaystyle \frac{1}{p}}$). Subsequently, we can calculate the coefficients of variation (CV) for the weight coefficients. The CV is typically used to compare the dispersion of different datasets. A larger CV indicates greater variability and higher dispersion, while a smaller CV suggests less variability and lower dispersion. By comparing the CVs of different datasets, we can evaluate their stability and consistency.

Finally, based on the CEI values generated by the maximum entropy-minimum residual model for each sample, we can rank the evaluation indices from highest to lowest to obtain the ranking results for all samples. In this example, the estimated CEI for each region, ranked from highest to lowest, are listed in the last part of Table 3. Based on the ranking results, a professional interpretation can be made along with a comparison of the positions of the evaluation objects. For example, in this case, Shanghai, Beijing, and Zhejiang rank as the top three, indicating the highest levels of consumption. They are followed by Jiangsu, Fujian, Guangdong, and Shandong, which have slightly lower consumption levels.

4. Comparisons of MEMR Model with Commonly Used Models

4.1. Technical Achievement Index

The original data for the construction of technical achievement index (TAI) by OECD [2] also adopted the min-max normalization, listed in Table A1 of the appendix for simplicity. Based on this data set, weight coefficients of eight TAI-based factors computed with the proposed MEMR model are compared with those computed with the methods of equal weighting (EW), principal component analysis (PCA) (Dunteman, 1989 [15]), factor analysis (FA) (Harman, 1975 [16]), and standard deviation (SD) (Mukhametzyanov, 2021 [17]), and criteria importance through intercriteria correlation (CRITIC) (Diakoulaki et al., 1995 [18]). Through the correlation analysis as shown in Figure 1, only the MEMR weights and the first principal component are highly positively correlated.

PCA is an industry-accepted method to reduce data dimensionality. The example data results show that the trend in the magnitude of the MEMR weights is consistent with the factor loadings of the first principal component in PCA, with a correlation coefficient of 0.8263. This indicates that the MEMR weights are essentially a form of data dimensionality reduction method. The MEMR and PCA weights are mostly negatively correlated with the weight coefficients obtained from other weighting methods. This suggests that, although these weights explain the variation of the evaluation factors, they do not serve the purpose of dimensionality reduction or combining the individual factors into a composite indicator.

Here, we also plot the relationships between the composite indicator, factor weights, and normalized values obtained using the MEMR method, entropy weighting (ET) method (Mukhametzyanov, 2021 [17]), CRITIC method, and SD method, in the form of a line-bubble combination chart as Figure 2. In the chart, the sample identification label (Region) is plotted on the horizontal axis, while the vertical axis represents the CEI values and the normalized values of individual factors, along with the factor weights displayed as bubbles. The diameter of the bubbles is uniformly defined as $0.025\times {\displaystyle \frac{w_j}{w_{max}}}$, where $w_j$ is the j-th factor weight coefficient, and $w_{max}$ is the maximum weight coefficient. The chart conveys four main points:

1.

Ranking and similarity of evaluation objects: the closer the points are, the more similar the evaluation objects are;

2.

Trend of the CEI line: the variation trend of the CEI along the line;

3.

Distribution of normalized values of each factor: the positioning of the normalized values of each factor around the CEI line indicates that the more concentrated the values, the better the ranking or evaluation effect;

4.

Factor weight coefficients: each factor is represented by a different color, with the size of each bubble indicating the magnitude of the weight coefficient. The larger the weight coefficient, the larger the bubble, and vice versa. This helps identify the distribution trends of different factors and their consistency with the CEI line.

As shown in Figure 2, in the model established using the MEMR method, factors with larger weights are distributed on both sides of the CEI line, while factors with smaller weight coefficients are located farther from the CEI line. The distribution of factor weight magnitudes aligns with the trend of the CEI values. This clearly reflects the representativeness of the CEI for the major factors with larger weight coefficients. However, the ET method, CRITIC method, and SD method do not exhibit a similar trend. This suggests that the methods of estimating factor weights for comprehensive evaluation using entropy weighting, criteria importance through intercriteria correlation, and standard deviation require further consideration. Remark that in the "Handbook on Constructing Composite Indicators" compiled by the OECD/EU-JRC (2008) [2], ET and CRITIC weighting methods are not mentioned.

Figure 2. Relationship diagram of composite indicator estimates and factor weights and normalized values for different methods: (a) The relationships of MEMR CEI, factor weights, and normalized values. (b) The relationships of ET CEI, factor weights, and normalized values. (c) The relationships of CRITIC CEI, factor weights, and normalized values. (d) The relationships of SD CEI, factor weights, and normalized values.

Figure 2. Relationship diagram of composite indicator estimates and factor weights and normalized values for different methods: (a) The relationships of MEMR CEI, factor weights, and normalized values. (b) The relationships of ET CEI, factor weights, and normalized values. (c) The relationships of CRITIC CEI, factor weights, and normalized values. (d) The relationships of SD CEI, factor weights, and normalized values.

4.2. Virtual Example Presented by an Orthogonal Table

An orthogonal table provides a virtual dataset comprising 24 orthogonal samples of four factors each having 2, 3, 4, and 8 levels respectively listed in Table A2 of the appendix, generated by the DPS software. All factors have no mutual correlation and hence are independent of one another. In the meanwhile, the importance of the qualitative factors can be intuitively distinguished by the number of levels since more levels implies more information. The original data are normalized with min-max method. Based on this virtual data set after normalization, the weight estimate of each factor computed by MEMR, ET, SD, CRITIC and PCA are listed in Table 4.

The PCA weighting method is used to correct for overlapping information between two or more correlated factors, rather than to measure the theoretical importance of the related indicators. If there is no correlation between the factors, this method should not be used to estimate weights, and in this case, PCA is not suitable for estimation. As the analysis results show, the normalized eigenvector used as the weight coefficient is a $4th$-order identity matrix, with all diagonal elements equal to 1 and all other elements equal to 0. The weight coefficients obtained using the ET method exhibit a trend where the weight coefficient decreases as the number of levels of a factor increases. This contradicts our intuitive analysis of the amount of information contained in the data. Specifically, if we treat each level of a factor as an ordinal variable, variables with more levels should typically have greater information content, providing more potential for analysis and inference. For example, $x_4$ has 8 levels, which is close to a quantitative indicator, and it should theoretically contain more information, thus having a larger weight in the comprehensive evaluation index. However, the weight coefficient here is less than half that of the two-level factor $x_1$. Moreover, since the correlation coefficients between the individual factors are zero, the weight results calculated using the CRITIC method are the same as those obtained using the SD method. Their results show a similar trend as the ET method, where the weight coefficient decreases as the number of levels of a factor increases. Therefore, this suggests that the ET, SD, and CRITIC methods, as ways of calculating weight coefficients, need further discussion.

Using the MEMR weights proposed in this paper, the trend shows that the weight coefficient increases as the number of levels of a factor increases. This is consistent with our intuitive understanding that variables with more levels typically contain more information than those with fewer levels. Additionally, with this orthogonal table as an example, if each factor in the orthogonal table adopts a z-score normalization by subtracting the mean and dividing by the standard deviation, then the mean of each factor will be zero, and the standard deviation will be 1. In this case, the weight coefficients of each factor, calculated according to MEMR model Equation (6), will be the same. This further illustrates that orthogonal tables have the characteristics of being uniformly distributed, orderly, and comparable.

4.3. Example for Comprehensive Development Status of Enterprises

In order to improve the investment efficiency, a large investment firm intended to evaluate 15 companies with 10 indicators that reflect the enterprise development status (Su et al., 2021 [19]). The evaluation results after min-max normalization are included in Table 5.

Su et al. (2021) [19] used various BoD methods and their improvements to construct weights and calculate the CEI. The corresponding BoD methods and the results of other methods are shown in the Table 6.

The non-compensatory BoD method introduced by Cherchye et al. (2007) [20] suffers from a significant 0-1 weight issue and often requires compensation rules of zero weights in the process of weighting. The BoD models without or with weight compensation are referred to non-compensatory or compensatory BoD. The CEI results of the non-compensatory BoD method for evaluating the comprehensive development status of enterprises are listed in Table 6. From the results, they cannot be used for ranking the enterprises. Cherchye et al. (2007) [20] gave five weight compensation methods by adding a rigid score compensation constraint to the non-compensatory BoD to address over-flexibility issues. Setting the constraint value to 0.85 as an example of absolute compensation, it is impossible to rank all the concerned companies (Table 6).

The generalized BoD method (Rogge, 2018 [21]) relaxes all computational symbols to general summation operator "⊕" and general multiplication operator "⊗" by means of a mathematical programming approach with geometric weighting aggregation patterns, but is still unable to rank the companies meaningfully. Zhou et al. (2007) [22] proposed a concept of "good indices" and "bad indices" from the positive and negative aspects of two-set BoD weighting method. This approach adds more information to a more discriminative bidirectional composite indicators in the BoD analysis. After the “good” and “bad” indices computed with the two-set BoD method are adjusted with the parameter $\lambda =0.5$, the resultant composite indicators remain undesirable for ranking the companies (Table 6).

If the importance levels of evaluation factors are known, the method of multi-criteria ABC analysis (Ng, 2007 [23]), close to the BoD analysis, can be adopted. Assuming the decreasing importance of each factor for the development status of those companies from left to right, the ABC analysis result in the rankings of the 15 companies from 0.55 to 1 (Table 6). The composite indicators computed with equal weighting method and the proposed MEMR method range from 0.47 to 0.62 and 0.46 to 0.63 (Table 6), respectively.

From the table, none of the BoD methods are suitable for the comprehensive evaluation of this dataset. A correlation analysis of the CEI computed with the ABC, EW, and MEMR methods shows that the correlation coefficients between the ABC method and the EW method, as well as between the ABC method and the MEMR method, are $-0.1694$ and $-0.2934$, respectively, both indicating negative correlations. The correlation coefficient between the MEMR method and the EW method is 0.9801, indicating a high positive correlation. This suggests that the ABC method does not reflect the trend of the composite indicator for each enterprise, whereas the composite indicator obtained through MEMR reflects the overall trend of each enterprise and can better ranking the 15 enterprises.

5. Conclusions

Multi-factor comprehensive evaluation is a widely used measurement method, whose results serve as an important auxiliary tool for decision-making. A key problem concerns how to assign weights to each factor when constructing comprehensive evaluation index. The fair and reasonable calculation of factor weights can provide a reliable foundation for decision-making. An unreasonable indicator, meanwhile, can make it difficult to provide objective evaluation results for decision-making bodies, and even result in misleading issue or controversy.

This study proposes an MEMR model based on the maximum entropy–minimum residual principle, directly applying the relationship estimation between multi-factor weights and the composite indicator. Based on the least-squares fitting of the theoretical component and the observed component, the model value of the theoretical component is the closest to the observed value of the actual component. The resulting CEI represents the central tendency and dispersion degree of each factor to the greatest extent, making it the most representative. The optimization process that minimizes the residuals while maximizing the information entropy is a method of extracting the common features of multiple factors or attributes under the premise of retaining as much information as possible. In a sense, this is the process of concentrating on a certain trend based on adopting as many suggestions as possible, and synthesizing the common points expressed by each factor, that is seeking unity while preserving diversity. This aligns with the basic principles of comprehensive evaluation and multiple attribute decision making.

Our proposed model can reflect the influence of each factor indicators better than the factor weight in the widely used equal weighting method and is closer to the actual meaning expressed by CEI. In the worst case, our model has the same evaluation outcome as the equal weighting method. PCA mainly considers the correlations between each factor. When the correlation coefficient between factors is small, the results of PCA are not very accurate. In addition, there is no standard regarding how many principal components need to be selected. When the number of principal components is different, the comprehensive scores might also be different. In this case, the evaluation object and its stakeholders will question the “rationality” of weight assignment.

The entropy weighting method, standard deviation method, and CRITIC method mainly estimate weight based on the dispersion degree of each factor, without considering the influence of the central tendency of each factor. These weighting methods do not take into account the quantitative relationship between each factor, its weight, and the estimated composite indicator. Instead, they focus more on presenting the data from a certain external characteristic to explain a particular phenomenon. They display data according to certain needs instead of deeply revealing the information contained within the data. There is no standard for when and how to choose these weighting methods, and they are not universally suitable for all comprehensive evaluation processes of quantitative factors. The problem becomes particularly salient when applying BoD for comprehensive evaluation.

Our proposed maximum entropy-minimum residual composite evaluation analysis modeling technique generates a composite evaluation index through numerical optimization, based on considering as much multi-factor information as possible while refining the central tendency of the factors. In the field of multivariate statistical analysis, the MEMR model employs nonlinear optimization techniques, allowing high-dimensional data to be more effectively mapped to one-dimensional space than principal component analysis. In various scenarios including big data analysis, we can apply the MEMR model for dimensionality reduction analysis of multivariate data.

The MEMR model is a new approach that combines information science and statistics. Accordingly, further theoretical research is needed in the following aspects.

1.

Inference and estimation of MEMR model parameters: this includes improving the current nonlinear numerical optimization method and inventing new methods in order to improve the effectiveness and robustness of optimization process; calculating inference accuracy and the confidence interval of parameter estimation; and further determining how to test the model hypothesis, diagnosing goodness of fit, and determining the rationality of the residual error;

2.

Estimation of weight coefficients in a MEMR model with attribute variables;

3.

Application to weight estimation in multiple attribute decision making (MADM): in MADM, it is necessary to consider how to evaluate the accuracy of each factor weight estimated by MEMR model for predicting future data or making new observations;

4.

Extensions of MEMR model: the model can be generalized to the problem involved panel data, or time series data; we also need to consider the applications of the MEMR model in different fields, such as medicine, ecology, and finance, and how to customize and improve the model according to the needs of specific fields; how to adjust the model if considering a scenario where the expert defines factor weights in advance.

Appendix A.1

Table A1. Original data for each factor in the technological achievement index of top 23 countries.

Region	Patents1	Royalties2	Internet3	Exports4	Telephones5	Electricity6	Schooling7	University8
Finland	187	125.6	200.2	50.7	3.08	4.15	10	27.4
US	289	130	179.1	66.2	3	4.07	12	13.9
Sweden	271	156.6	125.8	59.7	3.1	4.14	11.4	15.3
Japan	994	64.6	49	80.8	3	3.86	9.5	10
Korea	779	9.8	4.8	66.7	2.97	3.65	10.8	23.2
Netherlands	189	151.2	136	50.9	3.02	3.77	9.4	9.5
UK	82	134	57.4	61.9	3.02	3.73	9.4	14.9
Canada	31	38.6	108	48.7	2.94	4.18	11.6	14.2
Australia	75	18.2	125.9	16.2	2.94	3.94	10.9	25.3
Singapore	8	25.5	72.3	74.9	2.95	3.83	7.1	24.2
Germany	235	36.8	41.2	64.2	2.94	3.75	10.2	14.4
Norway	103	20.2	193.6	19	3.12	4.39	11.9	11.2
Ireland	106	110.3	48.6	53.6	2.97	3.68	9.4	12.3
Belgium	72	73.9	58.9	47.6	2.91	3.86	9.3	13.6
New Zealand	103	13	146.7	15.4	2.86	3.91	11.7	13.1
Austria	165	14.8	84.2	50.3	2.99	3.79	8.4	13.6
France	205	33.6	36.4	58.9	2.97	3.8	7.9	12.6
Israel	74	43.6	43.2	45	2.96	3.74	9.6	11
Spain	42	8.6	21	53.4	2.86	3.62	7.3	15.6
Italy	13	9.8	30.4	51	3	3.65	7.2	13
Czech Republic	28	4.2	25	51.7	2.75	3.68	9.5	8.2
Hungary	26	6.2	21.6	63.5	2.73	3.46	9.1	7.7
Slovenia	105	4	20.3	49.5	2.84	3.71	7.1	10.6

¹ Patents granted per 1 000 000 people; ² US $ per 1 000 people; ³ Internet hosts per 1 000 people; ⁴ % exports of high and medium technology products as a share of total goods exports (1999); ⁵ Telephone lines per 1 000 people (log); ⁶ kWh per capita (log); ⁷ Mean years of schooling (age 15 and above), which represents the basic education needed to develop cognitive skills; ⁸ % gross enrollment ratio of tertiary students enrolled in universities

Table A1. Original data for each factor in the technological achievement index of top 23 countries.

Region	Patents1	Royalties2	Internet3	Exports4	Telephones5	Electricity6	Schooling7	University8
Finland	187	125.6	200.2	50.7	3.08	4.15	10	27.4
US	289	130	179.1	66.2	3	4.07	12	13.9
Sweden	271	156.6	125.8	59.7	3.1	4.14	11.4	15.3
Japan	994	64.6	49	80.8	3	3.86	9.5	10
Korea	779	9.8	4.8	66.7	2.97	3.65	10.8	23.2
Netherlands	189	151.2	136	50.9	3.02	3.77	9.4	9.5
UK	82	134	57.4	61.9	3.02	3.73	9.4	14.9
Canada	31	38.6	108	48.7	2.94	4.18	11.6	14.2
Australia	75	18.2	125.9	16.2	2.94	3.94	10.9	25.3
Singapore	8	25.5	72.3	74.9	2.95	3.83	7.1	24.2
Germany	235	36.8	41.2	64.2	2.94	3.75	10.2	14.4
Norway	103	20.2	193.6	19	3.12	4.39	11.9	11.2
Ireland	106	110.3	48.6	53.6	2.97	3.68	9.4	12.3
Belgium	72	73.9	58.9	47.6	2.91	3.86	9.3	13.6
New Zealand	103	13	146.7	15.4	2.86	3.91	11.7	13.1
Austria	165	14.8	84.2	50.3	2.99	3.79	8.4	13.6
France	205	33.6	36.4	58.9	2.97	3.8	7.9	12.6
Israel	74	43.6	43.2	45	2.96	3.74	9.6	11
Spain	42	8.6	21	53.4	2.86	3.62	7.3	15.6
Italy	13	9.8	30.4	51	3	3.65	7.2	13
Czech Republic	28	4.2	25	51.7	2.75	3.68	9.5	8.2
Hungary	26	6.2	21.6	63.5	2.73	3.46	9.1	7.7
Slovenia	105	4	20.3	49.5	2.84	3.71	7.1	10.6

¹ Patents granted per 1 000 000 people; ² US $ per 1 000 people; ³ Internet hosts per 1 000 people; ⁴ % exports of high and medium technology products as a share of total goods exports (1999); ⁵ Telephone lines per 1 000 people (log); ⁶ kWh per capita (log); ⁷ Mean years of schooling (age 15 and above), which represents the basic education needed to develop cognitive skills; ⁸ % gross enrollment ratio of tertiary students enrolled in universities

Table A2. An orthogonal table with 24 runs and 4 factors.

No.	${\mathit{x}}_1$	${\mathit{x}}_2$	${\mathit{x}}_3$	${\mathit{x}}_4$
1	1	1	1	1
2	1	1	2	4
3	2	1	3	3
4	2	1	4	2
5	2	1	1	8
6	2	1	2	5
7	1	1	3	6
8	1	1	4	7
9	1	2	1	2
10	1	2	2	3
11	2	2	3	4
12	2	2	4	1
13	2	2	1	7
14	2	2	2	6
15	1	2	3	5
16	1	2	4	8
17	1	3	1	2
18	1	3	2	3
19	2	3	3	4
20	2	3	4	1
21	2	3	1	7
22	2	3	2	6
23	1	3	3	5
24	1	3	4	8

Table A2. An orthogonal table with 24 runs and 4 factors.

No.	${\mathit{x}}_1$	${\mathit{x}}_2$	${\mathit{x}}_3$	${\mathit{x}}_4$
1	1	1	1	1
2	1	1	2	4
3	2	1	3	3
4	2	1	4	2
5	2	1	1	8
6	2	1	2	5
7	1	1	3	6
8	1	1	4	7
9	1	2	1	2
10	1	2	2	3
11	2	2	3	4
12	2	2	4	1
13	2	2	1	7
14	2	2	2	6
15	1	2	3	5
16	1	2	4	8
17	1	3	1	2
18	1	3	2	3
19	2	3	3	4
20	2	3	4	1
21	2	3	1	7
22	2	3	2	6
23	1	3	3	5
24	1	3	4	8