Composition of Probabilistic Preferences in Multicriteria Problems With Variables Measured in Likert Scales and Fitted by Empirical Distributions

1. Introduction

In multicriteria decision aid problems, it is necessary to compose a decision matrix, in which two or more alternatives are evaluated according to two or more variables selected as decision criteria [1]. To solve the problem, those variables must be measured on some scale, so that a calculation algorithm can generate results and reach the objective of the research.

The scientific literature contains several studies on multicriteria decision aid problems, in which these variables are measured by psychometric scales, also called Likert scales, in reference to that author's research [2]. However, Likert assigned numerical values to a scale of five respondent perceptions (from “totally agree” to “totally disagree”) and considered the questionnaire responses through normal distributions, resorting to the use of means and standard deviation, for example, to draw conclusions.

From a statistical point of view, the use of parametric models (like the normal distribution, which depends on the mean and standard deviation of the data) with variables measured in ordinal scales is an error [3,4,5,6]. This common practice in scientific studies was even described as the first mathematical “sin” on a list of seven [7].

In general, the main criticism is the lack of mathematical/statistical rigor in relation to the measurement of variables on an ordinal scale. In this case, it is inappropriate to calculate averages, standard deviations, and other measures of central tendency to draw conclusions that have little or no meaning in the real world. For example, if a certain surgical procedure is rated on a four-point psychometric scale (e.g., poor, fair, good or excellent), it would not make sense to average the responses to the procedure, yielding a "fair and medium" result [7].

However, there are several authors who defend the use of descriptive statistics and parametric models in structured problems with Likert scales. Some argue that the use of scales with more than five points can better approximate the results to a normal distribution [8,9], or favor parametric models over nonparametric ones [10]. Other authors have mentioned the amount of research that “violates” the principles of statistics and obtained satisfactory results [11], the constraints on the use of parametric models with Likert scales [12], or the context in which such restrictions were proposed for the use of these scales [13], criticizing relaxation of mathematical rigor to the detriment of practical benefits.

The parameterization of data measured on ordinal scales has fueled controversy for decades between authors who favor mathematical/statistical rigor and argue against the procedure, considering that ordinal scales should not be parameterized, and academics from other areas who have reported gains in their studies that compensate for the relaxation of this rigor. We selected some studies defending the use of more flexible measures of central tendency in data generated by Likert scales and others against the relaxation of mathematical/statistical rigor in these problems to provide a better understanding of the arguments.

One of the most interesting papers on this controversy is from [13]. The author describes these discussions in the early 1900s, when physics was the model science and measurement followed mathematical principles to draw empirical conclusions about psychological data. Thomas [13] argued that psychology should treat measurement as the recording of empirical facts, which may not meet the mathematical properties of real numbers. There was then some error of interpretation, adaptation or excess of rigor in the use of the four measurement scales (nominal, ordinal, interval and ratio), proposed by Stevens [14], according to Table 1.

Thomas [13] pointed out that the article of Stevens [14] helped create a bridge to the use of mathematics to draw empirical conclusions about data without the properties of real numbers. However, this study was a product of its time and urged limitations on the mathematical operations for each scale, without explicitly anchoring those limitations in mathematics. Thomas [13] continued this criticism, noting that Stevens scales are flawed and encouraging the production of academic works that do not conform to the epistemology of science or the logic of mathematics.

Other authors have focused their research on finding solutions that allow the use of parameters based on Likert scales. Wu and Leung [8] tried to adapt ordinal scales as interval scales, increasing the number of points on the scale, preferably to eleven, to facilitate the adjustment of data to normal and other parametric distributions. Along the same lines, Awang et al [9] explored a parametric model of structural equations to suggest the use of ten-point Likert scales instead of the traditional five-point scales, to obtain more consistent results. Harpe [12] reviewed the literature, and based on empirical evidence, concluded that parametric analytical approaches are acceptable as long as certain criteria are met. The author also explored histogram densities, similar to the proposal of this article, but with adjustments to normal distributions instead of nonparametric ones. Mircioiu and Atkinson [15] analyzed ordinal data on Likert scales with high response rates, demonstrating that analysis using nonparametric methods causes loss of information. The addition of parametric methods, graphical analysis, subset analysis and data transformation leads to deeper analysis and better conclusions.

The authors who defend greater mathematical/statistical rigor condemn the use of models and parametric measures with data collected from Likert scales, regardless of the number of points on the scale. Pornel and Saldaña [16] discussed the characteristics and proper use of a Likert scale, examining 53 theses and dissertations in the Philippines. They identified four common misuses of the Likert scale, namely: unjustified length of the scale; asymmetrical verbal anchoring; uneven spacing in the verbal anchor; and unjustified interpretation of the mean. Jamieson [17] emphasized the common practice of assuming that the Likert scale is an interval scale, condemning the calculation of parametric measures, exemplifying the means and standard deviations, since these are typical of ordinal scales. Allen and Seaman [18] followed the same line. Sullivan and Artino [19] analyzed research in the medical field and made use of parametric tests more flexible, although they also criticized the use of means and standard deviations.

As can be seen, the subject is still controversial in the scientific literature. Thus, the purpose of this article is to illustrate the advantages of the Composition of Probabilistic Preferences (CPP) method in multicriteria decision aid problems, where the decision matrix uses evaluations measured on ordinal scales. CPP is capable of allaying the critiques described above and obtaining more precise results than descriptive statistics or parametric models can bring. There are basically three advantages of the proposed algorithm: (1) it is a nonparametric procedure, maintaining adherence to statistics with regard to the treatment of data in ordinal scales, without resorting to parameters to solve the problem (e.g., mean, deviation-default, etc.); (2) it dispenses with assigning numerical values to the Likert scale, since the frequency of responses to each item of the scale is sufficient; and (3) it explores an important property of the CPP, which probabilistically models responses to consider the uncertainty intrinsic to human opinion or judgment due to prejudices, doubts and biases, among other aspects that indicate a nondeterministic approach to the problem.

Two partial conclusions can be drawn from the studies for and against the full use of statistical tools with data measured on Likert scales: (1) the nonparametric approach of these data avoids difficulties present in parametric procedures; and (2) ranking of alternatives is consistent with the possible answers of ordinal scales, as defined by Stevens. The use of CPP fulfills these needs, since it is flexible regarding the choice of parametric or nonparametric models and can be used to define the probability of an alternative being better or worse than others, which corresponds to the empirical possibilities described in the ordinal scale of Stevens [14]. We present two cases to illustrate these characteristics numerically.

2. Materials and Methods

The proposed model is based on the CPP, originally developed by Sant’Anna and Sant’Anna [20] and later expanded by Sant’Anna [1]. This method incorporates the probabilistic nature of preference assessment into the multicriteria decision aid problem. The probabilistic characteristic can arise as a result of inaccuracies caused by subjective factors, leading decision makers to attribute different meanings to the same attributes of alternatives in different circumstances, or as a result of measurement errors that affect the evaluations of such attributes.

An initial and critical step in CPP is the transformation of the numerical vector, containing the evaluations of the various alternatives according to each criterion, into a vector containing preference probabilities. This transformation of exact values into probabilities is illustrated in the upper portion of Figure 1, where a triangular distribution emulates the preferences of an expert. The exact value that corresponds to an expert's assessment becomes the mode of a distribution of preferences, which varies between the extremes of the assessments by the criterion. In this example, the exact value “2” attributed to Alternative A and the range from “1” to “5” of Criterion 1 are converted into the triangular distribution of parameters (1,2,5), which indicate the minimum value “1”, the mode “2” and the maximum value “5”.

The new approach, in the lower portion of Figure 1, proposes a nonparametric procedure for this first stage of the CPP, for cases in which the respondents' original assessments are based on a Likert scale. Thus, instead of considering each respondent's preference for conversion into a parametric distribution, the histogram of the dataset is transformed into a nonparametric distribution. This conversion of the histogram into a nonparametric distribution is found in the literature [21,22,23,24], but its adaptation to the CPP is unprecedented.

This approach values CPP. The choice of nonparametric PDF avoids the aforementioned controversy, as it does not use descriptive statistics to extract parameters from the dataset. In addition, the new model simplifies CPP calculations in problems with a high number of respondents. For example, hundreds of evaluations of an alternative in a criterion can be generalized in a single histogram, avoiding the same number of calculations or the need to resort to a measure of central tendency to represent them.

The nonparametric distribution chosen in this approach is an empirical distribution, in which each point on the Likert scale is associated with a probability, calculated from the responses obtained from questionnaires or interviews. The R software provides several applications for fitting data to probability functions. The empirical distributions fitted here are based on the “mc2d” package [25].

The two subsequent stages of the CPP remain unchanged, and are described in detail in several published articles and books, with a wide range of applications, including sports science [26], management systems [27,28], security and defense [29], public health and social assistance [30,31], risk management [32], and bioenergy processes [33], among others. In summary, in the second stage, the joint probabilities of each alternative being superior or inferior to the others are calculated, maximizing or minimizing the preferences, respectively. In the third stage, these probabilities are aggregated according to some model, so that the order of individual or class preference is defined. These models range from the simplest, such as a weighted sum, to the most complex, such as aggregation by Choquet integrals [1].

Table 2 reports the pseudocode used to calculate the preference probabilities, referring to the second stage of the CPP, according to the proposed approach. The calculations were performed with the R software, with the code registered as open access on the Zenodo.org platform [34] (and available in the Appendix here). The third stage of the CPP was not included in the algorithm, since the examples in this article refer to the evaluation of alternatives under one criterion. This final aggregation of the third stage of the CPP requires evaluations of two or more criteria, beyond the scope of this proposal. Automation of the complete CPP procedure is available with open access in the “CPP” package [35].

The code registered on the Zenodo.org platform offers two functions: “PMax.Emp.Likert” to calculate the joint probabilities of maximizing the alternatives and “PMin.Emp.Likert” for minimizing the alternatives. Both are required for certain CPP compositions [1]. However, additional care must be taken when applying them to problems of ranking alternatives evaluated on Likert scales, related to the type of evaluation criterion. The “benefit” criteria indicate that the higher the score on the scale, the better the performance of the alternative is. In this case, evaluations with a value of “5” are more important than those with a value of “4”, and so on. On the other hand, the “cost” criteria indicate that the lower scores on the scale represent better performance of the alternative. Finally, it should be noted that the joint probabilities of maximizing and minimizing, considering three or more alternatives, are not complementary.

3. Applications

3.1. Dataset with the Same Means and Medians

We generated two datasets to illustrate the usefulness of the proposal in multicriteria decision aid problems with the alternatives measured by Likert scales. The first case contains ten alternatives, evaluated by 500 respondents, on a symmetrical five-point scale: “totally disagree”, “disagree”, “neutral”, “agree” and “totally agree”. The peculiar aspect of this case is that the responses of the ten alternatives (“A” to “J”) have the same average and the same median for any scale of values assigned to the five points, as long as they are equidistant. For example, if the five points assume values from “1” to “5”, the mean and median will be equal to “3”; if the points assume values (1, 3, 5, 7 and 9), the mean and median will be “5”. We chose this setup to demonstrate the limitation of descriptive statistics in comparison with the proposed model. Equations (1) to (10) indicate the frequencies of responses for each alternative, in which the first represents the option “totally disagree” and the last “totally agree”. For example, Alternative “A” received 50 “totally disagree”, 175 “disagree” responses and so on.

A = {50, 175, 100, 75, 100}

(1)

B = {150, 75, 75, 25, 175}

(2)

C = {100, 100, 125, 50, 125}

(3)

D = {100, 100, 75, 150, 75}

(4)

E = {75, 50, 200, 150, 25}

(5)

F = {25, 175, 175, 25, 100}

(6)

G = {75, 0, 325, 50, 50}

(7)

H = {100, 100, 100, 100, 100}

(8)

I = {125, 75, 125, 25, 150}

(9)

J = {150, 0, 175, 50, 125}

(10)

Figure 2 illustrates the histograms of Alternatives “A” and “B”, in which each column indicates the proportion of choice of the respective scale point, in relation to the 500 responses. The curve to the right of each histogram is the empirical PDF, which is nonparametric in nature. An interesting aspect to visualize is that the PDFs are unique for each dataset, differentiating them even for the situation of equality of their means and medians. This kind of data “DNA” is sensitive to the CPP, as its calculations depend on the PDF and its related cumulative function (CDF) to identify which alternative is likely to be superior or inferior to the others.

The PDF of the histograms in Figure 2 were defined by the “dempiralC” function, available in the “mc2d” package [25], according to Equation (11), where “x” are the scale values and “p” denotes their probabilities of occurrence, based on their frequencies. The “p” values are normalized to give the distribution one unit of area. The functions “dempiralC” and “pempiricalC” (CDF) were used in the calculations of PMax and PMin, according to the R code in the Appendix.

$$f(x)=p_i+\left(\frac{x-x_i}{x_{i+1}-x_i}\right)\left(p_{i+1}-p_i\right)\hspace{0.17em},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{x}_i\le x<x_{i+1}$$

(11)

Once the probability distributions that fit the evaluations represented by the histograms have been defined, the next step of the CPP refers to the calculation of the probabilities that the alternatives maximize or minimize their preferences, considering the type of criterion. Based on the code registered on the Zenodo.org platform, the functions “PMax.Emp.Likert” and “PMin.Emp.Likert” were applied to the ten alternatives, generating the results in Table 3. The highest frequency of extreme values of Alternative “B”, with 150 “totally disagree” and 175 “totally agree” responses, were decisive for its priority, both to maximize and minimize its preference over the others.

3.2. CPP Sensitivity to Likert Scale Cardinality

The second illustration of the usefulness of the model proposed here explores two alternatives (“K” and “M”), with variants K* and M*, according to the frequencies indicated in Equations (12) to (15). Alternatives “K” and “M” were rated on a 5-point scale and alternatives K* and M* on a 9-point scale. The extreme options remain with the indications “totally disagree” and “totally agree” and the intermediate ones as “neutral”. The K* and M* frequencies repeat the values of the “K” and “M” evaluations, only representing greater rigor on the part of the evaluators by using the 9-point scale. Both “K” and “M” and K* and M* have the same medians, but different means and modes, which would be enough to define the priority for K or K*, if decision-making privileged these measures of descriptive statistics.

K = {150, 50, 100, 25, 175}

(12)

M = {75, 50, 200, 150, 25}

(13)

K*= {150, 50, 100, 25, 175, 0, 0, 0, 0}

(14)

M* = {75, 50, 200, 150, 25, 0, 0, 0, 0}

(15)

Figure 3 and Figure 4 describe the histograms and empirical densities of alternatives K, M and their variants K* and M*.

In the literature review, we identified studies that sought to make the use of descriptive statistics more flexible in problems with Likert scales of higher cardinality, arguing that ten-point [9] or eleven-point [8] scales, for example, would be sufficient to relax the mathematical rigor /statistics of the scales of Stevens [14]. This application shows that the CPP is able to confirm this argument, but without resorting to statistical parameters for decision making.

Table 4 indicates that the expansion of the Likert scale points was not enough to change the parameters of the variables (median, mean and mode) in order to change the decision. However, the change in scale gives a new density to the data, even if the density profiles of K* and M* are similar (shifted to the left) to the respective ones of K and M. The CPP is able to capture this change in scale due to its nonlinear property, valuing the highest evaluations in the PMax calculations and valuing the smallest ones in the PMin calculations, as described in Gavião et al [36]. Thus, in the new 9-point scale, the PMax of Alternative K* becomes higher than that of M*, confirming the order of priority indicated by means and modes, without eliciting the historical critiques of using descriptive statistics in ordinal scales.

3.3. Unified Health System (SUS)

The national health system in Brazil is called the Unified Health System (SUS in the Portuguese abbreviation), ranging from basic care to complex procedures such as organ transplants. The SUS guarantees full, universal and free access to the country's population. Its chain of health units includes federal university hospitals, which are reference centers of medium and high complexity for the SUS. In addition, these hospitals are important training centers for human resources in the health area and provide support to teaching, research and extension of the federal institutions of higher education to which they are linked.

The network of federal university hospitals comprises 51 hospitals linked to 36 federal universities. Of these, 41 hospitals are linked to the Brazilian Hospital Service Company (Ebserh). This company was created by the federal government and is responsible for the management of federal university hospitals. Among the attributions assumed by Ebserh are the coordination and evaluation of execution of the hospitals' activities; technical support for the elaboration of management improvement instruments; and the distribution of resources to the hospitals. Several reports of satisfaction surveys carried out by Ebserh are made available for public access (https://www.gov.br/ebserh/pt-br/acesso-a-informacao/participacao-social/ouvidoria-geral/pesquisas-de-satisfacao).

In 2022, Ebserh published a panel of results with the satisfaction rates of different users of university hospitals, which included resident physicians in their units. The study consolidated the responses on a 5-point Likert scale, but drew conclusions based on calculating the averages of these responses. Thus, in this case study we sought to apply the proposed method to reassess the final ranking of teaching hospitals, based on the opinion of resident physicians.

Table 5 presents the results of calculating the PMax for each hospital, compared with the final ranking tallied by Ebserh. Kendall's ordinal statistical correlation was applied to the two rankings, indicating a moderate relationship of 49.2%. Three hospitals had the same position in the satisfaction assessment, while 18, highlighted in this table, improved their order in the probabilistic calculations. The results illustrate the greater discriminant power of the CPP for the final calculation. This is relevant, since these 18 hospitals could have priority in the distribution of public resources or in public educational policies, receiving new diagnostic equipment and medicines, among other benefits.

4. Conclusions

The purpose of this article was to illustrate with examples the advantages of the CPP in multicriteria decision aid problems that have used databases measured on Likert scales. Without addressing the criticisms and controversies that have fueled the debate for decades in the scientific literature, the CPP offers a model with full adherence to the characteristics of an ordinal scale, without involving any of the points raised for and against the use of data with that type of scale. The CPP can be used with empirical distributions that are nonparametric, or for data adjustment without the need to resort to data parameters, but only the histogram.

The code in R language was used to calculate the most important parameters of the proposal, namely the probabilities of preference of the alternatives in the criteria. To complete the CPP, it is necessary to collect data referring to more than one criterion and compose the probabilities under some rule. In this article, we did not extend the analysis to the last stage, instead identifying the literature on different forms and models of the CPP for this conclusion, including the “CPP” package of the R software.

A possible deepening of this study is the use of other nonparametric distributions in the code, such as kernel density models, comparing the results with those of the empirical distribution explored here.