A Comparative Analysis of Multi-Criteria Decision-Making Methods and Normalization Techniques in Holistic Sustainability Assessment for Engineering Applications

1. Introduction

Multi-Criteria Decision-Making (MCDM) methods have a wide range of applications across various fields. There are numerous MCDM techniques, including the Weighted Sum Model (WSM), TOPSIS (Technique for Order Preference by Similarity to Ideal Solution), ELECTRE (ELimination and ChoiceExpressingREality), PROMETHEE (Preference Ranking Optimization METHod for. Enrichment Evaluation), VIKOR (VIseKriterijumska Optimizacija I Kompromisno Resenje) COPRAS (Complex Proportional Assessment), AHP (Analytic Hierarchy Process), and Goal Programming (GP), among others. Additionally, many hybrid models have been developed to address specific decision-making challenges. A common application area of MCDM is the sustainability assessment, which spans multiple industries and sectors, providing a structured approach to evaluate and optimize decisions based on sustainability objectives [1]. Given their versatility, MCDM methods have been widely employed and compared in the context of sustainability assessment, showcasing their strengths and limitations in various case studies and applications. For example, a thorough literature review on the application of MCDM methods in sustainable engineering was conducted in [2]. It highlights that selecting an appropriate MCDM method is itself a multi-criteria problem, as each method has its own strengths and limitations. The choice of method largely depends on the preferences of decision-makers and analysts, with no single method universally superior to others. The review also concludes that there has been a significant increase in the application of MCDM models across engineering disciplines over the past decade, with the AHP method being the most applied one , followed closely by TOPSIS. In [3], a hybrid MCDM model combining TOPSIS, DEMATEL (Decision-Making Trial and Evaluation Laboratory), ANP, and GRA (Grey Relational Analysis) is proposed as a tool for selecting green and sustainable materials. Similarly, [4] emphasizes the use of MCDM in the transportation sector, highlighting the critical role of stakeholder involvement in the decision-making process. Meanwhile, [5] advocates for the application of MCDM to deliver sustainable products to consumers. Additionally, [6] applies MCDM to assess the sustainability of electricity production technologies, incorporating methods such as MOORA and TOPSIS. Finally, [7] provides a comprehensive review of MCDM methods for sustainable renewable energy development, analysing a wide range of models, including SAW, AHP, MAUT, ELECTRE, PROMETHEE, and others, to identify optimal solutions.

In the aviation sector, MCDM methods are widely used for sustainability assessments, particularly in areas such as aircraft evaluation, airport, and airline operations. The author in [8] identifies 280 articles applying MCDM methods for sustainability assessments, involving over 60 different techniques. This study highlights a lack of clarity in many cases, as no reasons are provided for why a particular MCDM method was chosen over others. Instead, justifications typically focus on why the selected MCDM interpretation method was suitable for the study, often citing capabilities such as ranking or scoring a set of alternatives based on multiple criteria—functions that almost all MCDM methods can achieve. In a comprehensive review of over 160 papers on MCDM in the aviation industry [9], many studies focus on assessing sustainability criteria at the aircraft, airport, and airline levels, with AHP and TOPSIS being the most widely used methods. Another study [10] highlights that MCDM is an appropriate tool for sustainability assessment in aviation, especially when compared to other decision support systems such as deep learning and neural networks.

At the aircraft level, a hybrid MCDM approach is proposed in [11] for aircraft selection based on sustainability-related aspects. Another study [12], has used SWARA (Step-Wise Weight Assessment Ratio Analysis) is utilised for weighting and COPRAS for aggregation, focusing on sustainability criteria in aircraft selection. A comparison of AHP and ESM (Elimination Et Choice Translating Reality) in [13] also results in the same rankings, with sensitivity analysis also applied on the involved weights. Additionally, [14] compares a novel MCDM method with the WSM and the WPM (Weighted Product Method) to select an aircraft using a fuzzy weighting approach, leading to consistent conclusions. Additionally, in [15] MCDM methods were applied to assess and compare aircraft in terms of sustainability, incorporating AHP for weighting and WSM and TOPSIS for aggregation, which resulted in consistent outcomes

At the airport level, [16] uses a hybrid MCDM approach incorporating DEMATEL, DANP (DEMATEL-based ANP), and VIKOR to assess sustainability performance in international airports. Similarly, [17] proposes MCDM for selecting a new airport, comparing methods like SAW, TOPSIS, and AHP, all yielding consistent results. The importance of sensitivity analysis to assess robustness is also emphasized in this study.

Additionally, [18] proposes a hybrid MCDM method for selecting sustainable aviation fuels in supply chain management. This method is compared with others, including SAW and TOPSIS, showing consistent rankings across all methods, with sensitivity analysis applied to both input data and weights. Similarly, in [19], MCDM is used to assess the sustainability of alternative aviation fuels, incorporating and comparing methods such as SAW, TOPSIS, and GRA. The study concludes that all methods provide consistent rankings, with slight deviations observed only in the SAW method. Finally, [20] highlights the importance of stakeholder weighting in decision-making by proposing the Best-Worst Method, a technique that accounts for the relative importance of various stakeholders in the decision-making process.

The effect of normalization techniques in the context of MCDM methods has been also extensively studied in the literature. The choice of normalization technique can significantly impact the final ranking of alternatives and the selection of the best option. As noted by Chatterjee and Chakraborty [21], the normalization method employed in a MCDM approach plays a crucial role in determining the outcome of the decision-making process. Moreover, a comprehensive analysis of various normalization techniques for MCDM methods such as SAW, TOPSIS, and COPRAS was provided in [22], highlighting the pros and cons of each method. The authors suggest that sum-based linear normalization and vector normalization methods are among the most preferred normalization techniques, although the final choice depends on the specific structure of the problem being analysed. In the context of ship design, [23] investigates the effect of normalization (sum-based and vector normalization) methods on MCDM techniques such as WSM, WPM, TOPSIS, and ELECTRE. The results indicate consistency in the outcomes based on correlation analysis. In a study on web services [24], the impact of normalization on the VIKOR method was assessed using techniques such as linear sum, linear max, min-max, and vector normalization. The findings revealed that different normalization methods resulted in distinct outcomes, emphasizing the importance of selecting the appropriate technique for the given context. Additionally, [25] examined normalization techniques like max, min-max, and vector normalization in conjunction with the SAW aggregation method, leading to different rankings for each technique. Similarly, [26] assessed various normalization methods for the TOPSIS, highlighting the challenge of identifying the most suitable normalization technique for each problem. In [27], different normalization techniques for the SAW method are evaluated, concluding that min-max normalization is the most appropriate compared to other methods, although the study could not offer concrete recommendations for practical applications. In [28], the authors assessed ten different normalization techniques for the PSI method, noting that each technique led to different results, though some could be correlated. The challenges of selecting the optimal normalization technique were also discussed in [29], where the authors pointed out that varying rankings can result from different techniques, complicating the decision process. Different normalization methods were compared in [30] in the presence of outliers, showing that each method yielded distinct results depending on the normalization approach. On the other hand, [31] assessed five normalization techniques in conjunction with the MARCOS aggregation method, finding relatively consistent results for all normalization methods. In the robotics sector, [32] compared different normalization techniques for the Weighted Aggregated Sum Product Assessment (WASPAS), alongside other MCDM methods, favouring min-max normalization for its effectiveness. In a study focused on time and complexity criteria, [33] examined various normalization techniques for the TOPSIS method. Lastly, in the aviation sector, three normalization techniques—min-max, sum, and z-score normalization—using the WSM method were assessed in [34]. The study found consistent rankings for both the min-max and z-score methods. In another study for aircraft selection problems [35], different normalization techniques were applied to the weighted sum aggregation method, with results showing that vector, linear sum, and linear max normalization produced similar outcomes, while min-max normalization led to different results.

The findings regarding both the MCDM methods and the selected normalization techniques underscore the importance of carefully choosing the appropriate method based on the specific characteristics of the problem. Different techniques can yield varying results, which highlights the need for a tailored approach in decision-making. The choice of normalization method can significantly influence the final rankings and the selection of the best alternative, depending on the structure and requirements of the problem at hand. Recognizing this impact, it becomes essential to systematically investigate how different normalization methods and decision-making tools perform in practical applications.

The aim of the present work is to explore and evaluate the effectiveness of various MCDM tools and normalization methods in sustainability assessments of engineering processes and structures, focusing on an aircraft component design with multiple variations. Data collected from an aircraft structural component, investigated in a previous study of the authors [36] is utilized to implement a Sustainability Index (SI), firstly introduced in [37], integrating key metrics across multiple sustainability-related dimensions, involving mechanical performance, environmental impact (emissions), economic factors (costs), social dimensions (social impact metrics), and circular economy principles (focusing on the recyclability of the materials), for a large set of aircraft component configurations. This research investigates how different MCDM methods, such as the WSM, TOPSIS, modified TOPSIS, VIKOR, and COPRAS, as well as various normalization techniques (min-max, median, vector, z-score, and sum normalization), affect the robustness, reliability and the interpretation of the final outcomes. The findings aim to enhance the decision-making process in engineering applications, with a particular focus on sustainability in aircraft design.

2. Dataset – Exploited Use Case for Sustainability Evaluation

To assess and compare the robustness and sensitivity of various MCDM methods and normalization techniques on the evaluation of the SI on aviation sector, a comprehensive dataset from study [36] has been utilized. This dataset is specifically centred on the design and evaluation of alternative configurations of a commonly used aircraft component: a hat-stiffened panel. This type of panel is a critical structural element in aerospace applications and is composed of two primary components—the skin and the stringer. A typical profile and geometry of a hat-stiffened panel are illustrated in Figure 1 [38]. Five distinct material combinations were analysed, as detailed in Table 1. The dataset includes 4487 distinct design alternatives, with variations in both the material properties and the geometrical features of the panel. Each configuration was systematically evaluated across multiple dimensions of sustainability: mechanical performance, environmental impact, economic cost, social impact, and circularity performance. These dimensions are quantified and aggregated into a sustainability metric, unique for each alternative, using a framework described in [36]. In the latter study, the aggregated SI has been calculated using a weighted sum approach, with each metric being normalized using min-max normalization. Therefore, this dataset has been utilized as a representative example of an aircraft component design procedure, providing a foundation for assessing the effects of different MCDM methods and normalization techniques within a decision support process. This is particularly valuable in scenarios where multiple, and often conflicting, aspects of sustainability—such as mechanical performance, environmental impact, economic costs, social considerations, and circularity—must be balanced.

3. Methodology

For the aforementioned dataset, a holistic SI was evaluated with respect to five pillars: performance, environment, cost, circularity and society, based on five (5) MCDM and five (5) normalization methods. Three different weighting scenarios were considered, one with equal weights on each pillar, one where the environment and performance were prioritized and one where cost and performance were prioritized. A Pearson correlation analysis was conducted for the five MCDM methods to verify the relationship between the alternative’s rankings, indicating a strong positive correlation among them. Moreover, a sensitivity analysis was conducted to test the robustness of the studied methods. Specifically, the data was perturbed by a zero-mean Gaussian error, with the standard deviation being a percentage of the standard deviations of the initial variables. Based on 1,000 simulated perturbed samples, the mean and standard deviation of the final rankings were computed and compared. The most robust method with respect to small departures from the initial data, as well as the most sensitive to high perturbations, was selected for further investigation, considering the normalization method and its effect on the holistic SI evaluation.

3.1. MCDM Methods Considered

The impact of various MCDM methods was assessed using the approaches outlined below. These methods are recognized as some of the most widely utilized and popular techniques in the literature, due to their effectiveness in handling complex decision problems where multiple conflicting criteria must be considered. By applying these methods, the study aimed to evaluate how different decision-making frameworks can influence the sustainability assessment of alternative design configurations.

3.1.1. Weighted Sum Method (WSM)

The WSM or alternatively, Simple Additive Weighting (SAW) method is a widely used MCDM method in practice, since it is an easy-to-implement and interpret method. The WSM was introduced by Fishburn [39] providing a scoring model based on the weighted average principle. It assigns weights to each objective and then takes the weighted sum of them, converting multiple objectives into an aggregated scalar objective function. The weights are usually determined by the Analytical Hierarchy Process (AHP) method [40,41,42].

More specifically, let $x_{ij}$ be the performance of the alternative A_i against the attribute Cj, i =1, …. I and j=1, …,J, $y_{ij}$the normalised rating and $w_j$ the weight of the Cj attribute, with $\sum w_j=1$. The normalized data is utilized since the data should be in a common scale. Then, the overall score for the alternative A_i is equal to

$${\mathrm{WSM}}_{\mathrm{i}}={\sum }_{j=1}^J{w_j\bullet y}_{ij}.$$

The attribute with the highest WSM score is the best alternative.

One disadvantage of this method is that it can lead to compromises between the pillars; high performance in one criterion can compensate for low performance in another.

3.1.2. Technique for Order Preference by Similarity to an Ideal Solution (TOPSIS).

TOPSIS was initially proposed by Hwang & Yoon [43] as a compromising model, based on the distance principle. TOPSIS proposed as an optimal solution the one that has the shortest distance from the positive ideal solution and the longest distance from the negative ideal solution. The most used distance in TOPSIS evaluation is the Euclidean distance. However, other distances, such as Manhattan and Tchebichev [44] and others [45] can be used to calculate TOPSIS.

Let $v_{ij}=w_j\times y_{ij}$, i =1, …. I and j=1, …,J, the weighted and normalized rating and ${\mathrm{v}}_{\mathrm{j}}^{+},{\mathrm{v}}_{\mathrm{j}}^{-}$ $\mathrm{b}\mathrm{e}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{i}\mathrm{d}\mathrm{e}\mathrm{a}\mathrm{l}\mathrm{s}\mathrm{o}\mathrm{l}\mathrm{u}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}$ respectively. Then the overall score for the alternative $A_{\mathrm{i}}$ is

$${\mathrm{V}}_{\mathrm{i}}={\displaystyle \frac{\sqrt{{\sum }_{\mathrm{j}=1}^{\mathrm{J}}{\left({{\mathrm{v}}_{\mathrm{i}\mathrm{j}}-\mathrm{v}}_{\mathrm{j}}^{-}\right)}^2}}{\sqrt{{\sum }_{\mathrm{j}=1}^{\mathrm{J}}{\left({{\mathrm{v}}_{\mathrm{i}\mathrm{j}}-\mathrm{v}}_{\mathrm{j}}^{-}\right)}^2}+\sqrt{{\sum }_{\mathrm{j}=1}^{\mathrm{J}}{\left({{\mathrm{v}}_{\mathrm{i}\mathrm{j}}-\mathrm{v}}_{\mathrm{j}}^{+}\right)}^2}}}$$

Finally, alternatives are ranked with respect to ${\mathrm{V}}_{\mathrm{i}}$ values. A higher ${\mathrm{V}}_{\mathrm{i}}$ score value indicates a better alternative performance.

A main limitation of TOPSIS is the rank reversal phenomenon where by adding or removing an alternative solution, the initial ranking of the alternatives changes. Several modifications for addressing this problem have been proposed in the literature, e.g. [46,47]. TOPSIS with Manhattan distance does not appear to suffer from the rank reversal phenomenon [44] as it inherits this property from WSM, with both methods exhibiting similar behaviour when this distance metric is used. Additionally, in this method, the interpretation of the weights is not straightforward, as the distances from the positive and negative ideal solutions are weighted by the square of the initial weights ($w_j$²). For this reason, modified TOPSIS has been proposed.

3.1.3. Modified TOPSIS

The main difference between the modified TOPSIS and the original TOPSIS is that, in the modified version, the ideal solution is identified first, and then the weighted Euclidean distance is calculated [48]. Thus, the overall score for the alternative A_i is

$${\mathrm{V}}_{\mathrm{i}}={\displaystyle \frac{\sqrt{{\sum }_{\mathrm{j}=1}^{\mathrm{J}}{{\mathrm{w}}_{\mathrm{j}}\left({y_{\mathrm{i}\mathrm{j}}-y}_{\mathrm{j}}^{-}\right)}^2}}{\sqrt{{\sum }_{\mathrm{j}=1}^{\mathrm{J}}{{\mathrm{w}}_{\mathrm{j}}\left({y_{\mathrm{i}\mathrm{j}}-y}_{\mathrm{j}}^{-}\right)}^2}+\sqrt{{\sum }_{\mathrm{j}=1}^{\mathrm{J}}{{\mathrm{w}}_{\mathrm{j}}\left({y_{\mathrm{i}\mathrm{j}}-y}_{\mathrm{j}}^{+}\right)}^2}}}$$

The difference between the two methods is that in TOPSIS, ${\mathrm{w}}_{\mathrm{j}}^2$is used, contrary to modified TOPSIS where ${\mathrm{w}}_{\mathrm{j}}$ is used while calculating the distances from the positive and the negative ideal solutions. This helps maintain a proportional and more balanced impact of each attribute on the final decision, avoiding the exaggerated influence of higher-weighted attributes seen in the traditional TOPSIS method. Yet, The TOPSIS and the modified TOPSIS methods result in the same ranking when equal attribute weights are used.

3.1.4. VIKOR

VIKOR (VIseKriterijumska Optimizacija I Kompromisno Resenje, that means: Multicriteria Optimization and Compromise Solution) has been proposed by Opricovic [49,50,51]. Unlike TOPSIS, which prioritizes solutions closest to the positive ideal solution and farthest from the negative ideal, VIKOR focuses on a compromise solution by minimizing the distance to both ideal solutions. This approach appeals to risk-takers, potentially leading to more innovative or bold decisions.

Let $y_j^{\mathrm{*}}and{y}_j^{-}$ be the best and the worst values for each criterion

Step 1: Compute ${\mathrm{S}}_{\mathrm{i}}$ and ${\mathrm{R}}_{\mathrm{i}}$

${\mathrm{S}}_{\mathrm{i}}={\sum }_{\mathrm{j}=1}^{\mathrm{J}}{\mathrm{w}}_{\mathrm{j}}\bullet {\displaystyle \frac{{\mathrm{y}}_{\mathrm{j}}^{\mathrm{*}}-{\mathrm{y}}_{\mathrm{i}\mathrm{j}}}{{\mathrm{y}}_{\mathrm{j}}^{\mathrm{*}}-{\mathrm{y}}_{\mathrm{j}}^{-}}}$ and ${\mathrm{R}}_{\mathrm{i}}=\underset{\mathrm{j}}{\max }\left\{{\mathrm{w}}_{\mathrm{j}}\bullet {\displaystyle \frac{{\mathrm{y}}_{\mathrm{j}}^{\mathrm{*}}-{\mathrm{y}}_{\mathrm{i}\mathrm{j}}}{{\mathrm{y}}_{\mathrm{j}}^{\mathrm{*}}-{\mathrm{y}}_{\mathrm{j}}^{-}}}\right\}$

Step 2: Compute ${\mathrm{Q}}_{\mathrm{i}}=\mathrm{v}\bullet {\displaystyle \frac{{\mathrm{S}}_{\mathrm{i}}-{\mathrm{S}}^{\mathrm{*}}}{{\mathrm{S}}^{-}-{\mathrm{S}}^{\mathrm{*}}}}+\left(1-\mathrm{v}\right)\bullet {\displaystyle \frac{{\mathrm{R}}_{\mathrm{i}}-{\mathrm{R}}^{\mathrm{*}}}{{\mathrm{R}}^{-}-{\mathrm{R}}^{\mathrm{*}}}}$where ${\mathrm{S}}^{-}=\underset{\mathrm{i}}{\max }{\mathrm{S}}_{\mathrm{i}}$, ${\mathrm{S}}^{\mathrm{*}}=\underset{\mathrm{i}}{\min }{\mathrm{S}}_{\mathrm{i}}$, ${\mathrm{R}}^{-}=\underset{\mathrm{i}}{\max }{\mathrm{R}}_{\mathrm{i}}$, ${\mathrm{R}}^{\mathrm{*}}=\underset{\mathrm{i}}{\min }{\mathrm{R}}_{\mathrm{i}}$ and $\mathrm{v}$ a weighting reference, where 0 ≤ $\mathrm{v}$ ≤ 1.

Step 3: Rank alternatives, sorted by the values S, R, Q in ascending order and produce three ranking lists.

S, R, Q ranking lists will propose the compromise solution or a set of compromise solutions. If the attribute A’ has the best score with respect to Q list, it could be considered as the best solution if it satisfies the following conditions

C1: Q(A’’) – Q(A’) ≥ (m-1)^-1

where A’’ is the attributed with second-best score in Q list and m is the number of attributes and

C2: A’ has also the best score with respect to S and/or R list. If the above conditions are not satisfied, a compromise solution set results.

3.1.5. COPRAS

COPRAS (Complex Proportional Assessment) method was introduced by Zavadskas and Kaklauskas [52] and it can be described as follows:

Step 1. Normalize the initial rating $x_{ij}i=1...Iandj=1\dots J.$Let $y_{ij}i=1...Iandj=1\dots J$ be the normalized rating.

Step 2. Weight the normalized rating $y_{ij}$by proper weight $w_j,\mathrm{j}=1\dots \mathrm{J}\mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h}\mathsf{\Sigma }{w}_j=1$

Step 3. Calculate the ${\mathrm{P}}_i,R_i,Q_i$where: R_i the summation of the attributes that need to be minimized and P_i the summation of the attributes that need to be maximized

$$\mathrm{Q}\mathrm{i}=P_i+{\displaystyle \frac{{\sum }_{\mathrm{j}=1}^{\mathrm{J}}{R}_j}{R_i{\sum }_{\mathrm{j}=1}^{\mathrm{J}}\frac{1}{R_j}}},U_i={\displaystyle \frac{Q_i}{Q_{max}}}\bullet 100\mathrm{\%}$$

Alternatives are ranked based on $U_{\mathrm{i}}$ values. A higher score value on U indicates a better alternative performance.

A main limitation of COPRAS is that it is less robust than other MCDM methods, even in the case of small data changes [53]. This lack of robustness means that even minor variations in the input data, such as slight changes in the values of criteria or the performance of alternatives, can lead to significant fluctuations in the final rankings or outcomes.

3.2. Normalization Methods

Normalization is a crucial step in any decision-making problem as it transforms heterogeneous data into data on a common scale. Data normalization involves scaling the attribute values to make them lie numerically in the same scale and thus have the same importance on the final index. There exist numerous normalization methods in the literature [54], but in the current work, the following five widely used techniques were considered and compared (Table 2).

Let x be the initial value, and x′ represent the normalized value. The $min\left(x\right)$, $max\left(x\right)$, $mean\left(x\right)$, $median\left(x\right)$, $stdev\left(x\right)$, $IQR\left(x\right)$ correspond to the minimum, the maximum, the mean, the median, the standard deviation and the interquartile range value of the criterion respectively.

4. Results

4.1. MCDM Impact Results

In Table 3, the first 10 ordered configurations are presented obtained from the five different MCDM methods. The numbers correspond to the configuration IDs as described in [36]. The first column indicates the material combinations, as shown in Table 1. Moreover, three different scenarios have been considered: equal weighting among the five pillars of sustainability, prioritization of costs and performance, and prioritization of environment and performance. Colouring has been applied to help identify the similar configurations obtained across the five different MCDM methods. To improve clarity, the configuration colours derived from the WSM method are highlighted in the rankings of the other MCDM methods.

In the equal weighting scenario, most of the top-ranked design configurations consistently appear across all five MCDM methods, although their positions within the rankings varied. This suggests a strong consensus among the different MCDM methods, indicating that these top configurations are consistently deemed optimal or high-performing, regardless of the specific method used. More specifically, TOPSIS and modified TOPSIS yielded identical results, as mathematically proved, and their final rankings closely aligned with those of VIKOR. SWM and COPRAS produced similar ranking orders. When greater weight was assigned to cost and performance metrics compared to the others, modified TOPSIS and VIKOR exhibited nearly identical performance, though the rankings of other configurations differed. Finally, when greater weight was assigned to performance and environmental factors, the top ten ranked attributes were identical in COPRAS and VIKOR, although their exact positions differed. These two methods differed by only two attributes compared to the corresponding list from SWM.

To verify the similarity of the entire ordered lists resulting from the five different aggregation methods, a Pearson correlation analysis was conducted, considering the three different weighting schemes. The Pearson correlation coefficient describes the strength and the direction of a linear relationship between two variables, with values ranging from -1 to 1. A value of 1 indicates perfect positive correlation (all data points align in a straight line with a positive slope), and a value of -1 indicates perfect negative correlation (all data points align in a straight line with a negative slope). The correlation analysis results (Table 4) indicate a strong positive correlation between the alternative rankings, meaning that all methods demonstrated equivalent rankings.

To test the sensitivity of the five MCDM methods to small variations in the initial data, 1000 perturbed samples were generated. The initial values were perturbed by a Gaussian error with zero mean and a standard deviation equal to 1% and 10% of the standard deviation of the initial data. The simulated samples were then normalized and ordered. Figure 2a-f present the boxplots of the first five ordered attributes with respect to SWM for each of the considered MCDM method.

For the scenario with equal weights, both VICOR and COPRAS exhibited higher variability, even with small perturbations (1%) in the initial data, which is undesirable in this specific context. Conversely, the WSM demonstrated minimal variation in response to small perturbations, indicating stability in the resulting order. TOPSIS and modified TOPSIS were relatively stable under small data perturbations, although they were slightly more sensitive than WSM. As the perturbation of the initial data increased (to 10%), the variation in WSM also increased, which is a desirable characteristic in this scenario. TOPSIS and modified TOPSIS showed high sensitivity to larger perturbations, although they were less sensitive than WSM. Finally, COPRAS and VIKOR exhibited extreme sensitivity to the ranking order under large data variations.

When greater weights were assigned to cost and performance metrics, the Weighted Sum Model (WSM) exhibited low sensitivity to small data variations but higher sensitivity to significant data variations. This behavior was also observed in TOPSIS, Modified TOPSIS, and VIKOR. In contrast, COPRAS displayed very high sensitivity to both small and large data variations.

In the scenario where greater weights were assigned to environmental and performance metrics, WSM again demonstrated low sensitivity to small data variations and high sensitivity to significant data variations. However, the other methods showed sensitivity to both small and large data variations, with VIKOR and COPRAS being particularly sensitive to substantial data changes.

In conclusion, SWM appears to be a more suitable method compared to the others, as its ranking order exhibits minimal sensitivity to small data perturbations, while it becomes more sensitive to larger data variations. This characteristic is particularly desirable in the context of sustainability assessment, where stability in the ranking of attributes is essential when dealing with slight variations in the initial data. However, the ability to reflect greater sensitivity with larger perturbations allows SWM to capture more significant shifts in the data, ensuring that it can account for more pronounced changes in underlying trends.

4.2. Normalization Impact Results

Following the analysis on the impact of the MCDM methods, the effect of normalization methods on WSM performance was assessed. For this analysis, the most robust MCDM method from the previous step, i.e., the SWM method, was considered, and the impact of different normalization methods on this method was evaluated. In Table 5, the first ten design configurations are presented for the SWM method across five normalization techniques and three weighting scenarios: equal weighting, performance and cost prioritized, and environment and performance prioritized. For all weighting scenarios, the rankings were very similar across all studied normalization methods, with the L1 and L2 norms yielding nearly identical results.

This consistent ranking is further confirmed by the correlation analysis (Table 6), where all correlation coefficients are very close to one, indicating a high degree of similarity between the rankings for the different normalization methods. This is also the case for all the studied weighting scenarios.

To test the sensitivity of the five normalization methods to variability in the initial data, a simulation similar to that conducted for the MCDM methods was performed. The simulated data were ordered, and the boxplots of the first five ordered attributes with respect to Min-Max normalization method are depicted in Figures 3a-f, for each normalization method

In the equal weight’s scenario, the L1 norm and L2 norm normalization methods appear to be sensitive to both small and large data perturbations. The ranking using min-max normalization seems to be insensitive to small data perturbations (1%) but shows sensitivity to larger data perturbations (10%). Z-score and median normalization methods demonstrate moderate sensitivity to both small and large data perturbations.

In the scenario where greater weight was assigned to both cost and performance metrics, all normalization methods exhibit moderate sensitivity to both small and large data perturbations. However, min-max appears to be more sensitive to large data perturbations compared to the other methods, which is a desirable outcome.

Finally, in the scenario where greater weights was assigned to both environment and performance, min-max normalization appears to be less sensitive to small data perturbations compared to the other methods. Additionally, Z-score and median methods seem to have low sensitivity to small data perturbations. For larger data perturbations, the ranking using min-max becomes more sensitive compared to the other methods, while L1 norm and L2 norm show high sensitivity to both small and large data perturbations.

Based on the analysis above, min-max normalization emerges as the preferred method in several scenarios due to its balanced sensitivity to variations in the data. This normalization technique is particularly advantageous because it exhibits stability when dealing with small data perturbations, while also being sufficiently responsive to larger shifts in the data. In the context of sustainability assessment, where accurate ranking and sensitivity to changes are crucial, min-max emerges as the most robust and reliable choice due to its ability to manage a range of data fluctuations. Its capacity to maintain consistent and reliable rankings under varying conditions ensures that it can provide meaningful insights, even when faced with uncertainties or variability in the input data. This adaptability is essential for ensuring the integrity and accuracy of sustainability assessments over time.

5. Conclusions and Future Perspectives

This study investigates the impact of various MCDM tools on the decision-making process and examines how prominent normalization methods affect the final outcomes in the context of sustainability assessment, specifically during the design of aircraft structures. Using a dataset from the aviation sector, a typical aircraft component was selected as a case study for a holistic sustainability assessment. A detailed sensitivity analysis was conducted to explore the influence of MCDM tools, normalization methods, and weight variations on the holistic SI.

The results indicate that the ranking obtained from the considered MCDM methods was consistent, as demonstrated by the correlation analysis. This finding suggests that the choice of MCDM tool does not significantly affect the ranking in this context. However, the sensitivity analysis of the impact of data variation under different weighting scenarios revealed that the WSM is more sensitive to small data variations, while exhibiting desirable sensitivity to larger data variations. This characteristic is advantageous in sustainability assessment, as it aligns with the need for robust responses to significant changes in the dataset.

When considering normalization methods, the analysis focused on five widely used techniques. The effect of normalization on the WSM was assessed, with the ranking results showing consistency, as validated by the correlation analysis. Notably, the min-max normalization method demonstrated smaller sensitivity to small data variations and greater sensitivity to larger data variations. This property makes it a particularly robust choice for this context. Additionally, the bounded nature of results using min-max normalization (e.g., within the range 0-1) enhances interpretability, transparency, and comprehensibility.

Overall, the weighted sum model using min max method is justified for calculating the Sustainability Index in the conceptual design phase due to its ability to provide a simple, transparent, and flexible framework for decision-making under uncertainty. The weighted sum model's clarity, and ability to provide a first-order approximation of sustainability make it a practical tool for conceptual design. Future research should address potential drawbacks of the weighted sum model, particularly its compensatory nature, where strong performance in one criterion can offset poor performance in another. Introducing minimum thresholds for critical criteria could help ensure that design alternatives failing to meet essential sustainability requirements are excluded from consideration.