Sustainability Assessment of Machinery Safety in a Manufacturing Organisation – Supporting Machinery Safety Decision Making with AHP and CART Methods

1. Introduction

Safety Related Data (SRD) are essential to maximise the effectiveness of the risk assessment process. SRDs need to be as accurate and detailed as possible at all stages of the system life cycle, however, at the design stage they are the basis for achieving the required level of Occupational Health and Safety (OHS). According to Badri et al. [1], to improve the process of Safety Decision-Making (SDM) to achieve safety level, SRDs can be considered as the most valuable assets for all safety professionals. The combination of historical and current data strengthens real-time decision making during the manufacturing process, which positively affects the performance, safety, reliability, and sustainability of industrial systems [2].

In the field of machinery safety, it is very important to identify all hazards associated with machinery by collecting accurate and relevant safety data. Accurate Machine Safety Related Data (MSRD) analysis is needed to use machine safety knowledge and to make more realistic risk assessments. However, the lack of appropriate communication between machine manufacturers and end-users further magnifies the existing problem and hinders sufficient transfer of MSRD and feedback from end-user experience [3].

Taking care of employee safety and minimising the impact of hazards when in contact with machinery is a multi-dimensional and necessary prerequisite for any sustainable production. In addition to the technical, economic, health, social, motivational, or legislative aspects, it also has a highly ethical dimension. According to Lee et al. [4], most organizations strive for sustainability by providing workplace conditions that contribute to employee health and safety. Therefore, developing a workplace safety culture is a competitive factor for sustainable companies.

Managers at all levels of management are responsible not only for their subordinates but also for managing safety within their scope of responsibility [5,6,7]. Any misconduct by a manager is usually perceived with great sensitivity by the injured person, but also by the wider community. On the other hand, all relevant measures to improve OHS require incurring considerable costs. It is often economically prohibitive to carry out all remedial measures at once. From the organisation's perspective, it should be a process that ensures the cost acceptability of fulfilling the requirements to maintain and improve the safety of machinery. However, this is no cheap proposition. Particularly in larger manufacturing operations, which contain number of machines at different levels of wear and tear (resulting from lifetime) operating within several production lines, usually on a single shop floor, the sustainability of their safety is almost always a serious economic issue.

Machinery in manufacturing creates a dynamic environment, especially with respect to the requirements of Industry 4.0 [8] where quality, accuracy and efficiency are paramount. However, this also poses specific risks for their operators, but also for other people in the vicinity of the machine. Ensuring safety in manufacturing is not only a legal requirement [9], but also a moral obligation to protect employees and sustain operations. Leveson [10] defines "safety as the absence of accidents, where an accident is defined as an event involving unplanned and unacceptable loss". Safety is also associated with low and acceptable risk [11,12,13].

Nowadays, the concept of any development is associated with the concept of sustainability. Some authors point out [14,15] that occupational health and safety (OHS) and injury and illness prevention are issues that may raise concerns in terms of sustainability. Boileau [14] highlights three areas to consider in sustainable safety: integrating safety requirements at the system design stage, incorporating OSH into strategic plans, and implementing a safety culture into the organization. According to Hogevold et al. [16] and Dyllick and Muff [17], sustainable business is business that is economically, socially and environmentally responsible, without negative impacts on all these areas. Currently, the direction of business from small to large organisations is contributing to a more sustainable future.

However, if organisations fail to procure and maintain machine safety, provide adequate Personal Protective Equipment (PPE), ensure proper training for employees, or implement a safety culture, a variety of undesirable events that threaten the sustainability of the business are likely to occur. If an undesirable event occurs due to insufficient approaches to OHS management, the manufacturing organization may incur various costs. Many of these costs can be direct and immediate, but often more significant are the so-called indirect costs [18]. Indirect costs (“hidden” or “invisible”) of an undesirable event can include negative impact on the employer's reputation, lost time due to injury investigation, cleanup of the incident site, and reduced productivity and morale among employees [19].

Bearing the initial costs of implementing and complying with a comprehensive occupational health and safety program (e.g., with the support of an ISO 45001 OHS management system [20,21]), complying with regulations, and providing employees with proper training and PPE is much more likely to save more costs than the costs associated with the occurrence of an undesirable event [21].

However, safety in manufacturing is not just about compliance, it is about promoting a safety culture, minimising risks and optimising productivity. It is a multifaceted effort that requires a holistic approach [22]. By implementing strict safety procedures, investing in appropriate machinery safety devices, and promoting a safety culture, machinery can significantly reduce the number of undesirable events in the workplace, protect employees, and ensure business continuity. Safety can be understood in a generic sense as a fundamental pillar of any manufacturing organization that adds value to the business sustainability [23].

A sustainable approach to machine operator safety in an organisation involves integration of environmental, social, and economic aspects into OHS management to not only protect the health and safety of workers, but also to ensure the long-term sustainability of the business and to minimise environmental impact. This approach requires comprehensive planning, implementation, monitoring and continuous improvement of safety procedures and systems. The perception of safety as the opposite of risk, i.e. that the lower the risk, the higher the safety and vice versa, has been challenged by several authors. Already Möller et al. [24] concluded that it is necessary to go beyond thinking in this relationship of “safety as an antonym of risk”. A different approach to risk is taken by Aven and Renn [25], who view it as uncertainty and its impact on activities. It is clear that health and safety of machine operators should be seen as part of a broader corporate sustainability strategy (Corporate Social Responsibility, CSR). This approach involves considering safety measures in the context of environmental responsibility (e.g. minimising waste and emissions, efficient use of energy, etc.) and social responsibility (e.g. ensuring fair working conditions, promoting the health and well-being of employees) [26,27].

Integrating safety objectives into the overall sustainability goals of the organisation allows better coordination and more efficient use of resources. Sustainable safety emphasizes preventive measures that focus on preventing accidents and injuries before they occur. This includes comprehensive risk identification, analysis and assessment, as well as the development and implementation of innovative technological solutions and safety systems. A preventive approach also encourages investment in the development and training of employees to be better prepared to identify and respond correctly to potential risk situations in machinery. The safety of machinery operators in an organisation is a complex area that requires a multidisciplinary approach involving engineering, ergonomics, occupational psychology and legislation.

The aim of our research was: (1) to apply risk assessment procedures based on the analysis of available data on the status of 124 devices in the manufacturing organization (Machinery safety related data MSRD); (2) to support the estimation of the cost-effectiveness of selected measures to reduce unacceptable risks using the AHP method; and (3) to use the CART technique to support the decision making process (Machinery safety decision making MSDM) to assess the effectiveness and efficiency of the proposed measures.

2. Material and Methods

2.1. Machinery Safety Analysis

The research was conducted at an organisation that is a global manufacturer of healthcare products and technologies. It focuses on the manufacture of market-leading medical devices. These devices are in the areas of advanced wound care, ostomy care, continence and critical care, and infusion devices.

A key step to ensure safety when working with machinery is to identify the potential risk scenarios associated with its operation (hazard → hazardous situation / event → harm). This process “Machinery safety related data” (MSRD) involves analysing all data on the machine's operation, including potential failures, accidents or hazardous situations that could cause an accident. After identification of the hazards, a risk assessment must be carried out, i.e. to determine the probability of occurrence of harm related to the hazardous situation ($P$) and severity of harm ($S$). These estimates are usually represented by a numerical scale corresponding to a verbal expression.

Note: To understand when data is collected by the organization's staff, probability of hazardous situation occurrence ($P$) is simplistically understood as the total probability of hazard exposure to persons during a given activity on a machine, combined with the probability of occurrence of a hazardous event, which in describing an accident scenario is related to e.g. unintentional human error or machine failure resulting in harm.

Due to the variety of machines in operation at the company, and their life cycle (some machines were more than 20 years old), a risk assessment methodology (ISO 12100 [28], see Table 1, Table 2 and Table 3) was designed by the research team and agreed by the OHS management of the organization. The specific score value for the assessed risk parameters was estimated based on the available safety information for individual machines and consultations with safety engineers, operators, and maintenance staff.

Risk ($\mathrm{R}$) is the expressed combination of the probability of occurrence of harm related to the hazardous situation ($\mathrm{P}$) and the severity of that harm ($\mathrm{S}$) [28] as shown in Equation 1:

$$R=P\mathrm{*}S$$

(1)

Based on the assessment of $\mathrm{R}$ according to the risk matrix, risk minimisation measures are then developed and implemented (see Table 3).

Although the organisation had previously written operational documentation in accordance with the current legislation on machinery (Directive 2006/42/EC [29]), due to the age of the machinery and the statistics of OHS accidents in the organisation, the management was not satisfied with its relevance. A new “Machinery Risk Assessment” document was prepared by experts for each machine, which included an analysis and risk assessment of the machinery based on the OHS methodology approved by the management.

An example of machinery risk analysis is selected in Table 4. Based on the results of the analyses (MSRD) from all 124 machines, a total of 5260 hazardous situations (HSs) and their resulting risks were identified. To assess the effectiveness and efficiency of the measures, the expert team, in cooperation with OHS management, focused on those risks whose value was higher than 9 and less than 20. Values $\mathrm{R}$ that were less than 9 and greater than 20 were not considered. Lines with a value of less than 9 represented a lower priority risk of harm from machinery, lines with a value of more than 20 represented the highest non-comparable priority, the so-called STOP-STOP category (see Table 3), meaning immediate stoppage of machine operation (a highly unacceptable risk). The result was a database containing 489 data reaching the so-called critical risks, for which it was important to prioritize their reduction or elimination by adding additional measures. From this database, 49 machines (see Table 5) with the most frequently recurring types of hazardous situations (HS) were selected by the experts. These were type-non-homogeneous, variously worn-out machines, some of which are part of different production lines. They are arranged in the production halls in such a way that their individual danger zones overlap each other. Therefore, the analysis included an assessment of the safety of the machinery in relation to both its layout and its interconnection within the production lines. Despite the heterogeneity mentioned above, these are machines that process the same material, contain similar components and their operation is also very similar. Some types of hazardous situations were often repeated, others were identical in content, just described differently.

This allowed experts to introduce a standard designation for the identified hazardous situations ($\mathrm{H}\mathrm{S}$). A total of 10 main $\mathrm{H}\mathrm{S}\mathrm{i}$ types ($i=1,\dots ,10$) could be obtained from the 489 HSs after standardization, which were recurrent in the machine analysis. Table 6 shows the $\mathrm{H}\mathrm{S}\mathrm{i}$ types with their number of occurrences O on the machines (see Table 6).

Possibility of avoidance of harm or impending HS is a very important parameter for risk management by appropriate measures. The expert group proposed to complement relationship (1) with the parameter $\mathrm{P}\mathrm{N}\mathrm{D}$ - Probability of non detection of HS, which consider whether the undesirable situation ($\mathrm{H}\mathrm{S}$) can be prevented by preventive measures or only after its occurrence (so called corrective measures). The experts proposed to extend the risk estimation with one column labelled “Probability of non-detection of HS” ($\mathrm{P}\mathrm{N}\mathrm{D}$), inspired by the approach applied in Failure Modes, Effects and Criticality Analysis (FMECA) [30,31]. The scores of this column are described in Table 7, applying the Braglia [32] approach. In collaboration with the OHS organization experts, the relation for risk estimation was modified (see Equation (2)) and then applied to all 489 data, see Table 8.

Note: HS – Hazardous situations; PND - Probability of non-detection HS/E.

Table 8 contains the assessment of ${\mathrm{R}}_{\mathrm{M}}$ (Machine Risk) based on two parameters obtained from the performed risk analysis and the third parameter $\mathrm{P}\mathrm{N}\mathrm{D}$ which we have added based on Table 7. Specific quantification ${\mathrm{R}}_{\mathrm{M}}$ is given by the relation:

$$R_M=P*S*PND$$

(2)

This is a similar relationship to the one used in the FMECA method [30,31], with the only difference that the parameters mentioned above are estimated using a 5-step scale, whereas FMECA uses standard 10-step scales. The individual rows of Table 8 contain the above-mentioned main $\mathrm{H}\mathrm{S}\mathrm{s}$obtained from the safety analyses of the individual machines. Numerical values of the parameters $\mathrm{P}$, $\mathrm{S}$ a $\mathrm{P}\mathrm{N}\mathrm{D}$ represent the maximum of the values of similar parameters from the risk assessment analysis of the machinery for a particular combination $\mathrm{H}\mathrm{S}$. Each risk presents the possibility of loss for the organisation [19], the impact of which can be of different nature e.g. loss of customer, penalties, loss of reputation, etc.). For this reason, Table 8 has been extended with a column allowing to assign a parameter to determine the significance value of negative impacts ($\mathrm{E}$) from the point of view of the organisation's management when a given hazardous situation occurs ($\mathrm{H}\mathrm{S}$) in the performance of the activity (maintenance or operation) and severity of harm.

2.2. Estimation of Expected Cost Using the AHP Method

Estimation of the expected costs ($EC$) caused by the severity of harm (e.g., direct, and indirect costs associated with injury caused by machine hazardous situation) generally cannot be estimated with sufficient precision [33]. On the other hand, the above-mentioned costs are an important indicator of the effectiveness of measures that reduce or even eliminate the assessed risks. This problem can be solved e.g. by using the Analytic Hierarchy Process (AHP) multicriteria decision-making method. In this method, it is not necessary to accurately estimate the amount of expected costs for a specific criterion. It is sufficient if we can perform a pairwise comparison of the mentioned criteria [34].

AHP belongs to multicriteria decision making methods, with the level of information about criteria preferences based on pairwise comparisons [35,36]. It is a powerful and flexible multicriteria decision making tool for complex problems where both qualitative and quantitative aspects need to be considered. It allows analysts to organize the critical aspects of a problem into a hierarchical structure similar to a family tree. By reducing complex decisions to a series of simple comparisons and rankings and then synthesizing the results, it supports come to the best decision, but also provides a clear reason for the decisions made.

Once the AHP hierarchy was defined (see Figure 1), it was possible to prioritize among the elements within each level of the hierarchy of our researched model. As mentioned above, both qualitative and quantitative criteria can be compared using informal judgments to derive priorities.

Part (a) of the Table 9 presents the pairwise comparison matrix of expected costs $\mathrm{E}\mathrm{C}$ for all combinations of Hazardous situation/Effect ($\mathrm{H}\mathrm{S}$/$\mathrm{E}$) of the analysed HS and the severity of their $\mathrm{E}$ described in Table 8. Using the AHP method, the calculated priorities of each combination are expressed as Expected Cost ($\mathrm{E}\mathrm{C}$), which form part (b) of Table 9.

These are the values obtained using the following steps [37]:

• The calculation of the normalized comparison matrix was performed.
• The average weights (priorities) for each element were determined.
• Calculated maximum eigenvalue (${\lambda }_{max}$).
• Calculated specific coherence indices ($CI$) and coherence ratio ($CR$) to verify the consistency of the comparison matrix.

The normalized matrix Table 9 part (a) was used to determine the priorities of each HS/E combination (Priority vector). Since the normalized matrix is of order 15 x 15, we used the Random Index $\left(RI=\mathrm{1,59}\right)$for the calculation:

• The priority vector for each HS/E combination (Table 9 part (b)) represents the priority or weight of the relevant element compared to the other elements;
• The maximum eigenvalue represents the value ${\lambda }_{max}=\mathrm{16,886}$;
• The consistent index is equal to $CI=\mathrm{0,1347}$;
• The coherence ratio is equal to $CR=\mathrm{0,0847}$.

Given that the coherence ratio ($CR$) is lower than $\mathrm{0,1}$, the comparison matrix can be considered sufficiently consistent by AHP standards. This means that the identified priorities can be considered reliable in the evaluation of alternatives.

In the next step of the analysis, we performed pairwise comparisons at the first level of the hierarchy to obtain value judgments. Table 10 part (a) shows the matrix A obtained from the pairwise comparison of criteria performed with the help of experts and OHS management. The comparison was performed in terms of the 5 criteria described in the previous section. The prioritization of the performance criteria was calculated similarly to the previous case. The prioritization vector (Table 10 part (b)) was determined by calculating the normalized components of the right eigenvector of the corresponding matrix.

The values in the Table 10 represent pairwise comparison weights, indicating how one criterion was evaluated against another. On the diagonal, the default values are 1 because each criterion is equally important in comparison to itself.

Similar to the first set, we proceeded as follows to calculate the priorities and consistency of this AHP matrix:

• We filled the diagonal with units and mirrored the values in the lower half of the matrix to the upper half to make it symmetric.
• We have normalized the matrix.
• We calculated the average weights (priorities) for each criterion.
• We verified the consistency of the matrix by calculating the consistency ratio ($CR$).

In the calculation described above it was calculated:

• The maximum eigenvector is ${\lambda }_{max}=\mathrm{5,1835}$;
• Consistent index is equal to $CI=\mathrm{0,0459}$;
• The coherence ratio is equal to $CR=\mathrm{0,0409}$.

Coherence ratio ($CR$) is again lower than $\mathrm{0,1}$, so the comparison matrix is considered sufficiently consistent by AHP standards. Also in this case, the identified priorities can be considered reliable in the evaluation of alternatives.

Table 10 part (b) shows the priorities for each criterion, i.e. their degree of importance, are as follows:

• Probability of HS: $P=\mathrm{0,2084}$;
• Severity of Harm: $S=\mathrm{0,1802}$;
• Number of HS occurrences: $O=\mathrm{0,2177}$;
• Probability of non-detection of HS: $PND=\mathrm{0,0889}$.
• The most important are the expected costs: $EC=\mathrm{0,3048}$;

The value of the inconsistency ratio is equal to $\mathrm{0,04}<\mathrm{0,1}$, i.e. it is acceptable. We call this variant of pairwise comparison Variant A.

After evaluating the different causes of undesirable events (HS) with respect to the criteria under consideration, all the evaluations must be aggregated in a hierarchical tree. Table 12 shows the priority of the probability of occurrence $\mathrm{H}\mathrm{S}$ with respect to the five criteria from a local and total perspective (i.e. criterion priority multiplied by local cause priority ($\mathrm{H}\mathrm{S}$)). It means, in the next step we set local and then total priorities for all 5 risk management parameters. The three “quantifiable” criteria can be evaluated by normalizing the quantitative assessments. These are the risk estimation parameters $\mathrm{P}$, $\mathrm{S}$ a $\mathrm{P}\mathrm{N}\mathrm{D}$ (see Equation (2)). The normalization is performed by using the sums of all their scores (see the last row of Table 9). For example, for the parameter $\mathrm{P}$ is the value of $\mathrm{T}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}=49$. The local prior for the parameter $\mathrm{P}$ and the value of $\mathrm{E}1$ we calculated as $3/49=\mathrm{0,061}$. We followed a similar calculation for all rows for the three risk parameters mentioned above ${\mathrm{R}}_{\mathrm{M}}$. In the same way, we calculated the row values of the local priority for the parameter $\mathrm{O}$. The local prior for the value of $\mathrm{E}1$ and parameter $\mathrm{O}$ we calculate as $41/489=\mathrm{0,084}$. The local priority for the parameter $\mathrm{E}\mathrm{C}$ is represented by the priority vector described in Table 10 part (b). In the following step, we computed the total priorities for each parameter. We used the values from the vector of priorities described in Table 11 part (b) for the calculation. E.g. for the value of $\mathrm{E}1$ a parameter$\mathrm{P}$ we calculated the total prior of the parameter $\mathrm{P}$ as the product of the local priority times the corresponding priority from the aforementioned priority vector, namely $\mathrm{0,061}\mathrm{*}\mathrm{0,2084}=\mathrm{0,013}$.

We followed a similar procedure for calculating the other values of the total priority parameter $\mathrm{P}$, but also all other parameters. After calculating all 5 total priorities by multiplying them in the corresponding row, we obtain the values of the last column of Table 11 under the name “Evaluation”. As soon as the prioritization of the performance criteria is achieved by composing pairwise comparisons, the last step involves using the AHP framework to evaluate the different causes of failure. The last column of Table 11 shows the final ranking of the 15 causes/effects considered, i.e., the hazardous situations analysed and their effects ($\mathrm{H}\mathrm{S}$/$\mathrm{E}$).

As indicated in the table above, falling objects during machine maintenance ($HS5$) in terms of expected costs E5 resulting from the severity of harm when this hazardous event occurs, it emerges as the most critical problem among the 15 alternatives with an overall priority score of $\mathrm{0,092}$. This is followed by ignition of combustible material during maintenance resulting in fire E2.1 with a value of $\mathrm{0,091}$ and ignition of combustible material E2.2 resulting in explosion, similarly with a value of $\mathrm{0,087}$ during maintenance of the machine.

As can be seen, in the case of E5 the final result of this analysis, is different than the results of the previous analysis based on the risk calculation alone $R_M$ (Table 8). The most critical undesirable events are E2.2 with a corresponding value of $R_M=4*5*5=100$ and E2.1 with the value $R_M=4*5*4=80$. These represent a high degree of probability of an undesirable event occurring that is both very serious and difficult to detect. This apparent contradiction is due to the way the table for Variant A has been constructed.

2.3. Sensitivity analysis

The solution based on the Table 10, for Variant A, reflected a possible scenario where the most important criteria are the expected costs ($EC$) and the number of undesirable event occurrences ($O$). In general, the model solutions may change in line with changes in the logic of the approach, given the focus of the analysis.

The sensitivity or robustness of the model solution to potential shifts in strategy priority creates the opportunity to analyse the sensitivity of the criteria weights to changes in the priority (relative importance) of the weights. In fact, each criterion is characterized by an important degree of sensitivity, i.e., the ranking of all causes of failure changes dramatically across the weight range [38]. Therefore, it is important to control to what extent the evaluation ratings changes may lead to more or less significant changes in the final order of priorities. Sensitivity analysis is used to examine the sensitivity of alternatives to changes in the priority of criteria immediately below the target. The proposed analysis highlights the priorities of the five criteria used in the model and how a change in the priority of one criterion affects the priorities of the others. Clearly, as the priority of one criterion increases, the priorities of the remaining criteria must decrease in proportion to their original priorities, and the global priorities of the alternatives (see Table 11) must be recalculated.

Table 12 shows the priorities for each criterion, which, in that order, are the most important:

• The chance of failure (Probability of HS): $P=\mathrm{0,127}$;
• Severity of Harm: $S=\mathrm{0,1802}$;
• Number of HS occurrences: $O=\mathrm{0,369}$;
• The chance of not detecting an undesirable event: $PND=\mathrm{0,08}$;
• Expected costs: $EC=\mathrm{0,279}$.

This variant of the pairwise comparison will be named as Variant B.

In the calculation described above it was calculated:

• The largest eigenvector is ${\lambda }_{max}=\mathrm{5,587}$;
• The consistency index is $CI=\mathrm{0,072}$;
• The consistency ratio is $CR=\mathrm{0,064}$.

Consistency ratio ($CR$) is less than $\mathrm{0,1}$, which is the generally accepted threshold for an acceptable level of consistency in AHP analysis. This means that the comparisons between criteria in this analysis are sufficiently consistent and the priority results can be considered as reliable.

From Table 13 we can see the priorities for each criterion, which are the most important in that order:

• Severity of harm: $S=\mathrm{0,395}$;
• The chance of failure (undesirable event): $P=\mathrm{0,258}$;
• Number of HS Occurrences: $O=\mathrm{0,16}$;
• Expected costs: $EC=\mathrm{0,109}$;
• The chance of not detecting an undesirable event: $PND=\mathrm{0,078}$.

This variant of pairwise comparison will be named as Variant C.

Similarly to the previous cases, we verify the consistency of the matrix by calculation:

• The largest eigenvector: ${\lambda }_{max}\mathrm{}=\mathrm{5,331}$;
• Consistency index: $CI=\mathrm{0,083}$;
• Consistency ratio: $CR=\mathrm{0,074}$.

Consistency ratio ($CR$) lower than $\mathrm{0,1}$ indicates that the comparison matrix is sufficiently consistent for the purposes of AHP analysis. This indicates that the comparisons between criteria are consistent and the results of the analysis are reliable (see Table 14).

We note that the sensitivity analysis is only relevant for the priorities of the five criteria. If we were to change the values of the only one attribute at a time, one at a time, only the "main effects" would be considered, thus the "interaction effects" of changes in two or more of the weights would be neglected. We therefore adopted an approach whereby we varied different combinations of parameters for each variant, depending on the focus of the analysis and in consultation with OHS management.

Even after different, radical changes in preferences in pairwise comparisons, some robustness can be observed for variants A, B and C. Albeit in a different order, the first four riskiest events in all variants are identical (E2.1, E2.2, E3, E5). Moreover, this (except for the E3 case) corresponds in some way to the assessment based on ${\mathrm{R}}_{\mathrm{M}}$ (see penultimate column of Table 8 or the last two columns of Table 14). We note that in the original evaluation based on ${\mathrm{R}}_{\mathrm{M}}$ only the three parameters $\mathrm{P}$, $\mathrm{S}$ and $\mathrm{P}\mathrm{N}\mathrm{D}$ were considered. Thus, the other two parameters were not considered $\mathrm{E}\mathrm{C}$, i.e. the expected unit cost of the alternative, as well as $\mathrm{O}$, i.e. the frequency of occurrence of the given alternative across all the machinery. In the third place, when assessing the parameter ${\mathrm{R}}_{\mathrm{M}}$ three alternatives E2.4, E4.1 and E8 were recorded immediately, but when evaluated according to the five parameters (Variants A to C) their priority was ranked at 4 and above. On the other hand, the parameter E3 was ranked ${\mathrm{R}}_{\mathrm{M}}$ ranked as high as 9, but when evaluated on a variant-by-variant basis, it was ranked up to rank 4. This apparent mismatch in priorities is due to a more complex view, where alternative E3, although with a low-risk number $R_M=36$, has an order of magnitude higher occurrence than the other alternatives.

3. Results of the Evaluation of the Effectiveness and Efficiency of the Proposed Measures

In the previous section, we used the AHP method to determine the priority of each $\mathrm{H}\mathrm{S}$/$\mathrm{E}$ according to the severity of the overall impacts on the whole group of considered machinery. Thus, it was about the amount of additional costs or losses that could potentially be incurred by the organization with respect to each of the considered alternatives. In this section we focus on the performance of the individual measures that have been proposed within the framework of the aforementioned "Machinery Risk Assessment" document. This is an important element in ensuring the sustainability of machinery safety. The MSRD process involves, in addition to the assessment of the assigned risks, the proposal of more or less effective and efficient measures. Within the framework of the document referred to above, experts have defined the most appropriate types of measures $\mathrm{M}\mathrm{s}$for each $\mathrm{H}\mathrm{S}$/$\mathrm{E}$.

This means that for each of the 489 identified $\mathrm{H}\mathrm{S}$ specific measures have been proposed as well as estimated how these measures will reduce the resulting risk number. We will refer to this reduced risk number as${\mathrm{R}}_{\mathrm{A}\mathrm{M}}$.

After consideration of the OHS by management and evaluation of all the proposed measures, it was found that in many cases they are repetitive and can be standardly divided into 7 basic types (see Table 15). All proposed measures represent some combination of the aforementioned measures and risk management hierarchy according to NIOSH (National Institute for Occupational Safety and Health) [39].

Two of them, i.e. $\mathrm{M}6$ a $\mathrm{M}7$ do not require any additional cost increase. These are measures that are routinely implemented in the organisation. However, they have almost always been recommended in risk analyses only because many accidents or hazardous situations have been reported where these generally legislated safety measures have not been followed. Compliance with these two measures is therefore considered to be the absolute basis for the risk management of machinery and has therefore not been taken into account in the following assessment.

Based on consultation with OHS management, we estimated the expected costs associated with implementing each machine and each of the five proposed measures. We note that the costs for each measure were estimated only as an approximation at approximately the same time, without taking into consideration possible material price fluctuation and labor changes, etc. The costs associated with the introduction of each measure were influenced not only by the specific machinery, but also by its location in the plant or its proximity to other machinery. Table 16 shows the minimum and maximum estimated unit costs. Thus, these are the CM costs budgeted for each event, the machinery as well as the type of measure proposed.

The effectiveness of the measures was assessed by the so-called index $\mathrm{A}\mathrm{E}\mathrm{I}$ (Action Effectiveness Index), whereby $\mathrm{A}\mathrm{E}\mathrm{I}$ is a metric used to assess the effectiveness of proposed or implemented corrective actions in reducing risk ${\mathrm{R}}_{\mathrm{M}}$. It aimed to quantify whether the proposed actions $\mathrm{M}\mathrm{i}$ reduce risk or improve detection capability $\mathrm{H}\mathrm{S}$ before it could have a negative impact on the machinery operator.

Calculation of the index $\mathrm{A}\mathrm{E}\mathrm{I}$ is based on a comparison of the values of ${\mathrm{R}}_{\mathrm{M}}$ before and after the implementation of the measures, in our case ${\mathrm{R}}_{\mathrm{A}\mathrm{M}}$(AM – after measure). The index of effectiveness of the measures ($\mathrm{A}\mathrm{E}\mathrm{I}$) can thus be calculated as a percentage change that shows how much the risk (${\mathrm{R}}_{\mathrm{M}}$) has been reduced as a result of the implementation of the measures. The following relation applies:

$$AEI=\frac{R_M-R_{AM}}{R_M}*100\%$$

(3)

This calculation provides a quantitative indicator of measures effectiveness. If a high percentage is achieved it is clear that the measures have significantly reduced the risk. On the other hand, a low or negative percentage could indicate that the measures have not been sufficiently effective or that the risk has not changed or has even increased.

It is important to note that the specific calculation methods and values may depend on the specific practices and policies of the organization. Therefore, when calculating $\mathrm{A}\mathrm{E}\mathrm{I}$ it is always advisable to follow internal guidelines. Calculation of the index $\mathrm{A}\mathrm{E}\mathrm{I}$ was used to estimate the effectiveness of the implementation of the 5 proposed measures mentioned above ($\mathrm{M}1$ to $\mathrm{M}5$). We note that for each $\mathrm{H}\mathrm{S}$/$\mathrm{E}$ were almost always a combination of several of the 5 measures $\mathrm{M}1$ to $\mathrm{M}5$. Thus, the estimation of the resulting costs implied an accumulation of the mentioned values. Therefore, for $\mathrm{M}\mathrm{i}$($\mathrm{M}1$ to $\mathrm{M}5$) we defined the value $\mathrm{C}\mathrm{M}\mathrm{i}$ for each $\mathrm{H}\mathrm{S}$/$\mathrm{E}$. These are the costs that potentially need to be incurred in relation to the measure $\mathrm{M}\mathrm{i}$. If any measures $\mathrm{M}\mathrm{i}$ are not for a given $\mathrm{H}\mathrm{S}\mathrm{i}$ proposed, the value $\mathrm{C}\mathrm{M}\mathrm{i}$ ($i=1,\dots ,5$) is zero. The resulting expected costs $\mathrm{C}\mathrm{M}$ for each $\mathrm{H}\mathrm{S}$/$\mathrm{E}$ are therefore given by the relation:

$$CM=\sum _{i=1}^{5}CMi$$

(4)

The costs assigned to each measure as well as the values $\mathrm{P}$ a $\mathrm{S}$ described in relation (1) were used as predictor variables in the Classification and Regression Trees (CART) method. This is a method focusing on regression trees to model the relationship between the dependent variable (in our case $\mathrm{A}\mathrm{E}\mathrm{I}$) and several predictor variables, i.e. $\mathrm{P}$, $\mathrm{S}$, $\mathrm{C}\mathrm{M}1$ to $\mathrm{C}\mathrm{M}5$).

3.1. Applying the CART Method for Assessing Effectiveness of $Mi$

This is a 7-node CART regression analysis, focusing on regression trees to model the relationship between the dependent variable ($\mathrm{A}\mathrm{E}\mathrm{I}$) and the predictor variables ($\mathrm{S}$, $\mathrm{P}$, $\mathrm{C}\mathrm{M}1$ to $\mathrm{C}\mathrm{M}5$). The key parameters of the CART analysis include [40]: Node Splitting, Optimal Tree, Model Validation, R_squared, RMSE (Root Mean Squared Error) and MSE (Mean Squared Error), MAD (Mean Absolute Deviation) and MAPE (Mean Absolute Percent Error), which allow alternative measures of prediction error, important for understanding model accuracy in practical terms.

Based on previous analyses related to machine risk assessment and their impacts on the organization, specified criteria for classifying subjects into the best terminal nodes were developed as predictor variables and their thresholds define the division of data and the assignment of cases to different subgroups for prediction. These criteria and the assignment to nodes were the basis for creating a model that maps the relationships between the predictor variables and the dependent variable. Minitab software was used to analyse the regression decision tree and applied to an extended database with 489 rows. As mentioned above, we considered the response index $\mathrm{A}\mathrm{E}\mathrm{I}$ and the predictor parameters $\mathrm{P}$, $\mathrm{S}$, a $\mathrm{C}\mathrm{M}1$ to $\mathrm{C}\mathrm{M}5$ characterizing the expected unit cost for one machine, one event, and the specific measure proposed.

Figure 2 is a graphical representation of the dependence between the number of terminal nodes in the network and the R-squared value. R-squared is a statistical measure that expresses the degree to which the variability of the dependent variable is explained by the independent variable. In this case, the dependent variable is the number of terminal nodes and the independent variable is R-squared. Axis $\mathrm{X}$ shows the number of terminal nodes in the network, with the values on the axis $\mathrm{X}$ range from 2 to 11. $\mathrm{Y}$ shows the value of R-squared and the values on the axis $\mathrm{Y}$ ranges from 0 to 85%. The curve in the graph shows the trend of the relationship between the number of terminal nodes and the R-squared value.

The curve has an increasing trend, indicating the number of terminal nodes increases, the R-squared value also increases. The optimal R-squared value, which is 84.08%, is also indicated in the graph. This value has been achieved with 8 terminal nodes. The graph shows the number of terminal nodes increases, the R-squared value also increases to explain the variability in the number of terminal nodes. This indicates the number of terminal nodes increases, the model becomes more accurate in predicting the number of terminal nodes. The optimal value of R-squared indicates the model is most accurate in predicting the number of terminal nodes at this point. Increasing the number of terminal nodes above 8 would lead to a decrease in the accuracy of the model. This shows there is a strong dependence between the number of terminal nodes and the R-squared value.

The overall importance of the nodes in the CART tree (see Figure 3) lies in their ability to partition the dataset into predictively homogeneous groups, allowing for more accurate predictions and a better understanding of the relationships in the data. It describes the specific data in the tree nodes and the criteria for classifying subjects into the best terminal nodes within the CART model.

Figure 4 compares the relative importance of each cost measure $\mathrm{C}\mathrm{M}1$ to $\mathrm{C}\mathrm{M}5$ and parameters $\mathrm{S}$ a $\mathrm{P}$. It shows that the costs of the measures $\mathrm{C}\mathrm{M}4$, $\mathrm{C}\mathrm{M}1$ a $\mathrm{C}\mathrm{M}2$ have a large impact on the formation of the predictive model.

, $\mathrm{S}$ and $\mathrm{C}\mathrm{M}1$ to $\mathrm{C}\mathrm{M}5$ (source: own design).

The following Table 17 provides quantitative measures of model accuracy, i.e., the performance of the CART regression model on the training and test sets. The low values of RMSE, MSE, MAD and MAPE indicate good prediction accuracy of the model.

The R-squared (86.75% on the training set and 84.08% on the test set) indicates the model explains the variability of the dependent variable relatively well. Overall, these figures show the CART model can accurately predict the dependent variable with a high degree of accuracy and has low prediction errors on both the training and test sets. The following Table 18 describes the criteria for the 7 Terminal Node.

The greatest efficiency in terms of the greatest percentage of the index value $\mathrm{A}\mathrm{E}\mathrm{I}$ is represented by terminal node 4, specifically 75%, but on the other hand, this node refers to the application of measures $\mathrm{H}\mathrm{S}$/$\mathrm{E}$. The path to it from the root goes through node 2 and node 4, i.e. that for each of the 4 events it is necessary to spend $\mathrm{C}\mathrm{M}4$ (from 0 to 50 Euros) + $\mathrm{C}\mathrm{M}2$ (from 44 to 88 Euros) + $\mathrm{C}\mathrm{M}1$ (78 Euros). This can be interpreted as the implementation of measures $\mathrm{M}4$: Education - training, $\mathrm{M}2$: E- STOP and $\mathrm{M}1$: LOTO - Safety padlock at unit costs ranging from 132 Euros to 216 Euros. This is therefore a high efficiency of the measures in relation at a relatively low cost. The problem is, as we have already mentioned, that only 4 undesirable events ($\mathrm{H}\mathrm{S}$/$\mathrm{E}$) are affected. Another of the nodes Terminal Node 7 achieves success in reducing the criticality of the event with measures M4 and M3 at corresponding costs ranging from 1200 Euros to 1500 Euros. In a similar way, we calculated the unit cost for the proposed measures with respect to their effectiveness as measured by $\mathrm{A}\mathrm{E}\mathrm{I}$ for all 489 undesirable events ($\mathrm{H}\mathrm{S}$/$\mathrm{E}$) considered. Thus, using the CART method, we divided the individual events $\mathrm{H}\mathrm{S}$/$\mathrm{E}$ into relatively homogeneous groups according to the relationship between $\mathrm{A}\mathrm{E}\mathrm{I}$and costs $\mathrm{C}\mathrm{M}$ (divided into 5 groups according to each measure; $\mathrm{M}1$ up to $\mathrm{M}5$) as well as parameters $\mathrm{P}$ a $\mathrm{S}$. We note that the parameters $\mathrm{P}$ a $\mathrm{S}$ were the basis for calculating the risk number $\mathrm{R}$ and thus naturally influence the parameter $\mathrm{A}\mathrm{E}\mathrm{I}$. In the following, we show how both applied methods can be used simultaneously.

In Chapter 2.3, we performed the ranking of the results of the AHP method for the 5 investigated parameters and 3 different pairwise comparison matrices (Variant A, B, C, Table 14). By comparing the aforementioned rankings, we found independently of the different variants, there are four basic types of negative effects (namely E2.1, E2.2, E3 and E5) once they occur, will imply the largest total costs (losses) related to failure. In the following, we have therefore reduced the identified 489 $\mathrm{H}\mathrm{S}$/$\mathrm{E}$ to 4 types of significant undesirable events mentioned. The resulting reduced database contained 188 $\mathrm{H}\mathrm{S}$/$\mathrm{E}$that corresponded to any of the 4 types of undesirable events mentioned above. We then applied the CART method to this database. We again used the index as the response $\mathrm{A}\mathrm{E}\mathrm{I}$, parameters were used as predictors $\mathrm{P}$, $\mathrm{S}$, $\mathrm{C}\mathrm{M}1$ to $\mathrm{C}\mathrm{M}5$. The results of the regression decision tree are presented in Figure 5.

We describe the resulting model parameters as follows. The R-squared (91.04% on the training set and 84.08% on the test set) indicates the model explains the variability of the dependent variable relatively well. Overall, these figures show that the CART model is able to accurately predict the dependent variable with a high degree of accuracy and has low prediction errors on both the training and test sets. The following Table 19 describes the criteria for the 3 Terminal Node.

The final evaluation of the decision tree is presented in summary in the following Table 20. It can be concluded which undesirable events can be expected to result in the greatest cost savings, i.e. both the cost of failure (losses from undesirable events) and the cost of the measures that will most reduce the existing risk. The presented results are obtained by the AHP method for 3 different variants A, B and C of the pairwise comparisons matrices. The table shows the order of importance of the 4 events with the highest priority for all three variants. These are the 188 events selected for E2.1, E2.2, E3 and E5. Three Terminal Nodes were obtained from the CART technique applied to the set of all events of these types. Each of them represents the percentage of expected criticality reduction for each event. The table also lists the measures that are expected to result in a given criticality reduction. The unit and total costs that will be incurred by implementing the prescribed combination of measures are also included in the table.

From the table is clear the event with the highest efficiency has the designation E5 and a frequency of 51. Approximately 65% reduction in criticality for all events of this type requires the implementation of combinations of measures M1, M2 and M4 with a unit cost of 666 Euros and a total cost of 33 966 Euros. In other words, up to about 65% reduction in criticality for the 51 most serious failures, as assessed by all three variants, can be achieved with the measures $\mathrm{M}1$, $\mathrm{M}2$ a $\mathrm{M}4$ at the lowest possible unit cost of € 666. With even lower unit costs, it is possible to implement the measures $\mathrm{M}1$, $\mathrm{M}2$ a $\mathrm{M}3$ to reduce by 1/3 from 46 events E3. We can estimate the costs for the other events in the same way. If the management of the organisation decides to reduce all 188 most critical events, it would need to cover costs of about 192 776 Euros.

4. Discussion

The presented five-parameter machinery safety analysis appears to be an effective tool to perform a comprehensive risk assessment analysis of machinery. The proposed procedure allows obtaining a ranking of the causes and consequences of undesirable events, which include several types of information ($\mathrm{P}$ - Probability of hazardous situation; $\mathrm{P}\mathrm{N}\mathrm{D}$ - Probability of non-detection of HS; $\mathrm{S}$ - Severity of Harm; $\mathrm{E}\mathrm{C}$ - Expected cost; $\mathrm{O}$ - Number of HS Occurrence). The approach based on the implementation of the AHP method for multi-attribute analysis provides a framework in which the selection of the most critical undesirable events can be performed using different relevant characteristics. The AHP method helps safety analysts to work in a systematic and analytical manner, addressing each aspect of the failure in a stepwise hierarchy. The qualitative and subjective judgments of multiple individuals can be incorporated into the prioritization process. Using a series of paired evaluations, the AHP can obtain a direct (quantitative) assessment based on intangible (qualitative) criteria. Specifically, the problem is how to assign scores based on tables listing vague and unreliable verbal ratings. However, the great advantage is that if reliable quantitative judgements are available for some criteria, they can easily be included in the AHP analysis. This possibility means that our proposed procedure can replace, complement or integrate into a more comprehensive approach to safety studies.

Using the AHP method and the 5 basic parameters, we have shown a way to estimate the most critical undesirable events ($\mathrm{H}\mathrm{S}$/$\mathrm{E}$). Risk management is based on $\mathrm{E}\mathrm{C}$ and is applied to the whole group of machinery also based on the estimation of the parameter $\mathrm{O}$. The result of such an approach is a procedure that allows designing an effective plan of action to minimize the expected losses due to undesirable events in an organization.

Another way to increase the effectiveness of machinery safety management is to maximise the effect of the measures taken at a cost-effective cost. For this purpose, we apply the CART technique. As a response we selected the parameter $\mathrm{A}\mathrm{E}\mathrm{I}$, which evaluates the effectiveness of the proposed or implemented corrective actions in reducing the risk of failure. As predictors, we worked with expected unit cost for each machine (cause or effects). In the case study we conducted that it is possible to develop a model we proposed, which can predict which group of measures, at what expected unit cost, will reduce the criticality of a particular undesirable event to what extent.

5. Conclusions

Machinery safety should be a priority for the management of every operation in an organisation. As significant resources are usually required to ensure safety or reduce risks, it is appropriate to carefully consider the priority of the relevant measures as well as the expected costs of their implementation. We have shown it is possible to determine, on the basis of existing safety analyses, in a relatively simple way, those undesirable events, the fulfilment of which entails the greatest risk of an often-unexpected increase in costs (losses). Although it is almost impossible to define these costs in an exact way, we have shown that by using the AHP multicriteria decision-making method, it is possible to prioritize them using pairwise comparisons. Increasing the efficiency of machinery safety assurance can also be achieved by efficiently spending resources on individual proposed measures aimed at reducing the risk of undesirable events. Using the CART tool, we have identified how the cost of individual measures affects the degree of criticality reduction for each type of undesirable event. Overall, the CART model in this analysis provides an accurate and reliable technique for prediction, indicating its suitability for the data analysed even when expanding operations to include similar machinery.

Finally, we have shown how the combination of the AHP method and the CART technique can be used to extract and cost-estimate those measures that will most effectively reduce both the unit and total cost of failures within the operations of a manufacturing organization.