Determine the Evaluation Rubric Weights using the Analytical Hierarchy Process Method for Oil and Gas Company Strategic Project

1. Introduction

XYZ Company, one of Indonesia's leading companies in the oil and gas sector, is planning a strategic project to transfer part of the demand from the ITJ Fuel Terminal (FT) to the JIGT Fuel Terminal (FT). This project aims to enhance the efficiency and effectiveness of the company's operations in response to the growing market demand. However, despite revenue analysis indicating that the move might not be immediately profitable, XYZ Company acknowledges the importance of considering other factors in decision-making.

Pure revenue analysis is not sufficient to determine the feasibility of the demand transfer project. Therefore, a comprehensive evaluation rubric is needed to assess this project from various perspectives. By using an evaluation rubric that encompasses several key aspects, it is hoped that the decision regarding the project's feasibility can be made more holistically and objectively.

1.1. Research Objectives

This research aims to:

• Develop a fair and comprehensive evaluation rubric to assess the feasibility of the demand transfer project from FT ITJ to JIGT.
• Calculate weights for each evaluation aspect using the Analytical Hierarchy Process (AHP) method.
• Evaluate the project based on five main aspects to determine whether the project is feasible to proceed, not only from a financial perspective but also from operational and strategic viewpoints.
• Provide recommendations based on the evaluation results to support management's decision regarding the continuation of the project.

In this project, the demand to be transferred from FT ITJ to JIGT will be evaluated based on three different scenarios. Each scenario considers various operational and strategic factors to provide a complete picture of the impact of the transfer.

1.2. Aspects for Evaluation

The evaluation rubric includes five main aspects that are considered crucial for assessing the feasibility of this project. These aspects are:

• Safety and Asset Integrity:

a.

Safety: Assessing the level of operational safety to be implemented at the new terminal.

b.

Asset Integrity: Evaluating the integrity of existing assets and the potential impact of the transfer on asset condition.

c.

Reliability: Examining the operational reliability of the terminal in supporting distribution needs.

d.

ESG (Environmental, Social, Governance): Considering the environmental, social, and corporate governance impacts.

2.

NPV Efficiency: Evaluating the efficiency of the Net Present Value (NPV) of the project to ensure that the investment provides optimal added value.

3.

Operational Supply:

a.

Supply Options (ships, pipelines, and trucks): Assessing various options for fuel supply, including ships, pipelines, and trucks.

b.

Port Capacity: Evaluating the port's capacity to handle the transferred volume.

c.

Storage Capacity: Assessing the available storage capacity to meet demand requirements.

4.

Operational Distribution:

a.

Weighted Distance to Consumers: Measuring the weighted distance from the terminal to end consumers to assess distribution efficiency.

b.

Distribution Options (multi-product single destination capability): Evaluating the terminal's ability to handle various products with a single destination.

5.

Terminal Flexibility & New Business:

a.

Flexibility of Utilizing Existing Facilities for Current Fuel (backloading, supporting growth, and SHS): Assessing the terminal's flexibility in utilizing existing facilities for current fuel.

b.

New Business (potential new business such as bunkering, SAF, petrochemicals, biofuels): Evaluating the potential for new business development that might arise from this project, including bunkering, Sustainable Aviation Fuel (SAF), petrochemicals, and biofuels.

2. Research Methodology

The methodology of this research aims to develop and apply a comprehensive evaluation rubric using the Analytical Hierarchy Process (AHP) method to assess the feasibility of the demand transfer project from the ITJ Fuel Terminal (TBBM) to the JIGT Fuel Terminal (TBBM). This methodology involves several key steps from data collection to analysis and weight calculation. AHP is a method developed by Saaty to address complex decision-making problems by breaking them down into simpler elements and assigning relative weights to each element based on expert preferences (Saaty, 1980).

AHP is a multi-criteria decision-making method used to solve complex decision problems by breaking them down into a hierarchy of criteria and sub-criteria. This method allows for subjective assessments and pairwise comparisons to determine the relative weights of various criteria. AHP is particularly useful for evaluations involving multiple aspects and objectives that are often difficult to measure directly.

2.1. Flowchart and Research Steps

• Analyze the Problem and Goal: Identify the objectives of this research, which is to evaluate the feasibility of the project based on various aspects.
• Structure the Hierarchy: Organize the hierarchy of relevant criteria and sub-criteria according to the determined aspects.
• Perform Pairwise Comparisons: Conduct pairwise comparisons between criteria and sub-criteria to determine their relative weights.
• Calculate Weights and Consistency: Calculate the relative weights using the comparison matrix and check consistency to ensure the validity of the results.

Figure 1. Flowchart Research.

Figure 1. Flowchart Research.

2.2. Data Collection

The questionnaire sampling uses proportional random sampling, which considers the proportion of each stakeholder. Since proportional random sampling was used, the number of respondents in the AHP method does not have a fixed formula, but there is a minimum limit of two respondents (Saaty, 1993). According to Melillo & Pecchia (2016), an appropriate sample size for AHP analysis ranges from 19 to 400 respondents. To obtain the necessary data for AHP calculations, 19 top management members at PT XYZ were selected as respondents. They possess in-depth knowledge and experience regarding the project and the evaluated aspects, allowing them to provide accurate and relevant assessments.

The questionnaire was designed to collect data on pairwise comparisons for each criterion and sub-criterion. It included questions requiring respondents to compare the importance of one criterion against another and one sub-criterion against another. Data was collected through questionnaires distributed to top management members, either directly or via online platforms to facilitate participation. After collecting the questionnaires, the data was analyzed to compute weights and check consistency.

2.3. Analysis of AHP

The pairwise comparison matrix was constructed based on data collected from the questionnaires and calculated using the Geometric Mean method. In the context of AHP, the Geometric Mean (GM) is used to consolidate results from pairwise comparisons performed by various experts, effectively integrating different opinions (Saaty & Vargas, 2001). Each element in the matrix represents the relative comparison between two criteria or sub-criteria. The rating scale used for pairwise comparisons follows a scale from 1 to 9, where these values indicate the level of relative preference. Matrix normalization is crucial to ensure that the resulting criterion weights are proportional and can be compared directly (Saaty, 1994). Weights for each criterion and sub-criterion were calculated using the eigenvalue method from the comparison matrix. This process involves computing eigenvalues and eigenvectors to determine the relative weights of each element in the hierarchy.The consistency of the pairwise comparisons was checked by calculating the Consistency Index (CI) and Consistency Ratio (CR). This ensures that the assessments provided by respondents are consistent and acceptable. A Consistency Ratio greater than 0.1 indicates inconsistency that needs to be corrected.

2.4. Evaluation of Weights to Evaluation Criteria

To ensure that the pairwise comparison matrix is consistent, it is necessary to perform a consistency check by calculating the Consistency Index (CI) and Consistency Ratio (CR) (Forman & Gass, 2001). The calculated weights are applied to each aspect and sub-aspect of the evaluation to assess the project's feasibility. Each aspect is evaluated based on the given weights, and the evaluation results are used to determine the project's overall feasibility. The research results will then be validated with feedback from respondents and experts to ensure accuracy and relevance. Data reliability will also be checked by performing consistency analysis and internal validation. Consistent and repeatable results indicate that the collected data and methods used are reliable.

3. Data Processing

After the questionnaire data is collected, the next step is to use the geometric mean (GM) to consolidate the results from the filled questionnaires and calculate the criteria weights more accurately, integrating the results from the pairwise comparisons performed in the AHP process. The formula used to calculate the GM is:

$$GM=\sqrt[n]{\left(X_1\right)\left(X_2\right)\dots \left(X_n\right)}$$

Ex. :

GM = Geometric Mean

$X_1$ = Expert 1

$X_2$ = Expert 2

$X_n$ = Expert n

Table 1 shows the GM results obtained from the calculations using the above formula.

The next step is to arrange the GM results into a weight matrix. This weight matrix is then normalized by dividing each element by the total sum of all elements to obtain the relative weight of each criterion, as shown in Table 2 and Table 3. After normalizing the weight matrix, the subsequent stage is to calculate the consistency of the matrix to ensure that the pairwise comparisons are consistent. This process involves calculating the Maximum Eigenvalue (λ_max), Consistency Index (CI), and Consistency Ratio (CR) using the following formulas:

• Maximum Eigenvalue (λ_max) is calculated using:

$$\lambda \max =\left(\sum {GM}_{11-n1}x\overline{X}1\right)+\dots +\left(\sum {GM}_{1n-n1}x\overline{X}n\right)$$

Ex. :

${GM}_{ij}$ = Element of the normalized weight matrix

$\overline{X}$ = Average weight value

$n$ = Number of criteria

2. Consistency Index (CI) is calculated using:

$$CI={\displaystyle \frac{\lambda \max -n}{n-1}}$$

Ex. :

$\lambda max$ = Maximum Eigenvalue

$n$ = Number of criteria

3. Consistency Ratio (CR) is calculated using:

$$CR={\displaystyle \frac{CI}{RI}}$$

Ex. :

$RI$ = Random Index depending on the number of criteria

Note that if the Consistency Ratio (CR) is less than 0.1, then the pairwise comparison matrix can be considered consistent. Conversely, if CR is greater than 0.1, the comparison matrix needs to be revised to improve consistency.

After normalizing the weight matrix, the results are as follows:

-

Maximum Eigenvalue (λ_max) = 5.123

-

Consistency Index (CI)  = 0.031

-

Consistency Ratio (CR)  = 0.027

Since the Consistency Ratio (CR) is below 0.1 (or 10%), the pairwise comparison matrix can be considered consistent. Thus, the comparisons used in the decision-making process are acceptable without requiring further revision. The next step is to calculate the priority vector, which aims to determine the weights or relative importance of each criterion used in the analysis. The priority vector is calculated by summing the values in each row of the normalized matrix and then dividing by the total number of rows to obtain the average value for each criterion. These priority weights represent the importance of each criterion in the decision-making context, as shown in Table 4.

3.1. Data Processing in Python

1. 

Importing Data 

-Use Python libraries such as pandas to import the pairwise comparison matrix and weights calculated in Excel.

Python code:

import pandas as pd

# Load data from Excel

df = pd.read_excel('pairwise_comparison_matrix.xlsx', sheet_name='Sheet1')

2. 

Calculation of Eigenvalues and Eigenvectors 

-Use NumPy or SciPy libraries to compute the eigenvalues and eigenvectors of the pairwise comparison matrix.

Python code:

import numpy as np

# Convert DataFrame to NumPy array

matrix = df.values

# Calculate eigenvalues and eigenvectors

eigenvalues, eigenvectors = np.linalg.eig(matrix)

-Calculate the Maximum Eigenvalue to determine the priority weights.

Python code:

# Find the maximum eigenvalue

lambda_max = max(eigenvalues)

3. 

Consistency Check 

-Identify the largest eigenvalue used to calculate the Consistency Index (CI).

Python code:

# Number of criteria

n = len(matrix)

# Calculate Consistency Index

CI = (lambda_max - n) / (n - 1)

-Compare the CI with the Random Consistency Index (RI) to get the CR. RI values are pre-determined based on matrix size.

Python code:

# Random Consistency Index (RI) for a matrix of size n

RI_values = {1: 0.0, 2: 0.0, 3: 0.58, 4: 0.90, 5: 1.12} # Example RI values

RI = RI_values[n]

# Calculate Consistency Ratio

CR = CI / RI

-Ensure that the CR is less than or equal to 0.1. If the CR exceeds this threshold, reassess the pairwise comparisons.

4. 

Final Calculation and Reporting 

-Gather the final weights and consistency results. Prepare data for reporting and decision-making.

Python code:

# Prepare final results

weights = eigenvectors[:, eigenvalues.argmax()]

weights = weights / weights.sum() # Normalize weights

-Save the results into a file for inclusion in the final report.

Python code:

# Save results to Excel

pd.DataFrame(weights, columns=['Weights']).to_excel('final_weights.xlsx', index=False)

4. Result and Discussion

After obtaining the priority vector, these values are used in evaluating the existing metrics, where each metric will be assessed based on the established criteria weights, as shown in Table 5.

The criteria weights derived from the priority vector highlight the relative importance of each aspect in evaluating the project feasibility. The criterion weights obtained through AHP reflect the relative importance of each aspect in project evaluation, enabling a more comprehensive and data-driven assessment (Triantaphyllou & Khamooshi, 1996). The "Safety & Asset Integrity" aspect holds the highest weight at 38%, indicating its critical importance in the overall evaluation. The "Operational Supply" and "Operational Distribution" sub-aspects also play significant roles, with weights of 15% and 13% respectively, underscoring their relevance in the project's assessment. The "Terminal & New Business Flexibility" aspect, while important, has a comparatively lower weight of 12%, reflecting its lesser impact relative to other criteria.

In the "Safety & Asset Integrity" sub-aspects, "Safety" and "Asset Integrity" have substantial weights of 35% and 34%, respectively, which emphasizes the importance of these factors in ensuring project success. Conversely, "Reliability" and "ESG" have lower weights but still contribute to the overall evaluation, with weights of 20% and 11%. For "Operational Supply," the "Supply Options" sub-aspect is prioritized with a weight of 42%, indicating its primary importance over "Port Capacity" and "Storage Capacity," which have weights of 31% and 28%, respectively.

In the "Operational Distribution" sub-aspects, "Distance" is given a higher weight of 51%, reflecting its critical role in logistics planning, compared to "Distribution Options" with a weight of 49%. Lastly, in the "Terminal & New Business Flexibility" sub-aspects, "New Business Potential" is valued more highly with a weight of 58% compared to "Terminal Flexibility" at 42%, demonstrating the significant role of new business opportunities in evaluating the project's feasibility. Overall, the results underscore the critical factors and their relative importance in evaluating the feasibility of the project, and guiding decision-making processes effectively. AHP has proven to be a valuable tool in project evaluation by providing relative weights for various criteria, thus assisting in more structured and objective decision-making (Kumar & Thakur, 2014)

These points and criteria weights have received feedback from experts and have been approved for use in developing the assessment rubric. The expert validation ensures that the criteria accurately represent the key aspects of the project evaluation, allowing for a more reliable and robust weighting process in the decision-making framework. Expert validation is crucial in the AHP process as it ensures that the criteria and weights set reflect expert consensus and practical relevance in the project context (Ranganathan & Hashemi, 2010). This consensus among experts strengthens the validity of the prioritization and provides a solid foundation for the evaluation process.