Multimodal Large Language Model-Based Fault Detection and Diagnosis in Context of Industry 4.0

1. Introduction

In modern technology, developing specialized fault detection and diagnosis (FDD) systems for industrial applications has emerged as a vital endeavor[1]. Manufacturing processes and industrial operations have been revolutionized by the advent of Industry 4.0 (I4.0), characterized by the integration of advanced technologies including artificial intelligence (AI), Internet of Things (IoT), and big data analytics. Digital transformation establishes opportunities and challenges for maintenance. Additionally, it optimizes complex industrial systems.

Large learning models (LLMs) represent a remarkable breakthrough in AI [2], transforming organizational decision-making, problem resolution strategies, and operational efficiency. Trained on vast amounts of textual data, these models, have exhibited noteworthy capabilities in understanding context, generating human-like text, and carrying out a wide range of language-based tasks. Although LLMs have been successfully applied in various domains, their potential for industrial FDD systems is still largely unexplored.

Existing FDD approaches are mainly dependent on traditional machine learning (ML) techniques [3], expert systems [4], or statistical methods [5]. The current methods often face difficulties in handling the intricacy and various fault scenarios in modern industrial systems. Additionally, they tend to concentrate on particular data types or predetermined fault patterns, which constrains their ability to adapt to new or unanticipated faults [6]. The crucial obstacles in FDD research are the incorporation of multimodal data and the capability to analyze different data types.

In our proposed work, these gaps are addressed via a novel multimodal FDD framework that uses LLMs, which offer several key contributions. First, an LLM-based FDD method is proposed using the GPT-4-Preview, which enhances the scalability, generalizability, and efficiency of complex systems and various fault scenarios. Second, by generating synthetic datasets using LLMs, we enrich the knowledge base and enhance the accuracy of FDD in imbalanced scenarios. Third, the approach optimizes the diagnosis accuracy and overall performance of the framework. Fourth, we introduce a hybrid architecture that integrates online and offline processing, which combines real-time data streams with fine-tuned LLMs for dynamic, accurate, and context-aware fault detection suitable to I4.0 environments. This comprehensive approach aims to overcome traditional FDD challenges and advance the field towards more adaptive and efficient fault diagnosis systems. In the following sections, a comprehensive literature review is presented, our methodology and system design are detailed, the results of our case study and model comparisons are discussed, and the implications of our findings for future developments in industrial FDD systems and I4.0 technologies are explored.

2. Background, Taxonomy, and Related Research

In recent years, FDD has seen remarkable advancements, particularly with multimodal LLMs (MM-LLMs) emerging as a cutting-edge technology. These advanced AI-based systems were created to integrate and process various data types, including text, images, audio, and video, which makes them more applicable across multiple industrial contexts.

Several studies have explored the use of LLMs and MMLLMs in FDD. For instance, combining fuzzy logic with ChatGLM2-6B has potential in electrical equipment fault diagnosis, enhancing the adaptability and accuracy of fault diagnosis mechanisms [7]. Likewise, integrating knowledge graphs with LLMs has been applied to compressor fault diagnosis, which demonstrates significant improvements in fault detection accuracy [8]. Prompt engineering approaches have also been investigated for spacecraft fault diagnosis, which highlights the potential of LLMs in managing complex multi-level fault scenarios [9]. Moreover, the use of LLMs in aviation assembly and autonomous driving in mines emphasizes the versatility and effectiveness of these models in diverse industrial applications [10,11,12].

Figure 1 illustrates the architecture of MM-LLMs, which consists of four key components: the modality encoder, which encodes inputs from various modalities to obtain corresponding features; the input projector, which aligns these features with the text feature space via methods such as linear projectors, multilayer perceptron, and cross-attention mechanisms; the LLM backbone, which processes the representations to perform semantic understanding, reasoning, and decision-making; and the output projector, which maps signal token representations into features that are comprehensible to the modality generator, which produces outputs in distinct modalities via latent diffusion models. Examples of MM-LLMs include Flamingo, which integrates NFNetF6 and Chinchilla for visual and textual processing; BLIP2, which combines CLIP and Flan-T5 for image-text tasks; and LLaVA, which employs CLIP and Vicuna for visual question answering. The training pipeline is divided into multimodal pretraining, which focuses on aligning modalities using X-Text datasets, and multimodal instruction tuning, which fine-tunes pre-trained MM-LLMs with instruction-formatted datasets to enhance their zero- shot performance and generalization [13].

This comprehensive architecture enables MM-LLMs to effectively integrate multiple modalities, leveraging pre-trained models to optimize computational efficiency and improve performance across various multimodal tasks.

Given the novelty and potential of MM-LLM-based FDD systems, a comprehensive search was carried out using the Scopus database to acquire references for related studies. To narrow down the search, the following query was employed, focusing on the terms “fault diagnosis” and “fault diagnostics” within the title or abstract: TITLE-ABS-KEY (“fault diagnosis” OR “fault diagnostics”) AND PUBYEAR > 2018 AND PUBYEAR < 2025 AND (LIMIT-TO (SRCTYPE, “j”)) AND (LIMIT-TO (LANGUAGE, “English”)) AND ( LIMIT-TO ( SUBJAREA, “ENGI”) OR LIMIT-TO (SUBJAREA, “COMP”)). The search was further refined to include publications from 2019 to 2024, ensuring the most recent and relevant findings. To maintain a high standard of research, only journal articles (indicated by “j”) were considered, and the language was restricted to English. Moreover, the subject areas were limited to Engineering (“ENGI”) and Computer Science (“COMP”), aligning with the theme of industrial fault diagnostic support systems. A total of 11,655 documents were identified using this approach.

Subsequently, comprehensive and insightful taxonomies in FDD in terms of methods and applications were utilized, as shown in Figure 2 and Figure 3, respectively. Utilizing BERTopic and GPT-4, this comprehensive review was essential because of the limited number of articles that explicitly address MM-LLM-based FDD, highlighting its status as an emerging technology and emphasizing the need for further research in this promising area. The resulting taxonomies provide a structured and easily navigable overview of the diverse FDD methods and their potential applications across various industrial sectors.

2.1. Industrial Equipment

In mechanical rotating machines, attention mechanisms and deep learning (DL) techniques including deep convolutional neural network (DCNN) and transfer learning enhance bearing fault diagnosis [3,5,6]. Data-driven condition monitoring of industrial equipment has been reviewed, with a focus on various ML models [14]. A Bayesian network model was proposed for the IoT-based fault diagnosis of intelligent instruments [15]. Industrial equipment is heavily dependent on FDD methods, such as instrumentation [15], gas analyzers, and oxygen monitoring [16]. Process control applications including SCADA [17], PLCs [18], and HMIs [19] integrate FDD algorithms with a hybrid customized chatbot HMI using ChatGPT for swift equipment information retrieval [20]. In the realm of hydraulic systems, a novel sensor fault estimation scheme for hydraulic servo actuators was proposed by Djordjevic et al. [21] using sliding mode observers, and its effectiveness was shown via simulations. This approach addresses the difficulties of nonlinear systems and uncertainties in industrial hydraulic equipment. For planetary gearboxes, Tao et al. [22] introduced a semi-supervised fault diagnosis method using a dynamic graph attention network with fuzzy K-nearest neighbors (KNNs). This method effectively addresses the challenge of limited labeled data in gearbox fault diagnosis, which achieves over 99% accuracy with very few labeled samples and outperforms traditional classification networks.

Additionally, a new multi-source partial transfer network model addresses real-world challenges in machinery fault diagnostics, including unknown label spaces and incomplete fault type coverage. By employing class-level weight mechanisms and particle swarm optimization, the model combines shared feature extraction with domain-specific modules for fault diagnosis and adaptation. It was verified across six datasets, and its effectiveness in industrial equipment fault diagnosis was presented [23].

2.2. Industry 4.0

I4.0 integrates AI, data mining, and IoT to revolutionize manufacturing. DL, particularly CNNs, has potential for FDD in smart manufacturing [24]. Digital twin collaboration aids in the auto-detecting of erratic data at an energy 4.0 [25]. A novel approach that combines circular economy, data mining, and AI has been proposed for FDD [26]. Transfer learning has emerged as a valuable technique for predictive maintenance [27]. Innovations have been developed in 3D printing fault diagnostics [28,29] and optimized CNN approaches for industrial robotics [30]. Data security in industrial fault diagnosis is addressed by blockchain-based decentralized, federated transfer learning [31]. Recent advancements in IoT-based fault diagnosis include an industrial IoT platform with an EML for transformer monitoring [32], an IoT-based system using a one-dimensional (1D) CNN for PV plant monitoring [33], and preprocessing techniques with DL models for industrial IoT (IIoT) devices [34]. These approaches have shown improved accuracy, efficiency, and cybersecurity in fault diagnosis across various industrial applications.

In the context of fault estimation and fault-tolerant control, Wang et al. [35] proposed a Q-learning-based approach using an iterative learning control framework. This method addresses the obstacles posed by unknown faults that vary with both the time and trial axes in repetitive tasks, exhibiting the effectiveness of adapting to changing faults and maintaining control performance in MIMO systems.

In the era of big data and advanced networking, the diagnosis of fiber optic cable faults in communication networks has become critical. A combined GAN-CNN approach obtained 98.5% accuracy in addressing these faults [36]. DL enhances fault location and network maintenance efficiency [37], whereas a CNN-based model using the KDD99 dataset acquires 98.1% accuracy for computer network fault diagnosis [38].

A Bayesian network-based mechanism ensures an efficient real-time fault diagnosis in communication systems [39]. For I4.0, a framework integrating LSTM, CNN, and GNN effectively handled heterogeneous data for fault detection [40]. Advanced orchestration techniques utilizing self-organizing maps and root cause analysis improve fault detection in 5G and beyond infrastructure [41]. A deep neural network-based method accurately diagnoses faults in millimeter-wave active phased arrays for 5G and 6G radios [42]. Moreover, a new federated learning framework known as FedCAE was proposed for edge-cloud collaboration in machine fault diagnosis. In this approach, the challenge of data islands is addressed by allowing multiple data owners to contribute to model training without sharing raw data, thereby exhibiting promising results in bearing fault diagnosis [43]. Furthermore, a comprehensive roadmap for performance-supervised plant-wide process monitoring in Industry 4.0 has been developed. This roadmap highlights the need to evaluate the impact of detected faults on plant-wide performance and proposes a technical route embedded in the cyber-physical-social system framework, which addresses key research questions and future directions in process monitoring [44].

2.3. Power Plants

FDD techniques are indispensable in thermal and nuclear power plants. Advancements have focused on intelligent methods, which include model-based, data-driven, and statistical approaches [45], which are applied to crucial equipment including boilers, turbines, and heat exchangers. A robust soft sensor for estimating heat exchanger fouling in combustion-thermal power plants was developed [46]. In nuclear power plants and photovoltaic systems, ML-based fault diagnosis techniques, such as KNNs, support vector machines (SVMs), bagged trees, and ensemble methods, detect faults in pressurized water reactors [47].

2.4. Electronic Systems

Numerous advanced techniques have been developed for the fault diagnosis and monitoring of electronic systems. Hybrid intelligent particle swarm optimization. Moreover, SVMs have been applied for fault diagnosis in analog circuits [48]. FRA-based parameter estimation has also been utilized to diagnose faults in three-phase voltage-source inverters [49]. Recent advancements include the application of transformer-based networks for consumer electronic sensor fault diagnosis in the Internet of Everything era, addressing the challenges posed by environmental factors [50]. In microelectronic manufacturing, a novel hierarchical tree-DCNN structure has been proposed to address unbalanced data diagnosis in flexible integrated circuit substrates, improving the efficiency and accuracy of traditional human visual interpretation methods [51]. Moreover, joint self-attention mechanisms and residual networks have been used for the automated monitoring of intelligent sensors in consumer electronics, showing the potential for enhanced fault detection and classification in complex electronic systems [52].

2.5. Energy Sector

Recent advancements in the energy sector fault diagnosis include enhanced methods for lithium–ion batteries [53], low-voltage direct current networks [54], and transformers [55]. CNNs perform superior power grid fault diagnosis [56], whereas KG-CNN enhances the motor fault diagnosis [57]. Photovoltaic systems benefit from infrared thermography and ML [58]. Advanced models such as GPT-4 are being explored for building energy management and fault diagnosis [59]. For integrated energy systems, a novel multiscale spatial–temporal graph neural network was proposed to address compound fault diagnosis with small sample sizes. This approach employs multiscale graph operators and gated recurrent units to learn label-specific fault features, hence mitigating the challenge of limited compound fault samples [60].

In high voltage circuit breaker fault diagnosis, a new method combining variational mode decomposition energy entropy and SVM has been developed. This approach, validated on a 35-kV SF6 circuit breaker, effectively identifies faults including actuator issues, buffer spring invalidity, and loose base screws [61].

2.6. Transportation

FDD systems are important for transportation, autonomous driving, aviation, railways, and ship engines. In autonomous driving, CNN-based information fusion aids in sensor fault detection [62]. Aviation utilizes biomimetic pattern recognition [63] and deep neural networks [64] for fault diagnosis. Railways benefit from hybrid methods [65] and RP-CNN [66] for track and wheel fault diagnosis. Digital twin-based methods [67] and advanced fault feature extraction techniques [68] optimized condition monitoring and fault diagnosis for ships.

2.7. Healthcare Systems

FDD systems have advanced in various applications. A hybrid HMM-SVM approach enables sensor fault diagnosis in IoT-based healthcare systems [69]. For medical IoT devices, a lightweight autoencoder with KernelSHAP and an ANN classifies anomalies [70]. Medical equipment failure prediction benefits from an optimized least squares SVM algorithm, which demonstrates high accuracy for ECG equipment and incubators [71]. Recent advancements include the development of hierarchical health assessment models for equipment with uncertain fault diagnosis results. Based on Bayesian networks, these models can assess health states at different hierarchical levels by incorporating identified and uncertain fault diagnosis results. This approach has been successfully employed in complex safety-critical systems, including diesel engine combustion systems, showing its potential for more comprehensive health monitoring and risk avoidance in medical equipment [72].

2.8. Aerospace

Various innovative approaches have been developed for aerospace FDD applications. A PWM harmonic current-based method detects interturn short-circuit faults in FTPMSM drives [73]. For liquid rocket engines, fault diagnosis methods can be categorized as signal processing, model-driven, and AI-based approaches [74]. The LSTM-GAN method effectively detected faults in LOX/kerosene rocket engines [75]. A hybrid AI technique, BGOA-EANNs, is superior for aerospace health diagnosis [76]. The EMFFI approach, which employs bispectrum technology, SURF, LDA, and random forest (RF), attains a high accuracy in gearbox fault diagnosis for aerospace systems [77].

2.9. Software System Engineering

For software fault diagnosis, diverse ML, XAI, and data mining methods have been developed. ChatGPT-based fault diagnosis using execution traces enhances the identification of bug causes [78]. Data mining algorithms debug and compare software performance metrics [79]. A combination of adaptive tracing, fault injection, and graph convolutional networks demonstrates superior accuracy in fault diagnosis for various software systems [80]. Among the ML models evaluated for software fault diagnosis, XGBR showed the best performance in terms of accuracy and error metrics [81].

Table 1 and Table 2 provide a comprehensive overview of related studies on FDD systems using various models. It details the models employed, their application domains, and sectors of implementation and evaluates them based on ML and DL paradigms. Moreover, the table examines the relevance of I4.0 technologies and their real-time capabilities, generality, and support for multimodal data. The applications range from industrial equipment and power plants to healthcare and aerospace, which reflects the diverse applicability of these models. These studies highlight the diverse range of FDD methods and their broad applicability in various industries. However, challenges including generalizability, multimodality, applicability to I4.0, and real-time detection and diagnosis require further exploration. The novelty of the proposed work lies in addressing these gaps by developing a robust multimodality FDD model that accounts for various types of data from different device applications while utilizing the LLM capability.

3. Methodology and Design

In this section, the methodology and design of the proposed multimodal LLM-based FDD system, which harnesses the power of multimodal architectures and LLMs, is discussed. The integration of these technologies aims to offer tailored and insightful diagnostic support solutions in the context of the I4.0 transformation and diagnostics for industrial systems. This section contains two main subsections: the design and data set acquisition.

3.1. A Multimodality Framework and Architecture

Figure 4 presents the proposed MM LLM-based FDD architecture. This novel approach integrates multimodal data sources and advanced ML techniques, mainly focusing on LLMs. The MM LLM-based FDD framework proposes stages that enable effective FDD in complex systems, which is promising for revolutionizing the industrial domain.

The initial stage involves data sourcing and collection of diverse data types including images, audio, vibration signals, video, and text. These multimodal data were extracted via web scraping and application programming interfaces (APIs). LLMs can be utilized for data collection and processing.

In the preprocessing stage, the extracted data undergo various transformations to improve their quality and prepare them for further analysis. This includes drop duplication, stop-word elimination, deep column transformations, and the handling of missing values. LLMs can be effectively utilized to enhance these tasks and improve the accuracy of duplicate removal, context-aware stop-word elimination, and intelligent handling of missing data.

The ingestion phase involves embedding techniques for converting text chunks into numerical representations that capture semantic or similarity relationships. These embeddings facilitate the effective utilization of LLMs in subsequent FDD stages. The knowledge base incorporates keywords, topics, embedding topics, and semantic or similarity search capabilities. Leveraging LLMs such as Llama 2 [82], GPT [83], and Mistral [84], the framework extracts relevant information from embedded text chunks, enabling the identification of key insights and patterns for FDD.

The FDD component integrates state-of-the-art techniques such as LangChain [85] or Llamindex [86], a framework for building applications with language models. These techniques, which work in harmony with LLMs, can accurately detect and diagnose faults within the system. The accuracy and relevance of FDD components were assessed. Unsatisfactory results lead to iterative refinement, which loops back to the data sourcing step, whereas satisfactory results are saved for decision-making and fault management. This feedback mechanism guarantees the framework’s long-term efficacy and ongoing improvement. The proposed framework offers generality, flexibility, and scalability to various domains. Its integration of advanced preprocessing, a comprehensive knowledge base, and multimodality LLM ensures adaptability to diverse fault scenarios. This design significantly improves the accuracy and performance. Using the LLM-based approach, a substantial improvement can be obtained over traditional methods.

3.2. Dataset Acquisition for LLM-Based FDD System

Our multimodal FDD system utilizes a comprehensive data acquisition approach that integrates diverse fault data, contextual information, and enriched technical knowledge. The dataset includes raw fault data, operational parameters, environmental conditions, maintenance records, and expert insights. Our data acquisition strategy encompasses two main types of sources:

3.2.1. Siemens Industry Online Support Forum

This primary source provides rich discussions on various Siemens products[87], including the SIMATIC S5/STEP 5, SIMATIC 505, and SINUMERIK CNC systems. We extracted 240,000 rows across 22 columns from the SIMODRIVE converter system and SIMATIC S7 series, significantly improving the knowledge base of the FDD system.

3.2.2. Diverse Industrial Equipment Manufacturers

To ensure diverse data, we gathered fault codes, descriptions, and corrective actions from major manufacturers such as Allen Bradley, ABB, CLICK and Yokogawa. This process added 1,398 new fault records from different devices, as listed in Table 3

Data Preprocessing and Diversity

Data preprocessing involves the removal of redundancies, noise, and irrelevant information. For example, we eliminated duplicate discussions and standardized fault codes and extracted key troubleshooting steps from the Siemens support forum data.

To facilitate contextual data cleaning and the semantic categorization of fault descriptions, we utilized LLMs. This process has been applied to various data sources from various manufacturers.

3.3. Hybrid LLM-Based Architectures for Online and Offline FDD Systems

Figure 5 shows a hybrid LLM-based FDD architecture that optimizes FDD systems by integrating online and offline models and improving diagnostic capabilities, data privacy, and cybersecurity. The system processes unstructured data, generates embeddings, and utilizes a custom GPT model for query handling. Semantic indexing facilitates efficient knowledge retrieval. Offline LLMs including Llama 2 and Mistral 7B are fine-tuned with synthetic data to improve performance and address security concerns. The integration layer incorporates real-time data from various industrial sources, which enriches the diagnostic process. The architecture manages IT and operational technology (OT) complexities and ensures secure data flow through firewalls. This hybrid approach leverages real-time data integration and offline LLM processing, advancing the FDD system’s capability to handle real-time data and enhancing fault detection accuracy in I4.0 environments.

4. Evaluation

In this section, a comprehensive approach for assessing the performance and effectiveness of the proposed LLM-based FDD system is outlined. The metrics used for the evaluation, including inference relevance accuracy, response time, and a custom performance score (PS) that combines accuracy and speed are detailed the metrics used for the evaluation, including inference relevance accuracy, response time, and a custom PS that combines accuracy and speed. In this section, our methodology for categorizing diagnostic questions into simple, medium, and complex types, thereby providing a nuanced understanding of a system’s capabilities across various fault scenarios is described. Additionally, the comparative process analysis between different LLMs, including ChatGPT4, Mixtral 8 x7B, Llama 2, and our proposed FDD GPT-4Preview model is explained.

Moreover, a comparative analysis of conventional FDD and LLM-based FDD is applied. In this subsection, traditional FDD methods against the LLM-based approach are evaluated using the same performance metrics. The comparative analysis helps emphasize the advancements and potential improvements offered by integrating LLMs into FDD systems. By analyzing both approaches, we aim to determine the specific benefits and limitations of LLM-based systems over conventional methods.

Additionally, the effectiveness of handling hybrid architecture performance in real-time and offline processing is evaluated. This includes assessing how well the system manages the immediate fault diagnosis needs versus a more detailed, offline analysis. In the section, the manner in which we assessed the system’s adaptability to different industrial contexts and its potential impact on I4.0 implementations is also addressed. By detailing these evaluation methods, we aimed to provide a clear and rigorous framework for understanding the strengths and limitations of the proposed FDD system.

4.1. Comparative Analysis of Different Models

In this section, four LLMs are thoroughly examined: ChatGPT4, Mixtral 8x7B, Llama 2, and our proposed model using GPT-4-Preview, all of which have been implemented into the FDD system. Our assessment considered various metrics, which include inference relevance, response time, and PSs. Using these metrics, we systematically compared LLMs via quantitative measurements and evaluated their strengths and weaknesses.

4.1.1. Assessment Metrics

This study selects performance evaluation indicators to comprehensively assess LLM use in FDD systems. The proposed metrics include inference relevance categories, response time measurements, and a composite PS, providing a balanced assessment of accuracy and efficiency. The inference relevance categories (Irrelevant, Generic, Relevant, Precise) enable detailed analysis of output quality, whereas the response time metrics address the critical aspect of real-time performance. The PS formula, which combines response quality and speed with adjustable weights for different response categories, allows for flexible prioritization based on specific industrial requirements, demonstrating the adaptability of our research to various contexts. We employed three key metrics to evaluate the model performance:

Inference Relevance

There are four categories of relevance metrics in fault diagnosis systems has four categorizations: relevant, irrelevant, generic, and precise. The relevant responses accurately address the specifics of the fault query, effectively understand the fault context, and provide solutions from a knowledge base. Irrelevant responses fail to address the fault, indicating misinterpretation or retrieval of unrelated information. Precise responses deliver detailed and specific solutions that directly address faults using the targeted information. Although correct, generic responses lack the specificity of precise responses and offer broader and more general information. Equations (1)–(4) are employed to calculate the average inference relevance accuracy by summing the counts for each response type and dividing it by the total number of responses.

$$I{R}_{generic}={\displaystyle \frac{{\sum }_{i=1}^{m}Respons{e}_{generi{c}_i}}{m}}$$

(1)

$$I{R}_{irrelevant}={\displaystyle \frac{{\sum }_{i=1}^{m}Respons{e}_{irrelevan{t}_i}}{m}}$$

(2)

$$I{R}_{relevant}={\displaystyle \frac{{\sum }_{i=1}^{m}Respons{e}_{relevan{t}_i}}{m}}$$

(3)

$$I{R}_{precise}={\displaystyle \frac{{\sum }_{i=1}^{m}Respons{e}_{precis{e}_i}}{m}}$$

(4)

The average value for generic inference relevance, denoted as $I{R}_{generic}$, is calculated by (1) summing all individual counts of generic responses, represented as ${\sum }_{i=1}^{m}Respons{e}_{generic,i}$, where $Respons{e}_{generic,i}$ refers to each generic response generated by the model and m is the total number of results in the dataset. The average value for irrelevant inference relevance, denoted as $I{R}_{irrelevant}$, was calculated using (2) where ${\sum }_{i=1}^{m}Respons{e}_{irrelevant,i}$ represents the summation of all individual irrelevant responses from the model, and m is the total number of results in the dataset. Each $Respons{e}_{irrelevant,i}$ refers to an individual irrelevant response generated by the model. This calculation measures the tendency of the model to generate irrelevant responses across the dataset. The average value for the relevant inference relevance, represented as $I{R}_{relevant}$, was determined using (3) where ${\sum }_{i=1}^{m}Respons{e}_{relevant,i}$ is the total sum of all relevant responses provided by the model and m is the total number of results in the dataset. Each $Respons{e}_{relevant,i}$ corresponds to the individual relevant response generated by the model. This formula effectively quantifies the ability of a model to produce responses that accurately address specific fault queries in a dataset. Equation (4) calculates the precise answer, in which ${\sum }_{i=1}^{m}Respons{e}_{precise,i}$ is the total sum of all precise responses provided by the model and m is the total number of results in the dataset. Each $Respons{e}_{precise,i}$ refers to the precise response of the model.

PS

We can establish a performance metric that considers response time and context relevance to evaluate performance One way to do this is by calculating a PS for each model using (5):

$$\begin{array}{c}PS={\displaystyle \frac{1}{R{T}_{avg}}}\times (W_{irrelevant}\times Irrelevant\%+\hfill \\ W_{relevant}\times relevant\%+W_{generic}\times Generic\%+\hfill \\ W_{precise}\times Precise\%)\hfill \end{array}$$

(5)

A higher PS indicates better performance, which reflects faster average response times and higher relevance in the given context, with a focus on precise and relevant categories. The models with higher PS values were both efficient and effective in their responses. They are also relevant, which makes them ideal for real-time applications where speed and accuracy are crucial. Conversely, a lower PS value indicates poorer performance, which is characterized by slower average response times and a less relevant context, as evidenced by the higher percentages of irrelevant and generic categories. Such models are less efficient and less relevant and, thus, more suitable for non-real-time applications.

Response Time Analysis

Our model performance evaluation analyzed the maximum and minimum response times across all the models. These metrics provide insights into the extremes of response latency, which helps us to understand the range of performance variability. To quantify these measures, (6) and (7) were employed:

$$Ma{x}_{RT}=max\left(RT\right)$$

(6)

$$Mi{n}_{RT}=min\left(RT\right)$$

(7)

$Ma{x}_{RT}$ represents the maximum response time observed across all models, and $Mi{n}_{RT}$ denotes the minimum response time. Function $max\left(RT\right)$ returns the highest value in response times, whereas $min\left(RT\right)$ determines the lowest value. These equations allow us to establish the upper and lower bounds of the response time performance, which is crucial for the assessment of efficiency and consistency of model responses.

Fault Categorization

We suggest organizing diagnostics to assess the troubleshooting requirements of different LLM industrial equipment models. The questions were divided into three levels, namely, simple, medium, and complex. Simple diagnostic questions involve basic tests or common issues that can be identified and resolved with minimal technical expertise, such as addressing the fault code T18. Medium diagnostic questions necessitate a more detailed examination, which potentially involves several steps or specific system knowledge, as observed with fault code T17, in which specific runtime diagnostics must be conducted initially. Complex diagnostic questions entail extensive troubleshooting, requiring an intricate understanding of the system’s inner workings and multiple diagnostic procedures. For instance, diagnosing a CPU fault in a Siemens S7-1500 system, as detailed in Table 4, involves comprehensive diagnostic procedures to isolate and resolve the issue, which reflects the high level of complexity involved.

4.2. Comparative Analysis Between Conventional and LLM-Based FDD

In this subsection, the methodology for comparing a conventional FDD system based on traditional ML algorithms with the proposed LLM-based FDD system is outlined. The comparison is grounded in quantitative metrics including accuracy, precision, recall, F1 score, and computational efficiency. To compare their effectiveness in classifying fault clusters, we utilized three models, namely, logistic regression (LR), RF, and neural network (NN). The conventional ML-based FDD system was trained on Siemens PLC semantic S7 datasets with five fault categories (CPU/processor issues, power issues, input/output issues, communication issues, and other issues). To measure accuracy, we assessed the proportion of correct predictions made using the model out of the total number of predictions, providing an overall perspective on its performance across all fault categories. It was calculated using (8) in which $TP$ = true positives, $TN$ = true negatives, $FP$ = false positives, and $FN$ = false negatives.

$$Accuracy={\displaystyle \frac{(TP+TN)}{(TP+TN+FP+FN)}}$$

(8)

To measure precision, we evaluated the accuracy of positive predictions by focusing on the ratio of true positives to the total number of positive predictions, which include both true and false positives. The precision was calculated using (9).

$$Precision={\displaystyle \frac{\left(TP\right)}{(TP+FP)}}$$

(9)

When evaluating the effectiveness of a model, recall (or sensitivity) measures its ability to correctly determine all relevant instances of a fault. This metric captures the ratio of true positive predictions to the total number of actual positives, including both true and false negatives. This was calculated using (10).

$$Recall={\displaystyle \frac{\left(TP\right)}{(TP+FN)}}$$

(10)

The data and resource requirements for LLM-based and traditional FDD methods vary significantly, as shown in Table 5. LLM-based approaches can leverage large volumes of unstructured data from diverse sources, excelling at processing natural language. At the same time, traditional methods are typically dependent on structured numerical data from sensors and predefined fault codes. LLMs are pre-trained on vast general knowledge and then fine-tuned for specific tasks, reducing domain-specific training data needs. By contrast, traditional ML models necessitate extensive labeled training data for each fault type and system. LLMs demand significant computational power, especially for training and real-time inference, whereas conventional methods are generally less computationally intensive. Nevertheless, LLMs offer greater adaptability to new fault types or systems through fine-tuning or prompt engineering, whereas traditional models often require retraining from scratch. LLMs provide natural language explanations, enhancing interpretability, and can integrate information across modalities more seamlessly.

5. Results and Discussion

In this section, the findings from our comparative study of four LLMs, namely, ChatGPT-4, Mixtral 8x7B, Llama 2, and our proposed FDD using the GPT-4-Preview model are presented and analyzed. We evaluated their performance in the context of FDD applications, focusing on three key metrics: inference relevance, response time, and overall PS. This section also assesses the effectiveness of our proposed hybrid architecture in creating a dynamic and responsive FDD system in both online and offline systems. Moreover, we compared conventional FDD methods and our LLM-based approach in terms of accuracy and response time, underscoring the advancements and potential benefits of employing large language models for FDD.

5.1. Comparative Analysis of LLMs: Assessment Metrics and Practical Use Cases

We conducted a comparative analysis of various LLMs, which include ChatGPT 4, Mixtral 8x7B, Llama 2, and the proposed FDD GPT-4-Preview. The core objective was to evaluate the efficacy of these models in generating contextual questions derived from a specified knowledge base. Our methodology involved the systematic injection of 10 uniquely constructed questions into each LLM and the proposed models to assess their ability to produce relevant, irrelevant, precise, and generic responses.

5.1.1. A Comparison of Different LLMs

In the analysis of the comparative language models, as shown in Table 6 and Table 7, the GPT-4-Preview showed superior performance, which generates correct answers with 60% precision. This marks a significant improvement in the specificity of the responses compared with other models, as exemplified in Table 8 (a), where a precise response is provided for fault code T04:C31. ChatGPT 4 balanced specific and vague responses at 40% each but lacked precision, as shown in Table 8 (b), in which an irrelevant response is given for the same fault code. Llama 2 and Mixtral 8x7B models demonstrated the quickest response times but suffered in contextual relevance, with the highest rate of irrelevant responses at 50%. Llama 2 had the highest generic response rate of 40%, which indicates a more generalized understanding of prompts, as shown in Table 9 (b). The GPT-4-Preview had a lower rate of irrelevant responses 20%, which suggests better contextual understanding. Although Mixtral 8x7B provided the fastest response times, this speed may come at the cost of contextual accuracy, although it still manages to provide relevant responses, as shown in Table 9 (a). Conversely, GPT-4-Preview offered more concise and accurate responses with moderate delay times, which indicates precise comprehension of complex inputs.

Table 6. Comparative analysis of response time and inference relevance across different LLM models.

Question	ChatGPT 4		Llama 2		Mixtral 8x7B		FDD GPT-4-Preview
	RT (Sec.)	IR	RT (Sec.)	IR	RT (Sec.)	IR	RT (Sec.)	IR
Q1	44.04	Irrelevant	10.79	Irrelevant	3.37	Irrelevant	12.7	Precise
Q2	23.58	Generic	12.04	Irrelevant	2.61	Irrelevant	15.29	Precise
Q3	26.75	Irrelevant	15.56	Irrelevant	5.65	Irrelevant	17.17	Precise
Q4	22.31	Irrelevant	17.66	Irrelevant	2.24	Irrelevant	21.27	Precise
Q5	34.66	Irrelevant	18.28	Irrelevant	2.58	Irrelevant	13.34	Precise
Q6	41.46	Relevant	22.93	Relevant	3.81	Relevant	17.71	Relevant
Q7	22.38	Relevant	7.8	Generic	3.2	Relevant	14.68	Precise
Q8	42.18	Generic	10.10	Generic	5.56	Generic	19.88	Generic
Q9	30.04	Generic	12.95	Generic	5.96	Generic	21.33	Irrelevant
Q10	38.52	Generic	13.85	Generic	4.16	Relevant	17.64	Irrelevant

Table 6. Comparative analysis of response time and inference relevance across different LLM models.

Question	ChatGPT 4		Llama 2		Mixtral 8x7B		FDD GPT-4-Preview
	RT (Sec.)	IR	RT (Sec.)	IR	RT (Sec.)	IR	RT (Sec.)	IR
Q1	44.04	Irrelevant	10.79	Irrelevant	3.37	Irrelevant	12.7	Precise
Q2	23.58	Generic	12.04	Irrelevant	2.61	Irrelevant	15.29	Precise
Q3	26.75	Irrelevant	15.56	Irrelevant	5.65	Irrelevant	17.17	Precise
Q4	22.31	Irrelevant	17.66	Irrelevant	2.24	Irrelevant	21.27	Precise
Q5	34.66	Irrelevant	18.28	Irrelevant	2.58	Irrelevant	13.34	Precise
Q6	41.46	Relevant	22.93	Relevant	3.81	Relevant	17.71	Relevant
Q7	22.38	Relevant	7.8	Generic	3.2	Relevant	14.68	Precise
Q8	42.18	Generic	10.10	Generic	5.56	Generic	19.88	Generic
Q9	30.04	Generic	12.95	Generic	5.96	Generic	21.33	Irrelevant
Q10	38.52	Generic	13.85	Generic	4.16	Relevant	17.64	Irrelevant

Table 7. Time response comparison of different LLM models.

Model	${\mathit{IR}}_{\mathit{irrelevant}}$	${\mathit{IR}}_{\mathit{relevant}}$	${\mathit{IR}}_{\mathit{generic}}$	${\mathit{IR}}_{\mathit{precise}}$	${\mathit{Max}}_{\mathit{RT}}$	${\mathit{Min}}_{\mathit{RT}}$
	%	%	%	%	Sec.	Sec.
ChatGPT 4	40	40	20	0	44.04	22.31
Llama 2	50	10	40	0	15.56	7.8
Mixtral 8x7B	50	30	20	0	5.65	2.24
FDD GPT-4-Preview	20	10	10	60	21.33	12.7

Table 7. Time response comparison of different LLM models.

Model	${\mathit{IR}}_{\mathit{irrelevant}}$	${\mathit{IR}}_{\mathit{relevant}}$	${\mathit{IR}}_{\mathit{generic}}$	${\mathit{IR}}_{\mathit{precise}}$	${\mathit{Max}}_{\mathit{RT}}$	${\mathit{Min}}_{\mathit{RT}}$
	%	%	%	%	Sec.	Sec.
ChatGPT 4	40	40	20	0	44.04	22.31
Llama 2	50	10	40	0	15.56	7.8
Mixtral 8x7B	50	30	20	0	5.65	2.24
FDD GPT-4-Preview	20	10	10	60	21.33	12.7

Table 8. Comparison of LLM responses to fault code T04:C31 (Part 1). (a) GPT-4-Preview response, (b) ChatGPT 4 response.

(a)

Q5: For fault code T04:C31, what is the best way to overcome it?

Response: The best way to overcome fault code T04:C31 is to ensure that the parameters of the JSR (Jump to SubRoutine) instruction match those of the associated SBR (SubRoutine) or RET (Return from SubRoutine) instruction. If you pass too many parameters, the extra ones are ignored without any error, but you should pass the appropriate number of parameters to avoid the fault

Response time: 13.34 Sec    IR: Precise

(b)

Q5: For fault code T04:C31, what is the best way to overcome it?

Response: To effectively address fault code T04:C31 in an ABB 800xA system, you should follow a structured troubleshooting process. This process will help identify and resolve the issue associated with this specific fault code. Here are the recommended steps: Consult the System Documentation: Begin by referring to the ABB 800xA system’s documentation or fault code manual. This documentation often provides detailed information about specific fault codes, including T04:C31, and may offer initial troubleshooting guidance or describe the typical causes of the fault. [Continue ...]

Response time: 34.66 Sec    IR: Irrelevant

Table 8. Comparison of LLM responses to fault code T04:C31 (Part 1). (a) GPT-4-Preview response, (b) ChatGPT 4 response.

(a)

Q5: For fault code T04:C31, what is the best way to overcome it?

Response: The best way to overcome fault code T04:C31 is to ensure that the parameters of the JSR (Jump to SubRoutine) instruction match those of the associated SBR (SubRoutine) or RET (Return from SubRoutine) instruction. If you pass too many parameters, the extra ones are ignored without any error, but you should pass the appropriate number of parameters to avoid the fault

Response time: 13.34 Sec    IR: Precise

(b)

Q5: For fault code T04:C31, what is the best way to overcome it?

Response: To effectively address fault code T04:C31 in an ABB 800xA system, you should follow a structured troubleshooting process. This process will help identify and resolve the issue associated with this specific fault code. Here are the recommended steps: Consult the System Documentation: Begin by referring to the ABB 800xA system’s documentation or fault code manual. This documentation often provides detailed information about specific fault codes, including T04:C31, and may offer initial troubleshooting guidance or describe the typical causes of the fault. [Continue ...]

Response time: 34.66 Sec    IR: Irrelevant

Table 9. Comparison of LLM responses for Symphony control system (Part 2), (a) Mixtral 8x7B response, (b) Llama 2 response.

(a)

Q10: Symphony control system reports a configuration error, what steps should be taken to verify and correct the configuration settings?

Response: If a Symphony control system reports a configuration error, here are the steps to verify and correct the configuration settings: Check the system status: Check the system status in the Symphony Plus operator interface or in the engineering workstation. Look for any error messages or fault codes that may indicate the cause of the configuration error. Check the configuration files [... Continue ...]

Response time: 4.16 Sec    IR: Relevant

(b)

Q10: Symphony control system reports a configuration error, what steps should be taken to verify and correct the configuration settings?

Response: If a Symphony control system reports a configuration error, it is important to take steps to verify and correct the configuration settings to ensure that the system is operating correctly and efficiently. Here are some steps that can be taken: 1. Review system logs: Review the system logs to identify the source of the configuration error. The logs can provide information about the specific configuration issue and the affected components. [... Continue ...]

Response time: 13.85 Sec    IR: Generic

Table 9. Comparison of LLM responses for Symphony control system (Part 2), (a) Mixtral 8x7B response, (b) Llama 2 response.

(a)

Q10: Symphony control system reports a configuration error, what steps should be taken to verify and correct the configuration settings?

Response: If a Symphony control system reports a configuration error, here are the steps to verify and correct the configuration settings: Check the system status: Check the system status in the Symphony Plus operator interface or in the engineering workstation. Look for any error messages or fault codes that may indicate the cause of the configuration error. Check the configuration files [... Continue ...]

Response time: 4.16 Sec    IR: Relevant

(b)

Q10: Symphony control system reports a configuration error, what steps should be taken to verify and correct the configuration settings?

Response: If a Symphony control system reports a configuration error, it is important to take steps to verify and correct the configuration settings to ensure that the system is operating correctly and efficiently. Here are some steps that can be taken: 1. Review system logs: Review the system logs to identify the source of the configuration error. The logs can provide information about the specific configuration issue and the affected components. [... Continue ...]

Response time: 13.85 Sec    IR: Generic

The response time analysis revealed significant variations. ChatGPT 4 presented a maximum response time ($Ma{x}_{RT}$) of 44.04 s and a minimum response time ($Mi{n}_{RT}$) of 22.31 s, which suggests deeper analysis but slower processing. Llama 2 improved this, with a ($Ma{x}_{RT}$) of 22.93 s and ($Mi{n}_{RT}$) of 7.8 s, showing enhanced efficiency. Mixtral 8x7B was notably fast, with a ($Ma{x}_{RT}$) of only 5.96 s and ($Mi{n}_{RT}$) of 2.24 s, showing rapid response generation. GPT-4-Preview balanced comprehensiveness and speed with a ($Ma{x}_{RT}$) of 21.33 s and ($Mi{n}_{RT}$) of 12.7 s. Figure 6 illustrates response times, inference relevance, and PSs across 10 questions (Q1–Q10) for various LLMs. ChatGPT 4 consistently showed the highest response times, often exceeding 40 s, mainly for Q1, Q6, and Q8. Mixtral 8x7B maintained the lowest response times, staying below 6 s. Llama 2 and GPT-4-Preview exhibited moderate response times, with Llama 2 typically performing faster than GPT-4-Preview.

Table 6 summarizes the comparative analysis based on inference relevance categories and response time ranges. GPT-4-Preview showed higher precision in inference relevance, whereas Mixtral 8x7B excelled in minimum response time. Nevertheless, Mixtral 8x7B’s speed did not translate to inference relevance, as it shared the highest percentage of irrelevant responses with Llama 2. This suggests that although Mixtral 8x7B and Llama 2 process queries rapidly, GPT-4-Preview provides more accurate responses at the cost of slightly longer processing times. As a result of PS calculations for all models, ChatGPT 4(PS=0.804) has the lowest PS, which indicates that it may be slower and less relevant than the other LLMs. Llama 2 (PS=0.935) performs better than ChatGPT 4 but is not the best among the given models. Mixtral 8x7B (PS=1.504) has a higher PS, which indicates that it is more efficient and relevant in its responses than the other two models. GPT-4-Preview (PS=3.255) demonstrated the highest PS, which signifies superior efficiency and relevance in its responses. These results are summarized in Figure 6c, which illustrates the LLM PSs.

The feasibility of implementing the LLM-based FDD method in the industrial domain is substantiated by the results, both from the implementation and computational perspectives. Through existing APIs and open-source frameworks, the integration of models such as GPT-4-Preview, ChatGPT 4, Mixtral 8x7B, and Llama 2 is achievable. The observed performance metrics, particularly the superior precision of GPT-4-Preview, indicate that model customization via fine-tuning, enriching the knowledge base, or prompt engineering is feasible and highly effective for FDD tasks. The range of response times (12.7-21.33 s) showed a trade-off between accuracy and computational efficiency. The PS metric exemplifies this trade-off, in which GPT-4-Preview’s higher PS of 3.255 justifies a suitable model selection for critical industrial applications.

5.1.2. A Comparative Review of Results: LLM Versus Conventional FDD

In our quantitative analysis, we evaluated the performance of conventional and LLM-based FDD models across several key metrics. In Table 10, the three traditional ML models (LR, RF, and NN) are compared with our proposed FDD GPT-4Preview model. The results reveal a notable trade-off between response time and accuracy.

Conventional models showed superior speed, with LR leading at 0.0526 s, RF at 0.3445 s, and NN at 2.7168 s. By contrast, our GPT-4-Preview model requires 9.3 s for response. However, this increased processing time was offset by the model’s exceptional accuracy of 96.3%, surpassing that of the next best performer, RF, by 5.94 percentage points.

Examining the confusion matrices provides further insight into the strengths and weaknesses of each model. As shown in Figure 7a, the NN model demonstrates strong performance in identifying “Other Issues” (97.75% accuracy) and moderate success with “I/O issues” (40% accuracy). However, it struggles with “Communication Issues” (27.27% accuracy) and “Power Issues” (25% accuracy), which indicates potential areas for improvement.

Figure 7b illustrates the RF performance model , which shows improved results across most categories compared to the NN. It excels in identifying “I/O issues” (70% accuracy) and “Power Issues” (66.67% accuracy). The model maintained high accuracy for “Power Issues” (97.75%) while showing significant improvement in “CPU/Prc. Issues” (72% accuracy).

The LR model, as shown in Figure 7c, exhibits mixed results. It performs exceptionally well in identifying “Power Issues” (99.10% accuracy) but struggles with “Power Issues” (16.67% accuracy) and “CPU/Prc. Issues” (48% accuracy). Its performance on “I/O issues” (40% accuracy) and “Communication Issues” (36.36% accuracy) is moderate.

Although these traditional models offer rapid response times, their variable performance across different fault categories highlights the challenges in achieving consistent accuracy across all fault types. Beyond this quantitative measure, the GPT-4-Preview model, despite its longer processing times, offers superior overall accuracy and enhanced explainability. Our adaptable LLM-based approach can address new fault types through fine-tuning or knowledge based utilization.

Traditional ML models (NN, RF, and LR) require rigorous training, testing, and validation. LLM-based FDD demonstrates superior contextual understanding when compared with conventional models. The combination of high accuracy, interpretability, and adaptability makes the GPT-4-Preview model particularly valuable for complex industrial diagnosis scenarios.

Use Case: Fine-Tuned LLM-Based Multi-Stage FDD Process Workflow

Figure 8 shows the use case of the multistage process workflow to address user queries regarding equipment faults that employ a custom GPT model and Llama 2 to enhance diagnostic precision. In this workflow, a user initiates a query regarding a specific fault code, which is processed by the custom GPT model accessing a vector index constructed from a comprehensive document corpus of unstructured data, such as device manuals and fault records. The system question generator module formulates pertinent questions to refine the context and extract detailed information from the knowledge base. Then, these refined data were fine-tuned using the Llama 2 model to ensure the accuracy of FDD.

The system subsequently generates a detailed response encompassing diagnostic information, recommended resolution steps, and the associated equipment model details. For instance, a query regarding fault code 17.31 for the Siemens STEP device would result in a response that identifies the fault as a minor diagnostic fault, provides steps to resolve the issue, and specifies the related equipment model. This use case demonstrates the integration of advanced AI models into FDD systems, significantly enhancing the accuracy and timeliness of fault diagnostics and recommendations, thereby supporting robust industrial operations. The system generates answers using both the base and fine-tuned models, highlighting the difference in accuracy between them. The process begins when the user enters a query. Custom GPT processes the user’s query by accessing the vector index and retrieving relevant information from the corpus, as shown in Figure 8

As part of refining the response, the system includes a question generator module that formulates relevant questions based on the initial user query, ensuring that detailed and relevant information is gathered from the knowledge base. Figure 9 presents a table of questions and answers that are likely utilized as training data for fine-tuning. The system generates answers using both the base and fine-tuned models, emphasizing the difference in accuracy between them. Figure 10 shows the base model answer, irrelevant to the fine-tuned model’s response. Furthermore, the system allows users to upload a JSON data file to fine-tune the model. The interface indicates the progress and completion of the fine-tuning process as shown in Figure 11a,b, giving the user complete control of the fine-tuning process. The fine-tuned Llama 2 model then processes this refined data further, enhancing the diagnostic accuracy and relevance of the system’s responses. By integrating these components, the workflow ensures that the FDD system can provide precise and contextually relevant responses.

5.1.3. Parameter Tuning and PS Optimization

Optimizing the performance of the proposed method requires a systematic approach to parameter tuning, focusing on maximizing the PS. Key strategies include reducing the average response time ($R{T}_{avg}$) by selecting appropriate LLMs, employing effective preprocessing techniques, using high-quality datasets for fine-tuning, and optimizing vector database configurations. Fine-tuning is critical, as parameters such as the learning rate, number of epochs, and batch size significantly affect model performance. Moreover, the choice of embedding technique and its dimensionality can significantly influence the quality of fault representation and retrieval efficiency from the vector database. The indexing method and similarity search algorithms within the vector database must be carefully considered to achieve an optimal balance between speed and accuracy.

5.1.4. Limitations and Challenges of LLM-Based FDD

Despite the GPT-4-Preview model achieving a 96.3% accuracy in fault diagnosis, outperforming conventional models, several limitations were determined. A significant challenge lies in the model’s lack of true understanding and commonsense reasoning. In approximately 3.7% of cases, the model produced responses that were syntactically correct but contextually inaccurate. Moreover, instances of hallucination were observed, where the model generated irrelevant or factually incorrect responses, were observed in approximately 10% of cases.

Bias is another concern. In ambiguous cases, the model tends to attribute faults more frequently to specific categories, such as CPU/processor issues, over others, like input/output issues. Although specific bias mitigation strategies were not implemented in this study, future efforts could focus on diversifying training datasets, using post-generation self-diagnosis techniques, and employing prompt engineering to encourage balanced fault attribution.

Computationally, fine-tuning a GPT-4 model on a dataset comprising 52,464 words (approximately 69,952 tokens), including fault data from diverse industrial equipment manufacturers over 10 epochs, is computationally intensive. Utilizing an NVIDIA V100 GPU, this operation is estimated to take around 3 h. Moreover, the fine-tuning process incurs significant financial costs due to the resources required for computation and model optimization.

Context management emerged as a significant challenge, but we were able to overcome it. We were initially limited by the model’s token window size of 8,192 tokens. However, we integrated tools including LangChain and LlamaIndex, which enable dynamic document retrieval and sophisticated text-splitting and vector storage techniques.

Finally, reliability in real-time diagnosis is still a significant area for improvement. The model exhibited inconsistent response times, with the most extended delay recorded at 21.27 s for specific fault scenarios.Such variability, especially in real-time industrial applications, can negatively impact reliability, which highlights the need to optimize both model selection and response strategies to meet the strict timing demands of industrial FDD systems.

6. Conclusion

In conclusion, in this research, a novel approach to FDD in industrial systems is presented, leveraging LLMs and multimodal data analysis within Industry 4.0. Our proposed multimodal framework, which incorporates diverse data sources, has significantly advanced industrial diagnostics. Specifically, our FDD GPT-4-Preview model showed superior performance with 96.3% accuracy in fault diagnosis, surpassing traditional methods such as RF ( 90.36% ) and NNs (86.07%).

The proposed FDD GPT-4-Preview model achieved a PS of 3.255, significantly outperforming other models such as Mixtral 8x7B (1.504) and ChatGPT 4 (0.804). It also demonstrated superior performance in relevant diagnostic information across various fault scenarios, with 60% of the responses classified as precise compared to 0% for other models. This study significantly contributes to the application of AI techniques in industrial settings. The proposed framework is a substantial step toward developing more sophisticated diagnostic support tools, offering a 5.94-percentage-point improvement in accuracy over the next best conventional model while providing enhanced explainability through natural language responses.

By contrast, a hybrid architecture has been proposed that combines real-time data streams with fine-tuned LLMs to effectively create a dynamic and responsive FDD system. Our fine-tuned model consistently outperformed the base model, which exhibits enhanced accuracy and contextual understanding in fault diagnosis scenarios.

Although the results are promising, further research is necessary to fully realize the potential of LLM-based FDD systems in various industrial contexts. To enhance data security and confidentiality in industrial settings, future work should explore integrating emerging technologies including blockchain. The decentralization and immutability of blockchain could provide a robust foundation for securing sensitive diagnostic data and ensuring the integrity of FDD processes.

Moreover, fostering vital collaboration between humans and machines through human-in-the-loop systems can further enhance the outcomes of LLM-based diagnostics. This collaborative approach leverages human expertise to validate and refine AI-generated insights, potentially leading to more accurate and contextually appropriate fault diagnosis. By combining advanced AI techniques with secure data management and human oversight, more trustworthy and effective FDD systems that meet the complex demands of Industry 4.0 environments can be created.