Evaluation of the Quality of ChatGPT’s Responses to Top 20 Questions about Robotic Hip and Knee Arthroplasty: Findings, Perspectives and Critical Remark on Healthcare Education

1. Introduction

Robotic-assisted knee and hip arthroplasty have emerged as innovative surgical techniques, revolutionizing the field of joint replacement. The integration of robotics into these procedures has promised a paradigm shift, aiming to enhance surgical accuracy (Figure 1), improve patient outcomes, and expedite postoperative recovery [1,2,3]. Robotic-assisted arthroplasty offers preoperative planning tools, intraoperative guidance, and real-time feedback mechanisms, enabling surgeons to execute procedures with meticulous accuracy [4,5]. (Figure 2) Indications for robotic-assisted knee and hip arthroplasty extend to complex anatomies, younger and more active patient populations, and revision surgeries, where their precision can be particularly advantageous. Nevertheless, the adoption of robotic systems in orthopedic surgery comes with a set of challenges, including cost considerations, a learning curve for surgeons, and limitations in specific clinical scenarios. Their cost-effectiveness and broader applicability remain subjects of ongoing research and debate [6,7].

This technological revolution has gained considerable interest and curiosity among various stakeholders, including patients seeking information about their potential treatment options, healthcare professionals striving to stay abreast of cutting-edge advancements, and researchers exploring the efficacy and nuances of robotic-assisted procedures.

Given the surge in interest and the ever-growing quest for credible information in the digital age, AI language models like ChatGPT have emerged as potential gateways to disseminate knowledge and address inquiries related to complex medical procedures [8,9]. The information provided by AI models has the potential to serve as a valuable resource for patients, caregivers, and healthcare professionals seeking knowledge and guidance regarding these surgical interventions. However, the reliability and accuracy of the information dispensed by such models are critical, especially in the context of medical procedures [10]. ChatGPT, powered by deep learning algorithms, has been trained on a diverse array of data sources, providing it with a breadth of information across multiple domains, including healthcare and orthopedic surgery. However, the reliability and accuracy of AI-generated information in the medical domain, especially concerning intricate procedures such as robotic hip and knee arthroplasty, have been subjects of scrutiny. Patients contemplating joint replacement surgeries often seek detailed insights into the procedure, potential benefits, associated risks, postoperative care, and long-term outcomes. Similarly, healthcare professionals rely on accurate information to counsel and guide patients through the decision-making process, while researchers and practitioners delve into the nuances of these technologies to advance the field further. The intersection of advanced medical technology and AI-driven information dissemination presents a unique opportunity to address the growing need for accessible and reliable medical information. Evaluating the proficiency of AI language models, like ChatGPT, in providing accurate and comprehensive responses to inquiries regarding hip and knee robotic arthroplasty is crucial. By analyzing the quality, accuracy, and depth of information presented by these models, we can better understand their capabilities and limitations in meeting the informational needs of diverse stakeholders.

This study endeavors to critically assess ChatGPT’s efficacy in addressing the top 20 questions frequently posed about hip and knee robotic arthroplasty. Through rigorous analysis and expert evaluation, this research aims to provide insights into the reliability of AI-generated information and its potential role in complementing traditional sources of medical knowledge. In doing so, we seek to illuminate the strengths and limitations of AI language models in the context of complex medical information dissemination, paving the way for informed decision-making and improved accessibility to reliable healthcare information in the era of advancing technology.

2. Materials and Methods

The methodology was designed to ensure a systematic and objective assessment of the AI language model’s proficiency in addressing queries related to hip and knee robotic arthroplasty. (Figure 3).

We selected a set of top 20 questions concerning robotic hip and knee arthroplasty on Google (extracted by OpenAI ChatGPT-4 version itself) to analyze the quality of the AI chatbot answers (Table 1). ChatGPT has been encouraged to elaborate informative and detailed responses, ensuring that the model has sufficient context to generate meaningful answers. These responses were logged and stored for subsequent analysis.

A panel of three expert orthopaedic surgeons evaluated the quality of ChatGPT responses. These reviewers possessed a background in orthopedic surgery (specifically in the area of hip and knee arthroplasty) and a laudable academic career (having achieved National qualification for Full Professor). They independently assessed the responses generated by ChatGPT in terms of accuracy and completeness with a predefined set of evaluation criteria. A five-point Likert scale (score 1-5) was used to assess the accuracy: 1 – completely incorrect, 2 – more incorrect than correct, 3 – approximately equal correct and incorrect, 4 –more correct than incorrect, and 5 – correct. A four-point Likert scale (score 0-3) was used to examine the completeness of the response: 0 – comprehensiveness not assessable for completely incorrect answers/accuracy scale 0; 1 – incomplete; substantial parts of the answer rare incomplete or missing, 2 – satisfying; it provides the minimum amount required to give complete information, addressing all aspects of the question and 3 – exhaustive; all aspects of the question are investigated with additional information beyond expectations.

Statistical tests were applied to assess the significance of differences in response quality across various questions and to evaluate the impact of fine-tuning on response quality. The evaluation was performed twice, at a distance of 15 days each (t1 and t2) by each of the three expert orthopaedic surgeons to get adequate internal consistency. Wilcoxon and Mann-Whitney tests were performed to check for possible differences both within and between surgeon evaluations of the answers.

Institutional Review Board approval was not necessary, since neither animal nor human subjects were involved in this study; all data utilized was available for public use.

3. Results

The average accuracy of answers for questions regarding robotic hip and knee replacement was 4.37 (0.89) and 4.6 (0.58) points out of 5 respectively, while the average completeness was 2.02 (0.42) and 2.07 (0.38) points out of 3. No significant differences were found among answers at t1 and t2 (all P-values are over the significance threshold) for each surgeon; detailed results of the comparison of evaluation within and between surgeons at t1 and t2 are reported in Table 2.

In aggregate, ChatGPT demonstrated a reasonable level of proficiency in addressing inquiries related to hip and knee robotic arthroplasty. The model’s ability to provide accurate and relevant information, particularly in outlining the fundamental aspects of the procedures and their potential benefits, was notable.

4. Discussion

The dynamic nature of robotic arthroplasty is reflected in the integration of emerging technologies. Artificial intelligence (AI) and machine learning algorithms are being employed to enhance preoperative planning and predict patient-specific outcomes [11,12]. Augmented reality (AR) systems are being explored to provide surgeons with real-time, 3D visualization of the surgical field, aiding in precise implant placement [13,14,15]. The integration of robotic technology offers the promise of enhanced precision, improved implant survivorship, and tailored patient care (Figure 2). While substantial progress has been made, further research is required to address challenges related to training, cost-effectiveness, and the integration of cutting-edge technologies.

As these challenges are overcome, robotic arthroplasty is poised to play an increasingly prominent role in the future of orthopedic surgery, ultimately benefiting patients and healthcare systems alike [16]. The general opinion of common consumers regarding robotic surgery is a mix of fascination, curiosity, and some common myths and potential disappointments. Many consumers believe that robotic surgery offers greater precision and accuracy compared to traditional methods, which can lead to better surgical outcomes (Figure 2). Robotic surgery is often seen as minimally invasive, with smaller incisions and potentially faster recovery times. Most consumers may have misconceptions that robots perform surgeries entirely autonomously (in reality, surgeons control the robotic systems). Furthermore, the idea of a quicker recovery and shorter hospital stays is attractive to patients who want to return to their daily lives sooner [17]. The significance of accurate and comprehensive patient education in robotic knee and hip arthroplasty is paramount within the realm of orthopedic healthcare. Correct patient education ensures that individuals undergoing robotic knee and hip arthroplasty are equipped with a thorough understanding of the surgical procedure, including potential risks and benefits. Patients who receive accurate education are more likely to adhere to prescribed treatments, exercise regimens, and follow-up appointments, ultimately resulting in improved surgical outcomes [18]. In addition, such education helps manage patient expectations, minimizing postoperative anxiety and enhancing overall patient satisfaction. Moreover, it contributes to the efficient allocation of healthcare resources by reducing the likelihood of complications and readmissions, thereby curbing healthcare costs. On these premises, the utilization of AI-driven models, such as ChatGPT, to provide information and answer questions about these procedures has gained prominence [19]. Common users traditionally look for information online (with Google being the most popular search engine) but an increasingly larger share interacts with chatbots even for healthcare understanding. ChatGPT and Google serve different purposes in patient education. ChatGPT offers a conversational and personalized approach with the advantage of accessibility and customization but lacks emotional support and may have limitations in medical expertise. Google provides a vast repository of healthcare information but requires patients to sift through search results, potentially leading to information overload and a higher risk of encountering misinformation [20].

The choice between these modalities depends on the specific needs of patients and the healthcare context, and in some cases, a combination of both may provide a comprehensive patient education strategy. It is important to note that while ChatGPT provides information based on a vast dataset, the quality of responses can vary depending on the complexity and specificity of the queries [21].

Our analysis reveals that ChatGPT provides a relatively high degree of accuracy and relevance, especially for factual questions. The completeness and depth of explanations can be considered satisfying since the AI-driven chatbot ensures the minimum amount required to give complete information. The evaluation of ChatGPT’s responses revealed several noteworthy findings. In assessing the accuracy of ChatGPT’s responses, it was observed that the model generally provided information that was consistent with established medical literature. For instance, the response to the question, “What is the typical recovery time and rehabilitation process after robotic knee replacement?” provided accurate information about the general recovery timeline and the role of physical therapy. This aligns with well-documented post-operative protocols in knee arthroplasty [22]. Similarly, responses related to patient eligibility criteria, risks, and benefits of robotic joint replacement were in concordance with medical literature [23]. ChatGPT demonstrated the capability to provide tailored information, especially in response to questions about customizing implant placement in robotic arthroplasty. The model appropriately emphasized the importance of customization based on the patient’s unique anatomy and the surgeon’s intraoperative adjustments. This aligns with the evolving trend in personalized medicine within orthopedics [24]. The results of our research are in line with the findings of the study performed by Kienzle et al. [25]. In this paper, the authors subjected ChatGPT to a set of questions, mirroring those commonly posed by patients in the preoperative assessment phase before undergoing total knee arthroplasty. All queries were framed in simple non-technical language and were presented to ChatGPT through a single continuous chat session. A panel of three orthopedic surgeons independently rated the responses elaborated by the AI chatbot, employing the DISCERN instrument to validate the accuracy of the answers. The authors found consistently high scores, suggestive of the good quality and accuracy of the information provided. Anyway, at a closer examination, the authors highlighted the generation of non-existing references for certain claims, emphasizing the necessity of cross-referencing information from established sources. Similarly, Mika et al. in their study examining ChatGPT responses to common patient questions regarding total hip arthroplasty [26] appreciated the quality of the chatbot responses, having provided evidence-based answers in a way that most patients could understand. The authors, based on these results, considered ChatGPT an adequate clinical tool for patient education before orthopedic consultation. On the other side, these results differ from the observations made by Yang et al. [27] who compared the responses of two AI chatbots (ChatGPT and Bard) regarding treatments for hip and knee osteoarthritis with the American Academy of Orthopaedic Surgeons (AAOS) evidence-based clinical practice guidelines (CPGs) recommendations. According to their results, ChatGPT and Bard do not consistently provide responses that align with the AOOS CPGs, encouraging the use of non-recommended treatments in 30 and 60% of queries, respectively. On this basis, the authors ask patients and healthcare professionals for prudence to not blindly trust the information provided by the AI chatbots. Assessing the credibility of information provided by a chatbot, like ChatGPT, is essential to ensure that users receive accurate and trustworthy information. Source verification plays a crucial role in healthcare patient education, as it directly impacts the accuracy, reliability, and trustworthiness of the information patients receive [28]. By steering patients toward credible sources, healthcare providers can help mitigate the negative effects of misinformation [29]. Continuing the reasoning, human interaction remains superior to ChatGPT and other AI-driven tools in several aspects of healthcare patient education. While AI can provide valuable support, there are certain qualities and capabilities that human interactions offer. Human healthcare providers can empathize with patients, offering emotional support, reassurance, and comfort during difficult moments. They can address patients’ fears, concerns, and emotional needs in a way that AI cannot replicate. They can adapt their communication style and content to meet the individual needs of each patient. They can recognize and respond to unique cultural, emotional, and cognitive factors that influence a patient’s understanding and engagement. Much of human communication relies on non-verbal cues such as body language, facial expressions, and tone of voice. These cues convey empathy, sincerity, and reassurance, enhancing the patient-provider relationship and the understanding of information. In complex medical situations, healthcare providers can engage in nuanced discussions, explaining intricate medical concepts and addressing patients’ questions in real time [30]. They can navigate uncertainty and adapt their responses to the patient’s level of understanding. Human healthcare providers can demonstrate cultural competency, taking into account a patient’s cultural background, beliefs, and values when delivering information. This cultural sensitivity fosters better communication and understanding. In a conversation with a human healthcare provider, patients can seek immediate clarification or ask follow-up questions, fostering a dynamic and iterative learning process. Healthcare providers can offer real-time feedback and gauge patient comprehension. Building trust and rapport with patients is a critical aspect of healthcare [31]. While AI, including ChatGPT, has made significant advancements in healthcare patient education, it should complement rather than replace human interactions. The unique qualities of human providers, including empathy, adaptability, cultural competency, and the ability to provide emotional support, make them indispensable in delivering patient-centered care and education. A balanced approach that leverages both human expertise and AI capabilities can provide the best outcomes for patients [32].

5. Conclusions

The evaluation of ChatGPT’s responses to the top 20 questions regarding robotic hip and knee arthroplasty indicates that the model offers accurate, well-informed, and tailored information that aligns with established medical literature and guidelines. However, it is essential to recognize that AI models, including ChatGPT, are not a substitute for professional medical advice and should serve as complementary sources of information. As the field of robotic joint arthroplasty continues to evolve, ongoing assessments of AI-driven models like ChatGPT can contribute to their refinement and utility in providing valuable insights to patients and healthcare professionals.