ChatGPT vs. Gemini: Which Provides Better Information on Bladder Cancer?

1. Introduction

Bladder cancer is the most common heterogeneous malignancy in the urinary tract, which has variable natural history, types, and treatment options [1]. In Saudi Arabia, the incidence of bladder cancer overall was 1.4 per 100 000 individuals [2]. Additionally, its incidence in Saudi has increased by tenfold in the last decade [3]. The risk factors behind the development of bladder cancer are smoking and exposure to certain chemicals, advancing age, and male gender. Patients usually present with hematuria, whether gross or microscopic, but require further investigations, such as cystoscopy and urography [4]. The decision on the most beneficial treatment option is based on the accurate staging and grading of the tumor as well as the type. Non-muscle invasive urothelial bladder cancer is preferably treated with Transurethral Resection of the Bladder Tumor (TURPT), while in muscle-invasive bladder cancer, radical cystectomy, i.e., removal of the bladder, is implemented. Favorable oncological outcomes of bladder cancer treatment, however, require a multidisciplinary approach [5]. Moreover, since it is a disease that results in severe consequences that affect several aspects of life, early detection through screening, in addition to engaging the patient in cancer care through education, are all essential to improve patient outcomes and quality of life [1,6]. For this reason, it has become essential to empower patients to take a proactive approach when it comes to their care by improving the educational resources and tools, and raising their awareness about their condition [6].

Artificial Intelligence (AI) systems have gained popularity in recent years for providing information and assisting users online [7]. Moreover, the utilization of the internet for health-related reasons, such as information about cancer, has significantly increased [8,9]. Some bladder cancer patients may utilize the internet as a means to get more information about their condition, and the current technological advancements in Large Language Models (LLM) chatbots have provided another tool from which health information can be obtained [10,11]. Chat Generative Pretrained Transformer (ChatGPT), which was recently developed by the company OpenAI, has captured the attention of many with its ability to provide information and build conversations with users due to its deep learning programming. ChatGPT provides answers in a contextualized manner that are relevant to the input of the user. These features made the application of ChatGPT as a source of medical information a reality [12]. Another emerging AI chatbot that was released by Google is Bard, which is powered by LaMDA and incorporates language recognition and conversation processing while interfacing with Google search [13].

This new era has raised both hopes for the future applications of these tools in healthcare as well as concerns about their current quality and accuracy of responses that deliver critical medical information, especially when it comes to cancer [9,14]. Furthermore, monitoring these platforms’ outputs for accuracy and their ability to keep up with the changing nature of medical sciences and its new and increasing advancements is a necessity, given the importance of this information in the cancer research field and to healthcare communicators [9].

However, limited studies have established data about the quality and accuracy of information that these chatbots provide to cancer patients. Therefore, the aim of this study is to assess the effectiveness of Large Language Models (LLMs), ChatGPT (3rd and 4th generations), and Gemini, in providing bladder cancer patient information about their condition, and evaluate the quality of the answers provided to these patients.

2. Materials and Methods

Frequently asked questions from various trusted educational websites, were compiled into an excel file. The criteria on which these questions were selected are (1) questions frequently asked by patients and the public (2) questions that target general disease knowledge, treatment, diagnosis, and prevention. These questions were then reviewed by board-certified urologists for optimum selection. The questions were then inputted into two LLMs (ChatGPT 3, Chatgpt-4, and Gemini) all available at https://chat.openai.com/chat and https://gemini.google.com/app website. The quality of responses was assessed based on accuracy, comprehensiveness, patient readability, and stability. A 3-point scale was employed to measure accuracy, with one indicating correct information, two indicating a mix of correct and incorrect, and three indicating completely incorrect information. For assessing the comprehensiveness of responses, a 5-point Likert scale was used, where one represented “very comprehensive” and five represented “very inadequate.” As for readability, the output answers, sentences, words, syllables per word, and words per sentence were the factors analyzed. Moreover, the Flesch Reading Ease Score and Flesch–Kincaid Grade Level were calculated for each text using the online calculator available at https://charactercalculator.com/flesch-reading-ease/ website. A higher Flesch Reading Ease Score signifies that the text is easy to read, whereas the Flesch–Kincaid Grade Level indicates the educational grade level required to comprehend the text. The Stability of the output text was analyzed due to the various responses generated for the same question by the LLMs. Stability was assessed by two independent reviewers who subjectively evaluated whether the second and third answers were accurate compared to the first generated answer. Three responses were generated for each question, with the chat history cleared after each trial. The first answers to these questions were evaluated separately by two board-certified urologists all referring to the same resource for answers for accuracy and comprehension. Any discrepancies in the evaluation were then independently resolved by a blinded third board-certified urologist.

Statistical Analysis

Statistical analysis was conducted using RStudio software (version 4.3.1). Descriptive statistics were employed to summarize the characteristics of different Large Language Models (LLMs) and their performance across various categories related to bladder cancer. Categorical variables were expressed as frequencies and percentages. To assess the association between LLMs and categorical variables, Fisher’s exact test was utilized. Additionally, continuous variables, such as grade level scores, were presented as medians with interquartile ranges (IQR), and the Kruskal-Wallis rank sum test was applied to evaluate significant differences among LLMs. All statistical tests were two-tailed, and p-values less than 0.05 were considered statistically significant.

3. Results

In the current study, three large language models (LLMs), including ChatGPT-3.5, ChatGPT-4 and Gemini, were assessed using 12 general questions, 6 diagnosis-related questions, 28 treatment-related questions and 7 prevention-related questions, which accounted for 22.6%, 11.3%, 52.8% and 13.2%, respectively (Figure 1).

Table 1 and Figure 2 present the analysis of the accuracy of different LLMs in providing information about bladder cancer. The overall analysis of 53 questions revealed no significant differences among the LLMs (p = 0.655). Across all categories, the proportion of correct responses was high, with 92.5% for ChatGPT-3.5 and ChatGPT-4, and 86.3% for Gemini. There were no significant differences in the proportions of correct answers among the three LLMs in terms of the questions of general domains (p > 0.999), diagnosis (p > 0.999), treatment (p = 0.848) and prevention (p = 0.079).

The overall analysis revealed a significant difference among the LLMs in terms of their comprehensiveness (p = 0.011). Comprehensive and very comprehensive responses were provided for 75.4% of questions by ChatGPT-3.5, 83.0% by ChatGPT-4 and 68.6% by Gemini. Additionally, in the treatment category, ChatGPT-3.5 displayed a higher combined proportion (75.0%) compared to ChatGPT-4 (78.6%) and Gemini (57.7%), and the difference was statistically significant (p = 0.007, Table 2). No other question categories showed statistically significant differences among the LLMs.

Table 3 presents an analysis of grade level scores among different Large Language Models (LLMs) regarding bladder cancer information. The overall comparison revealed a significant difference among the LLMs (p < 0.001), with college-level responses apparent in 34.0% of the questions by Gemini, 69.8% by ChatGPT-3.5 and 75.5% by ChatGPT-4. Furthermore, in the treatment category, ChatGPT-3.5 had a proportion of 71.4%, ChatGPT-4 had 75.0%, and Gemini had 39.3% categorized as college-level questions. This discrepancy was statistically significant (p < 0.001). In general questions, Gemini had significantly lower proportions of college-level questions (16.7%) compared to ChatGPT-3.5 (66.7%) and ChatGPT-4 (83.3%). In the diagnosis category, ChatGPT-3.5 had a proportion of 66.7%, ChatGPT-4 had 83.3%, and Gemini had 16.7% categorized as “College,” although this difference was borderline significant (p = 0.050, Table 3).

Table 4 illustrates the analysis of the reading note of different Large Language Models (LLMs). The overall comparison revealed a significant difference among the LLMs (p < 0.001), with difficulty in reading for 69.8% of responses provided by ChatGPT-3.5, 75.5% of ChatGPT-4 and 34.0% of Gemini. Notably, in the treatment category, ChatGPT-3.5 had 71.4%, ChatGPT-4 had 75.0%, and Gemini had 39.3% categorized as “Difficult to read”, indicating a substantial discrepancy among them (p < 0.001). Similarly, in the diagnosis category, ChatGPT-3.5 had 66.7%, ChatGPT-4 had 83.3%, and Gemini had 16.7% categorized as “Difficult to read,” and the difference was statistically significant (p = 0.019, Table 4).

There were notable variations observed among LLMs for the number of words, with ChatGPT-3.5 showing a median of 207.0 (IQR = 166.0 to 240.0), ChatGPT-4 with 295.0 (IQR = 232.0 to 342.0), and Gemini with 274.0 (IQR = 223.0 to 341.0), indicating an increasing trend from ChatGPT-3.5 to Gemini to ChatGPT-4 (p < 0.001). Similarly, a trend of increase was observed for sentences, syllables, and syllable/word ratio, with ChatGPT-4 consistently showing the highest values followed by Gemini and then ChatGPT-3.5. For instance, the median number of sentences was 10.0 (IQR = 7.0 to 13.0) for ChatGPT-3.5, 18.0 (IQR = 10.0 to 21.0) for ChatGPT-4, and 15.0 (IQR = 11.0 to 21.0) for Gemini (p < 0.001). Regarding the FRE Score, ChatGPT-3.5 had a median of 43.4 (IQR = 34.2 to 48.0), ChatGPT-4 had 40.3 (IQR = 35.0 to 44.9), and Gemini had 54.3 (IQR = 47.4 to 61.7). However, no significant difference was observed in the FK Reading Level among the LLMs (p = 0.093, Table 5 and Figure 3).

The stability analysis was performed on 10 questions, among which three questions were related to diagnosis, other three questions were related to treatment and four questions were related to prevention. Results showed no significant differences in stability between the three LLMs under investigation for the overall questions and for each subscale (Table 6).

4. Discussion

Our study focused on evaluating the performance of two LLMs in answering questions related to bladder cancer. While previous studies have reported that ChatGPT provides unsatisfactory results regarding bladder cancer information, our findings demonstrated promising accuracy rates, consistent stability, and varying levels of comprehensiveness among the three LLMs [15,16].

LLMs have demonstrated high accuracy in providing information about bladder cancer, underscoring their potential as valuable resources in the medical field. Our findings align with those of Ozgor et al., who reported that ChatGPT’s responses to questions about urological cancers exhibited high accuracy rates. However, when evaluated against the EAU 2023 Guidelines, Ozgor et al. found ChatGPT’s performance to be inadequate [15]. Additionally, ChatGPT’s responses to various urologic conditions, including bladder cancer, were generally well-balanced, though treatment-related answers only achieved a moderate quality score on the DISCERN questionnaire, scoring 3 out of 5 points [16]. Consistent with these findings, our study also identified that most inaccuracies were related to bladder cancer treatment. Similarly, Musheyev et al. assessed the quality and accuracy of information about urological cancers provided by four AI chatbots (ChatGPT, Perplexity, ChatSonic, and Microsoft Bing AI) using the top five Google Trends search queries. While these AI chatbots generally delivered accurate and moderately high-quality information, they often lacked clear, actionable instructions [17].

Although all LLMs delivered highly accurate responses regarding bladder cancer information, instances of errors were observed, which stemmed from factors such as misinterpretation of ambiguous questions, reliance on outdated or incorrect sources, and a lack of contextual understanding of nuanced medical topics. These errors, despite the overall high accuracy rates, underscore the importance of critical evaluation by medical professionals. While LLMs show great promise in providing accurate information about bladder cancer, it is essential to recognize and address their limitations and potential errors to ensure their safe and effective use in a medical context.

It is evident how crucial it is for readers to be able to accurately read, comprehend, and apply information about their condition. Literature has reported that understanding AI-based LLM results can be challenging, with some findings indicating that reading and comprehending this content requires adequate training [18,19,20]. Our study demonstrated that Gemini provided more easily readable answers compared to both ChatGPT-3.5 and ChatGPT-4. Multiple studies have confirmed that ChatGPT frequently produces responses at a post-secondary, particularly college, grade level [18,21,22]. Additionally, as shown by Abou-Abdallah et al., with a mean FRES value of 38.9, placing it in the fairly difficult category, ChatGPT’s readability was considered poor [23]. A study assessing the readability of AI chatbot responses to the 100 most frequently asked questions about cardiopulmonary resuscitation found that Gemini’s responses were the easiest to read, while ChatGPT’s were the most challenging [24]. Regarding the stability and reproducibility of the responses, all generated answers were consistent. However, since only ten questions were evaluated for stability and compared across the three LLMs, it is important to test all research questions in future studies to accurately determine their stability.

Limitations

This study has offered valuable insights into the potential integration of LLM chatbots in health education. However, certain limitations should be acknowledged and addressed in future research. For instance, ChatGPT’s knowledge base is only updated until September 2021, which may limit the relevance of its responses. Additionally, although independent evaluation was conducted to ensure blinding to the type of LLM, factors such as the formatting of the answers could inadvertently reveal the identity of specific LLMs. Despite these limitations, the study remains reliable. Future research should consider including other LLMs and focus on crafting contextualized questions that closely mimic real-world scenarios.

5. Conclusions

In conclusion, the application of AI in bladder cancer education is steadily emerging with promising potential. Our study contributes valuable insights into this field by highlighting the accuracy, comprehensiveness, and stability of the three LLMs in answering bladder cancer-related inquiries. These findings, together with future research, can further support the effective utilization of AI in medicine.