Evaluation of Replies to Voice Queries in Gynecologic Oncology by Virtual Assistants Siri, Alexa, Google, and Cortana

1. Introduction

The internet is widely used by patients to obtain health information [1]. Voice technology allows internet searches through verbal queries which are answered by a virtual assistant. Virtual assistants (VAs), such as Siri (Apple), Alexa (Amazon), Cortana (Microsoft), and the Google Assistant, are used ubiquitously. Google Assistant deploys searches on Google, as does Siri, while Alexa and Cortana use Bing as their search engine [2,3]. In 2020, Google Assistant was available on >1 billion devices and was used by > 500 million users monthly [4] providing 27% of all global web searches. More than 500 million Apple customers use Siri as a virtual assistant [5]. The Amazon Echo Home Speaker alone has >40 million users in the United States, which uses Alexa VA [6]. Siri, Alexa, Google and Cortana have been voted in the top 10 best VAs, hence why we have chosen these VAs to investigate [7]. Since the introduction of Siri as a feature of the iPhone 4S in 2011, VAs have become mainstream. Siri can make phone calls or send text messages when users cannot manually enter information while driving, biking, or walking [8]. VAs have utility for the elderly in situations that are simplified by speaking especially when there is physical impairment [9]. There have been steady declines in the personal computer market for the last eight years as people opt to go online with smartphones where voice queries are easier than keying in searches [10,11]. A surge in the use of VAs is underway in healthcare. VA usage increased to 21.0% of U.S. adults in 2021 with 54.4 million people using VAs for questions in healthcare about symptoms, medical information, and treatments [12]. VA responses to voice queries have been utilized for information on postpartum depression [12] healthy lifestyles [13], vaccinations [14], addiction [15], mental health, and interpersonal violence [16]. Several US hospitals have smart speakers installed in patients’ rooms enabling them to make requests that can relieve the clinical staff [17]. Widespread access and utilization can be contrasted by the quality of the results received from VA’s, especially in extending simple everyday voice queries to more complex questions related to GYN-ONC. The literature for VA queries in oncology or gynecologic oncology (GYN-ONC) does not exist, thus this paper investigates for the first time the degree to which voice queries related to GYN-ONC are accurately addressed by audible replies from Siri, Alexa, Google, and Cortana.

2. Materials and Methods

Power was calculated between the VAs using McNemar’s test for comparing the percent correct between two sources of information. The McNemar’s test was used because it was determined to be the best fit for our data set in which the data was paired and proportions were different. To calculate power, VA performance was estimated based on general queries made by two investigators as shown in the “Percent correct responses” line of Table 1 (bottom line). In this setting, Google was correct (83.3%) more often than Siri (45.8%: P value = 0.27, power = 0.90) and Cortana (20.8%: P value = 0.0003, power = 0.99) but not Alexa, and Alexa (66.7%) was correct more often than Cortana (20.8%: P value = 0.0023, power = 0.98). By enlarging the data set to additional rounds of 24 queries, assuming the results obtained in Table 1 would continue to hold, then to detect a difference between Google and Alexa, 90 queries would attain 90% power and this would hold for Alexa vs Siri, as well as Siri vs Cortana. To approach this power, 21 evaluators each presented a set of 24 questions specific to GYN ONC in querying the VAs. The general questions used for estimating power were selected from QUIZ Daily (1550 Larimer Street, Suite 431, Denver, CO 80202) and checked against Google and Bing for matching correct answers. Our baseline evaluation of VA performance was within 1-3% of the performance reported by others, except for Cortana for which our estimate was lower (20.8% vs 45%) [18,19]. We utilized a team of 21 evaluators making queries in anticipation of variability due to different devices and in the interpretation of VA replies by individuals that were provided with correct answers at the risk of the study being over-powered. Evaluators were chosen at random. Evaluators dialect, skills, and speech characteristics all varied, but we do not believe this alters the data because each VA is equipped to assess over 100 languages and dialects. Each evaluator accessed the most recent updated versions of Siri (iOS 14), Alexa (version 2.2.427375.0), and Google (version 1.9.28702) on their smartphones. To access Cortana (version 3.2106.14307.0), a Dell Latitude E6430 Windows 10 laptop was provided to each evaluator that did not have access to Windows 10, while evaluators who already had access completed the test on their own device. The panel of 24 questions related to GYN-ONC were posed in three different templates to all the VAs by 21 evaluators. A 3-tier template was used for each inquiry: “X?” (A), “What is X?” (B), and “Define X?” (C) in order to maximize the chances of a correct answer by each VA. The questions asked to each VA are listed in Table 2 in the 3-tier template. We weighted the 24 question panel toward ovarian cancer queries (16 questions) with the premise that the lower incidence of ovarian cancer might provide a more robust test of the VAs than more common gynecologic malignancies. In addition, six queries related to more prevalent cervical cancers were included. The remaining two questions were aimed at borderline epithelial tumors of the ovary and endometrial cancer. All questions are based on an expectation of correct knowledge by a fellow in training in GYN ONC.

Individuals making queries were provided answers to queries against which responses returned by VAs were graded. Correct answers were determined from consensus recommendations published by the Society of GYN-ONC, the GYN-ONC Group, and the American College of Obstetrics and Gynecology. Audible answers from the VAs were scored as incorrect = 0; does not understand or know or returns only web-links = 1, <40% correct = 2, 40-50% correct = 3, 50-85% correct = 4, 100% correct = 5.

3. Statistical Analysis

We calculated the overall intra-class correlation coefficient (ICC) using the results from a one-way ANOVA on Winstat (version 2012.1) and calculated individual coefficients of variation for each question, as well as results across VAs and templates. Finally, we quantified the responses across the VAs by aggregating the frequency of each score and have reported the respective results in percentages. Graded scores were averaged across all graders and expressed with the standard error of the mean (SEM). An average graded score + SEM was calculated across individuals that evaluated responses by each VA. An average % score was determined as a percentage of the total score possible. Median scores and 75th percentiles were determined as a function of the total score possible. Minimum and maximum scores were determined as a function of the total score possible and the difference between maximum and minimum as a function of the total possible score. Counts of total correct answers to VA queries are expressed as a percentage of total queries. Significant differences were determined as p<.05.

Chi-square and Fisher’s exact probability tests were used for non-parametric analyses, and Student’s t-test was performed to compare means in the time-series study.

4. Results

Google provided the most correct audible replies with the general questions. (n_correct = 20; 83.3% correct), followed by Alexa (n_correct = 16; 66.7% correct), Siri (n_correct= 11; 45.8% correct), and Cortana (n_correct= 5; 20.8% correct). Differences in correct audible responses to the general voice queries occurred with Google > Alexa > Siri > Cortana (Table 1). Regarding queries related to GYN-ONC, Google’s average graded score (2.88 ± 0.04, Table 3(B) mean ± SEM) was significantly higher (p<0.01) than the average graded scores for Alexa, Siri, and Cortana (1.60 ± 0.04, 1.52 ± 0.04, 1.28 ± 0.03, Table 3(B)). The graded score for Cortana was statistically lower than the other VAs. When the average graded scores were expressed as a percent of the highest possible score, Google graded almost twice as high as the other VAs (57.6% ± 0.9 vs. 32% ± 0.8, 30.5% ± 0.8, 25.7% ± 0.6%, Table 3(C)). Google’s performance was mirrored by comparing scores at the median (Table 3(D)) and at the 75th percentile (Table 3(E)). Although the range displayed between the lowest and highest maximum scores was wide (Table 3(H)), the agreement between graders showed acceptable reliability with an ICC score of 0.525. Totally correct replies to Google queries yielded a significantly higher number of responses graded as totally correct (n=222, 18.2%) than the other VAs (2.3-6.5%), Table 3(I). Examination of the query formats in terms of totally correct responses showed that there was no significant difference in the Google responses to the three formats, while Alexa, Siri, and Cortana had more correct responses to the “What is X” format, p<.05 (Table 4). Consequently, query formats can influence VA responses to queries. The number of queries varied due to dissimilar access devices for VA applications. The total number of evaluators for Google were 17, 16 for Alexa, 14 for Siri, and 14 for Cortana. Nevertheless, the data still proves that Google provided the most accurate responses for both general queries and Gynecologic Oncology related questions.

5. Discussion

In summary, audible replies by VAs to queries related to GYN-ONC have room for improved accuracy. Our findings support those of Bickmore et al. that patients should not rely solely on VAs to answer medical questions [20]. For the queries evaluated, a well-trained gynecologic oncologist would answer the queries with more accuracy than the VAs. Our results are consistent with surveys showing that the Google Assistant performed better than Siri or Alexa in addressing non-clinical queries [19]. VA responses related to family medicine revealed room for improvement [21], and did not provide addiction help [22]. We have not found reports on VA queries by GYN-ONC patients; however, virtual visits to gynecologic oncologists have increased during the pandemic [23] and should continue in the future [24]. A proof-of-concept utilization of a virtual assistant has been reported for creating a molecular tumor board in GYN-ONC to integrate automated methods in collaborative treatment decisions [25]. Certain GYN-ONC providers have already introduced “live chat” [26,27] which can be readily augmented with “voice chat” through specialized VAs. A data-driven approach, substituting virtual visits for in-person visits, has been suggested for the identification of symptoms related to ovarian cancer recurrence [28]. While not specific to GYN-ONC alone, the Alberta Health Service is the first public health care organization in Canada to offer health care information via voice queries on Google and Amazon devices [29]. In addition, the United Kingdom’s National Health Service is partnering with Amazon Alexa, to answer health-related questions [30]. Amazon is also partnering with a telemedicine provider to start a voice-activated virtual program that prompts a call back from a telemedicine physician [30]. Our view is that the availability and utilization of VAs for medical information is increasing. This paper indicates that improvements are needed in the accuracy of GYN-ONC information provided by the VAs considered here. Differences in VA performance may be related to the search engine employed by the VA, as well as the proprietary artificial intelligence employed by each. The degree to which physical devices can clearly interpret speech without interference from background sounds is also a factor in VA performance. In the present study, 21 evaluators made 24 queries in three different formats, using different access devices and hence should mirror performance expected in the real world. The present work provides an empirical standard for evaluating the reliability of information obtained from virtual assistants. Inaccuracy may be reduced by improvements at the search engine and VA artificial intelligence levels. Utilization of VAs internationally is a function of language and dialect [31]. Alexa supports 8 languages and 10 dialects, while Google Assistant supports 12 languages and 13 dialects and has been working on more than 115 languages capable of speech recognition and natural language understanding [32]. By early 2022, Google became conversant in 30 languages in 80 countries, with Siri supporting 21 languages in 36 countries, and Cortana supporting 8 languages in 16 countries [32]. VA availability in an expanding number of languages attests to the international relevance of the topic presented here.

6. Conclusions

In summary, audible replies by VAs to oral queries related to gynecologic oncology have considerable room for improved accuracy. The findings of this study support those of Bickmore et al., showing that patients should not rely solely on VAs to answer medical questions [20]. Overall, we recommend caution when using VAs to obtain information in gynecologic oncology.

In recent news, Microsoft has announced that it will launch an AI-powered Bing search engine on the Edge browser to “deliver better search, more complete answers, a new chat experience and the ability to generate content” [33]. Google plans a rollout of its AI chatbot named BARD which is intended to enhance Google Search; however in a preview demonstration Google Bard provided erroneous information about discoveries made by the James Webb Space [34,35], while Microsoft’s AI-powered Bing has also generated false information [36,37]. At present neither Microsoft nor Google has associated these AI-technologies with receiving and responding to voice instructions. With new technology like these, further analysis of such VAs may be required when the technology has been improved. The new AI-powered Bing search Engine and Google BARD will need to be examined for their validity in answering healthcare questions, specifically as related to gynecologic oncology.

In addition to Microsoft’s AI-powered Bing search engine, OpenAI has released an AI system called Generative Pretrained Transformer 4 (GPT-4) that has a chat interface [38]. The chatbot gives a natural-language “response”, normally within 1 second that is relevant to the prompt [38]. This opens up a new conversation for how the chatbot could pertain to the medical field. Just like with the VAs, there is concern for how accurate the information that the chatbot gives is. According to an article in the New England Journal of Medicine, a false response given by GPT-4 is referred to as a “hallucination” [38]. These errors can be dangerous in medical scenarios because they can often be subtle and stated in a manner in which the chatbot is very convincing. These concerns are the same with the VAs, and thus it is extremely important to verify the information from both the VA and GPT-4.