Design and Usability Assessment of a Cognitive Screening Digital Tool on Tablet: AlzVR Project

1. Introduction

Alzheimer’s disease (AD) is the first cause of neurodegenerative decline, affecting millions of people worldwide [1] with a considerable cost for countries. In AD, patients progressively lose their cognitive abilities, and behavioral troubles can occur. Without any current and efficient treatment, loss of memory and autonomy become an essential burden for caregivers and families. AD management is a global and public challenge for health systems that face a constantly increasing prevalence in aging populations.

Precocious screening of cognitive decline leads to better and early support for patients and their families. Unfortunately, general practitioners (GPs) do not always have enough time to perform initial cognitive explorations. They address their patients at specialized centers whose appointments can be long, delaying diagnostic but also symptomatic and social measures. These labeled memory consultations are primarily available in hospitals, and the diagnostic process still relies on paper tests that involve an exterior examiner.

Numerous tests exist to assess cognition (global or precise evaluation), but the Mini–Mental Status Examination (MMSE) [2] and the Montréal Cognitive Assessment (MoCA) [3] are widely used in primary screening, and most professionals know them. Both tests share several questions and explore approximately the same cognitive functions, even though MoCA evaluates frontal deficits more precisely. They both browse several cognitive functions quickly and can be repeated through the medical following of patients. More recently, the MoCA seems to have a higher sensitivity (Se) than the MMSE in differentiating healthy subjects from demented patients, whereas MMSE still performs a higher specificity (Sp) [4]. Nevertheless, there are good correlations between the two tests [5,6].

Besides these classical evaluations, several authors have developed new screening tools on numerical tablets showing good correlations with usual tests but without exceeding the experimentation stage [7]. Many applications are also available on commercial platforms such as Apple iTunes or Google Play Store [8], allowing an autonomous screening for patients. With the better accessibility of new technologies, most patients use a tablet or a smartphone, and the usability of numerical tablets has been globally demonstrated among large populations [9]. Using these novel opportunities, patients or GPs could benefit from an earlier evaluation before getting a specialized consultation.

Physicians need these innovative tools to evaluate cognitive functions to facilitate primary screening. Digital tablet assessment should be short, reliable, and understandable for patients with cognitive tasks reproducing classical questions from usual paper tests.

The AlzVR project is a digital multimodal program for cognitive screening, and we have previously developed an immersive assessment on Oculus Quest® [10,11]. We now aim to explore a new modality by creating a touchscreen application and evaluating its usability among a healthy population.

2. Materials and Methods

We constructed our program using Unity® (v.2021.3.11) for Android® (tablet). Three scenes compose AlzVR: the welcome menu, playing scene, and F-SUS questionnaire.

2.1. Welcome Menu

When launching the application, there are three possibilities (Figure 1):

• Supervised experience (“Expérience encadrée” ): medical questionnaire and cognitive assessment;
• Quick experience (“Expérience rapide” ): cognitive assessment only;
• Results (“Résultats”): results visualization.

The game consists of 3 main scenes:

• The “Menu” scene includes the main menu, medical questionnaires, and results consultation;
• The “InGame” scene contains the tutorial and all the user’s tasks;
• The “ Survey “ scene collects user feedback, which the administrator can only consult.

2.2. Medical Questionnaire

The supervised experience includes a primary medical questionnaire (Figure 2) to collect socio-demographic items (name, age, type of residence) and medical background (diagnostic, previous cognitive tests, treatments, and sensory loss).

2.3. Anonymization

After the last validation of the medical questionnaire, the program generates an automatic anonymized number composed of date and hour until seconds without integrating the name’s initials. A typical anonymous number looks like YYYYMMDDHHMMSS. This process safeguards the confidentiality of patient information (personal information) and allows a further blinded analysis. In “quick experience”, an “A” precedes all anonymized numbers, such as A-YYYYMMDDHHMMSS. In “supervised experience”, the letter of an eventual medical center can be automatically added before the number.

2.4. Playing Scene

2.4.1. Experiences Architecture

The main module of the “InGame” scene, the GameManager, references the list of nine tasks to be performed. Although each task has a different objective, each has textual and audio instruction and then proposes none, one or several answers in the form of images or text. So, the “JExperience” parent class groups all the attributes and methods common to all the tasks. However, the specific features of each task have necessitated the creation of new classes (“JExpMonoChoice”, “JExpChoiceTown”, “JExpImages”) inherited from “JExperience” (Figure 3).

2.4.2. General Aspect

The visual aspect should be simple without any cognitive surcharge. All scenes appear in a uniform blue background.

In all experiences, the user must select answers by touching one or several buttons. These buttons are big, allowing for an easy touch. A maximum of 8 buttons are on the screen, ensuring good visibility.

2.4.3. Answer Modality

Since the user selects all the answers, a confirmation screen appears with a button “Yes” (“Oui”) and “No” (“Non”). This step avoids inattentive answers and validates the choice (Figure 4). A “Yes” leads to the next question, and “No” allows a new chance to answer.

Each exercise lasts 30 seconds maximum. The next question automatically occurs if the user does not answer within the time (TimeOut). The choice of a “No” in the confirmation step reinitializes the time, but only three attempts are allowed.

In every case (success or failure), a message “Well done!” (“Bravo!”) congratulates the user (Figure 5). This message provides a cheerful ambiance and can reduce further false results of stress or fear.

2.4.4. Training Task

Before beginning the cognitive questionnaire, two tasks occur (touching shapes on the screen) to ensure a good understanding of the tablet’s functioning by the user (Figure 6). A failure in the training tasks leads to the assessment’s stopping, and the test cannot continue.

2.4.5. Cognitive Questionnaire

If the training tasks succeed, the cognitive assessment begins and comprises seven tasks from the MMSE and the MoCA. We wanted a varied assessment, so we selected questions from multiple cognitive fields presented in Table 1.

The first task is the «three words» test. Firstly, orally delivered by the program, the user must select the images on the screen corresponding to the three previously heard words (Figure 7). There is an immediate and delayed recall.

The clock recognition (Figure 8) task is inspired by the clock drawing test [12], which presents three different clocks and requires one to select the one representing a given hour (for example, “Select the clock indicating 12h15”).

Then come two spatial orientation tasks: selecting the French flag (Figure 9) and the town among several choices (Figure 10).

Temporal orientation tasks are relatively similar by selecting the current season (Figure 11) and year (Figure 12). Considering varying dates for season changes, we left a 48-hour margin for the answer.

The abstraction task consists of filling in a logic suit of fruits with four proposed answers (Figure 13).

2.5. Results Menu

The results’ menu allows a simple visualization of the patient’s score after the cognitive questionnaire (Figure 14). A password protects this section and only uses an anonymized number (ID patient). Three possibilities exist: “X” (failure), “V” (success), and “?” (Timeout).

2.6. F-SUS Questionnaire

After the cognitive tasks, we implemented the French translation of the System Usability Scale (Figure 15) [13], and the F-SUS questionnaire [14]. It evaluates global satisfaction through ten questions and five degrees of response from 1 (strongly disagree) to 5 (strongly agree). F-SUS results do not appear in the results menu and are directly stored.

2.7. Data Storage

All data are exported and stored in CSV format, allowing easy exploitation. We planned separate storage for personal information from other results (experiences and F-SUS). Thus, a blinded analysis is possible using only results files containing anonymized data (Figure 16).

2.8. Usability Assessment

2.8.1. Study Population

We carried out an experimental, qualitative study in IBISC Laboratory (University of Évry-Paris Saclay, Department of Sciences and Technologies) among healthy volunteers (staff and students) to assess the program usability using ISO 9241-11 norm [15] and the Nielsen method [16]. The tablet was a Samsung Galaxy Tab S7 FE® (screen of 315.0 mm, 2560x1600) running on Android 11 (user interface One UI 3.1).

The inclusion criteria were age > 18 years old, French language understanding, and the exclusion criteria were age < 18 years old, no understanding of the French language, and visual or hearing loss with no equipment.

2.8.2. Stages

Participants successively and anonymously achieved several stages:

• Pre-questionnaire: fill in an online questionnaire to collect socio-demographic data (age, profession, sex) and numerical habits (smartphone and tablets);
• Quick experience;
• F-SUS questionnaire;
• Post-questionnaire: online questionnaire to collect free comments about the program.

2.8.3. Data Collection and Exploitation

During the tests, we collected the following parameters: answer (success, fail), number of trials, and response time (ms).

We chose the total F-SUS score calculated on the author’s recommendation [13,14] as the primary endpoint to assess usability with a goal of 85.5 %, considered “excellent”.

All data were blinded, collected, and analyzed using the anonymized numbers of participants.

3. Results

3.1. Population

We included 24 healthy controls from Paris-Saclay University between 2022/09/27 and 2022/10/12. Their socio-demographics are presented in Table 2, and numerical habits are in Table A1.

3.2. Success Rate

Cognitive tasks were completed by 100% of participants. We observed a success rate for the questions of 97.4 % (187 correct answers out of 192). The two tests that presented failures were the clock task (2 failures) and the season (3 failures).

3.3. Time of Completion

The average test administration time (excluding training tasks) was 141.47 (± 18.77) seconds, and details of task completion times are presented in Table 3.

3.4. F-SUS Questionnaire

Ninety-six percent of participants completed the F-SUS questionnaire (one person left the application before completing it), and the results for each question are presented in Table A2.

The overall score on the F-SUS questionnaire was 89.24%, considered as “excellent.”

3.5. General Remarks

In the post-questionnaire, we collected general opinions about the computer program. Users overwhelmingly found the program to be easy to use. The negative remarks reported are the lack of fluidity of the oral instructions and the tests judged too simple. User reviews are shown in Figure 17.

4. Discussion

Several authors studied numerical tablet possibilities to assess cognitive decline and perform training tasks in healthy and cognitively impaired patients [17,18]. Facing an increasing prevalence of patients in the future decades [1] with a more and more precise diagnostic (biological, functional…) [19], there is a need to get simple, quick, and performing tools to help practitioners in cognitive decline screening.

Numerous paper tests assess cognition for a global screening or precisely evaluate a specific cognitive function [20].

During our conception, we chose to inspire from two primary used and recommended tests [21,22] to create our cognitive tasks: MMSE, MoCA, and the clock drawing test (CDT) (integrated into the MoCA). Among these tests, we selected several tasks transposable on a numerical tablet. In the context of a lack of practitioners and aiming to get an easily accessible screening, the program is conceived to be realized autonomously with no exterior examiner. Thus, the program orally delivers all questions, and we have limited written instructions.

Targeting an elderly population, we chose the Samsung Galaxy Tab S7 FE® because of its large screen and reasonable resolution.

Completion time should be short, and the mean time observed in our study (142 seconds) is a good result. However, the mean age of our healthy population (41.88 years) is relatively under the age of AD patients (> 60 years) [23], which can explain this result.

We are satisfied because we have reached our usability goal with a global score (89.24%), surpassing the initial objective of 85.5% and close to 90.9% (« best imaginable »).

The participants globally perceived the test as easy to use, corresponding to F-SUS scores (questions 3, 5, 7, and 8). It is a positive evaluation because participants did not know cognitive tests and thus discovered them for the first time. However, the questions were negatively perceived as « too simple » or « too slow ». Indeed, it can appear simple or slow for healthy volunteers, but tasks target people with potential cognitive decline and no daily numerical tablet use. Nevertheless, our population poorly used tablets, with less than 50% of people using them yearly (Table A1).

We noted errors in the clock recognition task, probably due to the needle shapes signaled in complimentary remarks (Figure 17). The original clock drawing test [12] must be drawn on paper. Thus, transposing it on a numerical tablet would have needed a stylet use and external review. To maintain autonomous functioning, we proposed a clock recognition test among three choices: good hour (10h30), « symmetrical clock » (05h50), and complex clock. Recent constatations showed that students had more and more difficulties reading traditional clocks [24], and our two failed users were 24 years old. These difficulties appear in the mean task time of realization (Table 3); indeed, it is the task with the most significant difference between the minimal and maximal time of realization. Although technical limitations may exist, digital clock drawing tests have shown reliable results in differentiating healthy subjects from demented patients [25]. CDT is also hugely used in daily practice and belongs to quick screening tools such as Codex [26] or Minicog [27]. Season errors may be explained by the recent season changes (summer/autumn) before the beginning of the study (Sept 27). We also found an extensive range in task time realization.

5. Conclusions

We have developed a new quick digital cognitive screening tool with good usability among a healthy population. The application could also be transposed onto smartphones to enhance its diffusion and utilization. This preliminary study belongs to the global study COGNUM-AlzVR, which aims to evaluate the efficiency and relevance of two numerical programs on tablets for cognitive assessment in AD patients. The Committee for the Protection of People of Ile de France approved the multicentric project in 2022, and the study began in April 2023 (NCT06032611).