Ontology-Driven Knowledge Sharing in Alzheimer's Disease Research

1. Introduction

The rise of human life expectancy during the past centuries is a remarkable accomplishment of science and medicine. However, living longer means that we are going to spend more time in our old age, which is going to make us more prone to neurodegenerative diseases such as dementia. In 2019 there were about 57 million cases of dementia globally, and it is estimated that by 2050 this number would increase and reach 152.8 million cases [1]. A characteristic hallmark of dementia is a progressive decline in cognitive functions, which gradually causes a loss of ability to perform basic activities of daily living [2]. The occurrence of the disease significantly impacts the life of the patient’s family and is also related to huge financial costs for society [3]. Alzheimer’s disease (AD) is known as the most common cause of dementia, and it has been implicated in about 80% of all dementia diagnoses [2]. The severity of AD is rooted in the complexity of the disease and the fact we are still unaware of what is causing it. As a result, AD remains a disease that we can neither prevent nor treat.

It is becoming evident that the multifaceted nature of AD calls for interdisciplinary collaborations capable of creating novel solutions, that go beyond the capabilities of single disciplines. There is a severe need for interdisciplinary collaborations to create a comprehensive definition of AD, understand the root causes of the disease and find interventions that can halt its progression or even prevent the disease altogether [4]. Furthermore, interdisciplinary AD research should expand beyond narrow collaborations within biology and medical sciences. For instance, addressing costs and equity in AD treatment would require not only clinicians and scientists but also economists and sociologists working alongside pharmaceutical companies and other stakeholders [4]. Assessing whether a new treatment is going to be accepted by patients, addressing caregiver burden and offering adequate support are all topics requiring deep comprehension of human beliefs, perceptions and behavior, a task suitable for psychology and cognitive science [4].

The present work is focused on the collaborative efforts of medical and science, technology, engineering, and mathematics (STEM) field experts towards researching Alzheimer’s Disease. According to a study from 2020, 73% of 250 machine-learning articles in biology and medicine were a result of interdisciplinary collaborations comprising authors from at least two of the three disciplines: computational sciences, biology, and medicine [5]. Furthermore, emerging evidence suggests that the rise of new technologies such as machine learning and artificial intelligence might be the key to unlocking new horizons in preclinical AD research, early diagnostics of AD and AD risk estimation [6,7,8,9,10].

Even though such collaborative endeavors give high promises for advancements in the field, fulfilling these promises depends on one’s ability to collaborate and communicate effectively. Facilitating effective knowledge sharing between individuals with different backgrounds is not a trivial task. Some of the most prominent issues interdisciplinary teams face are related to different communication and presentation styles, differences in vocabulary and general differences in concept comprehension [11]. Naturally, the further apart from each other the cooperating fields of science are, the more prominent these issues become.

These difficulties can be addressed by using ontology-based methods for facilitating knowledge sharing within interdisciplinary teams. Ontologies consist of concepts, their definitions and the relationships between these concepts. Importantly, ontologies serve as a description of a target domain while they also provide a standardized vocabulary for describing entities and semantic relationships, which can minimize the interpersonal variation of expression and ambiguity of representation [12]. Application ontologies bear all these properties, but they are built to address one specific task. Thus, they provide a minimal terminological structure to fit a particular task [13]. In this sense, they appear lightweight compared to reference ontologies, more compact and easier for navigation [13].

BioPortal is the main online biomedical repository for ontologies, developed by the National Center for Biomedical Ontology (NCBO) [14]. It contains a generalized disease ontology—namely, human disease ontology that classifies more than 10 800 rare, common and complex human diseases and combines epidemiological concepts, symptoms, genetic information, and findings into a disease scenario [14]. However, the usage of this ontology for specific diseases such as AD is limited due to its broad coverage and lack of depth. Among the domain specific disease ontologies in BioPortal are Cardiovascular Disease Ontology [15], Chronic Kidney Disease Ontology [16], Infectious Desease Ontology [17], Neurodegenerative Disease Data Ontology [18], Parkinson’s Disease Ontology [19], COVID-19 Infectious Disease Ontology [17], Oral Health and Disease Ontology [20] and others. Among them, there are four ontologies related to Alzheimer’s diseases. The Alzheimer’s Disease Ontology (ADO) covers preclinical, clinical, etiological, and molecular/cellular views of Alzheimer’s disease domain [21]. Its main purpose is act as a robust navigation tree for terminology integration and text-mining applications. The Bilingual Ontology of Alzheimer's Disease and Related Diseases (ONTOAD) is a bilingual (English-French) domain ontology for modeling knowledge about AD and related syndromes [22]. However, the degree of formalization of the ontology and its interoperability with other biomedical ontologies need to be improved. The alignment of the ontology with an upper level ontology like BFO is identified as a perspective for enhancement. The Disease Map Ontology (DMO) is an ontological upper model driving the conversion of a disease maps into a formal ontology. It is applied to convert AlzPathway into the Alzheimer Disease Map Ontology (ADMO) [23]. The latter can be connected with ADO into ADMO-plus as a resource able to store and interconnect biomedical data. Alzheimer Disease Relevance Ontology by Process (AD-DROP) is developed to classify disease-relevant processes according to their specificity, pathogenic intensity properties and frequency related to AD [24]. To the best of our knowledge there is no AD ontology created to facilitate the understanding of the AD knowledge domain in interdisciplinary teams and thus to speed up the process of delivering analytical models.

The present work starts from the hypothesis that application ontologies can facilitate knowledge understanding and transfer in interdisciplinary teams working on Alzheimer’s disease. In particular, we sought to demonstrate that such ontologies can serve as an effective source of expert knowledge to researchers and technology specialists who lack prior knowledge of Alzheimer’s disease. Furthermore, expert knowledge in medicine is a valuable commodity that is usually hard to find and access. It is costly to have expert clinicians always providing domain expertise and support. Additionally, such experts are often preoccupied and have little to no time available. Application ontologies, on the other hand, can be utilized as an on-site source of the expert knowledge that is available at all times and is related with minimal expenses.

The above considerations motivated us to propose the Alzheimer’s disease Ontology for Diagnosis and Preclinical Classification (AD-DPC) as a remedy for the difficulties of knowledge understanding and sharing in the context of AD-related interdisciplinary projects. AD-DPC aims to support nonmedical experts in three ways: (1) faster comprehension of the domain of AD, (2) provision of real-time expert knowledge to non-medical experts and (3) facilitating optimal communication with medical experts by providing a unified vocabulary. To validate the hypothesis that AD-DPC facilitate knowledge understanding and transfer in interdisciplinary teams, we conducted a user evaluation trial to test three dimensions vital to the successful interaction with AD-DPC – usability, applicability and correctness.

The rest of the paper is structured as follows. First, the methods used to build and evaluate AD-DPC are discussed. Next, the scope, structure and contents of AD-DPC are presented together with the results from the user evaluation. Then, the results and their implications are discussed, and the existing limitations are summarized. Finally, conclusions and directions for future work are outlined.

2. Materials and Methods

AD-DPC was built using the skeletal methodology for ontology building proposed by Unschold and Gruninger in 1996 [25]. According to their methodology, the process of building ontologies consists of four key activities: identifying ontology’s purpose and scope, building the ontology, ontology evaluation and ontology documentation [25]. Figure 1 depicts the skeletal methodology, representing the processes of ontology development as well as their temporal distribution. There are four key activities in ontology development – defining purpose and scope, building the ontology, evaluating the ontology and documenting the process. The ontology building can be partitioned into three tasks – ontology capture, ontology coding and integration of existing ontologies. Dashed lines represent the backward movement between the activities. Note that the process of documenting the ontology spans the whole cycle. The scheme is based on the original description of the methodology presented by Uschold and Gruninger.

AD-DPC is an extension of a top-level ontology, namely Basic Formal Ontology (BFO) [26]. AD-DPC was built in the Ontology Web Language (OWL) format using the Protégé OWL editor (version 5.5.0). Protégé is a widely used tool for ontology development and visualization [27].

2.1. Purpose and Scope

The scope of AD-DPC was defined based on several competency questions (CQ) that were later used also for the validation of the ontology. AD-DPC aims to facilitate knowledge exchange in interdisciplinary teams that perform research on Alzheimer’s disease. Therefore, we divided the stakeholders into two groups – (1) those with a medical background and (2) those without a medical background. While the first group is vital to the correct definition of the scope and the validity of the included terms, we expect that the second group would generally have higher benefits from using AD-DPC. Therefore, our competency questions were defined based on the input from both groups. We conducted 3 survey meetings with the stakeholder groups in order to establish the scope of the ontology. First, an individual meeting with each group was organized in order to outline initial questions and requirements coming from the group. Then a final meeting involving both groups was conducted in order to provoke a broader discussion between the stakeholders. As a result of this discussion and reviewing the questions and notes made by representatives of both groups, we constructed 5 competency questions that summarize all outlined points.

• What are the stages within the Alzheimer’s disease spectrum and their differences?
• How is Alzheimer’s disease diagnosed?
• Why is the preclinical stage important and how can we do preclinical research?
• What are the assessments that clinicians usually use whenever Alzheimer’s disease is suspected and which are the most informative ones?
• What are the symptoms and pathological hallmarks of AD?

2.2. Building AD-DPC

The initial collection of terms and concepts was gathered from various online resources such as scholarly articles, review articles and websites. The selection was carefully reviewed and edited by a neurologist. The resulting set of terms was arranged in a hierarchy with outlined relations between the concepts. A domain expert also validated this primary hierarchy. Terms and term definitions were either inherited from other ontologies, imported from other ontologies, adapted from other ontologies or defined with the assistance of a domain expert. Regardless, every term contains information about the source used to define it.

AD-DPC is defined as an extension of BFO – a top-level ontology designed to support the integration of scientific data [26]. Using shared top-level ontologies ensures interoperability by simplifying ontology integration and term reusing [26]. By building on top of BFO, we adopt the defined BFO principles and guidelines for defining terms and building ontological hierarchies such as using Aristotelian style definitions and adhering to the single inheritance principle [26].

In line with all best practices for ontology building [28], we set out to incorporate the available knowledge from other ontologies and keep the door open for future extensions. Therefore, apart from reusing BFO, we incorporated terms from several other ontologies – the ontology for precision medicine and investigation (OPMI) [29] and the information artefact ontology (IAO) [30].

Although term reuse was of high priority for us, there were instances in which we would find an appropriate term defined in another ontology, but the definition would significantly vary from our intended use case, and therefore we couldn’t fully reuse the term. In such cases, we would fallback to creating a term that is ‘adapted from’ the origin ontology.

AD-DPC was tested with regard to formal consistency with the HermiT 1.4.3.456 reasoner [31]. The current version of AD-DPC is published and available with open access on Zenodo [32].

2.3. Ontology evaluation

AD-DPC is meant to support non-medical STEM experts doing research in the domain of Alzheimer's disease. Therefore, it is important to validate that AD-DPC is fit to serve this purpose. However, one of the difficulties in ontology validation is the selection of the properties and characteristics that will be evaluated. Which ontology features to evaluate depends heavily on the type of ontology and the focus of the evaluation. In the case of application ontologies, it is enough to evaluate the features important to the task that the ontology was designed to fulfil [33]. Thus, by evaluating AD-DPC, we sought to demonstrate that AD-DPC can indeed support nonexperts in the domain of Alzheimer’s disease in handling medical data and making inference. Our secondary goal was to show that the interaction with AD-DPC induces knowledge acquisition in a particular domain.

For this purpose, we designed a user evaluation setup that focuses on usability, applicability and correctness. These three properties traditionally vary in their definitions across the literature. To address this variation, we adopted the definitions provided in [33]. Namely, we adopt the following definitions - usability to be defined as ‘a set of attributes that describe the effort needed by a human to make use of the ontology’, correctness as ‘the degree to which the information asserted in the ontology conforms to the information that need be represented in the ontology’ and finally applicability as ‘the quality of the ontology regarding its appropriateness for a particular application, task or purpose’ [33].

Since we also wanted to evaluate whether AD-DPC facilitates knowledge acquisition, we consider the perceived level of knowledge as the fourth component in our evaluation method.

2.4. Procedure and participants

We assessed the usability, applicability and correctness of the AD-DPC ontology by tasking experts in different STEM fields to answer a series of questions about Alzheimer’s disease with the help of AD-DPC. While the applicability and the correctness of the ontology were assessed directly from the performance on this task, we used the System Usability Scale (SUS) [34] to estimate the usability of the ontology.

We evaluated the baseline knowledge about Alzheimer’s disease and ontologies of each participant by asking them to fill in a baseline questionnaire. This questionnaire also contained questions about the participants’ field of work and their previous experience with biomedical projects and data. All participants were given a 15-minute walk-through Protégé and its functionality. Then participants were given alone time to get familiar and comfortable with the Protégé environment. There was no time limitation, instead, participants were instructed to signal the experimenter whenever they felt comfortable enough with Protégé. Once the self-learning phase was over, participants were given the two main questionnaires and tasked to answer questions about Alzheimer’s disease with the help of ontology. Finally, participants were asked to complete a usability questionnaire and to provide a self-assessment of the acquired knowledge.

The total number of participants was 10, all members of the scientific staff of Sofia University. The only inclusion criterion was that all participants should actively exercise a profession within а STEM field. The only exception to this were medical graduates since they would already be familiar with all of the concepts related to AD. All questionnaires were paper-based. However, to complete the main task participants had to interact in real-time with Protégé and the preloaded version of the AD-DPC ontology.

2.5. Questionnaires and scales

We assess the correctness of AD-DPC by empirically testing whether the implemented ontology indeed represents the scope that was initially set with the competency questions. We took the initial CQs and stripped them down to 6 narrow and specific questions (Table 1 left). Participants were instructed to give written answers to the 6 questions by interacting with AD-DPC in real-time. Each answer was evaluated and given a numeric grade: -1 for an incorrect answer; 0 for no answer; 1 for a correct answer and 2 for a partially correct answer.

Applicability was evaluated with the means of two scenario-based questions (SQs) that were given after completing the CQs-based questionnaire (Table 1 right). Unlike the questions used in the correctness task, scenario-based questions require a level of reasoning with the knowledge that one can acquire from AD-DPC. This task is designed to mimic a real-life situation where a person is supposed to navigate a more loosely defined task outside of their field of expertise. Since such situations are often observed in interdisciplinary dynamics, we consider their emulation as the ultimate test of applicability. The provided answers were graded on a scale from -1 to 2 as described above.

For both correctness and applicability, success rates were defined based on the quality of answers provided by participants. Success was defined as providing a fully correct answer. Partially correct answers were considered for partial success and incorrect answers were treated as failures. Finally, a lacking answer was considered for an omission. If participants were unable to answer a question they were asked to explicitly indicate this by writing ‘IDK’ or ‘I don’t know’.

Finally, to assess the usability of the AD-DPC we used a modified version of the System Usability Scale (SUS) [34]. SUS was originally proposed in 1986 as a general tool for assessing the usability of a wide range of systems and products. Therefore, to be suitable for ontology evaluation the scale needed to be modified to address the specifics of the task at hand. Previous work has already demonstrated that a modified version of SUS can be used to evaluate the usability of ontologies [33,35,36]. In the present work, we adhered to the SUS modification for ontology evaluation described in [33]. This modification contains 10 questions each with possible answers on a scale of 1 to 5 (1=strongly disagree, 2=disagree, 3=no preference, 4=agree, 5=strongly agree). The scale was given to the participants after they had answered the scenario-based questions. Therefore, the results from SUS reflect how participants have evaluated their experience of using AD-DPC to answer real-world questions related to AD.

3. Results

This section presents the scope, structure and contents of AD-DPC as well as the results from its evaluation following the procedure described in the previous section.

3.1. AD-DPC definition

AD-DPC encodes domain knowledge in two ways - through the hierarchical structure of concepts and relations between them, and through definitions and elucidations. The scope of AD-DPC covers 6 major conceptual groups relevant to the domain: (1) Alzheimer's disease pathology, (2) Alzheimer's disease spectrum, (3) Diagnostic process, (4) Symptoms, (5) Assessments, and (6) Relevant clinical findings. All concepts were annotated with definitions or elucidations and in some cases enriched with synonyms and additional resources. AD-DPC features a total of 126 concepts including inherited concepts from BFO. Figure 2 shows how AD-DPC is answering the predefined CQs.

Part A depicts the temporal sequence of the diagnostic process for MCI and AD. Several assessments are conducted to extract clinical findings that can support/reject a hypothesis about the patient’s condition. Together with collected anamnestic data these findings form the clinical picture of the patient’s condition. The clinical picture is formed with respect to a diagnostic guideline or diagnostic criteria, and subsequently, it is evaluated according to the same. Finally, this process of evaluation produces a diagnosis. Part B shows the process of detecting pathological features of AD as represented in AD-DPC. Each of the three hallmarks can be measured in at least two ways listed under ‘Assessments’. The output from these assessments undergoes interpretation which results in the production of clinical findings. These findings either confirm or reject a hypothesized diagnosis. However, AD-DPC lists clinical findings that confirm abnormalities related to AD (listed under ‘Findings’) since these are relevant to our work. Part C shows concepts that do not form rich taxonomies. These concepts convey information through their definitions and attached resources.

The following sections offer a detailed description of the 6 conceptual groups included in AD-DPC.

3.3.1. Alzheimer’s disease pathology

As a neurodegenerative disease, AD is a condition which slowly unfolds in time and presents itself with multiple deficits related to memory, reasoning and cognition. However, it is now well-established that AD runs a prolonged preclinical course characterized by a lack of symptoms [37]. Despite the lack of cognitive deficits, there are pathological changes in the brain that are already observable during the preclinical stage [37]. These early pathophysiological changes are considered to be distinctive hallmarks of AD. According to the current understanding AD pathological hallmarks include 1) accumulation of amyloid plaques; 2) aggregation of tau proteins into neurofibrillary tangles (NFTs); 3) neurodegeneration and brain atrophy [37]. In particular atrophy of the medial temporal lobe is highly associated with Alzheimer’s disease [38]. AD-DPC goes on to describe the pathological features of AD, commonly used methods of measurement and relevant clinical findings suggesting that a particular pathology is present (Figure 2B). As postulated by the CQs, AD-DPC underlines the distinction between symptoms of AD and pathological features of AD.

3.3.2. Alzheimer's disease spectrum

In line with the latest understanding of Alzheimer’s disease, AD-DPC describes the disease as a spectrum (Figure 2C). After a long asymptomatic stage, the clinical manifestation of AD begins with the gradual onset of cognitive impairment. The first clinical stage of the disease is mild cognitive impairment (MCI) where patients are exhibiting a mild form of cognitive deficits that are not causing functional impairment nor interfering with the daily life of the patient [39]. The gradual worsening of the symptoms leads to the more severe and final stage – Alzheimer’s disease. To emphasize the importance of preclinical research, AD-DPC provides concepts related to the amyloid/tau/neurodegeneration framework (ATN framework) used to classify the biological state of AD independent of observable symptoms [40]. AD-DPC also emphasizes that the ATN framework is used in research settings and is making use of biomarkers that are observable before the onset of symptoms.

3.3.3. Diagnostic process

Since diagnosis was a prominent topic among the questions collected before the construction of the final CQs, AD-DPC was designed to include classes and relations describing the diagnostic process for MCI and AD (Figure 2A). AD-DPC also describes the current diagnostic guidelines for MCI and AD and how they are applied in practice. While the inclusion of these guidelines might be considered as unnecessary detail, we believe that they are likely to enable non-medical experts to better differentiate between the stages of the disease. The descriptions of the guidelines were derived from the updated diagnostic guidelines for Alzheimer’s disease [41].

3.3.4. Symptoms

The early symptoms of AD include memory impairments as well as deficits in additional cognitive domains [42]. The manifestation of the symptoms is gradual, and their severity increases over time. Eventually, the severity of the symptoms disrupts patients’ ability to execute normal living activities. AD-DPC describes some of the common symptoms of AD by providing general concepts such as 'impaired learning', 'impaired memory function', 'impaired language capacity', 'impaired visuospatial function', 'impaired executive function'. While the concepts are general, their descriptions contain particular examples of how such impairments might be manifested. For example, 'impaired visuospatial function' can be manifested as difficulty in recognizing places or getting lost in familiar places. Note, that the severity of each of the symptoms can vary and this is not explicitly stated in AD-DPC in the form of having separate concepts for mild and severe symptoms. Rather it is communicated through the examples of particular manifestations. AD-DPC also provides information on how the severity of the symptoms is tested - namely with neuropsychological scales.

3.3.5. Assessments

While there is a variety of assessments that can be used during the evaluation of patients with a potential diagnosis of AD, we included the assessments used traditionally in clinical practice and considered as most informative. The selected assessments were segmented into the following classes – imaging assessments, neuropsychological assessments, CSF biomarkers, genetic biomarkers, physical examination and neurological examination (Figure 2A).

Brain imaging is a technique that produces brain images for diagnostic or treatment purposes. While there are several different imaging techniques, the most commonly used techniques in the context of AD are magnetic resonance imaging (MRI) and positron emission tomography (PET). These are the imaging techniques that AD-DPC is describing as well.

MRI imaging is used to identify brain atrophy and neurodegeneration in regions of interest. One way to evaluate atrophy from an MRI image is by using visual rating scales where a physician is manually grading the observed atrophy (Figure 2B). Visual scales are widely used in clinical practice. A visual scale commonly used in evaluating AD is Scheltens’ scale [43]. Scheltens’ scale evaluates bilateral medial temporal atrophy (MTA). A similar evaluation of atrophy can be achieved with volumetric MRI (vMRI) where one is directly working with the volumes of brain regions/structures of interest (Figure 2B). Unlike visual scales this process is automated. However, it is still predominantly used in research settings.

While MRI shows the structural state of the brain, PET is commonly used to image how the brain functions. In the context of AD, PET imaging is used to image the accumulation of amyloid-beta (Aß42) and tau proteins as well as the metabolism of the brain (Figure 2B). While PET imaging is the most accurate method for measuring amyloid and tau accumulations in the brain, it is still an expensive procedure with limited availability. A highly accurate alternative is measuring amyloid beta (Aß42) and tau from cerebrospinal fluid (CSF). Measuring CSF biomarkers is a far more affordable procedure widely used in clinical practice. AD-DPC reflects this by featuring concepts defining CSF biomarkers such as Aß42, phosphorylated tau (p-tau) and total tau (t-tau).

Neuropsychological evaluation is widely used in clinical practice to assess the cognitive state and abilities of a patient. While the diagnostic criteria for AD depends on the presence of cognitive impairment, typically the patient undergoes an additional assessment of functional capabilities, depression and overall dementia severity [44]. AD-DPC reflects the variety of assessments that is usually applied by listing 14 scales among which Mini-Mental State Examination (MMSE) [45], Montreal Cognitive Assessment (MoCA) [46], Alzheimer’s Disease Assessment Scale (ADAS) [47], Clinical Dementia Rating (CDR) [48], The Functional Activities Questionnaire (FAQ) [49]. The list includes commonly used cognitive assessments, scales assessing the functional capabilities regarding daily activities as well as scales assessing the neuropsychiatric state of the patient and dementia severity assessments. While the list of scales offered by AD-DPC is not exhaustive, it still provides information about the most commonly used scales and gives an overview of how cognitive functions and psychiatric state are measured in general.

To reflect the genetic component in the occurrence of Alzheimer’s disease, AD-DPC includes concepts of genetic biomarkers. However, currently, the ontology features only the APOE4 gene as a genetic risk factor. Future work will expand the concept to include other genes that have been implicated in the development of AD.

Finally, AD-DPC also accounts for first-line assessments such as physiological and neurological assessments. While these two concepts do not have child concepts, they are provided with definitions and additional resources which are informative of the procedure. We do not consider this a flow in AD-DPC since these evaluations detect unspecific to Alzheimer’s disease symptoms and are the least informative in the context of AD.

3.3.6. Relevant clinical findings

As seen in Figure 2, AD-DPC includes descriptions of the assessments as well as descriptions of the relevant clinical findings produced by the assessments. Clinical findings include neuropsychological, genetic, imaging and CSF findings. This way AD-DPC enables the reader to make sense of raw medical data and thus be able to make a basic inference as to whether there are abnormalities in the results or not. While the wording of some clinical findings might seem unspecific – for example ‘increased CSF p-tau’, note that there are a number of assays and kits to make such measurements. As a result, the cut-offs indicating abnormality are different depending on the assay that was used. Therefore, AD-DPC provides only a general description of what a relevant finding would be.

AD-DPC evaluation 

A total of 10 participants took part in the evaluation procedure. About 80% of them reported having an occupational background in science with half of them having an interdisciplinary background in science and one or two other STEM fields. Overall, participants reported low familiarity with ontologies (1.6 ± 0.69 points) and no experience with Protégé (1.0 ± 0.0 points). While the average reported familiarity with Alzheimer’s disease was slightly higher (2.6 ± 0.96 points) only two participants reported having previous experience with medical data and only one had been involved in biomedical research before the experiment (Table 2).

The correctness of AD-DPC was evaluated by measuring the ability of the participants to answer derivations of the competency questions with the assistance of ontology. The results are provided in Table 3. The overall performance was high with success rates per question ranging from 70% to 100%. The only exception was Q3 where only half of the participants could answer correctly, another 20% gave partially correct answers, and the rest answered incorrectly. Interestingly, 30% of the participants did not make an attempt at answering Q6 while the rest gave fully correct answers.

Applicability on the other hand was measured through the ability of the participants to answer scenario-based questions that require reasoning with the knowledge contained within the competency questions. The results from the applicability task are characterized by low success rates (30% and 50%) with partial success rates of 20% and 30% respectively (Table 3). Scenario 1 is marked by worse performance with half of the participants failing to provide even a partially correct answer. Curiously, it seems that participants were utilizing different strategies in answering the two scenario-based questions. Even though the first question has a higher failure rate, all participants made an attempt to provide an answer. The picture seems slightly different for the second question – despite the higher success rate, only 80% of the participants made an attempt at providing an answer and the rest committed omissions (Table 3). Out of these 80% percent, all answers were either correct or partially correct.

A modified version of SUS was used to evaluate the usability of AD-DPC in solving the main task. SUS produces a number between 0 and 100 where the higher the score is the better. However, understanding how this score translates to deciding the permissible levels of usability is not a straightforward task. Therefore, additional grading ranges were proposed to accompany SUS scores to provide a higher level of clarity as to which applications have an acceptable level of usability [38]. Table 4 displays the total SUS scores along with adjective and acceptability grades assigned according to the ranges described in [38].

The SUS scores for half of the participants fall within the acceptable range and 70% of the SUS scores have an adjective grade ‘OK’ or ‘Good’. However, one should note that 70.0 is the lower boundary for acceptability and therefore the majority of our acceptable SUS scores seem to be borderline.

4. Discussion

The presented work demonstrated that the AD-DPC ontology has the potential to at least partially mitigate the difficulties of interdisciplinary research in the field of Alzheimer’s disease. We conducted a user evaluation to assess the correctness, applicability and usability of AD-DPC by having 10 participants answer AD-related questions with the help of ontology. Our findings suggest that ontologies can potentially provide effective support to team members that lack medical training and expertise in the field.

The overall correctness of AD-DPC was high with success rates between 70% and 100% on the respective task. Since the questions in the task were derived from the competency questions, these results confirm that AD-DPC successfully covers the initially set scope. Furthermore, high success rates suggest that the information encoded in AD-DPC is accessible and discoverable whenever needed. Overall, the results suggest that individuals without a medical background benefit from using AD-DPC as a real-time consultant in answering questions from the AD domain. While on average AD-DPC scored high in correctness, Q3 was characterized by notably lower success rates and higher failure rates in comparison to all other questions from the questionnaire (Table 3). This might be, at least partially, due to the question’s wording. In particular, the usage of the word ‘hallmarks’ might have posed difficulties for the participants for two reasons: (1) the word ‘hallmark’ is not used in the ontology (2) our sample consisted of non-native English speakers. This theory is seconded by the fact that several participants explicitly asked for the meaning of the word ‘hallmark’.

Interestingly, Q6 was the only question to which 30% of participants did not provide an answer and gave no indication that they did not know the answer (performed omissions). We believe this result is a byproduct of having a paper-based questionnaire where Q6 was the only question printed on the back of the sheet. Despite the fact that all participants were instructed to check both sides of the sheets, it might have been the case that several participants failed to notice the last question.

In contrast to the correctness results, applicability scores derived from the context-based questions were not as positive. The estimated success rates were 30% for Q1 and 50% for Q2. Such results indicate that participants were unable to accurately answer questions that require a level of reasoning even with the assistance of AD-DPC. While the correctness task featured well-directed questions which can be answered simply by looking for their keywords in AD-DPC, the applicability task contained questions that required not only understanding the meaning of the keywords but also reasoning with the extracted knowledge. While we acknowledge that the low success rates might be due to the insufficient level of domain understanding on the side of the participants, the results nevertheless demonstrate that ontologies cannot provide the level of consulting that experts can. Therefore, AD-DPC can be used to partially consult non-specialists, but task definitions and execution procedures should still be provided by experts.

On the same note, the results suggest that participants were overall more confident in their capability to answer Q1 with 100% of the participants providing an answer. In contrast, Q2 was answered only by 80% of the participants. Despite exhibiting higher confidence in answering Q1, 50% of the participants actually gave wrong answers. This might signal that some concepts and relations in the ontology need better definitions, however further evaluations are required to confirm this hypothesis.

On the other hand, no wrong answers were given to Q2, and 80% of the participants answered either correctly or partially correctly. The lower confidence for Q2 is compensated by higher accuracy, suggesting that even though Q2 was perceived as more difficult, AD-DPC provided well-understood definitions and did not cause confusion among participants.

Next, we asked all participants to complete SUS and evaluate their experience with AD-DPC in answering AD-related questions. After processing the scores, we obtained a measure of usability relative to the particular task. The average usability score of AD-DPC was 63.0, and according to the classification presented in [50] it falls within the ‘OK’ band. However, 63.0 is well below the acceptance cut-off of 70.0, suggesting that while the score seems above average, the reported usability is still low compared to what is expected from a ready-for-use tool. Nevertheless, we still interpret this result as encouraging since the usability scores might have been influenced by the lack of experience with Protégé and ontologies in general. This assumption is in line with previous studies that have used SUS to evaluate ontology usability and have reported similar scores [33]. Furthermore, several participants gave feedback referring to the fact that there were difficulties arising from the interaction with Protégé. Other comments mentioned the need for additional training with Protégé and elaboration on the ontological structure. As a result, we conclude that while the usability scores are encouraging, the lack of experience with Protégé and ontologies, in general, may hinder the quality of user experience. Therefore, one should consider ensuring sufficient training time before incorporating ontologies as consulting tools.

Finally, participants were asked to report their perceived level of acquired knowledge during their interaction with AD-DPC. All participants reported that they had acquired some knowledge of Alzheimer’s disease. The perceived increase in acquired knowledge was 2.55 points on average. The increase in perceived level of knowledge suggests that AD-DPC may find use not only as a consulting tool but also as an educational instrument.

Our study has several limitations that remain to be addressed in future work. First, we evaluated AD-DPC with a sample of 10 participants. Even though such a sample can provide some indication of the usefulness of AD-DPC, there is a need for a bigger sample of evaluators to confirm our findings. Furthermore, a bigger and more diverse sample might shed light on various interaction and knowledge extraction strategies relative to the participant’s experience.

Another limitation is the lack of evaluators who are experts or at least well-versed with ontologies. However, such experts are indeed hard to find, and this remains to be addressed in future studies. Including experts as evaluators would provide more realistic feedback on the structure and the quality of the ontology in terms of conceptualization and implementation. Also, such experts would not suffer from the ‘lack of experience’ bias observed in the usability evaluation (SUS scale).

Finally, some of the scales we used were based on self-evaluation and perceived measures rather than objective ones. While we acknowledge that this approach might generate a level of bias, we still believe that this type of measure was fit to serve the purpose. However, future work might look into the actual level of acquired knowledge rather than the perceived one.

5. Conclusions

The current work proposes the application ontology AD-DPC as a tool for mitigating the difficulties of interdisciplinary research on Alzheimer’s disease. AD-DPC features 126 terms spanning five major branches relevant to the domain – Alzheimer's disease pathology, Alzheimer's disease spectrum, diagnostic process, symptoms, assessments and relevant clinical findings. The evaluation of the ontology showed that it holds the potential to provide effective support to team members that lack medical training and expertise in the field. The results from the evaluation demonstrate that participants can answer AD-related questions with the help of AD-DPC. However, whether AD-DPC is saving time compared to searches in traditional resources, such as scientific journals, is a question that remains to be answered.

Future work will be focused on improving AD-DPC and improving the methods of evaluation. First, while AD-DPC provides a comprehensive description of the domain of AD, the ontology can be extended to cover a wider range of concepts. For instance, AD-DPC can be easily integrated with Alzheimer's disease ontology (ADO) [21] since both of them are based on BFO. Such integration will enrich AD-DPC with various concepts covering the etiological, molecular and cellular mechanisms involved in the pathogenesis of AD. Furthermore, while AD-DPC describes main processes by fully utilizing the ontological structure of concepts and relations (Figure 2A, B), some concepts have underdeveloped taxonomy but compensate with a highly informative definition or elucidations (Figure 2 C). This decision was made for the sake of simplicity and ease of interaction. However, future work will improve the ontological structure by expanding such singular concepts to full-flown hierarchies without making the content heavier for interpretation. A more broad expansion should also consider the inclusion of terms describing the legal aspects of collection, storage and exchange of medical data.

In terms of evaluation methods, future work will be focused on evaluation with bigger and more diverse samples. Furthermore, the evaluation process might benefit from introducing a domain expert in the grading of the questionnaires. This would give an even more accurate idea of correctness and applicability since we expect experts to be stricter and detail-oriented in their grading. To complete the evaluation cycle, evaluators with ontology expertise should be involved. On one hand, ontology specialists will not be prone to interferences caused by the lack of experience with ontologies and Protégé. On the other hand, their evaluation will provide a clear picture of the structural and conceptual quality of AD-DPC.

Last, we believe that AD-DPC can also be used to support the interoperability and integration of medical data. It will be further used as a base for developing knowledge graph on top of the existing graph databases, such as Neo4j. However, this remains to be demonstrated in future studies.