Comparative Study of Two Commercial Brain Volumetry Software: Assessing Medial Temporal Lobe Atrophy

1. Introduction

As the worldwide population ages, neurodegenerative diseases, including Alzheimer's disease (AD), are rapidly becoming more prevalent; however, AD diagnosis remains challenging [1-3]. Since the revised guideline was published in 2011, medial temporal lobe atrophy (MTA) has been considered and used as an essential imaging biomarker in brain magnetic resonance imaging (MRI) in the diagnosis framework for AD [4-6].

Therefore, the visual grading of the MTA score including the atrophic change of the hippocampus represented by Scheltens' scale, has been a significant biomarker for diagnosing AD [7]. This scale showed moderate-to-good interobserver agreement and reproducibility among reviewers [8-10]. However, it is sensitive to the clinician's experience and perspective [11,12]. Therefore, to overcome the potential shortness of qualitative measurement, various commercial volumetry software have been developed. These software can aid in quantifying MTA and providing an age-appropriate percentile based on a normative database. For example, NeuroQuant® (version 3.0.1, CorTechs Labs Inc., California, USA) and Aqua® (version 2.0.0, NEUROPHET Inc., Seoul, Korea) are volumetry software. NeuroQuant® is one of the world's most validated volumetry software [13]; however, Aqua® remains unvalidated in clinical settings. Hence, performance regarding MTA for diagnosing AD should be assessed to help clinicians choose suitable volumetry software because the normative database varies among different software [4,14-16].

Therefore, we aimed to compare the performance of NeuroQuant® and Aqua® in detecting significant MTA in non-AD and AD groups. Additionally, we analyzed the correlation between the calculated mean quantitative hippocampal volume of each software and Scheltens' scale.

2. Materials and methods

This retrospective study was approved by our hospital's institutional review board and conducted in accordance with the Declaration of Helsinki's principles (Protocol number, PC21RESI0174; approval date, 11 November 2021).

2.1. Sample Eligibility

Between January and August 2021, 203 individuals who underwent brain MRI and visited the neurology or neuropsychiatry departments of our hospital's neuroscience center for subjective memory impairment or transferred from a community mental health center were enrolled as potentially eligible participants. The sample size was estimated based on the previous study and is summarized in Appendix E1 [17].

2.2. Clinical Diagnosis for AD

To determine whether participants had AD, clinical evaluations were conducted using the Korean version of the Mini-Mental State Examination (K-MMSE) and the Clinical Dementia Rating (CDR). Two dementia specialists (one neurologist and one neuropsychiatrist with over 10 years of clinical experience in diagnosing AD) diagnosed and classified the disease groups, including subjective cognitive impairment (SCI), mild cognitive impairment (MCI), and AD based on the 5th edition of the Diagnostic and Statistical Manual of Mental Disorders or the 2011 edition of the National Institute on Aging and the Alzheimer's Association [6,18]. Two clinicians classified participants with SCI and MCI into a non-AD group and the other participants with AD into an AD group.

2.3. Exclusion Criteria

Three neuroradiologists (J.K., S.-W.O., and J.-Y.K. with 8, 17, and 21 years of interpreting brain MRI experience) reviewed all MRI scans and excluded 59 samples that presented focal brain lesions or severe artifacts, which could interfere with the volumetry analyses of each software. Similarly, 17 samples were excluded because clinicians diagnosed the patients with diseases other than SCI, MCI, or AD.

2.4. MRI Acquisition Protocol

Each participant underwent a 3T MRI using a 64-channel head & neck coil (MAGNETOM Vida 3T, Siemens, Erlangen, Germany). Similarly, three-dimensional (3D) T1-weighted volumetric magnetization-prepared rapid gradient echo images were obtained with the following parameters: repetition time, 2090 ms; echo time, 3 ms; inversion time, 1200 ms; number of excitations, 1; section thickness, 1.0 mm; gap, 1.0 mm; matrix size, 256 × 230; flip angle, 9°; and field of view, 250 × 250 mm.

2.5. Volumetry Procedure

All MRI scans were analyzed using NeuroQuant® and Aqua® volumetry software. Each participant underwent one MRI examination including sagittal volumetric T1-weighted imaging, and the obtained images were uploaded to separate NeuroQuant® and Aqua® servers for automated volumetry analyses. After completing the analyses, the volumetry results of each software were returned to the picture archiving and communicating system of our hospital. The system displayed the volume, volume percentile based on the normative database, the proportion of volume to the total intracranial volume of brain substructure, and colored images of each software. More detailed volumetry analysis steps and examples of colored images of NeuroQuant® and Aqua® are described in Appendix E2 and Supplemental Figure 1. The cutoff of significant MTA was defined as <5% of the normative percentile.

2.6. Gold Standard for Significant MTA

All MRI examinations were reviewed by three neuroradiologists (J.K., S.-W.O., and J.-Y.K.). For hippocampal volume assessment, the 3D volumetric T1-weighted images were reconstructed into a coronal view perpendicular to the anterior commissure–posterior commissure line [19]. All reviewers were blinded to patients' history, clinical diagnosis, results of volumetry software, and other reviewers. MTA scoring was conducted according to Scheltens' 5-point Likert scale (0–4) based on assessing the width of the choroid fissure, the temporal horn of the lateral ventricle, and the height of the hippocampus [7]. Similarly, reviewers reported that individuals aged <75 years with an MTA score >2 and those aged >75 years with an MTA score >3 had significant MTA [7]. Consensus was made if there was a disagreement among reviewers.

2.7. Statistical Analyses

First, for comparing detection performance for significant MTA between NeuroQuant® and Aqua®, accuracy, sensitivity, and specificity with a 95% confidence interval (CI) were compared using the Chi-square test, defining the non-inferior margin as 0.05. A non-inferior test was performed for detection performance between the two software. We defined a -0.05 or -5% difference as a non-inferiority margin through consensus among participants' neuroradiologists, considering previous studies [20,21]. Second, for assessing the correlation between Scheltens' scale and the quantitative mean hippocampal volume of each software, Kendall's τ correlation was used. Additional inter-reviewer agreement among three reviewers was calculated using Fleiss' Kappa analyses with 95% CI. The Kendall's τ correlation coefficients, Kappa values with 95% CI were defined as follows: weak, <0.3; moderate, 0.3 ≤ and <0.7; and strong, ≥0.7. The baseline characteristics between non-AD and AD groups were compared using Student's t-test.

The statistics of sub-analysis for the mean volume comparison and inter-method reliability of each software is described in Appendix E3.

All statistical analyses were performed using SPSS (version 21.0, IBM Corp., Armonk, NY, USA) and MedCalc (version 20.215, MedCalc Software Ltd., Ostend, Belgium). P-value <0.05 indicated statistical significance.

3. Results

3.1. Baseline Characteristics of Study Sample

A total of 127 participants (mean age, 78.57±7.01 years; 38 men; 89 women) were enrolled. The number of non-AD (SCI, 23; MCI, 31) and AD were 54 (43%) and 73 (57%), respectively. The flow diagram of participants' enrollment is displayed in Figure 1. Participants in the AD group had significantly older ages with lower K-MMSE and higher CDR scores than those in the non-AD group (P<0.01). The baseline characteristics are summarized in Table 1.

3.2. Comparing Detecting Performance for the Significant MTA

The overall accuracy, sensitivity, and specificity of NeuroQuant® for detecting significant MTA were as follows: accuracy, 83.5 (95% CI, 77.2–90.0)%; sensitivity, 72.2 (95% CI, 60.3–84.2)%; and specificity, 83.8 (95% CI, 77.6–90.1)%. Aqua® had the following: accuracy, 87.4 (95% CI, 81.8–93.0)%; sensitivity, 72.2 (95% CI, 60.3–84.2)%; and specificity, 95.6 (92.1–99.1)%. Aqua® showed higher specificity and accuracy than NeuroQuant®; however, only specificity showed statistical significance. NeuroQuant® and Aqua® values differed insignificantly in the non-AD and AD groups. As the lower boundary of the CIs of accuracy (3.9%, 3.0–4.6%) and specificity (11.8%, 9.0–14.5%) were above the predefined non-inferiority margin, the non-inferiority of Aqua® was established after comparison with the NeuroQuant® in the non-AD and AD groups. However, Aqua® was not demonstrated to be non-inferior to NeuroQuant® regarding sensitivity. Similar results of the overall population, accuracy, sensitivity, and specificity of the non-AD and AD groups showed non-inferiority of Aqua® to NeuroQuant® except for sensitivity in the AD group. The detailed results are presented in Table 2. The Kappa value for MTA among three reviewers was 0.90 (95% CI, 0.86–0.92; P<0.01), indicating strong inter-reviewer agreement.

3.3. Correlation of Mean Volume with Scheltens’ Scale

The correlations between Scheltens’ scale and quantitative mean hippocampal volume were analyzed using each software. The correlation between Scheltens’ scale and NeuroQuant® showed a negatively weak correlation with statistical significance in the AD group (τ, - 0.20; P=0.02); however, the correlation was statistically insignificant in the non-AD group. The correlation between Scheltens’ scale and Aqua® showed a negatively moderate correlation with statistical significance in the non-AD (τ, - 0.35; P<0.01) and AD (τ, - 0.49; P<0.01) groups. All the τ of Aqua® in each group were significantly more correlated than those of NeuroQuant®. The scatterplot graphs are illustrated in Figure 2, and the results are summarized in Table 3.

3.4. Sub-analysis for Mean Volume Comparison and Inter-method Reliability

The sub-analysis comparing mean hippocampal volumes indicated that Aqua® was significantly larger than NeuroQuant® regardless of clinical diagnosis. The inter-method reliability of mean quantitative hippocampal volume between the two software was moderate-to-strong in the non-AD and AD groups. The statistics and detailed results of the other brain regions are described in Appendix E3 and Table E1.

4. Discussion

Aqua® and NeuroQuant® were compared in two ways in this study. In the former way, the comparison detecting performance for significant MTA, our results demonstrated that Aqua® showed significantly higher specificity than NeuroQuant®. Furthermore, detection accuracy and specificity were comparable between the two software according to a non-inferiority test in the non-AD and AD groups.

In the latter way, the correlation between Scheltens' scale and each software’s calculated mean hippocampal volumes demonstrated that NeuroQuant® showed a weak correlation; however, Aqua® showed a moderate correlation with Scheltens' scale, with a significant difference. This result is inconsistent with those of previous studies, which showed a more than moderate correlation of NeuroQuant® with Scheltens’ scale [22,23]. However, our sample has a larger number, and the gold standard was established by three neuroradiologists. Similarly, our mean MTA score and incidence of abnormal MTA were within the range of those in previous studies [7,17,22-24]. Therefore, our results are reasonable and more robust than those of previous studies. Considering that we did not aim to validate but compare two volumetry software, these results should be interpreted as Aqua® being more correlated with Scheltens' scale than NeuroQuant®. Visual assessment of MTA is essential for dementia work-up. Thus, Aqua® could be used by clinicians for monitoring the AD [5].

In sub-analysis for mean volume comparison, there was a significant hippocampal volume difference between both software with moderate to strong inter-method reliability. The hippocampus and most of the brain regions showed significant volume differences. There could be variable reasons for volume discrepancies between the two software. Different baseline anatomic atlases between NeuroQuant® and Aqua® could be a cause because NeuroQuant® and Aqua® were developed with the Talairach atlas and the Desikan–Killiany cortical atlas, respectively [25-27]. For example, the hippocampus has different anatomical definitions across the software. In the process of developing and establishing the gold standard through manual segmentation, different anatomical boundaries and tracing procedures can provide potential volume differences [28]. Similarly, different software applications involve using different algorithms to segment brain regions, which can cause variations in regional volume measurements [17,29,30]. Therefore, for clinicians, these factors could complicate the selection of a suitable volumetry software for acquiring a quantitative volume of brain substructure.

NeuroQuant® is one of the most validated volumetry software with good ability for brain volumetry [13,17]. To our knowledge, this study would be the first clinical study on Aqua® since the introduction of the software. However, the overall performance of Aqua® for detecting significant MTA and correlation with Scheltens' scale was not inferior to that of NeuroQuant® in our study. Because the brain morphology of the aging population could be affected by ethnicity owing to the geographic environment, implementation of a normative reference database could be a substantial factor for our results [16,27].

This study has some limitations. First, because of the study’s retrospective nature, there could be a selection bias. However, it was an inevitable limitation owing to the nature of the study design. Second, since a cutoff of 5% was used for significant MTA of each software, there could be variations in the detection performances when different cutoff values were used. Finally, although the specificity of Aqua® for significant MTA and correlation with Scheltens' scale were superior to those of NeuroQuant®, these results not indicate that Aqua® had better performance than NeuroQuant® for the other regions. Therefore, further comprehensive study would be required to generalize the performance of Aqua®, and clinicians should consider these limitations when selecting the volumetry software and interpreting the results.

5. Conclusions

The accuracy of Aqua® was not inferior to that of NeuroQuant® in detecting MTA. Similarly, Aqua® was significantly more correlated to the calculated hippocampal volume than NeuroQuant®. Therefore, Aqua® could be considered an alternative volumetry option and could aid clinicians in determining MTA.

Appendix E1. Calculation of Sample Size

The sample size for the two-tailed Student's t-test was calculated using the common formula for calculating sample sizes for proportions. Based on a statistical power level of 0.80 and a statistical significance level of 0.05, we estimated the sample size. Similarly, based on previous studies, the impact size was conservatively established at 0.5 [1]. The sample size for this study was calculated to be 128.

Appendix E2. Volumetry Process Steps of NeuroQuant® and Aqua®

NeuroQuant® is a spin-off of FreeSurfer, a software with a research focus used for volumetry. The steps in the NeuroQuant® process were as follows: the process involves removing the scalp, skull, and meninges, expanding the brain into a sphere, mapping that sphere to a shared space using the Talairach atlas coordinates, identifying the segmented brain areas, and contracting the brain back to its original shape [1].

Similarly, Aqua® is built on FreeSurfer using Split-Attention U-Net, a convolutional neural network with skip routes and a split-attention module that separates brain MRI images. Two open sources and three research sites provided a wide variety of cognitively normal East Asian populations' normative MRI data (age, 18–96 years). The 97 different regions of the brain are divided using Aqua®, which is used to examine structural details, such as atrophy, structural alterations, anatomical asymmetry, and white matter abnormalities associated with degenerative brain illnesses. The segmentation of the central nuclei, hippocampus, brainstem, and ventricles; lobar parcellation; and the detection of white matter abnormality. The process steps of Aqua® included skull stripping, a tissue classification to extract brain tissue compartments, such as the white matter, gray matter, and intra/extra ventricular cerebrospinal fluid [2,3].

Appendix E3. Mean Volume Comparison and Inter-method Reliability

We categorized study samples into three groups according to clinical diagnoses by two clinicians: subjective cognitive impairment (SCI), mild cognitive impairment (MCI), and Alzheimer's disease (AD) groups. We compared the mean volume of brain substructures between the two software using the paired t-test. Considering clinical diagnosis, most brain regions showed significant differences in mean volume except for the parietal lobe in the AD group and the frontal lobe and caudate nucleus in the SCI group, the white matter, parietal lobe, caudate nucleus, and cingulate gyrus in the MCI group. Intracranial volume, cortical gray matter, white matter of AD group, parietal lobe, hippocampus, amygdala, caudate nucleus, globus pallidum, and cerebellum were larger in Aqua® than NeuroQuant®. The other regions exhibited the reverse results.

Furthermore, we assessed the inter-method reliability between the two software using Pearson's correlation coefficient (r) and intra-class correlation coefficient (ICC). Pearson's correlation coefficient and ICC with a 95% confidence interval were defined thus: weak, <0.3; moderate, 0.3 ≤ and < 0.7; strong, ≥0.7. Most brain regions exhibited statistically significant values regardless of the clinical diagnosis, except for the caudate nucleus and accumbens of the SCI group and the cingulate gyrus and globus pallidum of the MCI group. These significant regions showed moderate to strong linear correlations across all groups (r=0.35–0.98). Most brain regions without group classification demonstrated moderate to strong inter-method agreement (ICC=0.45–0.96). Considering clinical diagnosis, the accumbens and globus pallidum of the SCI group and the cingulate gyrus and globus pallidum of the MCI group showed statistically insignificant values. On excluding these data, most brain regions showed moderate to strong inter-method agreement (ICC=0.37–0.97). These results of the mean volume comparison and inter-method reliability between the two software are summarized in Table E1.