The Usefulness of the Regression-Based Normed SKT Short Cognitive Performance Test in detecting Cognitive Impairment in a Community Sample

1. Introduction

In 2015, nearly 50 million people around the world were affected by various forms of dementia [1] According to the GBD report, this number is projected to increase by 10 million each year. These figures highlight the global significance of dementia as a major social issue. Implementing preventive and therapeutic measures in the early stages of the disease can slow down its progression and delay the need for long-term care [2,3]. Currently, there is no therapy available to cure dementia. However, there are therapies designed to slow down or alleviate the symptoms of the illness. Drugs based on monoclonal antibodies in particular offer hope for Alzheimer’s disease specifically. The controversial effects of early monoclonal antibody drugs in this class, like Solanezumab or Aducanumab, on cognitive decline was hotly debated [4] and studies have shown that both drugs don’t slow cognitive decline [5]. In contrast, the two drugs Lecanemab and Donanemab, introduced sometime later, have shown promising results when it comes to slowing cognitive and functional decline in people diagnosed early with Alzheimer’s disease [6,7]). However, it remains unclear how effective the new drugs are compared to possible side effects [7].

Early detection of dementia during the prodromal stage, when targeted interventions are most effective, is crucial. In order to achieve this goal, it is essential to have reliable tests that can detect cognitive decline as soon as possible. One potential option for such a test is the Short Cognitive Performance Test (SKT, German abbreviation for Syndrom Kurz Test;[8,9])

The SKT is a brief cognitive assessment that includes nine subtests, with a total application time of no more than 15 minutes. Among these subtests, three (subtests II, VIII, and IX) measure memory, while the remaining six (subtests I, and IV to VII) evaluate attention. Based on the extracted number of factors, additional factors such as executive function or word fluency have been identified [10]. By examining the validity and reliability of the SKT, many studies have confirmed its usefulness [11,12]). In Addition the SKT has been translated into several languages like spanish and norwegian [13,14].

The SKT offers several advantages compared to other brief cognitive assessments. Firstly, there are five parallel test forms, enabling the test to be repeated for monitoring the progression of cognitive decline over time without practice effects [8]. Another notable advantage of the SKT is that it takes into account a number of factors in its scoring. Several studies (e.g., [15]) have shown that performance on cognitive tests depends on demographic factors and that these factors may influence cut-off points for norming. It’s therefore important to include factors such as age, intelligence and gender in the norms. Since 2015, an updated norming process has enhanced the usability of the SKT.

A detailed description of the new norming procedure can be taken from the SKT Manual [8]. For the new norming procedure, multiple regression analyses were used to predict expected SKT raw scores for each subtest based on age, sex, intelligence, and their interactions, in a large sample of older healthy persons. Based on the magnitude of the difference between observed and expected scores, either 0, 1, or 2 deviation points were given for each subtest with more points indicating lower performance than expected. The resulting total scores range from 0 to 18. Higher points indicate lower-than-expected cognitive performance. Hence, the new norming approach embraced a new regression-based methodology [16]. Traditional norming procedures standardize an individual’s raw score, which is compared to typical scores from the reference group [17]. If, for instance, age is the norming predictor, an individual’s raw score is compared to scores of same or similar aged people. Such traditional norms of subgroups are fine for categorial norming predictors such as school type but become somewhat arbitrary for continuous variables like age. For a traditional standardization, test developers divide the continuous norming predictor into arbitrary intervals and compute standardized scores within each subgroup using linear (e.g., T or IQ values) or area transformations (e.g., percentiles of norms). However, traditional norming methods may be biased if the same raw score produces a different normed score solely because of the person’s age group, especially if individual’s age is near the threshold time point. Of course, the biases are strongly influenced by the width of the (age) subgroups [17]. Regression-based norming of the SKT therefore has an advantage in that there are no categorized age or intelligence groups, rather this statistical norming approach means it is possible to use the exact age (in years) or the exact IQ. This precise norming reduces the biases mentioned above. For more comprehensive information on the English norming sample used for the SKT, please refer to the English manual [8]

The updated norming procedure for the SKT was initially validated using a German-speaking sample, yielding satisfactory outcomes [18]. It showed that, as suggested, there are differences in norms between German and English speakers. Predictors such as age or IQ varied in the strength of their influence on test results. Based on these findings, it can be said that it is important to have language-specific norms in order to achieve high diagnostic accuracy. Thus, these new regression-based norms based on data from the English norming sample [18] are a first step towards using the SKT in English-speaking countries and achieving accurate results.

To our knowledge, there are no valid cut-offs to differentiate between the diagnostic groups of no cognitive impairment, Mild Cognitive Impairment (MCI), and dementia for the English SKT. Further, this new norming approach has not yet been applied to an English-speaking sample. The aim of this paper is therefore twofold: (1) to establish valid cut-offs for distinguishing between no cognitive impairment, MCI and dementia, and (2) to assess both the criterion and concurrent validity of the SKT in English-speaking older adults. To do this, an English validation sample was established using data from the Sydney Memory Ageing Study (MAS). Although some of the participants in the present study were already part of the original English norming sample which included only cognitively healthy participants. The data for the current validation study included therefore, healthy subjects and participants with diagnoses of MCI and dementia. In the search of appropriate cut-offs and for the validation of the English norming, we compared SKT scores also with scores on the Mini-Mental Status Examination (MMSE [19]), the Addenbrookes’s Cognitive Examination (ACE-III [20]), and a clinical consensus diagnosis made by an expert panel [21].

2. Materials and Methods

2.1. Participants

Participants for the original MAS were aged 70 – 90 years old and were recruited from two electoral districts in the Eastern Suburbs of Sydney, Australia, between 2005 and 2007. From 8,914 individuals approached, 1,037 were selected for the baseline sample. Inclusion criteria at baseline were the ability to read and write English sufficiently well to undergo psychometric evaluations and fill out self-report questionnaires. Individuals were excluded if they had psychiatric conditions, acute psychotic episodes, or had a current or previous diagnosis of multiple sclerosis, motor neuron disease, developmental disabilities, aggressive cancers, or dementia. Participants were further excluded if they scored below 24 on the MMSE at baseline. Participants gave written consent to partake in this research, which was approved by the University of New South Wales Human Ethics Review Committee (HC 05037, 09382, 14327, 190962).

Every two years MAS participants undertook a detailed assessment with a trained research assistant (called a ‘wave’) where they completed a comprehensive neuropsychological test battery, medical history, medical exam, and a series of questionnaires. Clinical diagnoses were made at each wave by an expert consensus panel who considered all available neuropsychological, clinical, and informant-reported data. At each wave MCI was diagnosed using international consensus criteria [22], and dementia was diagnosed according to the Diagnostic and Statistical Manual of Mental Disorders-IV [23]. All of the patients who were diagnosed with dementia according to the DSM-IV also met the criteria for a major neurocognitive disorder according to the DSM-V [24]. For the current study, participant data from waves 6 and 7 (approx. 12 - 14 year follow-ups) were considered as wave 6 was the first time the SKT was administered in MAS. Of the 317 participants who completed wave 6 and 7 of MAS, a subset of 140 completed the SKT from February 2018 to February 2020.

2.2. Cognitive Measures

2.2.1. SKT

The SKT is a short cognitive performance test assessing impairments in memory and attention. The test is scored on an 18-point scale, with lower scores indicating better cognitive performance and is adjusted for age, sex, years of education and IQ [8]. For the present validation study, age, sex, and years of education were obtained through MAS questionnaires and interviews, where IQ was conceptualized as crystalized and fluid intelligence. Crystalized intelligence was measured using the F-A-S Test, a verbal ability test which is a strongly influenced by school education. The F-A-S is a subtest of the Neurosensory Center Comprehensive Examination for Aphasia (NCCEA [25]), and fluid intelligence was measured using the Digit Symbol Coding (DSC) subtest of the Wechsler Adult Intelligence Scale [26]. For norming and reasons of comparisons, results from both intelligence tests were transformed into Wechsler value points, with a mean (M) of 10 and a standard deviation (SD) of 3 [27]

To ensure comparability to the original norming sample, the results for the current sample were also categorized into three groups according to the traffic-light coding method developed by Stemmler and colleagues [8]: no cognitive impairment (green), MCI (yellow), and dementia (red). Eighty-nine people (63 %) were classified as green (no cognitive impairment), forty-nine people (35 %) were classified as yellow (MCI), and two people (1.4 %) were classified as red (dementia).

2.2.2. MMSE

The MMSE [19]a well-established test of global cognitive function used to screen for dementia. The MMSE is an 11-question measure that tests different areas of cognitive function, including orientation, attention, recall, and language. It uses a 30-point scale, where a lower score indicates a higher level of cognitive impairment. Generally, a score of 24 or below is considered indicative of cognitive impairment (possible dementia), distinguishing it from normal cognitive function [19].

2.2.3. ACE-III

The ACE-III [20] is a well-validated brief assessment of cognitive functioning that takes 12 to 20 minutes to complete. The ACE-III consists of five subscales: attention/orientation, memory fluency, language, and visuospatial functioning, with a maximum total score of 100. Lower total scores indicate poorer cognitive functioning and a score of 88 or below recommended as a possible indicator of dementia [28].

2.2.4. Data collection

The SKT was administered to participants at either wave 6 or wave 7. One patient had SKT data from both waves and was excluded from the study. The MMSE was administered to participants at both wave 6 and wave 7. For those participants who were administered the SKT at wave 6, the wave 6 MMSE score was used for calculations. For those patients who were administered the SKT at wave 7, the wave 7 MMSE score was used for calculations. The ACE-III was administered to all participants at wave 6. Wave 6 ACE-III scores were also used for participants who completed the SKT at wave 7 (N = 106); see Table 1 for an overview of the data collection timeline².

2.2.5. Consensus Diagnoses

At each wave, participants were brought to a consensus review meeting where a panel of neuropsychiatrists, psychogeriatricians, and neuropsychologists discussed all available clinical, neuropsychological, laboratory and informant-reported data to reach a consensus diagnosis. The consensus diagnosis did not involve the SKT. A range of cut-offs indicated when cases should receive consensus review, including impaired performance on neuropsychological tests (at least 1.5 SDs below published normative data on two or more domains). To diagnose MCI, the consensus criteria defined by Windblad et al. [22] were used. According to these guidelines, a person was diagnosed with MCI if they reported subjective cognitive complaints and showed cognitive impairment (SD ≤-1.5) on one cognitive test, if they did not have a diagnosis of dementia, and if they had no or minimal impairment in instrumental activities of daily living (for more detailed and additional information see [21]). A diagnosis of dementia was based on the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV [23]) criteria which include the cognitive impairment (SD ≤ -1.5) on at least two cognitive tests, with one test being memory specific, that is sufficiently severe as to cause impairment in daily functioning (e.g., Bayer Activities of Daily Living score ≥ 3.0). Participants who had complete neuropsychological test data and did not meet criteria for an MCI or dementia diagnosis were classified as ‘normal cognition’ at each wave. For both waves, 111 consensus diagnoses were made. For the present study, eight participants were excluded because they had a non-amnestic form of MCI (nMCI). This decision was based on the finding that people with nMCI do not have significant memory impairments [29] which is what the SKT measures. Finally, seventy people (69 %) were classified as green (no cognitive impairment), 15 people (14 %) were classified as yellow (MCI), and 16 people (16 %) were classified as red (dementia). Figure 1 shows this distribution of the SKT classification.

2.2.6. Comparing Group Status Across Measures

Figure 1 presents a line diagram with the colors of the SKT traffic-light system as the categories, the SKT raw scores as the category axis, and the frequencies of the sum scores as the y-axis. Based on these values, we chose 3 and 11 as cut-off values for the traffic light system. That is, participants with a score from 0-3 were classified as green, indicating no cognitive impairment; participants with a score from 4-10 were classified as yellow indicating MCI; and participants with a score of 11+ were classified as red, indicating dementia. For the ACE-III, individuals with scores higher than 88 were categorized as not cognitively impaired, while individuals with scores lower than 83 were categorized as possible dementia. Those with scores between 83 and 88 inclusive were classified as possible MCI [20]. Regarding the MMSE, participants with scores above 26 were categorized as no cognitive impairment, those with scores of 26 and 25 as MCI, and those with scores lower than 25 as possible dementia [19].

Finally, diagnostic classifications from each measure were compared with participant’s consensus diagnosis, which was based on all available data and considered to be the gold standard.

2.3. Statistical Analysis

First, descriptive statistics were generated for the validation sample. Then, different specificity and sensitivity values were calculated by cross-classifying the three levels of cognitive impairment (no cognitive impairment, MCI, dementia) with the different cognitive test results. To assess concurrent validity, bivariate correlations (Kendall´s Tau) were calculated between the consensus diagnoses and the classification of the SKT, the MMSE and the ACE-III. To better understand the test results, all three tests were transformed into categorical variables (0 = no cognitive impairment, 1 = MCI, and 2 = dementia). Next, receiver operator characteristics (ROC) analyses were used to calculate the area under the curve (AUC) for the total SKT, MMSE and ACE-III scores. The AUC is a measure of the overall classification accuracy of a test, where 1 is perfect accuracy. AUCs were calculated for all three tests. Additionally, for the SKT, Mann-Whitney-U-Tests were performed, where the independent variable was the SKT categorization group (based on the traffic light coding system), while the dependent variable was the consensus diagnosis. To account for multiple comparisons, we employed the Bonferroni-Holm correction. All analyses were carried out using the software SPSS 29 or R-Software.

3. Results

3.1. Sample Characteristics

The sample consisted of 143 people aged between 83 and 98 (mean age = 87.7). 90 of the sample were female and the remaining 53 were male. Further characteristics of the sample are shown in Table 2.

3.2. Diagnostic Congruency

Table 3 presents the congruency between consensus diagnoses and the SKT diagnoses. The overall classification congruency was 71/101 (70.30 %). The SKT demonstrated the highest hit rate for the category of no cognitive impairment (82.9%) and the lowest hit rate for the category of dementia (12.5%). Importantly, the hit rate for MCI was much higher at 73.3% compared to the hit rate for dementia. In the group of participants without cognitive impairment, none received a red label from the SKT (indicating possible dementia), but 17% were given a yellow label (indicating possible MCI). Among those with a consensus diagnosis of MCI, none were assigned a red SKT label, though two participants received a green SKT label, suggesting no impairment.

3.3. Sensitivity and Specificity

Table 4 presents sensitivity and specificity values for the SKT, MMSE, and ACE-III. For all three tests, the consensus diagnosis is taken as the “true diagnosis” and is used to calculate sensitivity and specificity. Across each test, the sensitivity in distinguishing between no cognitive impairment vs. cognitive impairment (i.e., normal cognition vs. MCI or dementia) is the highest, while the sensitivity for distinguishing between normal cognition vs. MCI is lower. In addition, of the three tests, the SKT has the highest sensitivity in both categories. Regarding specificity, all three tests demonstrate relatively high values across both categories. The lower specificity is observed in the category of no cognitive impairment versus cognitive impairment for all tests. The ROC analysis was calculated using the cut-offs which were determined on the basis of Figure 1.

Finally, Mann-Whitney U tests were performed to analyze the extent to which the both categories (no cognitive impairment vs. cognitive impairment, normal cognition vs. MCI) differed significantly in their consensus diagnosis. The results of the tests correspond to the sensitivities and specificities. The bar plots for the corresponding tests are shown in Figure 2a,b. For the category “no cognitive impairment vs. cognitive impairment”, the result was significant (U=384.00; p < .001). The result suggests that there is a significant difference in the SKT scores between individuals with no cognitive impairment and those with cognitive impairment.

For the category “normal cognition vs. MCI”, the result was also significant (U = 239.0; p <.001). Again, these results indicate a notable difference in SKT scores between individuals with no cognitive impairment and those with MCI.

3.4. Concurrent Validity

In Table 5, correlations between the dementia tests and the consensus diagnoses, according to Kendall’s Tau, are presented. All correlations, both between the psychometric tests and the consensus diagnoses, as well as between the psychometric test scores themselves, were significant. Furthermore, these results indicate that of all three tests, the SKT has the highest correlation with the consensus diagnosis.

4. Discussion

After the establishment of the new English norms for the SKT in 2021, we aimed to determine optimal cut-off points for the SKT in an English validation sample that would distinguish between no cognitive impairment, MCI, and dementia in a sample of older adults. Based on a line graph, the optimal cut-off score for MCI was 3 and 10 for dementia. The line graph showed significant increases after the cut-offs but no comparable increases at other points. These results support the accuracy of the cut-offs and are consistent with the cut-offs found for the German version of the SKT [12] that is, the cut-off for MCI in the English version is one point lower than in the German version (3 instead of 4), but the cut-off for dementia is the same. For the English norming we recommend a cut-off of 3, especially if the mean age of the sample is high (c. f [30]) (here it was about 88 years); otherwise a cut-off of 4 should be considered. These results suggest that the basic structure of the SKT remains very similar, although further research [18,31,32] has shown that the strength of influence of different factors (e.g., age, gender, intelligence) varies between different language-specific norms.

The second objective was to validate the new regression-based norms for the SKT in a sample of older English-speaking adults. Together the results, indicate that the SKT is a valid instrument for discriminating between older adults with no cognitive impairment and those with cognitive impairment (i.e., MCI or dementia). In this context, the cut-off value of 3 seems to be effective in distinguishing between the two groups. This is a very encouraging result, as the primary aim of the SKT is to detect cognitive impairment as early as possible. The main focus of the SKT is not on distinguishing between MCI and dementia, but between no cognitive impairment and cognitive impairment. The SKT performs very well in this area, and indeed, was better at distinguishing between these two groups compared to the MMSE and ACE-III, two commonly used cognitive (screening) tests. We also decided to test the sensitivity between normal cognition and MCI because MCI is a valid predictor of developing dementia. However, MCI may be harder to detect than dementia because the cognitive impairment is not as severe as in dementia. Fortunately, the sensitivity for normal cognition vs. MCI was only slightly lower than the sensitivity for no cognitive impairment vs. cognitive impairment. These results suggest that the SKT is also able to detect milder forms of cognitive impairment. These findings underscore the importance of early use of dementia screening tests such as the SKT, because earlier detection of pathological cognitive decline provides an opportunity to start interventions that slow cognitive decline. Additionally, the SKT exhibits high psychometric properties, effectively distinguishing between cognitive impairment and normal cognitive function. This ensures that cognitively healthy older adults do not place undue strain on the healthcare system with unnecessary treatments. Additionally, results revealed that the differences between mean SKT scores across the groups of normal cognition, MCI and dementia were significant. This is an indication that, although the power is not overwhelming for some comparisons, there is a clear difference in SKT performance among people different clinical diagnoses.

The English norming calculation assumes that factors such as age, IQ, and sex have varying influences on SKT performance across different populations. The evidence supports the notion that language-normed dementia screening tests are relevant. Compared with other tests, the SKT has excellent sensitivity. However, there are only small differences in sensitivity and specificity between the SKT and the ACE-III, a well validated screening test for cognitive impairment. It is important to note, however, that the SKT is shorter to administer and easier to score than the ACE-III, which is an important benefit. The MMSE performed less well in this sample. The MMSE performs best in terms of specificity, but performs poorly in terms of sensitivity, which is a serious drawback.

These findings are supported by the correlations between the dementia tests and the consensus diagnoses. The correlation between the SKT and the consensus diagnosis is the highest, consistent with the good sensitivity of the SKT. The high correlation between the consensus diagnosis and the SKT suggests that they measure the same construct, cognitive impairment. In addition, the correlation between the ACE-III and the SKT is at a medium level. The ACE-III also correlates at a medium level with the consensus diagnosis, so it is evident that all three measure the same overall construct. These findings are consistent with the very good sensitivity of the SKT and the ACE-III. The fact that the correlation between the MMSE and the consensus diagnosis is the lowest, and that the correlations between the MMSE and the other two dementia tests are relatively low, is also consistent with the poor sensitivity of the MMSE. Thus, the correlations in combination with the other results support the hypothesis that the SKT and the ACE-III measure the construct of cognitive decline at a valid level.

4.1. Limitations

Although results indicate good validity of the SKT, there are some limitations. The first limitation is that the norms used for ACE-III and MMSE did not account for age or education, so it is possible that both tests would have performed better if these corrections were made. However, to our knowledge, there are only exploratory papers (e.g., [33,34,35] for specific samples with corrections for variables that could be associated with cognitive decline. As there is no manual with corrected norms available, we have decided to use the uncorrected norms that are commonly used by most applications of both tests. Another limitation of this study is that the ACE-III was administered only in wave 6 of the MAS, meaning that ACE-III test scores for 102 participants were obtained from wave 6 and compared to wave 7 MMSE, SKT, and consensus diagnoses for these participants. This could be problematic as there could be up to a 4-year gap between the two waves, during which time cognitive function may have changed. Therefore, it is possible that the sensitivity and validity of the ACE-III test may be underestimated in this paper. As the sample size would only consist of 33 individuals if we had used data from the same wave for the ACE-III, we acknowledged this limitation and suggest to obtain a larger sample. As stated earlier, the sample size was limited and the participants were relatively old. Furthermore, the participants were drawn from two different waves of a 12 to 14-year study, which may have resulted in non-random attrition bias. This implies that the majority of individuals who performed the SKT and were included in the sample had favorable cognitive abilities and/or an increased cognitive reserve, having remained in the study for 14 years, some with a diagnosis of MCI or dementia. Therefore, it is probable that the sample was not wholly representative of older adults, especially at that age. It is noteworthy that the intelligence was assessed using the aggregated scores from two tests, which represent the fluid and crystallised components of intelligence. Digit Symbol test is a clear measure of fluid intelligence [36]. The FAS tests verbal fluency, which is a measure of crystallised intelligence. However the time limited testing of the FAS includes also a fluid intelligence component [37]. Therefore both tests can be considered as reliable estimators of IQ. A full-scale measurement was too time consuming and burdensome for the older adults. Of course of part of the daily testing routine, the estimate of IQ can be based on educational years in the absence of a premorbid IQ value.

5. Conclusions

The results of the studies show that the SKT has very good psychometric properties and compares favorably with other established psychometric tests for dementia. The test is also quick and easy to administer, and different versions are available. This allows an individual´s performance to be measured over time and can help healthcare professionals to identify early changes that may indicate early signs of dementia pathology. The combination of these factors points to the high potential of the SKT in testing for cognitive impairment. Due to the relatively limited sample size, further studies may help to confirm the results of this study.