Introduction
The ageing population and increasing prevalence of age-related disorders such as mild cognitive impairment (MCI) and dementia have encouraged the development and improvement of discriminative neuropsychological instruments (Nilson et al., Reference Nilson, Bäckman, Sällsten, Hagberg and Barregård2003). The role of screening instruments, unlike extensive neuropsychological tests, biomarker assessments and imaging techniques, is to quickly detect cognitive impairment (Applegate et al., Reference Applegate, Blass and Williams1990; Appels and Scherder, Reference Appels and Scherder2010). Rapid detection of cognitive impairment is critical, particularly in primary care, because it quickly indicates a need for further investigation. Moreover, extensive neuropsychological testing is often difficult in older patients due to general factors, such as motivation and fatigue (Zhu and Tulsky, Reference Zhu and Tulsky2000). However, the applicability of currently available screening instruments is under debate. Only few have been validated in the populations for which they are intended to be used. Although many screening instruments have high accuracy for moderate to severe cognitive impairment and dementia, many have low accuracy for mild levels of impairment (Ritchie et al., Reference Ritchie, Artero and Touchon2001). Additionally, many existing cognitive screeners have been developed for detecting cognitive decline in Alzheimer's disease and may therefore be less sensitive to cognitive impairment of patients with non-Alzheimer's dementia. Furthermore, demographic characteristics such as age and education level have great influence on cognitive performance (Cullen et al., Reference Cullen, O’Neill, Evans, Coen and Lawlor2007), but age and education adjusted norms are often lacking. An ideal cognitive screening instrument should be rapid, tolerated by patients, easy to administer and score, relatively independent of culture, language, and education and psychometrically robust (Shulman, Reference Shulman2000; Solomon and Murphy, Reference Solomon and Murphy2005).
The Folstein Mini-Mental State Examination (MMSE) is internationally the most commonly used screening instrument for cognitive dysfunction (Milne et al., Reference Milne, Culverwell, Guss, Tuppen and Whelton2008; Ismail et al., Reference Ismail, Rajji and Shulman2010), and has been shown to be a valid measure of cognitive decline in moderate and severe dementia. The MMSE was originally developed to detect moderate to severe dementia and it has been criticized for its poor sensitivity and specificity in the detection of MCI, mild dementia and patients with frontal pathology (Gregory et al., Reference Gregory, Orrell, Sahakian and Hodges1997; Mitchell, Reference Mitchell2009). Other frequently reported limitations are lack of standardized administration and scoring, influences of age, educational, or cultural biases and poor psychometric properties (Scazufca et al., Reference Scazufca, Almeida, Vallada, Tasse and Menezes2009; Ismail et al., Reference Ismail, Rajji and Shulman2010). Notably, the MMSE has also shown insufficient test–retest reliability (Tombaugh, Reference Tombaugh2005). Hence, additional studies of cognitive screeners are warranted.
For decades the Wechsler Memory Scale has been a widely used memory battery in the United States (Rabin et al., Reference Rabin, Barr and Burton2005). A goal for the development of the latest update, the Wechsler Memory Scale – Fourth Edition (WMS-IV: Wechsler, Reference Wechsler2009), was to include a rapid cognitive screening tool to identify significant cognitive impairment. As a result, the Brief Cognitive Status Exam (BCSE) is now included as an optional subtest. The BCSE has been developed for rapid identification of individuals with cognitive deficits, and assesses a variety of cognitive functions including orientation to time, mental control, clock drawing, incidental recall, automaticity and inhibitory control, and verbal production. In the technical and interpretative manual of the English version of the WMS-IV (Wechsler, Reference Wechsler2009), it is reported that the BCSE yielded clinical sensitivity to dementia that was equivalent or better than the MMSE. For the BCSE, the authors found average sensitivity (77%), high specificity (98%), high positive predictive values (97%), and negative predictive values (81%) for dementia of the Alzheimer's type. However, they emphasized that the BCSE is an optional subtest that provides rapid screening for significant cognitive problems, and cannot distinguish between different clinical disorders. To our knowledge, this is the first study to investigate the psychometric properties of the BCSE.
The present study examines the validation of the Dutch version of the BCSE (Wechsler, Reference Wechsler2009; Hendriks et al., Reference Hendriks, Bouman, Kessels and Aldenkamp2014) in older adults with MCI or dementia, comparing it to the MMSE. We examine the internal consistency of the BCSE, the correlations with age and education level, construct validity with respect to the MMSE, and whether the BCSE total and section scores can distinguish between patients with dementia, MCI and healthy controls. In line with previous studies, we expect to find good sensitivity and specificity of the BCSE for mild to moderate dementia, and hope to find good sensitivity and specificity for patients with MCI. Finally, optimal cut-off scores for the BCSE will be calculated.
Method
Participants
A total of 108 patients with memory complaints were recruited from several memory clinics in the Netherlands (Academic Medical Centre Amsterdam (n = 16), Fransiscus Hospital Roosendaal (n = 10), University Medical Centre Utrecht (n = 6), Radboud University Nijmegen (n = 15), and Hospital Twente Almelo (n = 61)). Standard clinical diagnoses were established by a multidisciplinary team based on the current international guidelines (Román et al., Reference Román1993; Albert et al., Reference Albert2011; McKhann et al., Reference McKhann2011). This standard procedure included a thorough clinical interview and extensive neuropsychological and psychiatric screening, supported by neuroimaging findings. As these patients were recruited from various clinics, no standard clinical battery was performed in all patients. However, in all centers the memory assessment was supervised by certified clinical neuropsychologists with expertise in dementia. Typically, neuropsychological assessment included: a global orientation and cognitive screening, such as the Revised Cambridge Cognitive Examination; word-learning tests, such as the Rey Auditory Verbal Learning Test and the 8 Word Test; visual memory tests, such as the Visual Association Test and Rey-Osterrieth Complex Figure Test; and executive function tests such as Trail Making Test and Stroop Test, as well as additional tests such as Verbal Fluency and Digit Symbol Substitution (see Lezak et al., Reference Lezak, Howieson, Bigler and Tranel2012, for a detailed description of each test). Of these patients, 44 were diagnosed with MCI, 44 with Alzheimer's dementia (AD), 11 with vascular dementia (VD) or mixed dementia (combined Alzheimer's dementia and vascular dementia), and 9 with subjective memory complaints but no objective memory deficits. Patients with subjective memory complaints but no objective memory deficits were excluded from further analyses. Moreover, data from nine patients were excluded because they met one of the following exclusion criteria: a comorbid psychiatric disease; a significant hearing or visual impairment; inability to speak/understand the Dutch language; a comorbid neurological or other disorder or use of medicines affecting cognitive functioning. As a result, we included 39 patients with MCI who had a Clinical Dementia Rating (CDR: Morris, Reference Morris1993) of ≤ 0.5 and 51 patients with mild to moderate dementia (CDR of 1–2). Patients with different types of dementia were combined into one group because of the small subgroup size. The control group consisted of 96 healthy controls selected from the WMS-IV-NL standardization study matched for age (ranging from 60–90 year), sex, and education level (3 levels: low, average, and high) with the patient groups. Education level was assessed by classifying formal schooling in the Netherlands according to the grouping of the Central Office for Statistics of the Netherlands (CBS, 2011), which is based on the International Standard Classification of Education (ISCED: United Nations Educational, Scientific and Cultural Organization Institute for Statistics UNESCO-UIS, 2011). The three levels were composed of those with early childhood, primary and lower secondary education (Low), those with upper secondary, postsecondary non-tertiary and short cycle tertiary education (Average), and those with academic training: bachelor’s, master's and doctoral degree (High). Exclusion criteria included history of psychiatric, neurological, or neuropsychological problems or use of medicines affecting cognitive functioning. The MMSE was performed to exclude control participants with cognitive impairment using the established cut-off of 24. Moreover, all participants had normal or corrected-to-normal hearing and vision.
The WMS-IV-NL standardization study was approved by the Institutional Review Board of Radboud University Nijmegen. Patient data were collected as part of the routine assessment of each participating centre. Participant characteristics are summarized in Table 1.
Participant characteristics
Neuropsychological tests
The BCSE assesses a variety of cognitive functions including orientation to time and place (12 points), mental control (12 points), clock drawing (4 points), incidental recall (8 points), automaticity and inhibitory control (16 points), and verbal production (6 points). The total BCSE score ranges from 0 to 58 with lower scores indicating poorer cognitive function. Moreover, the total score can be converted to a classification level (Average, Low Average, Borderline, Low, and Very Low) taking age and education level into account. The MMSE consists of 30 questions in a number of domains such as orientation, registration, attention, calculation, recall, language, repetition, and complex commands (Folstein et al., Reference Folstein, Folstein and McHugh1975). The total raw score ranges from 0 to 30. The administration time of each of the screeners varied between different individuals, and ranged from 5 to 15 minutes.
Statistical analyses
To examine the internal consistency of the BCSE, a reliability analysis was performed using Cronbach's alpha. In the control-group, correlations between age and BCSE and MMSE total scores were calculated with Pearson product-moment correlations and correlations between education level (Low/Average/High) and BCSE and MMSE total scores were calculated with Spearman's rank correlations. Furthermore, Pearson product-moment correlations were calculated to examine the compatibility between the BCSE and the MMSE total and section scores, including all participants.
To examine the ability of both the BCSE and MMSE in differentiating between the patient groups and healthy controls, one-way analyses of variance (ANOVA) were conducted. A subsequent one-way multivariate analysis of variance (MANOVA) was conducted to examine the BCSE section scores. Significant differences were analyzed with Hochberg's post-hoc test.
Receiver Operating Characteristics (ROC) analyses were conducted to determine the discriminative validity of the BCSE and MMSE total scores. ROC analysis produces an Area Under the Curve (AUC), which indicates the discriminative power of the test. This value ranges between 0.5 (no discriminative power) to 1.0 (maximum discriminative power). Moreover, ROC analysis can help predict a number of diagnostically useful measures such as sensitivity, specificity, positive predictive value (PVV), negative predictive value (NPV), diagnostic odds ratio (DOR), and likelihood ratios. The optimal cut-off scores were considered based on the best balance between sensitivity and specificity levels. As the current study evaluates cognitive screeners, we chose > 80% as the cut-off for good sensitivity and NPV and > 60% as an acceptable low ratio of false positives resulting in specificity and PVV (Blake et al., Reference Blake, McKinney, Treece, Lee and Lincoln2002). ROCs were plotted for both instruments’ discriminative validity for distinguishing between any cognitive disorder (MCI, AD, VD, and Mixed) and no cognitive disorder (healthy controls). Secondary analyses included examining each instruments’ discriminative validity for distinguishing between MCI and no cognitive disorder and between dementia (AD, VD, and Mixed) and no cognitive disorder.
Data analyses were performed using the Statistical Package for Social Sciences version 19.0 and Sigmaplot version 12.5.
Results
The groups did not differ with respect to age, sex and education level (all p values > 0.30), indicating they were well matched and therefore no patient characteristics were included as covariates in the following analyses.
Internal consistency reliability analyses showed a Cronbach's alpha of 0.75 with the ten section scores of the BCSE together. The removal of section scores would not result in a substantial increase or decrease of the internal consistency (Cronbach's alpha of 0.72–0.76). There was a small negative correlation between BCSE total score and age in healthy controls (r = −0.22, n = 96, p = 0.03). No significant correlations between BCSE total score and education level (r = 0.02, n = 96, p > 0.05), MMSE total score and age (r = −0.11, n = 96, p > 0.05), and MMSE total score and education level (r = 0.15; n = 96, p > 0.05) were present.
Whole-group analyses showed a high significant positive correlation between BCSE and MMSE total scores (r = 0.79), n = 183, p < 0.001). Moreover, correlations between BCSE and MMSE section scores with same constructs revealed significant positive correlations ranging from r = 0.28 to r = 0.73 (see Table 2).
Pearson correlation coefficients for BCSE and MMSE section scores
**p < 0.001.
Table 3 shows the results of the MMSE total score and BCSE total and section scores for patients with MCI and dementia and healthy controls. A one-way ANOVA with group (MCI, Dementia, Controls) as the independent variable and BCSE total score as the dependent variable revealed a main effect of group (F(2, 183) = 109.24 p < 0.001, ηp 2 = 0.54), with dementia patients (M 29.82, SD 10.4) performing worse than MCI patients and healthy controls (MCI: M 37.31 SD 9.7; Controls: M 50.46, SD 6.34) and MCI patients performing worse than healthy controls as analyzed with Hochberg's post-hoc test (all p < 0.001). In addition, a one-way ANOVA with group (MCI, Dementia, Controls) as the independent variable and MMSE total score as the dependent variable revealed a main effect of group (F(2,180) = 139.91, p < 0.001, ηp 2 = 0.61), with dementia patients (M 20.58, SD 3.94) performing worse than MCI patients and healthy controls (MCI: M 24.62, SD 2.9; Controls: M 28.42, SD 1.65) and MCI patients performing worse than healthy controls as analyzed with Hochberg's post-hoc test (all p < 0.001). A subsequent one-way MANOVA with group (MCI, Dementia, Controls) as the independent variable and 10 BCSE section scores as dependent variables revealed a significant main effect of group on all BCSE section scores, except for Inhibition Commissions (F(2,183) = 1.58, p = 0.21, ηp 2 = 0.02). Mean scores on all BCSE section scores were highest in the control group and lowest in the dementia group, except for Inhibition Time (see Table 3).
Mean and standard deviations of MMSE total score and BCSE total and section scores for patients and healthy controls
Note. Groups with the same denotation (a, b, c) differ significantly (p < 0.05).
Figure 1 shows the ROC curves of the BCSE and MMSE for detecting patients with cognitive disorders. ROC analysis of the BCSE and MMSE total scores as predictors of cognitive disorders (i.e. MCI, AD, VD, and Mixed dementia) showed an AUC of 0.92 (p < 0.001) and an AUC of 0.93 (p < 0.001), respectively. Comparing the MCI group with healthy controls resulted in an AUC of 0.87 (p < 0.001) for BCSE total score and an AUC of 0.87 (p < 0.001) for MMSE total score; comparing the dementia group with healthy controls resulted in an AUC of 0.96 (p < 0.001) and an AUC of 0.98 (p < 0.001) for the BCSE and the MMSE respectively. Finally, comparing MCI and dementia resulted in an AUC of 0.66 (p = 0.01) for BCSE total score and an AUC of 0.79 (p < 0.001) for MMSE total score. For the comparison between MCI and healthy controls, an optimal cut-off score of ≤ 46 with a sensitivity of 81%, specificity of 80%, PPV of 61%, NPV of 92%, DOR of 17.4, and positive likelihood ratio of 4.1 was found for the BCSE; a proposed cut-off score of ≤ 27 with a sensitivity of 84%, specificity of 76%, PPV of 57%, and NPV of 92% was found for the MMSE. For the second comparison between dementia and healthy controls, an optimal cut-off score of ≤ 42 with a sensitivity of 96%, specificity of 92%, PPV of 86%, NPV of 97%, DOR of 264, and positive likelihood ratio of 11.3 was established for the BCSE. With the widely used cut-off score of ≤ 24 (Tombaugh and McIntyre, Reference Tombaugh and McIntyre1992), the MMSE had a sensitivity of 84%, specificity of 96%, PPV of 91%, and NPV of 92%. Results concerning sensitivity, specificity and predictive values for the proposed BCSE cut-off scores for discriminating between both MCI and dementia and the healthy control group are reported in Table 4.
ROC curves for distinguishing patients with cognitive disorders from healthy controls.
Sensitivity, specificity, and predictive values for different BCSE cut-off scores for both MCI and dementia compared to healthy controls
Discussion
The present study aimed to examine the clinical applicability and validity of the Dutch version of the BCSE in a sample of older adults with MCI or dementia. Our results show that the BCSE is a valid screening instrument, with psychometric properties similar to the widely used MMSE. In fact, we found high compatibility between the BCSE and MMSE scores, suggesting good convergent validity. Also, the BCSE has a high degree of internal consistency (Cronbach's alpha = 0.75). The BCSE total score can be converted to a classified level (Average, Low Average, Borderline, Low, and Very Low) taking age and education level into account. However, in this study we found only a small correlation between age and BCSE total score, no correlation between age and MMSE total score and no correlations between education level and BCSE and MMSE total scores. These results question the importance of age and education adjusted norms. By contrast, other studies showed influences of age and education level on cognitive screeners (Cullen et al., Reference Cullen, O’Neill, Evans, Coen and Lawlor2007). Future studies may clarify whether the BCSE total score should be adjusted for age and education level.
Furthermore, the analysis of group differences indicates that both instruments are capable of differentiating between patients with MCI, patients with dementia and healthy controls. Moreover, nine out of ten BCSE section scores are capable of discriminating patients with dementia from healthy controls and eight out of ten section scores are capable of discriminating patients with MCI from healthy controls. Notably, Orientation, Time Estimation and Clock Drawing seem to be suitable section scores for discriminating between patients with MCI and dementia. These results are consistent with studies reporting that similar tasks are useful diagnostic tools to identify dementia (Heinik et al., Reference Heinik, Solomesh, Shein and Becker2002; Papagno et al., Reference Papagno, Allegra and Cardaci2004). The inclusion of tasks that rely on “frontal” executive functioning may contribute to the BCSE being more sensitive in the detection of non-AD dementia than the MMSE. It would be an interesting topic for future studies to evaluate separate samples with different types of dementia (e.g. Alzheimer's and vascular dementia).
Finally, the ROC analysis reveals that the BCSE has good overall discriminative validity in the detection of cognitive impairment (i.e. patients with MCI, AD, VD, and Mixed dementia: significant AUC of 0.92), similar to the MMSE (significant AUC of 0.93). Although the differentiation between cognitive impairment and no cognitive impairment is often the primary comparison when assessing the validity of cognitive screening instruments, it is also important to evaluate the test in a broader spectrum of a disorder. Therefore, we also examined the discriminative validity for the detection of MCI and mild to moderate dementia for both instruments. The BCSE optimal cut-off score for detecting dementia had higher sensitivity (0.95), but somewhat lower specificity (0.92) than the MMSE cut-off point of ≤ 24 (sensitivity = 0.81, specificity = 0.96). These results are in line with the values reported in the technical and interpretative manual of the English version of the WMS-IV (Wechsler, Reference Wechsler2009). The BCSE optimal cut-off score for detecting MCI had somewhat lower sensitivity (0.81) and somewhat higher specificity (0.80) than the proposed MMSE cut-off point (sensitivity = 0.84, specificity = 0.76). Though BCSE and MMSE optimal cut-off scores could be established for patients with MCI, it should be noted that there is still a risk of approximately 20% false positive results (i.e. diagnosing MCI) in healthy participants and false negative results (i.e. classifying normal functioning) in patients with MCI. Another clinically relevant outcome in the analyses of MCI patients are the relatively low PPVs (BCSE: 61%; MMSE: 57%). The PPV is the proportion of participants classified as having MCI with the BCSE who were actually diagnosed with MCI. The low PPVs correspond to a relatively high risk of false-positives, which indicates that a positive test result may not be useful to confirm the presence of MCI.
Several limitations of this study should be mentioned. The selected study sample was based on clinical diagnoses. Patients with comorbidities that could adversely affect the test performances, and patients with only subjective memory complaints but no objective memory deficits, were excluded. In future, it would be beneficial to evaluate the clinical validity of the BCSE in participants with subjective memory complaints and other patient groups. A further limitation is that all clinical diagnoses had not received long-term confirmation. Therefore, despite following carefully established international guidelines, these clinical diagnoses are not impervious to error.
Second, although the diagnosis was made clinically, the clinicians were not blind for the MMSE. Therefore, the MMSE may have been used in the diagnostic identification of patients which may have resulted in a review bias (Gifford and Cummings, Reference Gifford and Cummings1999). That is, participants who scored below the established MMSE cut-off score for MCI or dementia may have been more likely to be allocated to the associated patient groups. This may have falsely increased the accuracy of the MMSE (Gifford and Cummings, Reference Gifford and Cummings1999). In future research, it is advised to administer the BCSE and MMSE, or another cognitive screener, independent of the standard neuropsychological assessment and to blind the administrators from patients’ diagnosis. Moreover, further research is needed to explore inter- or intrarater reliabilities, content validity, and divergent validity of the BCSE and to replicate convergent validity using other screening instruments.
In conclusion, findings from the current study suggest that the Dutch version of the BCSE is a clinically valid screening test for the detection of cognitive impairment in patients with dementia. Its psychometric properties are similar to the widely used MMSE. However, for distinguishing older adults with MCI from healthy controls both the BCSE and MMSE have limitations. Furthermore, both the BCSE and MMSE are not capable of discriminating between patients with MCI and patients with dementia. It is important to note, that the BCSE has been developed as a screener to quickly identify patients who may benefit from more extensive neuropsychological assessment; it should not be used as a stand-alone diagnosis tool.
Conflict of interest
None.
Description of author's roles
All authors designed the study. ZB and MPHH supervised the data collection. ZB wrote the paper and was responsible for the statistical design of the study and carrying out the statistical analyses. MPHH, APA, and RPCK assisted with writing the paper.
Acknowledgments
We thank Pearson Assessment B.V., Amsterdam, The Netherlands, for authorizing and funding the development of the WMS-IV-NL. We thank Hanneke Hoeijmakers for her assistance in the data collection and Daniel Sheppard for the final grammar check of our manuscript. This study was funded by Pearson Assessment B.V. and Kempenhaeghe, expertise centre for epileptology, sleep, medicine, and neurocognition.
