Introduction
There is ample evidence that cognitive impairment is underrecognized and misdiagnosed among community-dwelling older adults (Callahan et al., Reference Callahan, Hendrie and Tierney1995; Valcour et al., Reference Valcour, Masaki, Curb and Blanchette2000; Boustani et al., Reference Boustani2005; Bradford et al., Reference Bradford, Kunik, Schultz, Williams and Singh2009). Misdiagnoses or delayed recognition of cognitive disorders can deprive patients of treatments when they are most effective, place barriers to accessing community support services, and contribute to caregiver stress and depression (Olazarán et al., Reference Olazarán2010; Reuben et al., Reference Reuben2010; Cordell et al., Reference Cordell2013).
Of course, the increasing longevity of older adults, especially those in the “old-old” category, may compound these problems, likely resulting in higher incidence and prevalence rates of cognitive impairment (Corrada et al., Reference Corrada, Brookmeyer, Paganini-Hill, Berlau and Kawas2010). In the US healthcare system, the importance of creating valid and accurate assessment tools is clear, and encouraging healthcare professionals to use them has become a priority for the Alzheimer's Association (Alzheimer's Association, 2012) and the Centers for Medicare and Medicaid Services (CMS). CMS, a federal agency within the US Department of Health and Human Services that covers approximately 50 million Americans, has mandated that the newly created Annual Wellness Visit (AWV) include cognitive assessment as part of the benefit. The AWV requires the provider to assess “an individual's cognitive function by direct observation, with due consideration of information obtained by way of patient report, concerns raised by family members, friends, caretakers, and others.” (Patient Protection and Affordable Care Act of 2010, 42 CFR. ×410.15(a). 2010). Two important goals of the screening process are to identify patients with possible cognitive problems and make appropriate referrals for more comprehensive cognitive evaluations (Cordell et al., Reference Cordell2013). For this to be successful, it is important that primary care providers and other healthcare professionals use screening tools that have adequate psychometric sensitivity and specificity.
The US Preventive Services Task Force addressed this in their 2003 statement regarding routine dementia screening (US Preventive Services Task Force [USPSTF], 2003), but they also pointed out that most available cognitive screening tools were psychometrically suboptimal. Specifically, the task force observed that while many cognitive screening tools demonstrated adequate sensitivity for dementia (i.e. the ability to detect dementia in persons who actually have it), far fewer demonstrated satisfactory specificity (i.e. the ability to correctly identify persons who do not have dementia). Since the USPSTF 2003 statement, newer and psychometrically stronger cognitive screening measures have been developed. At present, there is a lack of consensus about which cognitive screening tool represents the “gold standard,” although there are potentially strong contenders such as the Addenbrooke's Cognitive Examination-III (ACE-III; Hsieh et al., Reference Hsieh, Schubert, Hoon, Mioshi and Hodges2013) and the Montreal Cognitive Assessment (MoCA; Nasreddine et al., Reference Nasreddine2005).
Another cognitive screening tool that shows promise is the Brief Cognitive Assessment Tool (BCAT; Mansbach et al., Reference Mansbach, MacDougall and Rosenzweig2012), which was originally validated in a US-assisted living sample. The BCAT is a 21-item cognitive screening measure that can be individually administered in 10 to 15 min. Scores range from 0 to 50. Items were chosen to emphasize contextual memory, executive functioning, and attention. In fact, the BCAT contains a multi-level memory component (e.g. word lists and story recall items) and complex executive functions features (e.g. judgment, set-shifting, and problem-solving). In previous research, it displayed robust psychometric properties, with strong score reliability (Cronbach's α = 0.92), sensitivity (0.99), specificity (0.97), and an AUC of 0.95 for detecting MCI versus dementia (Mansbach et al., Reference Mansbach, MacDougall and Rosenzweig2012). The BCAT produces a total score suggesting specific cognitive stages, and it yields a Contextual Memory Factor score (CMF) and an Executive Control Functions Factor score (ECFF), which were also found to be significant predictors of cognitive diagnosis and instrumental activities of daily living (IADLs).
The current BCAT cross-validation study required a measure to enable categorization of participants into “cognitively impaired” and “cognitively intact” categories, for which the MoCA (Nasreddine et al., Reference Nasreddine2005) was selected. The MoCA was designed to detect Mild Cognitive Impairment (MCI) in patients who were previously identified as performing in the normal range on the Mini-Mental State Examination (MMSE; Folstein et al., Reference Folstein, Folstein and McHugh1975). The MoCA assesses several cognitive domains including memory, attention, executive control, language, orientation, and visuospatial abilities. Scores range from 0 to 30. In a US community clinic and academic center, MoCA scores demonstrated adequate test–retest reliability (r = 0.92; average time between tests exceeding a month), internal consistency (Cronbach's α = 0.83), and convergent validity via a significant correlation with the MMSE (r = 0.87; Nasreddine et al.). Using a cut score of 26, Nasreddine et al. found the MoCA to have a sensitivity of 0.90, a specificity of 0.87, and an area under the ROC curve (AUC) of .92 for detecting MCI versus dementia. Luis et al. (Reference Luis, Keegan and Mullan2009) cross-validated the MoCA in a community dwelling older adult sample and found that the MoCA detected 97% of those with cognitive impairment at a cut score of 26.
The current study also required an investigation of the construct validity (convergent and discriminant validity) of the BCAT, for which the Geriatric Depression Scale (GDS; Yesavage et al., Reference Yesavage1983), the Neuropsychological Assessment Battery-Judgment subtest (NAB-JDG; Stern and White, Reference Stern and White2003), and the AD8 (Galvin et al., Reference Galvin2005) were selected. The 30-item GDS is likely the most commonly used scale to screen for depression in older adults. Scores range from 0 to 30, with higher scores indicating a greater likelihood of depression. Stiles and McGarrahan (Reference Stiles and McGarrahan1998) reviewed a decade of published research and found the internal consistency and test–retest reliability of GDS scores to be adequate. In a more recent psychometric analysis of GDS scores in 417 medical outpatients, Cronbach's alpha was .89 (Lopez et al., Reference Lopez, Quan and Carvajal2010). Wancata et al. (Reference Wancata, Alexandrowicz, Marquart, Weiss and Friedrich2006) systematically reviewed the screening accuracy of GDS scores across 42 studies and noted sensitivity to be 0.75 and specificity to be 0.77.
The NAB (Stern and White, Reference Stern and White2003) is a comprehensive battery of neuropsychological tests nationally standardized on over 1400 US adults aged 17–97. The 10-item judgment subtest (NAB-JDG), which is included in the Executive Functions module of the NAB, was designed to assess, “judgment as it pertains to critical aspects of independence in daily living. . . includes a series of questions about home safety, health, and medical issues likely to be encountered in everyday life” (p. 30). Scores on the NAB-JDG range from 0 to 20. Thirty protocols were randomly selected for interrater reliability analysis, resulting in an intra-class correlation coefficient of 0.85. Using a confirmatory factor analysis, NAB-JDG scores behaved as predicted, loading only on the Executive Function factor. A subset of 50 participants from the standardization sample completed multiple neuropsychological tests other than the NAB to demonstrate convergent and discriminant validity (White and Stern, Reference White and Stern2003).
The AD8 (Galvin et al., Reference Galvin2005) was designed to be a brief informant-based measure to differentiate individuals with and without dementia and is sensitive in detecting early signs of dementia. The eight-item screening interview was designed to assess memory, orientation, judgment, and function. Scores range from 0 to 8, with scores of 2 or higher predicting a likelihood of dementia in a sample of older adults with a 38% base rate of very mild dementia (i.e. CDR = 0.5). Galvin et al. (Reference Galvin, Roe, Xiong and Morris2006) investigated the psychometric properties of the scale through 225 patient-informant dyads, finding interrater reliability (ICC = 0.82; 95% CI 0.5 to 0.92), intrarater reliability (weighted kappa = 0.67, 95% CI 0.59 to 0.75), and construct validity through correlations of individual AD8 items and other neuropsychological tests. In the absence of an informant, Galvin et al. (Reference Galvin, Roe, Coats and Morris2007) demonstrated that a self-reported AD8 can differentiate individuals without dementia from individuals with dementia, albeit with greater utility in mildly impaired individuals.
The primary aim of this study was to confirm the diagnostic screening utility of the BCAT. To this end, we cross-validated the BCAT in a community-dwelling sample of older adults. In addition, we recommended score ranges for the CMF and ECFF to assist clinicians in identifying patients who likely have memory impairment or executive dysfunction. In so doing, we hope to better equip health care providers by improving the diagnostic accuracy and utility of their cognitive screening assessments.
Methods
Participants
We prospectively recruited 78 community dwelling participants from churches and senior centers in central Maryland. We chose to exclude only those individuals with any medical conditions or psychiatric symptoms that would prevent them from completing the questionnaires and cognitive tests (e.g. severe arthritis, global aphasia; significant vision, or hearing deficits). Three participants were deleted from the analyses due to incomplete BCAT data, leaving a final sample of 75. We used a conservative MoCA cut score of <26 to categorize participants as “with” or “without” cognitive impairment (Nasraddine et al., Reference Nasreddine2005; Luis et al., Reference Luis, Keegan and Mullan2009; Rossetti et al., Reference Rossetti, Lacritz, Cullum and Weiner2011).
Procedures
All procedures were approved by Hood College's institutional review board, and participants were not reimbursed for their participation. Participants reviewed and signed an informed consent document before completing a short battery of several individually administered cognitive tests, including the BCAT, the MoCA, the GDS, the NAB-JDG, and the AD8. One licensed psychologist and several trained research assistants collected all of the data for this study.
Statistical analyses
Descriptive and inferential statistical analyses were used to examine the demographic characteristics of the sample and to investigate the reliability and validity of the BCAT. In addition, a ROC curve analysis was used to investigate the diagnostic utility of the BCAT in this community-dwelling sample of older adults.
Results
Participants ranged in age from 50 to 92, with a mean age of 70.39 (±9.05). Seventy-three percent of the participants were female, 88% were Caucasian, and 49% were married. Years of education ranged from seven to 20, with an average of 14.55 (±3.12) (Table 1). Using a very conservative MoCA cut score of <26 for cognitive impairment (Nasraddine et al., Reference Nasreddine2005; Luis, et al., Reference Luis, Keegan and Mullan2009; Rossetti et al., Reference Rossetti, Lacritz, Cullum and Weiner2011), 35% (n = 26) of the sample showed evidence of cognitive impairment, while 65% (n = 49) were categorized as having “normal” cognitive functioning. Thus, the base rate of cognitive impairment in this community-dwelling sample was 35%.
Demographic characteristics of participants (N = 75)
There were no significant differences between participants with cognitive impairment (MoCA < 26; n = 26) and those without cognitive impairment (MoCA ≥ 26; n = 49) for age, sex, or education (Table 2). Likewise, the BCAT was not significantly related to age, sex, or education in this sample.
Age, education, and sex by diagnostic category
Note: Cognitive impairment is operationalized as a score < 26 on the Montreal Cognitive Assessment (MoCA)
Reliability
In this community-dwelling sample, the internal consistency reliability of BCAT scores, as estimated by Cronbach's α based on standardized items, was 0.76. This is lower than the Cronbach's alpha of 0.92 for BCAT scores reported in the original BCAT development and validation study with an assisted-living sample (Mansbach et al., Reference Mansbach, MacDougall and Rosenzweig2012).
In the current study, the MoCA had an internal consistency score reliability coefficient of 0.63 (Cronbach's alpha based on standardized items), and the GDS had an internal consistency score reliability of 0.84 (Cronbach's alpha). In addition, the NAB-JDG had an internal consistency reliability of 0.63 (Cronbach's alpha based on standardized items), and scores from a self-report version of the AD8 had an internal consistency reliability of α = 0.62.
Validity
Evidence for the convergent validity of BCAT score inferences was consistent with that found in the original development and validation studies, as the BCAT was significantly correlated with the MoCA (rs = 0.68, p < 0.001), and with the AD8 (rs = −0.26, p = 0.02). Likewise, discriminant validity was demonstrated by the absence of a relationship between the GDS and the BCAT (rs = −0.13). Additional evidence of construct validity was demonstrated by the significant correlation of the BCAT with a measure of judgment (NAB-JDG; rs = 0.55, p < 0.001). Finally, participants who exhibited cognitive impairment on the MoCA scored significantly lower on the BCAT (t(31.11) = 6.10, p < 0.001, Cohen's d = 1.62), on the BCAT Memory Factor (t(32.86) = 4.75, p < 0.001, Cohen's d = 1.25), and on the BCAT Executive Functions Factor (t(36) = 2.69, p = 0.01, Cohen's d = 0.70) than participants who were cognitively intact on the MoCA. BCAT, BCAT Memory Factor, and BCAT Executive Functions Factor scores by diagnostic category are presented in Table 3.
Descriptive statistics by diagnostic category for the BCAT, the BCAT memory factor, the BCAT executive functions factor, and validity measures
Note: BCAT (Brief Cognitive Assessment Tool); BCAT CMF (BCAT Contextual Memory Factor); BCAT ECFF (BCAT Executive Functions Factor); GDS (Geriatric Depression Scale); NAB-JDG (Neuropsychological Assessment Battery – Judgment subtest).
A receiver operating characteristic (ROC) curve was calculated from sensitivity and 1-specificity values for diagnoses of cognitive impairment versus no cognitive impairment. The area under the ROC curve (AUC) for the BCAT was 0.90, p < 0.001, 95% CI [0.83, 0.97] (Figure 1). A BCAT cut-score of 45 (scores below 45 suggesting cognitive impairment) resulted in the best balance between sensitivity (0.81) and specificity (0.80). The negative predictive value (NPV) indicates an 89% likelihood of correctly identifying those without cognitive impairment in this community-dwelling sample.
Predictive utility of BCAT scores for cognitive impairment.
Discussion
The results of this study with community-dwelling older adults confirm and cross-validate the original BCAT study findings. In the present study, adequate internal consistency and strong evidence of construct validity were found. Furthermore, predictive utility was established with analyses of sensitivity, specificity, positive predictive value, and NPV for the total BCAT score. From a clinical perspective, we recommend a cut-score of 45 for differentiating cognitively normal older adults from those who have probable cognitive impairment. That is, older adults with total BCAT scores of 45 and higher are likely to have normal cognitive functioning; whereas, those with scores of 44 and lower are likely to have cognitive impairment.
In the original BCAT study, a CMF and ECFF were established through a principal component analysis. Part of the design of the BCAT was to include complex memory and executive functions features, since impairments in these areas are key variables in establishing cognitive diagnoses. In addition, there is evidence that cognitive screening instruments that have broad-based memory items (Zahodne et al., Reference Zahodne2011) and a broadly complex executive functions component (Mansbach et al., Reference Mansbach, MacDougall and Rosenzweig2012) are related to cognitive stage and predictive of functional ability (i.e. IADLs).
In the present study, we build upon the original PCA analysis and report scores that can assist clinicians in identifying older adults with probable memory deficits and executive dysfunction. Based on participants whose BCAT scores were in the normal range, we suggest that CMF scores of 13 and below indicate possible memory impairment (score range 0 to 15), and ECFF scores of 5 and below indicate possible executive dysfunction (score range 0 to 7). Both of these values were determined by using the relatively conservative calculation of one standard deviation below the mean for the CMF and ECFF. The clinically accepted standard for identifying impairment on neuropsychological tests is one to two standard deviations below the mean. Although using one standard deviation below the mean as an indication of possible memory or executive function impairment may result in a number of false positive errors, we support this threshold for cognitive screening factors such as the CMF and ECFF, as we encourage clinicians to refer such patients for more comprehensive cognitive evaluation.
One surprising finding from this study was the relatively high rate of cognitively impaired participants in the sample. Approximately 35% of study participants tested in the mildly impaired range (based on the reported MoCA score range for MCI). This is much higher than what one would expect, as the prevalence of MCI is generally estimated to be approximately 15 percent of those age 70 and older (Petersen et al., Reference Petersen2009). One possible explanation is that 65% of our study participants resided in subsidized housing for independent-living older adults (age 62 and older). Previous research has demonstrated that the prevalence of cognitive impairment among older adults living in subsidized housing may be higher than among other community-dwelling older adults. For example, Simning et al. (Reference Simning, Conwell and van Wijngaarden2013) reported that 27% of their public housing sample screened positively for some type of cognitive impairment. In addition, in our attempt to ensure the purest possible sample of cognitively intact participants, we used an extremely conservative MoCA cut score to categorize participants as impaired versus not impaired. This may have pushed some participants with normal cognitive functioning out of the normal range for purposes of this study.
A limitation of this study concerns the demographics of the sample. Participants were relatively young older adults, largely Caucasian (88%), well educated (mean was 14 years of education), and married (49%). Therefore, one should be cautious in generalizing to older, less educated, non-white, and unmarried older adults. In addition, drawing participants from the community invariably leads to more homogeneous cognitive test scores, as a higher percentage of community-dwelling older adults exhibit normal cognitive functioning compared to those who live in institutional settings. This restriction of range of cognitive test scores, in turn, likely contributed to a smaller BCAT reliability estimate than was found in the original BCAT development study (Raykov and Marcoulides, Reference Raykov and Marcoulides2011). Finally, the standard for determining normal cognition was based on the MoCA, which is a screening instrument, and not determined by a more comprehensive evaluation (e.g. neuropsychological evaluation). In the original BCAT study, a comprehensive neuropsychological battery determined cognitive diagnoses.
For this cross-validation study, we chose to focus on differentiating between normal cognition and any cognitive impairment. In our view, the first priority of a cognitive screening tool is to detect any impairment so that a referral for more comprehensive evaluation can be made. It would be useful for subsequent research to investigate and confirm in a community sample the cognitive stages reported in the original BCAT study. In particular, cross-validating score ranges differentiating those with MCI from dementia would be clinically beneficial.
Conflict of interest
None.
Description of authors’ roles
E. MacDougall designed the study, supervised a portion of the data collection, analyzed the data, wrote the majority of the Methods and Results sections, and collaborated with W. Mansbach in writing the Discussion section. W. Mansbach assisted with the design of the study, supervised a portion of the data collection, wrote the Introduction section, and collaborated with E. MacDougall in writing the Discussion Section. K. Clark supervised data collection, collected data, and assisted with reviewing/revising the manuscript. R. Mace collected data and assisted with writing the manuscript. All authors read and approved the final paper.
