Introduction
The Addenbrooke's Cognitive Examination (ACE) is a brief cognitive test battery which was first published in 2000 (Mathuranath et al., Reference Mathuranath, Nestor, Berrios, Rakowicz and Hodges2000) as a theoretically motivated improvement over the Mini-Mental State Examination (MMSE) (Folstein et al., Reference Folstein, Folstein and McHugh1975). It aimed to address some of the recognized neuropsychological deficiencies of the MMSE by adding more tests of memory and visuospatial abilities and introducing tests specifically examining executive function. Although the ACE was widely and rapidly accepted in clinical practice, certain weaknesses were identified, prompting the development of the Addenbrooke's Cognitive Examination-Revised (ACE-R) in 2006 (Mioshi et al., Reference Mioshi, Dawson, Mitchell, Arnold and Hodges2006). The item content of ACE and ACE-R is compared with the MMSE in Table 1.
Item content of ACE, ACE-R, and MMSE
There have been a significant number of publications examining the accuracy of the ACE and ACE-R in various patient populations using both the original instruments and translations into a variety of languages (Davies and Larner, Reference Davies, Larner and Larner2013). Results from a National Dementia Register suggested that ACE-R items improved MMSE estimates of cognitive ability by 16% (Law et al., Reference Law2013). A systematic review of ACE/ACE-R studies identified nine of 45 studies as suitable for review, excluding translated versions, and concluded that ACE/ACE-R was capable of differentiating between those with and without cognitive impairment, but the evidence base on distinguishing dementia subtypes and mild cognitive impairment (MCI) was lacking (Crawford et al., Reference Crawford, Whitnall, Robertson and Evans2012). However, to date no meta-analysis of ACE/ACE-R studies has been published to our knowledge.
The aim of this study was to identify and quantify robust studies reporting the diagnostic accuracy of ACE and ACE-R in relation to dementia versus non-demented subjects.
Methods
Methods were similar to those used previously for meta-analysis of MMSE dementia studies (Mitchell, Reference Mitchell2009, Reference Mitchell and Larner2013). Guidelines for the performance and reporting of meta-analyses in the PRISMA statement (Liberati et al., Reference Liberati2009; Moher et al., Reference Moher2009) were observed.
Search criteria
A systematic literature search was undertaken using the following abstract databases: Medline 1966-May 2013, PsycINFO 1887-May 2013, and Embase 1980-May 2013. Five full text collections (Science Direct, Ingenta Select, Ovid Full Text, Blackwell Online and Wiley Interscience) were searched. The abstract database Web of Knowledge (4.0, ISI) was searched. The following search terms were used: “Screen* or test or instrument or measure or tool or diagnose* or Addenbrooke's Cognitive Examination or ACE or ACE-R” and “dementia or Alzheimer* or cognitive” and “sensitivity and specificity or accuracy or cut-off or receiver operator or ROC or Youden.”
Inclusion criteria
Studies were included that examined diagnostic validity of ACE or ACE-R in comparison to a validated diagnostic standard of either dementia or MCI (see Tables 2 and 3 for details).
Diagnostic validity of ACE in diagnosing dementia
Diagnostic validity of ACE-R in diagnosing dementia
Exclusion criteria
Studies were excluded that did not present adequate data for inclusion in the analysis. Based on previous sample size calculations, studies that recruited less than 160 patients (80 per group) were excluded to avoid studies with unreliable false negative or false positive results (Mitchell, Reference Mitchell2009). No exclusion was made solely on the basis of language of ACE/ACE-R publication (Crawford et al., Reference Crawford, Whitnall, Robertson and Evans2012), as these instruments have been translated into many languages (Davies and Larner, Reference Davies, Larner and Larner2013).
Analysis
Studies identified were subjected to critical appraisal and a meta-analysis of suitable studies was conducted, to give overall test accuracy, sensitivity, specificity, combined Youden score, positive and negative predictive values (PPV, NPV), positive and negative likelihood ratios (LR+, LR-), and positive and negative clinical utility index (CUI+, CUI-) (Mitchell, Reference Mitchell2009). The interpretation of the clinical utility index is 0.93–1.00 near perfect value; 0.81–0.92 excellent; 0.64–0.80 good; 0.49–0.63 adequate; 0.36–0.48 poor; and < 0.36 very poor. Publication bias was tested by Harbord method (Harbord et al., Reference Harbord, Egger and Sterne2006). Comparative accuracy was tested by conducting a relative risk comparison of pooled sensitivity and specificity and by comparing overall accuracy at equivalent prevalence rates of 25% and 50%.
Results
Search results
The literature search identified 61 possible papers on ACE and ACE-R in all (full list obtainable from corresponding author; see Figure 1 for PRISMA flowchart). Of these, 29 were reports of studies of the ACE, 13 using the English version and 16 using non-English versions. There were 32 reports of studies of the ACE-R, 17 using the English version and 15 using non-English versions. After application of exclusion criteria, ten studies were suitable for meta-analysis, five for ACE (Mathuranath et al., Reference Mathuranath, Nestor, Berrios, Rakowicz and Hodges2000; Garcia-Caballero et al., Reference Garcia-Caballero2006; Larner, Reference Larner2007; Stokholm et al., Reference Stokholm, Vogel, Johannsen and Waldemar2009; Yoshida et al., Reference Yoshida2011; Table 2), and five for ACE-R (Mioshi et al., Reference Mioshi, Dawson, Mitchell, Arnold and Hodges2006; Alexopoulos et al., Reference Alexopoulos2010; Dos Santos Kawata et al., Reference Dos Santos Kawata2012; Yoshida et al., Reference Yoshida2012; Larner, Reference Larner2013a; Table 3).
Flow diagram of search process (after PRISMA; Moher et al., Reference Moher2009).
All the studies identified were from high prevalence specialist secondary care settings. No study was identified from a low prevalence primary care or community setting. Most were experimental studies, applying the test to groups of patients with known diagnoses. Pragmatic studies, examining all clinic attenders before diagnosis was established, and hence reflecting the idiom of clinical practice, were in the minority.
Of the studies that examined ACE/ACE-R accuracy in relation to MCI, none identified was deemed suitable according to specified inclusion/exclusion criteria.
Criterion (gold) standard
The most common reference standard for dementia diagnosis was the Diagnostic and Statistical Manual (DSM) in all studies (usually version IV), with additional criteria for dementia subtypes.
Cut-off threshold
The original studies of ACE and ACE-R cited two different cut-offs, 88/100 and 83/100 (Mathuranath et al., Reference Mathuranath, Nestor, Berrios, Rakowicz and Hodges2000), and 88/100 and 82/100 (Mioshi et al., Reference Mioshi, Dawson, Mitchell, Arnold and Hodges2006), respectively. These cut-offs have been used in some of the subsequent studies, but not all, as different authors have examined different cut-offs according to the educational profile of their cohort (e.g. Garcia-Caballero et al., Reference Garcia-Caballero2006) or selecting the cut-off corresponding to the best overall accuracy of the test (Larner, Reference Larner2007, Reference Larner2013a). For the purposes of the pooled analysis, where studies had used more than one cut-off point the most sensitive was selected.
The accuracy of ACE in identifying dementia versus subjects without dementia
Across the 5 included studies there were 529 cases of dementia out of a population of 1,090, a prevalence of 49%. There was no evidence of publication bias (Harbord bias = –8.23, 95% CI = –29.1 to 12.6, p = 0.37).
Pooling the raw data from these studies demonstrated that 512 out of 529 cases were correctly identified using the ACE, giving a pooled Se of 96.8%. On meta-analytic weighting this was corrected to 96.9% (95% CI = 92.7% to 99.4%). Non-cases (377) were correctly ruled out from a sample of 561 comparison subjects to give a pooled Sp of 67.2%. On meta-analysis, this was corrected to 77.4% (95% CI = 58.3% to 91.8%). Unadjusted the PPV was therefore 74.7% and the NPV 95.5%.
The accuracy of ACE-R in identifying dementia versus subjects without dementia
Across the 5 included studies there were 560 cases of dementia out of a population of 1,156, a prevalence of 48%. Harbord bias was not significant (0.097, 95% CI = −18.95 to 19.14, p = 0.99).
Pooling the raw data from these studies demonstrated that 514 out of 560 cases were correctly identified using the ACE-R, giving a pooled Se of 91.8%. This was adjusted on meta-analytic weighting to 95.7% (95% CI = 92.2% to 98.2%). Non-cases (383) were correctly ruled out from a sample of 596 comparison subjects to give a pooled Sp of 64.3%. This was corrected on meta-analysis to 87.5% (95% CI = 63.8% to 99.4%). Unadjusted the PPV was therefore 70.7% and the NPV 89.3%.
Comparison of ACE and ACE-R with MMSE at given cut-offs
Five studies compared the ACE with the MMSE. For the MMSE, Harbord bias was 7.18 (95% CI = –24.77 to 39.14, p = 0.59). At a cut-off of <27 the adjusted meta-analytic sensitivity was 89.9% (95% CI = 81.9% to 95.7%). Specificity was 81.7% (95% CI = 71.6% to 90.0%). A head-to-head relative risk analysis suggested that the ACE was more accurate than the MMSE in terms of sensitivity (RRsens = 1.07, 95% CI = 1.00 to 1.14, χ2 = 4.3, p = 0.0383) but no different in terms of specificity (RRspec = 0.92, 95% CI = 0.75 to 1.13, χ2 = 0.62, p = 0.429).
Four studies compared the ACE-R with the MMSE. For the MMSE Harbord bias was –20 (95% CI = –115.6 to 75.6, p = 0.55). At mixed cut-off points, the adjusted meta-analytic sensitivity was 94.0% (95% CI = 81.0% to 99.8%). Specificity was 92.1% (95% CI = 86.3% to 96.4%). A head-to-head relative risk analysis suggested that the ACE-R was no more accurate than the MMSE in terms of sensitivity (RRsens = 1.05, 95% CI = 0.92 to 1.18, χ2 = 0.51, p = 0.470) and also no different in terms of specificity (RRspec = 0.87, 95% CI = 0.64 to 1.18, χ2 = 0.82, p = 0.364).
Combining all nine studies using the MMSE against the ACE or ACE-R generated a pooled sensitivity of 92.0% (95% CI = 84.9% to 96.8%) and specificity of 86.9% (95% CI = 80.5% to 92.1%) for the MMSE.
Comparison of ACE, ACE-R and MMSE at equivalent prevalence
Table 4 shows a direct comparison of the ACE, ACE-R, and MMSE at equivalent prevalence rates of 25% and 50%.
Comparison of ACE, ACE-R, and MMSE at difference prevalence (% with 95% confidence intervals)
* based on comparison data with ACE (n = 5)
** TP+TN/TP+TN+FP+FN
At 25% the overall accuracy of the ACE would be 0.823, the ACE-R 0.895, and the MMSE 0.882 (0.858 in the five studies compared with ACE alone). At 25% prevalence the PPV and NPV of the ACE would be 58.9% and 98.8% compared with 71.9% and 98.3% for the ACE-R, and 70.1% and 97.0% for the MMSE.
At 50% prevalence, the overall accuracy of the ACE would be 0.872, that of the ACE-R 0.916 and the MMSE 0.895 (0.838 in the four studies compared with ACE-R alone). The PPV and NPV of the ACE would be 81.1% and 96.1% compared with 88.5% and 95.3% for the ACE-R, and 87.5% and 91.6% for the MMSE.
According to the clinical utility index (see www.clinicalutility.co.uk), the ACE-R would have good utility at 25% prevalence and excellent properties at 50% prevalence, while the ACE would be qualitatively rated as good only at 50% prevalence. Figure 2 shows the accuracy at all possible prevalences and for rule-in (case-finding) and rule-out (screening) purposes separately. Overall the ACE-R and MMSE are almost equivalent in terms of rule-in accuracy but the ACE is somewhat inferior. The ACE and ACE-R have approximately the same rule-out ability whilst the MMSE is somewhat inferior. When using one tool for both ruling-in cases and ruling-out non-cases (as is common practice), then only the ACE-R is recommended.
Conditional probability plot of PPV and NPV at fixed sensitivity and specificity.
Discussion
In this meta-analysis, we examined the diagnostic accuracy of the ACE and ACE-R. Of 61 studies identified, only 10 (= 16.4%) fulfilled inclusion criteria; in a similar meta-analysis of MMSE diagnostic accuracy, of 775 studies identified, 34 (= 4.4%) fulfilled inclusion criteria (Mitchell, Reference Mitchell2009). Studies included here varied in terms of subjects (i.e. non-demented groups variously encompassed healthy controls, those with subjective memory impairment, and those with cognitive impairment without dementia) and test cut-offs used.
Study limitations included the small number of studies suitable for meta-analysis and the possibility of bias in these studies. Since the ACE and ACE-R may presently be superseded by ACE-III (Hsieh et al., Reference Hsieh, Schubert, Hoon, Mioshi and Hodges2013), it would seem unlikely that further large diagnostic accuracy studies of ACE and ACE-R suitable for inclusion in a meta-analysis will be forthcoming. As regards bias, use of normal control groups in some studies may have inflated test metrics. Although some studies clearly used non-overlapping patient cohorts (e.g. Larner Reference Larner2007, Reference Larner2013a), it is possible that the same participants were included in some studies (Yoshida et al. Reference Yoshida2011, Reference Yoshida2012). Data from different translations of the ACE/ACE-R used to test patients from different cultures may also be problematic. No studies were identified to permit analysis of ACE/ACE-R diagnostic accuracy for MCI.
From the ten valid studies using these tools (five each for ACE and ACE-R) to identify dementia compared with mixed subjects, both ACE and ACE-R were highly sensitive (> 95%), more so than simultaneous MMSE (< 90%). In terms of specificity, ACE-R was best (87.5%), followed by MMSE (81.7%) and ACE (77.4%). In a modest prevalence setting, such as primary care or general hospital settings, where the prevalence of dementia may be approximately 25%, the overall accuracy of the ACE would be lower than that of the MMSE and the ACE-R. While both ACE and ACE-R would have adequate rule-out performance (see Figure 2), the PPV of the ACE would be less than that of the ACE-R (58.9% compared with 71.9%). Thus the ACE cannot be recommended in low prevalence settings, rather the ACE-R is preferred.
In high prevalence settings such as memory clinics, where dementia prevalence may be 50% or higher, both the ACE and ACE-R can be used but again the ACE-R is preferable. In everyday terms, of 20 above threshold scorers on the ACE-R in a high prevalence setting only one person would have dementia (false negative); of ten under threshold scorers, only one would not have dementia (false positive).
The results for the MMSE in specialist settings may be compared with those found in a larger meta-analysis of MMSE diagnostic accuracy studies for dementia (n = 26) which showed sensitivity and specificity of 76.9% (95% CI = 70.1% to 83.1%) and 89.9% (95% CI = 82.5% to 95.4%) (Mitchell, Reference Mitchell and Larner2013). Results of the current study suggest much improved sensitivity for the MMSE (92.0% vs 76.9%), the interpretation of which is uncertain but might possibly reflect improved accuracy of the MMSE in recent studies, perhaps related to case selection in these studies. The close performance of the MMSE and ACE/ACE-R may not be surprising given that the ACE and ACE-R incorporate all of the MMSE stem questions (Table 1), unlike the ACE-III (Hsieh et al., Reference Hsieh, Schubert, Hoon, Mioshi and Hodges2013).
Results from a National Dementia Register suggested that ACE-R items improved MMSE estimates of cognitive ability by 16% (Law et al., Reference Law2013). Tests may be compared using the area under the curve of receiver operating characteristic (ROC) curves (AUC), but this method has been criticized in favor of weighted comparisons (Mallett et al., Reference Mallett, Halligan, Thompson, Collins and Altman2012). Such a weighted comparison of pragmatic studies suggested a net benefit for ACE-R versus MMSE and an equivalent increase in true positive patients of 121 per 1,000 tested (Larner, Reference Larner2013b). Clearly the MMSE is briefer and quicker to administer than ACE or ACE-R; there is inevitably a trade-off between time and accuracy with cognitive screening instruments, and so MMSE might be suitable for use in time-limited situations, particularly as a rule-out, initial screen.
ACE-III has only recently been published (Hsieh et al., Reference Hsieh, Schubert, Hoon, Mioshi and Hodges2013) and is yet to be robustly validated. At the current time, the evidence of this meta-analysis favors the ACE-R over ACE and MMSE in the identification of dementia.
Conflict of interest
None.
Description of authors’ roles
AJ Mitchell conceived and planned the study. Both authors abstracted the data, evaluated publications against the listed criteria, and resolved any disagreements about inclusion/exclusion through discussion. AJ Mitchell analyzed the data. AJ Larner drafted the manuscript. Both authors revised the manuscript for intellectual content.
