Introduction
Short cognitive screening instruments (CSIs) that can reliably detect dementia and mild cognitive impairment (MCI) are increasingly required to assess patients presenting to clinical services with memory complaints (Larner, Reference Larner2017). There are many brief CSIs described in the literature, differing in their exact item content in terms of neuropsychological domains covered, and administration time, ranging from about 1 to 20 minutes. The choice of which CSI to use in practice may be influenced not only by diagnostic efficacy, measured by parameters such as sensitivity and specificity, but also by subjective clinician preference. Well-established CSIs in widespread use include the Montreal Cognitive Assessment (MoCA) (Nasreddine et al., Reference Nasreddine2005) and the sequential iterations of Addenbrooke's Cognitive Examination (Hodges and Larner, Reference Hodges, Larner and Larner2017), a short version of which, Mini-Addenbrooke's Cognitive Examination (MACE) (Hsieh et al., Reference Hsieh2015), is the most recently described.
Many studies of the MoCA have been reported (Julayanont and Nasreddine, Reference Julayanont, Nasreddine and Larner2017), firmly establishing its place in the repertoire of CSIs in common clinical usage. Meta-analyses have confirmed the diagnostic utility of MoCA for dementia and for Alzheimer's disease (AD) (Davis et al., Reference Davis, Creavin, Yip, Noel-Storr, Brayne and Cullum2015; Tsoi et al., Reference Tsoi, Chan, Hirai, Wong and Kwok2015). For example, a meta-analysis of 20 MoCA studies found pooled sensitivity and specificity of 0.91 and 0.81, respectively, for the diagnosis of dementia. Nine of these studies also looked at MCI diagnosis, with pooled sensitivity and specificity of 0.89 and 0.75, respectively (Tsoi et al., Reference Tsoi, Chan, Hirai, Wong and Kwok2015). MoCA was specifically designed to detect MCI in part because of the low sensitivity of the Mini-Mental State Examination (MMSE) (Folstein et al., Reference Folstein, Folstein and McHugh1975) for MCI diagnosis (Arevalo-Rodriguez et al., Reference Arevalo-Rodriguez2015; Mitchell, Reference Mitchell and Larner2017). Various studies indicate the superiority of MoCA to MMSE, particularly in identifying earlier stages of cognitive decline (Roalf et al., Reference Roalf, Moberg, Xie, Wolk, Moelter and Arnold2013; Trzepacz et al., Reference Trzepacz, Hochstetler, Wang, Walker and Saykin2015). Effective identification of MCI, a possible prodrome of dementia, may have important therapeutic implications should disease-modifying therapies become available.
In contrast to MoCA, few studies of the MACE have been reported to date (Hsieh et al., Reference Hsieh2015; Larner Reference Larner2015a; 2015Reference Larnerb; Matias-Guiu and Fernandez de Bobadilla, Reference Matias-Guiu and Fernandez de Bobadilla2016). These suggest that MACE has screening utility for dementia and MCI which, based on the analysis of historical cohorts, may be comparable to MoCA in terms of metrics such as effect size (Cohen's d; Larner, Reference Larner2016a) and Q* index (Larner, Reference Larner2015c). However, meaningful comparison of CSIs should ideally be based on head-to-head studies, not on historical patient cohorts.
The aim of the current study was to compare directly MACE and MoCA (see Box 1 for item content of tests) in a pragmatic diagnostic test accuracy study (Larner, Reference Larner2015d). This was performed over a 12-month period and looked at a variety of measures of discrimination as well as global measures of test utility, including effect sizes and weighted comparison. Hence, this was a much more extensive analysis than that presented in a preliminary report of 6-month data that was confined to weighted comparison of the tests (Larner, Reference Larner2016b).
Item content of MACE and MoCA
Methods
Consecutive new patient referrals were recruited prospectively over a 12-month period (June 2015–May 2016 inclusive) from the Cognitive Function Clinic at a regional neuroscience center. There were no specific inclusion/exclusion criteria such that all referrals, other than those with a pre-existing diagnosis of dementia, were included, irrespective of age.
Patients were invited to complete the MACE (score range 0–30, impaired to normal; Hsieh et al., Reference Hsieh2015) and the MoCA (score range 0–30, impaired to normal; Nasreddine et al., Reference Nasreddine2005). Tests were administered sequentially in counter-balanced order to avoid bias by the same raters who had prior experience of use of these tests (Larner Reference Larner2012; Reference Larner2015a; 2015Reference Larnerb).
CSI administration was performed independent of, but on the same day as, patient assessment by means of semi-structured interview, collateral history where available, neuroimaging (CT +/– MRI), and neuropsychological assessment (Wechsler Adult Intelligence Scale Revised, National Adult Reading Test, Wechsler Memory Scale-III, Graded Naming Test, Rey-Osterrieth Complex Figure, Stroop colour-word test, and Hospital Anxiety and Depression Score).
Standard diagnostic criteria for dementia (DSM-IV) and MCI (Petersen) were used, as previously in this clinic. Criterion diagnosis of dementia, MCI, or subjective memory complaint (SMC), was by judgment of an experienced clinician based on diagnostic criteria but did not use MACE or MoCA score, in order to avoid review bias.
STARDdem guidelines for reporting diagnostic test accuracy studies in dementia were observed (Noel-Storr et al., Reference Noel-Storr2014). Standard summary measures of discrimination were calculated: correct classification accuracy, sensitivity, specificity, Youden index, positive and negative predictive values, predictive summary index, positive and negative likelihood ratios, diagnostic odds ratio, clinical utility indexes, and area under the receiver operating characteristic (AUC ROC) curve, using the test cut-offs specified in the index papers (MACE ≤25/30; MoCA ≥26/30) to avoid introduction of bias (Davis et al., Reference Davis2013).
Effect sizes (Cohen's d) were calculated for MACE and MoCA as the difference of the means of diagnostic groups divided by the weighted pooled standard deviations of the groups (Larner, Reference Larner2014; Reference Larner2016a).
MACE and MoCA were compared by means of weighted comparison performed as per the method of Moons et al. (Reference Moons1997).
$$\begin{eqnarray*}
{\rm{WC}} &=& \Delta \,{\rm{sens}}\, + [(1 - \pi /\pi )\\
&& \times \,{\rm{relative}}\,{\rm{cost}}\,{\rm{(FP/TP)}} \times \,\Delta \,{\rm{spec}}]
\end{eqnarray*}$$
where π = prevalence; FP = false positives; and TP = true positives. The relative misclassification cost (FP/TP), a parameter which seeks to define how many false positives a true positive is worth, was set at FP/TP = 0.1, following previous usage (Mallett et al., Reference Mallett, Halligan, Thompson, Collins and Altman2012; Larner, Reference Larner2013), and reflecting the clinical desire for high test sensitivity. From the WC values, equivalent increase in TP patients per 1,000 tested was also calculated, using the equation (WC × prevalence × 1,000).
As administration of MACE and MoCA was already routine in this clinic, the study was an audit of established practice, so neither institutional ethical review nor specific consent procedures were indicated.
Results
In total, 260 patients were assessed with both MACE and MoCA (F:M = 118:142; age range 22–89 years, median 59), of whom 43 were diagnosed with dementia (prevalence = 0.17) and 75 with MCI (prevalence = 0.29). Age and gender details per group are shown in Table 1; the patients diagnosed with dementia and with MCI were older than those diagnosed with SMC. This patient cohort was typical of those seen historically in this clinic (Larner, Reference Larner2012; Reference Larner2015a; 2015Reference Larnerb).
Patient diagnostic groups by age and gender
The distribution of MACE and MoCA scores by diagnosis is shown in Figure 1(a) and (b), respectively.
(a) MACE scores versus patient diagnosis. (b) MoCA scores versus patient diagnosis.
Standard summary measures of discrimination are shown in Table 2. Both MACE and MoCA were very sensitive for the diagnosis of both dementia and MCI with high negative predictive values but with low specificity and positive predictive values. Positive likelihood ratios were unimportant, whereas negative likelihood ratios were large for dementia diagnosis and moderate for MCI diagnosis. Using the classification of AUC ROC suggested by Metz (Reference Metz1978), both MACE and MoCA were excellent for dementia diagnosis, whereas MoCA was good and MACE fair for MCI diagnosis.
Measures of discrimination (with 95% confidence intervals) for diagnosis of dementia versus MCI and MCI versus subjective memory complaint (SMC) using MACE and MoCA at pre-specified cut-offs
Effect sizes (Cohen's d) were large for both tests for diagnosis of both dementia and MCI (Table 3).
Effect sizes (Cohen's d) for MACE and MoCA for diagnosis of dementia and MCI
Weighted comparisons showed a minuscule net loss for MACE versus MoCA for dementia diagnosis, resulting in an equivalent increase of less than 1 extra dementia case detected by MoCA per 1,000 tested. For MCI diagnosis, MACE showed a small net benefit versus MoCA, resulting in an equivalent increase of 13–14 extra MCI cases detected per 1,000 tested (Table 4).
Weighted comparison of M-ACE (cut-off ≤25/30) versus MoCA (cut-off ≥26/30) for diagnosis of dementia versus no dementia (n = 260) and MCI versus subjective memory complaint (n = 217)
Discussion
This direct comparison of MACE and MoCA, to the author's knowledge the first such diagnostic test accuracy study to be reported, has shown that both tests are very sensitive for the diagnosis of both dementia and MCI at the cost of low specificity and positive predictive value, with high negative predictive values and large to moderate negative likelihood ratios. MACE sensitivity for diagnosis of dementia (0.98) and MCI (0.95) was comparable to that seen in a previous report (1.00 and 1.00, respectively; Larner, Reference Larner2015a). MoCA sensitivity for diagnosis of dementia (1.00) and MCI (0.92) was comparable to that reported in a meta-analysis (0.91 and 0.89, respectively; Tsoi et al., Reference Tsoi, Chan, Hirai, Wong and Kwok2015).
Using various comparative metrics, MACE and MoCA appear comparable for dementia diagnosis (AUC ROC curve, effect size, and weighted comparison), whereas for MCI diagnosis, the metrics are somewhat contradictory, suggesting a slight benefit either for MoCA (AUC ROC curve) or for MACE (weighted comparison), the latter finding endorsing a previous analysis of 6-month data from this study (Larner, Reference Larner2016b). In this context, previous criticism of AUC ROC as a comparative metric, and preference for some form of weighted comparison (Mallett et al., Reference Mallett, Halligan, Thompson, Collins and Altman2012), may be noted. Head-to-head results from this study were comparable to results from historical studies, in terms of AUC ROC (Larner, Reference Larner2015c) and effect sizes (Larner, Reference Larner2014; Reference Larner2016a).
Study shortcomings include those inherent in any clinic-based study. This was an unavoidably selected patient population with a relatively high prevalence of dementia and MCI compared to primary care and population-based cohorts. These prevalences of dementia and MCI, although typical of historical cohorts from this clinic (Larner, Reference Larner2012; Reference Larner2015a; 2015Reference Larnerb), may be lower than those seen in dedicated memory clinics in old age psychiatry or geriatric practice. This is a reflection of the age distribution of patients seen in a neurology-led clinic: The median age (59) was typical of historical cohorts seen in this clinic (Larner, Reference Larner2012; Reference Larner2015a; 2015Reference Larnerb). These differences inhibit generalization of the current findings to older outpatient samples, although an examination of older patient subgroups in historical cohorts from this clinic showed better metrics for MACE and MoCA than in the sample as a whole (Wojtowicz and Larner, in press).
Another shortcoming is the use of clinical diagnoses. The reliance on a single clinician's diagnoses may increase precision but may also introduce systematic bias. The etiology of dementia and MCI was not addressed in this study, as reflected in the diagnostic criteria used, DSM-IV and MCI (Petersen), respectively. There was no pathological verification of clinical diagnoses and no access to CSF or imaging biomarkers recommended in current criteria for AD (Dubois et al., Reference Dubois2014). These deficiencies may challenge the validity of the diagnoses and hence the study results. However, this approach probably reflects current practice in many cognitive disorder clinics. This was a pragmatic study, in which the tests were applied to a population suspected to have the target disorder; hence, the results are likely to have external validity (Larner, Reference Larner2015d). Moreover, test cut-offs were not adjusted from those defined in index studies, thus avoiding risk of bias (Davis et al., Reference Davis2013). CSIs are typically used to screen for dementia and MCI with the application of more sophisticated investigations to those screening positive in order to establish etiological diagnosis.
As there are already many CSIs available (Larner, Reference Larner2017), it may be questioned whether another one, MACE, is required. Aside from the statistical arguments for diagnostic accuracy presented in this study, the item content of the MACE differs from MoCA (Box 1) and the MMSE. It may be judged less stringent in memory testing than MoCA (with three presentations of the material to be remembered, versus two in MoCA), although both registration and recall are scored in MACE meaning that a greater percentage of the total test score reflects memory performance. Likewise, testing of executive function, based on verbal fluency, accounts for a greater percentage of the total test score in MACE than in MoCA. Cognitive impairment characterized by executive dysfunction, as, for example, in cases of mild frontotemporal dementia, may therefore screen positive using MACE despite preserved memory and visuospatial function. Using the Bland–Altman methodology, MACE and MoCA scores were found to have broad limits of agreement, >10 points in 30-point scales, reflecting their different item contents (Larner, Reference Larner2016c). Both MACE and MoCA are free to use and have no copyright issues, unlike MMSE.
The implications of these findings for clinical practice are that the MACE may be used with confidence in place of MoCA for the assessment of patients with cognitive complaints referred to specialist cognitive disorders clinics.
Conflict of interest
None.
Description of author's roles
AJ Larner conceived and planned the study, collected and analyzed the data, and drafted the manuscript.
