Introduction
Mild cognitive impairment (MCI) is considered a transitional state between normal aging and mild dementia, and often precedes dementia (Petersen et al., Reference Petersen, Smith, Waring, Ivnik, Tangalos and Kokmen1999). MCI is defined as impairment in one or more cognitive domains with the preservation of independence in functional abilities and is recognized as an important target for Alzheimer's disease (AD) prevention studies (Bruscoli and Lovestone, Reference Bruscoli and Lovestone2004; Dubois and Albert, Reference Dubois and Albert2004). Early detection of deficits in patients with symptoms that are too mild to fulfil criteria for dementia is crucial in clinical practice and in research for successful preventive interventions and has been the object of discussion in several studies (Smith et al., Reference Smith, Petersen, Parisi and Ivnik1996; Petersen et al., Reference Petersen2001; Petersen et al., Reference Petersen2004).
The Montreal Cognitive Assessment (MoCA) is a cognitive screening instrument created and validated (Nasreddine et al., Reference Nasreddine2005) to detect MCI, and is considered superior to the Mini-Mental State Examination (MMSE) (Folstein et al., Reference Folstein, Folstein and McHugh1975). The validity of the MoCA has been studied in various clinical groups in many countries (Gill et al., Reference Gill, Freshman, Blender and Ravina2008; Lee et al., Reference Lee2008; Wong et al., Reference Wong2009; Fujiwara et al., Reference Fujiwara2010).
Little is known about how well the MoCA subscales assess the entire spectrum of cognitive dysfunction in AD. A thorough understanding of the psychometric properties across a broad range of cognitive dysfunctions is critical for optimal usage in clinical settings. As the development of therapies shifts to earlier interventions, correlated with milder cognitive dysfunction, it is especially important to understand which subscales provide the most reliable data and how the scale as a whole functions. Item response theory (IRT) provides a framework for analyzing this measure in psychometrically sophisticated ways (Hambleton et al., Reference Hambleton, Swaminathan and Rogers1991; Emberson and Reise, Reference Emberson and Reise2000).
The aims of the current study were to (1) evaluate the reliability and validity of the Taiwanese version of the MoCA (MoCA-T), and (2) apply IRT techniques to analyze how the subscales of the MoCA-T and the instrument as a whole function across the spectrum of AD severity, and to elucidate how individual items play a role in measuring disability.
Methods
Translation and cultural modifications of the MoCA-T
The MoCA is a brief (10 minute) test that evaluates seven cognitive domains: visuospatial/executive functions, naming, verbal memory registration and learning, attention, abstraction, 5-min delayed verbal memory, and orientation. MoCA scores range from 0 to 30. The English version MoCA was translated into Chinese/Taiwanese and independently translated back into English by a neurologist and a psychiatrist. Items on the Chinese/Taiwanese version were identical to the English version with the exception of the following five cultural and linguistic modifications listed below:
(1) Item 1 (Visuospatial/Executive Functions – Alternating Trail Making): We modified Trail Making Part B (Reitan, Reference Reitan1959) by using Chinese nominal sequential words instead of English alphabet. The number of steps required for completion of the task was retained (Lu and Bigler, Reference Lu and Bigler2002).
(2) Item 4 (Attention – Auditory Vigilance): Arabic numerals were used instead of English alphabet. The number and positions of responses remained identical to those in the English version.
(3) Item 5 (Language – Sentence repetition): English sentences were replaced with Chinese/Taiwanese sentences.
(4) Item 5 (Language – Verbal fluency): A fruit category fluency task replaced phonemic letter fluency, as there are no letter-equivalent linguistic units in the Chinese language.
(5) Item 8 (Orientation): District was substituted for city. In Taiwan, districts are the conceptual equivalent of cities in North American countries.
Translation and modifications in the final version of the MoCA-T were discussed with and approved by the original author of the MoCA (Z.N.). The MoCA-T test form and administration instructions are available for download at the MoCA official website http://www.mocatest.org.
Participants
The study participants were recruited from the outpatient Memory Clinic of Taipei Veterans General Hospital (TVGH). Diagnostic determinations based on the results of clinical interviews, physical examinations, laboratory findings and image investigations (CT and/or MRI) were made at a clinical consensus meeting. An AD diagnosis was made according to the criteria of National Institute of Neurological and Communicative Disorders and Stroke – Alzheimer's Disease and Related Disorders Association (NINCDS-ADRDA) (McKhann et al., Reference McKhann, Drachman, Folstein, Katzman, Price and Stadlan1984). Criteria for MCI included: (1) memory complaints, (2) impaired memory function as determined by the modified Chinese Version Verbal Learning Test, and (3) maintenance of daily living activities (Petersen et al., Reference Petersen2001). Criteria for participation included classification as amnestic MCI (single or multidomain); nonamnestic MCI (single or multidomain) patients were excluded. Other inclusion criteria for the study were: (1) probable clinical diagnosis of MCI or AD, and (2) 60 years of age or older. Exclusion criteria were: (1) an MMSE score of < 10 (severe dementia), (2) recent psychiatric comorbidity (clinically diagnosed in the 6 months prior to the current neuropsychological evaluation), (3) motor and/or sensory deficits that constituted confounding variables in the assessment of cognitive functions, and (4) current clinically important systemic or neurological illness that may affect cognition. The study controls were relatives of the participants or volunteers recruited from the hospital. Controls had no or mild memory complaints, were in the normal range on standardized neuropsychological batteries, and had no neurological abnormality. The study protocol was approved by the TVGH Institutional Review Board. All subjects provided informed consent before study participation.
Measures
Each subject completed the MMSE, the MoCA-T, and the Chinese Version Verbal Learning Test. To minimize interference, the MoCA-T was always completed 30 minutes before the neuropsychological battery. MoCA subscales were grouped into five domains: memory (delayed recall, 5 points); visuospatial (trail-making test, 1 point; cube, 1 point; clock drawing, 3 points); frontal functioning including attention, concentration, working memory and abstract thinking (digit span, 2 points; cancellation, 1 point; subtraction, 3 points; verbal abstraction, 2 points); language (nomination, 3 points; sentence repetition, 2 points; phonemic verbal fluency, 1 point); and orientation (time, 3 points; space, 3 points). The Chinese Version Verbal Learning Test was modified from the Rey Auditory–Verbal Learning Test (Vakil and Blachstein, Reference Vakil and Blachstein1993) and has been shown to have good validity in diagnosing AD and MCI (Chang et al., Reference Chang2010).
Statistics
All statistical analyses were performed using SPSS version 15.0. χ2 analyses were used to compare demographic and neuropsychological data between normal controls and patients. ANOVA with post hoc least significant difference (LSD) tests were performed as appropriate. The relationship between performance on the MoCA-T and MMSE and demographic data were assessed using the Spearman rank correlation.
The ability of the MoCA-T total score and MMSE scores to differentiate MCI patients from controls was examined using receiver operating curve (ROC) analysis. A cut-off score for optimal balance of sensitivity and specificity was derived. A four-week test-retest and inter-rater reliability of the MoCA-T among 20 randomly selected subjects was assessed using the intra-class correlation coefficient. Internal consistency between the ten domain scores was evaluated using Cronbach's α. Two-tailed tests were used throughout all analyses and a p-value < 0.05 was considered statistically significant. IRT, performed by version 9.1.3 of SAS (SAS Institute, Cary, NC, USA), was used empirically to assess the relationship between the total MoCA-T and each of the five domains represented by the subscales (Hambleton et al, Reference Hambleton, Swaminathan and Rogers1991; Scherbaum et al., Reference Scherbaum, Cohen-Charash and Kern2006; Oishi, Reference Oishi, Ong and Van Dulmen2007). These analyses will help to build a framework from which we can better interpret changes in raw scores that occur during clinical research.
Results
Study participants
A total of 207 participants completed the study, including 71 patients with MCI, 98 patients with mild to moderate AD, and 38 controls. The mean age was 77.3 years (SD = 7.5, range 60–92) and the mean educational level was 9.8 years (SD = 5.1, range 0–18); 44% of the participants were female and 71% were married. Participants had a mean MoCA-T score of 18.1 (SD = 6.8, range 2–30) and a mean MMSE score of 22.9 (SD = 5.8, range 10–30). The demographic characteristics of the study sample by diagnosis group are presented in Table 1. Age, sex, and educational year distributions did not differ among the three study groups (AD, MCI, and controls) (Table 1). The mean MMSE and MoCA-T scores were significantly different between groups (by post hoc LSD analyses, all p values < 0.001) with highest scores among normal controls, followed by MCI participants and lowest scores among participants diagnosed with AD. There was a significant negative correlation between MoCA-T score and age (Spearman's r = −0.31, p < 0.001) and a positive correlation between MoCA-T score and years of education (Spearman's r = 0.43, p < 0.001), while there was no correlation between MoCA-T score and gender (p = 0.83).
Demographic and clinical characteristics of the study participants
Note: *The χ2-test was used to examine associations among categorical variables, and ANOVA was used to compare the means of continuous variables. AD = Alzheimer's disease; CVVLT = Chinese Version Verbal Learning Test; MCI = mild cognitive impairment; MoCA-T = Taiwanese version of Montreal Cognitive Assessment; NC = normal controls; SD = standard deviation.
Reliability and validity of the MoCA-T
Cronbach's α of the MoCA-T was 0.86. The intraclass correlation coefficient between the baseline and 4-week follow-up data was 0.88 (95% confidence interval [CI] = 0.79–0.93, p < 0.01). There was a positive associated between MoCA-T scores and MMSE scores (r = 0.91, p < 0.001), and between MoCA-T scores and the Chinese Version Verbal Learning Test scores (r = 0.45, p < 0.001). The predictive accuracy of MoCA-T for cognitive impairment was high; the AUC was 0.91 (95% CI = 0.86–1.00, p < 0.001) for MCI and 0.99 (95% CI = 0.98–1.00, p < 0.001) for AD. In contrast, the predictive accuracy of MMSE for MCI was 0.81 (95% CI = 0.72–0.89, p < 0.001) (Figure 1), and for AD was 0.98 (95% CI = 0.97–1.00, p < 0.001). The best cut-off point for discriminating between the MCI group and the control group was 23/24 on the MOCA-T, with a sensitivity of 92% and specificity of 78% while the best cut-off point for discriminating between the AD group and the control group was 21/22 on the MoCA-T, with a sensitivity of 98% and specificity of 95%. True positive and negative predictive values for the MoCA-T were high for MCI (88% and 94%, respectively) and AD (88% and 100%, respectively). In contrast, the MMSE had high sensitivity (95%) and good specificity (98%) for screening AD at the cut-off point of 25/26, but it had poor specificity (63%) for screening MCI at the cut-off point of 27/28.
ROC depicting the ability of the MoCA-T and MMSE in detecting MCI patients from controls. (MoCA-T: AUC = 0.91, p<0.001; MMSE: AUC = 0.81, p< 0.001).
IRT analysis results
Results from the IRT analysis are presented in Figures 2–4. The item domain curves were rescaled for the purpose of visual inspection to account for differences in the total number of points possible in each domain, allowing for domain performance comparisons across the spectrum of cognitive dysfunction (x-axis).
Taiwanese version of Montreal Cognitive Assessment total scale item characteristic curves and test information curve. SD = standard deviation.
Taiwanese version of Montreal Cognitive Assessment subscale item characteristic curves. SD = standard deviation.
Taiwanese version of Montreal Cognitive Assessment subscale item characteristic and information curves.
The steepest aspect of the slopes of each of the domain curves (Figure 3) are as follows: frontal domain, between −2 and 1 SDs of cognitive dysfunction; orientation domain, between -2 and 0 SDs; language domain, between −1 and 1 SDs; visual-spatial domain, between -1 and 1 SDs; and memory domain, between 0 and 1.5 SDs.
In addition to generating domain functions, IRT analyses can be used to generate information functions (see the test information function in Figures 2 and 4). The information function indicates how well a particular instrument captures a latent phenomenon, in this case cognitive dysfunction. An instrument has optimal information at the point where the information curve peaks. Examination of Figure 2 shows that the information curve form MoCA-T total score peaks at the mild levels of cognitive dysfunction (approximately between −1 and 1 SDs of cognitive dysfunction) while information progressively drops off with lower information for measurements corresponding to much of the range of normal cognitive functions or advanced levels of cognitive impairment. The IRT analysis for the subscales revealed that the level of information provided by each of the five domains in the MoCA-T scale were consistent (Figure 4). The frontal domain and language domain provided high discriminating power and a high level of information across all cognitive ability levels. The memory domain provided high discriminating power and a high level of information at a range of mild cognitive impairment only, while the language and visual-spatial domains were not highly discriminative at advanced levels of cognitive impairment. The orientation domain provided a high level of information at moderate to severe levels of cognitive dysfunction but was poor at the lower range of cognitive dysfunction.
The information curve is steepest in the moderate range of cognitive dysfunction, where a unit change in impairment (x-axis) indicates a greater change in MoCA-T domain score (y-axis) than a unit change at mild or advanced ranges of cognitive dysfunction (Figure 3). This pattern suggests that the scale can discriminate best between smaller gradations of impairments at moderate levels of cognitive dysfunction. The left side of the graph (which demarcates between approximately −4.0 and −1.0 SDs, i.e. more severe levels of impairment) depicts a different pattern of results. Only the frontal and orientation domains have informative slopes at the severe range of cognitive dysfunction. As a result, the discriminative ability of the subscales remains low, suggesting that the domains and the scale as a whole perform relatively poorly at discriminating among degrees of cognitive dysfunction when the severity of cognitive impairment is advanced.
Discussion
This is the first psychometric study of the MoCA performed through IRT analyses. The current study examined the psychometric properties of the MoCA-T in a clinical setting. The results support the role of MoCA-T as a useful instrument in differentiating MCI subjects from cognitively normal controls.
The MoCA-T scores were negatively correlated with age and positively correlated with years of education, while there was no correlation with gender. The MoCA-T demonstrated a diagnostic sensitivity of 92% and specificity of 78% and performed better than the MMSE in differentiating MCI patients from normal controls when a cut-off of 23/24 was applied. The MoCA-T demonstrated high internal consistency, indicating a strong relationship between items. The strong validity of the MoCA-T was confirmed by a high correlation with the MMSE total score and the Chinese Version Verbal Learning Test scores. Several recent studies examining various populations have successfully used the MoCA to detect cognitive deficits in Parkinson's disease (Gill et al., Reference Gill, Freshman, Blender and Ravina2008), brain tumours (Olson et al., Reference Olson, Chhanabhai and McKenzie2008), AD (Tobinick and Gross, Reference Tobinick and Gross2008), and cerebral small vessel disease (Wong et al., Reference Wong2009).
The optimal cut-off point for the MoCA-T (23/24) was lower than that of the original MoCA (25/26) and of the Japanese version (25/26), and higher than that of the Korean version (22/23). The differences in optimal cut-off may be due to differences in years of education. The mean years of education (9.8 ± 5.1) in the current study was lower than that in the original and the Japanese version of the MoCA test, while slightly higher than that of the Korean version (original: 13.3 ± 3.4; Japanese: 12.4 ± 2.5; Korean: 8.3 ± 3.7) (Lee et al., Reference Lee2008; Fujiwara et al., Reference Fujiwara2010).
The current study is the first psychometric IRT study of the MoCA. The current analyses were conducted to examine how well each subscale of the MoCA-T and the test as a whole measures cognitive dysfunction at varying levels of cognitive impairment. Results indicated that the MoCA-T subscales measured AD-related cognitive dysfunction best at mild levels of cognitive impairment. Of the five cognitive domains designated a priori (frontal, orientation, language, visual-spatial and memory), performance on the frontal and memory subscales discriminated between levels of cognitive impairment better at the mild stage of cognitive dysfunction than the language and visual-spatial subscale, whereas the orientation subscale was not effective until moderate stages of cognitive impairment. Compared with other subscales, the frontal domain has the best discriminative ability across all levels of cognitive dysfunction. The tasks measuring frontal domain functioning in the present study included digit spans, cancellation, subtraction, and verbal abstraction. These tasks specifically measured attention, concentration, working memory, executive function, and abstract thinking. These findings suggest that frontal mental functioning was one of the first impairments demonstrated by the study population, supporting previous reports. Compared to healthy controls, studies have shown increased executive impairment, especially in the areas of response inhibition, cognitive flexibility, and attention switching in MCI patients (Perry et al., Reference Perry, Watson and Hodges2000; Kramer et al., Reference Kramer2006; Traykov et al., Reference Traykov2007). Impairments in digit span scores have been implicated as a first indication for conversion from subject memory complaints to MCI (Kurt et al., Reference Kurt, Yener and Oguz2011). The decline in working memory, simple sustained attention, and divided attention in the amnesic MCI group may be early indicators of a possible transition to dementia from MCI (Gagnon and Belleville, Reference Gagnon and Belleville2011; Saunders and Summers, Reference Saunders and Summers2011).
The memory domain test was particularly sensitive to a mild stage of cognitive impairment. Deficits in delayed recall are the cardinal feature of MCI (Petersen, Reference Petersen2004). Therefore, tests of delayed recall are especially useful in improving the psychometric properties of the MoCA-T in studies investigating the mild, preclinical stages of MCI, and recent clinical trial studies (Doody et al., Reference Doody2009) have taken this into consideration. While the 5-word delayed recall test was sensitive to mild stages of cognitive impairment, it had poor sensitivity to moderate and severe levels of cognitive dysfunction. The 5-word delayed recall is more difficult than the 3-item recall in the MMSE, and may cause a floor effect among more severe levels of cognitive dysfunction. The visual-spatial, orientation, and language domains demonstrated both ceiling and floor effects, and therefore the discriminative abilities of these subscales were relatively poor at the very mild and advanced stages of cognitive impairment.
The magnitude of cognitive impairment represented by each point on the MoCA-T score varied across levels of dysfunction. This phenomenon is most clearly demonstrated in Figure 2, which shows that at a range of high cognitive performance (MoCA-T total raw scores > 24), even small changes in raw MoCA-T score are associated with relatively large changes in cognitive function (θ). A similar pattern was observed in the low range of MoCA-T total raw scores. In the mild stage of cognitive dysfunction, between approximately −1.0 and 2.0 SDs of cognitive function, each MoCA-T raw score represents an incrementally smaller change in underlying cognitive dysfunction. These results will benefit clinicians and researchers alike in better understanding the information provided by MoCA scores of their patients with dementia and will help determine the influence of cognitive dysfunction on subsequent outcomes and response to treatment.
There are some limitations that must be considered when interpreting the results of the current study. Differences in language, variety of disease, and clinical setting make it difficult to directly compare our data with current publications. In addition, the current sample size was relatively small and did not allow a case match by age and level of education. Lastly, only subjects with AD and amnestic type MCI were enrolled in the current study which may limit application to other types of dementia and MCI.
In conclusion, the MoCA-T has high sensitivity and good specificity in the detection of MCI and mild to moderate AD, and performs significantly better than the MMSE. The MoCA-T effectively characterized subjects with AD, but performed poorly in normal or severely impaired participants. These results corroborate the MoCA's utility as a potential complement to the MMSE as a cognitive screening instrument, especially in the detection of MCI patients.
Conflict of interest
None.
Description of authors' roles
J.L. Fuh designed the study and supervised the data collection. C.F. Tsai and W.J. Lee wrote the paper. J.L. Fuh and S.J. Wang collected the data. J.L. Fuh and Z. Nasreddine assisted with writing the article. B.C. Shia was responsible for the statistical design of the study and for carrying out the statistical analysis.
Acknowledgements
The current study was supported by grants from Taipei Veterans General Hospital (V100C-059, VGHUST100-G7-1-3), NSC support for Centre for Dynamical Biomarkers and Translational Medicine, National Central University, Taiwan (NSC 99-2911-I-008-100), and the Ministry of Education (Aim for the Top University Plan), Taipei, Taiwan.
