Introduction
Mild cognitive impairment (MCI) refers to a clinical condition that represents an intermediate stage between early dementia and normal cognitive decline, but has not been diagnosed as dementia, although cognitive functions have deteriorated (Petersen et al., Reference Petersen, Smith, Waring, Ivnik, Tangalos and Kokmen1999; Morris et al., Reference Morris2001). Several studies have indicated that approximately 11 to 33% of elderly adults with MCI experience Alzheimer's disease (AD) within three years and about 80% of them show symptoms of dementia within six years (Morris et al., Reference Morris2001; Plassman et al., Reference Plassman2007). Therefore, it is important to screen elderly adults with MCI early before clinically evident AD occurs.
The Montreal Cognitive Assessment (MoCA) is the most widely used screening test for MCI (Nasreddine et al., Reference Nasreddline2005). The MoCA contains visuospatial ability, executive function, language, attention-concentration-working memory, short-term memory recall, and orientation to time and place (Nasreddine et al., Reference Nasreddline2005). The MoCA have been standardized and normative data have been obtained in tests.
However, the MoCA has several limitations in detecting MCI despite high test–retest reliability and excellent criterion and construct validity (Nasreddine et al., Reference Nasreddline2005). First, in the MoCA, memory items only represent 5 out of 30 points. The previous studies have shown that the relative weight of items for memory included in the MoCA is too low to sensitively detect elderly with MCI (Jung, Reference Jung2010; Park, Reference Park2017). Second, the MoCA as well as other paper-based test tools cannot measure reaction time of respondents, which can also be used for screening MCI (Phillips et al., Reference Phillips, Rogers, Haworth, Bayer and Tales2013). Finally, these tools are influenced by the tester's degree of training as well by the test environment. If the tester deviates from the standardized procedure and conducts the test poorly, grading errors may occur (Wouter et al., Reference Wouters2014).
To overcome these limitations, computerized cognitive test systems have been developed since the early 2000s (Zygouris and Tsolaki, Reference Zygouris and Tsolaki2015). Technical innovations for accurate scoring as well as measurement of reaction time enhance the use of these systems (Zygouris and Tsolaki, Reference Zygouris and Tsolaki2015). However, some studies indicated that these systems have less sensitivity for screening MCI than paper-based test tools (De Lepeleire et al., Reference De Lepeleire, Heyrman, Baro and Buntinx2005). This is because the criteria used to select elderly with MCI at the time of the development of a computerized cognitive function test system differed from the standard criteria for MCI (Park and Park, Reference Park and Park2016). In addition, the existing computerized cognitive function test systems do not give more weight to memory scores, and some of the items that are included do not reflect the cognitive characteristics of elderly adults with MCI (Park and Park, Reference Park and Park2016).
To address these problems, the mobile cognitive function test system for screening mild cognitive impairment (mSTS-MCI) was developed. This study assessed the reliability and validity of the mSTS-MCI, and to determine the optimal cut-off mSTS-MCI score that best corresponds to a clinical screening of MCI.
Method
Participant and procedure
All participants were aged 65 years and over and were recruited from communities and welfare center in South Korea. Eligibility criteria are as follow: (1) fluency in Korean; (2) no history of neurological and psychiatric disease or any other physical illness affecting cognitive function; and (3) agreement to participate in this study. All participants were allocated the normal control (NC) group or the MCI group.
The NC group consisted of 103 participants with no memory complaints, and they were in the normal range for standardized neuropsychological tests. The MCI group consisted of 74 elderly participants. MCI was defined according to the criteria of Petersen et al. (Reference Petersen2004): (1) a subjective memory complaint; (2) memory impairment relative to age- and education-matched healthy elderly confirmed by performance on neuropsychological tests (below 1.5 standard deviation); (3) intact general cognitive function confirmed by MMSE-K scores; (4) intact activities of daily living; and (5) without dementia confirmed by a physician. These criteria define the subtype of MCI known as amnestic-MCI. For the testing, the neuropsychological battery, the Seoul Neuropsychological Screening Battery, was applied to each participant in a quiet room (Ahn et al., Reference Ahn2010). Since the testing time is about 2 h, if the participant felt fatigue during the testing, the test was stopped and it was continued on the next day. All participants were screened by a blinded assessor who has five years of clinical experience and is familiar with all tests.
All participants provided written informed consent, and this study was approved by the institutional review board of Yonsei University (1041849-201611-BM-060-01). All participants were administered the mSTS-MCI, Korean version of MoCA (MoCA-K), Korean version of Mini-Mental State Examination (MMSE-K), and neuropsychological battery (Rey Auditory Verbal Learning Test and Delayed Visual Reproduction and Logical Memory, two subtests of the Wechsler Memory Scale) by one trained occupational therapists with more than three years of clinical experience. These measurements were conducted on the same day for each participant. Test–retest reliability tests were performed using 30 of the study participants at four-week intervals.
Measures
The mobile screening test system for mild cognitive impairment (mSTS-MCI)
For development of the mSTS-MCI, systematic review was conducted. In 10 screening test systems for MCI, the duplicated items were removed, and a total of 48 items were extracted and classified into seven cognitive domains (orientation, memory, attention, visuospatial ability, language, executive function, and other domains).
In the previous study, panel meetings were conducted to verify the validity of the extracted items. Panel consisted of professors with experience in computerized cognitive rehabilitation programs and occupational therapists with clinical experience of over five years. The validity of the items was verified through three Delphi surveys and preliminary items were selected. The mobile application was created by computerizing the preliminary items and installed on a tablet computer. After that, the feasibility study was conducted on ten healthy elderly adults and five elderly adults with MCI and then the mSTS-MCI was finally completed by removing items that showed no significant differences between both elderly groups (Park and Heo, Reference Park and Heo2017).
The mSTS-MCI consists of eight items for memory, 1 item for attention, and 4 items for executive function. It scores from 0 to 28 points, and the higher scores, the better cognitive functions. It can be administered in about 15 min. In detail, the memory scores range from 0 to 18, in the attention, reaction time is measured only without scores, and the executive function scores range from 0 to 10 points. All data of the mSTS-MCI can be stored automatically. The user manual written in English and Korean is included in mSTS-MCI, and it also can be implemented in English or Korean (Park and Heo, Reference Park and Heo2017).
The Korean version of Montreal Cognitive Assessment (MoCA-K)
The original MoCA is a paper-and-pencil brief cognitive screener used to assess cognitive domains (Nasreddine et al., Reference Nasreddline2005). The ten items summed to create the total score include visuospatial/executive function, naming, memory, attention, language, abstraction, and orientation. A total of scores of the MoCA range from 0 to 30, where higher scores mean better cognition and a score below 26 points indicates cognitive impairment corresponding to MCI and AD. In the MoCA-K, cut-off scores for MCI were 23 points, and the words and letters used in MoCA were changed. One point can be added for subjects with six years of education or less instead of 12 years of education or less.
The Korean version of Mini-Mental State Examination (MMSE-K)
The MMSE is a test developed to screen cognitive impairment (Folstein and Folstein, Reference Folstein and Folstein1975). It scores from 0 to 30 points, where higher scores indicate better cognitive function, and scores below 25 points indicate cognitive impairment. The MMSE-K is composed of orientation, short-term memory registration and recall, attention, naming, following verbal commands, judgment, and copying a double pentagon. The MMSE-K was validated in Korean old adults (Kwon and Park, Reference Kwon and Park1989).
Statistical analyses
SPSS for windows (version 22.0) was used to analyze the data in this study. Demographic characteristics of the participants were analyzed using descriptive statistics. The intra-class correlation coefficients (ICCs) was used for assessing test–retest reliability of the mSTS-MCI, Cronbach's α was used for evaluating internal consistency, and concurrent validity was confirmed using Spearman's rank correlation. Independent sample t-tests were used to determine the discriminant validity of the mSTS-MCI. The sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were confirmed through Receiver Operating Characteristic (ROC) analysis, and the cut-off scores for elderly adults with MCI were identified according to the highest Youden Index defined as sensitivity + specificity −1. This index provides a criterion for choosing the optimal cut-off scores (Fluss et al., Reference Fluss, Faraggi and Reiser2005).
Results
Demographic findings and group difference
Participant characteristics are indicated in Table 1. There were no significant differences in mean ages, sex ratio, and education years between both groups. Table 1 shows descriptive statistics for the MoCA-K and the mSTS-MCI. The MoCA-K scores were significantly different between both groups. The mSTS-MCI scores and reaction time are also significantly different between both groups.
General characteristics of participants (n = 177)
**p < 0.001.
Internal consistency and test–retest reliability
Cronbach's α for the mSTS-MCI was 0.87. This result indicates a high level of internal consistency of the mSTS-MCI. A similar high consistency was found for all the 13 items of the mSTS-MCI with Cronbach's α values ranging from 0.67 to 0.95.
Test–retest reliability data were collected from 30 participants at four week intervals. Mean change in the mSTS-MCI scores and reaction time from the first to second evaluation were 18.95 to 19.05 points and 14.51 to 15.27 sec, respectively. The mSTS-MCI scores (ICCs = 0.984) and reaction times (ICCs = 0.965) showed high test–retest reliability (Table 2).
Test–retest reliability (n = 30)
**p<0.001.
Concurrent validity
Correlations with the MoCA-K were high. The mSTS-MCI scores were found to be highly and positively correlated with the MoCA-K scores (r = 0.773), and the mSTS-MCI reaction time was negatively correlated with the MoCA-K scores (r = −0.511) (Table 3).
Concurrent validity (n = 177)
**p<0.001.
Sensitivity, specificity, and cut-off score
ROC curves were drawn for the MCI group versus the NC group to determine the discriminatory validity of the mSTS-MCI (Figures 1 and 2). Areas under the ROC curve for the MCI group by the mSTS-MCI scores and reaction time were 0.985 and 0.996, respectively. These results showed that the mSTS-MCI differentiates this group well, the corresponding values for the MoCA-K scores were 0.819 (Figure 3).
ROC curve of the mSTS-MCI scores.
ROC curve of the mSTS-MCI reaction time.
ROC curve of the MoCA-K scores.
Tables 4 and 5 presented the sensitivities, specificities, PPVs, and NPVs of the mSTS-MCI at by the different cut-off values. The optimal cut-off value for the mSTS-MCI scores and reaction time as determined by this study appears to be 18/19 and 13.22/13.32, respectively. At this cut-off, the sensitivity of the mSTS-MCI scores and reaction time in terms of screening MCI was 99% and 100%, respectively. The specificity of the mSTS-MCI scores and reaction time was 93% and 97%, respectively. At 22/23 cut-off points, the sensitivities of the MoCA-K for screening MCI was 94%, and the specificity of the MoCA-K to screen MCI was 33% (Table 6).
Sensitivity and specificity of total scores of the mSTS-MCI for detection of MCI (n = 177)
Sensitivity and specificity of reaction time of the mSTS-MCI for detection of MCI (n = 177)
Sensitivity and specificity of total scores of the MoCA-K for detection of MCI (n = 177)
For the mSTS-MCI scores, the PPV and NPV was 98.57% and 95.32%, respectively. For the mSTS-MCI reaction time, PPV and NPV was 96.10% and 100%, respectively. On the other hand, for the MoCA-K scores, PPV and NPV was 88.00% and 76.37%, respectively (Table 7).
Comparison of PPV and NPV based on cut-off between the mSTS-MCI and the MoCA-K (n = 177)
Discussion
The population of the elderly with dementia is rapidly increasing, but early diagnosis and intervention is not available to most patients because of the low sensitivity of existing tests, such as MMSE and MoCA (Wind et al., Reference Wind, Schellevis, Van Staveren, Scholten, Jonker and Van Eijk1997; Park and Park, Reference Park and Park2016). Therefore, more accurate and sensitive tests for screening MCI are needed. In this context, this study was conducted to verify the clinical usefulness of the mSTS-MCI, the novel computerized cognitive function test system for screening MCI.
In demographical characteristics, there were significant differences in the mSTS-MCI between the MCI and the NC groups, verifying the discriminant validity of the mSTS-MCI. These results may be attributed to the fact that the mSTS-MCI consisted only of the key cognitive domains in which declines occurred in the elderly adults with MCI. As the results of the test–retest reliability, it was confirmed that the ICCs value was higher than 0.965 in total scores and reaction time, and that the test results were thus stable over time. The high correlation observed between the MoCA-K and the mSTS-MCI proved good concurrent validity.
In the ROC curve analysis, the sensitivity and specificity of the mSTS-MCI was higher than those of the MoCA-K, suggesting that the mSTS-MCI is more sensitive and specific to screening MCI (Jung, Reference Jung2010). Considering the results of previous studies reporting that the sensitivity of existing computerized cognitive function test systems was lower than paper-based test tools, the high sensitivity of the mSTS-MCI is more appropriate for screening MCI than existing computerized cognitive function test systems as well as paper-based test tools (De Lepeleire et al., Reference De Lepeleire, Heyrman, Baro and Buntinx2005; Jung, Reference Jung2010).
In this study, the mSTS-MCI had the higher AUC value than MoCA-K because the mSTS-MCI is more focused on memory tests than previous test tools. The most prominent feature of elderly adults with MCI is that they do not completely recall the events that have just happened (Petersen et al., Reference Petersen, Smith, Waring, Ivnik, Tangalos and Kokmen1999). In order to test memory of elderly adults with MCI, items for testing episodic memory should therefore be included (Petersen et al., Reference Petersen, Smith, Waring, Ivnik, Tangalos and Kokmen1999; Albert et al., Reference Albert, Moss, Tanz and Jones2001; Tierney and Lerner, Reference Tierney and Lerner2010). However, in the MoCA-K, which is the most commonly used test for elderly adults with MCI, episodic memory tests only represent 5 out of 30 possible points. Moreover, in existing computerized cognitive function test systems, the proportion of memory items is not different from the proportion of items from other cognitive domains, which makes these systems inadequate for screening MCI (Clague et al., Reference Claque, Dudas, Thompson, Graham and Hodges2005; Ahmed et al., 2009). On the other hand, the mSTS-MCI memory scores account for 18 out of 28 possible points.
Another reason why the mSTS-MCI has a high AUC value is because it includes both auditory and visual memory items. Auditory and visual memory impairment are common in elderly adults with MCI (Petersen et al., Reference Petersen, Smith, Waring, Ivnik, Tangalos and Kokmen1999), but the MMSE-K and the MoCA-K are inadequate for screening MCI by testing only auditory memory (Kwon and Park, Reference Kwon and Park1989; Lee et al., Reference Lee2008). Furthermore, the mSTS-MCI was able to screen MCI more sensitively by testing memory through free recall as well as recognition. In the case of a test system that adopts the self-administration method among all existing computerized cognitive function test systems, memory is tested only through recognition, based on the characteristics of the administration method. However, since elderly adults with MCI show impairment of both free recall and recognition, computerized cognitive function test systems that test memory through recognition do not sensitively detect MCI (Memoria et al., Reference Memoria, Yassuda, Nakano and Forlenza2014). Therefore, the mSTS-MCI seems to be more useful than existing computerized cognitive function test systems evaluating memory based on recognition.
Since it is common to conduct a more detailed test after the screening test, it is desirable to lower the false NPV (Kwon and Park, Reference Kwon and Park1989). The NPV of the total scores and reaction time of the mSTS-MCI was higher than that of MoCA-K. Therefore, in order to screen MCI, it is recommended to screen MCI based on the total scores and reaction time rather than dividing it by domain. Specially, when the total scores and reaction time were used as the criteria, the NPV of the mSTS-MCI was 95.32% and 100%, respectively, whereas the NPV of the MoCA-K was 76.37, suggesting that the mSTS-MCI is appropriate for screening MCI.
The mSTS-MCI is a computerized cognitive function test system based on the items reflecting the cognitive characteristics of MCI. The items of the existing computerized test systems that did not reflect the cognitive characteristics of MCI were included, while in the mSTS-MCI, the items consisted of only memory, executive function, and reaction time, so that more sensitive testing is possible. The test results by cognitive domains showed a clearly characterized cognitive decline in MCI. In particular, there was a significant difference in reaction time between healthy elderly adults and those with MCI, as well as in the test scores, indicating that reaction time may be used as an important intervention measure in cognitive rehabilitation for MCI (De Jager et al., Reference De Jager, Schrijnemaekers, Honey and Budge2009).
This study has several limitations. First, the number of elderly adults with MCI was relatively low, which might limit the validity and reliability of the score analyses. In the future, it is necessary to collect data for a large number of elderly adults including different ages and education levels, and to set up selection criteria according to demographic characteristics (Han et al., Reference Han2010). Second, a learning effect in the mSTS-MCI was not evaluated. The order of the examples and stimuli was slightly different each time the mSTS-MCI was administered, which should minimize a learning effect (Falleti et al., Reference Falleti, Maruff, Collie and Darby2006). However, since a learning effect has not been verified, it is necessary to repeat the mSTS-MCI within a short period of time to confirm whether the score improves. Third, it has not been established whether the mSTS-MCI can be used effectively by other clinicians. Therefore, inter-rater reliability must be analyzed to confirm the effectiveness of the mSTS-MCI in clinical settings. Finally, further studies on the mSTS-MCI are also needed to evaluate the clinical efficacy of the system.
Conclusion
These results showed that the mSTS-MCI could be reliably used in primary care and geriatric outpatient clinics to screen MCI in just 10 min. Unlike the existing computerized test systems, the mSTS-MCI can be administered more quickly by testing only the core test domains reflecting the cognitive characteristics of MCI.
Conflict of interest
No potential conflict of interest was reported by the authors.
