Introduction
The proportion of elderly aged 65 and older in the population is rising rapidly in Hong Kong; it increased from 10.1% in 1996 to 12.4% in 2006 and is projected to reach 26.4% in 2036 (Census and Statistics Department, 2006a; 2006b). The prevalence of dementia among the elderly, those aged 70 years and above, in Hong Kong is estimated at 9.3% (HKSAR Department of Health, as cited in Ginkgo Group, n.d.). With the increasing size of the aging population, care for those with dementia is becoming an important and pressing issue in Hong Kong. It is vital to establish a reliable method of early detection of dementia and to implement further planning of appropriate intervention and treatment for the elderly.
The Mini-mental State Examination (MMSE), developed by Folstein et al. (Reference Folstein, Folstein and McHugh1975), is the most widely used instrument for assessing cognition among older adults. It briefly measures orientation to time and place, immediate recall, short-term verbal memory, attention, language, and construction ability. The maximum total score is 30; a cut-off point ≤ 23 was reported to discriminate between individuals who have cognitive impairment and normal individuals (Folstein et al., Reference Folstein, Anthony, Parhad, Duffy and Greenberg1985). It has been validated in many countries and innumerable translations of the MMSE are available, including those in French, Dutch, Spanish, Italian, Swedish, Finnish, Brazilian, Icelandic, Latvian, Japanese, Korean and Chinese (Blum, n.d.). Since cultural differences influence the scores and performance on the MMSE (Escobar et al., Reference Escobar, Burnam, Karno, Forsythe, Landsverk and Golding1986; Salmon et al., Reference Salmon, Riekkiner, Katzuran, Zhang, Jin and Yu1989), there should be minor adaptations of the original MMSE in the translation and validation process in order to suit the needs of different cultures. For Chinese regions, there are both Mandarin and Cantonese versions. The Mandarin version is widely used in mainland China (Katzman et al., Reference Katzman1988; Li et al., Reference Li, Shen, Chen, Zhao, Li and Lu1989; Yu et al., Reference Yu1989). In Hong Kong, two different Cantonese versions were developed by Fan (Reference Fan1992) and Chiu et al. (Reference Chiu, Lee, Chung and Kwong1994); both versions have undergone reliability and validity studies but the differences mainly lie in the linguistic translation for the repetition item. Besides cultural differences, a number of studies have revealed that education and age have a significant effect on scores on the MMSE (Anthony et al., Reference Anthony, Le Resche and Niaz1982; Holzer et al., Reference Holzer, Tischler, Leaf and Myers1983; Katzman et al., Reference Katzman1988; Magaziner et al., Reference Magaziner, Bassett and Hebel1987; Sahadevan et al., Reference Sahadevan, Lim, Tan and Chan2000). The study by Katzman et al. (Reference Katzman1988) suggests that different cut-off scores should be used according to the participants' educational level. For those with middle-school or higher education, a cut-off score of < 25 should be used, whereas for those with elementary education and no schooling, cut-off scores of < 21 and < 18 respectively should be used.
Although administering the MMSE is quick and easy, usually taking about 5–10 minutes, it still requires face-to-face administration. This makes it difficult to evaluate patients' performance if they fail to return to the clinic, which may be due to many reasons such as loss of ambulatory capacity, forgetting the date of follow-up, or reduced motivation due to high transport costs. Telephone follow-up provides an alternative to in-person clinic visits and has many advantages. It is relatively low in cost and takes a relatively short time to reach a large number of people (Sekaran, Reference Sekaran1992). Telephone follow-up also offers a way to increase retention in longitudinal studies, thereby bolstering the value of individual contributions to research (Newkirk et al., Reference Newkirk, Kim, Thompson, Tinklenberg, Yesavage and Taylor2004). Korner-Bitensky and Wood-Dauphinee (Reference Korner-Bitensky and Wood-Dauphinee1995) assessed self-care functioning by using the Barthel Index through telephone interview and found that the telephone version was comparable to a face-to-face interview with agreement being greater than 90%. For cognitive measures, Brandt et al. (Reference Brandt, Spencer and Folstein1988) reported a correlation of Pearson's r = 0.94 between the Telephone Interview for Cognitive Status (TICS), which is a 41-point cognitive assessment, and the MMSE. Roccaforte et al. (Reference Roccaforte, Burke, Bayer and Wengel1992) validated a telephone version of the MMSE that was originally administered as part of the Adult Lifestyles and Function Interview (ALFI-MMSE). This version includes 22 out of 30 of the original MMSE items, the majority of which were removed from the last section (language and motor skills). However, both the TICS and ALFI-MMSE have their limitations. The TICS does not assess the registration of words for recall and requires more time to administer. The ALFI-MMSE does not contain a three-step command, which is an important dimension of cognitive status. Therefore, Newkirk and colleagues (2004) developed a telephone version of the MMSE from the ALFI-MMSE. This is a 26-point adaptation, containing a three-step command: “Say ‘Hello,’ tap the mouthpiece of the telephone three times, then say ‘I'm back’.” It also contains a new question that asks the patients to give the interviewer a telephone number on which they can usually be reached. This 26-point telephone version of the MMSE (T-MMSE) has an excellent correlation (r = 0.88) with the original in-person MMSE (Newkirk et al., Reference Newkirk, Kim, Thompson, Tinklenberg, Yesavage and Taylor2004).
In Hong Kong, the Cantonese MMSE (CMMSE), which was developed 16 years ago, has been commonly used as a global measure to assess cognitive function in clinical and epidemiological settings. However, there is no similar kind of telephone-based cognitive measure in Hong Kong, and there is increasing need for a more convenient, yet valid and reliable method to assess cognition from a distance if face-to-face monitoring is difficult. The purpose of this study was to examine the convergent validity and external reliabilities of the 26-point T-CMMSE in relation to the CMMSE in elderly patients with or without dementia in Hong Kong, and to determine an optimal cut-off score on the 26-point T-CMMSE scale to discriminate between those with and without dementia.
Methods
Participants
A total of 65 participants between the ages of 65 and 95 were recruited for the study by convenience sampling from an acute regional hospital in Hong Kong (Table 1). There were 39 female participants (60%) and 26 male participants (40%) with a mean age of 77.8 (S.D. = 7.9). Their mean number of years of education was 2.4 years (S.D. = 3.2) with 50.8% being illiterate. There were two different diagnostic groups in the sample. The first group, consisting of elderly people (N = 34) who were diagnosed by geriatricians using the DSM-IV as having dementia or probable/possible dementia, were recruited from a geriatric rehabilitation ward of the hospital. The second group, consisting of elderly without dementia (N = 31) as confirmed by screening using the DSM-IV and their medical histories as recorded in their medical files, were recruited from an orthopedic rehabilitation ward in the same hospital. Other inclusion criteria for both groups were that they were medically stable, able to communicate fluently in Cantonese, and had no hearing impairment. Individuals with head injury, brain tumor or other brain diseases, acute delirium, psychiatric illnesses, or those taking medication affecting cognitive function were excluded.
Demographic characteristics of participants and the comparison of participants with or without dementia
*p ≤ 0.05; at-test for equality of means; NA = not applicable.
Instruments
The CMMSE (Chiu et al., Reference Chiu, Lee, Chung and Kwong1994) and the T-CMMSE were used in this study. The 26-point T-MMSE was first translated into Cantonese with some modifications by an expert panel, composed of six experienced occupational therapists of more than 10 years' clinical experience and one academic staff member from a university, using the linguistic style of the original version of the CMMSE. The CMMSE measures orientation to time and place, immediate recall, short-term memory, attention/calculation, language, ability to name objects, ability to follow verbal and written commands, and constructive ability. The maximum score is 30. The version of the CMMSE by Chiu et al. (1994) was used in this study for the following reasons. First, the sample size was larger in that study (N = 190) when compared with Fan's study (1992) (N = 29). Second, the study by Chiu et al. had a better description for the psychometric properties of the scales. Finally, the study by Chiu et al. had a control sample of normal participants for comparison.
The T-CMMSE measures the same constructs as the CMMSE except for the items “read and follow written command” and “constructive ability,” because these two areas cannot be examined during a telephone call. Besides these two items, there are four minor differences between the telephone and the original in-person version of the CMMSE. First, there are only four items of “orientation to place” in the T-CMMSE, whereas there are five such items in the CMMSE (the item “what floor of the building are we on?” was deleted from the T-CMMSE). Second, there is only one naming object item in the T-CMMSE, while there are two naming object items in the CMMSE. Third, the “follow a three-step verbal command” is different in the two versions. The three-step command in the T-CMMSE is, “Say ‘Hello,’ tap the mouthpiece of the telephone three times, then say ‘I am back’” (Newkirk et al., Reference Newkirk, Kim, Thompson, Tinklenberg, Yesavage and Taylor2004), whereas the command is “Take a sheet of paper in your right hand, fold it in half using both hands, and put it back on the desk” in the CMMSE (Chiu et al., Reference Chiu, Lee, Chung and Kwong1994). Finally, the T-CMMSE includes a question that asks the respondent to give the interviewer a telephone number, whereas in the CMMSE, the respondent is asked to “Say a sentence.” The maximum score on the T-CMMSE is 26 compared to 30 on the CMMSE.
Procedures
The study was conducted under the terms of the Declaration of Helsinki. Informed and written consent was obtained from all of the participants before data collection. For participants who were not mentally able to give their own consent, research investigators explained the study procedures to their next of kin in order to obtain their support and understanding. All participants were informed that they had the right to withdraw from the study at any time without giving a reason. The study was approved by the Departmental Research Committee of the Department of Rehabilitation Sciences, Hong Kong Polytechnic University (Ref: HSEAR20070907002) and the Research Ethics Committee of Kowloon Central/Kowloon East Cluster Hospital Authority, Hong Kong (Ref: KC/KE07–0160/ER-1).
In this study, two occupational therapists were responsible for administering the instruments. They were experienced in geriatric care and had prior experience of administering the CMMSE. All participants were assessed by an occupational therapist using the CMMSE before they were discharged from hospital. To establish the convergent validity, all participants were assessed one week later by the occupational therapist using the T-CMMSE in a telephone interview when they were in their own house. The occupational therapist made sure in the interview that both parties could hear clearly and were not distracted by other noises. The procedure for the flow of participant numbers is shown in Figure 1. Five participants (four with dementia and one without) dropped out from the study. Three of them could not be contacted by telephone within the assessment period. The remaining two were re-admitted to hospital due to medical problems. To examine the scale's intra-rater reliability, a random subset of 30 participants from the dementia group (N = 15) and the group without dementia (N = 15) was rated by the T-CMMSE for a second time one week after the first T-CMMSE assessment by the same occupational therapist. To examine the scale's inter-rater reliability, the remaining participants (15 with dementia and 15 without) were rated by the T-CMMSE for a second time by another occupational therapist.
Flow chart of the procedures.
Statistical analyses
The intra-rater reliability of the T-CMMSE was examined by using the first and second T-CMMSE scores of the same rater, and the inter-rater reliability was examined by using the first and second T-CMMSE scores of different raters. This was done using intra-class correlation (Model (2,k)) because this statistical technique is more conservative than Pearson's correlation analysis and therefore is more generalizable (Portney and Watkins, Reference Portney and Watkins2000). To compare the telephone and the original in-person versions of the CMMSE, item scores and total scores were compared by using the Kappa coefficient and Pearson's coefficient matrix. The convergent validity was examined by using multiple linear regression analysis enter method. The convergent validity, which is a form of construct validity, measured the extent to which the results of the T-CMMSE were in concordance with the results of the CMMSE that purported to measure the same concept (Portney and Watkins, Reference Portney and Watkins2000). The regression equations for predicting the CMMSE and the T-CMMSE, and the conversion table between the two versions of the CMMSE, were formulated. Finally, receiver-operating characteristic (ROC) analysis was used to determine the optimal cut-off points of the two versions of the CMMSE. All statistics were performed using SPSS version 15 (Chicago, IL).
Results
Baseline characteristics of participants
Table 1 shows the participants' characteristics including age, gender and scores on the CMMSE and the T-CMMSE for the two groups. The mean ages of the groups with and without dementia were 81.2 (S.D. = 7.0) and 74.1 (S.D. = 7.2) respectively, but the difference was statistically significant (t =−4.01, p < 0.05). The mean number of years of education was 1.8 years (S.D. 2.9) in the group with dementia and 3.1 years (S.D. 3.5) in the group without dementia; however, they were statistically insignificant between the groups (t =1.65, p > 0.05). The total scores of the dementia group on the CMMSE and the T-CMMSE were significantly higher than those of the group without dementia (t = 14.912, p < 0.05 on the CMMSE; t = 14.189, p < 0.05 on the T-CMMSE). The mean score on the CMMSE was 25.4 (S.D. = 3.1) for the group without dementia and 12.8 (S.D. 3.7) for the dementia group, whereas the mean score on the T-CMMSE was 22.5 (S.D. 2.9) for the group without dementia and 10.3 (S.D. 3.7) for the dementia group.
Reliabilities
The intraclass correlation (ICC) (2,k) was 0.99, showing that the scale was reliable over a period of one week. For the inter-rater reliability test, the ICC (2,k) was 0.99. This indicated that the T-CMMSE had excellent inter-rater and intra-rater reliabilities.
The κ coefficient was used to examine the degree of agreement between the telephone version and the original in-person version of the CMMSE. Twenty-three common items were examined with κ coefficients ranging from 0.219 to 1 (Table 2). There were five items with excellent agreement (κ > 0.8). Those items were “Year,” “Region,” “Registration of apple,” “Registration of newspaper,” and “Recall of newspaper.” Items with substantial agreement (κ > 0.6–0.8 ≤) were “Season,” “Date,” “Day,” “District,” “Registration of train,” “Calculation,” “Recall of train,” “Name watch,” “Name pen,” and “Repeat a sentence.” Those with moderate agreement (κ > 0.4–0.6 ≤) were “Month,” “Street,” “Building,” “Recall of apple,” and Step 2 and Step 3 of the three-step command. The remaining two items were Step 1 of the three-step command and “Say a sentence/give a telephone number,” which were in fair agreement (κ ≤ 0.4). When five items of “orientation to time” were grouped into one item of “orientation to time (overall),” four items of “orientation to place” were grouped into one item of “orientation to place (overall),” three registration items were grouped into one item of “registration (overall),” and three recall items were grouped into one item of “recall (overall),” all these “overall” items got substantial agreement between the two versions (Table 2).
Level of agreement between common items of the telephone version and the original face-to-face version of the CMMSE
κ > 0.8 indicates excellent agreement; κ > 0.6–0.8≤ indicates substantial agreement; κ > 0.4–0.6≤ indicates moderate agreement; κ ≤0.4 indicates fair agreement.
Validities
Table 3 shows that the mean score of the CMMSE is 18.8 (S.D. = 7.22), and the mean score of the T-CMMSE is 16.4 (S.D. = 6.94). The Pearson correlation coefficient between the T-CMMSE total score and the CMMSE score was 0.991 (p < 0.001).
Prediction of the CMMSE and T-CMMSE total scores
*p < 0.05.
Table 3 shows the prediction of the CMMSE total score from the T-CMMSE total score and vice versa for convergent validity. Linear regression analysis used the “T-CMMSE total score,” “age,” and “education” as predictors of the CMMSE total score. Neither “age” nor “education” yielded significant results, but “T-CMMSE total score” did yield significant results and was the only predictor of the CMMSE total score (B = 1.023, S.E. = 0.021, t = 48.317, p < 0.05). The regression equation for predicting the CMMSE total score from the T-CMMSE total score was: CMMSE = 4.472 + 1.023 (T-CMMSE). Using the same method of linear regression analysis, only the “CMMSE total score” was a predictor of the T-CMMSE total score (B = 0.954, S.E. = 0.02, t = 48.317, p < 0.05). The regression equation for predicting the T-CMMSE total score from the CMMSE total score was: T-CMMSE = −3.301 + 0.954 (CMMSE).
Cut-off scores
The ROC analysis was used to determine the optimal cut-off points on the T-CMMSE scale for differentiating the participants with dementia from those without. This was done by studying the corresponding sensitivity, specificity, false positive, and false negative rates for various scores on the T-CMMSE. An ideal instrument should have both high sensitivity and high specificity and at the same time a low false positive rate and a low false negative rate. Table 4 shows that ≤16 and ≤17 were excellent optimal cut-off scores for the T-CMMSE to correctly differentiate the participants with dementia from those without. A cut-off at ≤16 yielded a sensitivity of 100%, a specificity of 96.7%, a false positive rate of 3.3%, and a false negative rate of 0%. A cut-off at ≤17 yielded a sensitivity of 96.7%, a specificity of 100%, a false positive rate of 0%, and a false negative rate of 3.3%. The cut-off at ≤16 was therefore adopted because the T-CMMSE total score was most sensitive in identifying the participants with dementia, with the highest value of total specificity at a zero negative rate for the participants without dementia. On the other hand, the results show that ≤19 was an excellent optimal cut-off score for the CMMSE to differentiate correctly the participants with dementia from those without, with a sensitivity of 100%, a specificity of 100%, a false positive rate of 0%, and a false negative rate of 0% (Table 4). Therefore, ≤16 on the T-CMMSE scale and ≤19 on the CMMSE scale appear to be the best cut-off scores for our sample of participants in the study.
T-CMMSE and CMMSE cut-off scores
Discussion
The findings of this study yielded excellent external reliabilities (both inter-rater and test-retest reliability) for the new 26-point T-CMMSE, and demonstrated an excellent convergent validity of this T-CMMSE in relation to the CMMSE for elderly participants with or without dementia, which supported the use of this tool as a valid and reliable cognitive measure for the elderly in Hong Kong.
The T-CMMSE was found to have excellent inter- and intra-rater reliabilities, with an ICC (2,k) of 0.99. Both the total score and the common item score of the T-CMMSE were comparable to those of the CMMSE. The total score of the T-CMMSE strongly correlated with that of the CMMSE (r = 0.991, p < 0.001). Almost all of the common items between the telephone version and the in-person version of the CMMSE had moderate to excellent agreement in this study. There were 21 common items that had moderate to excellent agreement, whereas only seven common items had moderate to substantial agreement in Newkirk's study (Newkirk et al., Reference Newkirk, Kim, Thompson, Tinklenberg, Yesavage and Taylor2004). The lowest degree of agreement in this study was the item “Say a sentence/give a telephone number,” which had a κ value of only 0.219. This indicated that the results for this item were very different between the two CMMSE versions. However, there was no comparison of this item between the in-person and telephone versions of the MMSE in Newkirk's study. In the in-person Cantonese version of the MMSE, the participants were asked to say a complete sentence, for example “The weather is fine today,” which was already a variation on the original version of Folstein et al. (Reference Folstein, Folstein and McHugh1975) (the original version of this item was “write a sentence.”). In the telephone version, the participants were requested to give a telephone number on which they could usually be reached. Most of the participants could say a complete sentence in the CMMSE. However, only a few of them, especially in the group with dementia, could give a telephone number in the T-CMMSE. The low κ value may be explained by the different difficulty levels of this item between the two versions.
From the convergent validity value, it was found that the telephone version and the in-person versions of the CMMSE were largely interchangeable. The conversion table between the T-CMMSE and the CMMSE was formulated according to the regression equations. The T-CMMSE score can be predicted from the CMMSE score, and vice versa. This ensures that monitoring of cognitive function will not be affected by a patient's failure to attend a clinic for follow-up. Telephone follow-up can serve as an alternative if face-to-face clinic visits are not feasible.
Another significant finding of this study was the very high sensitivity and specificity of the suggested cut-off scores for the T-CMMSE and the CMMSE in our sample. The very high sensitivity and specificity rates may result from the fact that both groups of participants were very discrete. The mean total scores on the CMMSE and the T-CMMSE for the participants without dementia were 25.4 and 22.5 respectively, whereas they were 12.8 and 10.3 respectively for those with dementia. The suggested optimal cut-off score of the CMMSE was ≤ 19, which was similar to the results of the study by Chiu et al. (Reference Chiu, Lee, Chung and Kwong1994). This was the first study to explore the cut-off score for the telephone version. In this study, the suggested optimal cut-off score of the T-CMMSE is ≤ 16, which is three points less than that of the original CMMSE (Chiu et al., Reference Chiu, Lee, Chung and Kwong1994).
Influence of education and age on the T-CMMSE score
In this study, our participants had a high illiteracy level (50.8%), which is similar to the level of 46.3% reported in the study by Chiu et al. (Reference Chiu, Lee, Chung and Kwong1994) but which is higher than the levels reported in the study by Katzman et al. (Reference Katzman1988) of 26.5% and in the study by Li et al.(1989) of 33.7%. However, the mean period of education was 2.4 years and a few participants had six or more years of education. This made it difficult to further analyze the T-CMMSE cut-off scores according to educational level. Furthermore, there were only 60 valid participants with or without dementia in this study. The inadequate sample size made it difficult to stratify the participants into age-specific and education-specific subgroups to evaluate the influence of age and education on the T-CMMSE. A large-scale study in Shanghai by Katzman et al. (Reference Katzman1988) found that education had a significant effect on the score of the Mandarin version of the MMSE; different cut-off scores are therefore suggested for different educational levels when the telephone version is used in future. In the convergent validity test, education and age were not predictors of the T-CMMSE score. Neither education nor age yielded significant results in the linear regression analysis.
Limitations
Although the study yielded significant results, there were several limitations that must be addressed. The clinical sample included patients with mild to severe dementia, but the number of subjects with severe dementia was very low in this study. The main reason for this is that all of the participants with dementia were recruited from a single center (one geriatric rehabilitation ward of a single hospital). Further study of multiple centers/hospitals with larger sample sizes which include a greater severity of dementia is suggested.
A second limitation is that there is a significant difference in age between the groups with and without dementia. Since age may influence the cognitive performance as recorded by the CMMSE, this might have caused the group without dementia to achieve a better result and so favored the performance of this group in the study. Therefore, it is recommended that a cut-off score of ≤ 16 on the T-CMMSE scale to differentiate between participants with and without dementia be used with caution and that clinical interpretation of the test results be carried out carefully.
Another shortcoming lies with the sampling method. In this study, convenience sampling was used due to limited resources that would affect the generalization of the study results to a larger population. Also, since the principal investigator of this study was also one of the raters, no blinding procedures were used.
Conclusion
The 26-point telephone version of the CMMSE (T-CMMSE) is a reliable and valid instrument for assessing cognitive impairment among the elderly in Hong Kong. It can be used as an alternative to the original face-to-face CMMSE and makes a significant contribution both to the clinical care of people with dementia and to longitudinal research in the local context. The use of the T-CMMSE can reduce the default rate because cognitive function can be assessed in both clinical and home settings by the use of telephone follow-up when face-to-face clinic visits are not feasible.
Conflict of interest
None.
Description of authors' roles
S. S. Wong designed the study, collected and analyzed the data and wrote the paper. K. N. K. Fong designed the study, supervised the data collection and analysis, and helped to write the paper.
Acknowledgments
The authors thank Mary Chu, Anna Wu and Stella Cheng for their contribution to the Cantonese translation of the assessment in this study.
