Introduction
A definite diagnosis of Alzheimer's disease can only be made through histopathology after a brain biopsy or autopsy (McKhann et al., Reference McKhann, Drachman, Folstein, Katzman, Price and Stadlan1984). Even the syndromal diagnosis of probable or possible dementia requires a comprehensive assessment of several areas of functioning, which include – but go beyond – cognition. However, it is possible to make use of screening procedures that specify the likelihood of cognitive impairment by comparing a patient's scores to established norms (Cullen et al., Reference Cullen, O'Neill, Evans, Coen and Lawlor2007). Screening measures can be broadly grouped in two categories (Lonie et al., Reference Lonie, Tierney and Ebmeier2009): (i) well-established tests that cover a limited range of cognitive functions, such as the Mini-mental State Examination (MMSE, Folstein et al., Reference Folstein, Folstein and McHugh1975); and (ii) screening measures that are somewhat broader in scope, which more explicitly tackle executive functioning, such as the Rowland Universal Dementia Assessment Scale (RUDAS; Storey et al., Reference Storey, Rowland, Basic and Conforti2004). Tests like the MMSE are more suitable for use in primary care settings, whereas the broader instruments demand a little more time to administer and more training to interpret (Lonie et al., Reference Lonie, Tierney and Ebmeier2009). A third category of screening tests is the informant tests, which have gained popularity for the assessment of cognitive impairment in older adults due to their discriminant value (Ritchie, Reference Ritchie, Jacoby, Oppenheimer, Dening and Thomas2008), such as the Informant Questionnaire on Cognitive Decline in the Elderly (IQCODE, Jorm and Korten, Reference Jorm and Korten1988), which provides proxy information regarding cognitive functioning and decline.
Screening procedures estimate the likelihood of cognitive impairment by comparing patients’ scores to established norms (Cullen et al., Reference Cullen, O'Neill, Evans, Coen and Lawlor2007). As a consequence, screening is fallible, especially if it has been developed for a different setting than the one in which it is being used. For instance, an instrument might be useful as a brief assessment tool in primary care, but not be suitable for community screening if it has not been adapted for telephone or informant scoring (Cullen et al., Reference Cullen, O'Neill, Evans, Coen and Lawlor2007). The memory clinic is a specific clinical environment in which the performance of cognitive screening instruments can be readily assessed. In one common model, people referred to the memory clinic have already been seen by their primary care physician (general practitioner) at which time a question has arisen about their cognitive function. They may have current symptoms suggesting cognitive impairment and a history suggesting cognitive decline; and they may have clinical signs of cognitive impairment. These symptoms and signs might have been observed by the patient themselves, by their family or friends, or by their primary care physician. After completion of assessment procedures, people referred to a memory clinic are generally assigned to three main categories: subjective memory complaints; mild cognitive impairment (MCI); or dementia. Thus, cognitive screening instruments used in this context must distinguish people with dementia from the two other groups. From a psychometric perspective, this is arguably a much more difficult task than distinguishing people with dementia from normal older people. The memory clinic is thus a challenging environment in which to test the utility of screening measures.
The principal goal of this study was to compare the diagnostic accuracy of three brief instruments in memory clinic attendees. Two of the instruments were based on face-to-face clinical assessment (SMMSE and RUDAS), whereas the third used proxy information from an informant (IQCODE). Our hypothesis was that scores obtained on the two clinical test instruments (RUDAS and SMMSE) and the informant report instrument (IQCODE-SF) would provide equivalent diagnostic accuracy in distinguishing individuals with dementia from those without dementia as indicated by comparing the areas under their ROC curves.
Methods
Study design and participants
This was a cross-sectional study conducted at the multidisciplinary memory clinic of the Royal Brisbane and Women's Hospital (RBWH) in Brisbane, Australia. Brisbane is a sub-tropical city of approximately 2 million residents. Patients were referred to the memory clinic mainly by their primary care physicians (general practitioners) and their attendances were fully reimbursed by the Australian national health insurance system, which provides universal coverage to citizens and permanent residents. Participants or their substitute decision-makers gave informed consent and the RBWH and University of Queensland Human Research Ethics Committees approved the study. This study complies with the Standards for Reporting of Diagnostic Accuracy (STARD) steering committee guidelines (Bossuyt et al., Reference Bossuyt2003).
Instruments
Three screening instruments in common use were investigated in this study. The Standardized Mini-mental State Examination (SMMSE, Molloy et al., Reference Molloy, Alemayehu and Roberts1991) was developed to overcome some limitations associated with the original version of the MMSE (Folstein et al., Reference Folstein, Folstein and McHugh1975), and includes administration and scoring guidelines, as well as timing requirements for each question. The SMMSE assesses orientation to time and space, attention and calculation, language and immediate and delayed recall, with scores ranging from 0 to 30. Administration time is dependent upon patient characteristics, but averages 10 minutes. Psychometric properties for the MMSE have been extensively analyzed and suggested cut-off points vary according to sample characteristics and settings, with a cut-off of 23/24 being the most prevalent in dementia studies (Mitchell, Reference Mitchell2009), with a similar cut-off suggested for the SMMSE (Ward et al., Reference Ward, Caro, Kelley, Eggleston and Molloy2002).
The Rowland Universal Dementia Assessment Scale (RUDAS; Storey et al., Reference Storey, Rowland, Basic and Conforti2004) was developed in Australia to overcome cross-cultural constraints imposed by other screening instruments. The RUDAS contains six items (body orientation, fist-palm alternation, copying a diagram of a cube, judgment in relation to crossing a busy road, animal names, and recall of grocery items presented at the beginning of the test). The six items measure executive function, praxis, gnosis, recent memory, and category fluency. The RUDAS generates a total score of 0 to 30 points, with a suggested cut off point of 22/23 for cognitive impairment (Storey et al., Reference Storey, Rowland, Basic and Conforti2004).
The Informant Questionnaire on Cognitive Decline in the Elderly (IQCODE, Jorm and Korten, Reference Jorm and Korten1988) is one of the most used informant-rated instruments for cognitive screening. It was developed with the goal of assessing change in cognitive functioning over ten years, through a five-point response scale ranging from “much better” to “much worse”. Higher scores correspond to greater cognitive decline, with a recommended cut point for clinical significance between 3.3 and 3.6 out of 5 for community-living adults (Jorm, Reference Jorm2004). The IQCODE short form (IQCODE-SF), used in this study, has 16 items. Being an informant-rated instrument, the IQCODE score may be influenced by factors others than the patient's actual cognitive change, such as informant anxiety or carer burden.
Anxiety, mood, and general functioning were measured through the Geriatric Anxiety Inventory (GAI; Pachana et al., Reference Pachana, Byrne, Siddle, Koloski, Harley and Arnold2007), the Geriatric Depression Scale short version (GDS; Sheikh and Yesavage, Reference Sheikh, Yesavage and Brink1986), the Zarit Burden Interview (ZBI; Bédard et al., Reference Bédard, Molloy, Squire, Dubois, Lever and O'Donnell2001), the Modified Barthel Index (MBI; Shah, Reference Shah and Salek1998), and the Disability Assessment for Dementia scale (DAD; Gelinas et al., Reference Gelinas, Gauthier, McIntyre and Gauthier1999). The GAI is a 20-item questionnaire that was developed to assess dimensional anxiety in older adults. Presence or absence of anxiety symptoms is ascertained for the past week (Pachana et al., Reference Pachana, Byrne, Siddle, Koloski, Harley and Arnold2007). The GDS also uses a one week time frame to inquire about the presence or absence of depressive symptoms (Yesavage et al., Reference Yesavage1982). The short version used in this study has 15 items (Sheikh and Yesavage, Reference Sheikh, Yesavage and Brink1986). The ZBI short form has 12 items rated from 0 (never) to 4 (nearly always). It was developed from the Zarit Burden Interview (Zarit et al., Reference Zarit, Orr and Zarit1985) and is a measure of caregiver burden. The Barthel Index (Mahoney and Barthel, Reference Mahoney and Barthel1965) was created to assess dependency in ten areas of basic functioning, such as personal hygiene and bladder control. Limitations surrounding the quantification of impairment lead to the development of the modified Barthel Index (Shah, Reference Shah and Salek1998), which is a ten-item clinician-rated instrument that measures severity of impairment in basic activities of daily living on a five-point response scale. The DAD is a 40-item informant-rated instrument that measures functional impairment, including basic and instrumental activities of daily living, on a dichotomous response scale.
Procedure
Following referral from their primary care physician, each patient and informant dyad was seen by a research nurse who administered a standardized battery of measures, including the SMMSE, RUDAS, GAI and GDS to the patient and the IQCODE-SF, ZBI, MBI and DAD to the informant. As they are part of a standardized protocol, the measures were always administered in the same order (sociodemographic and medical information; cognitive functioning; mood and anxiety measures; and informant measures). The RUDAS was the first cognitive test measure to be administered, followed by the SMMSE. Screening blood tests and neuroimaging studies (CT or MRI brain scans) were ordered. Approximately two weeks later the patient was seen by an experienced senior physician (geriatrician, psychiatrist or neurologist), who conducted an independent examination of the patient and review of the existing information, including laboratory and neuroimaging studies as well as the standardized protocol previously collected, in order to provide a provisional diagnosis. Cases were subsequently presented by the physician at a multidisciplinary meeting. In selected cases, additional consultations, functional neuroimaging studies (SPECT, FDG-PET brain scans), additional laboratory studies (electroencephalograms, lumbar punctures) and formal neuropsychological testing were undertaken. After all investigations were complete, the consultant to whom the patient had been assigned arrived at a final diagnosis, using DSM-IV-TR criteria (American Psychiatric Association, 2000). For the purposes of this study the patients have been divided into those diagnosed with dementia and those without dementia. Thus, the latter category contains participants with a variety of problems including subjective memory complaints and MCI.
Analyses
Categorical variables were assessed through the use of contingency tables and the χ2 test. As many of the continuous variables had asymmetric distributions, the non-parametric Wilcoxon rank sum test was applied. Receiver-operating characteristic (ROC) analyses were then performed using the dichotomized diagnosis of dementia as provided by the consultant physician, and moderated by the multidisciplinary team meeting, as the gold standard, to compare the diagnostic performance of the SMMSE, RUDAS, and IQCODE-SF. The establishment of cut-off points was made through the Youden (J) index (Youden, Reference Youden1950). The J index is a function of sensitivity and specificity that ranges from 0 to 1, and can be defined as the vertical point on the curve that is more distant from chance, depicted by the diagonal line (Perkins and Schisterman, Reference Perkins and Schisterman2006). The main outcome measures for test performance were defined as the sensitivity (Se), specificity (Sp), positive (PPV) and negative predictive values (NPV), likelihood ratio (LR), and area under the curve (AUC) (Metz, Reference Metz1978). Ninety-five percent confidence intervals (95% CI) were calculated for those parameters. The utility index, defined as the clinical utility of a given test when both the accuracy and the occurrence are considered (Mitchell, Reference Mitchell2009), was also calculated for the three instruments. Positive utility index or rule-in accuracy was calculated using sensitivity and positive predictive value, whereas negative utility index or rule-out accuracy was obtained through specificity and negative predictive value (Mitchell, Reference Mitchell2009). Once the cut-off points for the instruments were established, κ agreement was calculated against the diagnosis provided by the consultant (Landis and Koch, Reference Landis and Koch1977).
The SMMSE and the RUDAS follow the same scoring procedure, where higher scores indicate better cognitive functioning, whereas the IQCODE-SF is based on a scoring method where higher scores reflect more impairment. To improve interpretability of the ROC plots, the SMMSE and RUDAS scores were inverted (Zweig and Campbell, Reference Zweig and Campbell1993). However, all other results for the SMMSE and the RUDAS are reported using their traditional scoring systems of 0 to 30. Significance levels were established using two-tailed tests and a critical value of α of 0.05. All analyses were performed using STATA 11.0 (StataCorp, 2009).
Results
Data were consecutively collected for 36 months, beginning in June 2007. During this period 243 patients were referred to the memory clinic. Of these, 35 patients (approximately 14%) were unaccompanied and were excluded from this study as several of the measures required completion by an informant. The likelihood of coming unaccompanied to the Memory Clinic decreases as the patient becomes more impaired. Thus, patients who came alone were less impaired (as measured by the SMMSE) than the ones who came with a carer. However, there were no significant differences in age, marital status, educational level or gender between those who came alone and those who came with an informant. Another four patients were excluded from this analysis due to missing data either on the SMMSE or the RUDAS. We thus studied 204 patients who attended the memory clinic with a caregiver or another person who was able to act as an informant. Sociodemographic and clinical characteristics are summarized in Table 1. The study sample was composed of 114 females (56%) and 90 males (44%) with a mean age of 76.90 years (SD = 8.85; range 49–97) all of whom were non-institutionalized. Of these, 123 (63%) were married or in a de facto marital relationship and 46 patients (23%) were widowed. Seventy-nine patients had graduated grade school (39%) and 79 (39%) had graduated high school, reflecting limited historical educational opportunities. Twenty participants (approximately 10%) had English as a second language. Around 40% of the patients had current cardiovascular disease and almost half (48%) suffered from hypertension.
Sociodemographic and clinical characteristics of participants (N = 204)
SMMSE = Standardized Mini-mental State Examination; RUDAS = Rowland Universal Dementia Assessment Scale; IQCODE-SF = Informant Questionnaire on Cognitive Decline in the Elderly, Short form; GAI = Geriatric Anxiety Inventory; GDS = Geriatric Depression Inventory; DAD = Disability Assessment for Dementia; MBI = Modified Barthel Index; ZBI = Zarit Burden Interview.
Of the 204 patients, 152 (75%) were diagnosed as having dementia (49% Alzheimer's disease; 13% mixed dementia; 6% vascular dementia; 2% dementia with Lewy bodies; 2% frontotemporal dementia; 3% other types). Twenty-seven patients (13%) received a diagnosis of MCI or the related diagnosis of cognitive impairment no dementia (CIND), and the remaining diagnoses included subjective memory complaints (3%), and anxiety or depression (5%). The relatively high prevalence of dementia is partially explained by the exclusion of patients who came to the memory clinic without a carer. Mean SMMSE score was 22.28 (SD = 5.33, range 4–30), mean RUDAS score was 20.56 (SD = 4.79, range 3–30), and mean IQCODE-SF score was 4.18 (SD = 0.57, range 3–5). Scores on the SMMSE and the RUDAS were highly correlated (ρ = 0.73, p < 0.0001), and both instruments were also correlated with the IQCODE-SF (ρ = ‒0.50, p < 0.0001 and ρ = ‒0.49, p < 0.0001, respectively).
Except for age (Z = ‒4.746, p < 0.0001) and marital status (χ2 = 12.549, df = 4, p = 0.014), there were no statistically significant differences in sociodemographic characteristics between patients diagnosed with dementia and patients who were not. In this way, widowed patients were more likely to have a diagnosis of dementia and to be older. When entered as a dichotomous variable (partner/no partner), marital status was not significantly associated with dementia (χ2 = 1.434, df = 1, p = 0.231). There were no significant differences between groups in gender (χ2 = 0.001, df = 1, p = 0.986), educational level (χ2 = 5.628, df = 5, p = 0.344), and having English as a first language (χ2 = 1.285, df = 1, p = 0.257). However, there were significant differences between patients who had English as a first language and those who did not on the SMMSE (Z = 8.73, p < 0.01), but not on the RUDAS (Z = 1.05, p = 0.30). As expected, patients diagnosed with dementia scored significantly lower on all cognitive screening measures (SMMSE: Z = 7.002, p < 0.0001; RUDAS: Z = 7.126, p < 0.0001; IQCODE-SF: Z = ‒5.909, p < 0.0001), as well as lower on self-reported anxiety (GAI: Z = 3.571, p < 0.001) and depressive (GDS: Z = 3.091, p < 0.01) symptoms, and functional abilities (DAD: Z = 4.313, p < 0.0001). There were no differences between groups on basic activities of daily living (MBI: Z = 1.324, p = 0.19), and caregiver burden (ZBI: Z = ‒0.578, p = 0.56).
ROC analyses indicated that both the SMMSE (area under curve, AUC = 0.82, 95% CI = 0.76, 0.87, p < 0.0001) and the RUDAS (AUC = 0.83, 95% CI = 0.77, 0.88, p < 0.0001) had a similar diagnostic utility, followed by the IQCODE-SF (AUC = 0.77, 95% CI = 0.71, 0.83, p < 0.0001). Figure 1 illustrates the ROC curves for SMMSE, RUDAS, and IQCODE-SF, compared with the diagnosis of dementia provided by the consultant. The AUCs for each instrument were compared through McNemar's χ2 test, indicating that there were no significant differences (χ2 = 2.57, df = 2, p = 0.28).
Comparison of the areas under the ROC curve for SMMSE, RUDAS and IQCODE, using dementia status as the gold standard (N = 204).
SMMSE = Standardized Mini-mental State Examination (inverted score); RUDAS = Rowland Universal Dementia Assessment Scale (inverted score); IQCODE-SF = Informant Questionnaire on Cognitive Decline in the Elderly, Short form. Obs.: SMMSE and RUDAS scores were inverted before entering the analysis.
In the present study, optimal cut-off points were <24 for the SMMSE (J index: 0.61; Se: 0.83, 95% CI: 0.76, 0.88; Sp: 0.73, 95% CI: 0.59, 0.84), <21 for the RUDAS (J index: 0.56; Se: 0.66, 95% CI: 0.58, 0.74; Sp: 0.90, 95% CI: 0.78, 0.96), and >4.1 for the IQCODE-SF (J index: 0.39; Se: 0.72, 95% CI: 0.64, 0.79; Sp: 0.67, 95% CI: 0.53, 0.79). Because the prevalence of dementia was high in the present sample, PPVs were very good or excellent for the three instruments (SMMSE: 0.9; RUDAS: 0.95; IQCODE-SF: 0.87), whereas NPVs were considerably lower, around 0.50. Conventional positive LRs were systematically small for the SMMSE and the IQCODE-SF and better for the RUDAS, and these values improved significantly when LRs were weighted for prevalence. Positive utility indices were good and the negative ones were slight or fair. Using these cut-off points, the SMMSE correctly classified 80% of the patients, while the RUDAS and the IQCODE-SF correctly classified 73% and 71% respectively. Table 2 summarizes these results. Agreement rates between the screening instruments and the diagnosis provided by the consultant were good, ranging from 68% to 76% (Table 3).
ROC curve analysis for dementia diagnosis, for SMMSE, RUDAS, and IQCODE (N = 204)
ROC = receiver operating characteristics; AUC = area under the curve; CI = confidence interval; SMMSE = Standardized Mini-mental State Examination; RUDAS = Rowland Universal Dementia Assessment Scale; IQCODE-SF = Informant Questionnaire on Cognitive Decline in the Elderly, Short form; PPV = Positive Predictive Value; NPV = Negative Predictive Value; CLR+ = Conventional Positive likelihood ratio; CLR- = Conventional Negative likelihood ratio; WLR+ = Weighed Positive likelihood ratio; WLR- = Weighed Negative likelihood ratio.
Kappa agreement rates (95% CI) between screening instruments and consultants’ diagnosis (N = 204)
CI = confidence interval; SMMSE = Standardized Mini-mental State Examination; RUDAS = Rowland Universal Dementia Assessment Scale; IQCODE-SF = Informant Questionnaire on Cognitive Decline in the Elderly, Short form; all results p < 0.0001.
Discussion
The main goal of this study was to compare the predictive value of three screening instruments in distinguishing individuals with dementia from those without dementia among community-dwelling older adults referred to a memory clinic. As referred patients, these individuals had a high prior probability of having dementia. Selecting a screening instrument requires a carefully considered compromise between sensitivity and specificity. Failure to identify dementia when it is present delays the opportunity for psychoeducation and therapeutic intervention (Maeck et al., Reference Maeck, Haak, Knoblauch and Stoppe2008), whereas an incorrect diagnosis attaches an illness label with potential for widespread negative consequences (Holsinger et al., Reference Holsinger, Deveau, Boustani and Williams2007). However, it can be argued that instruments with high specificity are preferable in the memory clinic environment.
Our findings indicate that the SMMSE and the RUDAS were equally accurate in predicting dementia in memory clinic attendees, with AUCs of approximately 0.83. These values are somewhat lower than the ones reported by Rowland and colleagues (Reference Rowland, Basic, Storey and Conforti2006), who reported AUCs of 0.91 and 0.92 for the MMSE and the RUDAS, respectively. However, it is relevant to highlight that those results were obtained by excluding participants with MCI, and that when these were included in the analysis the area under the curve decreased to 0.88 and 0.89. As for the IQCODE-SF, our analyses indicated an AUC of 0.77, a value superior to the one recently obtained by Hancock and Larner (Reference Hancock and Larner2009) of 0.71, although this difference might be within the range expected by chance, as the confidence intervals for both areas under the curve overlap. Our hypothesis was supported by our findings, as there were no statistically significant differences between the three instruments. However, among the three instruments the RUDAS had the highest specificity and the highest positive likelihood ratio.
The RBWH memory clinic attracts older people referred by their general practitioners for specialist assessment from the greater Brisbane metropolitan area. Because no patient payment is required, financial accessibility is good. Thus, the findings should be generalizable to other memory clinics accepting undifferentiated general practice referrals.
The psychometric values obtained in this study were slightly different from the ones previously reported, namely the optimal cut off points and the area under the curve, reflecting the characteristics of this specific sample. This is a feature observed in other studies that used institutionalized or clinical populations. For example, it is suggested that a score between 3.4 and 4 on the IQCODE-SF is used with inpatient adults, as opposed to the 3.3–3.6 range for community living adults (Jorm, Reference Jorm2004). Although our sample was composed of community-dwelling adults, all the participants were previously identified as having memory impairments either by their caregiver or their family, or by self-report. As for the areas under the curve, a recent study that assessed the diagnostic accuracy of the MMSE for diagnosing dementia in a sample of patients diagnosed with Parkinson's disease (Hoops et al., Reference Hoops2009) reported an area under the curve below those usually reported in the literature (AUC = 0.76, 95% CI: 0.67, 0.85), further indicating that the attributes of the setting constrain the diagnostic accuracy of the instrument.
Some limitations to this study should be acknowledged. First, patients who came to the memory clinic without an informant were excluded from the analyses, thus increasing the prevalence of dementia and eventually constraining the efficiency parameters (Zweig and Campbell, Reference Zweig and Campbell1993). Although the exclusion of these patients inflated the prevalence rates, these are not dramatically different from the ones reported in previous studies, especially if one considers that the majority of the patients who attend the memory clinic are older adults. Second, as this was a cross-sectional study there is the possibility of some misclassification between MCI and dementia, which might have been resolved by longitudinal follow-up. Nevertheless, diagnoses were provided following comprehensive assessment moderated by expert consensus. Finally, the sample size was insufficient to allow examination of scale performance by dementia disease type.
Although the diagnostic accuracy as provided by the area under the curve was similar for the three instruments, both the RUDAS and the IQCODE-SF displayed clear advantages within this specific setting. The RUDAS proved to be a particularly useful instrument to initially assess patients from a culturally diverse background, as those patients who did not have English as a first language performed similarly to those for whom English was their mother tongue. Additionally, the agreement rates between the screening instruments and the diagnoses provided by the specialist physicians were good (Landis and Koch, Reference Landis and Koch1977). While the RUDAS is a helpful tool to collect self provided information with memory clinic attendees, especially when multiculturalism is present, the IQCODE-SF proved to be an equally efficient instrument for the collateral assessment of cognitive decline. Considering that patients with cognitive impairment are often incapable of providing accurate information regarding their cognitive functioning, it is highly relevant to possess alternative methods of cognitive screening, in this case using proxy information. We recommend use of the RUDAS in multicultural situations in which a test with high specificity for dementia is needed.
Conflict of interest
None.
Description of authors' roles
Daniela Gonçalves designed the study, analyzed the data and wrote the paper. Elizabeth Arnold and Kana Appadurai collected data and edited the paper. Gerard Byrne designed the study, obtained funding, collected data, supervised data analysis and edited the paper.
Acknowledgments
Daniela Gonçalves was supported by a scholarship from the Fundação para a Ciência e Tecnologia, Portugal (SFRH/BD/30226/2006) and by a private benefactor. Gerard Byrne was supported by the Royal Brisbane and Women's Hospital Research Foundation, the Australian National Health and Medical Research Council (456182), and the U.S. Alzheimer's Association (IIRG-07-59015). The authors thank Suhail Doi for his perceptive comments, which helped to improve the final version of the paper.
The authors also acknowledge the essential contributions of Amanda Cox, Julie-Anne Jones, Claire McCabe, Vicki Naumann, Liane Nye, Robert O'Sullivan, Donna Spooner, Catherine Yelland and our late colleague Jonathon Chalk.
The authors thank the RBWH Memory Clinic patients and their families.
