Introduction
Neuropsychological testing to determine the presence of memory and other cognitive deficits is critical in the diagnosis of the earliest stages of Alzheimer's disease (AD) and mild cognitive impairment (MCI). In addition, cognitive deficits that are captured in neuropsychological evaluations can be critical to both differential diagnosis and identifying residual strengths and specific weaknesses useful in developing care management plans.
The need to assess cognitive function in the elderly is imperative since primary care physicians often do not detect early AD and MCI, with reports that more than half of the cases are unrecognized (Valcour et al., Reference Valcour, Masaki, Curb and Blanchette2000; Kaduszkiewicz et al., Reference Kaduszkiewicz2010). Reliance on a subjective memory complaint for early detection is not advisable because it is a common complaint that is neither sensitive nor specific to objective cognitive deficits (Purser et al., Reference Purser, Fillenbaum and Wallace2006). Screening measures, such as the Mini-Mental State Examination (MMSE), are particularly insensitive to distinguishing MCI from normal controls or those with AD (Mitchell, Reference Mitchell2009). Other tools, such as the Montreal Cognitive Assessment (Nasreddine et al., Reference Nasreddine2005), Test Your Memory (TYM) (Brown et al., Reference Brown, Pengas, Dawson, Brown and Clatworthy2009), and telephone screening batteries (Hill et al., Reference Hill2005; Manly et al., Reference Manly, Schupf, Stern, Brickman, Tang and Mayeux2011), require a trained administrator or clinician and provide only a global score without information on specific profiles of deficits across different cognitive domains.
Other assessments, such as the Brief Kingston Standardized Cognitive Assessment (Hopkins et al., Reference Hopkins, Kilik, Day, Rows and Hamilton2005) and the Philadelphia Brief Assessment on Cognition (Libon et al., Reference Libon2011), require trained personnel or additional technology such as computerized cognitive assessments (Doniger et al., Reference Doniger2005; Inoue et al., Reference Inoue, Jimbo, Taniguchi and Urakami2011). However, these assessments may not be convenient to a primary care office because they may use too much clinician time, be expensive, and may have poor acceptability to seniors because of frustration or unfamiliarity with technological demands (Sano et al., Reference Sano2010).
Schofield and colleagues (Reference Schofield, Lee and Davies2003) created the Audio-Recorded Cognitive Screen (ARCS), which uses an automated audio delivery of test stimuli for cognitive tests through earphones, and is administered to unsupervised patients with written responses captured in a booklet. The ARCS assesses cognitive domains important in the assessment of elderly individuals with AD and other dementias: memory, verbal fluency, naming, visuospatial function, and attention/executive function.
Validity and reliability have been described (Schofield et al., Reference Schofield2010) in a large sample of Australian residents from community and clinic settings. High correlations were found between the ARCS domain scores (memory, verbal fluency, naming, visuospatial function, and attention/executive function) and scores from in-person evaluations. The ARCS administration requires approximately 35 minutes, which is shorter than a standard neuropsychological test battery. Other advantages of a brief audio-recorded battery include low demand on clinician or data collector and a technology with which older people are familiar. The ARCS has been described by Schofield and colleagues (Reference Schofield2010) as a “hybrid” instrument, one that encompasses the brevity of a cognitive screen and the breadth of a comprehensive battery.
In the present study we compared the ARCS to an in-person battery to determine (1) its ability to distinguish between impaired and non-impaired participants, (2) the correlation of performance within domains, and (3) the ability to correctly detect participants within diagnostic categories. We also evaluated the feasibility of use of the ARCS in an elderly cohort of individuals living in the metropolitan New York area by assessing the acceptability of administration and testing procedures. We hypothesize that the ARCS will be well accepted in the elderly cohort and it will provide a sensitive measure of diagnostic discrimination.
Methods
Participants
Participants from the Mount Sinai Alzheimer's Disease Research Center (MSSM-ADRC) who previously consented to be approached for additional studies were invited to participate. Potential participants were aged 75 years or older, and had the most recent diagnostic evaluation of AD (non-AD dementias were excluded), MCI, or normal control, with a Clinical Dementia Rating (CDR) of 0, 0.5, 1, or 2. Consistent with other studies (Weintraub et al., Reference Weintraub2009), the CDR was administered to our normal controls to confirm the absence of cognitive and functional impairment. We recruited 76 of 109 consecutively approached participants. One additional participant signed consent but was not able to complete the study because of significant hearing loss. All participants had capacity to sign consent, and were given the option of being tested at the center or in their home. Participants were classified as either normal controls (n = 25) or impaired patients (n = 51). The impaired patients had an ADRC consensus diagnosis of MCI or mild AD (90% of the MCI and AD group had a global CDR of 0.5, 1, or 2, and 10% had a CDR of 0. The mean MMSE score was 23.5). Within the impaired patients group of 51, a subgroup of 33 mildly impaired patients was identified. This mildly impaired subgroup had a Global CDR ≤ 0.5 and included 21 MCI and 12 AD patients. Of the 33 mildly impaired patients, 27 had a CDR Sum of Boxes (CDR-SOB) score = 0–2 and 6 had a CDR-SOB score = 2.5–4.
Assessments/instruments
Uniform Data Set (uds)
The UDS battery of neuropsychological tests (Weintraub et al., Reference Weintraub2009), which takes approximately 60 minutes to complete not including scoring, is administered annually to all ADRC participants by a trained research coordinator or neuropsychologist. It is collected under a separate Mount Sinai IRB-approved protocol. This battery includes the MMSE, the CDR, the 15-item Geriatric Depression Scale (GDS), and standard tests of memory (logical memory), verbal fluency (animal naming) confrontation naming (the Boston Naming test, 30-item version), visuospatial function (Digit Symbol from WAIS-III), and attention/executive function (Trail Making A and B, Digit Span from WAIS-III).
Clinical assessment
The clinical evaluation was conducted by a geriatric psychiatrist or a neurologist. A consensus diagnosis was reached with a physician (geriatric psychiatrist or neurologist) and a neuropsychologist using the information from the clinical assessment, along with the UDS neuropsychological test data. No predetermined cut-offs were used for the purpose of consensus diagnosis.
Audio-Recorded Cognitive Screen
The ARCS (Schofield et al., Reference Schofield, Lee and Davies2003) is an audio-recorded battery of cognitive tests that takes approximately 35 minutes to administer and less than five minutes to score. A high school volunteer or other non-professional provided instructions on the use of the compact disk device and headphones, but the participants were otherwise unsupervised as they listened to test instructions and wrote responses in a booklet. The 26-page booklet consists of lined pages for freehand responses, blank pages between subtests, naming pictures, and a page with yes/no responses to be circled for recognition items. All instructions, including when to stop and turn to the next page, are on the audio-recording program. The ARCS comprises the Newcastle Auditory Verbal Learning Test (NAVLT), a three-trial 12-word list learning test, which includes a ten-minute delayed recall and recognition trial; 60-second verbal fluency tests (letter, category, and action fluency), where participants write responses freehand; a ten-item naming test, which uses pictures already printed in each response booklet; a clock drawing task; and the Hunter Attentional Task, an attention/executive function test, which is similar to the Trail-Making Test. The first section of the Hunter Attentional Task-A (HAT-A) is a 40-item test with a 30-second trial in which capital letters are written next to lower case letters. The second section of the Hunter Attentional Task-B (HAT-B) consists of 40 randomly distributed circled and non-circled lower case letters that are pre-printed on the page. The participant has 30 seconds to write capital letters next to the non-circled letters, and lower case letters next to the circled letters. To assess speed of writing, the participant is instructed to write the word “table” as many times as possible in a 30-second interval on a blank page. The response booklets for the first 17 participants were scored by the high school volunteer and the senior neuropsychologist to permit assessment of inter-rater reliability. The remainder was scored by the neuropsychologist.
To assess feasibility and acceptability of testing, the ARCS workbook includes questions (some answered yes/no, others answered on a Likert scale) regarding participants’ perception of testing and their performance, including: “How do you think you performed on the test of memory and concentration?,” “Could you hear the instructions on the recording clearly?,” “Did you find the directions on the recording confusing or difficult to follow?,” “Did you give your best effort during this test?,” and finally, a question asking whether the participants had any concerns that might have affected their test performance (e.g. nerves, fatigue, and technical difficulties). The voice of the version used for this study was that of a US English speaker. All participants were given the ARCS within six months of their most recent UDS testing date.
Data collection and analyses
In order to determine inter-rater reliability, we calculated the inter-class correlation coefficient (ICC) and the corresponding 95% confidence limits for the 14 ARCS variables that were analogous to the UDS variables. Demographic and global function scores were reported for the non-impaired and impaired groups. Feasibility (i.e. acceptability) items were summarized and compared across diagnostic groups. The ARCS and UDS scores were compared across impairment groups and using t-tests. Correlations between the UDS and ARCS domain scores were calculated using Spearman correlation coefficients to account for the potential non-linear correlation between the scores. For the ARCS, an overall score for the five cognitive domains of memory (three trials initial recall and one delayed recall of NAVLT), fluency (total of letter, category, action), naming, visuospatial (clock drawing), and executive/attention (HAT-A and HAT-B) were created by totaling the scores of the variables comprising each of these tests to compare them to analogous domains on the UDS. Several ARCS variables (e.g. the recognition trial and three error indices) were not included in our analyses as there are no analogous variables in the UDS battery. We compared the Receiver Operating Characteristic (ROC) curves of these two test batteries in terms of their ability to correctly classify the non-impaired and impaired individuals. We also compared ROC curves of the ARCS, the MMSE, and the Clock Drawing Test to determine if the ARCS was better able to classify impaired and non-impaired individuals than either the MMSE, the Clock Drawing Test, or the two combined. Individual scores from the ARCS and UDS were used in the regression analyses that generated the (ROC) curves. All the analyses were performed using SAS 9.1 for PC.
Results
Inter-rater reliability
Two raters, the senior neuropsychologist and a high-school ADRC volunteer double-scored 17 of the 76 protocols, with the neuropsychologist scoring the remainder. ICC scores were calculated for each of the 14 ARCS variables. The ICCs ranged from 0.98–1.00.
Clinical and demographic comparisons
The study successfully recruited 76 elderly participants with a mean age of over 80 years with no statistical difference between the two diagnostic groups on any demographic variable. Impaired and non-impaired individuals were well matched on age (impaired mean 81.9 years ± 4.7 versus non-impaired mean 81.2 years ± 4.0), education (better than high school: 51% versus 48%), gender (66% female versus 48% male), and ethnicity (80% white for both groups). Mean MMSE scores, as expected, were significantly lower in the impaired group (23.5 ± 4.3) than the non-impaired group (28.1 ± 2.0), and CDR-SOB was higher in the impaired group (3.4 ± 3.2) than the non-impaired group (0.1 ± 0.2).
Mean scores for the impaired group on the GDS (3.0 ± 2.9) were significantly higher than the non-impaired group (1.7 ± 2.0), although both raw scores were low.
Feasibility results
Using Chi-square analyses, no statistically significant differences were found between the impaired and non-impaired groups on any feasibility question. Results are summarized in Table 1.
Feasibility question resultsa
Note: an = 75 due to missing data; bn = 73 due to missing data.
To the question, “Could you hear the test instructions clearly?,” a quarter of the sample (n = 19) responded that they occasionally or frequently had difficulty hearing the test instructions (77% of the impaired and 68% of the non-impaired individuals reported no difficulty hearing instructions). The mean age of those who reported difficulty hearing the instructions (83.4 ± 3.9 years) was significantly higher than those who did not report hearing problems (81.0 ± 4.5 years) (t = –2.20, p < 0.03). Age was not a statistically significant factor for any other feasibility question.
Performance on ARCS and UDS variables
The results of Table 2 indicate that, as expected, the non-impaired group performed better than the impaired group on all measures with the exception of digit span forward. There was a significant difference (p = 0.01) between the impaired (mean 9.7 words) and non-impaired groups (mean 11.6 words) on writing speed.
Performance on ARCS and UDS tests
Note: UDS = Uniform Data Set; ARCS = Audio-Recorded Cognitive Screen; NAVLT = Newcastle Auditory Verbal Learning Test; HAT = Hunter Attentional Task.
Correlations between ARCS and UDS domain scores are summarized in Table 3. The bold entries are purported to assess the same domains of cognitive function. Some of the scores between domains (e.g. memory and executive function) are correlated, which is consistent with other neuropsychological test studies. The correlations within the expected domains are consistently modestly related. For example, the UDS and ARCS analogous variables were significantly correlated, most in 0.4 to 0.7 range, including verbal memory, executive function, naming, verbal fluency, and attention. HAT-A/B and Trails-A/B scores are negatively correlated because HAT-A/B scores represent number of items correct, and Trails A/B scores represent time. Information regarding how the ARCS domain scores were generated, including factor analyses, may be found in Schofield et al.'s paper (Reference Schofield2010).
Spearman correlation coefficients
Note: NAVLT = Newcastle Auditory Verbal Learning Test; HAT = Hunter Attentional Task; ARCS = Audio-Recorded Cognitive Screen.
ap-value ≤ 0.01; bp-value ≤ 0.05; cn = 75 due to missing data.
Ability to correctly classify participants as impaired or non-impaired
In order to gauge our ability to classify participants as impaired (MCI or early AD) or non-impaired, as defined by the ADRC consensus conference, we analyzed the ROC curves generated from the individual UDS and ARCS variables. We first fitted two logistic regression models, one with the individual UDS variables and the other with the ARCS variables, both adjusted for age, gender, education, and ethnicity. Then we generated ROC curves based on these two models and compared the Area Under Curve (AUC) of these two ROC curves.
The AUC for UDS variables was 0.988 and for ARCS variables it was 0.972. The difference was not statistically significant (p = 0.23). The AUC results may be found in Figure 1. Examination of the graph suggests that, for the same amount of specificity, there is better sensitivity in some score ranges for the UDS battery than the ARCS battery. There is some circularity in ROC analysis for the UDS but not for the ARCS, as diagnosis does include information from the ARCS. Because the AUC for the ARCS is not significantly worse than the AUC for the UDS, the existence of circularity makes our comparison more conservative.
Receiver operating characteristic (ROC) curves. The selection of a cut-off point of 0.80 results in a 14.3% false positive rate and a 30% false negative rate. This represents a significant improvement over the literature reporting that more than half of primary care physicians miss dementia diagnosis (Valcour et al., Reference Valcour, Masaki, Curb and Blanchette2000, Kaduszkiewicz et al., Reference Kaduszkiewicz2010).
Comparison of very mildly impaired and non-impaired participants
We also compared the AUC of ROC curves for the ARCS in very mildly impaired (N = 33 with a global CDR ≤ 0.5 and a CDR-SOB score ≤ 2 in 27 participants and a CDR-SOB score ≥ 2.5 in six participants) and non-impaired individuals for the commonly used screening instruments MMSE and Clock Drawing Test, controlling for age, education, and gender. The AUC for the ARCS full model was significantly better than for the MMSE alone (0.956. vs. 0.758; p = 0.018), the clock drawing alone (AUC = 0.717; p = 0.0002), or the MMSE and Clock Drawing Test combined (AUC = 0.792; p = 0.0035). Based on this new classification, we regenerated ROC curves for individual UDS and ARCS variables. The AUC for the ARCS and UDS variables were not significantly different (ARCS AUC = 0.956 vs. UDS AUC = 0.987; p = 0.13).
Discussion
This study describes the use of the ARCS in an elderly cohort of individuals with a mean age of above 80 years with and without MCI and AD. With the exception of one subject (not included in the analyses) who had to discontinue due to inability to hear adequately the audio-recorded instructions, all other participants were able to complete the ARCS satisfactorily. The ARCS data could be collected without the help of a professional in a variety of clinic and community settings. In general this supports the feasibility of assessment through the ARCS. A number of participants reported that the instructions were confusing but most reported giving their best effort. A majority of the non-impaired participants and half of the impaired participants perceived their test performance as satisfactory or better.
The ARCS took approximately 35 minutes to administer and five minutes to score. A high-school level research volunteer was adequately and reliably able to administer and score the ARCS. Although supervision is not needed for ARCS administration, in the current study, the volunteer or other staff member stayed with each subject while they completed the ARCS, initially to help adjust the volume on the compact disc player, and then to observe participants’ behavior. Since the participants were not alone, we were not able to assess whether participants “cheated” on the test (e.g. turn back to previous pages), a behavior that might be more likely if participants knew they were not being observed and something that might be important for assessment in group settings. Schofield and colleagues (Reference Schofield2010) reported very little systematic cheating (fewer than six over 2,600 administrations). The current results suggest that the ARCS may be useful in settings that rely on personnel with little or no professional training in neuropsychology.
One challenge to the feasibility of using ARCS in this elderly sample was hearing loss. Approximately 26% of the sample reported either occasional or frequent difficulty hearing the audio-recorded instructions. Although there were no differences in reported hearing difficulty between the impaired and non-impaired groups, there were more reports of difficulty hearing the instructions among the older participants. Though Schofield's studies do not specifically address hearing loss, our results may reflect the fact that the current sample (mean age 81.7 years) is considerably older than Schofield's original Australian sample (mean age was 59 years in one study (Schofield et al., Reference Schofield2010) and 74 years in another (Schofield et al., Reference Schofield, Lee and Davies2003)). The ARCS administration in the current study involved the use of a standard portable compact disc player and headphones. The ARCS may also be used with other devices, including MP3, iPod, or computer audio. In future studies it may be useful to investigate the use of hearing-assistive technologies (such as FM systems and infrared systems that provide better amplification or direct amplification into the listener's hearing aid).
Performance on tests of individual cognitive domains within the ARCS battery, which included measures of verbal memory, verbal fluency, attention/executive function, naming, and visuospatial function, was significantly correlated with corresponding tests on the in-person battery. Although inter-correlations were evident among many tests, we were pleased that the UDS and ARCS analogous domains were significantly correlated. These results expand the findings of Schofield and colleagues (Reference Schofield2010) by observing a significant correlation between the ARCS and an in-person assessment in a significantly older sample.
Using the overall score of all cognitive domains, the ARCS’ ability to distinguish between impaired (mild AD or MCI) and non-impaired diagnostic groups was found to be as good as the in-person test battery. These results were maintained when we examined a subset of mildly impaired patients whose CDR was ≤ 0.5, suggesting that the ARCS is useful in identifying mildly impaired individuals and distinguishing them from normal controls. The advantage of the ARCS over an in-person evaluation is that it reduces workload on the examiner and may be administered and scored by those with little training. The performance of our normal controls on the in-person battery was comparable to that reported by Weintraub and colleagues (Reference Weintraub2009) on the same tests in a sample of over 3,000 community-dwelling normal controls, suggesting that our normal controls were relatively similar to this large heterogeneous sample. Our normal controls comprised those recruited through flyers and community lectures or may be spouses of demented patients. It is possible that some normal controls came to the ADRC due to memory concerns, but a discussion of this is beyond the scope of our paper.
The ARCS was able to distinguish between diagnostic groups better than commonly used screening instruments, even in very mildly impaired patients. These results are consistent with Schofield's (Reference Schofield2010) description of the ARCS as a “hybrid” instrument with the time benefits of a cognitive screen in conjunction with meaningful clinical information, such as domain scores, associated with longer neuropsychological test batteries.
To our knowledge this is the first use of the ARCS outside Australia and the United Kingdom and the high correlation of audio and in-person assessment across critical cognitive domains suggests that common instruments can be developed and used across international settings. The ARCS may be particularly useful to medical generalists who are commonly required to make diagnostic decisions because of shortage of mental health specialists in low and middle income countries (Bruckner et al., Reference Bruckner2011) as well as in remote, rural areas of the United States (Smalley et al., Reference Smalley, Yancey, Warren, Naufel, Ryan and Pugh2010).
One limitation of the current study is that the sample was a self-selected group of research participants who may have been predisposed to participate in additional studies. A limitation of any study with more than one time point of cognitive testing is possible practice effects. The fact that administration of the ARCS and UDS was spaced one to six months apart raises the possibility of practice effects on similar subtests for the shorter time frames or cognitive decline in the longer time frames. Furthermore, the sample size was small and was characterized in a tertiary medical setting. This is a highly educated, primarily Caucasian cohort that limits the generalizability of the findings. The normal controls were devoid of comorbid conditions such as movement disorders and depressive disorders that may be common in primary care practice. While this is a limitation regarding generalizability of the findings, it is often the healthiest elderly individuals who seek very early diagnosis of cognitive problems that may be particularly relevant to the primary care setting.
The current study uses ARCS raw scores as norms were not available during the data collection phase of our study. Since our data were initially collected, a website for the ARCS was created (Schofield, Reference Schofield2012). The website provides a scoring template/calculator for entering raw scores to generate demographically adjusted scores for individual tests, specific cognitive domains, and the global ARCS score. Norms are provided for age, education, and gender, making interpretation of ARCS results more meaningful. The template/calculator can provide a quick indication of whether a patient is performing below expectation for their age on the domains most pertinent to dementia evaluations such as memory, verbal fluency, and executive functioning.
In summary, the ARCS provided excellent discrimination between normal and mildly impaired elderly individuals, and was able to evaluate cognitive performance within specific key domains. It was well tolerated and the relative ease of administration suggests that it may be useful in a wide range of settings.
Conflict of interest
None.
Description of authors’ roles
M. Sewell designed the study, collected the data, and wrote the paper. X. Luo analyzed the data and reviewed the paper. J. Neugroschl reviewed and edited the paper. M. Sano supervised study design, assisted in data analyses, and edited and supervised the writing of the paper.
Acknowledgment
The authors would like to express their gratitude to Talia Sandwick, BA, for her work with subject recruitment and data collection. The authors are also grateful to the National Institutes of Health for their grant no. NIH U01 P50-AG005138-28.
