INTRODUCTION
The impact of multiple sclerosis (MS) on neuropsychological (NP) function is well recognized, in that nearly half of all MS patients show deficiency on NP testing (Heaton, 1985; Rao et al., 1991a). Research also indicates that cognitive impairment gives rise to vocational disability (Benedict et al., 2005a; Rao et al., 1991b), social impairment (Benedict et al., 2001b; Rao et al., 1991b), and distress in persons caring for MS patients (Knight et al., 1997). Yet NP testing is not routinely employed in MS clinics for a variety of reasons. NP impairment is not a diagnostic criterion for MS diagnosis (McDonald et al., 2001). Board certified or well-trained clinicians with MS expertise are not available to many MS Centers. Most importantly, NP testing is often costly and time consuming. Combined, these factors impede the implementation of routine NP testing for MS patients.
In 2001, we reasoned that routine NP testing of MS patients might be more feasible if it were conducted in a more standardized and cost-effective manner. In collaboration with the Consortium of MS Centers, we convened a panel of neuropsychologists and psychologists with expertise in MS from the United States, Canada, United Kingdom, and Australia. The objectives of the meeting were to (a) propose a minimal NP examination for clinical monitoring of MS patients and research and (b) propose strategies for improving NP assessment of MS patients. Literature on MS-related cognitive dysfunction was reviewed with an eye toward both psychometric factors (reliability, discriminative validity) and practical considerations (e.g., alternate forms, brevity of administration). The meeting culminated in a publication (Benedict et al., 2002) summarizing the panel discussion and a 90-minute NP battery called the Minimal Assessment of Cognitive Function in MS (MACFIMS). Included were seven tests covering the five cognitive domains most commonly impaired in MS patients (Table 1). Tests emphasizing information processing speed, working memory, and unaided recall of recently presented information were judged to be most sensitive to the cerebral pathology of MS (Archibald & Fisk, 2000; Beatty, 1996; DeLuca et al., 1998; Demaree et al., 1999; Fischer, 1988; Rao et al., 1989a; Rao et al., 1989b). Reviewed research also showed that tests in other domains are also, if less frequently, affected in MS patients, especially those involving generative fluency (Beatty & Monson, 1994), spatial processing (Vleugels et al., 2000), and conceptual reasoning (Arnett et al., 1997; Beatty & Monson, 1996). While an encouraging first step, the authors acknowledged that the MACFIMS battery, merely a collection of recommended tests, had yet to be validated as a single battery in a large, diverse sample of MS patients.
Tests included in the Minimal Assessment of Cognitive Function in MS (MACFIMS) battery
In the course of preparing this manuscript, we searched the literature for studies that had included any of the individual NP tests comprising the MACFIMS battery to evaluate MS patients. Included in Table 2 are those studies comparing MS patients and healthy controls and reporting data that permit calculation of effect size using the well known Cohen d statistic (Cohen, 1988). Table 2 shows that of 28 between-group comparisons only 2 have effect sizes less than d = .5. Most show medium or large effects. Ten comparisons show a between-group difference of 1.0 SD or greater. Such strong effects are not surprising considering that the MACFIMS tests were chosen based largely on evidence for reliability and discriminative validity.
Effect sizes derived from previously published studies using the MACFIMS battery tests to discriminate MS patients from healthy controls
Here, for the first time, we examined the concurrent and construct validity of the entire MACFIMS battery in a large sample of 291 prospectively accrued MS patients. We predicted a priori that each of the individual tests that comprise the MACFIMS battery would discriminate MS patients from healthy controls, and that effect sizes and frequencies of impairment would parallel previous findings in the literature. We then explored the construct validity of these tests using principle components analysis. Because episodic memory was distinguished from processing speed in previous literature reviews (Benedict et al., 2002), we predicted that such an analysis would show a clearly discernible episodic memory component. Finally, we predicted that the MACFIMS tests would show good ecological validity regarding their associations with vocational disability status.
METHODS
Participants
Studied were 291 patients diagnosed with clinically definite MS (McDonald et al., 2001) by treating or research neurologists. Patients entered the study for one of three reasons: participation in brain imaging research (n = 77, 27%), routine monitoring of cognitive function (n = 106, 36%), or referral for evaluation of a specified management problem related to suspected cognitive impairment (n = 108, 37%). All patients met the following inclusion and exclusion criteria: (a) age 18 or older; (b) fluent in English; (c) able to provide informed consent to all procedures; (d) no neurological disorder other than MS; (e) no psychiatric disorder (American Psychiatric Association, 1994) other than mood, personality, or behavior change following the onset of MS; (f) no other medical condition that might influence cognition; (g) no history of developmental disorder (e.g., ADHD, learning disability); (h) no history of substance or alcohol dependence, or current abuse; (i) no motor or sensory defect (e.g., corrected near vision of at least 20/70) that might interfere with cognitive test performance; and (j) no relapse and/or corticosteroid pulse within four weeks of assessment. All research participants signed informed consent forms approved by institutional review panels prior to participating in the study.
Mean patient age (±SD) was 45.4 ± 8.9 years. On average, patients completed 14.4 ± 2.4 years of education. The majority were Caucasian (n = 277 or 95%) and women (n = 227 or 78%), consistent with the MS population which is primarily female (Jacobs et al., 1999). Scores derived from the Expanded Disability Status Scale (Kurtzke, 1983), which assess neurologic and physical disability, were available for 186 patients; the mean EDSS for this subset of patients was 3.0 ± 1.8, representing mild and moderate impairment. Most patients (n = 200 or 69%) had relapsing-remitting (RR) rather than secondary-progressive (SP) (n = 78 or 27%), progressive-relapsing (n = 6 or 2%), or primary-progressive (n = 7 or 2%) course.
Patients were compared with 56 healthy controls with a mean age of 43.8 ± 9.5 years and 14.6 ± 2.2 years of education. The majority of the controls were Caucasian (n = 50, or 89%) and women (n = 42 or 75%). The differences between the MS and control groups on these demographic variables did not reach significance by ANOVA or χ2 test.
MATERIALS AND METHODS
Each patient underwent NP testing in accordance with recent consensus panel recommendations (Benedict et al., 2002). The MACFIMS includes the following tests: Controlled Oral Word Association Test (COWAT; Benton et al., 1994), Judgment of Line Orientation Test (JLO; Benton et al., 1994), California Verbal Learning Test, second edition (CVLT-II; Delis et al., 2000), Brief Visuospatial Memory Test-Revised (BVMT-R; Benedict, 1997), Paced Auditory Serial Addition Test (PASAT; Gronwall, 1977), Symbol Digit Modalities Test (SDMT; Smith, 1982), and the Sorting Test from the Delis-Kaplan Executive Function System (DKEFS; Delis et al., 2001). In accordance with the position paper (Benedict et al., 2002), we employed Rao's adaptations (Rao, 1991) of the PASAT and SDMT.
The COWAT was administered in the standard manner, following the method of Arthur Benton (Benton et al., 1994). In successive one-minute trials, participants generated as many words as possible, beginning with each of three designated letters. The dependent measure was the total number of correct words over the three trials.
The JLO required participants to identify the angle defined by two stimulus lines from among those defined by a visual array of lines covering 180 degrees. Both oral and pointing responses were allowed. The dependent variable was the total number of correct responses over 30 items.
The CVLT-II and BVMT-R are both learning and memory tests with similar formats. Both require discrete exposures to new material that are followed by the participant's unaided recall immediately after presentation. There is a 25-min interval following the final learning trial, after which participants are asked to recall the information again without further exposure to the to-be-learned material. Delayed recall is followed by a yes or no, forced-choice recognition task. For the CVLT-II there were five learning trials. Examiners read 16 words and asked participants to repeat as many words as possible. The entire word list was repeated each time. For the BVMT-R, the stimulus material was a matrix of six visual designs, held before the participant for 10 sec. Participants were asked to render the designs using paper and pencil, taking as much time as needed for reproduction. Each design received a score of 0, 1, or 2 based on accuracy and location scoring criteria. There were three free-recall trials followed by 25-min delayed recall and yes or no recognition trials. In this study, we considered the following measures for each memory test: total recall over all learning trials (Total Learning or TL) and recall after the delay interval (Delayed Recall or DR, i.e., long delayed recall in the case of the CVLT-II).
The PASAT included 60 trials presented at inter-stimulus intervals of 3 (PASAT-3) and 2 (PASAT-2) sec. The dependent measure was the number of correct responses from each of the two trials. For the SDMT, participants were presented with a series of nine symbols, each paired with a single digit in a key at the top of an 8.5 × 11-inch sheet. The remainder of the page presented a pseudo-randomized sequence of symbols. Participants responded by voicing the digit associated with each symbol as quickly as possible. The dependent measure was the number of correct responses in 90 sec.
The DKEFS Sorting Test was employed to assess higher executive function. Participants were presented with six cards each depicting a single word. The cards varied in many ways, allowing conceptual sorting in accordance with at least eight different principles (e.g., card shape, card color, semantic association among words). This was a timed test with 4 min allowed for each of two card sets. We recorded the total number of correct sorts (CS) and calculated the verbal description score (DS), which was based on the abstractness and accuracy of the sort descriptions.
Depression was assessed using the Beck Depression Inventory—Fast Screen for Medical Patients (BDI-FS; Beck et al., 2000). The BDI-FS emphasizes thought content (e.g., negative self-evaluation or guilt) and mood (e.g., dysphoria) and avoids assessment of vegetative signs that can occur in medical illness without depression. This test was recently validated in a MS sample (Benedict et al., 2003).
Finally, we assessed vocational status by asking patients about their present work activities (Benedict et al., 2005a). There were two methods used to classify patients as either employed or disabled. A conservative approach to classification required that working patients be gainfully employed with pay, full time, without demotion, reported reprimands, or loss of pay due to MS related problems. Disabled patients were required to be receiving formal disability benefits from either public (e.g., social security disability income) or private (e.g., commercial benefits) sources. The liberal approach to classification required that working patients be gainfully employed with pay at least 20 hours per week. Disabled patients for this scheme were required to be receiving either formal disability as described above, or to be unemployed for reasons reported by them or informants to be disease related.
Procedure
The research participants (including all HCs) were contacted by mail or were approached during the course of their usual clinical care at an MS center. The clinical participants were patients undergoing evaluation either for routine monitoring of cognitive functioning, or for specific clinical reasons (e.g., concerns about driving or work capacity, disability evaluation, differential diagnosis of cognitive impairment vs. depressive disorder, etc). The tests were administered individually by a psychologist, trained assistant, or a graduate student under the supervision of a board-certified (ABPP-CN) neuropsychologist. The entire test battery, including tests of personality, required 90–120 min of face-to-face testing time.
Statistical Analysis
All statistical analyses were performed using SPSS 13.0. Group differences in continuous variables were assessed using unpaired Student's t-test or univariate ANOVA with alpha set at p < .01. The chi-square test was used to determine differences in categorical variables. We considered limiting the study to RR and SP patients, because there were few patients with progressive-relapsing and primary-progressive course. However, one of the goals of the study was to test the validity of the MACFIMS in a sample that represents all MS patients receiving clinical treatment. Therefore, normal versus MS comparisons included these rarer disease courses.
For descriptive purposes, we examined effect sizes using Cohen's d statistic, which is the difference between means divided by the pooled SD. Z scores were calculated for each individual NP test based on the normal control group. Impairment for a single test was defined as a z score < −1.5. NP impairment generally was defined as a defect on two or more test measures.
Principal Components Analysis (PCA) with varimax rotation was used to assess the construct validity of the MACFIMS battery. Because mean NP test scores and levels of impairment were found to differ by course for all NP tests in the MACFIMS battery, separate PCAs were conducted for RR and SP patients to evaluate whether the underlying components (dimensions) in the MACFIMS battery vary according to disease course. All 11 test variables were included in both models. Components were identified using a criteria of Eigenvalue >1.0. Each component was required to consist of at least two variables with loadings of .5 or more as well as explain at least 15% of the variance in the model. The NP variables most representative of a specific component were considered to be those with loadings [gE ] .500 for that variable.
Estimates of the risk of being disabled or unemployed associated with NP impairment were obtained from odds ratios calculated using unconditional logistic regression models. Like the PCA, this analysis did not include the 13 progressive-relapsing and primary-progressive patients. One of the criteria for inclusion in final stepwise regression testing was that NP variables significantly discriminate employed from disabled patients after adjusting for covariates that could also impact on vocational status. Thus, unconditional logistic regression models were first used to assess the association between individual NP tests and employment/disability status after adjusting for age, sex, years of education, depression status (depressed: BDF-FS Score >3; not depressed BDI-FS ≤ 3), and disease course (SP vs. RR). NP tests that were not significant after adjusting for covariates in the preliminary analyses were not entered into the final models. In order to further limit the number of predictor variables in the final models, we selected only one measure from each test (i.e., COWAT, JLO, CVLT-II-DR, BVMT-R-DR, DKEFS-DS, PASAT-3, SDMT) in the final stepwise model, specifically, the measure that showed the highest frequency of impairment. Vocational status (disabled/employed) served as the dependent variable for each model. Two sets of models were run using two different definitions of vocational status (conservative and liberal, see Methods). In preliminary analyses we noted that only 13% of SP patients were employed, leaving only 6 employed SP patients for analysis. Therefore, we also ran a separate set of models for RR patients only, using the same criteria and following the same steps outlined earlier.
RESULTS
Group Differences and Frequency of Impairment
Table 3 shows the group data for the MACFIMS tests. ANOVA showed significant between-group effects for all tests with effect sizes ranging from 0.49 for JLO to 1.31 for SDMT. Frequencies of impairment in MS patients (Figure 1) were as follows: COWAT 13.1%, JLO 22.3%, CVLT-II-TL 29.6%, CVLT-II-DR 34.4%, BVMT-R-TL 54.3%, BVMT-R-DR 56.0%, PASAT-3 27.4%, PASAT-2 27.5%, SDMT 51.9%, DKEFS-CS 15.8%, DKEFS-DS 25.8%. Based on the standard of impairment on two or more tests, 59.5% of MS patients were found to be impaired.
Group data comparing MS patients and normal healthy controls
Frequencies of impairment using a cut-off z score of −1.5 for the healthy comparison group.
MS patients also showed greater degrees of depression on the BDI-FS (MS mean = 3.6 ± 3.3, Normal mean = 0.9 ± 1.5; p < .001). However, there were no significant correlations between BDI-FS and NP test performance in either group (largest MS r = −.15 for SDMT).
ANOVAs also showed that all 11 NP variables significantly discriminated RR and SP patients at a probability of p < .01. In each case, RR patients performed better than SP patients. Effect sizes ranged from 0.42 for JLO to 0.92 for SDMT.
Principle Components Analysis
The PCAa
Because of likely correlation between NP measures, oblique rotation was also pursued. This approach resulted in the same frequency of variable overlap seen with varimax rotation. In addition to significant overlap between test scores, the oblique rotation resulted in components that did not meet the a priori criteria that a component must explain at least 15% of the variance in the model. In at least one instance, a variable that had loaded over 0.5 on a single component and less than 0.5 on the other components with varimax method now loaded as almost 0.5 on multiple components. Both rotation methods resulted in overlap among test scores, but with the varimax rotation we met the a priori criteria described above.
Results of Principle Components Analysis
The PCA yielded two components for SP patients: a general component encompassing language, processing speed, working memory, executive function, and a more specific memory component. Each explained over 15% of the total variance. Overall, 67.9% of the variance was explained by the two components combined. The only variable that loaded onto both components (i.e., loading >.5) was the SDMT.
Association with Vocational Status
When the more conservative definition of vocational status was used, 5 NP tests (COWAT, CVLT-DR, PASAT-3, SDMT, DKEFS-DS) were significant in the logistic models that included only a single NP test adjusted for covariates (i.e., age, sex, education, course, and depression), with odds ratios ranging from 3.07 for SDMT to 6.02 for COWAT (p < .01). The final model, which included covariates and the five NP variables found to be significant in preliminary analyses, retained CVLT-DR and PASAT-3 (Table 5).
Logistic regression models for MACFIMS tests associated with increased risk of vocational disability in all MS patients
When the liberal definition of vocational status was used, the following measures were found to be significant in preliminary logistic models that included only a single NP test: COWAT, CVLT-II-DR, BVMT-R DR, PASAT-3, SDMT, DKEFS-DS. After adjusting for covariates, odds ratios for these individual NP tests ranged from 2.05 (p = .03) for SDMT to 3.25 (p < .01) for COWAT. In the final model predicting disability/employment, which included all six of the NP variables found to be significant in the preliminary logistic regression analyses, after controlling for covariates, DKEFS-DS and CVLT-II-DR were retained (Table 5).
These analyses were repeated including only the RR patients. In the final model predicting the conservative disability/employment status for RR patients, there were 3 retained NP tests: DKEFS-DS (odds ratio 3.06, p = .10), CVLT-II-DR (odds ratio 2.85, p = .07), and PASAT-3 (odds ratio 3.57, p = .03). In the liberal model, BVMT-R-DR (odds ratio 2.25, p < .05) and DKEFS-DS (odds ratio 2.65, p < .05) were retained.
DISCUSSION
The primary purpose of this study was to investigate the validity of the MACFIMS battery in a single, prospectively-accrued sample. All seven NP tests discriminated MS patients from healthy controls and RR from SP patients. Effect sizes ranged from medium to very large, and tests emphasizing processing speed and memory were most sensitive to illness. PCA identified a single episodic memory component in both RR and SP samples, and components corresponding to either processing speed and working memory, or a more general component involving these constructs, as well as spatial and executive functions. For most of the MACFIMS tests, poor cognitive performance was associated with vocational disability; in logistic regression models accounting for all NP tests, tests emphasizing processing speed, verbal memory, and executive function were most predictive of vocational status, even after controlling for the effects of age, education, sex, depression, and disease course.
As predicted, the MACFIMS tests significantly discriminated MS patients and controls and showed a frequency of impairment consistent with the previous MS literature. The effect sizes were roughly parallel with those obtained previously using the same test measures (Table 2). Like the seminal work of Rao and colleagues (Rao et al., 1991a), we found that roughly half of MS patients exhibit cognitive impairment and that impairment is most common on tests that emphasize processing speed, working memory, and recent or episodic memory. Differences in the frequency of impairment can be expected to differ across studies because of variability in testing methods and sample composition. This sample, which includes patients undergoing routine monitoring of cognitive function, patients referred to address specific clinical problems (e.g., disability, cause of established memory disorder), as well as research volunteers, is representative of patients attending a hospital-based MS center. Lower frequencies of impairment would be expected in a population-based or community sample, and higher frequencies might be found in samples of patients seen only for specified clinical purposes.
In evaluating the PCAs, it is important to bear in mind that the MACFIMS tests were chosen to represent a minimal record of cognitive function in MS, and not to assess cognitive functioning in a comprehensive manner (Benedict et al., 2002). Therefore, these results should not be viewed as a representation of cognitive functioning in MS. Also, the MACFIMS includes only one test of language, the COWAT, which equally emphasizes lexical search and response speed. Indeed, this test is also commonly referred to as a test of executive ability. However, we did expect a priori that a single component representing episodic memory would be seen in the data, a hypothesis borne out by the PCA models for both RR and SP patients.
Fewer components emerged for the SP sample, which may be related, in part, to the low number of participants. It is also possible that as the disease progresses there is less variation in the NP presentation of MS. Although cognitive impairment can be found early in the disease, most studies show that patients with progressive disease are more neuropsychologically affected than RR patients (Comi et al., 1995; Heaton, 1985; Kraus et al., 2005). In a more demented state, impairment becomes more common across all NP tests, which makes identification of underlying components difficult.
It is interesting that the SDMT had a split loading in the PCA for SP patients, and nearly so in the RR analysis. This test is regarded to be very strongly correlated with brain atrophy in recent MS studies (Benedict et al., 2004; Christodoulou et al., 2003) perhaps because, like the PASAT (Lockwood et al., 2004), performance depends on rapid communication between widely dispersed cortical regions via long white matter tracts. However, in our previous work, the SDMT was correlated not only with general measures of brain atrophy but also regional temporal lobe atrophy (Benedict et al., 2005b). The explanation for this may lie in the fact that SDMT performance is also enhanced by remembering new associations between symbols and numbers. This multimodal aspect of the SDMT may account for its very high sensitivity in the present study.
The ecological validity of NP testing is increasingly emphasized in the MS literature (Goverover et al., 2006; Higginson et al., 2000; Beatty et al., 1995a). Evaluations are often requested to determine whether or not a patient has a legitimate claim for disability benefits. In this study, logistic regression analysis determined the extent to which these MACFIMS tests are associated with a dichotomous distinction of disabled versus employed. This is a difficult psychosocial construct to measure. Our method was based on patient and informant reported status ascertained at the time of NP testing. Whereas we have no reason to suspect that these reports were biased, our measurements do not account for why a person is disabled. To account for this, we eliminated patients for whom the decision to stay out of the work force is probably multifactorial (e.g., homemaker). Nevertheless, there were strong associations between NP testing and vocational outcomes, even after controlling for the influence of demographics, depression, and disease course. While depression and disease course also were significantly associated with employment status, the associations between NP testing and vocational outcomes remained even after accounting for the effects of these variables. As would be expected, verbal memory and reasoning tests demonstrated the strongest associations with vocational status. While the SDMT was the NP test showing the most frequent impairment in this sample, the PASAT-3 was the processing speed measure showing closest alignment with vocational disability. One explanation for this may be that the PASAT also involves numerical processing, a skill required in many occupations.
In addition to our concerns about the measurement of vocational disability in this study, there are other methodological limitations that merit comment. The study was cross-sectional in design, and whereas we can say something about the validity of these NP tests at a single point in time, their predictive validity, in a strict sense, is unknown. Quantitative assessment of neurological status using the EDSS was missing in about one-third of the sample, which means we could not control for this factor in the logistic regression analysis. However, we were able to control for disease course (RR vs. SP), which has been shown to be associated with EDSS (Jacobs et al., 1999). While the patients were screened for gross deficiency in upper extremity motor control, we did not administer the copy trial of the BVMT-R, and it is conceivable that mild motor dysfunction may have contributed to performance on this test. The effect size for this particular measure was large, and may have been inflated by drawing problems due purely to manual or motor dysfunction. In future work, we plan to study the contribution of motor control to BVMT-R performance using standardized tests of manual speed and dexterity. One of the advantages of these MACFIMS tests lies with their potential for using manifold variables from a single procedure. This is especially true of the CVLT-II and the DKEFS. Because the number of variables was intentionally limited to control for type 1 error, future studies might examine these tests in greater detail.
The above notwithstanding, we find support for the validity of the MACFIMS battery in MS patients. These tests demonstrate construct validity, discriminate MS patients from healthy controls and RR from SP patients, and are associated with greater risk for vocational disability. Future work will investigate the test-retest reliability of these tests in a large sample, and the longitudinal course of NP dysfunction in MS using these measures.
ACKNOWLEDGMENTS
The author acknowledges the support of the clinical and administrative staff from Jacobs Neurological Institute and Buffalo Neuroimaging Analysis Center. Some of the research was supported, in part, by an unrestricted educational grant from Biogen Idec.
