INTRODUCTION
Although cognitive complaints are frequent in cancer, systematic cognitive assessment is not routine in clinical practice, and mild alterations are seldom identified (Inouye et al., Reference Inouye, Foreman and Mion2001; Pisani et al., Reference Pisani, Redlich and McNicoll2003). These problems can be mild and subtle compared to delirium and dementia and thus more difficult to detect, understand, and treat, but they may still be harmful to patients' ability to make decisions regarding their own treatment, as well as interfering with their roles within the family structure, at work, and in society, consequently reducing their quality of life. The lack of easily applied validated assessment tools probably contributes to the failure of identifying these problems. There are several neuropsychological tests, simple and complex, computerized and not computerized, that have been developed to assess single or multiple cognitive domains; however, there is little information about their clinical validity and reliability in patients with cancer.
In addition, there is no consensus on which cognitive domains and tests are most clinically relevant and feasible to use (Kurita et al., Reference Kurita, Lundorff and Pimenta2009). Most of the tests do not have cutoff points based on specificity and sensitivity (Kurita et al., Reference Kurita, Lundorff and Pimenta2009) and may not be useful in a clinical context (dos Santos et al., Reference dos Santos, de Mattos-Pimenta and Kurita2014). Therefore, it seems justified to study the psychometric properties of a number of potentially relevant neuropsychological tests. Based on our experience and on studies regarding the cognitive effects of cancer, chronic pain, and opioids (Sjøgren & Banning, Reference Sjøgren and Banning1989; Banning & Sjøgren, Reference Banning and Sjøgren1990; Banning et al., Reference Banning, Sjøgren and Kaiser1992; Sjøgren et al., Reference Sjøgren, Banning and Christensen1994; Sjøgren,Reference Sjøgren1997; Sjøgren et al., Reference Sjøgren, Olsen and Thomsen2000; Sjøgren, Reference Sjøgren2006; Kurita & de Mattos-Pimenta, Reference Kurita and de Mattos-Pimenta2008; Kurita et al., Reference Kurita, Sjøgren and Ekholm2011), we decided to assess the validity and reliability of the Continuous Reaction Time (CRT) test, the Finger Tapping Test (FTT), the Digit Span Test (DST), the Trail Making Test – part B (TMTB), and the Mini-Mental State Examination (MMSE) in patients with cancer.
METHODS
Design, Sample and Settings
Our sample was composed of 80 patients with metastatic cancer treated by specialized palliative care services at Rigshospitalet, Copenhagen University Hospital (66 outpatients) and at Herning Region Hospital (14 home-based patients) from July of 2010 to November of 2015. A control group of 81 healthy subjects (related or not related to patients) was assessed to allow for validity and reliability comparisons. Power calculations were performed based on the standard deviations with the CTR test and the FTT from previous studies (Sjøgren & Banning, Reference Sjøgren and Banning1989; Banning & Sjøgren, Reference Banning and Sjøgren1990; Sjøgren et al., Reference Sjøgren, Banning and Christensen1994; Reference Sjøgren, Olsen and Thomsen2000). Assuming a power of 0.80 and a type I error of 0.05, the analyses indicated that a sample size of 50 patients would be sufficient to observe a mean difference of 4.0–8.5 ms on the CRT test and 5.1–5.7 taps on the FTT.
The inclusion criteria for our patients were: a diagnosis of metastatic cancer, aged ≥ 18 years, a Karnofsky Performance Status (KPS) index between 40 and 100%, at least 6 years of schooling, fluency in the Danish language, and stable medications for 4 days prior to first assessment (opioids, adjuvant analgesics, etc.). The exclusion criteria were: brain metastases, hepatic dysfunction, dementia, psychosis and/or delirium, misuse of drugs or alcohol, and last alcohol intake ≤ 24 hours.
The inclusion criteria were the same for healthy subjects with the exception of cancer diagnosis. Exclusion criteria were: treatment with medications with psychotropic properties, history of mental or physical disease or other chronic disease that could interfere with cognitive function, misuse of drugs or alcohol, and last alcohol intake ≤ 24 hours before testing.
Our study was submitted to the Danish Region Midtjylland Ethics Committee (no. M-20090221) and approved by the Danish Data Protection Agency for Health Science Research (no. 2007-58-0010/2012-58-0006). It was also conducted in accordance with the principles of the Declaration of Helsinki.
Neuropsychological Testing
Patients filled out and answered an identification form (sociodemographic data, cancer disease, KPS score, and treatments); two questions about sleep (“Do you feel rested?” and “How many hours of sleep last night?”); a visual numeric scale to assess “pain now” (0–10); the Hospital Anxiety and Depression Scale (HADS) (Zigmond & Snaith, Reference Zigmond and Snaith1983); and the European Organization for Research and Treatment of Cancer Quality of Life–Cancer 30 (EORTC QLQ–C30) (Aaronson et al., Reference Aaronson, Ahmedzai and Bergman1993). Healthy subjects filled in an identification form (socio-demographic data) and answered the two questions about sleep. Below are brief descriptions of each of the neuropsychological tests:
-
■ The CRT test measures sustained attention. Through headphones, 100 auditory signals (500 Hz, 90 dB) were delivered to the patient at random intervals (2–5 s) over a period of 10 min. Subjects were instructed to press a button as soon as they heard the sound. A computer registered the time from emission of the sound signal to pressing of the button. Scores were summarized in milliseconds using the 10th, 50th, and 90th percentiles, where the 10th percentile represented the fastest and the 90th the slowest values (Elsass, Reference Elsass1986). More prolonged times meant worse performance.
-
■ The DST assesses attention, concentration, and working memory (Wechsler, Reference Wechsler1981). Subjects were asked to repeat series of numbers of increasing length, both forward (direct) and backward (reverse). The score was calculated by the number of correct answers. Scores ranged from 0 to 14, with higher scores meaning better performance.
-
■ The FTT measures psychomotor speed. Participants were asked to use the second finger of each hand to make five 10-second trials tapping a key attached to a device that recorded the number of taps (Peters, Reference Peters1976). The score was calculated by the number of taps, with higher scores meaning better performance.
-
■ The TMTB has been utilized to assess visual scanning speed, motor function, attention, and mental flexibility. The test has two parts (A and B). We only applied part B in our study. Numbers and letters must be connected in a crescent and alternated sequence on a sheet of paper. The time spent to correctly complete the test was recorded. Shorter times meant better performance (Reitan, Reference Reitan1958).
-
■ The MMSE measures orientation to time and place, registration of words, attention, calculation, and word recall, as well as language and visual construction (Folstein et al., Reference Folstein, Folstein and McHugh1975). Scores can range from 0 to 30, with scores below 27 considered indicative of cognitive impairment.
Assessment Procedure
Eligible patients were identified during a medical appointment at the palliative care service. Patients' relatives were recruited via indication by the patients, and people not related to the patients were recruited from inside and outside of the hospital environment. They all received information regarding the study and signed written informed consent forms at the scheduled assessment appointment.
Patients and controls on stable medications were assessed and scheduled for a second assessment within an interval of 2 to 7 days (test and retest). The HADS and EORTC QLQ–C30 questionnaires were applied only to patients and answered once within a period of 2 days before or after the first assessment. Each cognitive test was explained in the standardized manner by the research assistant, and subjects were briefly pretested to ensure that they understood the tests. The control group was assessed regarding sociodemographic data and tested in the same way as the patients.
Data Analysis
Analyses were performed using SAS software (v. 9.1; SAS Institute, Cary, NC). The significance level was set at p < 0.01. The psychometric properties of the neuropsychological tools were analyzed with respect to the following:
-
■ The construct validity of the MMSE was planned to be investigated through factorial analysis to evaluate the instrument items and domains that should comprise the cognitive state in patients with metastatic cancer.
-
■ Criterion validity was examined by Spearman's correlation test between patients' results on the neuropsychological tests and sociodemographics, hours of sleep, and results on the HADS and EORTC QLQ–C30, and pain, which were applied at the first assessment. Our hypothesis was based on the idea that certain variables would present a significant positive or negative correlation with test results. Further, we analysed if a subject feeling rested at the time of the test would be associated with cognitive performance (Wilcoxon's test).
-
■ Discriminant validity was analyzed by a comparison between patients and controls in the first assessment (Wilcoxon's test). The hypothesis was that patients would have worse cognitive performance. The sensitivity and specificity of each instrument were also analyzed in order to determine the cutoff values for the instruments. The area under the ROC curve was produced for each cognitive test to evaluate an ability to predict MMSE score ≤ 26 (cognitive deficits) during the first assessment (test).
-
■ Reliability was analyzed using the intraclass correlation coefficient (ICC) (a two-way random single measure) for all of the neuropsychological instruments. Values were interpreted in accordance with the criteria described by Landis and Koch (Reference Landis and Koch1977).
Finally, as power calculations have only been performed to determine the detectable differences between patients with advanced cancer and healthy controls based on standard deviations on the CTR test and the FTT in earlier studies, we decided to carry out a post-hoc analysis with a sample power of α = 0.05 for each neuropsychological test.
RESULTS
Eighty patients were tested, of whom 48 did not have alterations in terms of medication use and were returned to be retested after a mean interval of 5.2 days (SD = 1.4, range = 2–7 days). A total of 81 subjects comprised the control group (41 patients' relatives and 40 people not related to the patients), of whom 71 were retested (32 and 39, respectively) after a mean interval of 4.7 days (SD = 1.3, range = 3–7). The majority of patients were treated with opioids (n = 61), anticonvulsants (n = 45), antidepressants (n = 31), corticosteroids (n = 19), and hypnotics (n = 19). Some received chemotherapy (n = 16) and radiotherapy (n = 1).
Patients and controls had similar sociodemographic characteristics but differed with regard to income (p = 0.00019) and sensation of rest (p = 0.0054) (Table 1). These differences were expected since it is not uncommon that patients with cancer have difficulty to carry on professional activities, and, as well, cancer disease/comorbidities/treatments frequently cause fatigue and exhaustion. Therefore, no specific statistical adjustment for these variables was done.
Comparison between patients and controls at the first assessment
1 DKK = 7.5 €.
Construct Validity
It was not possible to estimate a model for factorial analysis of the MMSE because of the skewed response distributions for many items.
Criterion Validity
The significant correlations followed the expected direction (positive or negative), but the majority of correlations were weak. Correlations were significant between MMSE, age, and income; CRT, nausea/vomiting, and KPS; DST, schooling, KPS score, and insomnia; and TMT, age, schooling, income, and KPS score (Table 2). In addition, men had better performance on the FTT (dominant p = 0.00000, nondominant p = 0.00011) than women, while no differences were observed regarding cohabitation and sensation of rest (data not shown).
Correlations between patients' cognitive results and variables that could interfere with cognitive function at first assessment (criterion validity)
Sleep in hours.
CF = cognitive functioning; CRT = Continuous Reaction Time; DST = Digit Span Test; EF = emotional functioning; FD = financial difficulties; FTT = Finger Tapping Test; MMSE = Mini-Mental State Examination; QoL = quality of life; PF = physical functioning; RF = role functioning; SF = social functioning; TMTB = Trail Making Test – part B.
Discriminant Validity
The cognitive test results for patients' relatives were compared to those for people not related to patients so as to ensure that they had similar performance to and could be included in control group. They were collapsed, as no significant differences were observed (Table 3). Patients had slower performance on the CRT 50th (p = 0.00483) and 90th (p = 0.00030) percentiles and FTT dominant hand (p = 0.00306) compared to controls (Table 3). Regarding sensitivity and specificity, only the DST and TMTB seemed to predict cognitive deficits in agreement with MMSE scores; however, our results were fair, at best (ROC curves ≤ 0.73).
Cognitive performance of the groups (discriminant validity)
Control = relative + not relative, CRT = Continuous Reaction Time; DST = Digit Span Test; FTT = Finger Tapping Test; MMSE = Mini-Mental State Examination; TMTB = Trail Making Test – part B.
Reliability
The patient group presented almost perfect and substantial conformity between the two cognitive assessments on 5 of the 10 measures: the TMTB, FTT dominant and nondominant hand, DST forward, and DST backward. The control group demonstrated almost perfect and substantial consistency/conformity in 8 of the 10 measures (Table 4). Approximately 45% of the control group had higher MMSE scores at the second assessment, which may reflect the poor consistency for this test due to a learning effect.
Agreement between cognitive results of test–retest in the two groups (reliability)
CRT = Continuous Reaction Time; DST = Digit Span Test; FTT = Finger Tapping Test; ICC = intraclass correlation coefficient; MMSE = Mini-Mental State Examination; TMTB = Trail Making Test – part B.
Post-Hoc Power Calculation
The calculations based on the present results demonstrated the following power for each test: CRT 10th percentile = 0.59; CRT 50th = 0.85; CRT 90th = 1.00; FTT dominant = 0.82; FTT nondominant = 0.72; FTT difference between hands = 0.12; TMTB = 0.72; MMSE = 0.52; DST forward = 0.09; and DST backward = 0.28.
DISCUSSION
Several factors may interfere with a routine assessment of symptoms in patients with advanced cancer, including barriers related to caregivers and patients (Ingham et al., Reference Ingham, Moore, Phillips and Cherny2015; Mitnick et al., Reference Mitnick, Leffler and Hood2010). Although assessment of cognitive function in patients with cancer is not standard-of-care in palliative care, the increase in life expectancy of patients with cancer due to improvement in diagnostics and treatments is beginning to draw the attention of clinicians to symptoms that may be controlled and/or prevented in order to preserve and improve quality of life, and even survival.
The tools that are being used to assess cognitive function in the cancer population were originally developed to screen or assess for a specific central nervous system dysfunction or disease, differentiating those with cognitive impairment from those without this disease/condition (a “normal population”) (Harvey, Reference Harvey2012). For several years, our research group has used neuropsychological testing in patients with cancer, especially to assess cognitive status before/after medications and to compare groups of patients and analyze the factors that may cause deficits (Kurita et al, Reference Kurita, Lundorff and Pimenta2009; Sjøgren & Banning, Reference Sjøgren and Banning1989; Sjøgren et al., Reference Sjøgren, Banning and Christensen1994; Reference Sjøgren, Olsen and Thomsen2000; Kurita & de Mattos-Pimenta, Reference Kurita and de Mattos-Pimenta2008; Kurita et al., Reference Kurita, Sjøgren and Ekholm2011; Reference Kurita, Ekholm and Kaasa2016). Due to the fact that many of the tools we have used for research were not designed to assess patients with cancer, a validation process is necessary to ensure their suitability and feasibility. Moreover, preestablished cutoff points may be difficult to apply to patients with cancer due to the fluctuating nature of their health condition.
In this present study, analysis of the 5 neuropsychological tests in 80 patients with metastatic cancer in specialized palliative care showed good reliability and some positive outcomes regarding examination of validity, indicating that the tests individually or combined have the potential to assess cognitive function in this population.
The attempt to examine the construct validity of the MMSE by confirmation of the different cognitive domains that comprise the instrument failed. There was a skewed response distribution for many of its items represented by high scores, indicating no cognitive deficits for most patients. This finding may indicate that the tool failed to detect milder cognitive deficits due to its lack of sensitivity, which is confirmed by the lack of statistical power found in the post-hoc power calculations. Considering the calculations, it would be necessary to involve a huge sample to demonstrate differences between groups that could capture mild cognitive deficits. The idea of validating this tool in patients with cancer is based on the premise that its original purpose was patient screening for dementia and other psychiatric conditions that involved severe cognitive impairment (Folstein et al., Reference Folstein, Folstein and McHugh1975). However, it has been widely used to assess cognitive function in several other health/disease conditions, including in patients with cancer (Hjermstad et al., Reference Hjermstad, Loge and Kaasa2004; Kurita et al., Reference Kurita, Sjøgren and Ekholm2011; Kurita et al., Reference Kurita, Ekholm and Kaasa2016). In addition, we found only one study regarding validation of the MMSE in patients with cancer that confirmed the good psychometric properties of the Greek version of the instrument by other validation methods (Mystakidou et al., Reference Mystakidou, Tsilika and Parpa2007). Thus, the main criticisms of this instrument are that it is not sensitive enough to detect milder but clinically relevant cognitive deficits and that it is subject to significant learning effects (Lange et al., Reference Lange, Rigal and Clarisse2014), although its simple and brief application may seem to offer an attractive advantage.
The criterion validity analyzed by correlations between the scores on the neuropsychological tests and variables related to patients' characteristics and symptoms demonstrated very few and weak relationships. Age, schooling, income, and KPS score seemed to be the main factors with some degree of relationship to at least some, but not all, of the tests. Similar to our study, a validation study with 90 Brazilian patients with cancer in palliative care observed a lack of significant correlations between the TMTB and such symptoms as pain, fatigue, depression, anxiety, and sensation of rest (dos Santos et al., Reference dos Santos, de Mattos-Pimenta and Kurita2014). On the other hand, a study analyzing the psychometric properties of the MMSE in 103 Greek patients with cancer found significant moderate correlations with age, schooling, marital status, metastasis, and treatments (Mystakidou et al., Reference Mystakidou, Tsilika and Parpa2007). This is a complex area that is still under investigation, but in the present study the absence of strong significant correlations with such factors as age, schooling, and income, previously considered factors to be reliably associated with cognitive performance (Glisky, Reference Glisky and Riddle2007; Tucker-Drob et al., Reference Tucker-Drob, Johnson and Jones2009), was a surprising finding. It was also unexpected that no significant association between a sensation of rest and performance on the cognitive tests was found, since 41% of the patients felt unrested. Fatigue and exhaustion have formerly been described during progression of cancer and its treatments, and they have been associated with cognitive dysfunction (Janelsins et al., Reference Janelsins, Kesler and Ahles2014).
Concerning discriminant validity, some measures of the CRT test (50th and 90th percentiles) and the FTT (dominant hand) were able to discriminate patients from controls, where patients had worse cognitive performance. In previous studies regarding opioid treatment, worse performance of adult patients with cancer was registered on the CRT test and the FTT when compared with healthy controls or less sick patients or those on different treatments. These studies also showed that differences are not always observed on all measures of the tests (Sjøgren & Banning, Reference Sjøgren and Banning1989; Banning & Sjøgren, Reference Banning and Sjøgren1990; Banning et al., Reference Banning, Sjøgren and Kaiser1992; Sjøgren et al., Reference Sjøgren, Banning and Christensen1994; Reference Sjøgren, Olsen and Thomsen2000). Post-hoc power calculations demonstrated that the sample of the present study was too small to capture significant differences between groups on the MMSE, DST, and TMT tests, and on the 10th percentile of the CRT and FTT nondominant hand and the difference between hands. An attempt to determine cutoff values for each instrument did not exhibit good sensitivity and specificity in our sample. The literature regarding validation of these neuropsychological tests in patients with cancer is scarce, which makes comparisons across studies most difficult. We found a study regarding validation of a simple auditory CRT test and a battery for attentional performance with simple and complex visual continuous reaction time tests. The comparison between healthy subjects (n = 74) and patients with cancer from the surgical, gastroenterology, oncology, and internal medicine departments (n = 70) demonstrated that reaction time was longer in those patients (Jakobsen et al., Reference Jakobsen, Sorensen and Rask2011). Another study evaluated the psychometric properties of the MMSE in patients with cancer, and the instrument was able to discriminate between subgroups of patients with different degrees of disease severity (Mystakidou et al., Reference Mystakidou, Tsilika and Parpa2007). Perhaps at the first assessment the patient sample in our present study did not present with a disease condition that would strongly interfere with cognitive function. Credence for this notion is supported by the mean KPS score being near 71% and the weak correlations with symptoms in our patient sample.
Reliability was almost perfect or substantial for the TMTB, DST, and FTT, but it varied between fair and moderate for the other tests. The disagreement between test and retest reliability may be explained by the vulnerable and unstable health condition in the advanced stages of cancer and the learning effect with some tests. Unfortunately, a second measure of quality of life and symptoms was not performed, so that we cannot confirm this hypothesis. In addition, as previously mentioned, the MMSE is highly criticized for its simplicity, easy memorization, and the possibility of learning effects.
This is one of the first and largest studies to investigate the validity of a battery of neuropsychological tests in patients with metastatic cancer. Considering that psychometric evaluation of assessment tools is an imperative step in order to select which measurement tools should be included in clinical as well as research endeavors in palliative cancer care, our study has produced new knowledge. The results from some analyses were less than definitive, but they offer information that should encourage further exploration of the area. We acknowledge that the abovementioned limitations of the tool characteristics (e.g., insensitivity to detect milder cognitive deficits, the learning effect, and methodological variances), the patient characteristics (e.g., low variation on cognitive performance, reflecting possible good cognitive performance), and the underpowered sample size for some of the tests may have contributed to some of the tests' weak properties observed during this validation process.
The neuropsychological tests analyzed demonstrated good reliability, and some measures of the CRT test and the FTT could discriminate patients from controls, indicating that they have the potential to assess cognitive function in patients with metastatic cancer in clinical and research settings. The results of the present analysis provide a warning to avoid hasty conclusions about the patients' mental condition. Further evaluation of these neuropsychological tools is mandatory to ensure that their properties are fitted for this population.
Our study also offers insight for recommendation of future studies with larger samples and analysis of subgroups with pre-confirmed cognitive dysfunction to provide more consistent results, especially regarding establishment of cutoff points for clinical and research use.
DISCLOSURES AND ACKNOWLEDGEMENTS
This study received no funding, and the authors hereby state that they have no conflicts of interest to disclose. We thank biostatistician Morten Aagaard Petersen for his assistance with the statistical analysis.
Our study was developed in partnership with the School of Nursing at the University of São Paulo, where a similar parallel investigation was conducted among Brazilian patients with cancer.
