Introduction
Neuropsychiatric symptoms (NPS) are an integral part of dementia and prodromal dementia symptomatology, with prevalence rates of 43%–90% in dementia outpatients, nursing-home residents, and patients with Mild Cognitive Impairment (MCI) (Lyketsos et al., Reference Lyketsos2001; Lyketsos et al., Reference Lyketsos, Lopez, Jones, Fitzpatrick, Breitner and DeKosky2002; Aalten et al., Reference Aalten2007; Zuidema et al., Reference Zuidema, Derksen, Verhey and Koopmans2007). NPS often impact patient and caregiver well-being and can be an important reason for nursing-home admission (Cohen-Mansfield and Wirtz, Reference Cohen-Mansfield and Wirtz2011; Onder et al., Reference Onder, Liperoti, Soldato, Cipriani, Bernabei and Landi2007). The Neuropsychiatric Inventory (NPI) is a well-known caregiver-based rating scale for NPS (Cummings et al., Reference Cummings, Mega, Gray, Rosenberg-Thompson, Carusi and Gornbein1994). However, it is unclear whether caregiver ratings of NPS concur with clinical judgment across the range of mild to severe cognitive impairment.
Limitations of the NPI are that (1) it is a retrospective measure and data are not based on direct patient observations; (2) symptom domains cannot be applied as a stand-alone measure; and, (3) few items are specific to mild and moderate dementia. Moreover, NPI scores are not impacted by clinical judgment. To rectify these limitations, the NPI–Clinician rating scale (NPI-C) was recently developed (de Medeiros et al., Reference de Medeiros2010). This new version includes next to the nested NPI, a clinician rating, two new symptom domains, and several new subquestions. Moreover, specific subitems were added, relevant to mild and severe dementia. The administration of the NPI-C provides flexibility, based on study needs. This means that domains can act as “stand-alone” measures. As a result, the NPI-C seems to be capable of measuring NPS accurately in different settings. Additionally, the NPI-C can optimize treatment and patient care (de Medeiros et al., Reference de Medeiros2010). So far, no other studies have reported on the psychometric quality of the NPI-C.
Caregivers can be biased in rating NPS. Caregiver burden, as measured with the NPI caregiver distress scale, is invariably associated with NPI symptom ratings (Kaufer et al., Reference Kaufer1998; de Jonghe et al., Reference de Jonghe, Kat, Kalisvaart and Boelaarts2003; Craig et al., Reference Craig, Mirakhur, Hart, McIlroy and Passmore2005; Huang et al., Reference Huang, Lee, Liao, Wang and Lai2012). Other factors that may influence caregiver NPI ratings are: cultural beliefs, educational level, psychological well-being, and cognitive impairments. These factors could all result in under- or over-reporting of NPS (de Medeiros et al., Reference de Medeiros2010). Moreover, it can be difficult to diagnose and measure psychopathological phenomena in moderate to severe dementia. Severely impaired patients may no longer be able to adequately report inner states and emotions, and symptoms may go undetected. So, in addition to the factors before mentioned, severe cognitive impairment per se may pose challenges to informal raters of NPS.
The objective of this study was to examine reliability of the NPI-C, and clinician–caregiver agreement. This is the first study examining psychometric aspects of the NPI-C Dutch version and the first to explore potential confounding effects of cognitive deficits on NPS assessment.
Methods
This is a cross-sectional observational study. It is part of a multicentre research project aimed at validating the NPI-C Dutch version. Concordance between psychiatrist and informant NPI-C severity ratings was examined in MCI and dementia patients. This study follows the guidelines of “good clinical practice.” Approval of the regional ethics committee was obtained. All patients or their relatives gave fully informed written consent.
Participants
The authors (MK) and (SZ) identified potentially eligible patients by screening clinical notes of outpatients attending the memory clinic of Medical Center Alkmaar (MCA) in the Netherlands, and nursing-home residents for whom MK and SZ provide psychiatric and psychological services. By sampling patients from both type of settings we intended to cover a wide range of mild to severe cognitive impairment. Patients were considered for inclusion in the study if they had an established diagnosis of dementia or MCI and if they had a knowledgeable informant. No other information about the caregiver was available prior to patient selection. Patients were excluded if they had severe communication problems due to cerebrovascular accidents (CVA) or if they had an acute somatic condition that might interfere with judgment of more or less stable NPS, e.g. in case of delirium.
Procedures
An experienced old-age psychiatrist (MK) interviewed all patients and caregivers separately using the NPI-C. Patient–caregiver dyad interviews were planned on the same day. The psychiatrist reviewed patient medical charts. A neuropsychologist (SZ) assessed the patient, using a short and comprehensive cognitive screen, in a separate room adjacent to the one the caregiver was interviewed in. Clinical ratings based on patient and caregiver interviews served as the gold standard for presence and severity of NPS, and were compared to caregiver ratings.
Two randomly selected NPI-C caregiver interviews were video-recorded and used to evaluate inter-observer agreement. Caregiver responses to questions inquiring about the frequency and severity of symptoms were independently rated by 11 MCA-affiliated psychologists, geriatricians, and geriatrician trainees.
Instruments
The NPI-C is a semi-structured interview which measures NPS in dementia patients. It consists of 14 symptom domains (total of 142 items), instead of the 12 domains (84 items) that are used in the NPI. The 14 NPS are: delusions, hallucinations, agitation, aggression, dysphoria, anxiety, apathy, irritability/lability, euphoria/elation, disinhibition, motor disturbances, sleep disorders, appetite and eating abnormalities, and aberrant vocalization. Aggression and aberrant vocalization are newly added domains. Each domain consists of a screening question and 6–16 subsequent questions pertaining to specific behaviors, which can be rated for frequency of occurrence (5 point scale, 0 = never to 4 = always) and severity (4 point scale, 0 = none to 3 = severe). De Medeiros et al. (Reference de Medeiros2010) report on two important rating changes in the NPI-C. The first is item-by-item scoring. In the NPI, a knowledgeable informant provides a global domain rating for frequency, severity, and level of caregiver distress related to the group of items within a given domain. The domain score is the product of the ratings for domain frequency and severity. In the NPI-C, ratings for frequency, severity, and distress are provided for each item and summed to create a total domain score. Item scores are better suited to clinical trials where assessment of change with finer detail is needed. The second important rating change is the use of ratings based on expert clinical judgment of all available data to make severity ratings for individual items. In the clinician rating approach, the knowledgeable informant is first asked to provide frequency, severity, and distress ratings for items as described above. Separately, the rater also interviews the patient. Caregiver distress is rated using a 6-point scale (Kaufer et al., Reference Kaufer1998). The NPI Dutch version can be scored objectively (Kat et al., Reference Kat, de Jonghe, Aalten, Kalisvaart, Droes and Verhey2002).
The Mini-Mental State Examination (MMSE) is a well-known screen for cognitive impairment (Folstein et al., Reference Folstein, Folstein and McHugh1975). The scores of the MMSE range from 0 to 30. Scores from 0 to 10 correspond with Clinical Dementia Rating scale severe dementia, 11–20 to moderate dementia, and 20+ to mild dementia (Perneczky et al., Reference Perneczky, Wagenpfeil, Komossa, Grimmer, Diehl and Kurz2006). The CAMCOG-R (Cambridge Cognitive Examination-revised) Dutch version is part of the CAMDEX-R/N (Cambridge Examination for Mental Disorders of the Elderly-revised (in English); Derix et al., Reference Derix2002) and it can be used as a comprehensive screen for cognitive impairment. It allows for calculating a total score and subscores for specific cognitive domains, i.e. orientation, memory, attention/calculation, language, praxis, abstract thinking, and perception. CAMCOG-R total scores range between 0 (severe cognitive impairment) and 105 (no cognitive impairment). The CAMCOG-R was successfully harmonized between European countries (Verhey et al., Reference Verhey2003).
Statistical analyses
We examined internal consistency reliability of items pertaining to specific NPI-C symptom domains using Cronbach's α. Caregiver video inter-rater agreement was examined using Intra-class correlations (ICC). Prevalence estimates were calculated based on NPI symptom domain scores, and a score >3 may indicate the presence of clinically relevant symptoms (Aalten et al., Reference Aalten2007). Caregiver- and clinician-based NPI-C total severity scores and symptom domain severity scores were examined using Pearson's correlations. Based on expected clinician–caregiver correlations of R = 0.50–0.60, a power of 80%, and P = 0.05, we calculated that 18–28 patients were needed for this study (StatPages.org). We calculated delta scores reflecting differences between caregiver and clinician ratings for all individual NPI-C symptom domain severity questions. A negative score indicates that the psychiatrist observed more severe psychopathology than the caregiver and vice versa. Delta score variance was compared across the range of cognitive impairment. With the aim of making group comparisons MMSE total scores were recoded mild (19 or over) or moderate/severe (0–18), and CAMCOG-R total scores and subscale scores were recoded into scores 0–50th percentile or above the 50th percentile.χ 2, Fisher Exact, t-tests and Mann–Whitney U tests were used whenever deemed appropriate. Levene's test for equalities of variances was used to examine differences in delta score variance. All data were analyzed using Predictive Analytics Software (PASW) for Windows, version 17.0 (SPSS Inc., Chicago, IL).
Results
Participant characteristics
A total of 30 patient–caregiver dyads with complete data protocols were included in the study. Most outpatient caregivers were family members, and for nursing-home residents, caregivers were staff members. Twenty patients were female and overall age range was 64–95 years (mean = 81.2, SD = 6.3). A known diagnosis of Alzheimer's disease was present in 43.3% of patients, 13.3% had mixed dementia, 6.7% had frontotemporal dementia, 3.3% had Lewy body/Parkinson disease dementia, 6.7% had dementia not otherwise specified, and 23.3% had MCI. A wide range of cognitive impairment was covered, as evidenced by MMSE score distribution, i.e. 13 patients had MMSE score >18 (mean = 23.0, SD = 2.3), 7 patients had MMSE scores between 10–18 (mean = 14.1, SD = 2.5), and 10 severely impaired patients had MMSE scores between 0–9 (mean = 5.6, SD = 3.6).
Neuropsychiatric symptoms were prevalent in this sample, as indicated by score of >3 on one or more NPI symptom domains in 19/30 of patients (63.3%). Mild to severe symptoms were most often observed for dysphoria (73.3% of patients), and least often for euphoria/elation (6.7%; Table 1). The average NPI total score of 15.8 (SD = 15.8, range = 0–59) shows that behavioral changes were relatively mild, a result that is statistically different from that found in a large-scale study using the NPI (mean = 22.8, SD = 16.4, n = 2,354, average group difference = 7 points, CI = 1.30–12.69; Aalten et al., Reference Aalten2007, personal comment).
Caregiver severity ratings NPI symptom domains (n = 30).
Reliability
NPI-C subscales are internally consistent as evidenced by acceptable or high Cronbach's α for the respective clinician and caregiver ratings of: delusions 0.65–0.77, hallucinations 0.86–0.76, agitation 0.76–0.74, dysphoria 0.76–0.84, anxiety 0.80–0.86, euphoria 0.86–0.71, apathy 0.92–0.91, disinhibition 0.86–0.83, irritability 0.71–0.78, motor disturbances 0.81–0.82, sleep disorder 0.71–0.66, appetite and eating abnormalities 0.64–0.85, and modest Cronbach's α for aggression 0.36–0.27 and aberrant vocalizations 0.42–0.48. Objectivity of the ratings was very high, as indicated by near perfect inter-rater agreement on ratings of two caregiver videos (ICC = 0.99–1.0).
Clinician–caregiver agreement
Psychiatrist and caregivers agreed highly on overall NPI-C symptom severity (r = 0.77). Furthermore, high agreement was found for 11/14 NPI-C symptom domains, with correlations ranging from 0.75–0.95. Moderate agreement was found for motor disturbances, appetite and eating abnormalities, and aberrant vocalization (r = 0.58–0.62). Nevertheless, clinician and caregivers disagreement was considerable in some instances, as was expressed by a range from –63 to 49 points in Delta NPI-C severity scores. Clinical judgment of overall symptom severity was higher than caregiver ratings in 73.3% of patients. The psychiatrist reported higher agitation severity (53.3%), disinhibition (46.7%), motor disturbances (36.7%), sleep disturbances (30%), eating behavior (40%), and aberrant vocalization (43.3%) compared to caregivers. Caregivers reported greater dysphoria severity compared to clinical judgment in 40% of the patients.
Clinician–caregiver agreement in relation to cognitive impairment
Clinician–caregiver agreement on NPI-C total symptom severity was lower when cognitive impairment was more profound, as was evidenced by greater Delta score variance in the MMSE moderate to severe range compared to mild range (Table 2). Similarly, in more severe cognitively impaired patients Delta score variance increased for most individual NPI-C symptom domains, except for hallucinations, dysphoria, euphoria, and motor disturbances (Table 2), which consistently showed no differences across any of the cognitive measures (Table 1). Clinician–caregiver agreement decreased when amnesia and aphasia were relatively severe, as measured by the CAMCOG-R Memory (F-test = 4.5, P = 0.05), and Language (F-test = 8.6, P = 0.01) subscales.
Clinician–caregiver variability expressed in F-statistic in mild versus severe cognitive impairment as measured with MMSE and CAMCOG-R (n = 30).
*P < 0.05.
**P < 0.01.
***P < 0.001.
aSummed severity scores of all individual NPI-C domain items: average (standard deviation), range.
Discussion
This study examined NPI-C reliability and clinician–caregiver agreement on symptom severity. Inter-rater agreement on videotaped NPI-C caregiver based ratings was very high, underlining the objectivity of the measure. We found that clinician–caregiver ratings were significantly associated, and that agreement between clinician–caregiver symptom severity ratings varied across the range of cognitive impairment. Although this is a small study, its strengths are the assessment of all patients by an experienced old-age psychiatrist, the use of a comprehensive cognitive screening test, and the complete data set.
The results of this study are important for understanding potential sources of systematic bias and precision of behavioral rating scales, in particular the NPI-C. Information on behavioral changes in neurodegenerative diseases, such as Alzheimer's disease, usually comes either from clinical observations or from caregiver reports. We found that these two sources of information often concur. However, caregivers reported less severe behavioral changes than the psychiatrist. Moreover, the standard deviation of clinician–caregiver agreement was 2.6 times higher in more severely cognitively impaired patients than in mildly impaired patients. The effects found were irrespective of type of cognitive impairments. These results may have implications for the precision of monitoring treatment effects in clinical trials, particularly when study outcomes are caregiver-based. As modest effects are usually found in clinical trials using the NPI (Cummings et al., Reference Cummings, Nadel, Masterman and Cyrus2001; Maidment et al., Reference Maidment, Fox, Boustani, Rodriguez, Brown and Katona2008; Rodda et al., Reference Rodda, Morgan and Walker2009), and some studies show that a reliable change index (RCI) for the NPI is relatively large (Iverson et al., Reference Iverson, Hopp, DeWolfe and Solomons2002; Zuidema et al., Reference Zuidema2011), potential systematic bias or imprecision of caregiver-based ratings of NPS in more severely impaired patients raises concern.
Our results concur with those found in the original NPI-C study (de Medeiros et al., Reference de Medeiros2010). In that study, clinical raters from different professional backgrounds evaluated 128 mild to severe dementia patients and showed that inter-rater agreement was moderate to strong. Selected NPI-C symptom domains correlated (R = 0.31–0.60) with other behavioral rating scales. However, de Medeiros et al. (Reference de Medeiros2010) did not examine agreement between NPI-C clinician and caregiver ratings. In our study NPI-C reliability was very high and clinician and caregiver ratings correlated highly. Furthermore, by comparing clinical and caregiver ratings across the range of cognitive impairment we were able to shed light on factors that may threaten validity of caregiver-based severity ratings.
Internal consistency of clinician- and caregiver-rated NPI-C symptom domains ranged from modest to high, with lower consistency findings for aberrant vocalization and aggression. Notably, de Medeiros et al. (Reference de Medeiros2010) also found that ICC was modest for the NPI-C aggression symptom domain. Probable explanations for internal consistency results for aggression in this study are the relatively small number of patients and the low incidence of some symptoms, which in turn could lead to reduced score variance. NPI-C aggression item “Is he/she unreasonably or uncharacteristically argumentative?” was present in only 16.7% of patients, and all other signs and symptoms of aggression were observed even less frequently. Large-scale studies that include severely behaviorally disrupted patients may show that NPI-C symptom domains are internally consistent.
Though clinician–caregiver ratings were significantly associated, we found that caregivers tended to report that symptoms were less severe. In almost three out of four patients, the psychiatrist judged symptom severity to be higher than the caregiver did. Dysphoria showed the reverse pattern, with higher severity scores given by caregivers. There may have been idiosyncrasies in this particular study that account for these results. Alternatively, it is likely that caregivers under-report symptom severity. A straightforward explanation for this hypothesis is that the NPI scoring format requires that the caregiver rates behaviors that are disruptive to the patient. We often noted that caregivers reported (e.g.) apathy to be present all day, but that the behaviors were not distressing to the patient at all, and they subsequently rated severity as mild. By contrast, caregivers sometimes observed symptoms of depression that, from a clinical perspective, would qualify as symptoms of apathy and emotional lability. Attributing behavioral changes to mood states might lead to over-reporting of depression or depression severity. Furthermore, studies consistently show that caregiver distress correlates >0.80 with NPI symptoms (Kaufer et al., Reference Kaufer1998; de Jonghe et al., Reference de Jonghe, Kat, Kalisvaart and Boelaarts2003; Craig et al., Reference Craig, Mirakhur, Hart, McIlroy and Passmore2005; Huang et al., Reference Huang, Lee, Liao, Wang and Lai2012). In summary, caregiver ratings were positively associated with clinical judgment.
A limitation of this study is its sample size. Though this study was powered for examining total sample clinician–caregiver correlations, it was not for separate subsamples of mild to severe cognitive impairment. Findings need to be replicated in a larger study. We are at present recruiting patients with the aim of comparing the NPI-C with other behavioral rating scales.
This study shows that the NPI-C may well be a better measure of NPS than the original NPI. In this first study of a Dutch patient sample, NPI-C objectivity was supported, and clinician–caregiver severity ratings were highly correlated. Present finding that disagreement between types of raters increases with cognitive impairment underlines the importance of the NPI-C as a clinician-based measure.
Conflict of interest
None.
Description of authors’ roles
Shirin Zaidi translated the NPI-C, collected the data, wrote the paper, performed the statistical analysis, and assisted as a research coordinator. Martin Kat translated the NPI-C, collected the data and supervised the data collection, and assisted as a research coordinator. Jos de Jonghe was the research coordinator. He translated the new NPI-C, assisted in designing the study, supervised analyzing the data, and assisted in writing of the paper.
Acknowledgments
The authors gratefully acknowledge the cooperation of patients with dementia, their families, and staff members of Medical Centre Alkmaar and Stichting SHDH. We are also grateful to Tjeerd van der Ploeg for statistical assistance.
