Introduction
Dementia is characterized by multiple severe cognitive deficits and disturbances that cause a decrease in occupational or social functioning (American Psychiatric Association, 2000). Therefore, selected disease-related symptoms (e.g. cognition, functional ability, associated disturbances, such as agitation) as outcome measures in intervention studies seem to be inappropriate. Quality of life (Qol) as an outcome measure reflects the meaning of the disease for an individual person as a whole. As a result it has become an often used outcome in intervention studies in dementia research (Rabins and Black, Reference Rabins and Black2007; Moniz-Cook et al., Reference Moniz-Cook2008).
The World Health Organization defines Qol as ‘individuals’ perceptions of their position in life in the context of the culture and value systems in which they live and in relation to their goals, expectations, standards and concerns’ (WHO, 1995). This broad concept stresses subjective experience, culture-specific influences and their interaction. Subjectivity and multidimensionality are the common denominators which emerged from different theoretical developments about the definition of Qol for people with dementia (Lawton, Reference Lawton1994; Jonker et al., Reference Jonker, Gerritsen, Bosboom and Van Der Steen2004; Ettema et al., Reference Ettema, Droes, de Lange, Ooms, Mellenbergh and Ribbe2005).
Self-rating by people with dementia is the gold-standard method for measurement of Qol (Brod et al., Reference Brod, Stewart, Sands and Walton1999; Rabins and Black, Reference Rabins and Black2007). However, deficits in memory and concentration and impaired decision making and communication competencies, which occur throughout the course of the disease, affect the reliability and validity of Qol self-reports (Brod et al., Reference Brod, Stewart, Sands and Walton1999). Therefore, proxy measures are preferred in advanced dementia and longitudinal Qol assessment (Moniz-Cook et al., Reference Moniz-Cook2008; Bouman et al., Reference Bouman, Ettema, Wetzels, van Beek, de Lange and Droes2011), although this technique is also associated with methodological uncertainties. Proxy-rating-generated Qol scores are systematically lower than self-rated Qol values (Ettema et al., Reference Ettema, Droes, de Lange, Ooms, Mellenbergh and Ribbe2005; Gräske et al., Reference Gräske, Fischer, Kuhlmey and Wolf-Ostermann2012) and correlate positively with proxy rater burden (Novella et al., Reference Novella2001; Huang et al., Reference Huang, Chang, Tang, Chiu and Weng2009) and rater attitudes (Sands et al., Reference Sands, Ferreira, Stewart, Brod and Yaffe2004). These challenges emphasize the relevance of a comprehensive and careful psychometric evaluation of dementia-specific proxy Qol instruments, and especially, their inter-rater and intra-rater reliability (test-retest reliability).
Studies on inter-rater and intra-rater reliability of dementia-specific Qol measurements lack internal validity: sample size calculations are not employed, and sample sizes are very small (n ≤ 25 participants) (Weiner et al., Reference Weiner, Martin-Cook, Svetlik, Saine, Foster and Fontaine2000; Smith et al., Reference Smith2005b; Lucas-Carrasco et al., Reference Lucas-Carrasco, Lamping, Banerjee, Rejas, Smith and Gomez-Benito2010); sample characteristics (Porzsolt et al., Reference Porzsolt2004; Sloane et al., Reference Sloane2007; Lucas-Carrasco et al., Reference Lucas-Carrasco, Lamping, Banerjee, Rejas, Smith and Gomez-Benito2010), selection criteria for reliability samples (Porzsolt et al., Reference Porzsolt2004; Falk et al., Reference Falk, Persson and Wijk2007; Lucas-Carrasco et al., Reference Lucas-Carrasco, Lamping, Banerjee, Rejas, Smith and Gomez-Benito2010) and the characteristics of proxy raters are insufficiently reported (Weiner et al., Reference Weiner, Martin-Cook, Svetlik, Saine, Foster and Fontaine2000; Porzsolt et al., Reference Porzsolt2004; Ettema et al., Reference Ettema, Droes, de Lange, Mellenbergh and Ribbe2007b); inappropriate statistical measures are applied (Sloane et al., Reference Sloane2007; Karim et al., Reference Karim, Ramanna, Petit, Doward and Burns2008) and measures of statistical uncertainty, e.g. Confidence Intervals (CIs) are not reported (Falk et al., Reference Falk, Persson and Wijk2007; Ettema et al., Reference Ettema, Droes, de Lange, Mellenbergh and Ribbe2007b; Wolak et al., Reference Wolak2009). A lack of prior reliability studies for dementia-specific Qol measures could be a reason for these methodological weaknesses, because reliability studies are most often part of a larger validity study.
One promising and relatively new measurement for dementia-specific Qol is the QUALIDEM (Ettema et al., Reference Ettema, Droes, de Lange, Mellenbergh and Ribbe2007b). The original Dutch instrument was developed and validated for the proxy rating of Qol throughout the entire course of dementia in nursing home residents (Ettema et al., Reference Ettema, Droes, de Lange, Mellenbergh and Ribbe2007a; Reference Ettema, Droes, de Lange, Mellenbergh and Ribbe2007b). Therefore, the QUALIDEM is recommended for Qol assessment in the late stages of dementia (Moniz-Cook et al., Reference Moniz-Cook2008) and for longitudinal assessment of Qol (Bouman et al., Reference Bouman, Ettema, Wetzels, van Beek, de Lange and Droes2011).
In 2008, both versions of the QUALIDEM were translated into German (Dichter et al., Reference Dichter, Bartholomeyczik, Nordheim, Achterberg and Halek2011). We demonstrated the construct validity and the moderate to high internal consistency of the German version in an initial exploratory study (Dichter et al., Reference Dichter, Bartholomeyczik, Nordheim, Achterberg and Halek2011) and a further confirmatory investigation (Dichter et al., Reference Dichter2013a). Apart from the data on internal consistency, no data on reliability are available. Taking the limitations of earlier reliability investigations into account, the objective of the present study is a comprehensive evaluation of the inter-rater and intra-rater reliability of the QUALIDEM as part of the Qol-Dem project (Dichter et al., Reference Dichter, Halek, Dortman, Meyer and Bartholomeyczik2013b).
Methods
Study design
This reliability study was conducted from July, 2011 to May, 2012 as part of the baseline data collection of a quasi-experimental trial (Halek et al., Reference Halek, Dichter, Quasdorf, Riesner and Bartholomeyczik2013). The sample consisted of people with dementia in nine nursing units at nine nursing homes located in the western part of Germany. To achieve the calculated sample size, data collection was performed at three additional nursing units at these nursing homes.
Sample size calculation
For the sample size calculation, we estimated the intra-class correlation coefficient (ICC) planning value, similar to the value calculated by Ettema et al. (Reference Ettema, Droes, de Lange, Mellenbergh and Ribbe2007b) in an earlier work. Based on the ratings of four independent raters (nurses), a width of 0.20 for the 95% CI and ICC planning values between 0.79 and 0.47 (37-item version) and between 0.79 and 0.55 (18-item version) was determined and, the calculated inter-rater reliability sample varied depending on the subscale being tested. Thirty-four to 106 residents with mild to severe dementia were needed for the 37-item version, and 34 to 93 residents with very severe dementia were needed for the 18-item version of the QUALIDEM (Bonett, Reference Bonett2002). To achieve precise results, we aimed to recruit 106 residents with mild to severe dementia and 93 residents with very severe dementia for the evaluation of inter-rater reliability.
The sample size calculation for the investigation of intra-rater reliability resulted in a range of 18 to 85 (37-item version) and 39 to 75 (18-item version) residents, depending on the subscale being tested. To obtain precise results for intra-rater reliability, we aimed to recruit 85 (37-item version) and 75 (18-item version) residents for the evaluation of intra-rater reliability.
Procedures
The evaluation of the inter-rater and intra-rater reliability was based on proxy ratings from nurses referring to the preceding two weeks. The inclusion criteria for residents with dementia were a Functional Assessment Staging (FAST) (Reisberg, Reference Reisberg2007) value ≥ 2 and living in the nursing home for at least two weeks. The exclusion criterion for people with dementia was a documented diagnosis of Parkinson's disease.
The qualification level of nurses (registered nurses and nursing assistants) depended on organizational conditions and staffing levels at the time of data collection. The inclusion criteria for nurses were as follows: a close relationship with the assessed resident and a working contract of at least half-time work. In addition, the nurses had to be at work on most days within the last two weeks prior to data collection to ensure a close relationship with the resident being assessed, which facilitated the collection of up-to-date information about the resident. Based on these criteria, the participating nurses were identified by the nursing home management.
For the examination of inter-rater reliability, four nurses from each nursing home assessed the Qol of people with dementia. Each nurse was blinded to the ratings of the other raters. One of these four nurses repeated the Qol assessment after one week for intra-rater reliability assessment. It was assumed that the nurses could not remember their prior Qol ratings in detail. The nurses were not allowed to make records of their ratings.
To ensure standardized data collection, QUALIDEM application was guided by external, trained study assistants who were registered nurses. The study assistants received 5 hours of prior training on data collection that was provided by one investigator (MND). The training included general information about the data collection and measurements, data collection exercises and a feedback round. A comprehensive instruction manual for data collection and a telephone hotline were provided. For intra-rater reliability assessment, the data collection was guided by the same rater at both measurement points.
Measurements
The QUALIDEM consists of two versions. Qol among people with mild to severe dementia is assessed using the 37-item version, which covers the following nine domains of Qol: care relationship, positive affect, negative affect, restless tense behavior, positive self-image, social relationship, social isolation, feeling at home and having something to do. The domains positive self-image, feeling at home and having something to do cannot be assessed in people with very severe dementia. The second version, for people with very severe dementia, consists of 18 items covering six domains of Qol. Three additional items were not scalable during the development of the QUALIDEM but were recommended for further research on the instrument. These include: enjoys meals, does not want to eat and likes to lie down (in bed) (Ettema et al., Reference Ettema, Droes, de Lange, Mellenbergh and Ribbe2007b).
The stage of dementia severity was classified according to FAST. This staging instrument ranges from 1 to 7 (1: free of cognitive impairment; 2–6: mild dementia to severe dementia; and 7: very severe dementia) (Reisberg, Reference Reisberg2007).
For the description of residents’ care dependency, the corresponding levels defined by the German statutory long-term care insurance were used (ranging from 1 = low to 3 = high). The age and gender of the residents with dementia were also assessed.
The sociodemographic data of the nurses, e.g. age, professional qualification and work experience, were recorded by study assistants as part of the data collection.
Statistical analysis
To gain detailed information regarding inter-rater and intra-rater reliability, the degree of inter-rater and intra-rater agreement on every individual item and each subscale of both versions of the QUALIDEM was calculated.
For the calculation of inter-rater reliability of every item, the overall proportion of agreement (po) was computed, which is defined as the number of the exact agreement between two raters divided by the total number of ratings (Kottner and Dassen, Reference Kottner and Dassen2008). Because four Qol ratings for each person were available, we computed the mean po for all possible rater pairs (n = 6). Po is limited because it does not take into account the fact that agreement could occur purely by chance. The sole consideration of the exact agreement for the investigation of the inter-rater reliability of at least ordinal data is the second limitation of po. Thus, all disagreements between raters are treated equally, regardless of their extent (Kottner and Dassen, Reference Kottner and Dassen2008). The extent of disagreement between different raters according to the QUALIDEM items is clinically relevant, so the multi-rater weighted κ statistics for ordinal data (κw, i.e. Conger's kappa) were also calculated to assess the inter-rater reliability of each item (Conger, Reference Conger1980). The strength of agreement for κw is based on a recommendation by Landis and Koch (Reference Landis and Koch1977). These authors defined ranges of κ-statistic as follows: 0.00–0.20, slight; 0.21–0.40, fair; 0.41–0.60, moderate; 0.61–0.80, substantial; and 0.81–1.00, nearly perfect. However, the two paradoxical properties of κ-statistic were also taken into account. The first paradox is the dependence of κ-values on the prevalence of the phenomenon of interest, and the second paradox is the dependence on extreme variations in the assessment compared with slight variations (Mayer et al., Reference Mayer, Nonn, Osterbrink and Evers2004). For assessment of the inter-rater reliability of the QUALIDEM subscales, ICCs based on a two-way random effects model for absolute agreement were calculated. This model includes systematic variability between raters and was chosen because the raters were randomly selected in this study. The results can be generalized to other nurses in the nursing unit (Kottner and Dassen, Reference Kottner and Dassen2008). Additionally, the average-measure ICC for four raters was calculated. This coefficient estimates the inter-rater reliability of a collaborative QUALIDEM rating by all four raters. This procedure allows the comparison of the results with previous findings from the Netherlands (Ettema et al., Reference Ettema, Droes, de Lange, Mellenbergh and Ribbe2007b). An ICC ≥ 0.70 was interpreted as a strong agreement (DeVon et al., Reference DeVon2007). To investigate the uncertainty of the inter-rater reliability, the 95% CIs for ICCs and κw were computed. The CIs for κw were based on 1,000 bootstrapped samples. Bootstrapping is a computationally intensive resampling technique that was first introduced by Efron (Efron, Reference Efron1982; Efron and Tibshirani, Reference Efron and Tibshirani1993) and allows estimation of the precision of a statistic to be calculated without making assumptions concerning its distribution. We draw 1,000 resamples with replacement of the size of the original sample. The κ statistic was then calculated for each resample. Its bootstrap 95% CI was determined with the percentile method (Carpenter and Bithell, Reference Carpenter and Bithell2000) while taking the 0.025 and 0.975 percentile levels of the estimated kappas distribution as interval limits.
For intra-rater reliability assessment, the overall proportion of agreement (po) and quadratic κw (weighted kappa) (Cohen, Reference Cohen1968) statistics were calculated for each item. For the evaluation of intra-rater reliability of the subscales ICCs based on a two-way random effects model for absolute agreement were calculated. For the evaluation of statistical uncertainty the 95% CIs for ICCs and κw were computed.
Descriptive statistics were used for the calculation of the sample characteristics. The software packages PASW Statistics version 18 and SAS 9.2 were used for all statistical analysis.
Ethics
Ethical approval was obtained from the ethics committee of the German Society of Nursing Science (2010–08–19). Written information about the study was given to the residents’ legal representatives, and written informed consent was obtained.
Nurses received written and oral information about the study before data collection. Voluntary participation in the data collection was considered as informed consent from nurses.
Results
Study population
The inter-rater reliability sample consisted of two subsamples of 108 people with mild to severe dementia and 53 people with very severe dementia. The Qol of each person with dementia was rated four times by 49 nurses.
For the evaluation of intra-rater reliability, one person with dementia was lost to follow-up in each subsample. The first and second Qol rating completed after one week, were performed by 12 nurses (Table 1).
Characteristics of the sample
a As determined by the expert raters of the medical service of the statutory long-term care insurance.
b Each QUALIDEM item was rated four times per participant (FAST 2–6 = 37 items, FAST 7 = 18 items).
c Each QUALIDEM item was rated two times per participant (FAST 2–6 = 37 items, FAST 7 = 18 items).
d FAST 2–6, n = 107; FAST 7, n = 52.
Missing-value analysis
From the 108 people with mild to severe dementia, nine of the 432 inter-rater reliability ratings (2.1%) and two of the 214 intra-rater ratings had one missing value (0.9%). Approximately 1.4% of ratings for inter-rater reliability and 1.9% of intra-rater reliability ratings are missing from the sample of people with very severe dementia. Table 1 shows the missing values in relation to the number of possible responses.
Inter-rater reliability
The results of the evaluation of the inter-rater reliability are displayed in Table 2. Briefly, none of the QUALIDEM subscales showed strong inter-rater agreement for the two raters based on an ICC, either for people with mild to severe dementia or for people with very severe dementia. The ICC values ranged from 0.35 to 0.61 (FAST 2–6) and from 0.28 to 0.67 (FAST 7).
Inter-rater reliability results for the German version of the QUALIDEM
a Mean overall proportion of agreement (the ratio of exact agreement between two raters divided by the total number of ratings) for all possible rater pairs.
b ICC: intra-class correlation coefficient.
c Single-measure ICC absolute agreement.
d Average-measure ICC for four raters in parentheses.
e Multi-rater κw (weighted kappa) values.
f CIs for κw were based on 1,000 bootstrapped samples.
Based on the average-measure ICC for four raters, eight of the nine subscales for people with mild to severe dementia showed strong or nearly perfect agreement (ICC: 0.72–0.86; care relationship, positive affect, negative affect, restless tense behavior, social relations, social isolation, feeling at home and having something to do). For people with very severe dementia, the following five subscales yielded ICC values from 0.72–0.89: care relationship, negative affect, restless tense behavior, social relations and social isolation.
The multi-rater weighted k values ranged between 0.17 (item 33) and 0.65 (item 32) for people with mild to severe dementia and between 0.18 (item 40) and 0.64 (item 32) for people with severe dementia. For people with mild to severe dementia, only item 23 (0.61 κw) and item 32 (0.65 κw) showed substantial agreement. For the 18-item version, this agreement was the same for item 32 (0.64 κw). The results for po ranged between 0.36 (item 22) and 0.69 (item 12) for people with mild to severe dementia and between 0.32 (item 22) and 0.71 (item 31) for people with very severe dementia, depending on the individual item.
Intra-rater reliability
The results presented in Table 3 indicate strong intra-rater agreement (ICC ≥ 0.70) for all subscales of both QUALIDEM versions. The ICC values ranged from 0.70 to 0.79 for people with mild to severe dementia and from 0.75 to 0.88 for people with very severe dementia.
Intra-rater reliability results for the German version of the QUALIDEM
a Overall proportion of agreement (the ratio of exact agreement between raters to the total number of ratings).
b ICC: intra-class correlation coefficient.
c Quadratic weighted kappa values.
The quadratic weighted κ values range from 0.52 to 0.77 (FAST 2–6) and 0.24 to 0.69 (FAST 7). In total, 17 of the 40 QUALIDEM items showed substantial intra-rater reliability for people with mild to severe dementia and 6 of the 18 items for very severe dementia show κw ≥ 0.61. For people with mild to severe dementia, po ranged between 0.56 (item 13) and 0.80 (item 12), and for people with very severe dementia, the value ranged between 0.52 (items 7, 20) and 0.75 (item 12).
Discussion
This study describes a comprehensive evaluation of the inter-rater and intra-rater reliability of the German version of the QUALIDEM.
The 37-item version of the QUALIDEM, intended for people with mild to severe dementia, shows moderate agreement (ICC: 0.52 to 0.61) for the five subscales, care relationship, restless tense behavior, social relations, social isolation and having something to do and fair inter-rater reliability for the subscales positive affect, negative affect, positive self-image and feeling at home. In comparison, Ettema et al. (Reference Ettema, Droes, de Lange, Mellenbergh and Ribbe2007b) found strong to moderate inter-rater reliability for eight subscales (ICC: 0.59 to 0.79). The researchers’ results for the subscale positive self-image were relatively comparable with our results, demonstrating fair agreement.
It remains unclear why our study presents different results than the previous study (Ettema et al., Reference Ettema, Droes, de Lange, Mellenbergh and Ribbe2007b). One reason could be the different preparation of the proxy-rating nurses. In the Dutch study, the proxy rating-certified nursing assistants received oral instructions at a 1 hour meeting at the start of the observation period (Ettema et al., Reference Ettema, Droes, de Lange, Mellenbergh and Ribbe2007b). In our study, the nurses received no instructions at the beginning of the observation period but were instructed during the Qol rating by a trained student assistant.
Similar to the Dutch study, our calculation of the average-measure ICC for four raters revealed higher inter-rater reliability than for two raters for all subscales.
The in-depth analysis of the inter-rater reliability results for each item provides a more accurate understanding of the subscale-related results. Based on κw and po, only items 23 (cries) and 32 (calls out) demonstrated sustainable inter-rater reliability. These items represent the relatively easily observable behaviors of people with dementia. In contrast, 20 QUALIDEM items yielded a poor and insufficient inter-rater reliability (κw: 0.17 to 0.40). These items are important for the assessment of psycho-social Qol domains. Instead of deleting of these items, the further development by a precise definition of items might improve the inter-rater reliability. These items represent the reactions of people with dementia to interactions or actions (items 8, 12, 20, 24, 25, 31, 36, 40) and the affects and mood (items 1, 5, 6, 10, 21), behavior (item 22) and direct interactions (items 13, 27, 28, 33, 35, 37) of people with dementia. The observation and assessment of these items are more difficult and are more dependent on individual situations, interactions and the relationship between the people with dementia and the assessing nurse.
Differences in nursing education and correspondingly different nursing tasks between Germany and the Netherlands might be another cause for the discrepancy in results. A further reason for the lower inter-rater reliability in our study compared with the Dutch study could be the difference in the focus on psychosocial aspects in the care of people with dementia between the countries. In Germany, a strong focus on physical and cognitive functioning is anchored in a long-term statutory definition of care dependency (SGB XI, 2008).
If the QUALIDEM inter-rater reliability results are compared with the reliability of other psychosocially oriented Qol instruments, the results are inconclusive. A similarly low coefficient (ICC between 0.02 and 0.61) was found for the subscales of the Alzheimer Disease-Related Quality of Life (ADRQL) instrument (Menzi-Kuhn, Reference Menzi-Kuhn2006), which assesses the emotional and social domains of Qol in particular. However, accurate interpretation and comparison of the results are not possible because the reliability coefficient for each item is not reported. Higher inter-rater agreement was shown for the Quality of Life-Alzheimer's Disease (QOL-AD) scale (between 0.60 and 0.74, based on Cohen's kappa) (Thorgrimsen et al., Reference Thorgrimsen2003). This scale consists of 13 items that assess psychosocial aspects and functional and cognitive abilities as domains of Qol. Unfortunately, the κ value related to each item is not reported (Thorgrimsen et al., Reference Thorgrimsen2003). A detailed interpretation of this result in comparison with the item content is therefore not possible.
The evaluation of inter-rater reliability for people with very severe dementia (18-item version) exhibits comparable results. None of the six subscales revealed strong reliability between two raters. The inter-rater agreement for the subscales care relationship, negative affect and social isolation was moderate and for the subscales positive affect, restless tense behavior and social relations was fair. Ettema and colleagues found strong (care relationship and social isolation) and moderate (positive affect, negative affect, restless tense behavior and social relations) inter-rater reliability (Ettema et al., Reference Ettema, Droes, de Lange, Mellenbergh and Ribbe2007b). However, the average-measure ICC for four raters indicated strong inter-rater reliability for care relationship, negative affect, restless tense behavior, social relations and social isolation and moderate agreement for positive affect.
The evaluation of each item for people with very severe dementia showed substantial inter-rater agreement for item 32 (calls out). The more difficult rating of items that represent the reactions of people with dementia to interaction and action (items 8, 12, 20, 25, 40), the affects and moods (items 5, 6, 21), behavior (item 22) and direct interactions (item 3) of people with dementia resulted in a poor and insufficient inter-rater reliability (κw: 0.18 to 0.38).
All QUALIDEM subscales, regardless of dementia severity, show strong intra-rater agreement. The ICC values range between 0.75 and 0.88 for people with mild to severe dementia and between 0.70 and 0.79 for people with very severe dementia. This high intra-rater agreement is similar to the Dutch findings and demonstrates strong intra-rater agreement for all QUALIDEM subscales (Ettema et al., Reference Ettema, Droes, de Lange, Mellenbergh and Ribbe2007b).
The intra-rater item analysis for people with mild to severe dementia revealed 17 items as substantial and the 23 remaining items as moderately reliable, based on κw and po. For people with very severe dementia, six items (2, 3, 8, 19, 23, 32) showed substantial intra-rater agreement, 10 items (5, 6, 7, 12, 14, 16, 21, 22, 25, 31) exhibited moderate intra-rater agreement and 2 items a poor intra-rater agreement (20, 40). Thus, the stability of ratings which are difficult to assess, such as the reactions, actions, and mood and feelings of people with dementia, is satisfactory within an interval of one week. The satisfactory results for intra-rater reliability are similar to the findings of several intra-rater reliability investigations of other self-rated (Brod et al., Reference Brod, Stewart, Sands and Walton1999; Smith et al., Reference Smith2005a; Matsui et al., Reference Matsui2006; Trigg et al., Reference Trigg, Jones and Skevington2007; Karim et al., Reference Karim, Ramanna, Petit, Doward and Burns2008) and proxy-rated (Logsdon et al., Reference Logsdon, Gibbons, McCurry and Teri1999; Weiner et al., Reference Weiner, Martin-Cook, Svetlik, Saine, Foster and Fontaine2000; Garre-Olmo et al., Reference Garre-Olmo2010) measurements, regardless of the operationalization of Qol.
Limitations
This is the first analysis of the inter-rater and intra-rater reliability of the German QUALIDEM based on a large sample. Whereas the calculated sample size was achieved for people with mild to severe dementia, the appropriate sample size was not reached for people with very severe dementia. Despite additional recruitment, our sample size was below the calculated sample size by 40 people with very severe dementia for the inter-rater evaluation and by 23 for intra-rater evaluation. As the resulting ICC values for people with very severe dementia seemed to be accurate enough according to maximum CI lengths of 0.30 (inter-rater reliability) and 0.29 (intra-rater reliability), we did not further expand the data collection. Given the observed low prevalence of people with very severe dementia in nursing homes, the required additional effort appeared disproportionate to the received beneficial statistical accuracy.
Unfortunately, only one previous study evaluated the inter-rater and intra-rater reliability of Qol measurements (Ettema et al., Reference Ettema, Droes, de Lange, Mellenbergh and Ribbe2007b), allowing an appropriate comparison and discussion of the study results.
Conclusions
This study showed insufficient inter-rater reliability and sufficient intra-rater reliability for all subscales of both versions of the German QUALIDEM based on individual ratings by nurses. The results are possibly due to the meaning of the items, which represent the reactions of people with dementia to interactions and actions and the affects, moods, behavior and direct interactions of people with dementia. The proxy rating of such items is difficult and depends on individual situations and interactions and the general relationship between the person with dementia and the proxy. According to our results, an individual nurse can define the meaning of such items over time, but the definitions of different nurses vary. A precise definition and, if necessary, a revision of items 1, 3, 5, 6, 8, 10, 12, 13, 20, 21, 22, 24, 25, 27, 28, 31, 33, 35, 36, 37 and 40 and instructions for item assessment within a new measurement manual might improve the inter-rater reliability of the German QUALIDEM. In addition, obligatory training of proxy-rating nurses on the Qol of people with dementia in general and on the items’ meanings is recommended. The results show sufficient inter-rater reliability for a collaborative QUALIDEM rating, which is a manageable but complex, time-consuming and expensive method for use in intervention studies.
The intra-rater reliability results are satisfactory for all subscales. However, multiple Qol ratings over time by the same nurse in longitudinal studies seem to be limited by the high turnover rates of nurses in German nursing homes.
Based on the results for inter-rater reliability, further development of the QUALIDEM based on a manual with clear and accurate definitions of the mentioned items and a standardized education program for proxy raters are needed.
Conflict of interest
None declared.
Description of authors’ roles
Study design: MND, SB, MH, GM
Data collection and handling: MND, CGGS, OD
Data analysis: MND, OD
Manuscript preparation: MND, CGGS, OD, GM, SB, MH
Acknowledgments
We thank the nursing home staff, the professional caregivers and the residents with dementia for their participation in our investigation of the inter-rater and intra-rater reliability of the German QUALIDEM.
