Introduction
The world prevalence of dementia has recently been estimated at 24.3 million people. This is expected to double over the next 20 years (Ferri et al., Reference Ferri2005). Services that increase performance abilities and well-being in both patients and caregivers are required (Gitlin et al., Reference Gitlin, Hauck, Dennis and Winter2005; Mittelman et al., Reference Mittelman, Haley, Clay and Roth2006; O'Connor et al., Reference O'Connor, Ames, Gardner and King2009). Decline in memory and executive functions are characteristics of the degenerative process in dementia. Disabilities in performing daily tasks are the most important reason for the need for care, including nursing home placement, and are a burden to caregivers and society (Froelich et al., Reference Froelich2009). Tailor-made, individualized, multi-component interventions that are aimed at improving daily functioning and quality of life for both patient and caregiver are shown to be most effective in the home setting (Graff et al., Reference Graff, Vernooij-Dassen, Thijssen, Dekker, Hoefnagels and Rikkert2006, Reference Graff, Vernooij-Dassen, Thijssen, Dekker, Hoefnagels and Olderikkert2007; Smits et al., Reference Smits, de Lange, Dröes, Meiland, Vernooij-Dassen and Pot2007; Campbell et al., Reference Campbell2007; Vernooij-Dassen et al., Reference Vernooij-Dassen, Vasse, Zuidema, Cohen-Mansfield and Moyle2010). To tailor interventions to individual needs, it is important to know the patient's ability to perform activities and their relationship to disturbances in the specific cognitive processes that underpin particular task performance. Neuropsychological tests have a low ecological validity and do not adequately predict difficulty in daily functioning within the home context (Bouwens, Reference Bouwens2009). This is why sensitive instruments for measuring everyday functional abilities are very important in dementia care (Katona et al., Reference Katona, Livingston, Cooper, Ames, Brodaty and Chiu2007; Black et al., Reference Black2009). Observation of the performance of daily activities within the home context gives unique information about the influence that cognitive disturbances have on daily life. These assessments form the basis of effective treatment interventions. There are several reliable and valid observation instruments available to the occupational therapist for this purpose, such as the Assessment of Motor and Process Skills (AMPS) (Nygård et al., Reference Nygård, Bernspång, Fisher and Winblad1994; Doble et al., Reference Doble, Fisk, MacPherson, Fisher and Rockwood1997; Fisher, Reference Fisher2003; Liu et al., Reference Liu2007) and the Arnadóttir OT-ADL neurobehavioral evaluation (Arnadóttir, Reference Arnadóttir1990; Arnadóttir and Fisher, Reference Arnadóttir and Fisher2008; Arnadóttir et al., Reference Arnadóttir, Fisher and Löfgren2009). Usage depends on the type of activities a patient prefers to perform. The AMPS mainly consists of standardized household tasks and some other standardized instrumental activities of daily living, and measures quality of performance in motor and process skills. The A-ONE observes primary self-care tasks and measures levels of independence and related cognitive dysfunctions. The A-ONE has a ceiling effect because of the lack of complexity in tasks. In both instruments, the observed activities are standardized, which may have the disadvantage that the whole task or the criteria to perform it do not fit to the clients’ routines. For dementia patients, it is highly important to perform meaningful activities they are motivated to do and in the routines and context they are familiar with. These components add to their quality of life (Teunisse and Derix, Reference Teunisse and Derix1997; Graff et al., Reference Graff, Vernooij-Dassen, Thijssen, Dekker, Hoefnagels and Rikkert2006, Reference Graff, Vernooij-Dassen, Thijssen, Dekker, Hoefnagels and Olderikkert2007). Therefore, the Perceive, Recall, Plan, and Perform (PRPP) system seems to be an instrument with potential to assess dementia patients in the home environment because it enables the patient to choose activities based on individual preferences and needs (Buikema et al., Reference Buikema, van Norel, Tigchelaar and Steultjens2009). The PRPP examines errors and the effectiveness of cognitive information processing in the performance of self-chosen tasks.
Psychometric properties of the PRPP for traumatic brain injury, learning difficulties, chronic pain, and schizophrenia are evaluated in small groups and show adequate outcomes on content validity, construct validity, internal consistency, concurrent validity, inter-rater reliability, intra-rater reliability, and responsiveness (Chapparo and Ranka, Reference Chapparo and Ranka2008; Aubin et al., Reference Aubin, Chapparo, Gélinas, Stip and Rainville2009; Nott et al., Reference Nott, Chapparo and Heard2009; Voigt-Radloff et al., Reference Voigt-Radloff2009). However, no data are available for the use of PRPP in older people with dementia, although this target group of patients needs a tailor-made assessment to evaluate their performance in familiar daily activities in their own context. Inter-rater reliability is an important feature of measures that assess performance of abilities in daily activities by observation. Criterion-referenced measures incorporate the variability in three foci: the demands of the task, the capacity of the person to perform that task, and the context of performance. Such measures therefore have high ecological validity. However, they have the disadvantage that absolute inter-rater agreement is hard to achieve (Nott et al., Reference Nott, Chapparo and Heard2009).
The objective of this study is to evaluate whether the PRPP assessment has reliable standardization criteria that are consistently used for the assessment of performing familiar daily tasks in community-dwelling dementia patients by multiple raters.
Methods
Patients
PRPP baseline data of the WHEDA trial were used in this study. This German multi-center randomized controlled trial evaluated the effectiveness of community occupational therapy for dementia patients and their caregivers. The design of the WHEDA study has been published previously (Voigt-Radloff et al., Reference Voigt-Radloff2009). Inclusion criteria of participants were mild to moderate dementia of the type Alzheimer's disease or mixed type, MMSE ranging from 10 to 24, people dwelling at home either together with their primary caregiver or the primary caregiver provides care at least twice a week. People with a Geriatric Depression Scale score (GDS 30) >12, in major need of nursing care, with unstable medical conditions or severe behavioral disturbances were excluded (Voigt-Radloff et al., Reference Voigt-Radloff2009). In this way a homogenous group was established with a focus on performance abilities.
Raters
The Dutch occupational therapists (OTs) who had completed a PRPP assessment course (N = 79) were asked by email to participate in this inter-rater reliability study. Twenty-five were motivated to participate and agreed to score the PRPP videos independently at home. No additional inclusion criteria for raters were used. Dutch PRPP trained OTs were chosen as raters instead of German PRPP trained OTs because of the availability of a PRPP alliance in the Netherlands and the use of the official translated (from English) Dutch PRPP scoring forms that are not available in German.
Measurements
Patients' characteristics were measured within the WHEDA trial by German research assistants. The baseline characteristics regarding sex, age, education, years since onset of dementia, and baseline outcomes measured with the Interview of Deterioration in Daily living activities in Dementia (IDDD), Dementia Quality of Life Instrument (DQoL), Short form of the SF-36 measuring health status (SF-12), GDS, and the Mini-Mental State Examination (MMSE) will be presented to provide a clear description of the group of dementia patients involved in this study.
The PRPP system of task analysis is a two-stage, criterion-referenced measure instrument that examines the effectiveness of cognitive information processing in performing meaningful tasks according to individualized standards (the criterion) set by the participant's particular situation. Stage One incorporates a task analysis procedure, in which an observed familiar task is subdivided into all relevant steps, and types of mistakes are measured, for example being inaccurate, repeating steps, omitting steps, and wrong timing, in the performance of steps in the chosen task. Level of independence in performing the task can be calculated in Stage One by dividing the amount of wrongly performed steps by the amount of all steps of the task. Stage Two consists of the analyses of information processing and incorporates 34 items divided in the subscales PRPP. Items cover information processing aspects such as attending; sensing; discriminating; recalling facts, schemes, and procedures; mapping; programming; evaluating; initiating; continuating; and controlling. Each item is rated on a 3-point scale (inadequate, inconsistent, adequate). Both therapist and patient decide together which task is relevant to observe given the performance difficulties in meaningful activities as expressed by the patient.
The PRPP was scored using the Dutch PRPP assessment scoring form including Stage One independent score and Stage Two information processing items. For Stage One, the activity steps to be scored were preset by the raters from the WHEDA trial based on individual performance of the filmed task according to the routines of the patient. Within the WHEDA trial, each patient performed two self-chosen tasks out of 29 preset meaningful tasks. Scoring criteria were used according the PRPP manual that is instructed in the PRPP assessment course (Chapparo and Ranka, Reference Chapparo and Ranka2008).
Procedures
Thirty video footages out of 258 baseline performances of two self-chosen daily tasks in the home environment by a dementia patient were randomly selected by a research assistant who was blinded for patient identity and performance characteristics. Exclusion criteria for video footages were duration > 15 minutes, comments from WHEDA rater on visibility of the video, and/or comments on therapist's influence on performance. An independent statistician randomized the videos by raters in such manner that each of the 25 raters scored 12 videos and that each of the 30 videos was scored by ten different raters. This resulted in 300 cases and provides a good power to establish inter-rater test reliability (intra-class correlation (ICC) > 0.70) (Walter et al., Reference Walter, Eliasziw and Donner1998).
Raters were provided with CD-Roms with the 12 video films and scoring forms and instructed to rate independently at home and not discuss findings with other raters. Independence of raters was enhanced by providing each rater with a unique set of 12 videos.
Statistical analyses
For Stage One independence scores and Stage Two subquadrant scales, quadrant scales and whole scale ICCs were computed with SPSS 18 (PASW statistics, Chicago, IL, 2009). A one-way model was used for ICC calculation since both raters and video's were randomly assigned and no videos were rated by all raters. Average measure ICCs reflect the consistent use of the PRPP standardization criteria by multiple raters. Scores ≥ 0.70 will be regarded as good inter-rater consistency.
Single measure ICCs reflect the absolute agreement between raters and are expected to be fair or moderate. An ICC from 0.20 to 0.39 is regarded as fair inter-rater reliability, 0.40 to 0.59 as moderate inter-rater reliability, 0.60 to 0.79 as substantial, and 0.80 or above as outstanding (Landis and Koch, Reference Landis and Koch1977).
Results
Patient characteristics
Table 1 summarizes the characteristics of the 28 dementia patients at the time of the video shoot. Recently diagnosed (mean 1.7 year, SD 1.5) mild dementia patients (mean MMSE score = 20, SD 2.5) with low to moderate need for assistance (IDDD performance mean = 14, SD 10.5) were included in this study.
Characteristics of included patients, N = 28
MMSE = Mini-Mental State Examination; GDS = Geriatric Depression Scale; SF-12 = Short Form of the SF-36 measuring health-status; DQoL = Dementia Quality of Life Instrument; IDDD = Interview of Deterioration in Daily Living Activities in Dementia.
All participating raters (N = 25) were OTs or researchers who had been trained in PRPP use. The majority (N = 24 therapists) had followed the assessment course one to three years prior to this research, only one had been trained 11 years before. All were experienced OT in adult healthcare.
Table 2 shows the ICC for Stage One and Stage Two: subquadrant scales, quadrant scales, and whole scale. The ICC for Stage One independence score showed good outcomes both for absolute inter-rater agreement and for consistency of test criteria standardization. Stage Two subscales, quadrants, and whole scale show good inter-rater consistency of test criteria standardization.
Inter-rater single measure and average measure reliability of the PRPP
Bold and Italic are used to refer to the specific scale. Bold refers to whole scales Stage One and whole scale Stage Two. Bold and Italic refer to quadrant scores. Italic refers to subquadrant scores.
Stage Two subscales and quadrants show fair inter-rater absolute agreement. The whole PRPP information processing scale shows moderate absolute agreement.
Discussion
This study shows that the PRPP system of task analysis has sufficient inter-rater reliability for the assessment of performance of familiar daily activities in the home context in mild dementia patients who required little to moderate assistance in performing daily activities. Both the independence score and the information processing scale show consistent test use by multiple raters indicating reliable standardization of the PRPP assessment of task analysis. Absolute inter-rater agreement is not expected to be high in ecologically valid, criterion-referenced measures. This is confirmed for Stage Two. The absolute agreement between raters for the independence score is good.
The results of this study for Stage Two correspond in general with other studies that have evaluated inter-rater reliability of the PRPP assessment for other diagnostic groups. Nott et al. (Reference Nott, Chapparo and Heard2009) evaluated inter-rater reliability of Stage Two of the PRPP assessment in two studies with, 15 and five traumatic brain injury patients respectively, who had been scored by three and nine raters, and found high reliability (single measure agreement ICC between 0.51 and 0.77; test consistency all ICC > 0.8). Nott et al. (Reference Nott, Chapparo and Heard2009) used a three-way ICC analysis incorporating the three factors of variability – patients, tasks, and raters – which explains the higher ICCs compared to the current single absolute agreement outcomes of the current study. Aubin et al. (1990) reported good (for PRPP total score; 0.77) and moderate (for PRPP quadrant scale scores; 0.63–0.69) outcomes for inter-rater reliability of the PRPP assessment Stage Two in 15 schizophrenia patients who were scored by three raters on two tasks each using a two-way mixed effects model for absolute agreement. This is the first study that also looked at inter-rater reliability for Stage One of the PRPP and showed good results on both absolute inter-rater agreement and standardization.
The reliability outcomes for Stage One of the PRPP assessments do show the power of the instrument to assess individual performance of routine tasks of dementia patients in their home context. In the PRPP, the necessity for individual choices regarding specific tasks and environmental demands is well recognized and therefore is an important tool in assessing ecologically valid performance levels in the performance of daily tasks. The criterion-referenced standardization of scoring rules is applied consistently by multiple raters as shown in this study for both stages. Therefore, the PRPP is an important addition to the available assessment tools for occupational therapists treating home-dwelling dementia patients.
A limitation of this study is that only patients with mild dementia who had been diagnosed in the last few years were included in the study. Therefore, the results on reliability of this instrument cannot be generalized to all dementia patients. However, this sample is comparable for mood and quality of life with the Dutch study of Graff et al. (Reference Graff, Vernooij-Dassen, Thijssen, Dekker, Hoefnagels and Olderikkert2007) in which mild and moderate community-dwelling dementia patients were included.
Another limitation of this study regarding Stage One is the use of preset task analyses by raters. This was done to make the ratings comparable regarding the number of subtasks, the range of subtasks, and the equation of the independence score. In routine care, each OT has to perform the task analyses for the individual situation and tasks observed, which increases variability in scoring the PRPP. Future research into Stage One should also evaluate the consistency in task analyses standardization applied by OTs.
This study was not designed to establish absolute inter-rater agreement on Stage Two information descriptors (item) level. To look at inter-rater reliability on item level of the PRPP, one should take into account all the variability that occurs in PRPP assessment. This means that the design should have enough statistical power; should use a nested design in which the variety in tasks, patients, environments, and raters are adequately addressed (Nott et al., Reference Nott, Chapparo and Heard2009); and that all raters should rate all available cases. At this stage of PRPP implementation in the Netherlands, the required number of experienced raters was not available to perform such a large study.
The absolute agreement ICC outcomes for Stage Two are fair (subscales, scales) and moderate (scales). These outcomes might be slight underestimates of real absolute agreement because of some specific study characteristics. The video footages were of German-speaking dementia patients living in Germany and filmed in their home environments. Although all Dutch raters have sufficient knowledge of the German language, which is generally taught in high school, and of German culture to rate these performances, the influence of a different language and less familiar routines in performances need to be taken into account. Habits in performing activities, such as cleaning shoes, are known to be different between adjacent countries. However, the good consistency outcomes of the PRPP show that the PRPP is a reliable tool to be used by multiple raters in observing various and less familiar tasks.
In conclusion, this study pertaining to inter-rater reliability of the PRPP for community-dwelling dementia patients shows that the PRPP is a consistent tool for use by multiple raters and has the potential to be a good measuring instrument to rate performance of patients with mild dementia in a variety of routines and tasks of meaningful daily activities in a variety of environments. The criterion-referenced standardized PRRP can be used to assess “real life” performances reliably according to individual standards. Future research should focus on other validity and reliability characteristics of the PRPP for dementia patients at different stages of the disease, with additional symptoms such as depression, and at different living environments. Rasch analysis procedures could help to address the several variety issues that occur in criterion-referenced measures.
Conflict of interest declaration
None.
Description of authors' roles
All authors were involved in research design, planning, and in writing the paper. In addition, Esther Steultjens organized the data collection and R. Leonhart performed the statistical analyses.
Acknowledgments
This research was made possible by all participants, patients, raters, and research assistants, whom we thank for their cooperation. The WHEDA trial is financed by a grant from the German Federal Ministry of Health, one of the “Lighthouse projects” (Grant number IIA5-2508FSB111/44-004).
