Introduction
Exercise undertaken regularly can have pronounced physical and functional benefits for adults with dementia (Ahlskog et al., Reference Ahlskog, Geda, Graff-Radford and Petersen2011; Littbrand et al., Reference Littbrand, Stenvall and Rosendahl2011). To understand these benefits, we are reliant on the sound psychometric properties of the measures utilized to assess functional and physical capacity. However, most psychometric assessments of these measures have been done among adults with no cognitive impairment. For those with dementia impaired balance confidence, elevated anxiety towards new or foreign tasks, decreased comprehension, and increased confusion may impede the individual's ability to follow instructions and complete the protocol, thereby significantly reducing the reliability of the measures (Brill et al., Reference Brill, Drimmer, Morgan and Gordon1995). Data in current literature investigating the reliability of common exercise intervention measures with older adults with dementia are mixed and inconclusive.
While some studies reported the relative reliability of physical performance measures among adults with dementia (Tappen et al., Reference Tappen, Roach, Buchner, Barry and Edelstein1997; Rockwood et al., Reference Rockwood, Awalt, Carver and Macknight2000; Binder et al., Reference Binder, Miller and Ball2001; Thomas and Hageman, Reference Thomas and Hageman2002; Lin et al., Reference Lin, Hwang, Hu, Wu, Wang and Huang2004; Conradsson et al., Reference Conradsson2007; van Iersel et al., Reference Van Iersel, Benraad and Rikkert2007; Ries et al., Reference Ries, Echternach, Nof and Gagnon2009; Blankevoort et al., Reference Blankevoort, Van Heuvelen, Boersa, Luning, De Jong and Scherder2010; Suttanon et al., Reference Suttanon, Hill, Dodd and Said2011; Blankevoort et al., Reference Blankevoort, Van Heuvelen and Scherder2012), most have failed to assess the absolute reliability, the measure of magnitude of change, and individual variability. Identifying or developing appropriate measures that can be employed with older adults with dementia will allow the accurate assessment of outcomes in exercise interventions aimed at improving functional well-being in this population.
The aim of this pilot study was to gain preliminary insight into the relative and absolute reliability of functional performance measures that have been commonly used in studies involving cognitively sound older adults, among a group of adults with dementia who live in residential aged care facilities (RACFs).
Methods
Design and participants
This study was undertaken to establish the feasibility of conducting a larger, powered study of functional performance test–retest reliability among adults with dementia. Participants were recruited from two RACFs in South-East Queensland, Australia. The inclusion criteria were: a diagnosis of dementia, not wheel-chair bound, and no unstable or terminal disease. Those with walking aids (frame or stick) were included as long as they could ambulate at least 10 m. Participants with pacemakers were excluded due to Bio-electric Impedance Analysis (BIA) measurement. Participants required medical practitioner or RACF-registered nurse health status approval and all participants’ substitutionary decision makers were contacted to provide informed consent for participant in the project. Verbal assent from participants was necessary on day of testing and they were free to withdraw from the project for any reason at any time, and were allowed to rest between measures if needed. The study was approved by (identified after review) Ethics Committee.
Procedures
Prior to the reliability assessment, RACF staff was consulted and confirmed dementia diagnosis, and information regarding gender and date of birth of participants was provided. A number of common functional performance, grip strength, and anthropometric measures that have been found to be highly reliable in older, cognitively sound adults were collected from participants in two sessions undertaken seven days apart. Assessment times, location, and order of testing were kept constant for both occasions. Measurement administrators, trained in the prescribed measures by a qualified exercise physiologist, experienced in measuring functional capacity in older adults, were assigned specific measures throughout to minimize inter-tester error. RACF staff assisted with moving clients between measures, and under the guidance of the assessor helped maintain client safety during measures that involved standing, walking, or balancing. To ensure safety, the measurer and a staff member stayed close to the participants during the standing measures and the participants were encouraged to sit and rest between trials. Prior to the assessments, participants were familiarized to the measure by both demonstration and verbal description.
Measures
Anthropometry
Height and weight were measured by stadiometer (Charder Electronic Co, Ltd., Taichung Hesin, Taiwan) and electronic scale (SECA Medical Scales and Measuring Systems, Birmingham, UK), respectively. Percentage body fat, lean mass, total body water, and body mass index (BMI) (kg/m2) were measured by Maltron 906 50-kHz tetrapolar BIA (Maltron International Ltd., Rayleigh, UK). Participants were instructed to lie supine with their hands and legs slightly apart, and four electrodes were applied to the right side of the body at the hand, wrist, foot, and ankle. Parameters specific to each client were entered and a non-invasive analysis undertaken.
Performance-based measures
Participants undertook two standardized physical performance battery measures designed to assess balance and functional capacity in older adults, and a measure of muscle strength. Measurement protocols have been published elsewhere, and will only be briefly detailed to here.
The Short Physical Performance Battery (SPPB; Guralnik et al., Reference Guralnik1994) is an assessment of functional capacity and contains three measures: a hierarchal measure of standing balance, a timed 2.4-m walk at habitual speed, and a timed repeated (five) chair stands. Measures are scored individually and an overall summary score is also obtained. The Balance Outcome Measure for Elder Rehab (BOOMER; Haines et al., Reference Haines, Kuys, Morrison, Clarke, Bew and Mcphail2007) is a measure of agility, dynamic and static balance consisting of four measures: a maximum step test in 15 sec, the Timed Up and Go (TUG) test, the static timed standing test, and a measure of functional reach. Hand grip muscle strength (kg) was measured using a Jamar dynamometer (Sammons Preston Roylan, Bolingbrook, USA). Participants were seated with their elbows at their sides and at 90°. When given a “GO” signal, participants squeezed the dynamometer as hard as they could. Both the left and right hands were tested (Taekema et al., Reference Taekema, Gussekloo, Maier, Westendorp and De Craen2010).
Notes were also taken during measurement, which included: deviations from the protocol, the use of additional verbal or physical assistance, difficulties in comprehension for participants, confusion and behavioral issues, and any other anomalies from the testing protocol. This information was used to draw subjective conclusions regarding the appropriateness and applicability of the measures and methods of assessment.
Statistical analysis
Data were analyzed using SPSS 21 (SPSS Inc., Chicago, USA). Intra-class Correlation Coefficient (ICC) determined the relative test–retest reliability of measures. Relative reliability was deemed acceptable if the ICC statistic was greater than 0.71 (Thomas and Nelson, Reference Thomas and Nelson2001). Absolute reliability was assessed by measuring the typical error of measurement (TEM) (see Figure 1), a within-subject measure of variation across measurement times (Domholdt, Reference Domholdt2000) and was deemed acceptable if the TEM value was less than 10% the mean cumulative test–retest scores (Schwenk et al., Reference Schwenk, Gogulla, Englert, Czempik and Hauer2012).
Typical Error of Measurement (TEM) equation.
Bland–Altman plots were used as a visual representation of pre–post agreement, with mean difference (MD) and Limits of Agreement (LoA) (±1.96 SD) imposed on the plots for each measurement. Limits of Agreement represents the expected difference between pre and post measurements to a 95% CI. Mean difference is a function of the average difference between test and retest measurement scores.
As a function of the difference between test–retest measurement result and the score during pre-testing, Bland–Altman plots helped establish whether there was good agreement at the group level (a MD close to zero), and at the individual level (narrow bands of LoA). A sample t-test was used to determine if MDs significantly deviated from 0. Acceptable LoA values were determined by expected variance of comparative measures (Schwenk et al., Reference Schwenk, Gogulla, Englert, Czempik and Hauer2012). All analyses were two-tailed and a value of p < 0.05 was required for significance. All values were expressed as mean ± standard deviation.
Results
Informed consent was received for 17 participants to be included in the study. The study had five dropouts, which represents an attrition rate of 29.41%. Of those who dropped out, one was removed from the study after becoming extremely agitated during the initial measurement, with another removed under similar circumstances during retest measurement. The remaining four either did not feel well or did not give verbal assent on the day. Twelve participants completed both test–retest measurements and have been included for statistical analysis.
Participants were predominantly female (n = 11, 91.66%) and were aged 83.25 ± 9.94 years. Dementia diagnosis was confirmed by aged care facility management, but information on dementia severity was not collected. However, the low walking speeds (0.41m/s at baseline, range = 0.62–0.24 m/s) in this sample are suggestive of more severe cognitive impairment, with previous links between walking speed and cognitive functioning already been established (Sheridan et al., Reference Sheridan, Solomont, Kowall and Hausdorff2003; Ble et al., Reference Ble2005; Atkinson et al., Reference Atkinson2007; Kemoun et al., Reference Kemoun2010). Poor time to complete the TUG, five repeated chair stand measures, and low SPPB summary scores at baseline also indicate the low level of function in this sample (Guralnik et al., Reference Guralnik1994; Guralnik et al., Reference Guralnik, Ferrucci, Simonsick, Salive and Wallace1995; Bean et al., Reference BEAN2002; Cesari et al., Reference Cesari2005; Bohannon, Reference Bohannon2006).
Acceptable levels of reliability were shown for all anthropometric measures, grip strength, chair stands, the SPPB summary score, TUG test, and the step test (left – L). Step test (right – R) approached acceptable levels. Acceptable levels of absolute reliability were shown for five of the six anthropometric measures (height, weight, BMI, lean mass, and total body water) and approached acceptable body fat percentage (10.70%). Grip strength (L and R) approached acceptable levels (10.00% and 11.66% respectively). All other measures showed unacceptable levels of absolute reliability (12.76% for chair stands to 43.33% for step test (R)). All test–retest, TEM, and ICC data are presented in Table 1.
Test–retest results and reliability for common functional performance measures
SD = standard deviation; TEM = Typical Error of Measurement; ICC = intra-class correlation coefficients; BMI = body mass index; SPPB = Short Physical Performance Battery; BOOMER = Balance Outcome Measure for Elder Rehab.
Note. Percentage value in brackets are presented as the percentage of average test–retest measurements.
Bland–Altman analysis results are presented in Table 2. There were no significant differences between group means and the expected value of zero for all 17 measures. The TUG (M = 4.77, p = 0.090) approached significance, while the functional reach (M = –4.08 cm, p = 0.146) had a large non-significant deviation from zero. Figure 2 is the Bland–Altman plot for the grip strength (R), with MD and LoA superimposed. With a zero mean (M = 0.17 kg) and narrow LoA (–4.55–4.88 kg), it is possible to conclude that the grip strength (R) is an appropriate measure in this population. By contrast, the Bland–Altman plot for the TUG is presented in Figure 3. Large, non-significant deviation of the mean and wide LoA (–11.74–21.29 sec) suggests that this measure may be inappropriate for use in a group of adults with dementia residing in a RACF. Body fat percentage, total body water, standing balance, five repeated chair stands, step test, TUG, functional reach and static timed standing, all had unacceptable LoA.
Bland–Altman analysis of levels of agreement between test–retest measures
MD = mean difference; SD = standard deviation; BMI = body mass index; SPPB = Short Physical Performance Battery; BOOMER = Balance Outcome Measure for Elder Rehab.
Note. p values presented are results of one sample t-test to determine if mean significantly differs from 0.
Bland–Altman plot for grip strength (right hand).
Bland–Altman plot for the Timed Up and Go.
A number of participants appeared to have reduced capacity in many of the assessments both as a component of dangerously reduced balance and balance confidence, and their ability to translate protocols to accurately undertake the measure. Deviations from strict protocol were also noted (including extra prompts) and are addressed in detail in the discussion. Generally, participants required extra prompts to be reminded of protocol (i.e. would forget what it was), or constant cueing throughout to be able to successfully complete the measure. Participants often required the contact with the arms of the chair to complete chair stands and the TUG test, which is outside the protocol, and required assistance in balancing during the step test. Many challenges also exist for participant with their comprehension of what the task was requiring of them often poorly translated to the movement or stance required of them. For example, the SPPB's semi-tandem position (see Figure 4) proved impossible to achieve for a percentage of participants, even with extensive assistance and prompting from researchers.
Three positions of the balance task within the SPPB. Participants are required to stand for 10 sec in the semi-tandem position. Measure progressed or regressed depending on the ability of participants in semi-tandem position. Image adapted from Guralnik (Reference Guralnik2012).
Discussion
This study appears to be one of the first to examine the psychometric properties of two commonly used measures of functional performance and common anthropometric measures in RACF adults with dementia. Results indicated acceptable levels of relative and absolute reliability for most anthropometric measures, while showing acceptable relative reliability at the group level and TEM values slightly above acceptable limits. In contrast, the functional performance measures of the SPPB, the BOOMER and hand grip appear less reliable. While grip strength, TUG, step test (L), SPPB summary score, and chair stands had demonstrably acceptable levels of relative reliability, unacceptably high TEM values question the applicability and usefulness of such tests in clinical practice and research. These data have significant implications for clinicians and researchers when deciding which measures are best used with a cognitively challenged older cohort. Bland–Altman plots confirmed that while group means do not significantly deviate from zero, demonstrating their applicability within a population setting, high levels of variability, and wide LoA confirmed that, at an individual level, significant shortcomings exist in the measurement of functional performance in older adults with dementia living in RACFs. Our data suggested careful consideration is required in choosing measures, and that there is scope for validating adaptations of the present measures to make allowances for the support required for the very old, low-functioning adult with dementia.
Currently no literature exists which examines the reliability (both relatively and absolutely) of anthropometric measures and grip strength in adults with dementia. Our data suggest that these measures are acceptable in terms of relative reliability and absolute reliability and demonstrated good levels of group and individual agreement. These measures were reliable and can be used with this population.
This also seems to be the first study to examine the reliability of the SPPB and BOOMER functional performance batteries with adults with dementia, although, studies have attempted to establish the reliability of the individual test items with varying success (Lin et al., Reference Lin, Hwang, Hu, Wu, Wang and Huang2004; Conradsson et al., Reference Conradsson2007; van Iersel et al., Reference Van Iersel, Benraad and Rikkert2007; Blankevoort et al., Reference Blankevoort, Van Heuvelen and Scherder2012). Some authors have only examined the ICC statistic and concluded tests are suitable for use within this population (Tappen et al., Reference Tappen, Roach, Buchner, Barry and Edelstein1997; Binder et al., Reference Binder, Miller and Ball2001; Lin et al., Reference Lin, Hwang, Hu, Wu, Wang and Huang2004; van Iersel et al., Reference Van Iersel, Benraad and Rikkert2007). While the relative reliability determines if differences exist at the group level (Suttanon et al., Reference Suttanon, Hill, Dodd and Said2011), it is necessary to examine the absolute reliability and variability of these measures in order to ascertain the consistency of the measurement and determine the magnitude of statistical relevant changes in a measure (Suttanon et al., Reference Suttanon, Hill, Dodd and Said2011). Both forms of reliability are necessary to determine the applicability of measures.
The TUG (Tappen et al., Reference Tappen, Roach, Buchner, Barry and Edelstein1997; Rockwood et al., Reference Rockwood, Awalt, Carver and Macknight2000; Thomas and Hageman, Reference Thomas and Hageman2002; Ries et al., Reference Ries, Echternach, Nof and Gagnon2009; Blankevoort et al., Reference Blankevoort, Van Heuvelen, Boersa, Luning, De Jong and Scherder2010; Suttanon et al., Reference Suttanon, Hill, Dodd and Said2011), step test (Suttanon et al., Reference Suttanon, Hill, Dodd and Said2011), functional reach (Rockwood et al., Reference Rockwood, Awalt, Carver and Macknight2000; Suttanon et al., Reference Suttanon, Hill, Dodd and Said2011), chair stands (Binder et al., Reference Binder, Miller and Ball2001; Suttanon et al., Reference Suttanon, Hill, Dodd and Said2011; Blankevoort et al., Reference Blankevoort, Van Heuvelen and Scherder2012), walking speed tests (Binder et al., Reference Binder, Miller and Ball2001; Ries et al., Reference Ries, Echternach, Nof and Gagnon2009; Blankevoort et al., Reference Blankevoort, Van Heuvelen and Scherder2012) have all been shown to have acceptable ICC values in dementia populations. While our data showed lower ICC for our walking test, step test, and functional reach than has been shown previously, Bland–Altman plots demonstrated their applicability at the group level. Low TEM values could be a function of the lower cognitive processing ability of the participants within this study. Our findings are consistent with previous research that unacceptable levels of absolute reliability limit the individual applicability of these measures in this population (Ries et al., Reference Ries, Echternach, Nof and Gagnon2009; Suttanon et al., Reference Suttanon, Hill, Dodd and Said2011; Blankevoort et al., Reference Blankevoort, Van Heuvelen and Scherder2012).
Part of this study was to qualitatively evaluate the ability of participants to comprehend and complete the measures prescribed. The ability of participants to complete the tasks was compromised by their inability to comprehend or maintain attention throughout, or require physical assistance to successfully complete the task. As consistent with previous research, participants needed constant cueing to remind them of the task, extra prompts to be able to comprehend the task or physical assistance (i.e. using the arms of a chair to stand or help from the researchers in ambulating) (van Iersel et al., Reference Van Iersel, Benraad and Rikkert2007; Blankevoort et al., Reference Blankevoort, Van Heuvelen and Scherder2012). This has severe implications for the reliability and validity of measures. Hauer and Oster (Reference Hauer and Oster2008) outlined similar difficulties of working within this population and have stated previously that acceptable relative reliability does not guarantee measure appropriateness. This is further evidenced by our lack of acceptable absolute reliability, the practicalities of which undermine the validity of the measures. Therefore, drawing any form of meaningful conclusion or comparison is difficult from using the SPPB or BOOMER tools and implies a paradigm shift away from protocol driven measures may be necessary if an accurate assessment of functional performance in older adults with dementia is to be achieved.
While this study demonstrated the poor psychometric properties of common measures of functional performance in this population, it operates under a protocol-driven measurement theory, which is considered outdated in the field of psychology (Embretson and Reise, Reference Embretson and Reise2000). The authors believe that an Item Response Theory (IRT) approach would provide an improved alternative in the assessment of functional performance in adults with dementia (Embretson and Reise, Reference Embretson and Reise2000) as it has already been postulated as potentially beneficial in psychomotor applications (Disch, Reference Disch1987; Safrit, Reference Safrit1987; Spray, Reference Spray1987). Item Response Theory is a mathematical probabilistic model which attempts to take an individual's response to measurement items and equate them on a spectrum of ability. The theory is not group specific, so tests can be calibrated on a wide range of functional abilities, including among those with dementia, and precedence already exists in motor development in young children (Hands et al., Reference Hands, Sheridan and Larkin1999). Moreover, the theory allows for mixed methodology assessments. Therefore, in a group of adults with moderate-to-severe dementia, a proxy survey could be completed and compared to individuals who were physically able to complete a functional assessment. However, while direct comparisons of, for example, balance could not be made, individuals would be compared on a latent trait spectrum of functional ability, in a similar way that intelligence is measured in the field of psychology (Embretson and Reise, Reference Embretson and Reise2000). While challenges still exist for measurement theorists in psychomotor applications (Zhu, Reference Zhu, Wood and Zhu2006), Spray's (Reference Spray1987) challenge is still pertinent and scope certainly exists to determine whether IRT offers greater advantages than more classical approaches in many psychomotor applications, including measuring older adults with dementia.
While this does not negate the issue of reliability, it does provide a flexible framework that allows for measures not being completed due to cognitive or behavioral issues, yet still provide information that can be assessed against functional improvement or decline.
There are several limitations to the present study. First is the sample size, which reduces the statistical power of the analyses. However, as a pilot, the study mirrored sample sizes from recent exercise intervention trials among adults with dementia (Santana-Sosa et al., Reference Santana-Sosa, Barriopedro, Lopez-Mojares, Perez and Lucia2008; Venturelli et al., Reference Venturelli, Scarsini and Schena2011; Coelho et al., Reference Coelho2013). Another limitation to this pilot is the aforementioned deviations to protocol. Hauer and Oster (Reference Hauer and Oster2008) suggested that levels of reliability may be more a function of additional prompts and continuous cueing than the actual measure itself.
Finally, data were not collected on level of dementia (Mini-Mental Score Examination (MMSE) score or equivalent) and aetiology of disease and, therefore, were not controlled for. While this is a major limitation of this study, it does not undermine the applicability of the findings herein. Future research, undoubtedly, should attempt to collect such information, and examine its effects on the reliability within this challenging population. However, the message remains clear that commonly used measures of functional performance are not well justified and are potentially psychometrically unsound in a sample of lowly mobile, older adults with a confirmed dementia diagnosis. Further, reiteration is necessary, that the low walk speed, demonstrated within this sample, is correlated to low cognitive function due to the complex, higher order executive nature of the initiating and maintaining movement (Sheridan et al., Reference Sheridan, Solomont, Kowall and Hausdorff2003; Ble et al., Reference Ble2005; Atkinson et al., Reference Atkinson2007; Kemoun et al., Reference Kemoun2010).
This study adds further weight of evidence of the difficulties of measuring functional performance in adults with dementia and establishes the inappropriateness of two common measurement batteries of functional performance. This is due to low levels of absolute reliability and high variance at the individual level. These measurement batteries, perhaps, are better employed in group comparisons, but caution must be extended at an individual level. Dementia should not preclude older adults from participation in interventions to improve functional performance. However, we suggest that future research investigate other assessment methods and that the use of IRT might be an applicable and useful alternative.
Conflict of interest
None.
Description of authors’ roles
Benjamin Fox carried out the study, analyzed the data, and wrote and edited the paper. Timothy Henwood and Christine Neville formulated the research question, designed and carried out the study, and proofed the paper. Justin Keogh designed the study, analyzed the data, and proofed the paper.
Acknowledgments
The authors would like to thank Carinity and Churches of Christ for their assistance throughout the project.
