Introduction
Stroke is a complex, heterogeneous syndrome with consequences that extend beyond physical domains to affect the social, occupational and emotional wellbeing of patients and their carers. Post-stroke consequences are unique to individuals and dependent on the interaction between the stroke pathology and associated impairments, mediated by factors such as networks of support, attitudes and motivation. Moreover, what is important to individuals in terms of outcomes is likely to be affected by their pre-stroke circumstances.
Large stroke trials, especially those considering pharmacological therapies, have traditionally focused on ‘hard’ stroke outcomes such as death and dependency. These endpoints have the advantage that they are definable and may be captured through relatively simple means (e.g. the ‘two simple questions’ methodReference Lindley, Waddell and Livingstone1). However, these measures are unlikely to be sufficiently sensitive to patient-centred outcomes that are often the primary focus of rehabilitation after a stroke. Conversely, capturing individualized (qualitative) data is not practicable or financially viable in routine care or in most stroke trials.
Quantitative measurement of post-stroke outcomes that are relevant to patients is likely to involve the use of either interview- or self-administered standardized questionnaires. The content of standardized questionnaires will require more than an assessment of functional ability. Difficulties in capturing multi-dimensional post-stroke outcomes are reflected in the number of potential instruments available, both stroke-specific and generic.
Interview-completed questionnaires are prone to observer bias. Even in carefully designed and blinded trials, assessors may become aware of treatment allocation.Reference Parker and Dewey2 Also, reporting bias may occur due to the presence of the researcher causing patients to respond to some questions differently. Postal collection of self-completed questionnaires is therefore an attractive alternative, but also has limitations. Generic instruments collected by post may be neither relevant nor valid in the specific context of stroke, whilst the reliability of stroke-specific outcome instruments may not have been investigated for postal administration. There are also issues of feasibility. Patients with severe impairments (physical, cognitive or language) may not be able to self-complete questionnaires and the consequent poor response rates or missing data can result in a selection response bias.Reference Parker and Dewey2 Systematic exclusion from studies of groups of patients with greater impairments (or, for example, patients residing in institutional care) can also result in non-representative samples, which may limit generalizability of study findings. Proxy completion may be an alternative, but unless of proven reliability, introduces another significant source of bias.
Restricting the use of instruments to those that are valid in stroke and with optimal psychometric properties in terms of postal reliability, acceptability and reliability for proxy completion would minimize the problems associated with postal data collection. Standardization of postal stroke outcome measurement may increase confidence in the assessment of outcomes by postal survey for future research and in routine care.
This review identifies outcome measures that have been demonstrated as valid and reliable for postal administration and acceptable to patients with stroke and their carers across a spectrum of domains and impairments. It also highlights the areas of absent evidence where further work on the psychometric properties of existing instruments is required.
Methodology
A review of the literature was undertaken to identify generic and stroke-specific outcome measures that have been used in quantitative stroke studies to collect patient outcomes data in any domain by self or proxy administered postal survey following discharge from hospital or rehabilitation facility.
Search strategy
The search strategy incorporated the Cochrane Stroke Group's highly sensitive search strategy for identification of stroke trials3 and included both controlled vocabulary and free-text terms. Search terms combined stroke with outcomes, methodology, survey type, measurement scale, collection and reporting methods. This search strategy was used in MEDLINE and modified for use in other databases (search strategy available on request from authors). Studies were identified through electronic searching of the Cochrane Controlled Trials Register (4th quarter, 2008), MEDLINE (1950 to November 2008), EMBASE (1980 to 2009 week 4), CINAHL (2005 to 2009), psychINFO (1987 to January 2009 week 3), AMED (1985 to November 2008) and British Nursing Index (1994 to November 2008). There were no constraints placed on language.
Citations were imported directly into reference management software (Reference Manager v.12). Studies were examined by one reviewer (ET) according to pre-specified criteria (Figure 1).
Inclusion and exclusion flow chart
Inclusion and exclusion criteria
Randomized and observational stroke studies were included. Studies not written in English and qualitative studies were excluded. Where methods of data collection were not specified, studies were excluded unless this information could be obtained through contacting authors. Studies reporting clinical or radiological diagnosis of stroke were included; those considering exclusively subarachnoid haemorrhage and traumatic brain injury were not.
Postal surveys included those where stroke patients (or their carers) were contacted by mail and required to return a self- or proxy-administered questionnaire by post. Studies where questionnaires were collected by researchers, or where researchers acted as proxy to complete pre-mailed questionnaires on the behalf of patients, were excluded. Studies involving hybrid data-collection techniques (e.g. part interview, part questionnaire) were included only if the questionnaires were collected by post and with the caveat that this is a likely source of response bias.
Studies using a study-specific instrument were excluded unless the instrument was adequately validated. Studies written in English using an outcomes instrument administered in another language were excluded unless the translated version of the instrument was specifically validated. Similarly, instruments developed in languages other than English were excluded, unless a validated English translation exists. Systematic reviews of randomized controlled trials were excluded as outcome measurement was not reported in sufficient detail. Reviews and their reference lists were examined as a quality measure to ensure that commonly used indicators had not been overlooked and that relevant studies had been included within the review.
Psychometric parameters
Data were extracted from the included studies to describe study methods, population sampling and recruitment, methods of data collection and postal outcome instruments. The list of outcomes measures was refined through evaluation of the evidence of validity in stroke populations, postal test-re-test reliability, internal consistency, acceptability, evidence of floor or ceiling effects and proxy reliability. Where it is critical that psychometric parameters are specifically evaluated for postal administration (e.g. postal test-retest reliability and rates of completion and return), data were extracted from identified studies. Information regarding validity in stroke populations, proxy administration and floor and ceiling effects was extracted from (identified) postal studies where available. Based on the assumption that these properties are unlikely to change significantly with the methodology of questionnaire administration and due to paucity of evidence in the included postal studies, data for these parameters were also extracted from the wider stroke literature. This included information from the Evidence Based Review of Stroke Rehabilitation (EBRSR)Reference Salter, Jutai, Zettler, Moses, Foley, Teasell and Teasell4 and the StrokEngine-Assess.Reference Blum-Zeltzer, Korner-Bitensky, Sitcoff, Salter and Teasell5 Both of these Canadian online resources have comprehensively reviewed the literature examining properties of stroke outcome measures.
The psychometric criteria applied to refine the outcomes measures were defined a priori. A correlation coefficient of greater than 0.6 was considered to demonstrate excellent,Reference Fitzpatrick, Davey, Buxton and Jones6 and between 0.31 and 0.59 adequate construct and criterion validities.Reference Salter, Jutai, Zettler, Moses, Foley, Teasell and Teasell4, Reference Blum-Zeltzer, Korner-Bitensky, Sitcoff, Salter and Teasell5, Reference Andresen7 Validated instruments that lacked specific evaluation of validity in stroke populations were retained for further consideration providing there was no conceptual reason why the replies from a stroke population might differ systematically to the population in which validity had been established. Instruments with no evidence for validity were excluded.
Where the intra-class correlation coefficient (ICC) had been used to assess test–retest reliability, values greater than 0.4 were considered adequate and greater than 0.75, excellent.Reference Salter, Jutai, Zettler, Moses, Foley, Teasell and Teasell4, Reference Blum-Zeltzer, Korner-Bitensky, Sitcoff, Salter and Teasell5, Reference Andresen7 An equivalent acceptable kappa statistic was a value greater than 0.2.Reference Altman8 A Crohnbach's alpha of between 0.7 and 0.9 was considered acceptable as a measure of internal consistency.Reference Fitzpatrick, Davey, Buxton and Jones6, Reference Bland and Altman9 Floor or ceiling effects were considered moderate if 10% or large if 20% of respondents scored at either extreme of the scale.Reference Salter, Jutai, Zettler, Moses, Foley, Teasell and Teasell4 Missing data rates were considered high if more than 20% of returned questionnaires contained missing data.
Results
The initial electronic search resulted in 18,947 citations. After removal of duplicates and screening of titles, the abstracts of 4577 papers were read, with 465 of these being accessed in full text for further examination.
Sixty citations (describing 51 study populations) fulfilled inclusion and no exclusion criteria.Reference Parker and Dewey2, Reference Bakas, Austin, Jessup, Williams and Oberst10–Reference Weir, Gunkel, McDowall and Dennis68 Forty-six study populations concerned patient outcomes, and five related to carer outcomesReference Bakas, Austin, Jessup, Williams and Oberst10, Reference Blake and Lincoln12, Reference Blake, Lincoln and Clarke13, Reference Pound, Gompertz and Ebrahim56, Reference Tommis, Seddon, Woods, Robinson, Reeves and Russell61 (not considered further in this paper). Thirty-six postal instruments that measured patient outcomes after stroke were identified (Table 1). One of these was an alternatively named version of another included instrument (and these were subsequently considered together). Twenty of the included studies aimed explicitly to evaluate validity and reliability, or to describe other psychometric properties of the instruments.
Instruments identified in review and their domains
Thirteen of the 35 different identified instruments had been poorly or incompletely validated in stroke populations (or lack evidence to support validity in stroke populations). Four instruments lacked evidence to support reliability (postal test–retest and internal consistency), and a further six had no evidence of postal test–retest reliability. Three measures had proven test–retest reliability but lacked evidence to support internal consistency. These were retained for further consideration as it was felt that absence of evidence of internal consistency alone, although imperfect, should not warrant their exclusion. Excluded instruments (with reasons) are included in Figure 2.
Refinement of outcome instruments and reasons for exclusion
Twelve instruments were therefore identified that demonstrated acceptable validity and reliability in stroke populations. Their performance according to other psychometric parameters (acceptability, proxy reliability and responsiveness) are summarized in Table 2.
Psychometric properties of outcome indicators valid and reliable for postal administration after stroke
*Where response rates are part of a test–retest study, the response rates to the first mailing have been presented. Other figures for response rates represent those after any additional measures to increase response rates (e.g. telephone or postal reminders). Response rates may be affected by the inclusion of additional questionnaires contained within a pack.
Floor and ceiling effects
The Barthel Index demonstrated a tendency to upper limits and some ceiling effects, especially in less dependent patients.Reference Parker and Dewey2, Reference Curry, Sethi, Ogilvy and Carter17, Reference Lincoln, Francis, Lilley, Sharma and Summerfield43, Reference Parker, Gladman, Drummond, Dewey, Lincoln, Barer, Logan and Radford55, Reference Watkins, Auton, Deans, Dickinson, Jack, Lightbody, Sutton, van den Broek and Leathley67 The SF-36 was prone to both ceiling and floor effects.Reference Dorman, Dennis and Sandercock22, Reference O'Mahony, Rodgers, Thomson, Dobson and James54 A further four instruments demonstrated either floor or ceiling effects in at least one subscale (Stroke Impact Scale (SIS) 2.0Reference Duncan, Reker, Horner, Samsa, Hoenig, LaClair and Dudley24), the Nottingham Extended Activities of Daily Living (NEADL),Reference Logan, Ahern, Gladman and Lincoln48 London Handicap Score (LHS)Reference Harwood, Gompertz and Ebrahim34 and the Subjective Index of Physical and Social Outcome (SIPSO)Reference Kersten, George, Low, Ashburn and McLellan39), but these findings were not consistent in all the papers that described their use. The ‘Two Simple Questions’ instrument was designed for use in large randomized trialsReference Lindley, Waddell and Livingstone1 and places patients into three broad categories (dependent; independent but not recovered; independent, fully recovered)Reference Dorman, Dennis and Sandercock69 on the basis of self-completed responses to two questions. The responsiveness of this instrument to small changes in an individual's health status is therefore likely to be limited.
Return rates and missing data
Twenty-five per cent of postal Nottingham Leisure Questionnaires were returned incomplete in one study.Reference Parker and Dewey2, Reference Parker, Gladman, Drummond, Dewey, Lincoln, Barer, Logan and Radford55 The number of missing data items per questionnaire was small (median 2), and most of the missing data were obtained at telephone follow-up. However, this still represents a significant additional resource burden in both research and routine care.
Heterogeneity in study methods (including methods of recruitment and reminders for non-responders) may be reflected in return rates. For this reason, instruments were not excluded from the review on the basis of return rates alone. In one study recruiting patients face to face, the SIS 2.0 (contained in questionnaire packs with four other instruments) was returned with, on average, 91% of items complete (5.63 missing data items per patient); 59% of returned questionnaires had some missing data.Reference Murray, Forster and Young51 Inclusion of the SIS 2.0 within a questionnaire pack, however, did not significantly affect return rates (81% with SIS vs 88% without), but may have adversely affected completion of other instruments (ibid.). In an earlier study, far lower return rates for questionnaire packs containing exclusively the SIS (63%) were reported.Reference Duncan, Reker, Horner, Samsa, Hoenig, LaClair and Dudley24 However, recruitment here was by letter of invitation and the non-responders tended to be more dependent. Low rates of missing data were reported (average 1.3 missing items per patient (range 0–20)) (ibid.) although the rate of proxy completed SIS questionnaires was high (45% of returned questionnaires).Reference Duncan, Reker, Horner, Samsa, Hoenig, LaClair and Dudley24 The same group found proxy reliability of the SIS to be adequate to excellent across items and although there were statistically significant differences in patient-proxy responses in some domains, these were not felt to be clinically significant.Reference Blum-Zeltzer, Salter, Korner-Bitensky and Sitcoff70, Reference Duncan, Lai, Tyler, Perera, Reker and Studenski71
Proxy reliability
The Reintegration into Normal Living Index (RNLI) demonstrated poor proxy reliability in its non-postal version,Reference Sitcoff, Blum-Zeltzer and Korner-Bitensky72, Reference Tooth, McKenna, Smith and O'Rourke73 and evidence to support proxy use in the postal version of the questionnaire was lacking. Evidence to support the reliability of proxy completed NEADL and LHS questionnaires was lacking, although studies that used these instruments reported high rates of proxy completion (22–58%Reference Parker and Dewey2, Reference Gladman, Lincoln and Adams28–Reference Gompertz, Pound and Ebrahim30, Reference Gompertz, Pound, Briffa and Ebrahim32, Reference Logan, Ahern, Gladman and Lincoln48, Reference Parker, Gladman, Drummond, Dewey, Lincoln, Barer, Logan and Radford55 and 36–71%,Reference Parker and Dewey2, Reference Harwood, Gompertz and Ebrahim34, Reference Parker, Gladman, Drummond, Dewey, Lincoln, Barer, Logan and Radford55 respectively). This may reflect the broad inclusion criteria of these studies, rather than specific properties of the instruments.
Optimal outcome measurement instruments
Three instruments demonstrate acceptable properties in terms of validity, reliability, acceptability, responsiveness (absence of consistent floor and ceiling effects) and proxy reliability. These are the Frenchay Activities Index (FAI), the SIPSO, and the EuroQoL (EQ5D). Two instruments lack evidence to support proxy reliability but have otherwise acceptable properties (the NEADL and LHS). One further instrument (the London (Pound) Stroke Satisfaction Questionnaire) has been shown to be valid and reliable for postal collection following stroke.Reference Pound, Gompertz and Ebrahim56, Reference Pound, Gompertz and Ebrahim57 This questionnaire is designed specifically to measure satisfaction with care rather than functional outcome.
Data extraction tables and tables of supporting evidence are not reproduced here, but are available on request.
Discussion
Postal questionnaire instruments are a common method of collecting stroke outcome data in research. Ideally, such instruments should not only be proven to be valid and reliable for postal administration in stroke populations, but should also demonstrate good psychometric properties in terms of acceptability, responsiveness and proxy reliability. These are demanding criteria and, pragmatically, a combination of instruments with caveats to accommodate imperfect psychometric properties will be required in order to capture relevant outcomes across a spectrum of domains.
This review identified 36 measurement instruments that have been used to collect postal outcomes in patients recovering from stroke, but only 12 were supported by evidence of validity and postal reliability. These 12 instruments encompass a wide range of outcome domains (Table 3), although the depth of questions varies. The optimal instrument or combination of instruments to measure outcomes for a specific study is dependent on the characteristics of the study population, study design, desired endpoints and available resources. Consideration of the relevance of the instruments identified in this review when designing studies should therefore be made at the point of application. Several instruments identified in this review have been used only in research studies conducted by the originators of the instruments. A lack of evidence to support the independent use of these instruments (e.g. the NLQ) raises issues of generalizability. The quality of studies included in this review, and heterogeneity in study methods, further limits interpretation of the supporting evidence and its applicability to a general stroke population.
Domains captured by stroke outcomes measurement instruments valid and reliable for postal administration
*Modified Two Simple Questions.
In an unselected post-stroke population, it is important for outcomes instruments to be sensitive to change across the entire spectrum of ability. Instruments with persistent ceiling or floor effects (e.g. the Barthel Index, SF-36) may therefore be of limited use despite their otherwise good psychometric properties. However, in studies where patients with extremes of impairment are excluded, ceiling or floor effects may be less problematic.
Proxy completion of questionnaires can increase the range of patients that may be reached through the postal route. High rates of proxy completed questionnaires may reflect a particular study population (e.g. those with cognitive impairment), or may represent poor acceptability of the questionnaire in stroke survivors. Providing that proxy responses have been demonstrated as reliable, this is an acceptable approach. However, patient and proxy assessments may differ systematically with either patient overestimationReference Salter, Jutai, Zettler, Moses, Foley, Teasell and Teasell4, Reference Tooth, McKenna, Smith and O'Rourke74 or proxy underestimation.Reference Dorman, Waddell, Slattery, Dennis and Sandercock75 Proxy assessments may be particularly inaccurate in subjective domains (ibid.). A failure to consider proxy reliability may result in a large volume of completed, but inaccurate questionnaires. Inclusive studies where high rates of proxy completed questionnaires are anticipated should aim to use instruments of proven reliability in proxy measurement of outcomes (e.g. the FAI or SIPSO).
Missing data is a major problem with questionnaires administered by post. For example, Murray et al. (2007) found the visual analogue scale of the EuroQoL questionnaire to be particularly poorly completed.Reference Murray, Forster and Young51 Visual analogue scales have been shown to be particularly unreliable in patients following a stroke.Reference Price, Curless and Rodgers76 More general issues contributing to missing data include poor acceptability (e.g. a question that patients do not wish to answer), ambiguity, or problems to do with layout or text. Increasing the number of instruments within a pack has a variable effect on missing data, and may reduce response rates.Reference Edwards, Roberts, Clarke, DiGuiseppi, Pratap, Wentz, Kwan and Cooper77 Studies that measure specific aspects of outcome in depth (using instruments such as the Nottingham Leisure Questionnaire) may therefore need to compromise on breadth of information collected such that the number of questionnaires within a pack is not too great. Global instruments that attempt to measure a breadth of domains in detail (e.g. the SIS 2.0) may result in higher rates of missing data.Reference Murray, Forster and Young51 Inclusion of one poorly completed instrument may affect the completion of others (ibid.). Shortening the length of questionnaires to improve response rates should be balanced against the potential risk of reduced precision and the random error that this may introduce.Reference Edwards, Roberts, Clarke, DiGuiseppi, Pratap, Wentz, Kwan and Cooper77
The amount of additional effort required to maximize responses may be considerableReference Parker and Dewey2 and impractical for data collection in routine care. Imputation techniques may be possible for small amounts of missing data. In smaller, well-resourced studies, obtaining missing data to complete questionnaires through telephone or face-to-face interviews may allow the use of postal instruments otherwise precluded through high rates of missing data and lower response rates. However, there may be a ceiling to the benefit of telephone and postal reminders.Reference Murray, Forster and Young51
Respondents to postal surveys are to some degree self-selecting, whether this be through motivation or ability to respond. When interpreting data collected by post, therefore, it is important to consider the characteristics of non-responders to ensure that there is no systematic omission of subgroups of patients with shared characteristics.Reference Parker and Dewey2 This is only likely to be possible where baseline assessments have been performed, or where patients are recruited from a pre-existing register.
Valid, reliable instruments to measure subjective problems post-stroke (such as pain, fatigue and sleep disturbance) are noticeably absent in this review. Questions pertaining to these problems are included in some generic questionnaires (e.g. the SF-36), but associations with depression, physical symptoms, sleep apnoea,Reference Barker-Collo, Feigin and Dudley78 and language or cognitive problems, may limit the usefulness of generic instruments in stroke. In addition, no postal-administered instruments that measure problems more reliant on objective assessment (e.g. cognition or language) were identified, although ‘screening’ questions for these problems were included within some instruments (e.g. the Stroke Impact Scale).
This review has considered a large amount of literature. The selection and analysis of studies for inclusion was made by one reviewer. Despite explicit and replicable methods for this process, this remains a potential source of selection bias.
Studies where postal collection of outcomes was not explicitly stated in the methods section were excluded from the review (except in two studies where authors were contacted directly). It is therefore possible that some studies where outcomes were collected by post may have been excluded. The assumption that some psychometric parameters (e.g. floor effects) are unlikely to be affected by the method of questionnaire administration may not be valid, and these instruments should be considered with this caveat.
Conclusion
Quality of results reported in stroke studies is dependent on the psychometric properties of the instruments used to measure outcomes. This review has identified that several stroke studies have measured outcomes by postal survey using instruments inadequately validated or lacking evidence of reliability for this method of delivery. Robustness of outcomes measurement instruments should be considered when designing stroke trials. Moreover, the absence of evidence to support additional psychometric properties of identified instruments does not necessarily equate to poor performance in this regard. Further research should focus on efforts to test, refine and optimize existing instruments to help standardize outcomes management in both rehabilitation research and routine care. The development of any novel stroke-specific postal outcomes measures should consider potential psychometric limitations a priori.
Acknowledgements
We are pleased to acknowledge funding from the National Institute for Health Research Collaborations for Leadership in Applied Health Research and Care (CLAHRC). The views and opinions expressed in this paper are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health. The paper represents the full report of an abstract presented in poster format at the British Geriatrics Society Spring Meeting, April 2010 (Edinburgh).
Competing interests
ET's post is funded by CLAHRC. This work was undertaken as part of the CIMSS project, the stroke theme of the Leeds, York and Bradford CLAHRC. The CIMSS project aims to develop and refine a core minimum dataset for stroke (including routine collection of postal outcomes) and an electronic system in which to capture it. We plan for this system (CIMSSPlus) to relate to existing electronic data collection systems. The authors received no financial support from commercial companies for this article.
