Introduction
Validated risk scores, also called prognostic scores or prediction rules, are widely used to guide decision-making in acute care and hospital settings. Most risk scores are used without any evidence of cost savings, effect on decision-making, or improvement in patient outcomes (Reilly and Evans, Reference Reilly and Evans2006; Moons et al., Reference Moons, Altman, Vergouwe and Royston2009). For example, only 11% of risk scores for traumatic brain injury have been validated, and none was studied for clinical impact (Perel et al., Reference Perel, Edwards, Wentz and Roberts2006). Ideally, such evidence would guide implementation of scores in clinical practice. Clinical impact studies assess risk score effectiveness in practice. They are particularly relevant because of the controversy surrounding risk scores (Graham et al., Reference Graham2001). Scores reduce complexities to a single percentage, while even guidance by guidelines, where abstraction of reality is more modest, is not always accepted (Formoso et al., Reference Formoso, Liberati and Magrini2001).
Risk scores are available for use in cancer populations, palliative care populations (such as the Palliative Performance Scale), and for dementia or nursing home populations (Mitchell et al., Reference Mitchell, Kiely, Hamel, Park, Morris and Fries2004; van der Steen et al., Reference van der Steen, Mitchell, Frijters, Kruse and Ribbe2007a; Lau et al., Reference Lau2007; Maltoni et al., Reference Maltoni2005; Glare and Sinclair, Reference Glare and Sinclair2008). The risk scores are mostly limited to providing probabilities without recommendation (“assistive” rather than “decisive” model; Moons et al., Reference Moons, Altman, Vergouwe and Royston2009). However, despite complex end-of-life decision-making in palliative care, reports on these risk scores still focus on the accuracy of the prediction, assuming or ignoring their clinical impact on decision-making. Currently, experts still prioritize validity over clinical impact (Stevinson et al., Reference Stevinson, Preston and Todd2010). Others have reported on the variable success of linking risk assessment to advance care planning and palliative consultation (Levy et al., Reference Levy, Morris and Kramer2008; Oliver et al., Reference Oliver, Bickel-Swenson, Zweig, Kruse and Mehr2009).
Up to two-thirds of dementia patients ultimately die from pneumonia (van der Steen et al., Reference van der Steen2006). The decision to cure with antibiotics versus providing comfort care without antibiotics is commonly faced in these patients. To support this decision-making process and better inform families, we have developed and validated an assistive risk score to estimate two-week mortality in patients with dementia and pneumonia (van der Steen et al., Reference van der Steen2006). To test whether use of the score meets these goals in practice and to assist with implementation strategies, we performed a series of quantitative and qualitative studies that explore the score's (barriers to) clinical impact in a dementia long-term care setting.
Methods
Development and validation of the mortality risk score is described in Appendix 1A. Each possible score between 0 and 31 was linked to estimated mortality (1.8% to >50%) and the original score did not include documentation on its background (van der Steen et al., Reference van der Steen2006).
Dementia long-term care setting
Our study setting is in the Netherlands, where nursing homes employ physicians who are certified as nursing home physicians after three years of vocational training. Special wards for psychogeriatric patients consist predominantly of dementia patients (96% had dementia in a previous study on pneumonia; van der Steen et al., Reference van der Steen, Ooms, Adèr, Ribbe and van der Wal2002) who usually stay until death. Residential care settings also have special units for dementia patients, cared for by nursing home or family physicians (Conroy et al., Reference Conroy, van der Cammen, Schols, van Balen, Peteroff and Luxton2009). Physicians and nurses discuss treatment and care, including palliative care, in regular multidisciplinary team meetings. Since physicians are responsible for decision-making, they are the primary focus for this study (Helton et al., Reference Helton, van der Steen, Daaleman, Gamble and Ribbe2006; Conroy et al., Reference Conroy, van der Cammen, Schols, van Balen, Peteroff and Luxton2009).
We used a mixed-method approach, employing a series of smaller quantitative and qualitative studies because the first quantitative physicians study did not suffice.
Study 1. Trial with physicians using and not using the score
Randomized controlled trials provide the highest level of evidence for clinical impact studies (Reilly and Evans, Reference Reilly and Evans2006), and we therefore conducted a trial. For evaluation of instrument development, we reviewed the literature on guideline implementation (Cabana et al., Reference Cabana1999; Halm et al., Reference Halm2000; Grimshaw et al., Reference Grimshaw2001; Reference Grimshaw2006; Naughton et al., Reference Naughton, Mylotte, Ramadan, Karuza and Priore2001), because of similarities between guideline and assistive risk score implementation.
Here we summarize results which in part have been described elsewhere (van der Steen and Ribbe, Reference van der Steen and Ribbe2008). Briefly, data were collected between July 2006 and March 2008. Almost all nursing homes (96%) agreed to participate. The main outcomes were the use and usefulness of the score, and physicians’ self-reported decision-making and information-providing behaviors. Nursing home physicians randomized to homes in the intervention group (n = 107) were asked to use the score for their next case of pneumonia if pneumonia developed in any of their dementia patients, and to complete a survey. The score was introduced without background other than that it was developed in a Dutch population and that it validated well in a Missouri (U.S.A.) population. The control group (n = 106) completed a survey only. Selected thesis titles were offered as incentives. This controlled design allowed for identifying possible bias if the score was used selectively.
Study 2. Physicians’ focus group discussions
Subsequently, two focus group discussion sessions were conducted in the fall of 2008, one with six nursing home physicians and a second with seven family physicians. We selected participants with variable experience and clinical responsibilities who had neither a hierarchical work nor collegial relationship. Participants received the risk score in advance and information was limited to the sessions’ and score's main goals. Results from Study 1 were unpublished at that time. We developed an interview guide with open-ended questions such as “do you believe the score will improve the quality of care?” We also developed three hypothetical clinical case scenarios and asked physicians about the score's usefulness in these cases. The experienced moderator (MTM) had been involved in the pneumonia study (van der Steen et al., Reference van der Steen, Muller, Ooms, van der Wal and Ribbe2000) from which the score was developed. One or two observers (researchers) took notes in addition to all discussions being audio taped. The discussions of the first focus group were transcribed verbatim. Independent analyses were performed by two researchers, one of whom attended both sessions (GA) and another who was familiar with nursing home practice (JTS), or family practice (EL-S). A framework conceptualized in the literature on guideline implementation (Cabana et al., Reference Cabana1999; Halm et al., Reference Halm2000; Francke et al., Reference Francke, Smit, de Veer and Mistiaen2008) was used to classify barriers related to patient, clinical setting, the score's properties, and physician. We added perceived family-related barriers. The two independent analyses of the focus group discussions were combined after discussing classifications and reaching consensus. We present the positive and negative aspects (e.g. (non)usefulness when knowing the patient) emerging from the focus groups, and counts of these aspects, as an indication of the discussions’ direction and breadth.
Study 3. Nurse and family surveys
In parallel with Study 2, we performed a survey study among nurses caring for dementia patients, and family members of dementia patients residing in nursing homes who were already participating in an ongoing prospective study (van der Steen, Reference van der Steen2010). Family surveys (53) were sent to participants scheduled for the study's bi-annual questionnaire between October 30 and December 16, 2008. A convenience sample of nurses – excluding nurse aids – who practice in dementia care in different parts of the country were recruited by the nursing home physicians participating in the focus groups. Eight survey forms per home, totaling 48, were distributed. Surveys included questions on families’ and nurses’ perspectives on being informed of prognosis. Nurses were also asked about their role in decision-making and informing families.
Revising the risk score and preparing for implementation
Based on the study results, we revised the risk score and introduced it to eight experts for implementation. These key persons had expertise in communication strategies, implementation of research findings in clinical practice, training of nursing home physicians, and nursing research in palliative care, or represented professional organizations of nursing home physicians and care providers or national patient advocacy groups.
Results
Study 1. Trial with physicians using and not using the score
We received 38 surveys (36%) from the control group, and 32 (30%) from the intervention group. A main reason for not returning the survey, as stated by the local coordinators, was related to the prospective design: many physicians did not diagnose incident pneumonia within the data collection period. Another main reason was forgetting to include a patient case. More important than rates per se is that the response was unselective and unrelated to score use because physician and patient characteristics (e.g. gender and age), treatment (e.g. antibiotics and opiates), and mortality for enrolled cases did not differ significantly between groups (P > 0.01 to account for multiple comparisons). The intervention group additionally reported practical barriers such as not having the score at hand so in only in 21 of 31 cases (one survey was lost in the mail) did physicians actually use the score; others completed the survey retrospectively (van der Steen and Ribbe, Reference van der Steen and Ribbe2008).
Most (64%; 44/69) physicians thought that a risk score would be a useful addition to their own estimate of the patient's risk of death and 71% who did use it found it helpful. However, usefulness related to learning how to prognosticate better in future cases (80%) and to better documentation of prognosis (53%), but less often to better informing families (33%), reducing uncertainty in prognostication (20%), or support in decision-making (13%). Physicians informed the majority of families (90%) and nurses (86%) on prognosis in some way, but no physician used a numerical approach with families, such as a percentage likelihood of death, and even informed only 14% of nurses in this manner, citing concerns that nurses might inform families about the prognosis without consulting the physicians. All physicians agreed that the format was acceptable. None of the nursing homes was previously using any risk score at the time of the study.
Study 2. Physicians' focus group discussions
Focus groups were conducted to provide more in-depth information on barriers that became apparent in the trial. Participating physicians were between 30 and 59 years old, and their clinical experience varied between 0 and 32 years. About half (50% of nursing home, and 42% of family physicians) were female. Most (67% and 86%, respectively) were involved in non-clinical activities such as training (under)graduates and research. One nursing home physician had previously used the score. The focus group sessions concluded after 85 minutes; when saturation was reached as no new issues appeared.
The main barriers to use of the score related to the score's properties and physicians’ attitudes, rather than to perceived patient, family or setting characteristics (Table 1). The score cannot identify patients at very high mortality risk nor provide estimates for antibiotic-untreated patients, and physicians believed they needed this for the score to support decision-making. With limited information provided, there was confusion on how the score had been developed, intended, and should be used. The other main barrier referred to physicians’ attitudes, including perceptions of usefulness and validity. The physicians were interested in the factors and the weighing of factors in risk estimation. However, they were reluctant to translate risk into numbers or percentages, and to use it in decision-making or informing families or nurses: “I would not confront them [families] with measures or numbers.” It was felt that the abstraction of the complexities of decision-making was inappropriate (“such a decision is not limited to statistics.”) This was also evident from the variety of factors other than prognosis put forward as important or more important in decision-making, such as quality of life, suffering, and family wishes. Perceptions of usefulness and trust in their own or the score's prognosis were heavily discussed, and ambiguity was evident (33 of 50 positive aspects and 36 of 65 negative aspects related to the physician level; Table 1).
Clinical impact of the risk score: aspects raised by physicians during two focus group discussions
Notes: Numbers refer to number of different aspects raised within each session. Occasionally, one aspect was raised multiple times during one session; this was not counted.
1Aspects indicated with “+/−“ held both positive and negative aspects.
Nursing home physicians in particular appreciated the score's usefulness in case of conflicts with families on best treatment – the only family physician who raised this benefit had been trained as a nursing home physician initially. From the discussions, family physicians seemed inclined to follow family wishes, and were more critical regarding usefulness in the particular work setting and the score's properties. However, similar issues were raised by both physician groups – the framework for analyses developed for nursing home physicians fitted well with the family physicians.
Study 3. Nurse and family surveys
Sixty percent of nurses (29/48) completed the survey. Most (93%) were female. Their average age was 42 years (SD 10) and they had 18 years (SD 9) of experience. Most (86%) were certified after a two-year training program; some (14%) had followed a four-year program.
All but one nurse stated they should be involved in treatment decision-making and all were at least sometimes involved (Table 2). Most nurses (76%) “often” received questions from families about prognosis in case of pneumonia (Table 2). There was considerable variability in how nurses responded to families’ enquiries, in role perceptions of informing families about prognosis, and some variability in perceived capability of doing so.
Nurses' views from surveying decision-making and prognosis in patients with dementia and pneumonia
Fifty of 53 families (94%) returned the survey. Most (68%) of the respondents were female. More than half (58%) were children, 22% spouses, 11% siblings, and 9% were otherwise related to the patient. Their average age was 62 years (SD 10) and 30% had at least a bachelor's level of education; 5% had no more than elementary school. The majority (88%) wished to be informed of prognosis if their relative in the nursing home should have a pneumonia; 2% could not tell; and 10% did not want to know, mostly because they felt comfortable if life was not prolonged (not in table). In families who wished to know the prognosis, many favored a numerical approach (43% preferred a percentage or frequency), but almost as many others wished to be told in words (35%). Only 19% of families thought that knowing the prognosis would not impact on decision-making (Table 3).
Families’ views from surveying decision making and prognosis in patients with dementia and pneumonia
1n = 42 who wished to be informed of prognosis; deselecting 6 who did not want to know or could not tell and 2 with missing responses).
Revising the risk score and preparing for implementation
Based on qualitative and quantitative work from Studies 1–3, we revised the risk score's format and drafted a score in card format (Appendix 1B and 1C). We omitted overall mortality outcomes of pneumonia because it was not well understood that treated and untreated patients were different, as well as advice on communicating prognosis because it was not accepted by the focus group participants.
We also developed a form that could be completed by nurses, omitting the weights because some physicians were uncomfortable with nurses’ possible interpretations. We further drafted a separate background section consisting of four pages of A5-format explanation. The explanation was structured according to the six questions and answers that were most salient in the focus group discussions. By answering those, we attempted to address the observed barriers (Appendix 1D). We included preliminary information of a recent study showing effectiveness of antibiotics (van der Steen et al., Reference van der Steen, Lane, Kowall, Knol and Volicer2010); such information was lacking according to the focus group participants. We also included a general checklist of broader issues considered important for decision-making (van der Steen et al., Reference van der Steen, Muller, Ooms, van der Wal and Ribbe2000) as an ethical framework to acknowledge the complexity of decision-making.
In consultation with experts, we moved the ethical framework, more prominently, to the risk score itself (Appendix 1C), grouped mortality rates to avoid suggesting over-precision (1B), and prioritized the order of the six questions (1D). The experts confirmed the results of the trial by preferring to use the score in educational programs rather than in practice, and recommended personalized implementation strategies.
Finally, we printed the scores in laminated pocket format and made them available to all four national training programs for nursing home physicians along with the developed explanation, and sent or brought these as personal incentives to physicians participating in research.
Discussion
Reports on risk scores for dementia patients at the end-of-life have not addressed the acceptability of such scores to its users nor its clinical impact. Three combined quantitative and qualitative studies in the Netherlands elicited substantial barriers to actual and potential clinical impact of our risk score to estimate mortality in antibiotic-treated patients with dementia and pneumonia. In addition to the practical barriers obvious from the trial, the focus groups revealed a reluctance to a numerical approach in end-of-life decision-making, and shortcomings related to the technical properties of the score. Roles of nurses in informing families of prognosis were unclear and perceptions of tasks and how to collaborate with physicians varied. Many – but not all – families wanted detailed, numerical information on prognosis, which physicians using the risk score never provided, showing substantial numbers of under-served families in this palliative care setting. Nevertheless, physicians accepted the risk score as an educational tool, and we were partly able to address barriers by relatively simple revisions of the risk score's format while maintaining its technical properties.
Palliative and long-term care settings
A Swiss study of family physicians reported similar barriers in using cardiovascular prediction rules, including physicians’ doubts concerning oversimplification of risk assessment, and limited benefit of numerical information resulting from prediction rules for decision-making in practice (Eichler et al., Reference Eichler, Zoller, Tschudi and Steurer2007). We also observed the perception that aggregated data can never be applied to the individual due to uniqueness and complexity as observed in primary care (Hadorn, Reference Hadorn1997; Eichler et al., Reference Eichler, Zoller, Tschudi and Steurer2007); indeed a simple cutpoint on which to base end-of-life decisions is too simplistic (Lemeshow et al., Reference Lemeshow, Klar and Teres1995). Furthermore, in end-of-life settings, a one-dimensional numerical approach may not fit intuitively with person-centered, holistic, palliative care and complex, sensitive decision-making. In palliative care settings, other outcomes may be preferred over mortality. However, more accurate prediction than by clinical judgment alone may be difficult to achieve for outcomes such as falls (Meyer et al., Reference Meyer, Köpke, Haastert and Mühlhauser2009), and possibly even more so for subjective outcomes such as quality of life or quality of dying, or comfort. In mortality risk estimation, combined risk scores and clinical judgment usually provide the best estimate of mortality risk (Viganò et al., Reference Viganò, Dorgan, Buckingham and Suarez-Almazor2000; Chow et al., Reference Chow, Harth, Hruby, Finkelstein, Wu and Danjoux2001; van der Steen et al., Reference van der Steen, Ooms, van der Wal and Ribbe2005; Maltoni et al., Reference Maltoni2005; Glare and Sinclair, Reference Glare and Sinclair2008). Of note, in specific cases, appropriateness of disclosure of the most accurate prognosis may be questionable, e.g. when families are not yet prepared emotionally, or when they state they prefer an optimistic attitude.
Our adding of the ethical framework for decision-making acknowledges its complexity, de-emphasizes a numerical approach, and conceptualizes decision-making in all its aspects, complementing decision-making support. This may be specifically needed in the setting and for “preference sensitive” decisions that include valuing benefits and risks (O'Connor et al., Reference O'Connor, Légaré and Stacey2003). Further, mortality risk scores were new to the physicians involved in the study. This may have invoked feelings of threatened autonomy in weighing of prognosis in decision-making even for an assistive score without recommendations such as to suggest palliative care with 50% or larger risk of 1-year mortality (Levy et al., Reference Levy, Morris and Kramer2008). This may explain why physicians tended to reduce the importance of prognosis for decision-making during the focus group sessions, while empirical data have shown that prognosis is very important, if not decisive, in Dutch physicians’ decision-making (van der Steen et al., Reference van der Steen, Helton and Ribbe2009).
However, even if well evaluated and appreciated, risk scores may not be used often in practice. For decision aids, including a booklet on feeding tube use in advanced dementia, it was noted that there was a wide gap between intention and behavior (Graham et al., Reference Graham2007). Risk score use may be a more sensitive issue than providing booklets, as other studies found that physicians are often unwilling to provide a frank estimate of prognosis even if cancer patients requested survival estimates (Lamont and Christakis, Reference Lamont and Christakis2001; Glare and Sinclair, Reference Glare and Sinclair2008); a fear of families’ adverse judgment for “prognostic errors” (Christakis and Iwashyna, Reference Christakis and Iwashyna1998; Glare and Sinclair, Reference Glare and Sinclair2008) may play a role. For a well-validated, well-accepted, and well-implemented risk score, such as the Ottawa Ankle Rules, to have clinical impact, local, active implementation strategies addressing specific local barriers were essential (Cameron and Naylor, Reference Cameron and Naylor1999).
Design considerations
Inexperience with the use of prognostic tools in the dementia long-term care settings in this study may have had a negative impact on the usefulness of trial evaluation results, because physicians lacked a reference against which to evaluate the score's format. The risk score format was found acceptable by all physicians – a fact that is surprising in view of the necessary revisions of the score that appeared from the focus group discussions, resulting in helpful discussion and questioning of the original format. This, and the insights developed from integrating different perceptions of other key parties in decision-making, emphasizes the desirability of a mixed-method design and the inadequacy of a single trial as initially planned and recommended so far (Reilly and Evans, Reference Reilly and Evans2006) to assess clinical impact, at least when impact is limited and a deeper understanding of barriers is needed, or when risk score use is new to participants.
The quantitative studies are limited by small sample sizes. Although adequate for description, it was underpowered to study correlates such as families’ preferences for a numerical approach (possibly in the female, younger and higher educated respondents). The trial design was not efficient with few evaluations of use in clinical practice, but this mimics prospective use in practice – it is easily forgotten and there may be prolonged periods with no cases of pneumonia. Trial outcomes were limited to self-report and physicians’ report about patients and did not include family or nurse outcomes.
Nevertheless, the mixed-method design guided necessary revisions and provided future directions for practice and research. We have integrated studies at the design and interpretation level and to some extent, at the level of analysis (Bryman, Reference Bryman2006). Ethnographic study of decision-making using the risk score, i.e. observing the roles of key parties, may have been a suitable, but time-consuming alternative. Our approach took a total of six months full-time equivalent of staff support, which is marginal compared to investments to develop and validate the score. The study was facilitated by access to the field due to a tradition of research in pneumonia and dementia. Research groups developing and validating prognostic instruments may wish to plan early for a (small) clinical impact study, ideally including it into a grant proposal.
Generalizability of the results to other countries
The results may be relevant to similar settings in other countries. In the U.S.A., family wishes are more important and withholding antibiotics occurs in patients with low mortality risk, in contrast to the Netherlands where antibiotics are withheld in high-risk patients (van der Steen et al., Reference van der Steen, Ooms, Adèr, Ribbe and van der Wal2002; Reference van der Steen2004; Reference van der Steen, Helton and Ribbe2009; Mehr et al., Reference Mehr, van der Steen, Kruse, Ooms, Rantz and Ribbe2003; Chen et al., Reference Chen2006; Helton et al., Reference Helton, van der Steen, Daaleman, Gamble and Ribbe2006). The mortality risk score may assist decision-making in the U.S.A., supporting the hospitalization decision and deciding on oral or parenteral antibiotics (parenteral antibiotics may reduce mortality risk in high-risk patients only; van der Steen et al., Reference van der Steen, Mehr, Kruse, Ribbe and van der Wal2007b). In our study, many families believed that prognosis at diagnosis would impact decision-making. This shows that re-evaluation of care plans when pneumonia develops is needed, at least in the Netherlands. This may be different in other countries or regions; in a U.S. local study in Massachusetts, prognosis in patients treated with and without antibiotics was similar, probably because of active advance care planning (van der Steen et al., Reference van der Steen, Lane, Kowall, Knol and Volicer2010). Further, cross-national differences to receptiveness of risk scores in general are relevant in this respect, with U.S. physicians being more critical than physicians in Canada, the U.K., France and Spain (Graham et al., Reference Graham2001). A more universal phenomenon of “distance” or “presence” may play a role. In our study, family physicians, who only occasionally visited nursing home residents as in the dominant model in the U.S.A., were more critical of the risk scores’ usefulness than specialized, on-site nursing home physicians. The latter seemed more willing to engage in conflicts with families over best interest issues rather than deferring to family wishes, which is more common in the U.S.A. (van der Steen et al., Reference van der Steen2004; Reference van der Steen2010; Helton et al., Reference Helton, van der Steen, Daaleman, Gamble and Ribbe2006), and to favor the score's decision-making support for this.
Future directions for implementation of the risk score in end-of-life care settings
Inclusion of the risk score in training curricula for physicians may help accommodate formal risk estimation in future generations of physicians. Providing laminated pocket cards may be helpful for inexperienced physicians entering practice. In previous work, inexperienced physicians were also more likely to adhere to pneumonia guidelines (Halm et al., Reference Halm2000) and to share their prediction with cancer patients (Lamont and Christakis, Reference Lamont and Christakis2001). Familiarity acquired during training was the best independent predictor of use of the Canadian Cervical Spine rule, in addition to perceived user friendliness (Brehaut et al., Reference Brehaut, Stiell and Graham2006). Alternatively, individual feedback on one's own estimate with retrospective risk score assessment may not only be useful in training programs, but also a legitimate way to improve prediction capabilities.
Implementation strategies are informed by a large body of literature on implementing continued medical education (Davis et al., Reference Davis, Thomson, Oxman and Haynes1995), decision support tools (Kawamoto et al., Reference Kawamoto, Houlihan, Balas and Lobach2005) and guidelines (Cabana et al., Reference Cabana1999; Halm et al., Reference Halm2000; Grimshaw et al., Reference Grimshaw2001; Reference Grimshaw2006; Naughton et al., Reference Naughton, Mylotte, Ramadan, Karuza and Priore2001; Hutt et al., Reference Hutt2006), and meta-reviews being available (Francke et al., Reference Francke, Smit, de Veer and Mistiaen2008). This may be condensed in a few key points relevant also to (assistive) risk score implementation: access should be made easier, e.g. by linking scores to electronic patient records (Kawamoto et al., Reference Kawamoto, Houlihan, Balas and Lobach2005); multi-component implementation strategies tend to work better, although single-component strategies may work too; and barriers should be studied locally because they may fail to generalize (Cabana et al., Reference Cabana1999; Cameron and Naylor, Reference Cameron and Naylor1999; Grimshaw et al., Reference Grimshaw2006).
Local strategies in our case may include, for example, clarifying nurses’ roles in communicating a prognosis, and involve nurse practitioners practicing in selected nursing homes; it could also involve clarifying the roles of social workers (Oliver et al., Reference Oliver, Bickel-Swenson, Zweig, Kruse and Mehr2009). We found that nurses’ and physicians’ roles in informing families were unclear, and there may even have been an element of distrust. Because multidisciplinary teamwork is especially important in palliative and long-term care settings, independent of risk score use, this finding may deserve further study.
Physicians were reluctant to provide numerical risk estimates, but many families preferred this. One study showed that on approaching the end of life, patients may wish to have less information, but families may wish to have more (Parker et al., 2009). The literature shows that numerical estimates are often misinterpreted due to low numeracy levels and yet patients often prefer quantitative over qualitative explanations of risk (Gordon-Lubitz, Reference Gordon-Lubitz2003; Cuite et al., Reference Cuite, Weinstein, Emmons and Colditz2008). Physicians may find that patients expect too much certainty (Christakis and Iwashyna, Reference Christakis and Iwashyna1998), but they should be capable of proper risk communication (O'Connor et al., Reference O'Connor, Légaré and Stacey2003; Gigerenzer and Edwards, Reference Gigerenzer and Edwards2003; Paling, Reference Paling2003). Of course, individual tailoring and explaining of the merits and limitations of prognostication are important.
Future directions for risk score research
Large, new investments will be required to further improve potential clinical impact of the score under study. This involves optimizing technical properties to better support decision-making, including development and validation of a new system in which risk factors are calibrated to treatment with and without antibiotics. The problem of estimates being skewed towards lower percentages, as found in other work (Mitchell et al., Reference Mitchell, Kiely, Hamel, Park, Morris and Fries2004), may be addressed for patients treated without antibiotics as preliminary analyses have shown, or by examining outcomes soon after diagnosis (e.g. one-week mortality). Following this, a new cluster randomized trial in which physicians used this system would deliver convincing evidence on the impact of decision-making which is not limited to physicians’ self-report. Sample sizes and power should allow for analyses by subgroups of experienced and less experienced physicians. Physicians’ own risk estimates would be compared with the risk score for accuracy of the prediction. Finally, a cross-cultural study of clinical impact would provide clearer evidence on generalizability of findings to other settings and the role of physicians’ presence in use of the risk score.
Conclusion
The combined quantitative and qualitative approach used in this study elicited substantial barriers to a numerical approach in end-of-life decision-making, produced crucial information for necessary revisions and further score development, and highlighted the need for implementation strategies that focus on education. The study results led to an improved risk score, while technical properties were maintained. Clinical impact studies are highly recommended for understanding specific barriers related to risk score use in dementia patients, and in palliative and long-term care settings. Such studies will also provide the evidence we need for an open debate on whether risk score use is helpful in practice. They may also help improve our current limited understanding of how best to integrate prognosis in end-of-life decision-making.
Conflict of interest
None.
Description of authors' roles
J. van der Steen formulated the research questions, contributed to study design, to data collection and data analyses, and wrote the paper. G. Albers, E. Licht-Strunk and M. Muller contributed to collection of data. G. Albers, E. Licht-Strunk, M. Muller and M. Ribbe contributed to study design and to data analyses. All authors reviewed drafts of the paper and contributed to interpretation of the data.
Acknowledgments
We thank Professor David R. Mehr, MD, MS, Professor Margaret R. Helton, MD, and Martijn Heymans, PhD for their suggestions to earlier versions of the paper.
The study was supported by grants from the Stichting Wetenschaps Bevordering Verpleeghuiszorg (Society Promoting Scientific Research in Nursing Home Care), Utrecht, the Netherlands, and by the National Platform Palliative Care from the Ministry of Health, Welfare, and Sport (VWS), the Hague, the Netherlands. J. van der Steen was additionally supported by a career award from the Netherlands Organisation for Scientific Research (NWO, the Hague; Veni 916.66.073).
Appendices
1A. Background and elements of the original, first, evaluated version of the risk score
Appendix 1B. Front side of the revised risk score
Risk score
To estimate risk of death within 14 days in patients with lower respiratory infection or pneumonia, and dementia when treated with antibiotics
© 2008 EMGO Institute for Health and Care Research
VU University Medical Center Amsterdam, the Netherlands
Appendix 1C. Reverse side of the revised risk score
Appendix 1D. Questions addressed in the explanatory note to the revised risk score
1. Why would I use the risk score?
2. To use the score as an aid to decision making, I would like to learn about treatment without antibiotics. Could the score be used in this manner?
3. How do I read the percentages?
4. Why doesn't the score discriminate higher percentages?
5. In what populations has the score been developed and validated?
6. How have the factors been selected, and why have, for example, comorbid diseases and fever not been included?
