Introduction
In recent years, health technology assessment (HTA) organizations such as the Canadian Agency for Drugs and Technologies in Health (CADTH) have shifted their strategic framing and direction towards health technology management (HTM) (1). This shift acknowledges the need to assess the value of health technologies, in terms of health outcomes achieved per dollar spent (Reference Porter2), throughout their lifecycle and the development and implementation of approaches to achieve this. Learnings from the fields of disinvestment (Reference Peacock, Ruta, Mitton, Donaldson, Bate and Murtagh3–Reference Calabro, La Torre, de Waure, Villari, Federici and Ricciardi9), de-adoption (Reference Niven, Mrklas, Holodinsky, Straus, Hemmelgarn and Jeffs10), and de-implementation (Reference Prasad and Ioannidis11–Reference Upvall and Bourgault13)—which tend to focus on mitigating low-value care—offer key lessons to move the HTM agenda forward. Of note are the importance of adopting a value-based lens, broad stakeholder engagement, a robust evidentiary foundation, and clear guidance for action. Given the growing scholarship in this area, an approach that can integrate aspects of and learnings across these fields may provide a more inclusive and acceptable means of promoting ongoing optimal technology use (Reference MacKean, Noseworthy, Elshaug, Leggett, Littlejohns and Berezanski14). One such consolidated approach that has been proposed is Health Technology Reassessment (HTR) (Reference MacKean, Noseworthy, Elshaug, Leggett, Littlejohns and Berezanski14–Reference Soril, Niven, Esmail, Noseworthy and Clement16).
HTR is defined as the systematic, evidence-based assessment of the clinical, economic, ethical, and social impacts of an existing health technology to inform its optimal use relative to its alternatives (Reference Noseworthy and Clement17). The primary objective of HTR is to support the development and implementation of evidence-informed policies and decisions to achieve optimal value for money of existing technologies throughout their lifecycle (Reference MacKean, Noseworthy, Elshaug, Leggett, Littlejohns and Berezanski14). Thus, HTR differs from HTA in that it is focused on not only evaluating an existing technology in use, rather than a new technology being considered for adoption into the healthcare system but also in the fact that it goes beyond the recommendation to directly enact a policy and/or practice change (Reference Soril, MacKean, Noseworthy, Leggett and Clement15).
The language to describe HTR is purposefully framed around optimizing value through the appropriate use of existing technology (Reference Noseworthy and Clement17); thus, HTR is intended to address overuse or misuse of low-value technologies and underuse of high-value technologies alike (Reference Soril, Niven, Esmail, Noseworthy and Clement16). To guide the practice of HTR, a model was developed and is comprised of three phases: Phase I Technology Selection, which includes identification and prioritization; Phase II Decision, involving evidence synthesis and development of policy or practice recommendation; and Phase III Policy Action, comprised of implementation, monitoring, and evaluation of the policy or practice recommendation (Reference Soril, MacKean, Noseworthy, Leggett and Clement15). Despite this conceptual grounding, practical advancements in the field of HTR are limited (Reference Pant, Boucher and Frey18). There have been international efforts to identify and prioritize candidate technologies for HTR (Reference Elshaug, Moss, Littlejohns, Karnon, Merlin and Hiller19–Reference Soril, Seixas, Mitton, Bryan and Clement21), yet there are few demonstrable examples of policy action to realize optimal use of those technologies (Reference Pant, Boucher and Frey18).
From a pragmatic and opportunistic lens (Reference Leon, Davis and Kraemer22), an ideal healthcare system context in which to test the HTR model would be one where there are clear opportunities for improving sub-optimal technology use. Critical care is a clinical area of high technology use and high cost owing to the severity of illness and specialized, resource-intensive care needs of its patient population (Reference Cullen, Sweitzer, Bates, Burdick, Edmondson and Leape23–25). With a history of rapid adoption of new technology, and inconsistent use of confirmatory research to support such decisions (Reference Kahn26;Reference Puri, Puri and Dellinger27), the need for active efforts to reassess existing technologies in the intensive care unit (ICU) has been increasingly recognized (Reference Niven, Mrklas, Holodinsky, Straus, Hemmelgarn and Jeffs10;Reference Stelfox, Niven, Clement, Bagshaw, Cook and McKenzie28). One technology that received much attention internationally for high rates of overuse or misuse, and thus serves as the ideal candidate for HTR (Table 1), is the practice of red blood cell (RBC) transfusions for critically ill patients (Reference Shander, Javidroozi and Lobel29).
Candidate technology for HTR: RBC transfusions in critically Ill patients
Therefore, to add to the practical HTR knowledge base, we set out to complete a multi-phase HTR of the candidate technology, RBC transfusions for critically ill patients. In alignment with the conceptual model, this HTR consisted of three phases and the findings from Phase I and II have been previously published (Reference Soril, Noseworthy, Stelfox, Zygun and Clement30–Reference Soril, Noseworthy, Stelfox, Zygun and Clement32). In this present study, we report the results of the third and final phase of the HTR: Policy Action. Specifically, the development, implementation, and evaluation of a bundle of behavior change interventions to address inappropriate RBC transfusion practices in one ICU in Alberta, Canada are described and key learnings are provided for the broader field of HTR.
Methods
Research Design and Setting
The HTR programme is comprised of three phases and the findings from the first two completed phases (Reference Soril, Noseworthy, Stelfox, Zygun and Clement30–Reference Soril, Noseworthy, Stelfox, Zygun and Clement32) are summarized in the Supplementary Methods. For the present study, Phase III, a small-scale pilot implementation study was conducted in a single ICU to evaluate the feasibility of implementing a bundle of behavior change interventions aimed at optimizing RBC transfusions (Reference Leon, Davis and Kraemer22). The pilot was designed as a controlled before-and-after implementation study and included one intervention site and one control site. The study was conducted over a 14-month period, which consisted of a pre-intervention (1 January 2018–14 November 2018) and a post-intervention phase (15 November 2018–28 February 2019).
The intervention site selected for this pilot initiative was the ICU at the University of Alberta Hospital in Edmonton, Alberta. The control site was the ICU at the Foothills Medical Centre in Calgary, Alberta; no intervention was administered at the control site. Site selection was based on the previous analysis of RBC transfusion events in stable, non-bleeding adult ICU patients completed in Phase I (Reference Soril, Noseworthy, Stelfox, Zygun and Clement30). Between 1 April 2014 and 31 December 2016, the average proportions of RBC transfusions with a pre-transfusion hemoglobin of 70 g/L or more to be approximately 58 percent and 57 percent at the intervention and control sites. These findings suggest a significant opportunity for improvement. The control site was also chosen as it is located a reasonable geographic distance (i.e., approximately 300 km) away from the intervention site to avoid contamination bias. Both sites are mixed medical and surgical ICUs situated in university hospitals in the two largest urban centers in the Province. They are similar in terms of capacity (28−30 beds) and in the number of board-certified ICU physicians (18−20 full-time equivalent intensivists). Post-graduate medical education trainees (i.e., medical fellows and residents) also provide similar clinical coverage at each site (Reference Bagshaw, Wang, Zygun, Zuege, Dodek and Garland33).
Stakeholder Engagement
The research team consisted of academic researchers and healthcare professionals with diverse expertise in HTR, implementation science, and critical care medicine. Local clinical leaders from the intervention site, including the ICU Medical Director, the Chair of the Department of Critical Care Medicine, and a Clinical Nurse Educator, were integral members of the research team and co-led the design, dissemination of information, and implementation of the intervention.
Multi-Modal Intervention
Theory-Informed Intervention Design
The intervention was designed using a theory-informed approach to target facilitators of and barriers to RBC transfusion practice change at the intervention site. In Phase II of the HTR, key behavioral determinants were previously identified through a survey of intensivists in Alberta completed (Reference Soril, Noseworthy, Stelfox, Zygun and Clement32). The survey was informed by the theoretical domains framework (TDF) to explore intensivists’ perceptions of current RBC transfusion practices and identify relevant behavioral determinants to practice change within the theoretical domains of the framework (Reference Soril, Noseworthy, Stelfox, Zygun and Clement32). Among intensivists at the intervention site, specific facilitators of and barriers to RBC transfusion practice change were identified in the theoretical domains of knowledge, behavioral regulation, motivation and goals, and beliefs about consequences. Relevant intervention modalities were then mapped to theories in the identified domains using the TDF taxonomy (Reference Michie, Johnston, Francis, Hardeman and Eccles34) (Supplementary Table 1). Possible intervention modalities to modify provider RBC transfusion practices were identified through a systematic review and meta-analysis also completed in Phase II of the HTR (Reference Soril, Noseworthy, Dowsett, Memedovich, Holitzki and Lorenzetti35). Details for the two studies completed in Phase II are briefly described in the Supplementary Methods.
Tailoring Intervention Modalities
The mapped list of possible intervention modalities, such as education, feedback, protocols, algorithms, etc., was reviewed with the clinical leaders to determine their receptivity and the suitability of proposed clinical practice behavior change techniques for the intervention site. Through discussion and consensus, the research team selected group education initiatives and audit and feedback of aggregate-level clinical performance data. This was provided to all healthcare providers responsible for ordering or involved in the decision to order, an RBC transfusion (i.e., ICU physicians, nurses, and medical trainees) as the intervention modalities. Details of the selected intervention modalities are provided in the Supplementary Methods.
Implementation
The multi-modal intervention was only implemented at the intervention site. During the post-intervention phase, three in-person educational sessions were held at the intervention site (one specific to ICU physicians; two lunch-and-learn events for nurses and trainees) and led by two members of the research team. The audit and feedback process was implemented in two cycles and, within each cycle, two-forms of written feedback were provided: detailed physician feedback reports (emailed to physicians only) and graphical feedback posters (displayed throughout the ICU for all healthcare providers). All of the content for the group education and audit and feedback were directly informed by and co-designed with the clinical leaders. Implementation details and timelines are outlined in Supplementary Figure 1.
Monitoring and Evaluation
Data Source and Outcomes
Clinical data were retrospectively obtained from the established critical care information system eCritical Alberta (Reference Brundin-Mather, Soo, Zuege, Niven, Fiest and Doig36). eCritical prospectively captures comprehensive data on each ICU admission and was used for the audit and feedback component of the intervention and for the overall study evaluation. Patients for whom a restrictive transfusion strategy has not been proven safe, nor superior, to a liberal transfusion strategy (e.g., chronic anemia, active blood loss, acute coronary syndrome, myocardial infarction, and neurological injury or traumatic brain injury) were excluded (Reference Carson, Stanworth, Roubinian, Fergusson, Triulzi and Doree37). Complete details regarding the included data elements and definitions, as well as the comprehensive inclusion and exclusion criteria of ICU admissions (Reference Soril, Noseworthy, Stelfox, Zygun and Clement30), are described in the Supplementary Methods.
The primary outcome measure was the proportion of RBC transfusions associated with a pre-transfusion hemoglobin value (i.e., patient's hemoglobin level measured within 24 h prior and most proximal to the transfusion (Reference Soril, Noseworthy, Stelfox, Zygun and Clement30)) greater than or equal to 70 g/L. Secondary outcomes included the costs of RBC transfusion events, and ICU and hospital mortality and LOS.
Data Analysis
Descriptive statistics were calculated for patient admission and outcome data (Supplementary Methods). Aggregate frequencies and percentages of RBC transfusion events were stratified within pre-transfusion hemoglobin ranges. The monthly percentages of RBC transfusion events with a pre-transfusion hemoglobin value at or above 70 g/L with 95 percent confidence intervals (CI) were also plotted. A two-sample z-test of proportions was conducted to compare the primary outcome before and after the intervention at each site; the significance level was set at .05. The cost of RBC transfusion events is reported as the cost per 10 admissions to account for the different number of months in the pre-intervention and post-intervention phases. It was assumed that 1 RBC unit was equal to 300 mL and the cost of transfusing 1 RBC unit is $666.10 in 2017 Canadian dollars (CAD) (Reference Lagerquist, Poseluzny, Werstiuk, Slomp, Maier and Nahirniak38). Costs were inflation-adjusted (39) and reported in 2019 CAD. Secondary mortality outcomes were also reported as the number of ICU and hospital deaths per ten admissions. The primary and secondary outcomes were reported as unadjusted values. All analyses were performed using STATA IC statistical software, V13.1.
Evaluation Survey
One week after the intervention period, healthcare professionals at the intervention site were asked to complete an online survey (Survey Monkey) to evaluate their experiences and the perceived usefulness of the intervention. The survey was administered between 6 March 2019 and 27 March 2019. Responses were analyzed using descriptive statistics and for questions with Likert scale response options, the percentages of selected responses were also reported. Survey questions and details regarding survey administration are provided in the Supplementary Methods.
Results
Included Patient Admissions and RBC Transfusions
During the 14-month study period, there was a total of 3,061 ICU admissions in which a patient received at least 1 transfusion at the intervention (n = 1,852) and control (n = 1,434) sites (Supplementary Figure 2). Of these, 482 admissions (intervention: n = 296; control: n = 186) met the inclusion criteria for the study. The characteristics of included patients are summarized in Supplementary Table 2.
Specifically, at the intervention site, patients received 2.2 transfusions per admission pre-intervention and 1.9 transfusions per admission post-intervention (Supplementary Table 3). The number of RBC transfusions per admission at the control site was similar throughout the study period (pre: 2.1; post: 2.0). In both sites, 1 RBC unit was administered per transfusion event before and after the intervention.
Pre-Transfusion Hemoglobin Values
The mean pre-transfusion hemoglobin values for included patients at the intervention site (pre: 69.0 ± 6.9 g/L; post: 69.4 ± 6.9 g/L) and control site (pre: 70.6 ± 7.2 g/L; post: 69.7 ± 6.9 g/L) were similar before and after the intervention (Table 2). Prior to the intervention, the proportion of RBC transfusions with a hemoglobin value at or above 70 g/L at the intervention site was 30.3 percent; this proportion did not change post-intervention (28.2 percent, p = .64). Similarly, at the control site, the proportion of RBC transfusions with a hemoglobin value at or above 70 g/L was not significantly different between the pre-intervention (47.2 percent) and post-intervention phases (40.0 percent, p = .22). Throughout the study period, the monthly proportion of RBC transfusions with hemoglobin values at or above 70 g/L was also highly variable at both sites (Figure 1).
Monthly proportion of RBC transfusions with pre-transfusion hemoglobin of 70 g/L or more.
Summary of pre-transfusion hemoglobin values
a RBC transfusions with pre-transfusion hemoglobin values.
b p-value from z-test of proportions between pre-intervention and post-intervention phases for each site.
RBC Transfusion Costs, Mortality, and Length of Stay
The costs of RBC transfusions are presented in Figure 2. The total estimated cost of RBC transfusions per ten admissions at the intervention site was $15,110 pre-intervention and $13,383 post-intervention. The cost attributed to the RBC transfusions with a hemoglobin value at or above 70 g/L was $4,466 per ten admissions pre-intervention, and $3,723 per ten admissions post-intervention. At the control site, the total estimated cost of RBC transfusions per ten admissions was $14,832 in the pre-intervention phase and $13,629 in the post-intervention phase. The cost attributed to the RBC transfusions with a hemoglobin value at or above 70 g/L was $6,627 and $5,143 per ten admissions in the pre- and post-intervention phases, respectively.
Cost of RBC transfusions per ten admissions in control and intervention sites.
Supplementary Table 4 summarizes the ICU and hospital mortality and LOS. At the intervention site, the number of ICU deaths was 1.8 per 10 admissions pre-intervention and 2.2 per 10 admissions post-intervention, whereas the number of hospital deaths was 2.3 per 10 admissions pre-intervention and 2.9 per 10 admissions post-intervention. Both the median ICU LOS (pre: 6.8 days, IQR: 9.1; post: 7.6 days, IQR: 11.9) and hospital LOS (pre: 18.8 days, IQR: 27.2; post: 17.0 days, IQR: 26.1) were similar before and after the intervention. At the control site, there were fewer ICU deaths (pre: 2.2; post:1.5) and hospital deaths (pre: 2.8; post: 1.9) per 10 admissions during the post-intervention phase. Further, the median ICU LOS (pre: 7.5 days, IQR: 14.0; 10.0 days, IQR: 13.0) and hospital LOS (pre: 20.5 days, IQR: 23.0; post: 18.2 days, IQR: 18.4) were relatively consistent throughout the study.
Experiences and Perceptions at the Intervention Site
Forty-three healthcare providers at the intervention site completed the evaluation survey and the majority of respondents were registered nurses (n = 25) (Supplementary Table 5). The feedback posters were most frequently observed by respondents (77 percent), and the poster information reported as most useful for their practice were the cost (67 percent) and percentage (56 percent) of RBC transfusions with a pre-transfusion hemoglobin value above 70 g/L (Supplementary Figure 4). The physician feedback reports were received by 10 respondents and, of these, eight respondents reported that information about the proportion of RBC transfusions at or above 70 g/L was useful for their practice. Thirteen respondents indicated that they attended an educational session and, of the information presented, the cost of RBC transfusions with a pre-transfusion hemoglobin value above 70 g/L (26 percent) was useful for their practice. Twenty-three respondents did not attend an educational session, whereas only four respondents did not observe the feedback posters. Only 7 percent of respondents reported that none of the information from the intervention was useful.
The majority of survey respondents agreed (19/38) or strongly agreed (11/38) that they appreciated receiving information about the RBC transfusion practices in their ICU through the intervention (Q1, Supplementary Figure 3). Most respondents also learned information about the RBC transfusion practices in their ICU that they did not know before (22/38 agreed; 3/38 strongly agreed) (Q2, Supplementary Figure 3). Nineteen out of thirty-seven respondents (51 percent) neither agreed nor disagreed that knowing the information about past RBC transfusion practices in their ICU changed the way they order transfusions for their patients (Q3, Supplementary Figure 3). Further, twenty-two out of thirt-eight (58 percent) neither agreed nor disagreed that their practice change will continue (Q4, Supplementary Figure 3).
Discussion
In this HTR, we describe the development, implementation, and evaluation of a bundle of behavior change interventions aimed at optimizing RBC transfusion practices in 1 pilot ICU setting. Working collaboratively with local clinical leaders, we found that it was feasible to implement a tailored, multi-modal intervention targeting ICU physicians, nurses, and trainees over the 14-month study period. We were able to leverage an existing ICU electronic information system to access routine clinical data and efficiently monitor and evaluate utilization and cost of the technology, as well as patient and healthcare system outcomes. Of the outcomes measured, however, we did not identify a statistically significant difference in the proportion of potentially inappropriate RBC transfusions (i.e., associated with a hemoglobin of 70 g/L or more) before and after the intervention. There was also marked variability in the monthly proportion of potentially inappropriate RBC transfusions and no significant difference in the average proportion before and after the intervention. Despite no observable change in technology use, we found that the intervention was appreciated by most respondents of the evaluation survey.
A principal gap to advancing the field of HTR is the relative paucity of practical experiences (Reference Haas, Hall, Viney and Gallego4;Reference Daniels, Williams, Robinson and Spence6;Reference Elshaug, Hiller, Tunis and Moss8;Reference Polisena, Clifford, Elshaug, Mitton, Russell and Skidmore40). Leggett et al. (Reference Leggett, Noseworthy, Zarrabi, Lorenzetti, Sutherland and Clement41;Reference Leggett, Mackean, Noseworthy, Sutherland and Clement42) and Polisena et al. (Reference Polisena, Clifford, Elshaug, Mitton, Russell and Skidmore40) found that despite increasing international initiatives to identify and prioritize areas of low-value care, there continues to be limited efforts and methods to actualize optimal technology use in real-world practice and policy settings. This HTR was developed in direct response to these knowledge gaps. Guided by the three-phase HTR model (Reference Soril, MacKean, Noseworthy, Leggett and Clement15), we demonstrated that an HTR process can go beyond the selection of candidate technologies and include the implementation of change management efforts at the clinical practice-level. The use of the TDF, in particular, offered an approachable framework to apply behavioral theory to assess and develop techniques that target potential drivers of low-value RBC transfusion practices (Reference Michie, Johnston, Francis, Hardeman and Eccles34;Reference Cane, O'Connor and Michie43;Reference Michie, Johnston, Abraham, Lawton, Parker and Walker44). The TDF has been recognized as a valuable tool for de-adoption and de-implementation initiatives (Reference Stelfox, Brundin-Mather, Soo, Parsons Leigh, Niven and Fiest45;Reference Voorn, Marang-van de Mheen, van der Hout, Hofstede, So-Osman and van den Akker-van Marle46), and we have demonstrated the utility of the TDF to guide intervention design in the context of an HTR.
A strength of this HTR was the early and meaningful engagement of the local champions or opinion leaders. The clinical leaders at the pilot intervention site supported this HTR initiative from the outset and, as such, were integral to the design and implementation (Reference Miech, Rattray, Flanagan, Damschroder, Schmid and Damush47). Given the interdisciplinary nature of care in the ICU, clinical leaders also advised that targeting the ICU physician group was necessary but not sufficient for ensuring uptake of evidence-based best practices of RBC transfusions. Consequently, our interventions modalities targeted practice change at the team-level. The lunch and learn sessions, for example, are common to the culture of the intervention site and were selected to increase intervention acceptability. Of the thirteen respondents of the evaluation survey who attended an educational session, approximately 92 percent reported that the information presented was useful for their practice.
The audit and feedback modality was tailored to the intervention site and informed by the clinical leaders and educational session attendees. For instance, we found that stakeholders were most interested in receiving feedback on the proportion and cost of potentially inappropriate RBC transfusions. We developed feedback to include not only the cost of RBC transfusions that could have been potentially avoided (i.e., associated with a hemoglobin of 70 g/L or more) but also examples of ICU-specific technologies that could have been funded with the foregone resources. These technologies were of high value and underused (or not currently available) at the intervention site. Feedback of clinical performance alongside costs has been infrequently reported in audit and feedback interventions (Reference Ivers, Jamtvedt, Flottorp, Young, OdgaardJensen and French48). However, with the rise in efforts to mitigate low-value care, healthcare providers are increasingly regarded as de facto stewards of healthcare resources (Reference Soril, Clement and Noseworthy49–Reference Colla, Kinsella, Morden, Meyers, Rosenthal and Sequist51). The perceived usefulness of this tailored feedback information to clinical practice was validated by responses from the evaluation survey.
Limitations and Considerations for Future Research
As noted, we did not observe a practice response to the planned intervention. This finding may reflect insufficient power (due to length of the follow-up period) of our pilot study to detect a difference or, given the observed decreases in the proportions of RBC transfusions associated with a hemoglobin of 70 g/L or more at the intervention and control sites, simply a lack of intervention effect (Reference Craig, Dieppe, Macintyre, Michie, Nazareth and Petticrew52). Ultimately, this questions whether it is worthwhile pursuing the implementation of the intervention for a province-wide HTR initiative given the required time and resources (Reference Kaur, Figueiredo, Bouchard, Moriello and Mayo53). It is, therefore, necessary to reflect on the limitations of this pilot study when considering decisions for future HTR initiatives.
One limitation related to the baseline RBC transfusion practices at the start of the pilot study. Through our initial exploratory analysis of transfusion practices (1 April 2014 and 31 December 2016) in Phase I of the HTR, we found that, on average, over 50 percent of RBC transfusions were associated with a pre-transfusion hemoglobin of 70 g/L or more at the intervention and control sites (Reference Soril, Noseworthy, Stelfox, Zygun and Clement30). However, by the pre-intervention phase of our pilot (1 January 2018 to 14 November 2018), both sites had since exhibited improvements in their RBC transfusion practices and to varying degrees. These changes may reflect contamination from other initiatives prior to our study (e.g., hospital quality improvement initiatives, regional presentations) or a secular trend, due to growing knowledge, evidence, and guidance in this area, resulting in increased awareness of or attention to RBC transfusion practices (Reference Soril, Noseworthy, Stelfox, Zygun and Clement30). Secular trends in RBC transfusion practice have been described in the literature in other jurisdictions. From a large, retrospective observational study conducted in the United States, Netzer et al. (Reference Netzer, Liu, Harris, Edelman, Hess and Shanholtz54) found that mean pre-transfusion hemoglobin levels significantly decreased from 79 ± 1.3 to 73 ± 1.3 g/L over a 10-year follow-up period and there was a significant decrease in the proportion patients who were transfused at a hemoglobin level less than 70 g/L (Reference Netzer, Liu, Harris, Edelman, Hess and Shanholtz54). Thus, re-evaluation of baseline practices and site characteristics should be performed prior to initiating Phase III of the HTR. This step can serve as a “go or no-go” point to confirm whether there was still an appreciable opportunity for improvement and determine whether the Policy Action phase should proceed.
The differing lengths of time for the pre-intervention and post-intervention periods were also a limitation. Although the objective of the pilot study was not to assess the effectiveness of the intervention, if the study were sufficiently powered wherein the pre-intervention and post-intervention phases were of sufficient and comparable lengths, it would be ideal to assess secular trends or seasonal variations versus intervention effect (Reference Wagner, Soumerai, Zhang and Ross-Degnan55). To evaluate the effectiveness of a future intervention using interrupted time series analyses, for example, there should be at least twelve data points before and after the intervention (Reference Wagner, Soumerai, Zhang and Ross-Degnan55). A minimum of 100 observations at each data point is also recommended to minimize estimate variability (Reference Wagner, Soumerai, Zhang and Ross-Degnan55). Considering the extent of variability, we observed in the estimated monthly proportion of RBC transfusions with a pre-transfusion hemoglobin of 70 g/L or more, increasing the number of sites, and hence number of monthly observations, in the intervention and control groups may help to minimize variability in an effectiveness study.
Another limitation to our work was that we did not prospectively collect data. As a result, we did not know the exact reason why an RBC transfusion was ordered because this information is not routinely collected with the electronic RBC unit orders. Instead, we identified pre-transfusion hemoglobin values from the available laboratory data and used these as proxy measures of appropriateness for most of the included RBC transfusion events. Our data access was also limited to the ICU patients that received an RBC transfusion during the study period. Therefore, we do not know how the characteristics of the transfused patients varied from those who were not transfused but were still at risk of receiving a transfusion. Nor do we know whether there were unanticipated consequences to their care, or potential harm to patients, as a result of the intervention or the HTR programme as a whole. For example, ICU physicians may have overcorrected in response to the intervention and inappropriately reduced the use of RBC transfusions for all ICU patients, including those who were actively bleeding and/or low baseline hemoglobin levels (i.e., less than 70 g/L). Given these limitations with the secondary data, prospective electronic collection of study data should be considered in future research. For instance, in the case of RBC transfusions, this may be actualized by adapting the existing electronic blood product ordering systems and the eCritical information system to collect additional data elements of interest to the study (e.g., rationale for each RBC transfusion order). Access to data for all admissions to the study sites would enable more robust monitoring and evaluation, and if chosen as a behavior change intervention, real-time audit, and feedback.
Due to the non-randomized nature of the controlled before-and-after study design, and lack of adjustment for differences in patient characteristics in our analysis, the present findings may also be subject to selection bias and confounding. Further, respondents of the post-study evaluation survey reported variable exposure to the intervention modalities; the educational sessions and detailed physician feedback reports reaching fewer of the respondents compared to the feedback posters. This limitation may also reflect selection bias due to under-coverage or non-response bias in our sample of respondents (Reference Johnson and Wislar56). Alternatively, poor educational session attendance may have resulted from scheduling arrangements and/or limited availability of healthcare providers on those dates. With regards to the physician feedback reports, some intensivists may have simply chosen not to view the reports (Reference Lord and Taylor57). In addition, over half of the evaluation survey respondents neither agreed nor disagreed that their practice changed, or that their practice change would continue following the intervention. These findings may indicate respondents’ uncertainty as to whether clinical practice change can be achieved and sustained through audit and feedback alone (Reference Payne and Hysong58). However, we were unable to conduct follow-up interviews or focus groups to explore the reasons underlying these responses due to limited human and financial reasons. Future HTR initiatives would, therefore, benefit from incorporating a thorough process evaluation as part of their Phase III. This may involve primary qualitative data collection (e.g., surveys, interviews, and/or focus groups with stakeholders exposed to the intervention) and periodic quantitative data collection to evaluate intervention fidelity (Reference Carroll, Patterson, Wood, Booth, Rick and Balain59) and identify factors influencing observed quantitative outcomes (Reference Hulscher, Laurant and Grol60). Although a process evaluation would be resource-intensive, the evidence generated would allow investigators to remain nimble and adapt their intervention—or pivot the HTR process as a whole—to respond to changes in knowledge and behaviour that may potentially outpace the ability of the HTR to enhance practice change.
Conclusions
This study represented the final phase in a body of work that sought to actualize the conceptual HTR model (Reference Soril, MacKean, Noseworthy, Leggett and Clement15) and conduct an HTR of an existing health technology in a real-world healthcare system context. We found that it was feasible to design, implement, and evaluate a tailored, multi-modal behaviour change intervention to optimize RBC transfusions in the ICU. Therefore, we demonstrated that an HTR process can go beyond the identification and prioritization of technologies and veritably include the implementation of change management efforts at the clinical practice-level. The TDF, in particular, offered an approachable framework to apply behavioural theory to assess and develop techniques that target potential drivers of low-value RBC transfusion practices. However, through the small-scale pilot evaluation, we did not identify a significant difference in the proportion of potentially inappropriate RBC transfusions before and after the intervention. The findings from this first real-world application of the HTR model uncovered important methodological and practical considerations, particularly when initial efforts to manage existing technology use may not go as expected. However, there remain a number of unknowns that require further study, such as determining how frequently an HTR should occur throughout the lifecycle of a technology, as well as understanding when an HTR programme may be considered truly complete or even unwarranted. Thus, more research employing the HTR model in other healthcare contexts, with other technologies, and by other users is required to address such unknowns and optimize the current model. To advance the field of HTR, we must learn from and build upon local, national, and international experiences and continue to develop innovative and thoughtful ways to implement evidence into policy and practice. The need to manage technologies throughout their lifecycle will not cease and neither must the pursuit of approaches to effectively optimize their use.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462321001653.
Acknowledgment
We wish to thank Ms. Samantha Taylor and Ms. Nadia Baig for their assistance with the administration, development, and implementation of this research.
Funding
LJJS was supported by an Alberta Innovates – Health Solutions (AIHS) Graduate Studentship Award. The authors have no other funding details to disclose.
Conflict of Interest
There are no conflicts of interest.
