Over the past 20 years, the expansion of the health technology assessment (HTA) infrastructure in Canada has seen the establishment of hospital-based HTA units to support evidence-based decision making at an organizational level (Reference Gagnon1). The goal of such units is to formulate recommendations directed at hospital administrators and healthcare professionals to facilitate informed decisions about the integration of health technologies or procedures into the healthcare system, while ensuring appropriateness of care. Hospital-based HTAs are particularly needed for medical device assessment, which often takes place at the hospital level, incorporating factors specific to the individual institution (Reference Facey, Boivin and Gracia2). These units can serve as gatekeepers to ensure that health technologies with unproven safety or efficacy are not adopted into routine practice.
The formulation of recommendations in HTA is a complex and inherently subjective process, necessitating the integration of several diverse concepts, including clinical benefit, impact on patient and personnel, and strategic and ethical considerations. Part of the difficulty in developing an effective framework for translating scientific evidence into recommendations lies in the context-specific nature of the task. While several helpful guidelines have been published, such as the Ontario Health Technology Appraisal Committee's (OHTAC) framework for health technology decisions (3;Reference Johnson, Sikich and Evans4); the Canadian Agency for Drugs and Technologies in Health (CADTH) process for developing recommendations (5); the EVIDEM framework linking HTA and multiple-criteria decision analysis (MCDA) (Reference Goetghebeur, Wagner and Khoury6;Reference Goetghebeur, Wagner and Khoury7); and the GRADE (Grading of Recommendations Assessment, Development, and Evaluation) guidelines to rate the strength of recommendations (Reference Andrews, Guyatt and Oxman8), these guidelines do not address all the relevant domains that need to be considered for the uptake of technologies within a hospital-based setting (Reference Poulin, Austen and Scott9–Reference Tanios, Wagner and Tony11). Some factors considered relevant when evaluating a health technology in a hospital setting, such as efficacy and safety, may overlap with others that are relevant at a provincial or national level. However, other factors may be peculiar to hospital settings, for example, surgical expertise due to a high volume of cases, or the availability of dedicated funding from a donor, and may influence whether a technology is recommended for use at a given hospital. The challenge lies not only in identifying these different factors, but in determining how they are considered in developing the final recommendation.
The overarching goal of this study is to develop a practical framework for formulating recommendations that can be applied in a systematic manner, and that ensures all relevant decision criteria are considered. Our objectives are: (i) to identify decision criteria specific to the context of hospital-based health technologies and interventions, (ii) to determine the extent to which the expert community agrees on the importance of the identified criteria, (iii) to incorporate the identified criteria into a decision-aid tool, and (iv) to illustrate the application of a prototype decision-aid tool in the context of a recently completed HTA.
Methods
Context
The Technology Assessment Unit (TAU) of the McGill University Health Centre (MUHC) is an advisory body that makes recommendations to the hospital administration regarding implementation of medical interventions using the best available scientific evidence. The TAU is composed of two distinct bodies: a scientific research staff, and an inter-disciplinary policy committee representing physicians, nurses, allied health professionals, and patients. A policy question is first researched by the scientific staff who gather and analyze the necessary scientific evidence pertaining primarily to the three main factors of effectiveness, safety, and cost. Other factors, such as feasibility or local expertise, may be considered, according to particular contexts. This evidence, in the form of an HTA report, is reviewed by the TAU Policy Committee, who issue a final recommendation. TAU has issued recommendations for over seventy health technologies and interventions since its inception in 2001. Recommendations fall into one of three categories: Approved: The overall assessment supports a recommendation that the technology be approved for routine use with support through the hospital operating budget; Approved for evaluation: The overall assessment does not support recommendation for permanent approval but is promising enough to warrant an evaluation, with support from the hospital budget for a limited time, often with local data accrual, before re-evaluation; Not approved: The technology is not approved for financial support through the hospital operating budget, but may be re-evaluated based on new evidence.
To assess the need for identifying relevant decision criteria, we first conducted a pilot exercise to determine whether a simple framework defined by only three factors (efficacy, cost and safety) was sufficient to inform the final recommendation. We reviewed twenty-three of our past reports, which were chosen with an attempt to include a wide variety of interventions from different hospital departments (Supplementary Table 1), and documented the association between these three factors and the final recommendation. We created an overall rating of the strength of the evidence based on the individual ratings for efficacy, safety, and cost. Thus, a technology with at least two weak ratings (e.g., high cost and poor evidence for safety) received a weak overall rating. The results, shown in Figure 1, illustrate that there were several instances where the recommendation appeared to contradict the available evidence. For example, among the seven projects that received a final recommendation of “Approved” by the committee, one had a weak overall rating (red) because, although there was sufficient evidence for efficacy, the evidence for safety and cost was not favorable. Among the nine projects that received a final recommendation of “Not approved”, two had favorable evidence for efficacy, safety, and cost, indicating good overall strength (green), but were nonetheless rejected. This exercise underscored two points: (i) that only three broad factors were clearly insufficient in informing the final recommendation, and (ii) that a structured and transparent framework was needed to document the decision-making process.
A simple framework consisting of three criteria (efficacy, cost, and safety) to evaluate twenty-three TAU reports. The graph shows the number of health technology assessments (n = 23) evaluated by TAU that received a recommendation of “Approved,” “Approved for Evaluation,” or “Not Approved,” categorized by the overall strength of efficacy, safety, and cost (red = weak overall strength; yellow = moderate strength; green = strong). If, for a given report, at least two of the three criteria had the same rating (e.g., weak efficacy and weak safety), then the overall strength corresponded to this rating (weak overall strength).
Identification of Relevant Decision Criteria Specific to Hospital-Based HTA
We aimed to identify an exhaustive list of decision criteria relevant to the decision-making process using four sources: an existing framework for HTA recommendations (3), our past experience at TAU, a literature search, and feedback from a survey. We used the framework for evidence-based across a diverse recommendations published by OHTAC in 2010 as a starting point (3). It included four decision-criteria: overall clinical benefit, consistency with societal and ethical values, value for money, and feasibility of adoption, which are further divided into sub-criteria (Supplementary Table 2). Two TAU members (N.A. and N.D.) independently reviewed the twenty-three TAU reports identified during the pilot exercise, a heterogeneous sample including a diversity of interventions evaluated between 2007 and 2014.
We recorded whether all OHTAC decision-criteria were relevant, and identified additional important criteria that played a role in influencing the decision-making process; this process was repeated for all twenty-three reports until no new criteria were identified, an approach known as theoretical saturation (Reference Tanios, Wagner and Tony11). We then reviewed the literature to identify other relevant criteria. Our literature search included the Medline and Embase databases using the following high-level concepts: HTA, decision-making criteria, framework for decision-making, priority setting, and resource allocation. We reviewed all articles that focused on guidelines or decision making within HTA, published until 2018, giving special consideration to a survey of hospitals in France. The latter was based on an extensive literature review that included forty-seven decision criteria grouped under eight domains: factors associated with the disease, impact on patient, impact on user, economic impact, social and ethical impact, organizational impact, strategic considerations, and other considerations (Laurent Piazza; unpublished data 2016).
Evaluating the Importance of the Identified Criteria
To evaluate the extent to which our stakeholders agreed on the relevance of the identified criteria, and to ensure that other important criteria were not excluded, we polled experts within and outside the MUHC community, including clinicians, researchers, administrators, patient representatives, and experts in HTA. The goal of this exercise was to identify those criteria that were considered universally important (indispensable) across a representative sample, while also gathering information on criteria that generated a lack of consensus.
An online survey including all the identified criteria was created and emailed to survey participants, who were asked to rate the importance of each criterion using a 5-point Likert scale (5 = indispensable, 4 = important, 3 = somewhat important; 2 = not very important; 1 = not at all important; 0 = do not know), and to suggest any other criteria that they considered relevant. To obtain a measure of which criteria were considered universally important across the expert community polled, we calculated the percentage of respondents rating each domain as “Important” or “Indispensable.” We did not have specific rules to retain or drop criteria; the distribution of importance ratings was mainly used to restructure the domains. The aim was to include any criteria deemed relevant by the expert community.
Creating and Applying a Decision Aid
Based on our survey results, we developed a final list of twenty-three criteria sub-divided into seven domains to be included in our tool.In creating a decision aid, previous frameworks have attempted to take a quantitative approach, using a weighting or scoring process such as multi-decision criteria analysis, and thus attempting to translate a subjective process into a more objective one (Reference Poulin, Austen and Scott9;Reference Sampietro-Colom, Morilla-Bachs, Gutierrez-Moreno and Gallo10). We believed that such a process was impractical because the weights used for each criterion would vary depending on the population surveyed and the technology being evaluated, requiring the creation of new weights for each technology assessment, thus rendering the tool overly cumbersome.
We defined an approach that relied instead on a visual means for arriving at a final recommendation. In the first step, data gathered for each criterion during the HTA are rated by the scientific staff. Members of the multi-disciplinary policy committee then rate the importance of each of the twenty-three decision criteria, and arrive at a preliminary recommendation by juxtaposing their importance rating against the results of the HTA. A final recommendation is reached by consensus of the whole committee, after deliberation of criteria associated with discord. To illustrate the use of our decision aid, we applied it to a recently completed health technology assessment by TAU on extra-corporeal membrane oxygenation (ECMO).
Results
Identifying and Evaluating the Importance of Relevant Decision Criteria
Figure 2A shows the decision-criteria identified after applying the OHTAC framework and from reviewing our past reports and the literature. We initially identified twenty criteria, grouped into five global domains, namely: clinical benefit, value for money, feasibility, ethics and values, and strategic considerations.
Initial decision criteria identified from literature and review of past TAU reports (A) and final criteria included in framework after survey of stakeholders (B).
The ensuing survey to establish the importance of each criterion was emailed to a total of sixty-one panelists representing the diversity of hospital stakeholder group (clinicians, administrators, HTA members, and patients), of whom fifty-two responded (85 percent). Thirty-eight respondents (73 percent) were MUHC members, sixteen (31 percent) were members of HTA organizations, twelve (23 percent) were administrators, and four (8 percent) represented the patient perspective (mutually nonexclusive groups). A total of 96 percent of respondents were Canadian. Of the nine who did not respond, five (55 percent) were MUHC health professionals, three (33 percent) were members of HTA organizations, and one (11 percent) was an administrator.
The results of our survey on the importance of the identified criteria are shown in Figure 2A. Six criteria (effectiveness, quality of the evidence, safety, costs related to the technology, improved hospital efficiency, and Impact on patient-important outcomes) were considered indispensable or important by greater than 90 percent of respondents (highlighted in green), while three criteria (external financial support for the technology, Prior hospital experience with the technology, and ability to increase attractiveness of the hospital to patients/health professionals) were considered important or indispensable by fewer than 40 percent of respondents (highlighted in red).
Eight criteria (burden of illness to patient, ease of implementation, need for evidence of effectiveness in the local setting, disease is a public health priority, benefit of technology to society, impact on delivery of equitable care, ability to offer a cutting-edge technology or new alternative treatment, and availability of the technology in other local centers) were generally considered relevant because the percentage of respondents considering them indispensable or important was between 62 and 79 percent. However, for three criteria (absence of alternate treatment options, disease is rare, and impact on creating research opportunities and external collaborations) (highlighted in gray), there was less agreement on their importance (40 to 60 percent considered them important or indispensable) because there was greater variability in the distribution of the importance ratings. For example, for the “disease is rare” criterion, 46 percent of respondents considered it important or indispensable, 42 percent considered it somewhat important, and 12 percent rated it as not very important (results not shown).
In their comments, respondents highlighted the need for a domain pertaining to patient-important criteria, as well as a domain that captures the impact on the healthcare system of integrating the health technology or intervention under evaluation.
Based on these data and the detailed comments of the respondents, we reorganized some criteria and added new ones. In particular, we redistributed the criteria within the “ethics and values” domain into two new domains: “impact on patient” and “impact on healthcare system.” Figure 2B presents the twenty-three criteria, now grouped into seven global domains: clinical benefit, impact on patient, value for money, feasibility, impact on healthcare system, strategic considerations, and ethical considerations.
Consolidating the Identified Criteria into a Final Framework
We conceived our final framework tool as a resource to be used in three steps, illustrated in Figure 3. In the first step, upon completion of the HTA report, the TAU research staff evaluate the new technology against the identified decision-criteria, and record their findings in the tool. The research staff also rate whether the findings are favorable for approval of the intervention by answering the question “Do these findings favor the approval of the technology for the intended purpose?” For each criterion, the research staff may choose from four color-coded options (Yes, No, Maybe, Need more information).
Steps in the integration of a recommendations decision aid at the Technology Assessment Unit of the McGill University Health Centre.
In the second step, a link to our online decision aid including the TAU findings and the completed TAU report is emailed to each member of the TAU Policy Committee. Members are requested to read the report and then rate the importance of each criterion based on their values and preferences and within the context of the technology. In considering how important each criterion is in shaping the final recommendation pertaining to the particular technology, the committee member is asked to rate the importance of each criterion as Important, Somewhat important, or Not at all important, again using a color-coded schema. The goal of such an exercise is to provide the Policy Committee members with a visual aid for arriving at a final recommendation.
By juxtaposing the importance rating of each criterion against the results of the health technology assessment, it should become apparent whether factors considered important by the committee member also have favorable findings from the assessment report. A technology would be expected to receive a recommendation of “not approved” if the majority of criteria considered important by the committee member did not have favorable findings. Any other recommendation would require documentation of the extenuating factors justifying its approval. At this stage, each committee member is also asked to make a final recommendation and document their reasons. Individual survey responses are anonymous to members of the Policy Committee, and are only available to TAU research staff.
In the final step, a policy committee meeting is convened where TAU research staff present the distribution of the importance rating for each criterion across the committee, as well as the distribution of the suggested final recommendation. This ensures that the focus of the meeting is on those criteria that caused dissent. Committee members will have the opportunity to discuss extenuating reasons and any disagreements until a consensus on the final recommendation is reached. All reasons will be explicitly documented, resulting in a structured and transparent process.
Applying the Framework
Figure 4 illustrates the use of our tool for the assessment of ECMO for cardiac life support in adult patients at the MUHC. ECMO is a life-support technique to circulate and oxygenate a patient's blood outside their body during cardiac or respiratory failure. TAU evaluated the policy question of whether the routine use of ECMO should be incorporated at the MUHC to support three different populations: adult patients in cardiogenic shock, patients with in-hospital cardiac arrest, and patients with out-of-hospital cardiac arrest.
An illustrative example of the application of our tool for the evaluation of extra-corporeal membrane oxygenation (ECMO).
Figure 4 shows the completed tool for ECMO. After completion of the HTA report, TAU research staff recorded their findings in column 3 of the tool. They also indicated whether the findings for each criterion were favorable for the approval of routine use of ECMO in adults at the MUHC (column 4). The tool was then emailed to one member of the TAU policy committee, who was asked to rate the importance of each criterion (column 5). Figure 4 illustrates that the majority of criteria considered important by the committee member did not have corresponding favorable findings. For example, the committee member rated effectiveness, quality of the evidence, safety, and cost as very important; however, the TAU findings for these criteria were either unfavorable or moderately favorable.
TAU findings were favorable for criteria such as innovativeness and number of patients affected, which the committee member rated as somewhat important in shaping the final recommendation. Thus, the juxtaposition of the importance rating against the TAU findings would enable the reviewer to quickly evaluate the strengths and weaknesses of ECMO against his or her individual values and preferences; in this case it led the reviewer to conclude that ECMO at the MUHC should be considered for evaluation, that is, neither approved nor rejected outright. Although our tool was not ready to be used during the final TAU policy committee deliberations for ECMO, it was able to identify those criteria (innovativeness and strategic considerations; Figure 4) that eventually swayed committee members toward a final recommendation of approved for evaluation rather than an outright rejection, despite the weak effectiveness and safety evidence.
We have recently confirmed the feasibility of using our tool among all members of the TAU policy committee, and these results will be described in a forthcoming manuscript. The goal of this exercise was to assess ease of use of the tool and the impact of the tool on the decision-making process and on transparency.
Discussion
The development of recommendations in HTA is a complex process requiring the assimilation of diverse concepts of a contextual nature. It is evident from the literature that there is considerable interest in developing a practical framework for translating scientific evidence into recommendations (3;5;Reference Goetghebeur, Wagner and Khoury6;Reference Andrews, Guyatt and Oxman8–Reference Ritrovato, Faggiano, Tedesco and Derrico12). However, a decision-aid tool that is easy to use, captures the subjectivity inherent in HTA, and is specific to a hospital setting has not yet been developed. GRADE (Reference Andrews, Guyatt and Oxman8) and OHTAC (3) have published basic frameworks enumerating broad domains to be considered, while other groups, such as CADTH (5) and the European network for Health Technology Assessment (EUnetHTA) (Reference Ritrovato, Faggiano, Tedesco and Derrico12), have attempted to identify more specific criteria. In an exhaustive summary, Tanios et al. (Reference Tanios, Wagner and Tony11) surveyed 140 clinical and policy decision makers from twenty-three countries on the importance of forty-three criteria relevant to healthcare decision making, which included the consideration of vulnerable populations such as children, and the impact on the environment.
In an attempt to develop more concrete ways of arriving at a final recommendation, most frameworks, such as those by EVIDEM (Reference Goetghebeur, Wagner and Khoury6;Reference Goetghebeur, Wagner and Khoury7) and EUnetHTA (Reference Ritrovato, Faggiano, Tedesco and Derrico12), have used MCDA and weighted analyses. These methods calculate an overall score by assigning weights to each criterion based on their relative importance. For example, Poulin et al. (Reference Poulin, Austen and Scott9) used MCDA to develop a decision-aid tool to support decision making within the Alberta Health Services in Canada, wherein reviewer scores on 28 criteria were incorporated into the final decision-making process. Sampietro-Colom et al. (Reference Sampietro-Colom, Morilla-Bachs, Gutierrez-Moreno and Gallo10) created a value/risk tool to aid in decision making by assigning weights to twelve criteria (six value and six risk criteria) using median scores from a survey of twenty-eight decision makers. These weights were then multiplied by a score for each criterion, obtained by comparing the new technology to an alternative. The overall scores were plotted on a value vs risk matrix to aid in determining the final recommendation.
While there is a certain attractiveness in using scores to render the decision-making process more objective, a scoring system obscures the subjectivity implicit in the assignment of weights, which is dependent on the technology being assessed, and on the population used to create the weights. Although HTA is rooted in an evidence-based approach, the decision-making process is inherently subjective and dependent on value judgements, which should not be perceived as threats to this scientific objectivity (Reference Hofmann, Cleemput and Bond13). Thus, instead of a scoring system, we opted for a tool that would render the value judgements explicit. We created a visual aid that would serve to guide each individual reviewer toward a final recommendation, by weighing his or her own values against the collected evidence. It should become apparent to the reviewer that criteria considered important should also have favorable findings from the HTA for the technology to be approved. Instead of assigning an arbitrary cutoff score to determine whether a technology is approved or rejected, our tool allows us to identify sources of discordance across committee members, and encourages an open discussion until a consensus is reached. The explicit consideration of value judgements ensures a transparent and structured process because each reviewer is asked to document their reasoning for their recommendation; the tool thus provides a structure for the discussion of sources of disagreement.
The illustrative example using ECMO highlights several strengths of our tool. First, our tool was able to identify those criteria that influenced the final recommendation of approved for evaluation, despite the limited scientific evidence for effectiveness and safety. This underscores our initial premise that further criteria in addition to effectiveness, safety, and cost are required to shape the final recommendation. Our tool is thus particularly useful in evaluating less straightforward technologies by allowing stakeholders to address the underlying nuances in a methodical manner, while promoting transparency and accountability. Second, the juxtaposition of the evidence against the importance rating obliges stakeholders to acknowledge the lack of evidence for domains they consider important, thus tempering their championing of an unproven technology. Our tool compels members to consider all the various criteria that go into shaping the final recommendation. A further advantage is that our tool encourages increased participation from all members by soliciting their input in an anonymous manner before the meeting, free of group pressure and the impact of influential members.
In conclusion, decision making is an inherently subjective process, and it is difficult to recognize and quantify all individual values and preferences. As evidenced from the literature, there is enormous interest in developing a practical tool to create a standardized and structured way to translate scientific evidence into policy. A decision aid such as the one we have developed, which explicitly incorporates value judgements, can render the process more structured and balanced. The advantages of our tool are that it is simple and easy to use, generating low user-fatigue and potentially greater compliance, while creating a methodical process that encourages transparency and reproducibility. Future work will include testing our tool on a diverse range of technologies to evaluate its robustness; refining the steps in our framework to test whether policy decision makers should be initially blinded to the scientific assessments when rating the importance of criteria; and aligning the reporting of our final HTA with the decision criteria included in our tool.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462319000254.
Conflicts of interest
None.
