Various frameworks are available to help draw health technology assessment (HTA) recommendations (Reference Johnson, Sikich and Evans1;Reference Guyatt, Oxman, Schünemann, Tugwell and Knottnerus2). A commonality among these frameworks is the prerequisite that the decision to recommend or not recommend a technology should be based on the strength of the evidence. This prerequisite is largely advocated for by scientists, and for HTA stakeholders in particular, it is a very important point (Reference Baum3–Reference Janoudi, Amegatse, McIntosh, Sehgal and Richter6). Consequently, it is generally recognized that the more the evidence indicates that the benefits exceed the disadvantages, the higher the likelihood that the technology will be recommended.
In recent decades, the strength of the evidence has been assessed in various ways, and the design of the study, number of patients and length of follow-up are generally considered to be key elements (Reference Campbell, Barnes and Kirby7). Although there may be some discrepancies regarding the exact figures associated with these elements, they remain central to the strength of the evidence (Reference Burns, Rohrich and Chung8). However, if evidence is one consideration in the process of drawing HTA recommendations, it is not the only consideration. Indeed, uncertainties regarding a range of factors do not always preclude drawing a clear recommendation. Along with the strength of the evidence, and sometimes in addition to this, many other factors are considered in the process of deciding whether or not to recommend a given technology. This raises the question of which criteria and which weights to consider in the recommendation process, and, more generally, what to do in the absence of evidence.
Most of the time, recommendations are guided by clinical efficacy and cost-effectiveness (Reference Clement, Harris and Li9;Reference Allen, Walker, Liberti and Salek10). However, other factors are also frequently considered, such as safety (Reference Kolasa and Wasiak11), budget impact (Reference Mauskopf, Chirila and Masaquel12), consistency with societal and ethical values (Reference Johnson, Sikich and Evans1), availability of alternative interventions (Reference Chambers, Morris, Neumann and Buxton13), feasibility of adoption (Reference Johnson, Sikich and Evans1), etc. In their review, Bossers et al. (Reference Bossers and Van Engen A14;Reference Bossers, Ivanescu and Heemstra15) identified up to 111 factors or criteria grouped into three categories: clinical, economic, and societal factors. Their targeted literature review retained twelve studies (Reference Bossers, Ivanescu and Heemstra15) from different countries (United Kingdom, United States, Australia, Canada, and others from OECD). This multiplicity of factors illustrates the fact the recommendation process is multifaceted and many aspects related to the technology need to be considered.
Moreover, the review by Bossers et al. (Reference Bossers and Van Engen A14;Reference Bossers, Ivanescu and Heemstra15) was conducted at the level of national HTA agencies, and not at a lower level, where many more considerations can influence the recommendation process. In the local context of a hospital, in particular, the “traditional” criteria used by national HTA agencies to make a recommendation may be insufficient (Reference Martin16). In such a case, this will preclude the making of a recommendation that is adapted to the context in which it is inserted. Considering that the context of the particular institution at which a recommendation was requested is one of the factors with the greatest influence on implementation (Reference Busse, Orvain and Velasco17), it clearly appears that to fail to consider the local context in making a recommendation would go against efficient and effective decision making. In this setting, it is all the more important to consider what Martin et al. call the SLEEPERs (Social, Legal, Ethical, Environmental, Political, Entrepreneurial/Research issues) when drawing recommendations (Reference Martin, Lal, Moodie, Zhu, Cheng, Sampietro-Colom and Martin18).
Recommendations from HTA are brought at different levels of management. As a consequence, there may be a difference between recommendations and outcomes of interest from a national HTA agency and a Hospital-Based HTA (HB-HTA) unit in the assessment of similar technologies. For example, a Canadian agency and a HB-HTA Unit have performed an assessment of reuse of single-use medical devices independently. While the agency assessment was quite broad and included several devices (19), the HB-HTA unit focused on a single device (20). Recommendations provided by the agency were more general and their recommendations were at the ministry and healthcare institutions level, while the HB-HTA unit focused on the needs of a single institution. Local data in a specific setting is an important source of data to adequately formulate recommendations in a better applicability.
In this article, we present a case study conducted in a Canadian hospital to illustrate the process that is undertaken at a local HTA level and the issues considered when there is limited published evidence on a technology.
CONTEXT
In our institution, monoplane imaging devices are used to perform all angiographic procedures, including neurointerventions. However, these interventions have evolved to the point that simultaneous, real-time biplane imaging is now considered by experts necessary to ensure procedure quality and to reduce imaging-system manipulations (21). Because our monoplane system is reaching the end of its useful life, our institution is now planning to replace this device with a biplane system for vascular neurointervention. In addition, our institution has a high level of expertise in vascular neurointervention. Biplane technology is increasingly being used (21) and the specialists at our institution indicated that for safety reasons (i.e., risk of complications) they may refrain from performing certain procedures if the institution were to maintain monoplane technology. This would result in patients from our region being referred to other specialized centers in the province for treatment.
Therefore, the angiography department wishes to replace a monoplane angiography device in one of the angiography rooms with a biplane system. The room would be used for all angiographic examinations, but the biplane technology would be dedicated exclusively to vascular neurointervention, particularly aneurysm embolization, subarachnoid haemorrhage, and the treatment of arteriovenous malformations, primarily to avoid the risk of perioperative complications based on current projections. Moreover, because it would be impossible to perform certain types of vascular neurointervention without a biplane system, our institution's teaching capabilities in neurosurgery could be called into question.
In addition, neurointervention must figure into our institution's offer of services to maintain its status as a tertiary-care facility for stroke. The technological shift toward a biplane system represents, however, a major investment for our institution, an additional $800,000 (Canadian dollars), as well as increased service contracts (approximately $25,000 per year). The question our hospital policy makers face is whether our institution should acquire a biplane device for vascular neurointerventions. For some specialists, biplane technology is already considered standard clinical practice and should be purchased by a hospital. At our institution, however, we have a process to review evidence from literature and discuss with stakeholders. This was done before to decide if a biplane system should replace a monoplane in our particular context (population, skill, development, teaching, etc.).
METHODS
This assessment aimed at comparing biplane to monoplane technology for vascular neurointervention. The first step of this assessment consisted of a collection of data useful for decision making. A systematic literature review was conducted, and details can be found in Bellemare and Poder (Reference Bellemare and Poder22). The systematic review was conducted for neuro-intervention procedures and/or diagnosis (Reference Bellemare and Poder22). While the systematic review publication focused on three main outcomes of interest (radiation dose, medical complications, and procedure time), the extracted data were wider according to the research strategy used (see supplementary data S1 and in Clinical Radiology Journal). All available outcomes from included studies were extracted and used in this current analysis.
Other sources of data were also collected to help decision making, such as a list of hospitals using the biplane systems in Canada, as well as the breakdown of hospitalizations and day surgeries in our institution in the last 2 years for aneurysm or subarachnoid haemorrhage. Consultations with physicians, managers, and biomedical engineers were also conducted. Through these different data sources, we sought information regarding the following outcomes: radiation doses, clinical complications, procedure times, purchase cost, impact on teaching program, the confidence of clinicians in the technology, quality of care, accessibility, and the volume of activity. Keeping in mind the SLEEPERs model (Reference Martin16), these outcomes were considered to be the most important for stakeholders to inform decision making. To note that the SLEEPERs model was not used as a multi-criteria decision analysis (MCDA) tool but as a model to help us considering factors that may be of importance for decision making. In addition, the relative weight of each criterion was not formally established during the process of decision making.
In a second step, a committee was created to develop a recommendation regarding whether or not to purchase the biplane technology, considering the evidence, context, stakeholder experiences, and population equity. The committee comprised managers, clinical experts, physicians (neurosurgeons and radiologists), physicists, and HTA experts. Two rounds of discussion were conducted: one to discuss the results of the systematic review and of Canadian and local data, and another one to discuss the benefits and disadvantages of purchasing the biplane system over the monoplane system. Each discussion was scheduled to last an hour and a half and was facilitated by HTA experts from our hospital-based HTA unit. The recommendation was approved by this committee after a consensus was reached.
RESULTS
The systematic literature review yielded a total of 257 references, and nine were selected for analysis (Reference Bellemare and Poder22). The analysis of these studies showed a very low level of confidence, with a median of 3/32 on the Downs and Black checklist (Reference Downs and Black23). This low score was associated with the presence of bias reducing the confidence in the results from the literature. Most studies were descriptive and did not provide accurate data. In particular, older studies indicate benefits from the biplane system, but it was mostly based on approximation. Considering the two most recent studies, published after the year 2000 and showing the highest level of evidence, no or weak benefits were found. Despite a high level of uncertainty, three elements stood out: a likely reduction in doses and number of injections, a likely reduction in risk of complications because of simultaneous biplane visualization, and a likely reduction in procedure time (Reference Bellemare and Poder22).
The analysis of the list of hospitals using the biplane system in Canada showed that most major hospitals in Canada use this system and that our institution is the only university institution in the Province of Quebec that does not have a biplane system for vascular neurointervention. Some university hospitals in Quebec were also contacted to get information about the reasons for adoption of the biplane system. Respondents clearly indicated that the adoption of the biplane was performed in the absence of scientific evidence because it was obvious that “two planes are better than one”. Of course, it is impossible to generalize for other hospitals that were not contacted, but it seems that the spread of this technology was mostly based on the belief that a biplane is better than a monoplane because it was obvious for managers and physicians that it should be more effective and safe; thus possibly leading to an effect of mimicry between hospitals.
Regarding the data about hospitalizations and day surgeries in our institution for aneurysm or subarachnoid hemorrhage, an in-depth analysis with physicians indicated that the biplane system would reinforce our capacity for fifty complex cases per year. Indeed, when the risk of complications is too high with a monoplane device, these cases are currently transferred to centers with a biplane system (i.e., 130 km away or more). Additionally, consultations with neurosurgeons inside and outside our institution reinforced our belief that the shift toward biplane systems was simply due to the fact that this system constitutes technological evidence of efficacy and safety, which may also explain why very few studies were conducted in this field. Based on this, members of the committee believe that using a biplane system could improve the operator's confidence and reduce the risk of clinical complications.
A biplane system would also be useful in maintaining the neurosurgery teaching program. Additionally, acquiring a biplane system would enable our institution to further develop the practice of vascular neurointervention and to strengthen its capacity to achieve its objectives related to population responsibility, particularly by performing procedures that previously had to be transferred to other centers in Quebec equipped with this technology. Consequently, the biplane system would be an asset in maintaining the service offer required to qualify for status as a tertiary-care center for diagnostic and therapeutic services for individuals at risk of stroke or who have already experienced stroke. Moreover, it is believed that such technology will help to attract expert radiologists and residents to our institution.
On the other hand, acquiring this technology will lead to an extra cost of $800,000 (i.e., purchase) and an additional $25,000 per year for a service contract, as well as an increase in the number of patients treated at our institution. However, some of these extra costs could be financed by the Ministry of Health and Social Services in Quebec, given that our institution is the only university hospital in the province that lacks such a system, thus potentially being eligible for a special fund.
Finally, the committee recommended acquiring a biplane system for vascular neurointervention based on the advantages this system can provide to our institution in terms of safety, quality of care, support for a university teaching program based on best practices, increased accessibility, and enhanced capability in performing complex procedures for cases currently transferred to other centers in Quebec that have this technology. Given the low level of evidence of this technology in the literature, the committee also advocated creating a local registry to collect data for optimizing clinical protocol in the doses of ionizing delivered.
DISCUSSION
Scientific evidence of quantitative measurement (assessment) is not sufficient to produce HTA recommendations. Most of the time, recommendations are based on a value judgement of multiple sources of data. As a consequence, a paradigm shift toward a value judgement (appraisal) method must be considered in the social role of HTA agencies and hospital-based units. In fact, there is a need to consider other evidence provided by contextual data and local expertise. In this study, with the help of the SLEEPERs model developed by Martin (Reference Martin16), we considered extra evidence that led the committee in charge of providing a recommendation to address uncertainties surrounding clinical outcomes and to perform a trade-off between the benefits and disadvantages of the biplane technology. This last point is fundamental in HTA, especially in hospital-based HTA, as decision makers must appraise, accept, and manage the level of risk associated with each new technology.
As noted by Dillon (Reference Dillon24), making the right choices with limited resources requires that decisions be informed by data and analyses that help to manage the risks involved. In our study, the main point was that decision makers believed that the evidence from the literature review was unconvincing to solely base their decision, particularly because of the extra-costs incurred by the biplane technology. The difficulty can be that often when a technology becomes accepted into clinical practice, fewer studies are published despite the fact that there still may be uncertainty around outcomes. This is particularly true when the technology has evolved and why contextual factors are important in such decisions. Consequently, after an analysis of the scientific evidence and contextual data, the recommendation committee considered that the disadvantages involved (i.e., mainly cost) were not sufficiently high to surpass the potential advantages of the biplane in our setting (i.e., clinical confidence, teaching, accessibility, and equity).
Additionally, the value of hospital-based HTA was that it helped the institution justify the need for a biplane to the Ministry of Health and Social Services and apply for extra funding. Moreover, it noted the potential weaknesses of the technology and made it possible to draft a plan to monitor the implementation and follow-up of the technology. Overall, the analysis of the context served to combine the rigor of the HTA process with the ambitions of our hospital in terms of quality of care and of population responsibility. This was done without compromising our objectivity and independence because all key elements in the recommendation process were thoroughly appraised and balanced by considering each one in relation to the others. Nevertheless, more transparency could have been obtained with the formal use of MCDA in which each member of the committee assigns a weight to each criterion (Reference Baltussen, Jansen and Bijlmakers25). This was a limitation of our approach because the SLEEPERs model was not used as a MCDA tool.
We also recognize that, in a different context, the recommendation provided by the committee may have been different, as decisions must be made based on local considerations regarding infrastructure, existing technologies, health professional skills, and competing priorities (Reference Martin16). For example, the decision could have been negative in an Italian context in which cost-containment is a major driver of hospital-based HTA recommendation (Reference Callea, Armeni, Marsilio, Jommi and Tarricone26). Indeed, decision-making processes and societal willingness to pay vary from one institution to another, and it is all the more relevant to consider in this fragmented environment that technological innovations are mainly concentrated in the hospital setting and must be considered in the particular context in which they may be implemented (Reference Allen, Walker, Liberti and Salek10,Reference Drummond27–Reference Abrishami, Boer and Horstman29).
As noted by Allen et al. (Reference Allen, Walker, Liberti and Salek10), “differences in recommendations could be considered to be due to an individual agency's approach to risk perception, and the comparator choice used in clinical and cost-effectiveness studies. Their activities, however, vary because of different mandates as well as unique political, social, and population needs”. All these elements belong to a rationale (value judgement) that may explain why a positive recommendation can be drawn despite low scientific evidence. In this setting, contextual data are of utmost importance for local decision makers. As reported by Halmesmäki et al. (Reference Halmesmäki, Pasternack and Roine30), the applicability of HTA reports in hospital setting will be limited without an analysis of local considerations. Without these data, low level of evidence and uncertainty surrounding a range of factors may lead to reject a technology, as reported by Allen et al. (Reference Allen, Walker, Liberti and Salek10) at a national level.
In conclusion, providing recommendations is a complex process that should be performed with a value judgement based on multiple sources of data. Scientific evidence of quantitative measurement is one consideration, although it is not the only one. Expanding health technology assessment with SLEEPERs or other tools can help producers to incorporate criteria to reach an appraisal-based method of producing recommendations. In this particular case, we demonstrate that technology can be recommended in a given context despite a lack of evidence in the scientific literature by considering local data and expertise in a collaborative, appraisal-based manner. Finally, we must recall that the entire HTA process is itself value-laden.
SUPPLEMENTARY MATERIAL
To view supplementary material for this article, please visit https://doi.org/10.1017/S0266462317004469.
CONFLICTS OF INTEREST
The authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Authors have no potential conflicts of interest to disclose.
