The background knowledge that underpins hospital-based health technology assessment (HTA) is the same as that used for national/regional HTA. However, the tools and the data required for making a decision with regard to the introduction of a given new health technology (HT), its timing frame, and the weights given to the different criteria used in the assessment may differ from one setting to another. Hospital decision makers need a quick and tailored answer to make a decision on the acquisition of specific health technologies. Traditional time-consuming HTA reports may not suit their needs and constraints. Additionally, the relative importance given to the different dimensions in the assessment at hospital level may vary from that at national or regional level in the healthcare system. Indeed, whereas safety, efficacy/effectiveness, and cost-effectiveness are the main criteria incorporated in a traditional national/regional assessment; safety, organizational impact, efficacy, and budget impact are the dimensions most frequently present in hospital level HTA (Reference Cicchetti, Marchetti, Di Bidino and Corio6). Strategic considerations, such as the innovativeness of the HT in a teaching hospital and the investment effort may also play a prominent role in the final decision at hospital level. In brief, hospital-based HTA combines relevant information on the clinical outcomes coming from scientific evidence, and the unique organizational and economic implications of a new HT at hospital level, thus providing shorter and more timely HTA reports to hospital managers.
The first publication discussing the need to create multidisciplinary committees in hospitals that would assess the appropriateness of acquiring new HTs is dated 1979 (Reference Mamana15), and it was not until 1986 that the experience of these committees was formally described (Reference Millenson and Slizewski16). These early committees used some of the core elements of the HTA process, but did not apply current standards of HTA methodology. Nowadays, HTA within the hospital setting is widespread including valuable experiences in Canada, Denmark, Italy, and Spain.
The first tool aimed at implementing HTA at hospital level was designed by the Andalusian HTA Agency (AETSA, Spain) back in 1999 (Reference Briones, Loscertales and Pérez Lozano4). Nowadays, one of the probably most well-known tools in this respect is the so-called “mini-HTA” developed by the Danish Centre for Health Technology Assessment (DACEHTA) in 2005 (Reference Ehlers, Vestergaard and Kidholm7). Mini-HTA is a management and decision tool edited as a check-list with several questions grouped into four relevant HTA aspects (technology, patient, organization, and economy), allowing for short answers to questions, and with the potential to be adapted to local objectives, decision criteria, and time schedules (Reference Ehlers, Vestergaard and Kidholm7).
Mini-HTA is useful for the local assessment of innovative HTs and in the standardization of decision-making processes regarding the introduction of new HTs, making them both transparent and accountable. Under fixed budget conditions, once the different HTs have been assessed using the mini-HTA, hospital decision makers face the challenge of how to best prioritize resource allocation among potentially very different HTs. A tool that visually compares HTs and uses mini-HTA criteria to inform decision making could be of valuable assistance. In addition, this tool should be able to adequately compare and discriminate among HTs once assessed.
The objective of this study is twofold. First, to convey the development of a new decision support tool to aid hospital decision makers in their investment decisions regarding new competing HTs. Second, to test whether this tool visually discriminates among competing HTs according to their attributable risks and benefits.
METHODS
We developed a computer program to plot the risk and value scores of new HTs that had previously been assessed with the mini-HTA. It is a two-layer software program based on (i) the different weights given to the mini-HTA variables, and (ii) the results from the comparative assessment of the new HT and the available alternative.
From the twenty-six questions in the Danish mini-HTA tool (Reference Ehlers, Vestergaard and Kidholm7), we selected those nine deemed most relevant from a hospital's point of view based on the relative interest shown by clinicians and managers when commissioning mini-HTA reports to our unit during the past 3 years. We added three more variables as a result of incorporating cost-effectiveness, innovativeness, and investment effort concerns coming from these same clinicians and managers at the time of acquiring new HTs. The resulting twelve variables were grouped into two categories named “value” and “risk.” Value variables refer to HT-related clinical and patient implications, while risk variables include those that consider the impact of the new HT on the hospital management dimension. Table 1 shows these 12 variables together with their definitions.
Value and Risk Variables.
When a new HT is under consideration for acquisition in a given healthcare system, decision makers at different levels (i.e. macro, micro level) may assign dissimilar relevance (hereafter called “weight”) to the risk and value variables assessed. As an example, hospital managers may attribute a higher weight to budget impact in their decision than national-level payers would. So as to identify potential weight differences and ascertain the final weight for each of the variables in our study, we designed a specific questionnaire. Respondents were asked to score on a Likert scale (from 1 = low importance to 9 = very important) the weight attributed to each of the twelve variables mentioned above (Table 1). To test its validity, the questionnaire was discussed with professionals working at the Information, Assessment and Quality Agency (AIAQS is the HTA Agency of the Catalan Ministry of Health [MoH]), and with a range of professionals working in our hospital. The questionnaire was then refined according to the comments received. The final version of the questionnaire was sent to a convenience sample of twenty-eight key decision makers in the healthcare system representing two decision-making levels: (a) national/regional (eight of them working in the Catalonian MoH planning and management department; seven working in the seven HTA Agencies/units in Spain – AETS, Lain Entralgo, AETSA, AVALIA-T, AIAQS, OSTEBA and the HTA Unit of the Canary Islands MoH) and (b) local (thirteen working at our hospital - ten Clinical Institutes Directors and three members of the Hospital Executive Committee). All twenty-eight decision makers were asked to attribute a weight (from 1 to 9) to each of the twelve variables selected. Non-normal data distributions were observed and the median and interquartile ranges were estimated for each variable using SPSS 17.0. Statistical differences among the weights assigned by the two groups of decision makers were investigated using the Mann-Whitney U test. The estimated median weights of each of the twelve variables constitute the first layer in the software program. Participants’ level of agreement regarding the weights assigned to the twelve variables was explored using a correlation matrix, strong clustering suggesting a high level of agreement among participants, while wide dispersion suggested a low level of agreement.
The second layer in the computer program includes the results obtained by performing a mini-HTA for each of the new HTs assessed. This second layer consists of providing a score to each of the twelve value and risk variables for the new HTs, as compared with the technology currently used (i.e., comparator). The score assigned could be (a) higher, (b) equal (if no differences were observed, or a conclusive answer could not be obtained due to the nature of the evidence), (c) not available (when no information regarding the variable was found), or (d) lower. An algorithm combining the results of the two layers was developed to obtain an overall score for the new HT under assessment. In the algorithm, the weight given to each of the twelve variables is named W (W, i.e., mean values from 1 to 9), and the results from the comparison between the new HT and the currently available alternative through the mini-HTA are called scores (S) (i.e., new versus comparator for each mini-HTA criteria, Higher = 1; Equal = 0; Not Available = 0; Lower = −1). The algorithm reads as follows: Overall Score = Risk point value (R) + Value point value (V) = [W*S(Staff impact) + W*S(Physical Space Impact) + W*S(Process of care impact) + W*S(Incremental cost) + W*S(Net cost) + W*S(Investment effort)] + [W*S(Safety) + W*S(Clinical Benefit) + W*S(Cost-Effectiveness) + W*S(Patient Impact) + W*S(Quality of Evidence) + W*S(Innovativeness)]
We made use of three specific HTs of great interest to our hospital to test whether the decision support tool discriminated correctly between competing HTs according to the risks and benefits variables mentioned above. The mini-HTA processes performed for each of these HTs followed the methodological standards established for any HTA systematic review (5). Clinical variables and patient impact were assessed by a systematic and exhaustive search for evidence, and by rating the selected evidence using the available quality scales (21). Organizational and economic variables information was obtained by interviewing hospital financial managers and physicians on the impact of the new HTs. A team of clinicians, managers, and HTA analysts work together throughout the process and, after a process of deliberation, attributed final scores to risk and benefit variables for the new HT as compared with its comparator.
The specific hospital new HTs assessed were: (a) Gravitational Platelet Separation of GPS-III Biologics Biomet® based on platelet-rich plasma (PRP) used in total knee arthroplasty (TKA), (b) Robot Da Vinci used for prostatectomy, and (c) a new generation (programmable) deep brain stimulator (DBS) for Parkinson's disease. These HTs were selected as they represent the portfolio of technologies our hospital provides, that is, high-cost large-size equipment (Robot Da Vinci); moderately priced, small-size equipment/prosthesis (DBS); low-cost, small-size product used in a procedure (PRP). Results from the three mini-HTA reports were entered into the program second layer and a visual dot plot of the value and risk level for each HT was generated. The tool was named matrix4value.
RESULTS
We obtained a 100 percent response rate from the decision-maker survey aimed at assessing the weights for the twelve selected mini-HTA variables. Because no significant differences were found among the scores from decision makers, the medians of all responses were taken as the weights (W) for the program algorithm. Table 2 shows the median value and interquartile range for each of the variables obtained from the decision-maker survey.
Weights Given by Decision-Makers to Value and Risk Variables.
The results from the exploratory analysis using the correlation matrix shows a clear clustering of the data among four of the variables included within the value dimension (i.e., safety, clinical benefit, cost-effectiveness, and quality of evidence), which suggests agreement on the weights given by decision makers to these variables. No clear clustering is observed in the variables included in the risk category (Supplementary Figure 1, which can be viewed online at www.journals.cambridge.org/thc2012062).
The systematic search for scientific evidence performed for each of the three HTs assessed produced the following results: (i) PRP: five articles (3;8;Reference Everts, Devilee and Oosterbos9;Reference Gardner, Demetrakopoulos, Klepchick and Mooar11;Reference Peerbooms, de Wolf and Colaris19) and one unpublished article (unpublished data, 2009), (ii) Da-Vinci Robot: three articles and four evaluation reports of which the most recent evaluation report was used (Reference Maeso Martínez, Reza Goyanes, Blasco Amaro and Guerra Rodríguez14) and (iii) DBS: two articles, one systematic review, and one clinical practice guideline (17), which incorporates a comprehensive high quality review of the evidence including the studies addressed by the mentioned systematic review; the clinical practice guideline and one article (published after the appearance of the mentioned guideline and providing new high quality evidence) were selected (Reference Weaver, Follett and Stern24). Table 3 shows the results obtained by comparing each new HT and its available alternative using mini-HTA (mini-HTA results available upon request to authors) and the subsequent deliberative process by the mini-HTA team. These results (i.e., scores) were introduced into the second layer of the matrix4value program.
Result from the Mini-HTA for each of the HT Assessed and Deliberative Process (Second Layer Decision-Support Tool).
1. Comparator: non use of platelet rich plasma for total knee arthroplasty.
2. Comparator: endoscopy surgery
3. Comparator: conventional surgery
Finally, Figure 1 shows how the decision support tool and algorithm visually discriminates competing HTs according to their risks and benefits. It shows that DBS is the HT with the highest value and lowest risk for the hospital, that PPR is a low value and moderate risk technology, and that the Robot Da Vinci has an almost neutral value and a high risk for the hospital. Thus, matrix4value presented good visual discrimination capacity between very different HTs.
Matrix4Value: value and risk dot plot.
DISCUSSION
The matrix4value program developed in our study makes possible the comparison and discrimination of new competing and considerably different HTs. This program therefore constitutes a valuable tool for hospital decision makers needing to make decisions on technology acquisitions under fixed budget constraints.
The tool makes use of information derived from decision makers regarding the relative value and risks of competing HTs. In this respect, previous surveys have highlighted the fact that the relevance given by decision makers to the different variables considered when assessing HTs may differ (Reference Cicchetti, Marchetti, Di Bidino and Corio6). To estimate the relative weight for each criterion we selected decision makers at macro and micro (hospital) levels, and explored whether they assigned different weights according to their decision-making level. If differences did exist, two overall scores for the same HT would be obtained, one of them based on the weights given by macro decision makers, and another by micro decision makers. This would probably lead to different (perhaps contradictory) investment decisions. In our healthcare system, the hospital's budget and the decision to acquire some types of HTs, particularly big ticket technologies, are made by policy makers at the macro level. We thus need to ensure that the decision made at hospital level supported by this tool would be in line with funding decisions made at macro level. The results from our study show that the weights given by macro and hospital decision makers to the different value and risk variables largely coincided. This is not in line with previous findings, which show that, while the variables most frequently assessed by hospital HTA relate to safety, efficacy/effectiveness, organizational impact, and budget impact (Reference Cicchetti, Marchetti, Di Bidino and Corio6), some of these variables (i.e., organizational and budget impact) are rarely assessed by national/regional HTA agencies (Reference Nielsen, Funch and Kristensen18), suggesting that differences in weights may exists among macro and micro decision makers. Further research with a larger sample size could elucidate whether significant differences actually exist.
The matrix of correlations among variables exhibits a clustering of data points in the upper-right corner of the correlation plots. This suggests that decision makers mostly agree that all variables are important to some extent. Of interest, variables with a tighter clustering are those included in the “value” dimension (i.e. safety, clinical benefits, patient impact, cost-effectiveness, quality of scientific evidence, and innovativeness). This means that weights for the “value” dimension are consistent among all decision makers. Such strong clustering is no longer observed among “risk” dimension variables, which means that decision makers differ more on the weights given to “risk” dimension variables than on the “value” dimension variables. This leads us to hypothesize that, when considering the introduction of a new HT, decision makers tend to give similar weights to “value” variables, while they differ on the weight of “risk” variables (i.e. staff requirements, physical space impact, process of care impact, incremental cost, net cost, and investment effort).
Weight differences between risk variables do not follow a trend in the two stakeholder groups, that is, National/Regional and Hospital. This situation could reflect the well-known risk attitudes (risk-seeker / risk-averse / risk-neutral) of individuals (Reference Gafni and Torrance10). Finally, risk trends and weights differ among hospital respondents to the survey. We expected responses from hospital clinicians and managers to be more homogeneous, particularly in reference to organizational impact and economic impact variables, because these are very context-specific and relevant variables at the hospital level. Our expectations were not met in the study, maybe as a result of a small sample size of the group or due to existing differences in risk attitudes.
Regarding the use of mini-HTA reports in assessing investment decisions at hospital level, Kidholm et al. (Reference Kidholm, Ehlers, Korsbek, Kjarby and Beck13) suggested that the data provided by such reports might sometimes be deemed insufficient, especially when information about the selection and interpretation of the clinical literature is missing. If credible advice were to be used in the acquisition of new HTs, we would agree it is necessary to guarantee the quality of mini-HTAs reports. In fact, a valuable matrix4value needs to be based on high-quality processes and contents information. The mini-HTA reports undertaken in this study complied with international quality recommendations and standards for HTA reports (Reference Velasco Garrido, Borlum Kristensen, Palmhoj Nielsen and Busse23).
New decision support tools have recently been developed for the identification and prioritization of HTs accounting for obsolescence (2) and market introduction (1). We were also able to identify one decision support tool in the literature that allows for the comparison of the characteristics of different brands of the same HT (Reference Sloane22). However, we were unable to identify any HTA-based decision support tool that helps in the prioritization of very different competing HTs. The HTA-based tool developed here has demonstrated its capacity to visually discriminate different competing HTs according to their value and risks dimensions. This visual display could considerably aid hospital managers in the deliberations to make decisions on investment priorities. It is important to mention that this tool is not intended for use in isolation but to be integrated in the standard evidence-based decision-making process, where there is usually a debate involving other issues relevant for the final investment decision (e.g., total budget, hospital strategies, and priorities).
Decision makers’ preferences may change with time and context. This decision support tool allows for changes in weights and results when needed, making it a dynamic tool which facilitates the incorporation of new scores as a consequence of new evidence becoming available on the topic, different contexts and constraints, and changes in decision-makers’ preferences. This is a relevant feature because it has been shown that while values are rather more static measures, preferences are more likely to change over time (Reference Kean12).
This study has certain limitations. The small sample size may have precluded detecting statistical differences in variable weights between the different decision makers (i.e., intergroup weights). However, ultimately, the validity of the results from this convenience sample may be guaranteed by its representativeness of the whole spectrum of decision makers regarding the introduction of new HTs in our healthcare system. Another limitation of this study could be the lack of randomization of respondents. Participants were selected according to their recognized professional careers and involvement in decision making at different levels in the healthcare system. This option was chosen to ensure that responses to the questionnaire were based on professional experience; therefore, selection bias should not be of practical importance. It could also be argued that this tool does not include patients’ views when deciding on weights. Previous literature (Reference Sampietro-Colom, Espallargues and Rodriguez20) has shown that patients tend to give more weight to those criteria that have a close relation with patient-related issues (e.g., pain). Because the weight attributed to patient-related aspects is already high in our study, we understand this limitation has been minimized here. However, it is recommended that further qualitative work, including patients’ views and preferences, should be pursued in the future. Finally, another limitation concerns the mini-HTA approach. Traditionally, systematic reviews are usually performed by two independent reviewers to ascertain the quality of the review (5). In the present study, all the mini-HTA reviews were performed by the same researcher. However, the reviewer followed the international procedures and standards requested for a high quality HTA (Reference Velasco Garrido, Borlum Kristensen, Palmhoj Nielsen and Busse23). A second researcher closely supervised the work done thus minimizing concerns about the quality of the mini-HTA reviews.
POLICY IMPLICATIONS
Performing mini-HTAs at hospital level has several advantages. The main advantage is that it makes possible the application of HTA methodology and processes in an easy, practical, and timely way, to support decision makers involved in the introduction of new HTs. However, direct comparison among mini-HTA reports can be difficult for hospital decision makers at the time of prioritizing investments. This tool visually shows how HTs are placed in a risk-value matrix and thus assists when the decision-making process involves competing HTs previously assessed with mini-HTA, providing more clear-cut information for the prioritization of HTs investments under the fixed budget scenarios of a hospital setting.
ACKNOWLEDGEMENTS
The authors would like to thank HealthTech for the initial idea on the development of this tool, and Marie Huc for her technical assistance throughout the process.
SUPPLEMENTARY MATERIAL
Supplementary Figure 1: www.journals.cambridge.org/thc2012062
CONTACT INFORMATION
Laura Sampietro-Colom, MD, MScPH, PhD, Directorate of Innovation, Hospital Clínic of Barcelona, (lsampiet@clinic.ub.es), Deputy, C/ Villaroel 170, 1.7. 08036 Barcelona, Spain
Irene Morilla-Bachs, MPH, Department of Innovation Management, Fundació Clínic per la Recerca Biomèdica
Santiago Gutierrez-Moreno, BSc, PhD, Department of Innovation Management, Fundació Clínic per la Recerca Biomèdica
Pedro Gallo, Gallo, PhD, Department of Sociology and Organizational Analysis, University of Barcelona
CONFLICTS OF INTEREST
All authors report they have no potential conflicts of interest.
