The Italian National Healthcare Service (NHS), within all its Regional components, is facing the challenge of modernizing the current paradigms of care to deliver “more for less” (Reference Hood1). In this regard, the capability to select and implement the most promising innovations has emerged as a priority for both policy makers and scholars of technology assessment, because healthcare technologies represent one of the leading causes of increased expenditure (Reference Greenberg, Peterburg, Vekstein and Pliskin2). The paradox of financing the continuous innovation of current services, while avoiding an unsustainable increase of expenditure in the short-term, may be overcome through the wide diffusion of health technology assessment (HTA) practices (Reference Drummond, Schwartz and Jönsson3). HTA aims at providing decision makers with relevant and reliable information that matches different perspectives in a coherent and evidence-based overview (Reference Zanaboni and Lettieri4).
This modus operandi should be agreed and implemented across the different levels where decisions regarding innovations are taken (i.e., national or regional healthcare system, organizational/hospital, individual/professional) (Reference Battista5). Although these levels agree with this vision, their maturity regarding the implementation of HTA practices still differs (Reference Frønsdala, Faceya and Klempa6). At the institutional level, prestigious and well-established HTA Agencies such as the National Institute for Clinical Excellence (NICE) in England and the Comité d´Evaluation et de Diffusion des Innovations Technologiques (CEDIT) in France have paved the way for the definition of generally accepted good practices for the production of HTA reports that are useful to policy makers for designing their policies. At the professional level, the diffusion of practices, inspired by the evidence-based medicine philosophy and of clinical guidelines generated by systematic literature reviews and meta-analyses, have legitimized decision-making processes regarding technologies and health strategies that should be preferred in terms of effectiveness, efficiency and safety.
However, at the hospital level, little has been investigated to date, and hospital decision makers are left without clear guidelines and good practices regarding how to select investments in healthcare technologies that are value for money (Reference Lettieri7;Reference Gagnon, Desmartis, Poder and Witteman8).
On the one hand, the various frameworks developed at the institutional level such as the EUnetHTA Core Model (9), the EVIDEM model for a multi-criteria appraisal (Reference Goetghebeur, Wagner, Khoury, Levitt, Erickson and Rindress10–Reference Thokala, Devlin and Marsh12), as well as the traditional cost-effectiveness analysis (CEA), have proved to: (i) be inconsistent with the severe constraints hospitals have in terms of time and resources available for the assessment exercise; (ii) be disconnected with respect to the administrative processes already in place in hospitals, such as budgeting and clinical risk management (Reference Lettieri, Masella and Nocco13); and (iii) overlook the organizational impacts resulting from the adoption of new technologies/routines (Reference Battista5). On the other hand, the various frameworks developed specifically for HBHTA, for example, the Danish Mini-HTA (Reference Ehlers, Vestergaard and Kidholm14), the essential questions check-list (Reference Uphoff and Krane15), the value/sustainability matrix (Reference Lettieri, Masella and Nocco13), and the criteria for disinvestment (Reference Mitton, Dionne, Damji, Campbell and Bryan16), have not emerged as widely diffused practices and have remained local or anecdotal.
Despite the relevance of all the above-mentioned frameworks, they share similar limitations that have encouraged further research. First, they suggest a qualitative approach to technology assessment that does not provide decision makers at the hospital level with quantitative syntheses, as advocated by the consolidated Multi-Criteria Decision Analysis (MCDA). Second, they do not crystallize the metrics that should be used to measure the impacts of different comparable technologies, thus limiting the replicability of the analyses. Third, they are stand-alone proposals for HBHTA that do not guarantee the coherence, as well as the potential integration between the organizational/hospital and the healthcare system/institutional levels.
Some of these limitations, the last two in particular, have recently been addressed within the European project “AdHopHTA” that aims at providing decision makers at the hospital level with guidelines to implement HTA practices in hospitals (Reference Sampietro-Colom, Lach and Cicchetti17).
The proposed study presents the results achieved through a 3-year experiment developed in the Northern Italian Regions, concerning the implementation of an original HBHTA framework that mitigates the three limitations described above This framework, namely IMPAQHTA (IMPlementation of A Quick hospital-based HTA), (i) has been developed in adherence to the institutional HTA program established in the Lombardy Region (Reference Radaelli, Lettieri, Masella, Merlino, Strada and Tringali18); (ii) provides hospital decision makers with quantitative data to support their assessments, thus endorsing the diffusion of the MCDA philosophy; and (iii) crystallizes the rationale and the metrics for all dimensions and sub-dimensions of analysis.
MATERIAL AND METHODS
Research Scope
The present study, focusing on the wide implementation in the field of the IMPAQHTA framework, aimed at testing its performances for supporting HBHTA practices in terms of: (i) efficiency, (ii) feasibility, (iii) utility and completeness, considering the related dimensions and sub-dimensions. This “real world” experiment is part of a broader strategy in the Lombardy Region for the institutionalization of HBHTA practices, linked with the Regional HTA activities.
The framework leverages on the dimensions required by the generally accepted Core Model developed by the EUnetHTA Consortium (9). In addition, the IMPAQHTA proposes thirteen sub-dimensions (i.e., the criteria) that should be taken into account for an effective and efficient HBHTA practice. The framework balances completeness (i.e., the capability to cover all the EUnetHTA dimensions) and timeliness (i.e., the capability to deliver the assessments timely) as required in hospitals (Reference Lettieri, Masella and Nocco13). Furthermore, the framework suggests quantitative metrics for each of the sub-dimensions to provide decision makers with a quantitative and coherent synthesis, as required by EVIDEM (Reference Thokala and Duenas11;Reference Thokala, Devlin and Marsh12) and in the use of Multi-Criteria Decision Analysis (MCDA).
Table 1 summarizes its dimensions, sub-dimensions and quantitative metrics.
IMPAQHTA Dimensions and Sub-dimensions
Research Design
The testing aimed at verifying the capability of the framework to provide hospital decision makers with relevant information regarding the technologies under assessment against severe time constraints. In this regard, a rich empirical experiment was designed for the collection of data about several technologies (e.g., drugs, large versus small size equipment, clinical procedures, clinical pathways, ICTs, and medical devices) at a different level of their maturity (e.g., innovative versus consolidated), assessed by different types of HBHTA teams (e.g., mono-disciplinary versus multi-disciplinary; mono-dimensional versus multi-dimensional), within different hospital settings (e.g., small versus large, rural versus city, teaching versus nonteaching, private versus public). This heterogeneity of situations offered the opportunity to verify the generalizability of the IMPAQHTA framework. Further details regarding the empirical testing are provided in the Results section.
All the professionals within the HBHTA teams attended a dedicated 12-day pretraining course aimed at providing them with an in-depth understanding of technology assessment methodology at both the national/regional and organizational/hospital level, in terms of goals, technical approaches, and international experiences, as well as the dimensions and metrics proposed by the IMPAQHTA. In particular, 9 days were dedicated to the detailed explanation of the EUnetHTA Core Model dimensions. Theoretical and practical sessions were designed to evaluate the understanding on these topics, including written exercises. Any professional attaining unsatisfactory results was required to retake the evaluation before being involved in the empirical testing regarding the production of the final HBHTA report.
Data Collection
The empirical testing was conducted over a 3-year period (2012–2014). Data were collected from multiple sources.
First, all the professionals completed an ad-hoc questionnaire, administered through the “LimeSurvey” online tool, aimed at collecting information regarding the technology assessment exercise, the performance of their team and their general level of satisfaction with the IMPAQHTA framework. Second, a representative from each HBHTA team was selected for a semi-structured interview, lasting an average of 90 minutes, aimed at gathering a better understanding of the “pros” and “cons” of the IMPAQHTA framework. Third, it was possible to gain access to the HBHTA reports delivered by the teams, the evidence on which these reports were based, and the presentations delivered to the hospital Board by the HBHTA teams.
Data Analysis
Data were first described using means and frequencies. For each HBHTA exercise, data were stratified according to: (i) professionals involved, (ii) technologies under assessment, (iii) hospital setting in which the HBHTA evaluation was conducted.
The main performances of the IMPAQHTA framework were analyzed with respect to: (i) its utility, feasibility and completeness; (ii) feasibility of the dimensions and sub-dimensions; (iii) efficiency, in terms of time spent to implement the framework and produce an HBHTA report; and (iv) replicability in every hospital setting, for the evaluation of different healthcare technologies.
RESULTS
After a detailed description of the sample, results are presented as follows: (i) main results in terms of utility, feasibility, and completeness of the IMPAQHTA framework, dimensions, and sub-dimensions; and (ii) main results in terms of efficiency, feasibility, and overall quality of the IMPAQHTA framework.
Description of the Empirical Sample
The IMPAQHTA framework was empirically tested over a 3-year period (2012–2014) within twenty-two hospitals in Northern Italy, for assessing thirty-two different healthcare technologies and involved seventy-four professionals organized into HBHTA teams. The HBHTA teams were composed of three professionals on average: 66 percent of teams were composed of more than a single individual, each from a different specialty, while the remaining 34 percent was composed of a single person, acting as an “Ambassador” (Reference Cicchetti, Manca and Ruggieri19), demonstrating the need for professionals who are able, or trained, to conciliate and/or coordinate different perspectives in the assessment. The majority (66 percent) of professionals was physicians or healthcare professionals (nurses and technicians).
The HBHTA teams assessed different healthcare technologies: drugs (19 percent), large-size equipment (19 percent), clinical procedures (16 percent), ICTs (16 percent), and clinical pathways (13 percent). These technologies were innovative (56 percent), as well as mature/consolidated (44 percent).
The HBHTA teams were affiliated to hospitals (50 percent), Local Health Agencies (18 percent), and Scientific Institutes (18 percent). The empirical sample included: 73 percent of healthcare organizations with more than 400 beds (versus 27 percent with less than 400 beds), with a multispecialty case-mix (59 percent versus 41 percent mono-specialty organizations). The assessment was conducted in public (86 percent versus 14 percent private ownership) and urban settings (55 percent versus 45 percent in rural areas).
More details are shown in Table 2.
Sample Description
Perceptions of Professionals Concerning the IMPAQHTA Framework, in Terms of Its Dimensions and Related Sub-dimensions
The results presented in this section are based on the self-reported perceptions of the seventy-four professionals involved as evaluators in the HBHTA teams, concerning completeness, utility and feasibility. Their judgement was reported using a 7-item Likert scale (1 = low, 7 = high) regarding the utility (e.g., the relevance) and the complexity (e.g. the difficulty to cope with their assessment) of the proposed eight dimensions. In addition, they stated their perception regarding the easiness and the completeness of quantitative metrics proposed for the various dimensions. The results are shown in Table 3.
Healthcare Professionals’ Perception Concerning the IMPAQHTA Dimensions and Sub-dimensions
Note. Results are expressed by mean values according to a 7-item Likert scale (1 = low and 7 = high).
The eight dimensions were perceived as relevant to produce an exhaustive HBHTA report to inform decision making within hospitals (scores were always higher than 5.5 over 7 with the notable exception of the “equity” dimension). The professionals declared that they did not experience particular difficulties in evaluating the eight dimensions, remarking a difference in the “legal, social and ethical impact” whose average score was 3.61 over 7.
Taking into account the thirteen sub-dimensions, the professionals were positive regarding the easiness and completeness of the proposed quantitative metrics, whose scores on easiness were higher than 4.53 (this score was for “effectiveness data retrieval”) and on completeness were higher than 5.31 (this score was for “social and ethical impact”).
The professionals’ perceptions in terms of completeness, utility, and feasibility were compared with respect to: (a) the professionals’ background, (b) the technology under assessment, and (c) the hospital setting where the HBHTA exercise was performed, as shown in Table 4. No significant differences emerged with respect to the professional background and the nature of the technology under assessment.
Healthcare Professionals’ Perception Concerning the IMPAQHTA Completeness, Utility, and Feasibility
Note. Results are expressed by mean values according to a 7-item Likert scale (1 = low and 7 = high).
With regard to hospital settings, the data show that teaching hospitals and scientific institutes appreciated more the IMPAQHTA framework in terms of completeness (6.38 versus 5.92), utility (6.38 versus 5.61) and feasibility (6.38 versus 5.40).
Other Self-reported Performance Measures
The IMPAQHTA framework was tested with respect to its efficiency. In this regard, the HBHTA teams reported that they spent an average of 85 hours (e.g., 11 working days) to collect past evidence regarding the technology and the comparator, to draft the HBHTA report and to synthetize their assessment in a concise, quantitative index, in accordance with the MCDA approach.
Within this timeframe, all the HBHTA teams performed a complete HBHTA evaluation covering the eight dimensions proposed by the IMPAQHTA framework. Considering the sub-dimensions, the HBHTA teams found them feasible, covering on average twelve sub-dimensions: all the thirteen sub-dimensions were examined in 59 percent of cases and eleven sub-dimensions were examined in 16 percent of cases. When a sub-dimension was not included in the HBHTA reports, the reason was that this sub-dimension did not apply to the technology under assessment (e.g., considering the introduction of an innovative drug, the “effectiveness” dimension was not always evaluated due to the lack of “real world” data deriving from observational study or empirical testing).
The delivered HBHTA reports showed an acceptable level of evidence: in fact, the “general relevance” dimension was supported by an average of 6 references from past studies, the “efficacy” dimension by an average of 5.4 references and the “safety” dimension by an average of 2.8 references.
The organizational, equity, and social impact dimensions were supported by collecting the opinions of an average of five senior healthcare professionals through a structured questionnaire.
DISCUSSION
The IMPAQHTA framework derived from the deploying at the hospital level of the HTA program that the Lombardy Region activated at the Regional level in 2008 (Reference Thokala, Devlin and Marsh12;Reference Radaelli, Lettieri, Masella, Merlino, Strada and Tringali18). Translating the framework from the Region to the hospitals required taking into account the difficulties hospitals face in adopting HTA-based practices for decision making, in particular the severe constraints hospitals have in terms of time available for the assessment and human resources needed. In addition, hospital decision makers require HBHTA reports that provide evidence regarding the organizational impacts due to the adoption of new technologies/routines (Reference Lettieri, Masella and Nocco13).
According to these needs, the present study reports the main results achieved by the IMPAQHTA framework in a large-scale empirical implementation in Northern Italy. Results were satisfactory in terms of feasibility, that could in the future pave the way for the adoption of this HBHTA framework as reference practice within any Italian hospitals. During the 3-year empirical experiment, thirty-two different technologies were assessed by the HBHTA teams that analyzed the eight dimensions defined by the Lombardy regional HTA program. This was facilitated by the crystallization of thirteen sub-dimensions measured through quantitative methods that make clear to all evaluators “where” and “how” to focus their efforts, with the result of a “quick” but complete HBHTA. The completeness of the HBHTA reports is encouraging for the future: the well-established Danish Mini HTA check-list, although composed only of four dimensions and twenty-six questions, never produced complete HBHTA reports (Reference Kidholm, Ehlers, Korsbek, Kjaerby and Beck20).
The recent lessons learned from the 10-year Argentinian experience (Reference Demirdjian21) with eighteen “quick and dirty” HBHTA reports show that hospitals need to focus their HTA practices on a limited list of necessary dimensions of assessment as the time constraints they have to face, do not allow the implementation of highly sophisticated, and holistic, frameworks. The IMPAQHTA framework allowed for the production of relevant, multidimensional reports within a limited period of time, according to what hospitals need to take informed choices (Reference McGregor22).
In this regard, it could be useful to discuss the results with respect to the recent evidence emerging from the European “AdHopHTA” project (Reference Sampietro-Colom, Lach and Cicchetti17). Both projects used the EUnetHTA Core Model (9) as a “backbone” for designing the framework and deploying the multidimensional assessment. This confirms the establishment of the EUNetHTA Core Model as a valuable and reliable reference for both HTA and HBHTA in Europe. In this regard, the IMPAQHTA and the AdHopHTA sub-dimensions are consistent with the information hospital managers in Europe need within their decision-making processes, as reported in literature (Reference Ølholm, Kidholm, Birk-Olsen and Christensen23;Reference Kidholm, Ølholm and Birk-Olsen24).
Furthermore, it could be interesting to report how the two frameworks deal with political and strategic aspects that past studies (e.g., Reference Thokala, Devlin and Marsh12;Reference Lettieri and Masella25) argued relevant for hospitals. While the AdHopHTA project proposes the creation of a new dimension to take these topics into account, the IMPAQHTA framework includes them in the “safety” dimension, claiming that in hospitals the concept of “safety” is not limited to adverse events or deaths due to the innovation of care delivery, but should also include the consistency with local policies and hospitals guidelines/protocols/procedures. Similar differences may be found with the comparison of the AdHopHTA “clinical effectiveness” versus the IMPAQHTA “general relevance,” regarding the sub-dimension “characteristics of evidence.” As result, it emerges that the two frameworks are grounded on the same sub-dimensions, even though aggregated in different dimensions, leading to a complete assessment.
Despite the similarities mentioned above, one main difference emerges. While the AdHopHTA framework suggests qualitative methods to assess the ten dimensions, without proposing specific metrics for them, the IMPAQHTA framework(being designed for allowing MCDA approaches) endorses a more quantitative approach to provide hospital decision makers with a quantitative synthesis of the assessment, thus facilitating the discussion regarding the impacts on the different dimensions and the convergence toward an agreed decision (Reference Lettieri and Masella25). An example is the “ethical, safety, social, and legal aspects” that, as claimed by the AdHopHTA project are not always included in the assessment (Reference Ølholm, Kidholm, Birk-Olsen and Christensen23), have been operationalized within the IMPAQHTA framework to facilitate their inclusion in the HBHTA implementation.
The proposed HBHTA framework, however, has some limitations. The sample was composed of professionals, all of them volunteers, with specific motivation to learn how to assess a technology within the healthcare sector; this could limit the replicability of the experiment, even if it does not weaken the effectiveness and the efficiency of the IMPAQHTA model. Therefore, training courses are recommended to ensure a specific HTA background and a correct implementation of the proposed framework.
As the organizational impact is currently based on the perception of each professional interviewed regarding their own work setting, an objective sub-dimension for a quantitative evaluation of the change management process could overcome this possible limitation.
CONCLUSION
The present study offers new insights, that advance the ongoing debate regarding the importance and feasibility of technology assessment in hospitals, as they are at the forefront of technological innovations and are thus the first required to take decisions and actions for the adoption of new healthcare technologies. However, despite the relevance of these choices, hospitals have less time, knowledge, and resources compared with HTA Agencies to carry out these assessments.
Due to the lack of HBHTA practices that produce relevant and timely reports to support an evidence-based and multi-dimensional choice, the IMPAQHTA framework has narrowed the previously mentioned gap and has shown that it could support the implementation of technology assessment practices within hospitals. Results, in terms of perceived relevance and easiness of use, as well as efforts required are promising across different HBHTA teams, different healthcare technologies, and different hospital settings. This supports the argument that the IMPAQHTA framework could be considered as a valid alternative for current HBHTA practices, within hospitals, thus leading to the coverage of all the EUnetHTA Core Model dimensions, and avoiding the need to select only few domains due to time and resources constraints.
The adoption of the proposed framework within every hospital setting would facilitate a rapid and evidence-based assessment, supporting top management with quantitative indicators useful for the determination of the comparative value of technologies. To make this possible, the introduction of ad hoc training programs is, therefore, highly recommended, to develop healthcare professionals’ skills.
Finally, it could be an interesting topic for future researches to study whether and how team effectiveness, or other organizational factors, interact with the IMPAQHTA framework, as well as the creation of appropriate criteria to allow clinicians to communicate better with management, thus facilitating the evaluation and the immediate introduction of innovative technologies within healthcare organizations.
CONFLICTS OF INTEREST
The authors have nothing to disclose.
