The present study describes the health technology assessment (HTA) framework implemented in 2008 by Regione Lombardia (RL) to decide on the introduction and delisting of health technologies. The framework (named VTS, Valutazione delle Tecnologie Sanitarie) applies the EUnetHTA Core Model® (1;Reference Lampe, Makela and Garrido2) to define the procedures, responsibilities, and criteria that the Health General Directorate follows for evidence-based and legitimate decisions.
The introduction of a formal HTA framework stemmed from a strive to balance goals of continuous innovation with needs of steady cost containments. While introducing state-of-art technologies is crucial for higher standards of care, their expenditures are critical factors for the increase of healthcare costs (Reference Chernew, Sabik, Chandra and Newhouse3–Reference Chandra and Skineer5). Policy makers have thus emphasized the introduction of tools that manage which, why, and how technologies get in or out the system (Reference Busse, Orvain and Velasco6).
RL targeted these concerns by using the EUnetHTA Core Model to legitimize a structured approach in the adoption and delisting of technologies. Specifically, the Core Model is not used as a generic guideline for HTA, but formalized in three regional laws that compel RL into following a standard procedure.
The Core Model in the VTS framework supports the entire decision-making process of RL, i.e. prioritizing technology requests; assessing the impacts of the technology; appraising the decision to invest in the technology (Reference Busse, Orvain and Velasco6). Two changes were introduced to fulfill this purpose. First, the framework makes explicit the implementation of a cost-opportunity logic according to which the adoption of new technologies must be balanced by the delisting of obsolete ones. Second, the framework incorporates tools and techniques for Multi-Criteria Decision Analysis (MCDA) (Reference Goetghebeur, Wagner and Khoury7). The Core Model concentrates on data collection and technology assessment, but does not endorse explicit tools to translate the latter into decisions. MCDA served this purpose, and its inclusion required a few changes to the Core Model, that is, (i) map EUnetHTA domains into dimensions that RL set up to legitimize the prioritization of technologies and (ii) include criteria from an open source framework (EVIDEM) that could best support the systematic appraisal of the assessment report into a final decision (Reference Goetghebeur, Wagner and Khoury7). Both “dimensions” and “criteria” are consistent with the Core Model as they do not distort its logic, structure, and contents.
The present study will describe the VTS framework in all its components. The study is structured as follows. In section one, we outline the context of RL, in which the framework takes place. Then, we describe the key principle of cost-opportunity that grounds the framework. Next, we outline the elements composing the VTS architecture, and the process of its functioning. Last, we discuss current experiences with framework implementation.
CONTEXT OF APPLICATION
Under the Italian Constitution, the primary responsibility for Healthcare is granted to Regions, which are autonomous in the organization of services and can independently choose the nature and quantity of technological investments (Reference France and Taroni8). The regional nature of the Italian Healthcare System leads to the lack of a national agency that could mandate and control the implementation of a shared HTA framework. Responsibility to assert HTA frameworks is delegated to each Region, with the result of having twenty-one distinct HTA applications in Italy (Reference Favaretti, Cicchetti, Guarrera, Marchetti and Ricciardi9).
RL represents one of the leading systems in terms of technology investments and quality of care. RL includes almost 10 million residents and 30,000 beds; and employs 150,000 healthcare professionals, 8,000 general practitioners. RL is organized as a quasi-market system (Reference France and Taroni8), with a neat distinction between purchasers (fifteen Local Health Authorities) and providers (twenty-nine hospitals, four research institutes, and almost 200 other healthcare organizations). Public healthcare-related expenditures amass to 17,000 mil € in 2012, which represent the vast majority of RL budget. The need to monitor local expenditures on the budget triggered the formalization of a HTA framework to decide over the introduction/delisting of technologies.
COST-OPPORTUNITY AND THE VTS FRAMEWORK
The VTS framework aims at supporting the continuous refurbishment of technologies under the constraints of fixed budgets. Such constraints magnify the role of opportunity costs associated with new investments, because the resources spent in a new technology might prevent investments somewhere else. RL conceived the VTS framework as a structured tool to identify obsolete technologies whose disinvestments could clear financial space for new investments. The VTS framework is thus structured to provide knowledge on (i) the comprehensive improvements associated with new technologies, vis-à-vis established ones; (ii) the possibility and extent of substituting obsolete technologies with the new ones.
Traditional cost-effectiveness analyses (CEA) fall short in both regards. The attention on costs and effectiveness is significantly less informative for policy makers than HTA methods that assume a multidimensional perspective. Also, CEA stimulate the accumulation, rather than the substitution, of technologies because they provide a rational for investment, but not for disinvestment (Reference Donaldson, Currie and Mitton10–Reference Brouwer, van Exel, Baker and Donaldson12).
The VTS framework adopts instead a principle of cost-opportunity which considers how new technologies substitute established technologies and, thus, push disinvestments. The VTS framework incorporates this principle by explicitly requiring proponents of new technologies to indicate which set of established technologies would become obsolete after the investment and, thus, candidates for disinvestment. This approach is applied at its best when a new technology replaces established ones. In this case, resources necessary to cover the new investment can be found from the disinvestment. The approach can be also applied when the new technology does not replace, but adds, to established ones. In this case, while no obsolete technology is fully disinvested, it is still possible to identify activities, processes, and resources that would become redundant or unnecessary with the new technology. Removing them, it is then possible to reduce the budget costs necessary for the new investment.
ELEMENTS OF THE VTS FRAMEWORK
The cost-opportunity analysis is performed through a structured model for (i) the collection and assessment of scientific evidence about technology impacts and (ii) the appraisal of results in ways that can be promptly communicated to and understood by decision makers.
VTS Framework Combines EUnetHTA and EVIDEM Models for These Purposes
The EUnetHTA Core Model represents an important European reference for technology assessment because it is the result of joint work from multiple agencies (1). It delineates a nested structure of issues, topics, and domains that organizes the collection and assessment of evidence. At top level, the model outlines nine domains (Table 1). Each domain represents “a wide framework within which technology is considered. It provides an angle of viewing the use, consequences and implications of technology” (EUnetHTA, 2011). Domains reflect the fact that data for technology impacts should be managed by experts in different disciplinary areas. Each domain is populated with several topics, representing the specific areas in which different impacts can be expected. For instance, the domain “clinical effectiveness” includes topics such as mortality, morbidity, and quality of life. Each topic includes several issues which represent the questions that experts should answer to assess technology impacts. For instance, to assess the impact on quality of life, an expert should answer to issue D0012: “What is the effect of the technology on health-related quality of life?” and D0016: “How does the use of technology affect activities of daily living?”
Comparison between EUnetHTA Domains and VTS Dimensions (with One Illustrative Example of Operationalization)
The EUnetHTA Model was adopted with two changes. First, the nine EUnetHTA domains were re-elaborated into eight dimensions (cf. Table 1). The changes were made to discern the impacts of the novel technology—while domains focus on aspects of the technologies. The differences between the domain “organizational aspects” and the dimension “organizational impacts” (Table 1) can be illustrative. The domain includes issues that originate from organizational aspects, but might exert impacts elsewhere. The issue G0006—that is, economic investments to guarantee viable organizational premises—is moved in the dimension “economic impacts” because it exerts an impact on the costs spent for the technology.
Second, EVIDEM criteria are incorporated to support the appraisal stage. The Core Model does not define a structured approach to appraise the results of its assessment; EVIDEM does so by implementing the MCDA approach. The appraisal is structured in a set of criteria (cf. Table 2) which “reflect explicitly the thinking process underlying appraisal/prioritization of healthcare interventions” (13). The MCDA approach replicates decision-making process by: (i) considering together all relevant criteria that intervene in the decision, (ii) assigning a weight to each criteria that reflect how relevant that criterion is for decisions, (iii) recognizing how strongly a new technology makes improvement in each criterion. To our knowledge, EVIDEM provides the most advanced and used definition of criteria for appraisal (Reference Goetghebeur, Wagner and Khoury14;Reference Goetghebeur, Wagner and Khoury15), and was thus adopted as reference model. The VTS framework provides minor changes to EVIDEM criteria to maintain consistency with EUnetHTA issues (cf. Table 2).
Correspondence between EVIDEM and VTS Criteria
Note. Asterisks indicate implicit criteria.
The changes respect the four properties required to apply MCDA, that is, mutual independence, completeness, nonredundancy, and operationalizability of criteria. In particular, criteria are operationalized through EUnetHTA issues (cf. Table 3), and independence and nonredundancy between criteria are guaranteed by imposing that an issues could be included in only one criteria.
Illustrative Criteria in the VTS Framework
PROCESS OF VTS FRAMEWORK
The VTS framework is applied in a three-step process comprising (i) a prioritization of requests, grounded on a “quick and dirty” assessment limited to VTS dimensions; (ii) a full assessment of the prioritized technologies, provided by answering EUnetHTA-based issues; (iii) an appraisal of the assessed technologies, grounded on the analysis of multiple criteria. Each step is performed by a specific agent that is formally assigned by RL after getting clearance over conflict of interest. Figure 1 provides an overview of VTS elements, process, and actors.
Actors, processes, and components of the VTS (Valutazione delle Tecnologie Sanitarie) framework.
STEP 1: PRIORITIZATION
RL receives every year a large amount of requests from different stakeholders about different types of new health technologies (drugs, devices, equipment supplies, procedures, and organizational systems). The Regional Act 7856/2008 formalized who can ask for the assessment, and how. Requests can originate only from “accredited actors,” ranging from no profit (i.e., care-giving organizations included in the Regional Health Database, scientific and professional societies, patients associations, and universities) to for-profit organizations (i.e. pharmaceutical, biotech, diagnostic firms). Proponents must fill a form with information about: (i) their details, (ii) the current technologies that will be considered as comparators and candidates for disinvestment, (iii) evidence on the impacts of the new technology on the eight dimensions of the VTS framework. Incomplete documentation will meet a desk-rejection. Appropriate requests are prioritized to identify which technologies deserve full assessment and which should be assessed first.
Process of Prioritization
Requests are sent to the Regional Unit of Prioritization and Conflict of Interests (NVPCI from the Italian wording) which includes twenty-four members—three members from Regional Health Directorates; eight from Hospital Directorates; eight from Health Authority Directorates; five from Care Scientific Centers. The members are formally appointed in regional deliberations and serve for 3 years. Each prioritization involves eight members—two members from each typology of directorate.
The NVPCI gathers twice—at the beginning to define the object of investigation and consolidate the comparators that would be used as candidates for disinvestment and at the end to deliver the prioritization decision. The stage lasts normally 45 days. Most of their work can be done online, through a dedicated Web-platform. The NVPCI provides a first rapid assessment, according to the eight dimensions (cf. Table 1).
The prioritization follows two steps. First, the NVPCI checks the attached documentation and removes all requests for new technologies which (i) cause a significant deterioration in efficacy/effectiveness, (ii) endanger patient safety, (iii) are irrelevant for healthcare purposes. Later, The NVPCI applies a Multi-Dimension Decision Analysis (MDDA) which takes the following form: Rank (technology j) = Σi=1 8 [weight (dimension i) * point (dimension i, technology j)]. Moving from right to left in the equation, each health technology receives a 1–5 point for each of the eight dimension of the VTS framework. The score defines to which extent the novel technology improves the status quo in that particular dimension (i.e., 1 = high aggravation; 3 = no differential impact; 5 = high improvement).
Each dimension has a weight that tells how relevant it is compared with the other seven dimensions. The weight is elaborated by the NVPCI and is valid for any technology used within a specific care pathway. Each member defines to which extent the dimension was relevant for decision making, on a 1–8 scale (1 = irrelevant; 8 = essential). Individual ranks are then gathered and elaborated to define for each dimension a relative measure of importance (a weight).
The health technology under prioritization has specific points for each dimensions (i.e., how enriching the technology is for a specific dimension) and each dimension has a specific weight (i.e., how relevant the dimension is for prioritization). The product of these factors gives an overall rank that asserts how relevant the technology is for prioritization. Technologies can be then put in a ranking and divided into high, moderate, and low priorities, and move forward to full assessment. Technologies that have a rank <2 (i.e., low improvement) are rejected.
The decision is then communicated to all interested stakeholders—who are given: (i) an executive summary of the prioritization, which explains the decision; (ii) an explanation of the process followed to produce the prioritization decision; and (iii) the weight, points, and comments of the NVPCI—with contributors properly anonymized. Proponents of the technology are given an opportunity to appeal against a rejection decision.
STEP 2: FULL ASSESSMENT
Health technologies that pass prioritization proceed to the “full assessment” stage. Evidence about the multidimensional impacts of the technology is systematically collected and processed by a panel of experts identified by NVPCI. Specifically, the NVPCI assigns experts to one of the eight dimensions of the framework and have no conflicts of interest.
The panel gathers on average four times for each assessment—at the beginning to gather initial information about the goal of the assessment, typically two more times to discuss critical issues in the assessment and at the end of a process to consolidate the assessment. The process lasts on average 3/4 months.
Within the panel, one actor is appointed as the conductor and given the responsibility to assign specific criteria to each expert and synthesize their contributions. The panel of experts is responsible for elaborating knowledge on each of the twenty-one criteria of the framework, by collecting evidence and answering to the relevant issues in each criterion.
Review and Grading of Evidence
The starting point in full assessment is the systematic review of scientific evidence. The experts are required to search for contributions in databases of primary studies (e.g., MEDLINE, Embase, Cochrane Controlled Trials Registry, Clinicaltrias.gov) and of secondary studies (e.g., Cochrane Reviews Database, Web sites of HTA Agencies (e.g., ANSM, CADTH, DHMA, DIMDI, HAS, IQWIG, NICE, Osteba, etc.), specifying the research strategy—that is, keywords and database queries—and the criteria for inclusion and exclusion of contributions.
The resulting publications are registered according to the typology of study design—meta-analyses, randomized controlled clinical trials, observational studies, etc. The classification is the first step in the grading of evidence. Relevant included contributions are graded on a 1–4 scale according to the degree of scientific validity and applicability to Lombardy of their findings. In practice, the panel of experts assesses the validity and applicability of evidence using distinct assessment models for: (i) epidemiological documentation; (ii) clinical outcomes; (iii) economic/organizational documentation; (iv) budget impact analysis.
Assessment of Impacts
The collected evidence is used to address the issues in each criterion. As we observed, issues represent in fact the basic questions that experts must address to inform the assessment. The inclusion of issues in criteria and dimensions allows breaking down the technology assessment into smaller units that can be assigned separately to field experts. All the answers to the issues provided by the experts are included in a shared document, called Multidimensional Impacts Form (MIF). The contents of the MIF document are processed and later synthesized into a comprehensive assessment of technology impacts. The MIF represents the primary input of the next stage of appraisal.
STEP 3: APPRAISAL
The appraisal is performed by the Regional Technical Roundtable for Medical Appropriateness (TTRAM from the Italian wording). The TTRAM includes forty members selected from curricula by the Regional Health Directorate; from Hospital Directorates; Health Authority Directorates; Care Scientific Centers, Professional Associations and Universities. The members are formally appointed in a regional deliberation and serve for three years. Each appraisal “uses” twenty members.
The TTRAM gathers twice—at the beginning and at the end of a process that normally lasts 45 days. Most of their work can be done online, through a Web-platform. The TTRAM appraises the results from the assessment by using EVIDEM-based criteria in the MCDA approach.
The appraisal follows two steps: (i) the definition of a quantitative score for fifteen explicit criteria, (ii) the definition of qualitative comments for six implicit criteria. Consistently with EVIDEM, in fact, the VTS framework distinguishes between fifteen explicit and six implicit criteria (cf. Table 3).
Criteria are deemed explicit if they can be measured using approach and indicators that are independent from the specific technology, context of application or the time of decision. The fifteen explicit criteria can thus be treated in a linear addictive equation, that is: Score (technology j) = Σk=1 8 [weight (explicit criterion k) * point (criterion k, technology j)].
Each explicit criterion is first weighted on a 1–15 scale (1 = criterion k does not contribute to value; 15 = criterion is the most important one for value). Then, based on the assessment included in the MIF, each member of the TTRAM assigns as EVIDEM a 0–3 point to each criterion that reflects how significant s/he considers the improvement made by the new technology. The individual points are then aggregated, and the regional board finally applies the Multiple Criteria Decision Analysis (MCDA) to quantify the multidimensional impact of the novel technology.
The linear addictive model cannot include the six implicit criteria. These criteria are deemed implicit because no “universal” score can be assigned to them. Their evaluation is entangled with an evaluation of the setting of application and moment of adoption. The criterion “Historical and Political Context” is illustrative, in this regard, as the consistency of the new technology with the context depends on where, when and by whom the appraisal is performed.
Each implicit criterion can add important clues to the overall appraisal, but their inclusion in the linear addictive equation would make the MCDA spurious. The appraisal of implicit criteria is, in fact, “subjective” and contextual while the MCDA manifests “objective,” evidence-based impacts of the technology. Implicit criteria are then elaborated with a qualitative assessment of whether the technology would produce a positive, negative, or nonsignificant impact.
The appraisal thus results in a quantitative result—that is, the score for fifteen criteria—and six qualitative evaluations that the regional policy maker will use to substantiate and legitimize the final decision and to communicate it to the public, the industry, the providers and any relevant stakeholder.
At the end, all information produced in the assessment and appraisal stages are made public, that is, (i) the contents of the MIF, (ii) the scores and comments made by the TTRAM (properly anonymized), (iii) an executive summary of the final appraisal decision. Proponents of the new technology are given the opportunity to appeal against a negative decision.
STATE OF PRACTICE
The application of the VTS framework started at the end of 2011. As we write, almost 2 years passed from its inception and twenty-six technologies have been signaled: seven diagnostic devices, eight interventional devices, two vascular procedures, six categories of drugs, one imaging technology, and two special food formulations.
Thus far, fourteen proposals were rejected at step 1 (prioritization) because their effectiveness could not be sufficiently documented or was judged inferior to available alternatives, while two proposals are currently awaiting prioritization. Nine of the remaining ten proposals have been admitted to step 2 (full assessment): three assessments proceeded to step 3 (appraisal, one finished and two ongoing) while six assessments are ongoing. Another proposal was judged clearly superior to alternatives at priority phase and proceeded directly to quick appraisal, which ended positively.
At present stage, no relevant issue has emerged in neither the prioritization stage, for instance, priority rejections have been accepted as legitimate, nor the assessment stage, for instance, the groups of experts are working steadily in data collection, issue assessment, and criteria weighting.
DISCUSSION
Regione Lombardia (RL) sought to institutionalize HTA methods and pragmatics through: (i) high standardization of activities, actors, criteria involved in the process; (ii) reliance on normative regulation to compel actors into performing VTS standards; (iii) central role assigned to scientific evidence.
RL had to deliver a framework that was both an effective decision-support system and a powerful agent of change in stakeholders’ behaviors. Facing stakeholders that were reluctant, unaccustomed, or unaware of HTA (Reference Lettieri, Masella and Nocco16), RL designed interventions—for example, the mobilization of regulatory support, conferring status to NVPCI and TTRAM—to legitimize HTA as a new routine.
In this regard, a crucial intervention was the application of the EUnetHTA Core Model in a normative process that includes prioritization, assessment, and appraisal of requests. The operationalization of the Core Model occurred through (i) a full adoption of its issues and topics for data collection and impact assessment and (ii) the inclusion of EVIDEM criteria and MCDA for the appraisal of requests. The integration between EUnetHTA and EVIDEM models is unproblematic, as the former provides elements for collecting data and assessing impacts while the latter instruments for supporting decisions. Moreover, the integration ensured the implementation of the Core Model in RL, at the expense of a few changes such as the redefinition of EUnetHTA domains in dimensions and the repositioning of topics and issues in a different, goal-oriented, structure of criteria, and dimensions.
This approach has both strengths and drawbacks. On the positive side, the standardization and explicit regulation of the HTA process instill uniformity and transparency in a process that may be highly elusive. While subjectivity cannot be completely removed, the framework seeks to minimize discretion in decision making and to produce decisions perceived as legitimate by the stakeholders. The adoption of EUnetHTA and EVIDEM models, the commitment to systematic reviews and the reliance on field experts all manifest an intention to legitimize a rigorous decision making. On the negative side, the VTS framework unfolds a complex process whose standards and requirements might limit flexibility, adaptability, and timeliness. Time is, at present, the key issue under consideration. The process now lasts 6/8 months from prioritization to the final appraisal and significant time containments have been achieved by a recent introduction of a Web-platform that allows members to collaborate on remote without frequently meeting face-to-face; and by focusing on technologies which already have advanced evidence bases. RL is presently seeking to move a step further and couple situations in which RL needs to perform the whole process of prioritization, assessment and appraisal (acting as “HTA-Doer”) with others in which it can use pre-existing assessments and focus only on the appraisal of results (thus acting as “HTA User”). The use of external assessments (properly validated and adapted) thus represents the second stage in the institutional change of HTA for RL—with the expectation of ramping-up the production of evidence-based decisions to a steady high level.
CONCLUSIONS
The VTS framework was designed in RL as an application of the EUnetHTA Core Model to inform investment/disinvestment decisions. The Core Model is adopted unabridged in the topics and issues that are meant for data collection and technology assessment. The VTS framework provides however changes in those components – dimensions and criteria—that are meant for prioritization and appraisal. These changes move in addition to, not in substitution of, the Core Model. They serve the purpose, in fact, to address those areas that are not explicitly covered by EUnetHTA and allow the direct use of the Core Model also for decision making. Changes leave unaltered the core aspects of the model, such as breaking down the assessment into small, standardized and replicable units, the nature of collected data and the process of full assessment.
The adoption of EUnetHTA issues into a structure of criteria can suggest that the Core Model can be used not just for assessment purposes, but also to support directly the appraisal. The VTS framework hints to a promising implementation of the EUnetHTA assessment tools with the EVIDEM multiple criteria appraisal methods, without compromising their respective objectives and approaches.
CONTACT INFORMATION
Giovanni Radaelli, PhD (giovanni.radaelli@polimi.it), Postdoc Researcher, Department of Management, Economics and Industrial Engineering, Politecnico di Milano, Milano, Italy
Emanuele Lettieri, PhD, Research Fellow, Department of Management, Economics and Industrial Engineering, Politecnico di Milano, Milano, Italy
Cristina Masella, MSc, Professor, Department of Management, Economics and Industrial Engineering, Politecnico di Milano, Milano, Italy
Luca Merlino, MB, BS, Deputy Director, General Health Directorate, Regione Lombardia, Milano, Italy
Alberto Strada, MSc, Junior Officer, General Health Directorate, Regione Lombardia, Milano, Italy
Michele Tringali, MB, BS, Deputy Officer of HTA Program, General Health Directorate, Regione Lombardia, Milano, Italy
CONFLICTS OF INTEREST
Michele Tringali serves in the Board of Directors of the EVIDEM Collaboration, a NGO non-profit organization, without receiving funding for the work. The other authors report they have no potential conflicts of interest.
