The use of health technology assessment (HTA) to inform best practice or reimbursement and coverage decisions has increased rapidly over the past three decades (Reference Banta1). As the use has expanded internationally, there has been increasing interest in exploring the potential for harmonizing HTA (Reference Hutton, Trueman and Facey11). Harmonization of HTA can refer to three different elements: (i) harmonization of approaches and processes; (ii) harmonization of methods and evidence requirements; (iii) harmonization of decisions.
Significant advances have been made in harmonizing many of the methods involved in HTA. Best practice guidelines and consensus papers have been published on many elements of HTA, including the conduct of systematic reviews of clinical research (4), economic modeling methodologies (Reference Weinstein, O'Brien and Hornberger16) and the reporting of HTA findings (Reference Busse, Orvain and Velasco3).
Less attention has been paid to developing harmonized approaches to health technology assessment or harmonization of decisions. This can be partly attributed to politics; HTA and decisions on the reimbursement and coverage of new technologies which it often informs remain under the autonomy of individual countries who wish to develop or maintain their own processes, with minimal influence from outside bodies.
Several international bodies, many with a remit for promoting the adoption of HTA, have sought to explore whether it is possible to develop a set of common principles and methodologies for adoption by international HTA bodies (Reference Cranovsky, Matillion and Banta6;Reference Jonsson, Banta, Henshall and Sampietro-Colom12). The most recent significant initiative is the EUnetHTA program (10). EUnetHTA is a pan-European initiative, established in 2006, which has sought to explore the possibilities for harmonization across European HTA agencies. As part of this process, EUnetHTA has developed a framework for health technology assessment, specifically for medical and surgical interventions, along with an illustrative case study of how this could be put into practice (9). The choice of case study for surgical interventions was drug eluting stents (DES) (5), a technology which has been considered by multiple HTA bodies across Europe. The EUnetHTA approach is expected to provide a “core HTA” that can act as a basis for individual country level HTAs with minimal adaptation.
This study considers the degree to which harmonization of HTAs across jurisdictions is possible. To consider this, we have conducted a post hoc analysis of published HTAs of DES from several countries to determine the degree to which the HTAs converge in their approaches, evidence considered, and resulting recommendations.
METHODS
Selection of HTAs
Four health technology assessments of drug eluting stents were considered for inclusion in the study. The HTAs were selected on the basis that they were all undertaken after 2006. The rationale for this being that evidence emerged in 2005 which cast doubt over the long-term safety of DES, specifically relating to the potential increased risk of late stent thrombosis after implantation (Reference Ong, McFadden and Regar14). Whereas safety concerns might have a direct impact on the findings of HTA, the concerns relating to drug eluting stents also impacted on their cost-effectiveness, as they had implications for co-medication in patients receiving DES. HTAs of DES published before this time were excluded on the grounds that these safety concerns would not have been taken into account and, therefore, would not be directly comparable with HTAs after this time.
Health technology assessments from the following bodies were considered: (i) The Liverpool Reviews and Implementation Group (LRIG) commissioned by the National Institute for Clinical Excellence (NICE), England and Wales (Reference Hill, Boland and Dickson8); (ii) The Belgian Health Care Knowledge Centre (KCE), Belgium (Reference Vinck, Neyt and Thiry15); (iii) The Ludwig Boltzmann Institute (LBI), Austria (Reference Kvas13); and (iv) Programme for Assessment of Technology in Health (PATH), Ontario, Canada (Reference Bowen, Hopkins and He2).
Information was derived from the recommendations on drug eluting stents, published by each of the HTA bodies, as well as any relevant documentation on the assessment that informed the development of the recommendations. For example, whereas NICE publishes guidance on new technologies, the health technology assessments used to inform this process are conducted by independent academic bodies. Where the full assessments were accessible, as well as the recommendations, then these were reviewed.
Evaluation
The HTAs were assessed using a framework developed for the purposes of this study. The framework was developed to systematically capture information on the process adopted, the evidence considered, and the recommendations of each HTA. Information captured in the framework included the following: (i) the purpose of the HTA, where explicitly stated; (ii) the number of clinical studies included in the HTA and their type (e.g., randomized controlled trial [RCT], non-RCT); (iii) other sources of clinical evidence considered (e.g., observational studies, published registries, and so on); (iv) clinical end points considered; (v) the number of economic studies considered and their type (e.g. cost-effectiveness analysis, cost utility analysis, and so on); (vi) economic end points considered; (vii) the recommendations on use; and (viii) estimates of the budget impact of the recommendations, where available.
The assessment framework allowed each aspect of the HTAs considered to be easily compared and areas of agreement and disagreement to be readily identified.
RESULTS
Approaches Adopted
Some of the HTAs considered were intended to inform resource allocation decisions. For example, NICE produces guidance on the how technologies should be used in the English health service and recommendations from PATH are intended to inform coverage decisions taken by the Ontario Ministry of Health and Long Term Care. Analyses produced by LBI and KCE are expected to inform best practice but have no direct link to reimbursement and coverage.
Of the agencies considered, some conduct or commission primary evidence generation as part of the health technology assessment, whereas others rely largely on analysis of published evidence. For example, NICE commissioned an academic unit at the University of Liverpool to assess the published evidence on drug eluting stents and also generate primary research on their cost-effectiveness. In so doing, the academic group undertook further analyses of previously unpublished data on the clinical effectiveness of drug eluting stents to help inform their assessment.
Similarly, the PATH assessment, commissioned by the Ontario Health Technology Assessment Committee, involved both analysis of published evidence and the development of new evidence through field evaluations of drug eluting stents. This additional evidence was critical to informing the recommendations on the appropriate use of this technology in Ontario.
KCE and LBI considered published evidence on drug eluting stents but made no attempt to generate primary research as part of their assessment. KCE adopted a fairly rigorous approach to identifying relevant studies on drug eluting stents, considering data from local registries where available. LBI appeared to adopt a less systematic approach to identifying evidence and focused mainly on recently published studies of drug eluting stents, particularly meta-analyses of published trials, although search strategies were not reported. Little attempt was made to identify additional data sources or undertake any extrapolation of the published evidence.
Evidence Considered
Clinical Trials. All the assessments included in the analysis considered the clinical effectiveness of drug eluting stents (DES) in comparison with bare metal stents (BMS). All the assessments, with the exception of LBI, included reviews of evidence from individual trials as well as meta-analyses, combining the findings of multiple trials. The LBI assessment was reliant on a recently published meta-analysis of the evidence.
A summary of the evidence considered is reported in Table 1. KCE considered the greatest number of RCTs, with a total of twenty-four trials being reviewed. Thirty-three RCTs comparing DES and BMS were considered in total across all the assessments considered, while thirteen trials were considered by more than one assessment group. A “core” set of seven trials comparing DES and bare metal stents are reported in all the assessments. Four of the trials used in all the assessments compared Sirolimus DES against BMS (Ravel, Sirius, C-Sirius, and E-Sirius), while three compared Paclitaxel DES with BMS (Taxus I, Taxus II, and Taxus IV).
Clinical Evidence Considered
RCT, randomized controlled trial; DES, drug eluting stents; BMS, bare metal stents; NICE, National Institute for Clinical Excellence; KCE, The Belgian Health Care Knowledge Centre; PATH, Programme for Assessment of Technology in Health; LBI, The Ludwig Boltzmann Institute; HTA, health technology assessment.
All the agencies considered head to head comparisons of different types of DES, although the degree to which this was considered varied. These comparisons were partly informed by evidence submitted by manufacturers as well as published sources such as the BASKET and ASPECT trials comparing Paclitaxel DES against Sirolimus DES.
The findings of clinical studies were combined into meta-analyses by NICE and PATH. Both KCE and LBI considered a recently published meta-analysis but made no attempt to conduct their own meta-analyses.
Clinical end points considered by the agencies varied considerably. The meta-analyses conducted by the NICE assessment group included up to eight different end points, while the most commonly reported end points considered across agencies included 1 year mortality, myocardial infarction, target lesion revascularization (TLR), and device associated adverse events. Target vessel revascularization (TVR) was considered by three of the assessments with the exception of KCE.
Conclusions based on the clinical evidence varied somewhat across agencies, particularly in relation to mortality. While LBI reported that Sirolimus Eluting Stents may result in increased noncardiac mortality, based on the findings of the BASKET Late Trial, other agencies concluded that there is no significant difference in rates of mortality between DES and BMS. While PATH and NICE reported no significant difference in the rate of myocardial infarction between stents, KCE concluded that Sirolimus Eluting Stents have lower rates of MI than Paclitaxel Eluting Stents. Nonetheless, all agreed that DES reduce the rate of TLR significantly, and there was no overall elevated risk of stent thrombosis with DES compared with BMS.
Use of Published Non-RCT Data Sources
The handling of non-RCT sources of clinical evidence varied across the assessments. Only NICE considered non-RCT evidence comparing DES with BMS. These studies were mainly concerned with less widely used drug eluting stents such as the dexamethasone coated stent. None of the other agencies made explicit reference to these studies.
With the exception of LBI, all the assessments considered published findings derived from stent registries which, in some cases, were sponsored by manufacturers. A total of seventy-four publications on forty-six registries were included in the assessments. (Note: the number of publications, rather than the number of registries, was considered in this analysis. For some registries there was more than one publication on the findings). Six of these publications were considered by more than one assessment group. The size, duration, and populations considered in the registries varied significantly.
Within Country Registry Data
In addition to considering published studies on registries, three of the assessments included reference to local registry information which was unpublished before the assessment. Both NICE and KCE considered data generated from within country registries specifically to help inform their recommendations. The assessment conducted for NICE included analysis of patient level data from a single cardiothoracic centre in England. This source provided follow-up data on patients receiving DES in a naturalistic setting in the National Health Service. KCE used the Belgian Working Group of Interventional Cardiology registry data for the year 2004 to establish the outcomes associated with DES.
PATH was most active in generating additional evidence and carried out its own field evaluation in collaboration with the Ontario Ministry of Health and Long Term Care and twelve Ontario Regional Intervention Cardiac Care Centers over a period of 2 years to support the evaluation of clinical and cost-effectiveness of DES compared with BMS. PATH analyzed retrospectively the type of stents provided to patients and correlated this with their outcomes.
These registries considered significant populations, with the sample size ranging from 2,000 to 20,000 patients. These local registry data were used to supplement the published evidence which was found to be heterogeneous and in many cases of limited relevance to local practice. Registries also provided additional evidence on long-term outcomes and re-intervention rates that were not always adequately captured in clinical trials. The registry data were also essential to the development of the economic evaluations of DES compared with BMS.
Consideration of Published Economic Evidence
Published economic evidence was formally reviewed by NICE and KCE. As the remit of PATH is focused on field based clinical evaluations, no attempt was made to assess the published economic evidence, although evidence was generated based on the findings of the field evaluation. LBI considered one German study but made no attempt to undertake a more systematic review of published sources of economic evidence.
Published economic evidence retrieved by NICE and KCE included studies carrying out both cost-effectiveness analyses and cost utility analyses. End points considered varied between TVR avoided, TLR, major adverse cardiac events (MACE), rate of restenosis, and repeat rate of revascularization. Some of the studies adopted a cost utility approach, reporting outcomes in the form of incremental cost per quality-adjusted life-years (QALYs). Incremental cost-effectiveness ratios reported varied widely, with the cost per QALY ranging from £3,000 to as high as £1.0 million. This can be partly explained by differences in the type of patients considered, that is, patients with diabetes, multivessel disease or single vessel disease, lesion characteristics, and so on. There was significant heterogeneity among the studies in terms of the approaches adopted, the end points considered, and their conclusions as can be seen in the Table 2. This created confusion in the interpretation of the available evidence.
Summary of the Economic Evaluations Considered by the Assessment Groups
DES, drug eluting stent; BMS, bare metal stent; CE, cost-effectiveness; QALY, quality-adjusted life-year; UK, United Kingdom; US, United States; RR, repeat revascularization; RRA, repeat revascularization avoided; RA, revascularization avoided; TVR, target vessel revascularization; TLR, target lesion revascularization; MACE, major adverse cardiac events; RCT, randomized controlled trial.
Primary Economic Evaluations
Primary analyses using data generated from literature and local registries were carried out by NICE and KCE, while PATH used data from its field evaluation to generate additional evidence on the economics of DES. A secondary analysis using the Danish HTA was carried out by LBI.
Direct comparisons of the evaluations are difficult due to differences in approach, end point, and populations considered. All the agencies carried out cost-effectiveness analyses, while cost utility analyses were performed by all except LBI. Other than using the QALY as an end point, NICE considered repeat revascularization, while revascularization rates were considered by KCE, LBI, and PATH.
Handling of the prices of DES, relative to BMS, varied across the agencies with consideration being given to list prices, average selling prices supplied by manufacturers, and “actual” prices paid by procurement bodies. For example, PATH derived an average weighted cost of DES using the relative market shares of the different DES stents used in Ontario.
To account for uncertainty, sensitivity analyses using methods such as probabilistic sensitivity analysis (KCE and PATH) or extreme values analysis (NICE) were carried out.
Table 3 summarizes the key findings from the economic evaluations. Despite significant differences and very wide confidence intervals around the cost-effectiveness ratios, the interpretation of the economic evaluations is broadly similar across the agencies, with DES being economically justified only for high-risk individuals. However, there remain some differences in the definition of high-risk individuals.
Summary of Economic Evaluations Carried Out by Agencies
DES, drug eluting stent; BMS, bare metal stent; CE, cost-effectiveness; QALY, quality-adjusted life-year; KCE, The Belgian Health Care Knowledge Centre; NICE, National Institute for Clinical Excellence; PATH, Programme for Assessment of Technology in Health; LBI, The Ludwig Boltzmann Institute; DAHTA, German Agency for Health Technology Assessment.
Recommendations
Despite differences in the clinical evidence considered, the patient populations considered and using different sources of health outcomes data, agencies nonetheless came to fairly similar conclusions. NICE, PATH, and LBI all identify high-risk patient groups who could benefit most from DES in their recommendation. KCE came to a more negative conclusion which advocates clearly that DES should not be reimbursed. Table 4 summarizes the recommendations made following the appraisal of the HTA of DES compared with BMS.
Summary of Recommendations
Note: All clinical trials retrieved by agencies. Either retrieved from literature or recent meta-analysis.
NICE, National Institute for Clinical Excellence; PCI, percutaneous coronary intervention; DES, drug eluting stents; BMS, bare metal stents; PATH, Programme for Assessment of Technology in Health; LBI, The Ludwig Boltzmann Institute; KCE, The Belgian Health Care Knowledge Centre.
DISCUSSION
The findings highlight areas of convergence and divergence in the approaches, evidence considered, and recommendations of the four agencies. The approaches adopted differed substantially, particularly in respect to the degree of primary evidence generated for the purposes of the HTA. This reflects the differing remits and capacities of the HTA bodies considered in the analysis.
There were some similarities in the evidence considered. Although there were differences in inclusion and exclusion criteria for the clinical trials considered, a common set of pivotal studies was considered by all the HTA bodies. The inclusion of these studies is unsurprising, given that DES remain a relatively novel technology and have been subject to a relatively small number of large, robust RCTs.
However, in all cases, the HTA bodies were critical of the relevance of the RCTs to their own country settings and found the published clinical evidence to be too heterogeneous to inform local decision making. As a result of this, the HTA bodies resorted to identifying evidence from local registries to supplement the information derived from RCTs. In the case of PATH, a significant effort was put into generating and analyzing a large body of evidence on the use of DES in Canadian practice. In the case of NICE, additional evidence was identified from a single hospital to help provide further evidence on the long-term outcomes of DES. This evidence was critical to the development of the economic evaluations conducted as part of the assessments.
The implications of this are twofold: first, manufacturers should consider how best to generate rigorous yet relevant evidence that meets the needs of HTA bodies. Second, payer bodies need to engage in further discussion with manufacturers about what level of evidence is desirable and acceptable for HTA purposes.
Despite significant differences in the evidence considered, the resulting recommendations display some degree of convergence. With the exception of KCE, all the agencies recommended the selective use of DES in individuals at elevated risk of cardiac events. Limitations on their use arise due to differences in their effectiveness across patient subgroups.
Relevance to the Debate on the Harmonization of HTA
What are the implications of these findings for the harmonization of HTA? The research identified a “core” data set of clinical and economic evidence considered by the HTA bodies, endorsing the EUnetHTA approach. However, it is noticeable that the evidence derived from the core data set was criticized by the HTA bodies and appears to have had limited influence on the resulting recommendations. The need to supplement a core data set with evidence generated locally, with relevance to local practice was critical to the development of the recommendations. This appears to challenge the hypothesis of the EUnetHTA approach which suggests that a core HTA can be readily adapted for use in local settings and rather, that significant effort is required to undertake meaningful local adaptations, often involving the generation of primary evidence.
Evidence from local settings or evidence generated as part of the assessments was critical to the development of the recommendations in each case. Previous authors have commented on the extent to which health technology assessment can be considered context-free or context-specific (Reference Culyer and Lomas7). This analysis suggests that, whereas some elements of the assessment process may be context-free (e.g., consideration of safety), the majority of the elements are context-specific, at least to some extent. In the case of the assessments considered in this analysis, assessment of the clinical evidence, cost-effectiveness evidence, organizational and budgetary implications of adopting DES were all informed by local evidence identified or generated as part of the assessment. In each case, this evidence appeared to have a significant influence on the resulting recommendations.
The analysis also highlights the different approaches that are currently adopted in HTAs to generating primary research to inform the assessment. PATH was the most advanced of the organizations considered at generating clinical evidence on technologies in practice, and they have an active portfolio of studies under way to help inform the appropriate use of new technologies. NICE too routinely generates primary evidence in the form of economic evaluations commissioned to support the development of their guidance and has recently begun to work with manufacturers to help identify appropriate study methodologies for new technologies before launch. More widespread adoption of this approach could lead to increased demands for country-specific evidence and possibly increase divergence in the evidence requirements of assessment bodies.
It is interesting to consider whether the use of drug eluting stents provides a typical case study of HTA. It could be argued that the effectiveness of medical devices is more susceptible to differences in local practice patterns than pharmaceuticals. If it is accepted that pharmaceutical trials are more generalizable across settings, then there may be less need for reliance on primary evidence generation and local registry data. As such, a greater degree of harmonization may be achievable for these technologies. Further comparisons of this type are warranted to explore whether this is the case.
The findings of this research paint a somewhat pessimistic view of the potential for harmonization of evidence requirements across HTA bodies. The demand for evidence that is relevant to local practice means the core dataset is relatively small and that significant efforts are required to generate additional data to inform local coverage decisions. Although this is good news for those involved in the HTA industry, it creates a challenging environment for manufacturers struggling to address the evidence demands of regulatory bodies, HTA agencies, and payer bodies while also looking to contain research and development costs.
CONCLUSIONS
DES provide a useful case study to assess the degree to which HTAs converge across jurisdictions. Based on our analysis, there is relatively limited convergence in the approaches to HTA. Although a common set of clinical and economic evidence was considered by all the assessments, ultimately this evidence had limited impact on the resulting recommendations. Evidence generated from local data sources, particularly registries, was necessary to inform the assessment and the resulting recommendations. Despite this, the resulting recommendations, from three of the four HTAs considered displayed similarities. This case study suggests that, although there is some potential for harmonizing the assessment of evidence across jurisdictions, this is relatively limited due to the desire to generate additional evidence relevant to the needs of local practice.
CONTACT INFORMATION
Paul Trueman, BSc, MSc (pt507@york.ac.uk) Director, Manjusha Hurry, BSc, MSc (manjushahurry@gmail.com) Research Consultant, York Health Economics Consortium, University of York, 2nd Level Market Square, Vanbrugh Way, York, North Yorkshire YO10 3EY, UK
Matthew W. Bending, MSc Econ, BSc Econ, (mwb501@york.ac.uk) Research Fellow, York Health Economics Consortium, University of York, Heslington, York YO10 5NH, UK
John Hutton, BSc, BPhil, (jh602@york.ac.uk) Professor, Department of Health Sciences, University of York, Project Director, York Health Economics Consortium, University of York, Vanbrugh Way, York, North Yorkshire YO10 5NH, UK
