An increasing number of countries are using health technology assessment (HTA) to inform pricing and reimbursement decisions for new health technologies, especially pharmaceuticals. These HTAs normally incorporate and economic evaluation (EE), in which the costs of the new intervention are compared with their benefits (Reference Drummond1). Although there are well-established methods for EE (Reference Drummond, Sculpher, Claxton, Torrance and Stoddart2), derived in part from the underlying principles of welfare economics (Reference Tsuchiya and Williams3), there are still controversies concerning its use in healthcare decision making (Reference Coast4), or its use for particular types of healthcare interventions (Reference Shiell, Hawe and Gold5). Despite these controversies, the use of EE is expanding worldwide, driven mainly by the need to use healthcare resources wisely, given the growing pressures on healthcare budgets.
This study explores the problems faced by decision makers in middle-income countries in attempting to use EEs to inform pricing and reimbursement decisions at the national or jurisdictional level. The conduct of a comprehensive EE requires a considerable amount of time and financial resources as well as technical expertise and data, all of which may pose challenges in these environments. Therefore, decision makers in middle-income countries may have to use analyses or data from other jurisdictions in their decision-making processes. While adapting or using EEs from other jurisdictions has the potential to save time and budget, inappropriate transfer evaluations can result in incorrect decisions which may delay patient access to the latest medical advances, or lead to an inefficient use of scarce healthcare resources.
There is a growing literature on the transferability of economic evaluations. A Good Research Practices Task Force force established by the International Society for Pharmacoeconomics and Outcomes Research reviewed the various approaches for interpreting analyses or data from other jurisdictions and how they might be made relevant to the local context. The Task Force also reviewed the advice on dealing with transferability issues provided in various international methods guidelines for economic evaluation and made some recommendations for good practice (Reference Drummond, Barbieri and Cook6;Reference Barbieri, Drummond and Rutten7). Because many economic evaluations use decision-analytic models to estimate the costs and benefits of healthcare interventions (Reference Caro, Briggs and Siebert8), the most common approach is to attempt to re-populate the model with local data.
Therefore, the objectives of this study are (i) to review the current practices of authorities in middle-income countries in using economic evaluations from other jurisdictions in decision making, using examples from select countries in Central and Eastern Europe, Latin America, and Asia; (ii) to map out the various methods of adaptation, examining the extent to which some middle-income countries simply “copy and paste” EEs, versus rigorously assess the degree of transferability, following a scientific methodology for adapting studies to the local context; (iii) to consider the various factors affecting the geographic transferability of EEs, focusing on the risks and limitations of using nonnative economic models or data in local decision making; and (iv) to explore the obstacles to transferability and the potential solutions.
METHODS
In preparation for the study we reviewed the major publications on the transferability of economic evaluations, including the EUnetHTA Adaptation Checklist (9), the ISPOR Task Force Reports on the Transferability of Economic Evaluations Across Jurisdictions (Reference Drummond, Barbieri and Cook6;Reference Barbieri, Drummond and Rutten7), a review of existing checklists (Reference Goeree, Burke and O'Reilly10), and two papers discussing the use of economic evaluations in Latin America (Reference Augustovski, Iglesias and Manca11;Reference Pichon-Riviere, Augustovski, García Martí, Sullivan and Drummond12). We also consulted the Web sites of the key organizations using economic evaluation in decision making concerning health technologies in the three regions of interest. The literature review was supplemented by structured interviews with representatives of the organizations concerned. Although several jurisdictions in middle-income countries undertake HTAs, we wanted to concentrate on those jurisdictions where it was likely that decision makers were partially reliant on economic evaluations conducted abroad. Therefore, the criterion for selecting the key organizations was that they were in jurisdictions that had been using economic analyses as part of their pricing and reimbursement processes for health technologies for at least 1 year. In each case, an approach was made either to the director of the organization, or to the person heading the division dealing with economic evaluations, explaining the objectives of the research, the researchers involved, and the source of funding for the study. In the majority of cases, the person contacted participated in the interview themselves, although in some cases they designated one of their staff to be the respondent.
An interview schedule with structured questions and response options was agreed among the study team and formed the basis of the interview. Whenever feasible, the schedule was sent to interviewees in advance, so that they could prepare before the interview. The interview covered the following areas (i) the role of the organization concerned in conducting or using economic evaluations, the expertise and skills of personnel available and the existence of methods guidelines for EEs; (ii) the current use of data, analyses of economic models from other jurisdictions and views on the transferability of each; and (iii) views on the main obstacles to the transferability. The list of obstacles was constructed based on our own experience of the problems of attempting to use economic evaluations conducted in other jurisdictions. (The interview schedule is available as a Supplementary Material file.)
While the interview generally consisted of a series of close-ended questions there were also some open-ended questions. However, as the researchers conducting the interviews were fairly experienced in their own region they further explored any responses that they believed required further clarification during the interview. In addition, where possible, the information obtained was verified by the researchers themselves, and/or by consulting with other academic researchers in the jurisdictions concerned. In the case of discrepancies between the interviewee's responses and the researcher's knowledge, a final response was agreed that adequately reflected the situation in the jurisdiction concerned. Therefore, while relying mainly on information reported by the organizations concerned, reasonable attempts were made to ensure its accuracy.
RESULTS
Sample of Organizations Studied
Representatives were interviewed from the relevant organization in each of the following twelve countries from the three regions that met the inclusion criteria: Asia (South Korea, Taiwan, and Thailand), Central and Eastern Europe (Croatia, Hungary, Poland, and Slovakia), Latin America (Argentina, Brazil, Colombia, Mexico, and Uruguay). In most cases, the question of inclusion or exclusion was straightforward. However, there are several other countries, such as Chile, that have some experience of the use of economic evaluation, but have not yet incorporated it into a formal process for pricing and reimbursement of health technologies (The list of organizations surveyed is given in Table 1).
Organizations Surveyed
Official Methods Guidelines for Economic Evaluation
Methods guidelines for EEs are typically produced in jurisdictions that plan to use economic evaluations as part of an official decision-making process. Normally, they are intended to be followed by both those making submissions and those evaluating them. Of the jurisdictions in our sample, all twelve had official guidelines, of which five had been developed by the organization surveyed.
Role of the Organizations Surveyed
The organizations surveyed could have one or more roles. First, they evaluate submissions of data or analyses from other parties, typically industry. Second, they could commission studies to be done by others, typically independent researchers. Finally, they could conduct their own EEs. The most common role was to evaluate studies submitted by others such as manufacturers (eleven organizations), although nine organizations also conducted studies and seven commissioned them.
Uses of Economic Evaluations
In some jurisdictions, economic evaluations are conducted to provide general information to inform resource allocation decisions. However, increasingly studies are being performed for a specific purpose. It was found that the use of economic evaluations closely followed the role of the organizations themselves, with a strong emphasis on studies to inform reimbursement or coverage decisions (all twelve organizations). In addition, it can be seen a common use of EEs was also to inform price negotiations and decisions (eight of twelve), reflecting the trend toward considering price as a variable in cost-effectiveness assessments, although not necessary indicating the existence of formal “value-based pricing” schemes.
Access to Skills and Expertise
Any organization evaluating, commissioning, or conducting HTAs or EEs requires access to personnel with a relevant range of skills and expertise. Many of the organizations had access to the relevant range of skills, such as physicians/clinical specialists (eleven of twelve organizations), pharmacists (10/12), health economists (10/12), medical statisticians (9/12), and epidemiologists (8/12).
Use of International Web sites
It is common for HTA bodies to consult several key Web sites to check whether particular assessments have been conducted in other jurisdictions, or more generally to search for evidence relating to the technology of interest (e.g., systematic reviews of the clinical evidence). The organizations studied reported that they considered a wide range of international Web sites. The five most frequently consulted Web sites were those of the National Institute of Health and Care Excellence (UK) (n = 10), the Centre for Reviews and Dissemination (UK) (n = 6), the Canadian Agency for Drugs and Technologies in Health (n = 4), the Scottish Medicines Consortium (n = 4), and Australia's Pharmaceutical Benefits Advisory Committee (n = 3).
Use of Studies from other Jurisdictions
Of the eleven organizations whose role included evaluating economic evaluations, nine believed they had enough experience to answer the survey questions about their practice in consulting economic studies with content from another jurisdiction. Studies from other jurisdictions could potentially be used in several ways, ranging from a more general use to gain a better understanding of the background of the technology concerned or the decision problem in hand, to more specific uses, such as to check specific items of data. In the extreme, a study conducted elsewhere could be used as a basis for making a local decision if no local studies are available. Because the use of existing studies may vary from case to case, the organizations interviewed could give a graded response, from “Often” to “Never.” Table 2 summarizes the responses obtained. It can be seen that, reassuringly, results from studies conducted in other jurisdictions were rarely used as the sole basis for making a local decision.
In What Ways Are the Results from Studies Conducted in Other Jurisdictions Used?
In addition, the organizations were asked to name the jurisdictions they often turned to as a “reference country” for local decisions. The responses to this issue are interesting in that very rarely did respondents name countries within their own region. The three countries named as reference countries were the United Kingdom (eight times), Australia (five times), and Canada (four times)
Use of Transferability Checklists
All of the organizations studied reported that the economic evaluations submitted to them contained data generated in other jurisdictions. Several checklists have been developed to assist those wrestling with the challenges of adapting studies or data from other jurisdictions. Respondents were asked indicate whether they had consulted any of those checklists published in the literature. In general, the checklists were not used, with the EUnetHTA Adaptation Toolkit (www.hta.ac.uk/links/finaladaptationtoolkitnetscc.pdf) being the most frequently mentioned (i.e., by three of the organizations studied).
Perspectives on the Use of Data from other Jurisdictions
Previous studies (Reference Barbieri, Drummond and Rutten7) have shown that there are differing perspectives on the use of data from other jurisdictions. Sometimes, there is an official position on the use of foreign data, stated in the jurisdiction's methods guidelines for conducting EEs. Nine of the organizations answered questions on this topic, relating to their role in evaluating economic evaluations, most commonly in industry submissions. Their views on the use and transferability of different categories of data are shown in Table 3.
Use of Categories of Data from Foreign Studies and Views on their Level of Transferability
These responses are consistent with previous research in that data on relevant treatment effect are considered more transferable than those on prices/unit costs. The views on the transferability of epidemiological data, data on resource use, and health state preference values are more mixed. With respect to the use of decision-analytic models from other jurisdictions, eight of the nine respondents to this question stated that analyses submitted to them had used models developed in other jurisdictions. In the majority of cases, the models were adapted to reflect local circumstances, in some cases not. Only four of the nine organizations had an official policy on the use of “foreign” models. Where there was an official policy, this was that models developed in other countries were acceptable, providing they were adapted to reflect local circumstances.
Similar questions were asked of those organizations that stated that their activities in HTA included commissioning or conducting EEs. The responses to many of the questions were fairly similar to those given in the case of evaluating studies. For example, studies undertaken in other jurisdictions were always consulted, but the existing transferability checklists were rarely used. However, in commissioning or conducting local studies there was a slightly lower tendency to use foreign data or foreign models, with a greater emphasis on using local data and analyses.
Obstacles to Transferring Studies from other Jurisdictions
Finally, respondents were asked to select from a list of potential obstacles to transferring data or analyses from other jurisdictions. The most frequently selected obstacles are shown in Table 4. It can be seen that the most important obstacles are those relating to the differences in current standard of care, practice patterns, or gross domestic product (GDP) between the developed countries where the majority of these studies are conducted and those jurisdictions that have a need to make local decisions. The second most important group of obstacles relate to inadequacies in reporting, including the lack of access to electronic copies of models.
Obstacles to Transferring Economic Evaluations from Other Jurisdictions
DISCUSSION
Although this study covered only twelve countries in Asia, Central and Eastern Europe, and Latin America, these countries represent the only jurisdictions in these regions that have had more than 1 year's experience with the formal use of economic evaluations in the pricing and reimbursement of pharmaceuticals. Therefore, this study should be viewed as a pilot study. However, although one must be careful of extrapolation, the experience of these organizations in dealing with the challenges of using studies or data from other jurisdictions may be predictive of the issues that other jurisdictions will face in the future, as they begin to use economic evaluations in their decision-making processes.
Main Findings
First, establishing the infrastructure to use economic studies in pricing and reimbursement decisions does not seem to be a major problem. Most of the organizations studied had access to the appropriate set of skills and were in jurisdictions where official guidelines have been established for the conduct of studies. However, previous research (Reference Neumann, Drummond and Jönsson13) suggests that the processes for conducting and evaluating HTAs (including economic evaluations) might need close scrutiny. That is, how good is the technical review of company submissions, how extensively are stakeholders involved, and how transparent is the process? Audit criteria for HTA organizations conducting HTA to inform resource allocation decisions have been previously suggested (Reference Drummond, Neumann and Jönsson14).
Second, HTA organizations in these regions do look to external resources when undertaking their activities, by consulting Web sites and locating studies conducted in other jurisdictions. This is probably no different from the practice of HTA organizations in Western Europe and North America. However, one key difference is that the jurisdictions where the majority of studies are conducted often have different treatment patterns and resource availability than the jurisdictions covered in this study. Indeed, the respondents reported that they mostly turned to developed countries as “reference” countries for their decisions, as opposed to countries within their own region, probably reflecting the fact that economic evaluations of the technology of interest were much more likely to exist in developed countries than in other middle-income countries, even if the latter jurisdictions were considered to be more relevant. It is also possible that evaluation reports from the countries named were readily available and/or considered to be reliable.
This suggests that issues relating to the transferability of economic evaluations are likely to be important. Therefore, it is somewhat surprising that the organizations studied had made very little use of existing transferability checklists. One explanation offered was that issues of transferability were adequately handled by local submission guidelines, so it was unnecessary to apply a checklist to the studies submitted. Another potential explanation is that HTA organizations in middle-income countries rarely have direct access to the economic models as applied in other jurisdictions. Therefore, they cannot assess directly the transferability of international economic models. On the other hand, the transferability of economic models can more easily be assessed by those organizations conducting economic evaluations, as opposed to those organizations that merely undertake a critical appraisal of submitted models only after partial or full local adaptation.
Third, the economic evaluations submitted to the organizations studied here often contain models developed in other jurisdictions and data from other countries. With respect to the models, the prevailing view is that they normally require adaptation to local circumstances and half the organizations that evaluate manufacturer submissions had official policies on this matter. However, more consideration is required of what “adaptation” entails and how one would determine the appropriate amount of adaptation in a given situation. A key issue is whether the adaptation required is confined merely to re-populating the model with local data (where available), or whether it extends to changing the model structure. The latter has more far-reaching consequences for technology manufacturers operating on a global level. More case studies are required to address this issue.
Turning to the data, the perceptions on the transferability of different categories of data mirrored the statements in official methodological guidelines for economic evaluation (Reference Barbieri, Drummond and Rutten7). By and large, the perceptions expressed were consistent and understandable, although more exploration of the transferability of relative treatment effect is probably required. For example, in some cases, the treatment effect seen in clinical studies undertaken in Western Europe and North America may be different (higher or lower) in middle-income countries because of differences in healthcare provision. However, paradoxically, more clinical trials of pharmaceuticals are now being conducted in Asia, Central and Eastern Europe, and Latin America, primarily for financial reasons.
Fourth, the analysis of the major obstacles to the transferability of foreign data or analyses showed that concerns about the differences in healthcare resources and practice patterns were the major reasons why respondents questioned whether studies conducted in other jurisdictions were relevant to their setting. Many of these issues could be handled by re-populating models with local data. However, if the concern is that the current standard of care in the country of interest was not compared with the new technology in the existing clinical trials, this suggests the use of network meta-analysis (Reference Jansen, Fleurence and Devine15;Reference Hoaglin, Hawkins and Jansen16), with the resulting, unavoidable, uncertainties.
As the use of economic evaluations to inform reimbursement and coverage decisions increases in middle-income countries, this study suggests that it will be increasingly important to ensure improved practices in evaluating and conducting transferability of foreign data. Inadequacies in dealing with transferability issues may lead to inaccurate estimates of the local cost-effectiveness of technologies, resulting in inappropriate coverage decisions and the inefficient use of healthcare resources.
Limitations of the Study
The main limitation of the study is that, at present, relatively few jurisdictions in middle-income countries are formally using economic evaluations in reimbursement and pricing decisions for health technologies. Therefore, current experience with using economic evaluations conducted elsewhere is quite limited. Second, the findings of our study are mainly based on interviews with key country experts, with the inherent shortfalls of the survey approach. However, the researchers conducting the interviews did their best to verify the responses and there is no a priori reason to suspect that biases in the responses could have been introduced, either through the objectives of the study or its funding.
Recommendations to those Evaluating Economic Evaluations
What could local HTA organizations do to improve on their use of economic evaluations conducted elsewhere? Certainly, they could make more use of the current published transferability checklists (Reference Goeree, Burke and O'Reilly10). In addition, they could consider investing in local data generation for those categories of data normally considered to have low transferability, such as unit costs, health state preference values, and epidemiological data. Also, as the number of HTA organizations in middle-income countries increases, they could collaborate more fully within their region, because the transferability of economic evaluations within the region is likely to be greater than that between regions. Finally, although the organizations studied reported that they had access to a wide range of skills, there probably still a need to invest in training in the relevant expertise in economic evaluation, as its use in reimbursement and coverage decisions increases.
Recommendations to those Conducting Economic Evaluations
Technology manufacturers operating on the global level could try to gain a better understanding of the operation of healthcare systems in Asia, Central and Eastern Europe, and Latin America to gain a greater appreciation of the likely problems in the transferability of studies. In addition, when developing a global decision-analytic model, they could better anticipate the need for local adaptation, including the possibility that the model structure may need to be adapted. They could also, invest in data themselves, either as individual companies, or collectively within a given disease area (e.g., in generating unit costs or epidemiological data relevant to the disease in question). The fact that respondents in this study reported that they placed a lower reliance on foreign data when developing their own models suggests that there may be a greater possibility of making studies locally relevant than is exhibited in some of the studies submitted to HTA organizations for evaluation.
Many of these initiatives could be better pursued if there were more active engagement between manufacturers and decision makers in these regions. This interaction has greatly increased in Europe in recent years, but requires a certain degree of trust on all sides. However, one might expect it to increase in the jurisdictions studied as decision makers gain more experience with the use of economic evaluations in their decision-making processes.
CONCLUSIONS
In the jurisdictions studied, it is commonplace to deal with the issue of transferring analyses or data from other jurisdictions to inform reimbursement of pricing decisions. Decision makers in these countries have used data or analyses from foreign studies in several ways. They are aware of the various factors affecting the geographic transferability of economic evaluations. However, they face several challenges, mainly due to differences in current standard of care, practice patterns or GDP between the developed countries where the majority of the studies are conducted and their own jurisdiction. Knowing these issues, there are several actions that those conducting or evaluating economic evaluations could do to increase their transferability.
SUPPLEMENTARY MATERIAL
Transferability of HTA Interview Schedule http://dx.doi.org/10.1017/S0266462315000604
CONFLICTS OF INTEREST
The authors have no conflicts of interest with respect to the content of this paper.
