Health technology assessment (HTA) agencies employ processes that are increasingly used for evidence-based decision making in low- and middle-income countries, which have an arguably greater need to prioritize the use of scarce resources (Reference Singer21). Latin America and the Caribbean (LAC) is a region with increasing use and influence of HTA in decision making (Reference Banta3;Reference Iglesias, Drummond and Rovira15;Reference Pichon-Riviere, Augustovski and Rubinstein19) with HTA agencies affiliated to INAHTA (International Network of HTA agencies) in four countries: Argentina, Brazil, Chile and Mexico (16). Furthermore, HTA was formally used to shape benefit packages in Argentina, Uruguay and Chile; while countries like Brazil, Mexico, and Colombia have a formal fourth hurdle system in place that requires proper evaluation of new technologies (Reference Augustovski, Bastardo and Caso2).
Producing comprehensive technology assessment (TA) reports on the medical, social, ethical, and economic implications of development, diffusion, and use of health technology requires a considerable amount of time and financial resources. Even in high income countries, most agencies are unable to conduct the desired number of assessments at a satisfactory level of quality (Reference Turner, Chase and Milne23). Adapting or using TA reports from other jurisdictions could reduce the need for multiple country reports on the same health technology, saving time and resources (Reference Turner, Chase and Milne23).
Several authors have addressed the problem of generalization and transferability of clinical and economic evidence across jurisdictions (Reference Augustovski, Iglesias and Manca1;Reference Barbieri, Drummond and Rutten4;Reference Goeree, Burke and O'Reilly13;Reference Sculpher, Pang and Manca20). Toolkits and checklists have been developed to guide the adaptation of reports and to assess the degree of transferability of the results from one region to another (Reference Boulenger, Nixon and Drummond6;Reference Chase, Rosten, Turner, Hicks and Milne8–Reference Essers, Seferina and Tjan-Heijnen10;Reference Nixon, Rice and Drummond18;Reference Turner, Chase and Milne22;Reference Urdahl, Manca and Sculpher24;Reference Welte, Feenstra, Jager and Leidl25).
Decision makers and researchers have direct and immediate access to TA reports produced in other countries. The global production of TA reports markedly increased in the last decade, and many of these reports can be readily accessed through Internet search engines. Given the scarcity of resources for developing local TA reports, decision makers and researchers in LAC may be using, formally or informally, reports from other jurisdictions when faced with decisions that cannot be avoided or postponed.
OBJECTIVES
The objectives of this research are to explore the extent to which decision makers and researchers in LAC are seeking and using TA reports from other jurisdictions. In particular, we sought to determine precisely how the reports were adapted and whether these reports supported both in-country HTA processes and local policy and resource allocation decisions. Finally, we wished to ascertain specific barriers and limitations of using external TA reports within LAC.
METHODS
Transferability usually refers to the potential for the analysis to be adjusted to make the results relevant to different settings (Reference Drummond, Barbieri and Cook9). We will use the term here in a broad sense and understand it as transferability experiences if researchers are using TA reports from other jurisdictions as an input/source when developing local reports or if they are explicitly adapting an HTA report to the local setting. The same is true if decision makers are using or applying TA reports from other jurisdictions to guide decisions. For the present study, these transferability experiences will be categorized as between developed and developing countries (“north-south” or trans-regional) or among the different Latin American countries (“south-south” or intra-regional).
We developed and distributed a confidential, self-administered, Web-based survey to a sample of 13,031 decision makers and researchers in nineteen Latin American and Caribbean countries (9,989 in Argentina and 3,042 in the other eighteen countries).
The survey had a component for decision makers and a component for researchers to evaluate to what extent they were using or applying the information of TA reports from other jurisdictions to guide decisions and as an input/source when developing local TA reports. Furthermore, by using this latter component it was possible to see how relevant and adaptable they found the information in the reports in different domains (e.g., efficacy, safety, economic evaluation). All items had a 1 to 10 scale, with anchors at 1 (not useful/adaptable) and 10 (extremely useful/adaptable). The survey also included a component both for DMs and researchers where we explored the perceived usefulness of the transferability of TA reports from other jurisdictions and the perceived barriers that limit transferability in the region.
Additional data retrieved included information on the respondent, and the kind of decisions for which the reports were used. The survey was administered in neutral Spanish and was piloted with potential respondents to ensure technical feasibility, relevance and understanding of the questions (the full survey can be viewed at the following link: http://www.iecs.org.ar/eets/).
LAC Survey recipients were identified through our network of researchers, managers, policy makers, and other users that are registered at the Institute of Clinical Effectiveness and Health Policy (IECS) Web page (www.iecs.org.ar). Registration is free but mandatory to access IECS's technology assessment documents. We enhanced the survey sample by including possible respondents from national health ministries, local and regional HTA agencies, university/research organizations, third party payers, medical device and pharmaceutical industries, regulatory authorities, and the Pan American Health Organization Listserv on health technologies. An e-mail with a brief explanation of the project and a link to the survey Web site was sent to the aggregated list between May and December 2010 and up to three reminders were sent to non-respondents.
Descriptive statistics were computed for all variables measured. We used the chi-square test to determine differences in categorical variables, and the independent-sample t-test or Wilcoxon-Rank Sum Test for continuous variables. All p values of less than .05 (two-tailed) were considered to be statistically significant. Data analyses were performed using Stata (version 8.0).
RESULTS
We received 671 responses with a response rate of 5.2 percent. Fourteen responses were eliminated as they were from individuals located in non-LAC countries. The characteristics of the respondents are summarized in Table 1. Fifty-five percent of the responses were from Argentina, and 45 percent were from eighteen other LAC countries, mainly Colombia (10.5 percent), Brazil (4.7 percent), Uruguay (3.8 percent), Chile (3.5 percent), Mexico (3.3 percent), and Peru (3.3 percent). Twenty-four percent of the respondents defined themselves as HTA researchers and 76 percent as DMs. Forty-three percent of respondents belonged to the public or social security health sectors.
Respondents' Characteristics
LAC, Latin-America; HS, Health Sector; Pharm, Pharmaceutical.
Around one-third stated that they use the TA reports at an institutional level for decisions related to coverage and reimbursement of health technologies; one third used them at an institutional level for other decisions not directly related to coverage (e.g., Clinical Practice Guideline development); and another third used them for clinical decisions at the patient level.
Of DMs that reported using TA reports for a policy decision, 75.7 percent reported using reports from non-local sources. In 24.3 percent of the cases they used TA reports from their own countries, in 23.1 percent they used reports from other LAC countries and in 52.6 percent of the cases the reports used came from other regions outside LAC such as Europe, United States, Canada, or Australia. When using TA reports from other jurisdictions, the respondents considered the information regarding the description of the technology (7.4), safety (7.0), and efficacy/effectiveness (7.5) to be more useful, applicable and adaptable (mean score of 7.0 or more on a 1 to 10 scale). They found the information regarding ethical/legal/social aspects (5.9), budget impact (5.9), economic evaluation (6.5) and organizational issues (6.4) to be less applicable or less adaptable (mean score below 7) (Table 2).
Mean Rating Score of the Usefulness and Applicability of the Information Found in Health Technology Assessment Reports (HTAs) From Other Jurisdictions for Different Domains: Responses From Researchers and Decision Makers
Scores in a 1 to 10 scale (1 = not useful at all, 10 = extremely useful).
HTAs: Health Technology Assessment reports; N/E: not evaluated; ns: p value not significant (>.05).
When decision makers rated the applicability and usefulness of the reports for decision making, they considered the information from other regions to be more useful, followed by the TA reports from their own countries and then from other LAC countries (7.1, 6.3, and 6.0, respectively, p < .01).
Surprisingly, decision makers also considered the information in TA reports from other regions to be more applicable and useful, even in domains that could be considered as having low potential for transferability between regions, such as information on cost-effectiveness or budget impact (6.5 versus 5.5; p < .01).
Researchers reported that they frequently use (63 percent) TA reports from other countries as an input/source when developing local TA reports. Nineteen percent of these reports came from other LAC countries and 80.5 percent were from other regions outside LAC.
As expected, the researchers indicated that when using TA reports from other regions, they considered the information regarding the description of the technology (8.4), safety (7.5), efficacy/effectiveness (8.3), as an aid in the comparisons of results (7.4), the methods of the systematic review (7.7) and as a starting point from which to develop a new report (8.0) to be more useful, applicable and adaptable. Furthermore, they found the information regarding ethical/legal/social aspects (5.9), budget impact (5.6), economic evaluation (6.8) and organizational issues (6.2) to be less applicable or less adaptable (Table 2).
Not surprisingly and consistent with the views of decision makers, the researchers considered the information produced by non-LAC HTA bodies, as compared to other LAC countries, to be more useful and applicable (7.6 versus 6.7; p < .01). Again, this was seen even in domains that could be considered to have low potential for transferability between regions, such as the information from the economic evaluations (6.8 versus 6.0; p < .01).
Researchers, as compared to decision makers, valued the usefulness of TA reports from other jurisdictions more highly (Table 2). This was observed mainly for the information regarding the description of the technology (8.4 versus 7.4; p < .01) and for efficacy/effectiveness data (8.3 versus 7.5; p < .01).
Regarding the adaptation of TA reports, 33.1 percent of the researchers stated that they had adapted an HTA report to the local setting at least once. In 78.5 percent of the cases, the reports adapted came from non-LAC countries.
In the general section of the survey, all respondents considered TA reports from other jurisdictions to be potentially very useful with a mean of above 7 of 10. However, the potential usefulness, for both decision makers and researchers, was considered to be greater for TA reports from LAC compared with other regions (7.4 versus 6.9; p < .01).
Usefulness of using TA reports from other jurisdictions scored highly (above 7) in all the domains explored. Respondents considered it useful in improving the decision-making process (7.5), reducing the resources needed (7.3), obtaining faster results (7.2), and avoiding duplication of work (7.1). Researchers rated the usefulness significantly higher when compared with decision makers, mainly when using it to avoid duplication of work (7.8 versus 6.9; p < .01) and to obtain faster results (7.7 versus 7.0; p = .01).
When the respondents were asked to identify and rate the importance of the current barriers for the expansion of the transferability of non-local TA reports, the importance given to each barrier varied significantly (Table 3). For example, low methodological quality was considered to be a restriction to the transferability of an HTA report from any origin, but this barrier was considered significantly more notable in reports from LAC than from other regions (6.7 versus 5.3; p < .01). The same was true for the low availability of reports (7.1 versus 5.2; p < .01). On the contrary, a different epidemiological context was considered a significantly smaller barrier for reports from LAC compared with other regions (6.0 versus 7.4; p < .01). The same was true, for example, for different healthcare costs (6.5 versus 7.9; p < .01). In general, for TA reports from LAC the main barriers identified by the respondents were the low quantity of reports available, lack of guidelines/methods on how to adapt reports to local conditions, and low methodological quality and lack of transparency in the published reports. Respondents uniformly considered the barriers regarding differences in healthcare costs, different epidemiological contexts and different healthcare system characteristics to be more important in the case of non-LAC TA reports.
Mean Rating Score of the Importance of the Different Barriers That May Be Currently Limiting a Wider Transferability of Health Technology Assessment Reports (1–10 Scale, Higher Scores Corresponds to Barriers Considered to Be More Important)
HTA, health technology assessment; HTAs, health technology assessment reports; LAC, Latin America and the Caribbean.
*Regions outside Latin America (e.g., Europe, Australia, Canada).
DISCUSSION
Most countries in the Latin American and Caribbean region have major healthcare problems related to both equity and efficiency, in a context of pluralistic and fragmented healthcare systems. Many of these countries have implemented or are actively considering formal HTA programs with the hope that more deliberative and evidence-based evaluation processes will help prioritize the use of scarce health resources (Reference Augustovski, Bastardo and Caso2;Reference Banta3;Reference Iglesias, Drummond and Rovira15;16;Reference Pichon-Riviere, Augustovski and Rubinstein19;Reference Singer21). However, HTA activities require time and financial resources, and therefore, adapting or using TA reports from other jurisdictions could, at least theoretically, result in improved efficiency, more timely reports and cost savings (Reference Turner, Chase and Milne23). The usage of TA reports from other jurisdictions could be done through an explicit and methodologically supported process of adaptation (Reference Boulenger, Nixon and Drummond6;Reference Chase, Rosten, Turner, Hicks and Milne8–Reference Essers, Seferina and Tjan-Heijnen10;Reference Nixon, Rice and Drummond18;Reference Turner, Chase and Milne22;Reference Urdahl, Manca and Sculpher24;Reference Welte, Feenstra, Jager and Leidl25) but also, as observed in this study, the results of foreign HTA reports are being applied directly to assist in the decision-making process when local information is not available.
Our results suggest that decision makers in LAC countries are using and applying TA reports from other jurisdictions in the majority of situations and that they consider the information from outside LAC to be more useful and applicable. This could be explained by the greater availability of reports from non-LAC countries, or it could be a reflection of the poor quality of local reports.
In addition, researchers in the LAC countries are increasingly relying on evidence from other countries as an input/source to produce local TA reports. They too, consider the information from outside the LAC countries to be more useful, applicable and adaptable than the reports produced in Latin America. This is likely because the existence of credible TA reports helps researchers complete their work more quickly.
This result is expected as researchers use the information contained in other reports as a starting point and they are in a position to adapt or extend this information if necessary. Decision makers, on the other hand, have to apply the information, as in the original report, often under the pressure of a policy or a clinical decision which they cannot avoid or postpone; making it less valuable to them.
In this study, the use of TA reports from other regions exceeds that from Latin America and Caribbean countries. However, the respondents clearly identified local reports from LAC with the greatest potential for transferability and applicability, provided that certain barriers such as lack of transparency, low methodological quality and low production of reports, are overcome. North-South transferability is playing an important role in Latin America, and the respondents are taking advantage of the significant production and availability of TA reports from other regions.
The results obtained in this study may also draw attention to potential threats from using evidence reports from outside the LAC region. Transferability involves the concept of making the results relevant to different settings (Reference Drummond, Barbieri and Cook9) and there are many known limitations to transferability (Reference Augustovski, Iglesias and Manca1;Reference Barbieri, Drummond and Rutten4;Reference Goeree, Burke and O'Reilly13;Reference Sculpher, Pang and Manca20). The application of evidence from other jurisdictions, without first going through a formal process of adaptation, may produce errors. Additionally, what would happen if the original report were to contain errors? Will the errors be passed down to local staff with limited training and very little ability to detect them? In the current situation, and in the best scenario, decision makers, in the absence of local information, are limited to using the information considered to be more transferable (such as description of the technology, the effectiveness or safety). This is not a serious problem but it does suggest that decisions may be being taken without considering important information, such as the economic evaluation or the social and organizational impact of the technology, just because it is not locally available and is regarded as not transferable.
A challenge for the geographical transferability of TA reports is the level of transparency in reporting the study. This means transparent reporting of the methods and results, and in the case of cost-effectiveness studies may also involve access to information such as the underlying mathematical model (Reference Essers, Seferina and Tjan-Heijnen10). The low quantity of available TA reports, lack of local guidelines/methods on how to adapt the evidence, low methodological quality and lack of transparency in the published reports were identified in our study as the main barriers limiting transferability. Fortunately, there have been significant changes in the region and many of these barriers can be largely overcome in the near future. The region is actively moving toward HTA and it is expected that both the quantity and quality of locally relevant evidence reports will significantly improve. Many countries in the region have begun to establish HTA units in their ministries of health, and research units and academic institutions dedicated to supporting these HTA bodies have risen in the last few years. Brazil, Colombia, Cuba, and Mexico have all issued methodological guidelines on the conduct of economic evaluations (Reference Bertozzi, Valencia Mendoza and Hernández Garduño5;Reference Carrasquilla Gutierrez7;11;Reference Gálvez González12;Reference Neto and Nita17), largely based on existing guidelines from developed countries. The Mercosur (Southern Common Market), an economic and political agreement between Argentina, Brazil, Paraguay, and Uruguay, is also about to launch official guidelines for the conduct of HTA and economic evaluations (14). The Andean Community of Nations, a customs union comprising the South American countries of Bolivia, Chile, Colombia, Ecuador, Peru, and Venezuela also is making progress on a similar regional initiative. It is expected that these changes will improve both the quality and quantity of the local production of HTA in the near future, thereby improving the transferability of information across countries. This then has the potential to avoid duplication of work, save efforts and resources and obtain more timely information resulting in a better decision-making process for health resource allocation in the region.
The research reported in this manuscript has limitations we wish to acknowledge. We did not have access to a comprehensive, generalizable, or probability weighted sampling frame from which to select survey recipients. Rather, we used a convenience list of researchers and decision makers in LAC countries supplemented with a selected group of others. In addition, we used a Web-based format to deliver the survey, which limits responses to those who have access to the Internet and who have a Web Browser that can display the interactive survey.
Although the distribution by country and sector of work was very similar to non-respondents, the sample may not be representative. In the majority of cases, the IECS databases contain contact details of those that are interested in HTA (this is one of the main reasons why they are in the database) when in fact a vast majority of decision makers in the region are still not aware of HTA. Further limitations of the study involve the predominance of responses from one country (Argentina), although the main results remained unchanged in the subgroup analyses by countries; and the fact that the survey and the metrics have not been validated. It is also important to note that the respondents’ interpretation of the questions could have been heterogeneous, and the understanding of Spanish by the Portuguese speaking respondents (Brazil) may have been a limitation.
CONCLUSIONS
Countries within Latin America and the Caribbean are rapidly expanding their interest in and use of the tools of health technology assessment in clinical and resources allocation decisions. Lack of technical knowledge, difficulties in accessing and adapting evidence reports, and the institutional fragmentation of the sector are all barriers to the use of HTA methods in the region. Our findings suggest that LAC HTA bodies are increasingly relying on technology assessment reports from outside the region, even when the evidence is considered to be less relevant.
The use of TA reports from other jurisdictions requires adequately trained researchers and decisions makers, and even under the best conditions, may involve risks and limitations. Despite its potential value as a tool in assisting the decision-making process, it is by no means a way of replacing the local production of good quality information, which is one of the greatest challenges facing the region today.
CONTACT INFORMATION
Andres Pichon-Riviere, MD, MSc, PhD, Executive Director, Director of Technology Assessment and Economic Evaluations, Federico Augustovski, MD, MSc, PhD, Director of Health Technology Assessment and Economic Evaluations, Institute for Clinical Effectiveness and Health Policy, Emilio Ravignani 2024 (C1414CPV) Buenos Aires, Argentina; School of Public Health, University of Buenos Aires, Argentina
Sebastián García Martí, MD, MSc, Coordinator of Health Technology Assessment and Economic Evaluations, Institute for Clinical Effectiveness and Health Policy, 2024 (C1414CPV) Buenos Aires, Argentina
Sean D. Sullivan, PPh, PhD, Professor of Pharmacy and Health Services, Pharmaceutical Outcomes Research and Policy Program, University of Washington, Seattle, Washington
Michael Drummond, BSc, MCom, DPhil, Professor of Health Economics, Centre for Health Economics, University of York, York, United Kingdom
CONFLICTS OF INTEREST
Andres Pichon-Riviere has received a grant to his institute from Global Health Research Initiative (a partnership of the Canadian International Development Agency, the Canadian Institutes for Health Research, Health Canada, and the International Development Research Centre). Michael Drummond has received funding for Board membership from International Group for HTA Advancement (funded by Merck and Co.), and has been a member of an expert committee for NICE, an HTA organization. The other authors report they have no potential conflicts of interest.
