A systematic evaluation of the level and trends in development of health technology assessment (HTA) is lacking. Previous studies generally focus on the best way to perform HTA (1). These include initiatives that aim to support the development of co-operation among HTA institutions such as Eur-Assess (1994–97), HTA Europe (1997–99), ECHTA/ECHAHI (European Collaboration for Assessment of Health Interventions) project (1999–01), EUnetHTA Project (2006–08), EUnetHTA Collaboration (2008–09), EUnetHTA Joint Action 1 (2010–12), and the International Network of Agencies for HTA (INAHTA). For example, the ECHTA/ECHAHI project developed a best practice of the HTA process based on well-recognized and internationally agreed characteristics along with a common understanding of the HTA process (Reference Børlum Kristensen and Palmhoj Nielsen2;Reference Busse, Orvain and Velasco3). This guidance was further developed and implemented in the EUnetHTA Collaboration and its successors, EUnetHTA Joint Action 1 and 2 (Reference Lampe, Makela and Velasco Garrido4).
Complementary to these initiatives, the objective of our study was to develop and apply an instrument to map the level of HTA at country level in selected countries.
METHODS
Development and Adjustment of the Instrument
In developing the instrument we distinguished two key elements of HTA: (i) the institutionalization of HTA and (ii) the HTA process itself.
The main sources used include EUnetHTA tools such as the Handbook on Health Technology Assessment Capacity Building (Reference Busse, Orvain and Velasco3), the report on best practice in undertaking and reporting HTA of the ECHTA/ECHAHI project (5), guidance of the International Information Network on New and Changing Health Technologies (EuroScan) regarding effective early warning systems (Reference Murphy, Packer and Stevens6), the INAHTA checklist to present HTA information (Reference Hailey7) and the principles for HTA programs in different countries of the International Working Group for HTA Advancement (Reference Drummond, Schwartz and Jonsson8).
The draft instrument consisted of the following domains: (i) institutionalization of HTA; (ii) identification; (iii) priority setting; (iv) assessment; (v) appraisal; (vi) reporting; (vii) dissemination; and (viii) implementation of HTA in policy and practice. The categories ii–viii have been used in the EUnetHTA Collaboration to describe the HTA process (5).
For each domain a set of criteria was determined along with indicators on how to measure each criterion using the key documents stated above. For example, an indicator for institutionalization is membership of the international society for the promotion of HTA (HTAi) and membership of INAHTA. The latter requires that the agency is not-for-profit, funded at least 50 percent from public sources and has a national / regional function (Reference Busse, Orvain and Velasco3). However, not all countries have formal HTA programs in place. We therefore added two indicators that reflect a less well institutionalized situation. In addition, a simple scoring system for each criterion was developed consisting of a grading scale that differentiates between the different levels of HTA development in a country. For example, we decided that an established agency that is a member of INAHTA reflects a more institutionalized level of HTA than a government-advising group outside INAHTA or the presence of elements to establish a (formal) HTA program if no agency or group is in place. Therefore, the score of a country with at least one national or regional agency/organization that is a member of INAHTA is higher (25) than the score of a country without a formal HTA program (i.e., a government-advising group outside INAHTA (score: 20) or no agency or group but elements of establishing a formal program are in place (score: 24 at the maximum) (see Table 1).
The Mapping Instrument
The instrument was adjusted by asking HTA experts with a strong interest in middle-income countries to give feedback. First, the instrument was sent by email to members (n = 75) of the interest sub-group on HTA in developing countries of HTAi (HTAi DC ISG) in June 2011. Second, the attendees (approximately 25) of the HTAi DC ISG meeting in Brazil (July 2011) were asked to review the instrument. Third, we asked a small sample of active HTAi members (n = 10), including HTAi DC ISG members as reviewers. In total, we received comments from six experts: two from India, two from Australia, one from China, and one from Poland. The respondents are well-known HTA experts in their respective countries. Their comments were used in developing the final version of the instrument.
Selection of Countries
To identify middle-income countries, we used the definition of the World Bank. Lower-middle-income countries are defined as countries with GDP/capita between $1,006 and $3,975 and upper-middle-income countries range between $3,975 and $12,275 (9). We aimed to include countries from different regions (Latin America and the Caribbean, Asia, and Europe) that undertake, according to our experience and knowledge at the start of the study, different levels of HTA activities. Argentina, Brazil (recently well developed HTA systems), Malaysia, Mexico (moderately developed HTA systems), India, Indonesia, and Russia (no or some elements of HTA systems in place) were selected.
For benchmarking purposes three reference countries with well developed HTA systems from different regions (Asia-Pacific, North America, and Europe) were selected, namely, Australia, Canada, and the United Kingdom (England and Wales). These countries are often used in international comparisons of HTA systems.
Document Review
To map the level of HTA relevant information from the (gray) literature for each country was collected and reviewed. For some countries we had collected information in a previous study on the role of HTA in middle-income countries, including Argentina, Brazil, and Mexico (Reference Oortwijn, Mathijssen and Banta10). In the current study we used the same search strategy as published in 2010, avoiding overlap and duplication to the maximum. In addition, multiple databases were searched (World Health Organization, European Observatory on Health Systems and Policies, Organization for Economic Cooperation and Development, World Bank, and PubMed/Medline) to identify recent publications. Also, the International Journal of Technology Assessment in Health Care was searched for relevant articles. We used the following search terms: <country> AND health system OR healthcare system; health insurance; reimbursement of pharmaceuticals; and health technology assessment. Searches were limited to publications from 2000 to 2011 (inclusive) that are available in English and in the public domain. Moreover, an Internet-based review of wider information sources was performed (i.e., professional organizations at international level: INAHTA, HTAi, and International Society for Pharmacoeconomics and Outcomes Research regarding specific information on the selected countries) using the same search terms. Reference lists of retrieved documents have been hand searched to identify additional publications of interest. Articles or information that were targeted at either the institutionalization of HTA or elements of the HTA process were included in the study.
Web-Based Survey
A Web-based survey (in English) was distributed to key HTA experts in the selected countries. The survey consisted of forty-four close-ended and open-ended questions focusing on the main domains of the instrument. The final two sections of the survey provided opportunities for further comments on future developments regarding HTA in their country, as well as general comments on the survey.
Before launch the survey was piloted with two potential respondents (one from a middle-income country and one from a reference country) to ensure that it functioned properly from a technical point of view, as well as to confirm that the questions could be understood and were relevant. Some revisions were made after this step.
We targeted 449 key informants, representing Ministries of Health, HTA agencies, university/research organizations, third party payers, medical device industry, pharmaceutical industry, and regulatory authorities. Potential respondents were identified by means of our own networks, the survey database from our former study (Reference Oortwijn, Mathijssen and Banta10), the HTAi annual meeting in Brazil (2011), HTAi Membership Directories (2010, 2008, and 2007), authors of relevant articles, presenters at relevant conferences, and members of NEVALAT (thematic network on the economic evaluation of healthcare program and its application in decision making in Latin American countries). After correcting for bounces and invitations sent to people with no or limited knowledge of the subject, the final panel consisted of 385 persons.
The survey was distributed by email through Check Market on September 30th 2011 and was live for completion until November 4th 2011. To maximize response rates two reminders were sent to respondents who had not or only partially completed the survey. Responses with at least 50 percent of the survey completed were considered as partial and included for analysis. The survey responses (65/385, see Table 2) were analyzed by country using IBM SPPS Statistics (version 19.0 for Windows).
Response Rate per Country
Level of HTA Development per Domain
*Scoring mainly based on survey.
Determining the Level of HTA
The information from the document review and Web-based survey was used to determine the score for each criterion (see Table 1). The scores were assigned by the lead author (WO) on the basis of the answer given by the majority of the respondents in a country, in conjunction with the evidence found in the literature. The scoring was reviewed by another researcher (PB). Any disagreement was resolved in a consensus meeting. These scores were then added resulting in a total score per domain. These scores were reviewed by the other members of the project team (H.V. and D.B.). We used this score for determining the level of HTA in the countries selected and for benchmarking purposes.
RESULTS
Institutionalization of HTA
The level of institutionalization is the highest in the reference countries. These countries all obtained the maximum score of 28. Brazil has a comparable degree of institutionalization. The institutionalization of HTA in Argentina, Mexico, and Malaysia is lower, as these countries have a limited number of HTAi members. Institutionalization is lowest in Russia, Indonesia, and India as indicated by the lack of a formal HTA program, the limited number of HTAi members, as well as lack of political commitment to HTA (see Table 3).
In the countries with a formal HTA program, there is at least one organization that is a member of INAHTA. They differ, however, in the way these agencies interact with providers and purchasers of health care. Canada is complex, with several programs at federal, provincial, and local levels. Also, in Latin America (Argentina, Brazil, Mexico), more actors are involved in decision making (11). In the countries where no formal HTA program exists, decision making about the adoption and use of health technologies may be carried out by health authorities and health service providers. Decisions, however, are frequently based on interests of individuals or “gut feelings.” At best, decisions take into account experience generated in other countries or selective expert advice. The challenge is to shift to a decision-making process that follows the principles such as those of evidence-based medicine, cost-effectiveness, and patient centered services (5). With regard to institutional set-up, both Indonesia and Russia have made some progress. There is some, but limited, capacity available to search the Internet and review international literature. In addition, there are several training opportunities regarding pharmaco-economics and outcomes research in India and Russia.
Identification of Health Technologies
We determined if a country has an early warning or horizon scanning system in place. If not, we mapped the extent to which early warning activities are carried out. The maximum score that could be obtained was 19. Australia, Canada, and the United Kingdom have an early warning system(s) or horizon scanning system(s) in place. For India, Mexico, and Russia, we found no literature on identification of health technologies. Of the survey respondents from the middle-income countries, 75 percent answered that at least one monitoring system to identify technologies is in place in their respective countries. For Argentina and Brazil, we found evidence of such a system but not for Indonesia and Malaysia. In Mexico and Russia, half of the respondents do not think that such a monitoring system is present in their country. More than half (55 percent) of the respondents confirm the presence of other identification activities (such as review of medical journals) described in a publicly available document.
Priority Setting
We collected information on the existence of a priority setting process including: (i) An explicit and transparent process that is replicable (criteria and procedures); (ii) A process that reflects the goals of (national, regional, local) health policy; (iii) Attention to stakeholder involvement; (iv) Explicitly stated information on priorities; (v) and An evaluation of processes and outcomes.
In addition, we identified countries that have a system for reviewing international evidence (literature) to set priorities. For each indicator, a maximum score of 3 could be obtained, resulting in a total maximum score of 18.
We found a mixed picture with regard to the different elements of a priority setting process (see Table 3). An explicit system exists in Australia, Canada and the United Kingdom, as well as in Brazil and Malaysia. We found limited evidence in the literature on priority setting in India, Indonesia and Russia. The majority of the survey respondents in all countries believe that the process reflects the goals of health policy, that there is, at least to some extent, stakeholder involvement and that the information on priorities set is available. It appears that evaluation is often not performed in the countries under study, except in the United Kingdom and Brazil. This observation also applies to the presence of a system for reviewing the international evidence base.
Assessment
A document describing a clear goal and scope of HTA is available in Australia, Canada, the United Kingdom, as well as in Malaysia and Mexico. All countries except India provide a clear description of the safety and clinical effectiveness of the technology in their assessments. Cost and cost-effectiveness information are well described in the reference countries, Brazil and Mexico. The other aspects (ethical, organizational, social, and legal aspects) are less examined, as in the rest of the world.
Argentina, Brazil, India, Indonesia, Mexico, and Russia score low with regard to an explicit and systematic approach to classify and critically appraise the quality of available studies. Generalizability and transferability are not well described in Argentina, Indonesia, Russia, and Mexico.
According to the survey results, the collection of new primary data is well described in Australia, the United Kingdom, as well as in Brazil. Conducting systematic reviews / meta-analysis is covered in the reference countries and in Brazil and Mexico. This also applies to literature searches in different languages, which are also described in Malaysia (see Table 3). For each of the 13 indicators reflecting the assessment phase, a maximum score of 3 could be obtained. This means that in total each country could obtain a score of 39.
Appraisal
The concept of appraisal was consolidated and formalized with the establishment of the National Institute of Health and Clinical Excellence in 1999. Appraisal is a consideration of the outputs of the assessment process within the context of additional information supplied by relevant parties. Although appraisal is often not separated from the assessment phase, we tried to identify whether a transparent and deliberative system exists. In total, countries could obtain a maximum score of 9. It appears that the reference countries all have appraisal systems in place, although there is some criticism regarding transparency of these systems in Canada. Also, Brazil is taking steps toward a more explicit process of decision making. Argentina, India and Russia do not have a clear and transparent system in place. Indonesia, Malaysia, and Mexico have some elements of an appraisal system (see Table 3).
Reporting
With regard to the reporting of HTA information, we examined whether reporting is unbiased and according to a transparent process. Also, we identified the total number of HTA reports produced annually using public resources and the total number of HTA reports produced by applicants for reimbursements. With regard to these indicators, countries could obtain a maximum score of 11.
In all countries, except for India and Russia, it appears that there is at least one agency/organization that uses (some sort of) guidelines on reporting (e.g., ECHTA report or INAHTA checklist for HTA reports). In Australia, Canada, and the United Kingdom, several HTA reports (i.e., more than twelve reports per year) are published on an annual basis, which is likely due to a well-established HTA infrastructure in those countries. This also includes a relatively high number of reports produced by applicants for reimbursement (i.e., more than twelve reports per year) (see Table 3). We found similar results for Brazil.
Dissemination
HTA information needs to be timely and effectively communicated to be valuable for decision makers. A publicly available dissemination strategy is important. We, therefore, examined whether the information (HTA report) is disseminated to decision makers before the decision on a particular technology is made (see Table 1). The maximum score for each of the indicators was 3, which totals the maximum score to 12 for this domain.
Dissemination strategies are well developed in the reference countries, and to a lesser extent in Brazil, Argentina, Malaysia, Mexico, and Indonesia. This might be due to the existence of HTA agencies or HTA unit(s) in those countries. More than 80 percent of survey respondents state that clear recommendations for target groups are at least to some extent present in publicly available dissemination strategies. Clear information on dissemination strategies in India and Russia is lacking (see Table 3).
Implementation in Policy and Practice
HTA tends to have a higher profile in coverage decision making when these processes are explicit, deliberative and formalized (i.e., HTA is integrated and enforced by law). We examined whether (a) at least one organization involved in HTA has a legal mandate in the healthcare system, (b) there is a link between HTA and the regulatory process, and (c) an implementation plan is used. In addition, we looked at the extent to which the actual impact of HTA is measured.
Almost 63 percent of the survey respondents believe that HTA organizations in their countries have a legal mandate. In the literature, we found evidence for such a mandate in the reference countries, Argentina, Brazil, Indonesia, and Mexico.
There is a clear link between HTA and a regulatory framework in the reference countries as well as in Argentina, Brazil, and Malaysia. The other countries showed divergent outcomes, although the majority believes that there is a link between HTA and the regulatory process, with the exception of Russia and India. This was also confirmed by the literature.
Regarding implementation plans, almost 90 percent of the respondents state that factors influencing the implementation of HTA are taken into account when HTA is used in their country, at least to some extent. However, this does not apply to India and Russia (see Table 3).
Measuring the impact of HTA is becoming more frequent in the HTA community. However, only a few studies have been undertaken to better understand the issue (Reference Wilsdon and Serota1,Reference Oortwijn, Mathijssen and Banta10). Measuring HTA impact by using monitoring systems is currently most advanced in the reference countries.
For this domain, a maximum score of 10 could be obtained (see Table 1).
DISCUSSION
With this study, we go beyond previous studies on the organization of HTA by focusing on a more systematic evaluation of the level and trends in HTA development at country level, taking into account the characteristics of the healthcare system. Of particular interest is Brazil, which is developing rapidly toward the standards of best practice in HTA. Political commitment in Argentina and Mexico will determine how quickly these countries can further develop and how the actions already undertaken will have an impact. For Malaysia, the main challenge is to sustain the established HTA framework while training (new) personnel. Indonesia has taken some steps toward developing HTA capacity, although the focus is mainly on pharmaco-economics. India and Russia are still at the very beginning of introducing HTA in their countries. Political will is obviously important for introducing and maintaining HTA in a country.
There are strengths and weaknesses to our approach. A strength is that we received minor comments from relevant stakeholders and respondents during the development and adjustment phases of the instrument. These focused on the level at which HTA should be mapped in each country: national and/or provincial level and/or local (e.g., Canada).
One of the restrictions of this research lies in the scope of the literature retrieved (limited to publications that are available in English). A Web-based survey was performed to collect additional information for each country to support and enrich the profiles with information which is not publicly available.
Implementing the survey highlighted several challenges. First, we address the issue of identifying key persons working in the HTA field. For countries with more established HTA activities (e.g., Brazil), it was easier to identify a large number of contact persons compared with countries with less established HTA activities (e.g., Malaysia, Indonesia, India, and Russia). Second, the initial response rate of the survey was relatively low. We found that this was partly due to the fact that our survey was likely distributed as SPAM (with survey as a trigger word). Efforts have been made to ensure a good response rate, with reminder emails sent 10 days after the first invitation was launched to all those stakeholders who had not responded to the survey. Reminders were also sent to stakeholders on the panel who had only partially responded to the survey, 1 day after their survey was received. A subsequent reminder was sent to all stakeholder panel members who had not responded as we suspected that our survey had been filtered as SPAM. After sending the reminder, we expected the response rate to increase. As we encountered no significant increase in response rate, we decided to search for additional contact information. After sending a second reminder through our own system, it was found that a large number (n = 52) of the initial email addresses were bounced. As a result, we decided to sent a final reminder to contacts from countries with a very low response and to contacts with corrected email addresses (n = 82). Furthermore, the nonresponse in most countries with less established HTA activities was relatively higher than in other countries. This might be explained by a lack of interest for the subject or people considering themselves not ‘qualified enough’ to answer the survey. Another issue related to the relatively low response rate could be the language barrier (e.g., in Argentina, Brazil, Malaysia, Mexico, Russia, and Indonesia). Also, a few respondents indicated that they had difficulties with the phrasing of the questions. In addition, we found that the results of the survey demonstrated sometimes contradictory views regarding several questions and often conflicted with the evidence retrieved from other sources, especially for the developing countries. For some of these countries, HTA activities are rather new (e.g., India, Indonesia, and Russia). In other cases, the number of survey respondents was relatively low for some of the countries (e.g., Argentina, India, Indonesia, and Mexico). In general, the survey scores for Canada appeared to be relatively low compared with the other reference countries. This might be explained by the fragmentation in the HTA process in Canada; it was not always clear whether the respondent related to the federal or provincial perspective. The low number of respondents has implications for the reliability of the findings from the survey. For future use, it is necessary to require a minimum number of informants per country.
Another limitation of the study is the assignment of scores using desk research and survey as main sources. Although for some indicators the evidence is clear (e.g. membership of INAHTA, EuroScan), for others (e.g., (elements of a) system for priority setting) it was less clear. In these cases, we used the experiences of the project team, but this might result in an over- or underestimation of the score for each country. Taking these limitations into account, we base our summarizing conclusions mainly on the findings from the document review.
CONCLUSIONS
Mapping of HTA at the country level is feasible and can be done by focusing on the level of institutionalization and the process of HTA. This includes the identification, priority setting, assessment, appraisal, reporting, dissemination, and implementation of HTA results in policy and practice.
The results of this first mapping exercise can be used as a baseline measurement for future evaluation. It would be beneficial to evaluate how the different selected countries would score on the different indicators after 3–5 years (especially those countries that have announced healthcare reforms and changes in the HTA processes). Ideally, progress should be monitored on a regular basis to identify trends in the indicators (e.g., publication of reports and HTAi memberships could be monitored on a yearly basis). This can help different actors (governments, HTA organizations, industry, other stakeholders) to assess the development of HTA at country level, help inform HTA strategies, and justify expenditure for HTA.
The selection of countries includes both lower-middle-income countries and upper-middle-income countries from different regions. This strengthens the argument that the instrument can be applied to other middle-income countries and probably low-income countries too. Before applying the instrument to other countries, it needs, however, further validation.
CONTACT INFORMATION
Wija Oortwijn, PhD (wija.oortwijn@ecorys.com), ECORYS Nederland BV, Rotterdam, the Netherlands
Pieter Broos, MSc, ECORYS Nederland BV, Rotterdam, the Netherlands, Dutch Order of Medical Specialists, Utrecht, the Netherlands
Hindrik Vondeling, Professor, PhD, University of Southern Denmark, Odense, Denmark, University of Twente, Enschede, the Netherlands
David Banta, Professor, MD, MPH, ECORYS Nederland BV, Rotterdam, the Netherlands
Lora Todorova, MPH, Novo Nordisk International Operations A/S, Zurich, Switzerland
CONFLICTS OF INTEREST
Wija Oortwijn reports consulting fees, travel support and payment for writing to her institution from GSK and Novo Nordisk International Operations A/S; and employment at and stock or stock options from Ecorys Nederland B.V. Pieter Broos reports consulting fees, travel support and payment for writing to his institution from GSK and Novo Nordisk International Operations A/S. Hindrik Vondeling reports consulting fees, travel support and payment for writing to his institution from Ecorys Nederland B.V., partly provided by Novo Nordisk International Operations A/S. David Banta reports consulting fees and payment for writing to his institution from Ecorys Nederland B.V., partly provided by Novo Nordisk International Operations A/S. Lora Todorova reports support for travel from Novo Nordisk International Operations A/S which is her employer.
