Health technology assessment (HTA) has been one of the most popular policy tools in healthcare reimbursement, pricing, and procurement over the last few decades (Reference Löblová1). Several developed countries within and outside Europe have developed national or regional public HTA agencies by the 90s (Reference Banta2). These bodies deal with conducting and critically appraising assessments that examine the short- and long-term consequences of the application or use of technologies. More recently, several low- and middle-income Asian, Latin American, and Central and Eastern European countries have followed suit (3;4). The implementation and the development of HTA structures are well documented. For example, a recent special issue in the International Journal of Technology Assessment in Health Care described the national healthcare systems and the practices of pricing and reimbursement decision making with a special attention on experience and perspectives of HTA implementation in twelve Central, Eastern, and Southern European countries based on accounts from national HTA experts and professionals (Reference Zawada and Mäkelä5). Although the special issue captured a wide scope of countries from the region, one of the largest European countries with a population of over forty million, Ukraine, was not covered. Former studies about Ukraine focused on the pharmaceutical system in general (Reference Zaliska, Piniazhko, Maksymovych, Sichkoriz and Tolubaiev6) or proposed reimbursement frameworks (Reference Zaliska, Piniazhko, Ilyk and Irynchyn7), but no scientific research has been performed specifically on the perspectives of HTA in Ukraine.
Ukraine is facing major challenges in implementing universal health coverage, which is illustrated by limited accessibility and affordability of medicines and the high out-of-pocket expenditure (46 percent of total health expenditure in 2014) (Reference Piniazhko, Zaliska, Ilyk and Stasiv8). The first attempt to introduce HTA was the establishment of the Ukrainian Agency of HTA in 2013, but it was never implemented nationally (9). The need for improving the evidence base of healthcare decision making has been recognized in recent years by the Ministry of Health. In this context, a reform of the regulation of pharmaceutical pricing and reimbursement took place in 2016–17 including the reimbursement program called “Affordable Medicines” introduced in April 2017. The focal points of the national reforms were the development of the National Essential Medicines List (NEML) and the establishment of the NEML Expert Committee (10). Along with the NEML development process, the first steps of HTA implementation within the reforms were a series of HTA training for the NEML Committee members, proposals for HTA institutionalization, and methodological guidelines.
By definition, essential medicines are those that satisfy the priority healthcare needs of a population. According to the WHO guidance, the development of the NEML must be transparent, conducted within a specific timeframe, and should be based on objective and verifiable criteria in order to justify the allocation of public resources (11). HTA standards introduce objectiveness and transparency into the development of the NEML process by its clear problem formulation, consideration of different domains (e.g., efficacy, safety, cost-effectiveness, budget impact), and explicit methodology (Reference Drummond, Schwartz and Jönsson12). Consequently, to make evidence informed decisions on the procurement and reimbursement of medicines as well as to establish NEML, HTA could be a useful policy tool in Ukraine. Notably, a recently published summary of a development project related to the Ukraine NEML establishment declared that “Recommendations on further elaboration and updating of the NEML have been implemented, which [are] the basis for HTA implementation into the public health system of Ukraine” (13).
Although there are similarities between national approaches to implement HTA, there are also major differences in HTA practices among European countries (Reference Allen, Liberti, Walker and Salek14). Even among Central and Eastern European countries, the development status of HTA is heterogeneous, including the adopted assessment and appraisal methods and the implemented procedures (Reference García-Mochón and Espín Balbino15). Given this heterogeneity, a tailor-made HTA implementation roadmap is necessary in the early phases of HTA implementation in countries such as Ukraine by taking into account the opinion of different national stakeholders. Such an approach may facilitate consensus about the establishment of organization structures, procedures, and legal frameworks, and on the other hand, it may restrain potential resistance from key domestic actors.
Our study aimed to describe the environment of HTA at the initiation point of reforms in 2016 and to explore the long-term perspectives for HTA in Ukraine. The current and the preferred HTA status in 10 years (i.e., by 2026) was summarized by capturing the opinion of a wide spectrum of Ukrainian stakeholders.
Methods
We conducted a survey among different stakeholders: the NEML Committee members (with different primary affiliations including academia and healthcare providers), Ministry of Health representatives, and representatives of pharmaceutical companies and patient organizations. We applied purposive sampling, aiming to include individuals with an in-depth understanding of pharmaceutical pricing and procuring in Ukraine. Respondents were identified by the NEML Committee leadership who had an extensive network of Ukrainian individuals with expert knowledge. We used an HTA implementation survey designed to support the formulation of HTA roadmaps (Reference Kaló, Gheorghe, Huic, Csanádi and Kristensen16). The survey has been applied earlier successfully at national and regional level in other countries (17;18).
A glossary of the key terms and expressions related to the HTA implementation roadmap was also distributed with the questionnaire electronically to facilitate the common understanding and prevent misuse of the survey. The glossary contained explanations in a few sentences and examples or where relevant references to the key terms. The survey was distributed electronically with personal invitations to participate in the study. The survey informed participants in a written form that the individual survey participation will be kept strictly confidential and only aggregated results will be disseminated. Respondents were anonymized after receiving the completed answer sheet. All invited stakeholders filled in the survey. Answers were received between June and July 2016. Opinions of the respondents about the HTA implementation in Ukraine were collected and aggregated in eight dimensions: capacity building, HTA financing, process and organizational structure, scope of HTA, decision criteria, standardization of methodology, use of local data, and international collaboration.
Completed surveys were considered valid if not more than three answers were missing or invalid (e.g., providing multiple answers for a single choice question). Results of an invalid survey were excluded from the data set. During the data processing of valid surveys, the invalid answers of the included surveys were not considered; therefore, those had no influence on the results presented in this paper. Individual results were aggregated and the distribution of survey answers was calculated as percentages of the total sample.
As a final step, the interpretation of aggregated results was discussed with the members of the NEML Committee.
Results
A total of thirty-three Ukrainian stakeholders filled in the survey and one survey was considered invalid due to the high number of missing answers. From the total sample of thirty-two included surveys, twenty-one (66 percent) of the participants were employed in the public, nine (28 percent) in the private sector, and two (6 percent) indicated employment in both sectors as multiple affiliations were allowed to report. More specifically, among the respondents, there were ten potential HTA users (decision maker, policy maker, or payer roles), ten individuals with academic affiliations, one public healthcare provider, nine healthcare industry representatives, one consultant, and three representatives of patient organizations. All survey respondents represented Ukraine. Table 1 contains further background details of the survey participants.
Background details of the survey participants
Aggregated answers are reported in three tables, including Table 2 on capacity building and international collaboration, Table 3 on funding and legislation of HTA, and Table 4 on various topics of HTA implementation.
Results on capacity building and international collaboration from HTA implementation survey
Results on funding and legislation from HTA implementation survey
Results on implementation and use of local data from HTA implementation survey
HTA Capacity Building
Regarding HTA education, 48 percent of respondents were not aware of any type of HTA-related training, whereas 42 percent indicated that there was some project-based training in Ukraine. On the other hand, 91 percent preferred having either a graduate or postgraduate training program (41 and 50 percent, respectively) established in 10 years.
International Collaboration
Although 87 percent of participants indicated that collaboration on international HTA courses was limited, everyone preferred changing this practice by participating or even developing such courses with other countries or international organizations. Limited capacity of HTA expertise can be partially overcome by participation at joint international work or reuse of HTA outputs from other countries. Most respondents (79 percent) indicated that Ukraine was not involved in joint international work on HTA or adaptation of HTA documents from other countries. However, in the future, 66 percent of the respondents preferred having active involvement in joint international work (e.g., European Network for Health Technology Assessment [EUnetHTA]). However, only half of the participants (50 percent) indicated that joint works could be adapted to the Ukrainian context and 47 percent preferred adapting works performed by foreign HTA bodies.
HTA Funding
Experts reported limited funding for HTA as around half of the respondents indicated a lack of funding for the critical appraisal process (55 percent) and for the research part of HTA (48 percent). Nevertheless, in the future, the majority of participants (81 percent) would like to ensure sufficient funding from public resources for HTA research activities. For financing the critical appraisal process in HTA, the majority of participants preferred the public funding to private funding (e.g., submission fees by pharmaceutical companies) in the future (83 vs. 17 percent).
Legislation on HTA
More than half of the participants (61 percent) indicated that HTA had no formal role in the decision-making process, whereas some indicated that international HTA evidence was considered. All participants preferred relying more HTA evidence based on local data in the future. However, 69 percent of them preferred that locally developed evidence should have an additional role without mandating it, and on the other hand, 31 percent indicated that it should be mandatory in 10 years. Regarding the organizational structure, the most preferred option among respondents (52 percent) was the establishment of a public HTA institute with academic support. Only the minority of the respondents (26 percent) considered that several HTA bodies with central coordination would be the preferred option.
Scope of HTA Implementation
Although most respondents reported that HTA was not applied for any type of health technologies in 2016, all of them preferred using it for pharmaceuticals in the future. The majority of respondents also preferred expanding the scope of HTA beyond pharmaceuticals for evaluating medical devices (74 percent) and prevention programs (81 percent). Some respondents (55 percent) indicated that HTA should be used even for evaluating surgical interventions. The majority of the participants (84 percent) would introduce HTA for all new health technologies applying for public reimbursement together with an additional revision process of the former assessments. Only the minority (9 percent) of experts considered that only those new technologies should be evaluated in the future, which indicate high budget impact.
Decision Criteria
Almost all of the respondents (97 percent) preferred applying the cost-effectiveness and the healthcare priority criteria for decision making of public reimbursement. Budget impact (80 percent) and therapeutic value (77 percent) were rated as the third and fourth most important criteria in future decisions, whereas unmet medical need received the lowest score (30 percent). None of the participants preferred introducing implicit thresholds for these criteria in the future. The majority of experts (87 percent) preferred applying soft explicit thresholds, which would allow some deviation in exceptional cases (e.g., acceptance of higher incremental cost-effectiveness ratio for orphan medicines). Few experts (10 percent) preferred using hard explicit thresholds without exceptional cases. Seventy-two percent of the respondents indicated that multi-criteria decision analysis (MCDA) would be a preferred method in the future HTA framework.
Quality and Transparency of HTA Implementation
In 2016, there was a lack of quality assurance elements implemented to support the use of HTA for decision making in Ukraine. In the future, 87 percent of respondents preferred having a publicly available critical appraisal checklist for standardizing the evaluation process of HTA submissions. Sixty-one percent preferred having published methodological guidelines for the preparation of HTA documents, and 58 percent would introduce the practice of regular follow-up research on previous HTA recommendations. Almost all respondents (88 percent) preferred having full transparency by making the HTA agency's recommendations and the related appraisal reports available in the public domain, whereas the remaining participants preferred publishing only the HTA recommendations. Nearly all participants (97 percent) preferred introducing transparent and continuous timelines for the appraisal of the HTA submissions and for publishing the following recommendations.
Use of Local Data
The great majority of respondents (93 percent) indicated that in 2016 using local data as the inputs for the assessment of health technologies was not mandatory in Ukraine. On the other hand, 94 percent preferred using local data to develop HTA evidence, or in case it was not available, they would mandate the transferability assessment of international HTA evidence whether it could be used for local decisions. Respondents reported that there was a lack of local/national registries for data collection, and the accessibility of payers' database was limited. Overwhelmingly, they (90 percent) preferred changing this practice and investing into up to date registries and practices to make payers' databases available for preparing evidence based on local data for HTA submissions.
Comparing the Opinion of Public versus Private Sector Respondents
Survey answers were analyzed separately for those who indicated employment in the private or in the public sector. The opinion of these two groups of stakeholders related to the capacity building, HTA funding, scope of HTA, local data, and international collaboration was similar (i.e., they preferred similar options for all the questions in these dimensions). They also had the same preferences regarding transparency (i.e., publishing HTA recommendations with appraisal reports) and timeliness (i.e., transparent and continuous timelines for the appraisal and issuing recommendations).
Respondents' opinion differed more substantially in five areas. More public sector respondents preferred introducing follow-up evaluations of former HTA recommendations (65 percent public vs. 40 percent private). Their opinion also differed related to the decision criteria that should be applied. Although for both groups, the cost-effectiveness and the healthcare priority were the most important criteria, budget impact ranked higher for public stakeholders (86 percent) compared to private stakeholders (60 percent). The difference was also observed in the decision threshold question. Although for both groups, soft explicit thresholds were the most preferred option (95 percent for public vs. 64 percent private), some private sector stakeholders preferred having explicit hard threshold (27 percent) or oppositely no thresholds (9 percent). Their opinion also differed regarding HTA legislation. Less private sector respondents preferred mandating the use of local data for HTA evaluation (39 percent public vs. 18 percent private) and consequently more private sector respondents preferred only an additional (i.e., not primary) role for the local data used for HTA evidence (61 percent public vs. 82 percent private). Regarding the organizational structure, more private sector stakeholders preferred introducing a public HTA institute with academic experts (41 percent public vs. 73 percent private). Some public sector respondents preferred several HTA bodies with central coordination, whereas this option was less preferred by private sector respondents (32 percent public vs. 9 percent private).
Discussion
Recommendations for Facilitating HTA Implementation
Our manuscript describes the initial results of an ongoing policy initiative, which has to be followed by additional steps to clarify further details. However, our findings could already identify several key points which could facilitate implementation of HTA in Ukraine.
First, the availability of trained human resources is crucial to implement HTA. Our survey indicates that graduate and postgraduate education should be established. Experts from such programs may contribute to different stages of the HTA process: preparing submissions in line with official requirements, delivering high-quality appraisals of submissions, and conducting HTA-related policy research activities (e.g., continuous improvement of HTA methodologies or timely revision of former submissions). Trained experts enable meaningful participation in international collaborations and adoption of international research in the national context. There are several options for increasing the number of HTA professionals, including graduate and postgraduate education and training programs.
Second, stakeholders conceded that sufficient public resources should be available in the development and operation of HTA in Ukraine. Therefore, political commitment to allocate public funding for continuing and expanding already initiated activities is paramount.
Third, survey results indicated that in certain categories (e.g., cost) the locally collected data in HTA submissions should receive higher priority and decisions should not depend solely on evidence generated in other countries. Further, public resource allocation decisions of new health technologies should be based on the local priority ranking of disease areas and cost-effectiveness of the intervention, as these were the most important criteria by survey respondents.
Fourth, the set-up of the HTA process should be designed in collaboration with academic professionals. National stakeholders had a clear preference for a transparent HTA system. Investment and academic input are needed to develop quality assurance elements in order to increase the objectivity and transparency of HTA recommendations.
Fifth, to maximize the potential for HTA in Ukraine, a multi-stakeholder approach in designing and periodically revising the HTA system is necessary. Our findings and recommendations were formulated according to the opinions of national healthcare policy makers, members of the NEML Committee, pharmaceutical industry representatives, and even patient organization representatives. Although for most dimensions of HTA implementation there were no differences in the preferred long-term objectives, in some questions (e.g., legislation of HTA, decision criteria, and introducing revision process of former decisions) further research and discussion are needed between public and private sector stakeholders to find the common ground.
Recent Developments in Ukraine
Recently, some improvements have been implemented regarding the quality and transparency of the HTA process in Ukraine, which are in line with the proposed recommendations and the envisioned 10-year perspectives in the current HTA roadmap. After the Ukraine NEML was established (19), HTA was included in the Fundamentals of Ukrainian Health Law, which was approved by the Ukraine Parliament in 2017 (20). Following these legal steps of HTA institutionalization, the NEML Committee developed a draft guideline for outlining the requirements of HTA submissions, which was made publicly available to receive feedback (21). The NEML also moved toward transparency by making the HTA applications by pharmaceutical companies and the corresponding recommendations by the Committee available on its website (22). Furthermore, EML Committee members have initiated research on quantifying the cost-effectiveness threshold for Ukraine and considering its applicability for healthcare decisions (Reference Topachevskyi, Piniazhko, Lebega and Oleshchuk23). International collaborations have been also established by experts in the NEML Committee who are active participants of the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) organization. The Committee also joined the EUnetHTA as an associate partner.
Opportunities for Learning from Other Countries
Ukraine can certainly learn a lot from countries with the similar healthcare system, economic status, or cultural and social values (e.g., Central and Eastern European countries) and take advantage of some good practices. To increase the number of experts in the country, the example of Hungary could be followed where the spectrum of available HTA education ranges from 1-week short courses to accredited postgraduate programs (Reference Drummond, Schwartz and Jönsson12). Regarding the participation in international networks to increase the awareness of HTA, positive experiences of Croatia or Slovakia could be mentioned (24;25). The transparency of the Polish HTA system shows that with appropriate legislation and corresponding online platform, the full HTA submissions, corresponding critical evaluations, and recommendations can be published. Notably, in certain aspects, the Polish system was shown to be more transparent than the NICE in England, such as providing summaries of expert opinions on the HTA submissions (Reference Ozierański, Löblová and Nicholls26).
On the other hand, the potential challenges and suboptimal policies can also be identified based on the examples of other countries so Ukraine can learn from the shortfalls in the practices followed. The examples of Bulgaria and Poland show that there is an important need to harmonize the organizational responsibilities related to HTA and the corresponding legislation of the pricing and reimbursement of health technologies. In Bulgaria, there was a controversy concerning which public institute should deal with HTA methodology and implement HTA procedures and how HTA recommendations should be taken into account in pricing and reimbursement decisions (Reference Benisheva-Dimitrova, Sidjimova, Cherneva and Kralimarkov27). Reportedly, in Poland, it is still required to achieve better consistency between HTA recommendations and national reimbursement decisions (Reference Lipska, McAuslane, Leufkens and Hövels28). In Romania, only international cost-effectiveness evidence is taken into account in reimbursement decisions without considering transferability, which makes the results of national cost-effectiveness studies irrelevant (Reference Scintee and Ciutan29). Finally, in Hungary, the lack of transparency and the low level of stakeholder engagement limit the potential of HTA to support evidence-based decision making and marginalize the influence of the national HTA agency on pricing and reimbursement decisions (Reference Csanádi, Löblová and Ozierański30).
Limitations
The most important limitation of our study is the relatively low sample size of survey respondents. Unfortunately, room for recruitment of more participants was fairly limited due to the low number of experts with a good understanding of HTA principles in Ukraine. This limitation has an even more important implication on the findings of the subgroup analysis by different employment status (public vs. private), where sample sizes were even lower.
Conclusion
To reflect specific national and regional characteristics, it would be important for Ukraine that HTA methodology development and implementation are based on a clear roadmap. Frequent changes in strategic directions of healthcare governance make the consistent implementation of longer-term investments, such as HTA capacity building or institutional setup, challenging. This study attempted to define these long-term objectives based on the opinion of Ukrainian professionals with in-depth knowledge of HTA practices and thorough understanding of the national health policy objectives, clinical practices, unmet medical needs, and economic constraints. This process might be supported by the guidance and experience of international experts and by understanding the good practices and pitfalls from other countries. There were important improvements recently but further steps outlined by our roadmap are required to complete. Learning from the mistakes or the good practices from other countries, however, may make the long and bumpy road of HTA implementation for Ukraine smoother and more impactful.
