Health technology assessment (HTA) is the systematic evaluation of the properties and effects of a health technology, addressing the direct and intended effects of this technology, as well as its indirect and unintended consequences, and aimed mainly at informing decision making regarding health technologies (1). As science and technology are advancing rapidly, it has become increasingly important to conduct HTA to promote the appropriate use of new health technology while avoiding potential negative impacts.
Since HTA was introduced to China in the 1980s, some progress has been made in terms of establishing HTA research agencies and developing HTA as an academic discipline (2;3). HTA was initiated in China through several health systems development projects funded by the World Bank. In 1994, the first Medical Technology Assessment Research Center at Shanghai Medical University was established with the support of the Ministry of Health (MoH) (Reference Chen, Banta and Tang3). However, HTA was not successfully systematically integrated into the policy cycle. Although it has been used occasionally by policy makers, it most often existed in the academic sector. A recent analysis by Chen et al. of the barriers to HTA development in China found that government leadership has limited awareness of HTA and does not value its importance for informing health policy and decision making (Reference Chen, He and Chi2). Furthermore, there has been no systematic evaluation of the level of development of HTA in China and how it compares to that in high-income countries and other middle-income countries. We conducted this study to elicit diverse stakeholder views in China in order to inform policies and decisions to promote further development of HTA in China.
Methods
Study Design and Participants
This was a cross-sectional study in which we sent a questionnaire to stakeholders who were familiar with the field of HTA in China from June to November 2016. Specifically, we surveyed policy makers, researchers, and representatives from the pharmaceutical and medical device industry and health providers. The potential respondents were identified through different means, starting with convenience sampling and snowball sampling. We started with known HTA experts, and asked them to identify other stakeholders who fit the criteria to enroll in the study. In addition, we searched attendant lists of major HTA conferences in China as well as Chinese journals in the field of HTA that were issues between 2013 and 2015.
Survey Instrument
In the questionnaire that we distributed, we included the Mapping of HTA Instrument that has been used to assess respondents' views regarding the level of HTA development in ten countries: Argentina, Australia, Brazil, Canada, India, Indonesia, Malaysia, Mexico, Russia, and the United Kingdom (Reference Oortwijn, Broos and Vondeling4). The Mapping of HTA Instrument is based on multiple HTA guidelines and standards that had been adopted broadly as well as discussions among many international HTA experts. The instrument distinguishes two key elements: (1) the institutionalization of HTA and (2) the HTA process. In particular, the instrument measures HTA status or performance in eight domains, that is, institutionalization (max. score 28), identification of health technology (max. score 19), priority setting (max. score 18), assessment (max. score 39), appraisal protocol (max. score 9), reporting (max. score 11), dissemination of findings and conclusions (max. score 12), and implementation in policy and practice (max. score 10). The eight domains and their indicators as well as the scores (max. total score 146) that can be obtained for each domain are listed in Supplementary Table 1. In our questionnaire, we asked respondents to what extent each of the domains of HTA development were present in either dichotomous (Yes/No) or Likert format in a range of 0 through 3. Before distributing the questionnaire, we translated the Mapping of HTA Instrument into Chinese and added a section on demographic information. We then piloted the questionnaire among a group of Chinese experts in order to receive feedback on how we could improve or adapt the questionnaire.
To accommodate the differences in how people would prefer to fill out the questionnaire and to maximize the response rate, we distributed the questionnaire in two formats. One was an online version that we created and managed using WJX (Enterprise Edition), one of the major online platforms for investigation, surveying, and examination in China, enabling participants to fill out the questionnaire by using a computer or mobile device. The other format was an electronic version that we created in Microsoft Word, which participants could download and fill out either on a computer or on hardcopy, and then return via email, mail, or in person. The questionnaires that were answered online or returned as an electronic file were encrypted and double-checked to ensure data privacy, security, and validity. To make the questionnaire user-friendly, we adopted a simple layout and sophisticated conditional questions and skip logic based on participants' responses.
Data Analysis
Since some respondents submitted their results through channels other than WJX, we first entered those results into WJX. We then exported all results from WJX as a Microsoft Excel file and used IBM SPSS Statistics (version 21.0) to analyze the data. We conducted a descriptive analysis of the questionnaire results to show the level of HTA development in China in terms of the eight domains in the Mapping of HTA Instrument, as well as the overall level of HTA development.
The Mapping of HTA Instrument has been used to assess the level of HTA development in seven middle-income countries (Argentina, Brazil, India, Indonesia, Malaysia, Mexico, and Russia) and three high-income countries (Australia, Canada, and the United Kingdom). Using the findings from Oortwijn et al., we compared the level of HTA development between China and the seven middle-income countries and between China and the three high-income countries. We ran two two-sample t-tests, one that compared whether the mean level of development was statistically significantly different between China and other middle-income countries (Argentina, Brazil, India, Indonesia, Malaysia, Mexico, and Russia), and another that determined whether the difference between China and high-income countries (Australia, Canada, and the United Kingdom) was statistically significant.
Results
From a total of 1,835 identified stakeholders to whom we sent the questionnaire, we received 222 complete responses (i.e., a response rate of 12.1 percent). Table 1 displays the detailed characteristics of the 222 respondents. Most respondents were researchers (71.2 percent), younger than 50 years old (93.7 percent), living in East China (59.5 percent), or having a master's degree (53.2 percent) or doctoral degree (41.9 percent). The response rate from policy makers was especially low. This may be because policy makers are relatively unaware of HTA (Reference Chen, He and Chi2), may not be very familiar with HTA terminology, or have concerns about the interpretation of their responses.
Characteristics of Survey Respondents (n = 222)
Institutionalization
The majority of respondents (64.0 percent; n = 142) reported that China does not have an HTA agency that fits the criteria specified in the survey, which refer to reporting to a Minister of Health/Human Resources or other relevant authority, producing and/or endorsing HTA reports, and informing decisions about the introduction, reimbursement, and/or disinvestment of health technologies. The remaining 36.0 percent of the respondents reported that China has already established at least one public agency, reports to the central government, produces or endorses HTA reports, and informs decisions related to health technologies. Among a list of agencies listed, the Key Lab of Health Technology Assessment, National Health Commission (Fudan University), and the National Health Development Research Center were reported most frequently. The respondents (n = 142) who believed that China does not have a formal HTA program scored the presence of the essential elements for establishing a formal HTA program in China (Table 2). The mean score for each element was 1.54 out of 3, and the total score for all elements was 12.31 out of 24. The elements that had the lowest scores were those that are related to public funding for HTA.
Scores Regarding the Presence of the Indicators in the Mapping of HTA Instrument
Note. Two indicators with (*) are different in the scores, which range from 0 through 4.
Identification
Questions regarding the identification of health technology were focused on the existence of a horizon scanning system (HSS) for emerging technologies and the characteristics and capacities of HSS. Fewer than one-third of the 222 respondents were aware of the concept of HSS (32.0 percent), and most of them either thought there was no HSS in China (40.9 percent) or were unsure whether HSS existed in China (50.7 percent). Regarding indicators other than HSS, respondents indicated that these were somewhere between largely present and present to some extent (Table 2).
Priority Setting
The total mean score for China's priority setting process was 10.37 out of 18. The mean score for the six indicators of the priority setting process was 1.73 out of 3, suggesting that each indicator was viewed as almost largely present. The score of measuring the transparency in processes was 1.74 out of 3, but 6.8 percent of the respondents rated 0, that is, no transparency (Table 2), and most of these respondents represented the industry.
Assessment
The mean score for this domain was 25.3 out of 39, which makes this domain the highest-rated in terms of the percentage of the maximum score. The mean score for the indicator of whether HTA plans in China addressed generalizability and transferability was the lowest among the five indicators (1.66 out of 3; Table 2).
Appraisal
This domain scored 4.59 out of 9. Each of all three indicators scored below 2, which means that this part of the HTA process was regarded as less well developed. The indicator that scored the lowest was the extent to which mechanisms for the appeal of HTA findings were in place (1.23 out of 3; Table 2).
Reporting
This domain scored 6.53 out of 11. The score for using the guideline on best practice in conducting and reporting HTA was 1.64 out of 3, which means that this element of HTA reporting was considered well developed in China. The score for the number of HTA reports produced per year was 2.73 out of 3, and the score for the number of HTA reports that were related to the National Drug Reimbursement List (NDRL) per year was 2.17 out of 3. These scores reported by the respondents suggested that the number of all HTA reports was close to 12 per year and that of the NDRL-related reports was more than 6 (Table 2).
Dissemination of Findings and Conclusions
This domain scored 6.76 out of 12, and the mean of the scores for all indicators was 1.69 out of 3 (Table 2). This means that HTA dissemination strategies are considered to be moderately well developed in China.
Implementation in Policy and Practice
This domain scored 4.39 out of 10. Regarding whether there is a legal mandate of an HTA agency or organization to get involved in HTA in the healthcare system, 14.4 percent of the 222 respondents indicated that this exists, but the majority indicated there is none (85.6 percent). Therefore, the indicator of the presence of a legal mandate scored .14 out of 1. In addition, the mean score for the other three indicators was 1.42 out of 3 which is the lowest among all domains.
International Comparison
We found that the level of HTA development in China is higher compared to that of the middle-income countries and lower than that in the high-income countries (all p-values <.05) based on the survey's respondents views. However, regarding the level of HTA institutionalization, China scored lower than all ten countries in terms of the level of institutionalization (Table 3, Figure 1 and Supplementary Tables 2 and 3).
Normalized scores for the level of HTA development in China, seven middle-income countries, and three high-income countries.
Total Scores for the Level of HTA Development in China, Seven Middle-Income Countries, and Three High-Income Countries
Note. Data of the middle-income and high-income countries are from the research of Oortwijn et al.
Discussion
Using the Mapping of HTA Instrument to determine the perceived level of HTA development in China allowed us to collect views of a broad set of survey respondents in China. These findings offer valuable, multi-stakeholder insights into decisions about the future development of HTA in China, along with other information about the current status of HTA in China. Although a considerable gap exists between the level of HTA development in China and that in high-income countries, China is relatively well developed relative to the aforementioned middle-income countries regarding most domains based on survey respondents' views. This reflects a recognition that China has made important achievements in developing HTA since the concept was introduced in China three decades ago (Reference Chen, Banta and Tang3). Shi et al. found that the number of articles and research projects on HTA were increasing and are continuing to increase, and that HTA development has made substantial progress in China including a growing number of research institutions and qualified HTA workforce. However, notable weaknesses remain, such as low capacity for conducting HTA, limited experience in HTA workforce, and lack of knowledge translation (Reference Shi, Mao and Tang5). As described by Chen et al., although achievements have been made by HTA activities among academic, providers, and policy makers, development of the field of HTA in China is fragmented and not yet formally integrated into health policy-making processes. The 2015 Global Survey on Health Technology Assessment by National Authorities found that (a) China does not have a legislative requirement to consider the results of HTA, (b) decision makers rely partly on HTA advice, (c) there are no guidelines for preparing HTA, and (d) there is a lack of regulation regarding transparency and a mechanism for public HTA dissemination and communication (6;7). These findings are generally aligned with the views of the survey respondents reported here.
The survey finding across the eight domains of HTA development strongly suggests that China needs to act promptly to improve HTA institutionalization. Although the survey findings indicate that the overall level of HTA development in China is higher than that in the seven middle-income countries, the survey findings also indicate that the level of institutionalization in China is significantly lower.
In addition, compared to other middle-income countries, China has a lower level of HTA progress in health technology identification and the existence of an early warning system or HSS. HSS detects potentially important emerging technologies that will enter the market in the coming years and conducts snapshot analyses of the emerging technologies (Reference Hu, Lin and Chen8). Many high-income countries have already established their own HSSs and are collaborating in establishing the International Information Network on New or Emerging, Appropriate Use and Re-Assessment Needed Health Technologies, that is, EuroScan International Network, currently known as EuroScan International Network e.V (Reference Douw and Vondeling9). In 2001, the Chinese Cochrane Center translated and published the horizon scanning reports of the Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU). In 2015, Zhou et al. at the Key Lab of Health Technology Assessment, National Health Commission (Fudan University) published an article on the global trend of new and emerging health technology development based on EuroScan's database, but the article still received little attention (Reference Zhou, Lin and Xia10).
The respondents of this study consisted of three types of stakeholders, namely HTA researchers, industry, provider representatives, and decision makers in the healthcare sector. Differences in their assessments of China's HTA development are likely due to their varying levels of awareness or interest in HTA. For instance, technical personnel from the industry or hospitals scored the indicators with regard to appraisal and priority setting of HTA lower, whereas researchers scored the indicators related to HTA research and policy translation lower. In future research, we can analyze the perspective of each type of stakeholders to illustrate their assessment of China's HTA development in detail, and we can also compare the assessment across different types of respondents from different countries.
Implications
Based on the Mapping of HTA Instrument, this survey-based analysis shed light on the development of HTA in China, including a gap analysis between China and other countries, and identified key issues for strengthening HTA development in China. Among these issues are the government's commitment and engagement, HSS development, guidelines for conducting HTA, assessment and appraisal relationship, and so on. As opposed to informal qualitative opinions, such analysis should provide more solid information on HTA development to policy makers, academics, providers, industry, and so on.
Limitations
There were several limitations in this cross-sectional study conducted in late 2016. First, the convenience sampling and snowball sampling methods expose our findings to selection bias. The respondents were assessing the level of HTA development from different perspectives and levels of awareness of HTA in China. The interpretation of our study findings should therefore consider the large variation in respondents' characteristics. Moreover, the 12.1 percent response rate suggests that the findings could be subject to response bias. Second, the questionnaire was adapted from an international study of mapping level of HTA development. There might have been imprecision and misinterpretation in the process of translating the Mapping Instrument from English to Chinese. Third, this is a cross-sectional study that examined the associations between the level of HTA development and different characteristics of HTA institutionalization and HTA process at a certain moment of time. Fourth, although the opinions of a broad set of survey respondents can provide important insights about the perceived status of HTA, these must be accompanied and substantiated by objective information about the current capacity, functioning, and roles of HTA in China in order to inform decisions about how to further develop HTA in China.
Conclusion
The Mapping of HTA Instrument offers a highly useful tool for assessing the level of HTA in China, through constant systematic measurement and comparison. The findings of this study suggest areas for improvement, serve as the baseline of future research on the level of HTA development in China, and provide points of reference for international comparison of HTA development. In terms of immediate action, China needs to tackle the issue of low HTA institutionalization to strengthen the foundation of HTA development. Future government initiatives that institutionalize HTA, for example, establishing a national HTA system or consortium, will improve the development of HTA in China. Indeed, China is currently further developing its national HTA centers, and an HTA mechanism for informing health policy and decision making is underway. It could therefore be valuable to conduct a second mapping exercise in the near future to observe any important impacts of these changes.
Acknowledgments
We sincerely thank the China Medical Board Collaborating Programs for supporting this project as well as the Key Lab of HTA (Fudan University), NHC continuously.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462319000709.
