Since the introduction of health technology assessment (HTA) to China in the 1980s, its development was focusing on studying technologies and experiences in developed countries to address their changing healthcare needs, appraising new technologies in terms of safety, effectiveness, and cost-effectiveness. Since the 1990s, some HTA research units have been established by the Ministry of Health (MoH), some universities, or some health services providers to conduct HTA researches, education and training, and technical services (Reference Chen, Banta and Tang1).
However, conducting HTA is one thing, using HTA results to influence health policy making is another. Unless the knowledge gained from HTA can reach policy makers and be used by them in decision making, HTA is largely an academic exercise with a little real impact on health policies and practices. In China, although some HTA evidence has been used to inform policy decisions, such as the adoption of assisted human reproductive technology, gamma knife technology, and prenatal diagnosis technology (Reference Chen, Banta and Tang1;Reference Chen, Qian and Tang2), health technology-related decision making tends not to be based on HTA evidence. Some university-based or government-led institutions have been set up to facilitate evidence-informed policy making in China within the context of the ongoing healthcare reforms, such as Key Lab of Health Technology Assessment in Fudan University designated by Ministry of Health (MOH) and a research unit in MOH's China National Health Development Research Center. Despite their efforts, application of HTA findings to policy making is not yet widespread and the integration of HTA in the policy-making processes is still in its infancy in China (Reference Chen, Banta and Tang1). For HTA to have maximum policy impact and for health technology-related policy making to be truly evidence-based, they need to be more closely linked. The linkage between these two sets of activities can be best accomplished by KT, which has been defined by the Canadian Institutes of Health Research as “the exchange, synthesis and ethically-sound application of knowledge—within a complex system of interactions among researchers and users.” (Reference Straus, Tetroe and Graham3)
There are studies on KT measurement, especially in measuring use of research findings. Larsen developed a utilization scale to measure knowledge use and non-use with seven ranked stages: “considered and rejected”, “nothing done”, “under consideration”, “steps toward implementation”, “partially implemented”, “implemented as presented”, and “implemented and adapted” (Reference Larsen4). Estabrooks proposed three types of research use (symbolic use, conceptual use, and instrumental use) and developed a measurement scale to separately measure each type of use (Reference Estabrooks5). Landry and his colleagues measured the use of social science research from knowledge producers’ perspective with a stages-of-use approach. The approach is in the form of a scale that includes transmission, cognition, reference, effort, influence, and application (Reference Landry, Amara and Lamari6). Additionally, they developed a similar scale to measure knowledge utilization from the perspective of policy makers (Reference Landry, Lamari and Amara7).
Some researchers have explored KT determinants. Landry et al. divided the determinants into four categories: science push variables, demand pull variables, dissemination variables, and interaction variables. Perceptions of users’ context, adaptation of products to users, and intensity of linkage between researchers and users were found to be positively related to knowledge utilization in their study (Reference Landry, Amara and Lamari8). Relevance to decisions was also reported by many researchers as an important KT determinant. Dobbins et al. argued that decision makers tended to adopt evidence that were relevant to their decisions (Reference Dobbins, Cockerill and Barnsley9). Dobrow et al. reported that the decision-making process could directly or indirectly be affected by personal factors, interpersonal relationships and individual and/or collective conflicts of interest (Reference Dobrow, Goel and Upshur10). The positive effect of users’ attitudes toward applying research knowledge was shown in a study by Squires et al. (Reference Squires, Estabrooks, Gustavsson and Wallin11). McIntyre believed that providing decision makers with evidence that clearly answered their questions would facilitate the use of research evidence (Reference McIntyre12). The importance of timeliness was also emphasized in Almeida and Báscolo's research (Reference Almeida and Báscolo13). Regarding dissemination, the format of transmitting knowledge was identified as an important factor of KT. Lavis et al. believed that the form in which research results were presented could affect the effectiveness of research knowledge transfer to decision makers (Reference Lavis, Robertson, Woodside, McLeod and Abelson14;Reference Jacobson, Butterill and Goering15). Hancock and Easen argued that research results must correspond to practical knowledge to be recognized as useful (Reference Hancock and Easen16). Finally, several studies have demonstrated the importance of exchange mechanisms between researchers and users (Reference Belkhodja, Amara, Landry and Ouimet17–Reference Ross, Lavis, Rodriguez, Woodside and Denis19).
However, much of the KT research targets clinicians (e.g., clinical guidelines, clinical decision support), there is not much about the perspective of policy makers. Additionally, in China, there is a dearth of empirical literature making comparison on KT from HTA to health policy making between researchers and policy makers to address the gaps between these two groups and to expand HTA usage in health policy making. Therefore, this study aims to: (i) Examine the gaps between HTA researchers and policy makers, which include their perceptions of HTA research and their KT activities. (ii) Compare HTA researchers and policy makers with respect to their views on the determinants of effective HTA KT.
METHODS
Survey Instruments
Two questionnaires—one for HTA researchers and one for policy makers—were developed, and most questions were similar with different perspective. Questionnaire for HTA researchers was divided into two sections. The first section is measuring perceptions of HTA research. According to the categorization by Landry et al. (Reference Landry, Amara and Lamari8), this section was divided into four dimensions: science push (1 item: scientific rigor of research), demand pull (four items: relevance to policy making, timeliness, compatibility with existing policy directions and role of HTA on policy making), dissemination (two items: practicality and in easy-to-understand language), and cooperation (two items: contacts between organizations and collaboration in research development). The second is measuring KT activities. According to the previous studies (Reference Landry, Amara and Lamari6;Reference Landry, Amara and Lamari8) and expert consultation, KT activities were measured in six stages in this study: academic translation, nominal translation, cognitive translation, reference translation, adoption translation and application translation. Academic translation means that research knowledge is published in academic journals. Nominal translation means that research knowledge is transmitted to policy makers by submitting the research report, etc. Cognitive translation means that research reports are read and understood by policy makers. Reference translation refers to research knowledge having been cited as a reference by policy makers. Adoption translation refers to the adoption of research knowledge by policy maker to underpin policies. Application translation means that research knowledge has resulted in policy application by policy makers. A similar questionnaire for policy makers was also developed.
A series of 5-point Likert scales were used in the survey. Possible responses to the questions were: 1 = strongly disagree, 2 = moderately disagree, 3 = neither agree nor disagree, 4 = moderately agree, 5 = strongly agree; or 1 = very bad, 2 = fairly bad, 3 = intermediate, 4 = good, 5 = excellent; or 1 = never, 2 = seldom, 3 = occasionally, 4 = often, 5 = always. Demographic characteristics of HTA researchers (gender, age, academic title, etc) and policy makers (gender, age, official rank, etc) were also collected.
Respondents
We surveyed policy makers in the health technology field at central government departments of the Ministry of Health (MoH), the Ministry of Health Resource and Social Security (MoHRSS), and the State Food and Drug Administration (SFDA), as well as the provincial counterparts of MoH, MoHRSS, and SFDA.
HTA researchers were those who met the following criteria (i) working at HTA agencies/units including both agencies/units with HTA as part of their titles and those with HTA research or related activities, such as academic units focusing on evidence-based medicine, pharmaco-economics, health economics, healthcare management at universities and research institutions, etc; (ii) conducting research that examines the medical, economic, social and ethical implications of the application or use of health technology, such as the pharmaceuticals, devices, procedures, and organizational innovations in health care.
Sampling Method
There was not a complete and reliable list of policy makers in the health technology field in China that could be used as a sampling frame. Thus, a snowball sampling technique was adopted. We started with a few individuals who were known to us as policy makers in the health technology field. We then asked each of them to identify others who should be surveyed. This process repeated till no more new potential policy makers were identified.
For the same reason, snowball sampling technique was also applied to the HTA researchers. We also started with some HTA researchers and asked each of them to identify other potential one, which was very similar to the sampling process for the policy makers.
Administration of the Surveys
According to feedbacks from the pilot study, both questionnaires were moderately revised to improve their clarity in wording. Some notes were also added in the questionnaires to ensure the respondents understand what they needed to do and how to do it. The survey was conducted between October 2012 and April 2013. The questionnaires and informed consent forms were sent to the HTA researchers and policy makers by means of email. If the respondents agreed to participate in the survey, they would fill out the questionnaire at their convenience and returned directly by means of email or mail. The research has been reviewed and approved by the ethics board of School of Public Health, Fudan University (IRB Approval Number: #2012–11–0382).
Scoring KT Activities
As mentioned above, an index score for each of six KT stages was obtained from the participants’ responses in a range from 1 to 5. The response indicated on the each question on the 5-point Likert scale ranged from one to five points and higher scores representing more positive assessments. Because each stage is rank ordered from low to high level of KT, the scores were further “weighed” for each stage (by 1 for stage 1, by 2 for stage 2, and so on) to produce a summary score for a possible highest score of 105. Therefore, the total score of KT activities for each respondent can range from 21 to 105 (Reference Landry, Amara and Lamari8).
Statistical analysis
Descriptive analysis (frequencies and percentages) was used to describe the demographic characteristics of the respondents separately for HTA researchers and policy makers in this study.
Wilcoxon rank sum test was used to compare differences between the HTA researchers and policy makers regarding perceptions of HTA research and KT activities in six stages.
Before the multivariate linear regression analysis, correlation analysis was performed to address co-linearity in model specification. For significant correlations with high correlation coefficients (>0.4) existed among some demographic characteristics (e.g., age, experience, academic title [researchers] / official rank [policy makers]), only three demographic characteristics (gender, education, academic title [researchers] / official rank [policy makers]) were used in subsequent analysis. Multivariate linear regression was performed to explore the association between the dependent variable (total score of KT activities) and the independent variables (respondents’ demographic characteristics, collaboration in research development, relevance to policy making, practicality, importance of HTA on policy making, and etc). Also, the comparative contribution of each independent variable to HTA knowledge translation was determined by their standard regression coefficients. Because of differences between the HTA researchers and policy makers, a multivariate linear regression model was run for each group separately.
Statistical analyses were carried out using SPSS/version 13.0 statistical software. Two-sided p-levels of < .05 were considered to be statistically significant.
RESULTS
Demographic Characteristics of Respondents
Of 561 HTA researchers approached, 382 HTA researchers completed and returned the questionnaires. The response rate was approximately 70 percent of the target respondents. The respondents were mostly in the 30–49 age group and 179 respondents were male (46.9 percent). Most of the respondents had postgraduate degrees. The researchers focused mostly on the assessment of drug (27.8 percent) and the healthcare system (55.8 percent). Regarding academic titles, 29.3 percent, 28.3 percent, and 35.1 percent of the researchers were lecturers, associated professors, or full professors, respectively (see Table 1).
Demographic Characteristics of Respondents
Regarding policy makers, the response rate was 55.2 percent (112 of 203 policy makers approached). Their age distribution was found to be similar to HTA researchers. Half of them had bachelor degrees and the other half had master's or doctoral degrees. With respect to organizational affiliation, 42.0 percent of the respondents were with MOH, 19.6 percent with MoHRSS, and 38.4 percent with SFDA. Additionally, 27.7 percent of the officials belonged to central agencies. Regarding official ranks, most of the respondents were directors (33.9 percent) or section chiefs (46.4 percent) (see Table 1).
Perceptions of HTA Research
Table 2 shows differences between policy makers’ and researchers’ perceptions of HTA research. Comparisons between the two groups were all statistically significant, except the items of “collaboration in research development” and “in easy-to-understand language”. Among the statistically significant items, policy makers’ perception about compatibility with existing policy directions was much more positive than those of HTA researchers. However, other significant items were perceived more positive among HTA researchers than policy makers. The widest gaps existed in perceptions about “practicality”, “scientific rigor of research”, and “relevance to policy making”.
Researchers’ and Policy-Makers’ Perceptions of HTA Research
aThe percentage was calculated by the number of positive responses (e.g. “excellent” or “good,” “always,” or “often”) divided by the number of valid responses on each item.
bRating average was calculated by the sum of rating divided by the number of valid responses on each item.
Assessment of KT Activities
Table 3 displays results regarding KT activities of policy makers and researchers in six stages. In the stage of nominal translation, only 19.7 percent of the policy makers claimed that they had “often” received research outcomes from HTA researchers. This was much less than the proportion of HTA researchers who claimed that they had “always” or “often” transmitted research findings to policy makers. The difference was statistical significant (p < .01). In the stages of cognitive translation, reference translation, adoption translation and application translation, the situation was just the opposite, with policy makers more likely to report that they had understood, referred, adopted and applied HTA research outcomes. The differences between policy makers and researchers were all statistical significant, especially for reference translation and adoption translation.
KT Activities of Policy Makers and Researchers
aThe percentage was calculated by the number of positive responses (“always” or “often”) divided by the number of valid responses on each item.
bRating average was calculated by the sum of rating divided by the number of valid responses on each item.
The total scores of KT activities for HTA researchers and policy makers were shown in Figure 1. Approximately 60.0 percent of policy makers had a total score of between 63 and 83 while only 40 percent of HTA researchers had a total score in the same range. For the range of total score between 42 and 62, there were similar proportions of respondents (policy makers: 26.8 percent versus HTA researchers: 35.0 percent).
Total score of knowledge translation activities of health technology assessment (HTA) researchers and policy makers.
Multivariate Linear Regression Analysis for HTA Researcher
The analysis showed that eight of the twelve independent variables contributed to the prediction of HTA knowledge translation by HTA researchers (the eight independent variables are academic title, contact between organizations, collaboration in research development, compatibility with existing policy directions, relevance to policy making, practicality, importance of HTA on policy making, in easy-to-understand language). They were all KT facilitators and accounted for 48 percent (46 percent adjusted) of the variance in the prediction of HTA knowledge translation, (R2 = 0.48; F(12,369) = 27.85; p < .001; see Supplementary Table 1, which can be viewed online at http://dx.doi.org/10.1017/S0266462314000737). As indicated by the standardized regression coefficients (Standard Beta) of the independent variables, collaboration in research development, relevance to policy making, academic title, contact between organizations, and in easy-to-understand language were the five most important factors in predicting HTA knowledge translation.
Variable inflation factors (VIFs) were also computed for each independent variable to detect multi-colinearity. As a guideline, a VIF of >10 indicated a problematic co-linearity (Reference Kennedy20). Statistical tests indicated that multi-colinearity was not a significant problem as the maximum VIF for the independent variables was only 2.12.
Multivariate Linear Regression Analysis for Policy Makers
The regression model for policy-makers only identified four of the independent variables significantly associated with HTA knowledge translation. Three of them, namely contact between organizations, relevance to policy making and importance of HTA on policy making, were KT facilitators. And scientific rigor of research was KT barrier. Together, these four variables accounted for 42 percent (35 percent adjusted) of the variance in the prediction of HTA knowledge translation, (R2 = 0.42; F(12,99) = 6.01; p < .001; see Supplementary Table 1). Based on the standardized regression coefficients, scientific rigor of research was the most important independent variable, followed by contact between organizations, importance of HTA on policy making, and relevance to policy making. Also, multi-colinearity was not a significant problem, as the maximum VIF of the independent variables was only 3.17.
DISCUSSION
The primary aim of this study was to identify the differences between HTA researchers and policy makers with respect to perceptions of HTA research and KT activities in China, and to compare the two groups in relation to the determinants of HTA KT. Our findings help the two groups understand each other better about HTA and KT. They can also be used to help bridge the gap between these two groups and facilitate HTA utilization in policy making. These would further improve the accessibility and equity of health technology and benefit the performance of healthcare system in China. As one of the largest developing nations, China's success in HTA knowledge translation also has an important impact on the world.
Gaps in Perceptions of HTA Research and KT Activities
Regarding the perceptions of HTA research, huge gaps between the researchers and policy makers were found in this study and the gaps were especially wide in relation to scientific rigor of research, practicality, and relevance to policy making. Although many HTA researchers believed they had done high quality research with great pertinence to policy making, many policy makers did not think so. To maximize the impact of HTA, it may be advisable for researchers to take policy-makers’ perceptions of HTA evidence into consideration. Our results suggest that some researchers might have failed to make their findings, conclusions or recommendations specific, meaningful or operational enough for policy makers to understand or use.
Regarding KT activities, major gaps between HTA researchers and policy makers were also detected. Researchers were reportedly more active in nominal translation, while policy makers indicated they had done more in other aspects, particularly reference translation and adoption translation. What the policy makers said in the survey appears to contradict Landry's argument that the latter stages of KT (e.g., reference translation and adoption translation) are built on earlier stages (e.g., academic translation and nominal translation) (Reference Landry, Amara and Lamari6). A plausible explanation is that because policy makers are not academics and do not typically engage in research activities, they tend to be less involved in academic or nominal translation. Instead, they are more likely to be interested in the “application” aspects of KT, such as adopting research results, where appropriate. Further work is needed to elucidate this point.
Determinants of HTA Knowledge Translation
For both researchers and policy makers, relevance of HTA research to policy making, contact between organizations, and importance of HTA on policy making were facilitators of KT. And two categories of variables, namely, demand pull variables and cooperation variables, were important to successful KT. These results are supported by previous studies that have found research knowledge more likely to be used when it is seen as pertinent to users’ needs and when users see its potential applicability to their practice (Reference McIntyre12;Reference Hancock and Easen16).
Also, our findings confirm recent studies’ suggestions that close relationships between knowledge producers and users, are a key factor leading to successful KT (Reference Lavis21). It is, therefore, not hard to understand the importance of the “demand pull” variables in promoting KT. Additionally, this study reveals the importance of the “cooperation” variables. Collaborative experiences, especially frequent exchanges and close linkages between HTA researchers and policy makers, are likely to promote trust between the groups and develop more productive cooperation. As a result, policy-makers’ needs and HTA investigators’ research questions and designs are more likely to be aligned with each other. Such alignment could be conducive to the production of needed HTA evidence, which will further increase its adoption and use (Reference Belkhodja, Amara, Landry and Ouimet17–Reference Ross, Lavis, Rodriguez, Woodside and Denis19). Our study's findings highlight the importance of promoting collaborations between HTA researchers and policy makers. Some international HTA institutes, such as the National Institute of Health and Clinical Excellence (NICE) in the United Kingdom, the Canadian Agency for Drugs and Technology in Health (CADTH), the Health Intervention and Technology Assessment Programme (HITAP) in Thailand, have provided bridges between researchers and policy makers by ensuring researches implemented pertinent to the policy making and stressing much importance to the areas with little or poor quality evidence for future research. The operating model of these institutes has played an important role in bridging the gap between research and policy, which further expands the impact of HTA research on policy making (Reference Hanney, Buxton, Green, Coulson and Raftery22).
Regarding HTA researchers, practicality of HTA reports, presenting results in an easy-to-understand language and compatibility with existing policy directions were also significantly associated with KT. These results are consistent with previous studies that emphasize the adaptation of research products for users and the importance of the decision-making context (Reference Dobrow, Goel and Upshur10;Reference Squires, Estabrooks, Gustavsson and Wallin11;Reference Lavis, Robertson, Woodside, McLeod and Abelson14;Reference Jacobson, Butterill and Goering15). To increase the use of HTA research, it is suggested that researchers make their reports easier to understand by nonresearchers and make conclusions and recommendations more specific and operational. Likewise, to further expand the impact of HTA in decision-making organizations, it is also very important for researchers to improve HTA evidence's compatibility with existing policy directions.
It is interesting that, contrary to the other factors, the scientific rigor of research was found to be negatively associated with successful KT. This appears to be counter-intuitive because HTA, like all research, needs to be done scientifically, objectively, meticulously, and competently. One possible explanation is that HTA reports written in typical academic style and full of technical details and jargons are hard for nonresearchers to understand and less acceptable by them. This finding also highlights the importance of presenting HTA evidence in ways that are acceptable to policy makers and other target audiences (Reference Thompson, Cullum, McCaughan, Sheldon and Raynor23). This may require presenting research findings in plain and precise language and in ways that take policy making needs into consideration (Reference Dobbins, Rosenbaum, Plews, Law and Fysh24). Because many researchers do not have policy-making experience and may not even be aware of the policy-making process, research organizations need to find ways to bridge this research-policy gap. One approach that has been attempted in countries such as Canada is for research organizations to have KT specialists whose task is to work with researchers to turn highly technical research reports into policy-maker–friendly documents. Another approach is to expose researchers, as well as those aspiring to become researchers, to policy-making environments and KT activities and, likewise, to make available opportunities for policy makers to better understand and appreciate research (Reference Sheps, Pong, Lavoie-Tremblay and MacLellan25).
Limitations
The main limitation of this study is the sampling method. Snowball sampling method was applied to the policy makers in this study, not a probability sampling. Therefore, the sample may not be representative of the populations of policy makers. Additionally, it is noteworthy that approximately 30 percent of the approached HTA researchers did not complete or return the questionnaires, and those willing to participate in this study were more likely to be aware of the importance of KT and more likely to engage in KT activities. Thus, in real situation, the HTA researchers’ KT activities or their perception of KT importance may be less positive than reported by this study. Besides, this study focuses on certain factors at the individual and organizational level, and yet the other potential determinants at broad context still need to elucidate in further studies.
CONCLUSIONS AND POLICY IMPLICATIONS
Significant gaps exist between HTA researchers and policy makers in China with respect to their perceptions of HTA research and their activities in HTA knowledge translation. There are also differences between the two groups in the perceived determinants of effective KT. For researchers, cooperation with policy makers in research development and using suitable format to present research findings are key facilitators of KT. Policy makers, on the other hand, pay more attention to the importance of HTA on policy making. They regard reports that are overly technical or pedantic as not conducive to translating research outcomes to decision making. Specific to China within the context of the ongoing healthcare reform, close cooperation between HTA researchers and policy makers should be emphasized, especially during the process of health policy making. For policy makers in China, it is recommended to transform the traditional experience-based decision-making process and realize the importance of HTA evidence in technology-related policy making. For HTA researchers, it is advised to focus more on the topics relevant to the need of policy making, communicate more fully with regarding stakeholders and present research findings in more appropriate format according to the needs of audiences. Only under the joint efforts of HTA researchers, policy makers and other stakeholders, can the usage of HTA be facilitated during the process of health policy making. Although this research and the findings are China-based, they probably have wider applicability in the sense that researchers and policy makers in other countries could benefit from what this study has found.
SUPPLEMENTARY MATERIAL
Supplementary Table 1 http://dx.doi.org/10.1017/S0266462314000737
CONTACT INFORMATION
Wenbin Liu, PhD, Key Lab of Health Technology Assessment (Ministry of Health), School of Public Health, Fudan University, 197 Mailbox 138 Yixueyuan Road, Xuhui District, Shanghai 200032, China
Lizheng Shi, PhD, Associate Professor of Health System Management, School of Public Health and Tropical Medicine, Tulane University, New Orleans, Louisiana 70112, USA
Raymond W. Pong, PhD, Senior Research Fellow, Professor of Rural Health Research, Centre for Rural and Northern Health Research and Northern Ontario School of Medicine, Laurentian University, Sudbury, Ontario P3E 2C6, Canada
Hengjin Dong, PhD, Professor of Health Policy, Executive Director of Center for Health Policy Studies, School of Medicine, Zhejiang University, Hangzhou, Zhejiang 310058, China
Yiwei Mao, Master, Key Lab of Health Technology Assessment (Ministry of Health), School of Public Health, Fudan University, 197 Mailbox, 138 Yixueyuan Road, Xuhui District, Shanghai, 200032, China
Meng Tang, Master, Key Lab of Health Technology Assessment (Ministry of Health), School of Public Health, Fudan University, 197 Mailbox, 138 Yixueyuan Road, Xuhui District, Shanghai, 200032, China
Yingyao Chen, PhD (yychen@shmu.edu.cn), Professor of Health Technology Assessment, Director of Key Lab of Health Technology Assessment (Ministry of Health), Collaborative Innovation Center of Social Risks Governance in Health, School of Public Health, Fudan University, 197 Mailbox, 138 Yixueyuan Road, Xuhui District, Shanghai 200032, China
CONFLICTS OF INTEREST
None declared.
