DISCUSSION

This study provides information on the relative importance of barriers and facilitators regarding the use of HTA, and the differences between policy makers and HTA experts in the Netherlands. Not surprisingly, many of the most important barriers and facilitators were opposites, meaning that the most important facilitators were the factors that reduced the most important barriers to the use of HTA by policy makers. For instance, no explicit framework for the decision-making process (that uses HTA evidence) was an important barrier, while the availability of an explicit framework for the decision-making process (that uses HTA evidence) was perceived as the most important facilitator.

This was also true for support by stakeholders, support within the organization for the use of HTA, and the availability of appropriate incentives (for the implementation of cost-effective interventions). Interestingly, these most important factors seem also somewhat tangible (e.g., strategies to enhance awareness and support within the organization), making it possible to reduce the research/policy gap. In a systematic review by Oliver et al. (Reference Oliver, Innvar and Lorenc5), which identified barriers and facilitators regarding the use of evidence in public policy, all factors were categorized into six themes: “Organizations and resources,” “Contact and collaboration,” “Research and researcher characteristics,” “Policy-maker characteristics,” “Policy characteristics,” and “Other.”

Categorizing the most important barriers and facilitators for these themes showed that “Organizations and resources” and “Policy characteristics” factors were the most important. Generally in line with Oliver et al. (Reference Oliver, Innvar and Lorenc5), attention should thus be drawn to enhance support and create an explicit framework (policy guideline, or policy statement) for the decision-making process, in which HTA is more transparently used in setting priorities (Reference Oliver, Innvar and Lorenc5). However, surprisingly, policy characteristics were reported only moderately often in the literature, while this study found that the creation of an explicit framework for the decision-making process is both the most important barrier and most important facilitator.

In addition, in line with Oliver et al., the generalizability and quality of research findings are important factors (Reference Oliver, Innvar and Lorenc5). Interestingly, the literature suggests that the factors of “Contact and collaboration” (e.g., appropriate timing between HTA research and policy making, improving longstanding relations between professionals/policy makers and HTA researchers, and more collaboration between the groups) (Reference Oliver, Innvar and Lorenc5;Reference Garrido7) may be important, as these factors were among the most frequently reported barriers and facilitators regarding the use of evidence in literature (Reference Oliver, Innvar and Lorenc5). Our findings, however, suggest that this frequency of reporting may not reflect the relative importance of the factors.

Despite being among the important factors (as it is frequently reported), policy makers and HTA experts deem other factors more important. Accordingly, findings suggest that it is highly important to increase awareness of the need for creating an explicit framework for the decision-making process, stressing the need to include HTA evidence. Policy statements on the importance of including HTA in decision making may facilitate this process. This is also in line with a study in Denmark that identified the need for a political or managerial decision to use (mini-) HTA (Reference Ehlers and Jensen30). In addition, when processes are explicit, deliberative and formalized, HTA tends to have a higher profile in the decision-making process (Reference Oortwijn, Broos and Vondeling31).

On the organizational level as well, It is important to create awareness and motivate managers to support the inclusion of HTA evidence in setting priorities and to place incentives in line with this. Here as well, the Danish study reflects the importance of a managerial decision to systematically use HTA in policy making. One interpretation of the study's finding is that HTA researchers should engage more directly with decision makers and create awareness and motivation to systematically incorporate HTA evidence in decision making (Reference Ehlers and Jensen30). This may also explain in part why findings suggest that contact and collaboration were relatively less important to participants in our study, as these factors may be the mediators of political and managerial decisions (and thus opinions which are reflected only in the outcome factors).

Subgroup analyses showed some differences between gender, age groups, and roles. Differences between policy makers and HTA experts in perceptions are relevant as they may indicate how to address the research/policy gap. Although HTA experts mostly recognized the relative importance of barriers and facilitators regarding the use of HTA by policy makers, they underestimated the importance of the absence of policy networks or observatory agencies that might promote the joint efforts of researchers and policy makers, and the lack of contact and interaction among policy makers, HTA researchers, and other stakeholders.

On the other hand, HTA experts seem to overestimate the relative importance of the absence of appropriate incentives for the implementation of cost-effective interventions, the lack of support within the organization for using HTA, the lack of an explicit framework for a decision-making process that uses HTA evidence, and lack of financial resources for conducting relevant HTA research. Moreover, HTA experts perceived the quality and credibility of the research to be more important than did the policy makers. This means that HTA experts may need to enhance efforts to involve policy makers in the early stages of research, reinforcing contact and collaboration. Policy makers consider this to be one of the most important factors, while HTA experts may have failed to recognize this issue.

Another explanation related to the noted inconsistency that, in the literature, contact and collaboration were not found to be one of the most important factors overall (Reference Oliver, Innvar and Lorenc5), may be that HTA experts perceive contact and collaboration to be relatively unimportant, or that expert opinion is focused more on the outcomes (i.e., the need for an explicit framework in decision making or putting a policy statement together) rather than on the mediator (contact and collaboration).

To better understand the most important factors related to the uptake of HTA by policy makers, a framework can be used to show the underlying mechanism for using HTA. The I-Change Model may help in understanding the position of the most important factors regarding the uptake of HTA in policy making (Reference Cheung, Evers and Hiligsmann32). The I-Change Model integrates concepts of various health behaviors, communication and promotion models, to explain behavior (Reference de Vries, Mudde and Leijs33), and has been used widely in identifying the determinants of adoption behavior (Reference Cheung, Evers and Hiligsmann32;Reference de Vries, Eggers and Bolman34;Reference De Vries, Eggers and Jinabhai35). According to the I-Change Model, behavior is the result of intentions, which are influenced by motivational and awareness factors. Motivational factors are attitude (perceived advantages and disadvantages of the behavior), social influence (perceived social influence on the behavior), and self-efficacy (the perceived capability to conduct a certain behavior).

First, it is important to increase awareness within the organization regarding the relevance of HTA. Then motivational factors need to be addressed; it is important to conduct high quality, relevant research that will enhance positive attitudes toward the use HTA. In addition, it is important to align HTA usage with appropriate incentives. To improve perceptions of social support, efforts need to be made to establish policy networks that promote the joint efforts of researchers and policy makers, to create personal contact and interaction between policy makers and researchers, and to involve policy makers in the research process. To improve the policy maker's self-efficacy, it would be beneficial to create an explicit guideline regarding the use of HTA in decision making.

Our study could have several limitations. One limitation is the potential lack of generalizability of the results, as the study was conducted with Dutch policy makers and HTA experts only. It is feasible that factors perceived as important for the use of HTA in the Netherlands may be different in other countries, especially in emerging countries, due to differences in the decision-making context (Reference Hoffmann and von der Schulenburg36). Middle-income countries (e.g., India and Russia) are less advanced in the use of HTA than more industrialized countries (e.g., United Kingdom), although the community that uses HTAis growing in middle-income countries (Reference Oortwijn, Broos and Vondeling31). Our findings may, therefore, not be transferable to all other countries, particularly to middle-income countries. Future research should investigate the relative importance of barriers and facilitators regarding the use of HTA in different countries, and compare results. In addition, despite recruiting decision makers on all levels within the health system, it is unknown whether certain levels were overrepresented in the responses. This may be a potential limitation regarding the generalizability of our findings.

Another limitation is that participants were not familiar with the BWS tasks (as indicated by the difficulty scores), which may have led some to drop out. As 58.3 percent of the participants who started the survey actually completed it, our sample may be biased. However, analyses showed no differences in demographic characteristics between participants who did not complete the surveys and those who did. Moreover, in searching for barriers and facilitators to draft the initial list of relevant factors, a scoping review rather than a full systematic review was conducted, as systematic reviews on the barriers and facilitators for HTA and evidence use have been conducted recently (e.g., Oliver et al.) (Reference Oliver, Innvar and Lorenc5). Therefore, this study potentially excluded some relevant articles regarding barriers and facilitators. However, it is not expected that many additional barriers and facilitators would have been identified, as data saturation was observed during the draft of the initial list of factors. In addition, the initial list was validated by experts.

In conclusion, this is, to our current knowledge, the first study that has assessed and quantified the relative importance of barriers and facilitators regarding the use of HTA and compared the views of policy makers and HTA experts in the Netherlands. This resulted in a list of factors that are most important in bridging the gap between policy and research. Barriers and facilitators related to organization, resources, and policy characteristics were particularly important. It is of utmost importance to create a pulse on the national level of the Netherlands to make a political statement and create an explicit framework for the decision-making context to include HTA evidence. Due to some differences in the perceptions between policy makers and HTA experts, it is necessary to create a framework for closer collaboration and informed decision making.