Introduction
Formalizing how universal health coverage (UHC) priorities are set through written rules can enhance consistency and fairness of decisions and improve accountability in decision making and implementation (1–Reference Greer, Vasev, Jarman, Wismar and Figueras3). Health technology assessment (HTA) guidelines are written documents that set out one or more components of the methods, process, or broader governance for how decisions are made (Reference Botwright, Sittimart and Chavarina4). HTA guidelines are often used as an indicator of progression toward institutionalized systematic, evidence-informed priority-setting (5–Reference Oortwijn, Determann, Schiffers, Tan and van der Tuin8) on the basis that they promote transparency (Reference Oortwijn, Determann, Schiffers, Tan and van der Tuin8–Reference Almazrou, Alaujan and Al-Aqeel10) and create an enabling environment for the use of evidence in decision making (Reference Mbau11–Reference Leelahavarong, Doungthipsirikul and Kumluang13).
Despite this, there is evidence that the presence of guidelines does not necessarily translate into a change in decision-making practices (Reference Denzau and North14–Reference Pahl-Wostl16). HTA guidelines can overemphasize administrative procedures (Reference Gulácsi, Rotar and Niewada17), be challenging to implement in fragmented decision-making systems (Reference Almomani, Hammad and AlQutob9), and/or focus on the technical aspects of evaluation without accounting for context-specific processes (Reference Suharlim, Kumar and Salim18). A cross-country comparison of health priority-setting in Asia, for example, showed that the mere presence of guidelines did not predict systematic and evidence-based decision making (Reference Kumar, Suharlim and Amaris Caruso6).
Many studies have evaluated guideline quality or compliance (Reference Daccache, Karam, Rizk, Evers and Hiligsmann19–Reference Decimoni, Leandro and Rozman24). However, current research has not addressed the features that distinguish HTA guidelines that successfully change decision-making practice from those that do not. Identifying these features is important to ensure that HTA guidelines achieve their intended impact.
To address this gap, we conducted a comparative multi-case study to identify the differentiating features of HTA guidelines that have successfully changed decision-making practice across diverse health systems in eight countries. The overarching goal was to understand how HTA guideline development teams can ensure that the guidelines contribute to more systematic, evidence-informed priority-setting.
Methods
Study design and sample selection
We adopted a qualitative comparative case study design to develop transferable findings while accounting for different country contexts over time (Reference Yin25). We included 24 publicly available guidelines from eight countries across different regions and income levels into the following three categories, based on their status in 2022, which was chosen as the year of data collection (see Supplementary Table S1):
-
(1) those that had completed at least one revision of their HTA guidelines (Brazil, Canada, Indonesia, and the Netherlands),
-
(2) those that had developed and implemented a single version of HTA guidelines (The Philippines and Tunisia), and
-
(3) those that went through the process of developing guidelines but for which a version of the HTA guidelines had not yet been officially launched (South Africa and Vietnam).
We used the term “HTA guidelines” to refer to laws, regulations, manuals, or operating procedures that define one or more elements of how evidence and decisions are (or should be) made. For each guideline, we classified the stage of HTA institutionalization in the country at the time of guideline development (Table 1), since in many settings HTA institutionalization has increased over time.
Classification of the stages of HTA institutionalization and guidelines included in the analysis
Table 1. Long description
Four classifications of HTA institutionalization stages are laid out in each row with each definition and which HTA guidelines included in the analysis fit into to these classifications.
Data collection
We used two data sources: semi-structured interviews to understand stakeholder perceptions of guideline development and implementation, supplemented by published literature and policy documents to provide historical information with less recall bias or retrospective sense making (Reference Yin25). We sought diverse perspectives by interviewing different types of stakeholder and identifying literature written by different stakeholder groups, where possible (e.g., HTA agency, international researchers, local academia) (Reference Patton67). The search was conducted in 2022–2023 and included documents in any language. To ensure accurate data interpretation, bilingual researchers or interviewees reviewed findings for documents published in languages other than English.
We interviewed guideline developers (authors or focal points coordinating/overseeing the guideline development process) and guideline users (researchers and personnel overseeing the policy process). We identified interviewees through consultation with relevant networks (HTAsiaLink, ISPOR, and HTAi). We invited twenty-five potential interviewees, of which twenty agreed to participate in the study. Supplementary Table S2 shows the profile of interviewees.
Interviews were conducted between 01 December 2022 and 07 February 2023, using a semi-structured interview guide (Supplementary Information 1). Formal consent was collected in writing and verbally from all interviewees. Each interview was conducted online in English for approximately 60 min, via Zoom or in person, and recorded. Member checking was conducted, and repeated analysis was conducted in case of disagreement (Supplementary Table S4). Further details of the data collection process are provided in Supplementary Information 2.
Analysis
A research team (SB, MS, and KKC) applied content analysis and pattern matching to iteratively review the data to develop, test, and refine hypotheses, while an external advisory group (YT, AC, WO, TM, GS, and CS) periodically reviewed findings to assess whether emerging hypotheses were logical and were supported by the evidence, as well as to propose alternative hypotheses to investigate further.
We conducted two steps of analysis: (i) a within-country analysis to understand the process for developing and revising guidelines, the broader HTA context, and guideline impact and (ii) cross-country analysis to systematically compare similarities and differences in terms of how success of the guideline was defined and measured and factors influencing whether the guideline was successful according to these measures. Success was evaluated qualitatively according to the principles described in Table 2, through discussion and agreement across the research team. The resulting hypotheses were tested against three additional cases (Colombia, India, and New Zealand; 1–2 interviewees per country) to evaluate generalizability and whether theoretical data saturation had been reached. We shared findings (including success ratings) with interviewees to ensure validity of findings. In case of disagreement, the research team reviewed the data again and, if relevant, revised the analysis. Full details of the analysis are provided in Supplementary Information 2 and a list of changes following interviewee validation can be found in Supplementary Table S3.
Criteria to measure the success of the included guidelines. Full definitions of success are provided in Supplementary Table S4 and supporting evidence to measure the success of each guideline is provided in Supplementary Table S5
Table 2. Long description
Four criteria that indicate success level of included guidelines and the rating principles of each criteria. Full definitions of success are provided in Table S4 and supporting evidence to measure the success of each guideline is provided in Table S5.
Ethics
The study was approved by the Institute for the Development of Human Research Protection (COA No. IHRP2022115), Health Research System Institute (IHRP no. 106–2565). All data were managed in accordance with Thailand’s Personal Data Protection Act B.E. 2562 (2019). All transcripts and audio files were de-identified and stored on HITAP secure servers for 5 years.
Results
Indicators to measure HTA guideline success
From our analysis, we defined and measured the success of guidelines by three components: (i) whether the stated objectives in the guidelines were achieved; (ii) the extent to which the guidelines advanced the institutionalization of systematic, evidence-informed decision making; and (iii) compliance with the guidelines (Supplementary Table S4). We evaluated each guideline according to these criteria (Supplementary Tables S5 and S6) to distil factors influencing guideline success.
Factors influencing HTA guidelines success
Factors influencing the success of HTA guidelines differed between two contexts: (i) guidelines developed by stakeholders with some level of constraint on their ability to influence decision making (e.g., institutions without a direct link to the decision-making process) and (ii) guidelines developed by an agency with a mandate to develop and implement guidelines. In the following sections, we discuss the success factors for each. Findings are summarized in Figure 1.
Summary of factors influencing the success of guidelines from the analysis.
Figure 1. Long description
At the top, two scenarios are presented. Context 1 features four purple human icons and states that guideline developers have limited ability to influence change in decision-making. Context 2 features four orange human icons and states that guideline developers have been mandated to develop, implement and revise guidelines.
Below these contexts, two large, symmetrical blue ribbon-like arrows curve downward toward a central dark blue circle represent the process of guidelines development/revision with the endpoint: promoting change in decision making. The development/revision process is described by an upper segment,labeled as Development / revision phase, and a lower segment,labeled as Implementation phase.
On the left side, two thin purple arrows point away from the blue ribbon represent factors influencing the success of guidelines for Context 1. One points from the development phase to the text Strategic stakeholder engagement to achieve shared understanding. The other points from the implementation phase to the text Stakeholders agree on a mandated authority with capacity to oversee guideline implementation and revisions.
On the right side, a thin orange arrow, represents factors influencing the success of guidelines for Context 2, points from the implementation phase to the text Maintain channels for feedback.
In the center of the diagram, between the two large ribbons, Effective accountability mechanisms were highlighted as the common point between the two contexts:through strategic stakeholder engagement, agreement on mandated authority, and maintaining channels of feedback throughout the process will promote change in decision making.
Context 1: Guideline developers had limited ability to influence decision making
One of the main barriers to guideline success in our analysis, especially in settings without systematic decision making, was the limited ability of the guideline developers to influence change. In certain cases, this was because guidelines were initiated by stakeholders that did not have a mandate to oversee and set standards for decision making, but it was also due to unclear or overlapping mandates, such as lack of clarity between the benefit package committee and the national formulary, or federal standard-setting with autonomy of provincial-level processes (Supplementary Table S7).
In these settings, successful guidelines strategically engaged stakeholders who could facilitate or block guideline implementation and engaged in an open dialogue to come to a shared understanding, through a series of consultations throughout guideline development. Importantly, these consultations did not impose how the HTA system should work, but instead actively shaped guideline content with stakeholder input. We refer to this as “joint learning.” Secondly, guideline developers ensured that, at the end of the guideline development process, key decision makers appointed an agency with the relevant expertise, capacity, and independence from the decision maker to oversee guideline implementation and future revisions (Table 3).
Level of joint learning and profile of the body mandated to oversee guideline implementation, in settings where the guideline developer had a significant constraint on their ability to influence change (further details in Supplementary Tables S7 and S8)
Table 3. Long description
Examples of HTA guidelines with different success ratings are listed, along with an explanation of how institutional arrangements before and after the guideline development in the respective setting. Further details in Tables S7 and S8.
These features are illustrated by the 1994 pharmacoeconomics guidelines in Canada. Although the Pharmaceutical Manufacturers Association of Canada (PMAC) did not have the authority to impose use of HTA guidelines, they established government ownership by convening a workshop to identify consensus areas and co-leading guideline development with the recently established Canadian Coordinating Office for HTA (CCOHTA, later to become CADTH then CDA) (Reference Menon, Schubert and Torrance69). As a result of this collaborative process, the federal and provincial ministries of health endorsed the guidelines within a year and officially requested CCOHTA to lead implementation and revisions of the guidelines (Reference Baladi, Menon and Otten70–80). Assigning CCOHTA as the guideline owner meant that CCOHTA could work with researchers to improve compliance (Reference Baladi, Menon and Otten70) and ensured that the guidelines were updated with experience to better meet user needs (Reference Glennie, Torrance and Baladi71;Reference Lim, Bowen and O’Reilly81). This process arguably strengthened CCOHTA’s position as a normative body, enabling the establishment of the Canadian Drug Review process and pan-Canadian HTA strategy in 2003 (Reference Menon and Stafinski82).
By contrast, the unreleased HTA methods and process guides developed from 2014 to 2018 in Vietnam did not show features of local leadership and open collaboration. Throughout guideline development, stakeholder consultation meetings were held, but these focused on guideline content as opposed to two-way dialogue to come to a shared understanding and were supported by international partners who had proposed cuts to the benefit package and, therefore, were unpopular with influential stakeholders (Reference Lee, Nguyen, Park, Hoang and Kim28;29;77;Reference Vo83). Despite support from the Vice Health Minister (29), support for HTA among policy makers remained low. As a result, the lead guideline developer in Vietnam could not justify dedicating staff time and resources for guideline completion.
VN001: Human resources were primarily prioritized for other missions, rather than [guidelines] … if there’s a strong demand from the government, the official requirement from the MOH or from policymakers, it’s going to be strong motivation to push this work [to] move forward better.
Context 2: Guideline developers have a mandate to develop and implement guidelines
If the guideline developer had a mandate to develop and implement guidelines, continued stakeholder dialogue and revisions after publication were important for advancing HTA institutionalization (Table 4). Contrary to the targeted approach to stakeholder engagement in context 1, opportunity for dialogue and input across all stakeholder groups (including professional societies, patient groups, local government, and the public) was important in context 2. This finding was consistent across process and methods guidelines, as well as across mandatory and recommendatory guidelines. Conversely, adherence to methods guidelines in research studies was most closely related to the levels of HTA institutionalization and maturity of HTA research networks at the time of guideline development, not to the level of stakeholder engagement (Supplementary Table S10).
Mechanisms for ongoing stakeholder dialogue following guideline publication in settings with a mandated authority to oversee guideline implementation (further details in Supplementary Table S9)
Table 4. Long description
Examples of guidelines with high, moderate, and low success levels in supporting HTA institutionalization and mechanisms of ongoing dialogue after guideline release.Further details in Table S9.
In Brazil, using HTA methods guideline development as a platform for ongoing dialogue with stakeholders has successfully supported progressive HTA institutionalization. DGITS, a department within the Ministry of Health (MOH) plays a central coordinating role between the benefit package advisory body (CONITEC), the decision maker for incorporation of technologies (MOH), and the network of HTA researchers and practitioners (REBRATS) (Reference Colpani, Kowalski and Stein84;Reference Lima, de Brito and de Andrades54). This coordination role has allowed DGITS to successfully cultivate a system for HTA in the country, coordinating across policy makers, HTA producers at national, state, municipal, and hospital levels, and HTA researchers conducting studies and advancing HTA methods.
BZ001: We are very close to the HTA network, and we are the executive secretariat and responsible ones for the methodological guideline, so it makes things way easier. You know the fact that we hold events, we hire for the HTA network, we hire the courses for educating them on the most recent methods, and etc. So, we have an agenda power on this, and it’s easier to disseminate this knowledge and to guarantee that the guidelines are being used. We are so directly connected to them, so fostering the HTA network of experts closely is very helpful.
The Brazilian methods guides have been successful not only in increasing the number and quality of HTA studies in the country but also in improving the quality of evidence used for policy making.
The third edition of the Canadian guidelines for the economic evaluation of health technologies, however, was drafted exclusively by Canadian health economists, with external peer review. The revision was undertaken after CCOHTA obtained a substantial increase in funding and support, and therefore expanded the guidelines to non-drug technologies. Although the updated guidelines were associated with an increase in published economic evaluations of vaccines between 2010 and 2015, there is limited evidence that these evaluations were incorporated into decision-making processes (Supplementary Table S6).
Across contexts, guidelines were most influential when they were treated as living instruments embedded in formal institutional systems rather than static technical documents. Sustained stakeholder dialogue, transparent revision processes, and responsiveness to feedback helped build legitimacy and, therefore, influence change in decision making.
Discussion
Our findings suggest that successful HTA guidelines were developed by those who were open to engaging with stakeholders holding very different perspectives and willing to adapt their views through such engagement. Where developers faced constraints in implementing change, open stakeholder engagement was most important during guideline development to align across key stakeholders with the power to influence change. Where developers were responsible for guideline development and implementation, maintaining channels for engagement across a broad range of stakeholders was most important after publication for continued updates and minor revisions.
Open dialogue with stakeholders may help maintain credibility with professional associations and researchers by keeping abreast of new methods and thinking, while also allowing for incremental change to processes as other stakeholders engage, learn, and change their views on how decisions should be made. Our findings are consistent with institutional theories that written rules and guidelines only influence behavior if they align with how stakeholders believe decisions should be made (Reference Aoki15), and that iterative discussion can gradually align stakeholders (Reference Pahl-Wostl16;Reference Cairney85).
It is often recommended to use legal frameworks to promote adherence to HTA processes (Reference Peacocke, Frønsdal and Heupink86–Reference Jeffery, Chi and Stewart89). However, we found no evidence that presence of laws or regulations improved HTA guideline success. In certain cases, rules and regulations may even constrain the ability to make continued updates to guidelines with stakeholder inputs. Aligned with international guidelines, our findings suggest that countries looking to institutionalize HTA would benefit from understanding the legal environment in which HTA will be implemented in their setting, in order to identify and make necessary legal amendments to uphold the values and principles necessary for effective HTA (Reference Bertram, Dhaene and Tan-Torres Edejer90). For example, without effective accountability mechanisms, a legal mandate may not translate into changed decision-making practice. Our findings highlight that it is the design and content of legal frameworks that are important, not merely putting laws in place.
A final implication concerns methods used to evaluate guideline success. Current research typically assesses methodological adherence in published studies; however, our analysis found that such adherence reflects the maturity of the HTA ecosystem in a setting, rather than the impact of guidelines on changing decision-making practice. Guideline compliance could also be measured by reviewing evidence presented during the decision-making process and assessing adherence to defined procedures.
A strength of our research is that we considered different types of guidelines developed over time and across countries with different geographies, healthcare financing and governance mechanisms, and norms for decision making (Reference Pahl-Wostl16). Nonetheless, alternative explanations for variations in success observed between guidelines may not have been fully captured. Data constraints limited our ability to assess the relative success and contextual factors of each guideline. For historical data, we were often unable to triangulate data sources; for more recent guidelines, we had access to rich interview data, but often insufficient time to properly assess the level of success. Our analysis could not fully address sociocultural norms and attitudes toward the use of evidence for priority-setting, perceptions around what constitutes good practice (Reference Brown91), or institutional culture that shapes the use of evidence in priority-setting (Reference Aoki15).
Another limitation relates to potential selection and respondent bias. Our sample was purposively drawn from countries with documented HTA guidelines, potentially over-representing systems with a narrative of progress. Within these countries, interviewees were predominantly guideline developers, agency staff, and affiliated experts who have perspectives on understanding design intentions and institutional history but may only reflect “official narratives.” Some observations, particularly those concerning the relative importance of legal framework, governance arrangement, and institutional design, might place greater emphasis on factors perceived as important by system architects than on factors relevant to patient groups, industry, and the public. To alleviate this potential bias, we triangulated interviews with a critical review of published literature (including academic critiques and implementation evaluations) and policy documents and actively sought disconfirming evidence, examining cases of failure with the same rigor as successes. Nonetheless, the views of frontline decision makers, patient groups, or industry stakeholders are less represented and could provide an alternative lens on the factors influencing practice change.
We recommend further research on techniques for successful cross-stakeholder learning in HTA. Most literature on health priority-setting addresses evidence synthesis and capacity to manage the decision-making process, rather than establishing the decision-making architecture itself (Reference Castro, Kumar and Suharlim87). Although a recent good practice report provides a checklist for deliberation in both the design of and within the decision-making processes (Reference Oortwijn, Husereau and Abelson92), there remains limited research on how stakeholder deliberation may be conducted effectively and efficiently given a specific context and set of goals. The role that technology may play in sustaining stakeholder engagement also needs exploration. Further research could identify features of successful approaches to guideline implementation used across different settings. This study also highlighted the importance of effective accountability mechanisms while remaining open to learning and change, particularly in settings where healthcare priority-setting may mature faster than higher-level institutions and governance mechanisms. Finally, this study measured success of guidelines in changing decision-making practice, but it did not consider the link between better decision making and improved quality and accessibility of health services at the population level, which is another area for future research.
Conclusion
The promotion of systematic, evidence-informed decisions in health is often encouraged by establishing formal rules for decision making through laws, codes of practice, or operational guidelines. Our study suggests that HTA guidelines influence change not merely by their existence, but through a process of open stakeholder dialogue that underpins their development and revision.
Supplementary material
The supplementary material for this article can be found at http://doi.org/10.1017/S0266462326104218.
Acknowledgments
The authors thank all study participants for their time and insights. We acknowledge the contributions of Kanchanok Sirisorn, Diana Beatriz Bayani, and Angela Kairu to this work. We also thank Oresta Piniazhko (HTA Department, State Expert Centre of MOH, Ukraine), Manuel A. Espinoza (Pontificia Universidad Católica de Chile, Chile), Gihan Hamdy El-Sisi (HTA office and Faculty of Pharmacy, Future University, Arab Academy for Science and Technology and Cairo University, Egypt), Jasmine Pwu (Fu Jen Catholic University and Taipei Medical University, Taiwan), Ying-Li Chen (Centre for Drug Evaluation, Taiwan), Izzuna Mudla Bt Mohamed Ghazali (the Malaysian Health Technology Assessment Section, Malaysia), Somsak Chunharas (National Health Foundation (NHF), Thailand), Hugo Turner (Imperial College London, UK), and Anna Vassall (London School of Hygiene and Tropical Medicine (LSHTM), UK) for their guidance and continued collaboration throughout the work. We are sincerely grateful to the project collaborators who provided coordination and contributed to this work. These include Antonio Migliore (HTAi), Kelly Lenahan (ISPOR), Robert Selby (ISPOR), Wanrudee Isaranuwatchai (HTAsiaLink), and the HTAsiaLink Secretariat team. Grammarly 1.175.0.0, WebUI 2.16.22 was used to check grammatical errors and harmonize spelling upon revision of this manuscript.
Funding statement
This research has received funding support from the National Science, Research and Innovation Fund (NSRF) via the Research and Innovation Acceleration Agency for Competitiveness and Area Development (RCAD) Program Management Unit for Frontier Brainpower and Future Industries) (grant number: B41G670025). HITAP staff in this work is supported by the Thailand Science Research and Innovation (TSRI) Fundamental Fund. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests
All authors declare no conflicts of interest related to the content of this article. C.S. declared no conflict of interest at the time the study was conducted, analyzed, and the manuscript was written. Before manuscript submission, there was a subsequent change in affiliation to Thermo Fisher Scientific.
Open access
