Introduction
Governments around the globe are under pressure to ensure efficiency in the provision of their health system, while safeguarding the quality of care, equity, access, and choice. As such, priorities have to be set, either explicitly or implicitly, to determine which health technologies to assess. Health technology assessment (HTA) is increasingly used to inform such decisions. Due to the broadness of the concept of what health technology means (1), HTA has undergone various interpretations and has been recently redefined by an international joint task group (Reference O'Rourke, Oortwijn and Schuller2) as a multidisciplinary process that uses explicit methods to determine the value of a health technology at different points in its life cycle. The purpose is to inform decision-making in order to promote an equitable, efficient, and high-quality health system. The impact of a health technology may be assessed by examining the intended and unintended consequences of using such a technology compared with existing alternatives. This can include health outcomes achieved for money spent, (Reference Porter3) social or psychological aspects of living with a disease, organizational changes in healthcare provision, or ethical and legal implications associated with using a health technology. Thus, ideally HTA is a mechanism linking research with policy and practice, considering the best available evidence. Furthermore, the process is formal, systematic, and transparent, providing high-quality reliable information on the health technology used.
As HTA is increasingly used, the diverse need within and across health systems for capacity building for HTA emerges. Recently, the Scientific Development and Capacity Building Committee (SDCB) of Health Technology Assessment International (HTAi) defined capacity building (Reference Pichler, Oortwijn, Ruether and Trowman4). It highlighted that capacity building for HTA includes a process by which individuals and organizations develop or strengthen abilities related to understanding, providing input to, conducting, or utilizing HTA for health policy and decision-making, as well as developing awareness and support in the environment within which HTA is used. This means that HTA must be considered within a country's decision context, taking into account the country's priorities, its health system, and its societal and cultural characteristics. HTA information utilization is dependent upon the necessary “hard” and “soft” skills acquired (Reference Mueller, Gutiérrez-Ibarluzea, Schuller, Chiumente, Ahn and Pichon-Riviere5), its value for and linkages to policy- and decision-making, and its acceptability in policy and practice. Due to the multidisciplinary nature of HTA and the application across the life cycle (technology inception, pre market, during market approval, post market and disinvestment), a broad range of competencies is required (Reference Mueller, Gutiérrez-Ibarluzea, Schuller, Chiumente, Ahn and Pichon-Riviere5;Reference Wild, Stricka and Patera6). However, the main focus of this paper is on the competencies needed for conducting HTA.
According to Bloom's taxonomy of learning, individual and global core competencies are defined as an essential minimal set of combined attributes encompassing specific knowledge, multidisciplinary skills, and required attitudes (KSAs) (Reference Bloom, Engelhart, Furst, Hill and Krathwohl7). These competencies enable an individual to perform a set of tasks to an appropriate standard in order to work efficiently and effectively (8–Reference Moynihan, Paakkari, Välimaa, Jourdan and Mannix-McNamara10). Bloom's taxonomy (Reference Bloom, Engelhart, Furst, Hill and Krathwohl7) of learning and its revised version (Reference Anderson, Krathwohl, Airasian, Cruikshank, Mayer and Pintrich11) are divided into three distinct learning objectives of educational activities: cognitive, affective, and psychomotor. Acquiring knowledge (K) is at the heart of the cognitive domain, the affective deals with feelings and emotions about the topic under study and can be assessed by the ability to listen and actively work in a team and provide the required feedback to each individual, whereas the psychomotor domain is about developing skills of “action.” This taxonomy also applies to the field of HTA. One of the challenges that organizations employing HTA professionals may face is that recruitment choices and the skill set of people filling the roles might not be in alignment (12). Competency frameworks have been found to provide a powerful way of ensuring this alignment (Reference Calhoun, Ramiah, Weist and Shortell13;Reference Greiner and Knebel14).
In order to initiate a discussion about competencies’ requirement of HTA agencies toward staff recruitment and their approach toward continuous professional development, we convened a workshop at the HTAi annual meeting in Washington, DC, 2014, under the umbrella of International Network of Agencies for HTA (INAHTA). INAHTA is a network of public non-for profit HTA agencies supporting evidence-based decision-making affecting their respective countries. These agencies cooperate and share information on the production and dissemination of HTA reports through the support structure made available by the INAHTA. The outcome of this workshop was the identification of several basic “hard” and “soft” competencies that would be essential for those who produce and those who need to understand and implement HTA (Reference Mueller, Gutiérrez-Ibarluzea, Schuller, Chiumente, Ahn and Pichon-Riviere5). Furthermore, they can be broken down into individual KSAs and differentiated into core and secondary competencies to be itemized for the various members of the HTA community, such as users, producers, partners, or knowledge brokers. We were aware that such a formal listing of competencies needed for the full scope of HTA is not available. Hence, the results from this workshop were utilized to shape scientific investigation to define competencies, establish consensus, and link these competencies to the KSAs.
As a follow-up of the workshop mentioned above, we undertook this study, specifically aimed to
(1) understand and list current competencies offered at universities and/or HTA institutions (i.e., HTA agencies, patient organizations, or organizations offering different HTA courses),
(2) identify existing educational and training programs for HTA around the globe, and
(3) create a common understanding of the various competencies which could be embedded in (future) HTA educational and training programs.
Currently, there are additional initiatives underway, for instance by the HTAi interest group on Ethics, whose focus is on developing specific competencies for a particular dimension that is being assessed when conducting HTA.
Methods
We conducted four interlinked research activities between 2016 and 2019 to gather information on the KSAs needed for HTA-specific expertise. These activities (A–E) are depicted in Figure 1.
Chronological order of interlinked research activities between 2016 and 2019.
Below we elaborate on each of these activities.
Activity A: HTA Handbooks/Toolkits Review
A two-step process was undertaken: determination of relevant documents followed by a review. The literature search was conducted on key resources including PubMed, Embase, and Google scholar, as well as fifty INAHTA member HTA agency Web sites for HTA handbooks or toolkits. Here, “handbook” means instructions or toolkit, especially for learning a subject, a handbook or guidance covering what HTA means (epistemology) and which dimensions and competencies are embraced or required. These handbooks or toolkits do not refer to “methods papers” of HTA organizations but are background papers used for building capacity. The search was first carried out in 2016 and updated in 2019. The main inclusion criteria were “HTA” and best practices in HTA targeting specific domains. Guidelines on conducting HTA were excluded. Supplementary File A contains further details on the criteria used for retrieving HTA toolkits and handbooks. Any disagreement on inclusion was resolved by discussion between two authors, and if consensus was not achieved, a third author was consulted. At first, twenty-four documents were retrieved, out of which six were selected based on the inclusion and exclusion criteria as listed in Supplementary File A, Table A1a.
We used the domains of HTA specified in the EUnetHTA Core Model (15): safety, clinical effectiveness, costs and economic evaluation, organizational aspects, ethical and social aspects, and HTA processes to group competencies specified in “handbooks/toolkits.” Thematic analysis was applied to extract and summarize the topics. The purpose was to obtain maximum clarity in the concepts and definitions of the competencies, mainly knowledge (K). We used a combination of sources to define competencies, as a single and complete database of such definitions, necessary for classification specific to the domains, was not available. The main source was the HTA glossary (1). When the definitions were unavailable in the HTA glossary, we used the EUnetHTA glossary of HTA adaptation terms (16). Information on secondary competencies such as skills in writing, project management, or communication was obtained from general or business-specific dictionaries.
The definitions are presented in Supplementary File A. The main challenge faced in the handbook/toolkits review was the difficulty in retrieving information from various webpages. In some instances, when descriptions were not readily available, we directly contacted the staff.
Activity B: Survey to Universities Offering MSc Programs
In parallel to the HTA programs and handbooks/toolkits review, we surveyed universities offering Master's degree courses in HTA (Supplementary File B). The inclusion criteria were university-level programs on HTA and covering more than only one aspect of HTA, such as economic evaluation, as this refers to a particular aspect within the plethora of domains under HTA definition. We excluded institutions that did not provide a description of their program on their Web site or lacked details regarding objectives and competencies to be achieved. To identify additional courses, we also considered the Vortal of HTAi, the list of educational activities created and maintained by the Pan American Health Organization (PAHO) (17), and the list of courses maintained by INAHTA (18). In addition to the survey responses, information on the courses/programs from the respective Web sites of the eight graduate-level programs, which fulfilled the inclusion and exclusion criteria, was captured.
Activities C and D: Workshops II and III
In 2016, we conducted a workshop at the HTAi conference in Tokyo (Figure 1: activity C) under the title “Capacity building for effective and efficient HTA: engaging stakeholders to determine competencies and skills toward professionalization of HTA.” At this follow-up workshop, information was shared about (i) the rationale behind competency-based programs as initially captured during a previous workshop in Washington, DC (see the Introduction for further information), (ii) the preliminary results from the analysis of the handbook/toolkits review (activity A), and (iii) the result of the universities’ MSc program survey (activity B). The eighteen workshop participants came from different HTA institutions and universities. Building on Bloom's taxonomy of knowledge (Reference Anderson, Krathwohl, Airasian, Cruikshank, Mayer and Pintrich11), the workshop participants were asked to share their views on core and secondary or “desirable” competencies classified into building capacity standards of concepts, procedures, and attitudes. See Supplementary File C for the questionnaire and the responses of the participants. We used this categorization, as according to Bloom's updated taxonomy, concept, or principle, procedure and metacognitive are the different types of knowledge that could be learned and taught (Reference Anderson, Krathwohl, Airasian, Cruikshank, Mayer and Pintrich11).
Supplementary File C includes the compiled and evaluated responses and the evaluation criteria (Reference Busse, Orvain, Drummond, Felix, Malone and Alric19), including the steps of conducting a HTA. These were a general understanding of HTA, scoping, prioritization, evidence collection, and appraisal, including the HTA dimensions such as clinical and economic aspects, ethical, legal, patient, and social and organizational (ELSO) aspects, and more recently, environmental aspects (Reference Polisena, De Angelis, Kaunelis and Gutierrez-Ibarluzea20).
In 2017, a third workshop on “Building consensus on defining core competencies required for high quality effective and efficient practice of HTA” was conducted during the HTAi meeting in Rome (Figure 1: activity D). This workshop gave us again the opportunity to share the outcome of the analysis of the handbook/toolkits review (activity A) and the universities’ survey (activity B). Moreover, at this workshop, the participants were asked to reach a consensus on KSAs using a comprehensive list of competencies identified by the authors from activities A and B and enlarged by new ideas generated from the previous workshop and analysis (activity C). Besides the usual attendee enrollment for the workshop, we purposely invited senior representatives from universities and senior members from HTA agencies. Of the twenty-three participants, seven participated in the scoring of the competencies using a Likert scale ranging from 1 (strongly disagree) to 10 (strongly agree). To present the results, we calculated the mean of the responses per question.
Activity E: Questionnaire to Universities, Institutions, and HTA Agencies
This 2018–19 survey using Google forms updated the information collated from the previous survey (activity B), augmented with questions on the type of course, the mode of teaching, and the kind of training programs offered by HTA agencies. The survey questions are detailed in Supplementary File D. These ten questions were based on those from the survey conducted by the ECHTA/ECAHI Project – Working Group 5 (Reference Douw, Vondeling and Bakketeig21). The aim was to gain an understanding of the competencies currently covered by educational programs. It covered topics such as teaching methodology, type, mode, and frequency of courses, in addition to the content of the curriculum and learning outcomes. It also included queries on collaboration with external institutions to shed light on the linkages between educational institutions and between agencies as employees of HTA professionals. The SDCB committee of HTAi distributed the survey together with a cover letter communicated by email to the members of HTAi. Additionally, Web sites of universities offering courses in HTA or related fields were hand-searched for their educational programs. We used a purposive and snowball sampling approach to invite staff from key educational institutions who may not have been members of HTAi or INAHTA. In total, we received sixty responses, of which seven were staff of HTA agencies, and forty respondents were affiliated to, or were staff of, universities or research institutions. The thirteen respondents who were not based at universities or research institutions recommended universities or institutions to be contacted. Following this, twenty-four courses were added to our initial list.
A separate questionnaire (available in Supplementary File D) was sent to INAHTA member HTA agencies particularly asking for opportunities on internal professional development programs. In this case, INAHTA distributed the survey and the cover letter among their members using their newsletter. In addition, the survey to the universities and HTA agencies was conducted as part of a broader review of programs and curriculum offered. Therefore, the scope of questions posed in the online questionnaire extended beyond the aims of this study. The complementary survey, which specifically targeted HTA agencies, led to 17 responses.
Results
Activity A: HTA Handbooks/Toolkits Review
From the review of the six selected handbooks/toolkits, fifty-five KSAs were obtained and these were distributed among the domains and processes.
The results obtained from the handbooks/toolkits review are listed in Supplementary File A.
Activity B: Survey to Universities Offering MSc Programs
An overview of the competencies covered in the postgraduate programs surveyed is presented in Supplementary File B. These programs were mostly offered in Europe; one was offered in Canada and another had an intercontinental program. The duration of the courses varied and so did their learning outcomes. All courses had compulsory and elective modules, and lasted between 1 and 2 years. Of the eight programs examined, all included clinical effectiveness and economical evaluation; three programs had a focus on the ethical domain; most courses partially included organizational, patient, and social themes and only two programs fully integrated these domains into their curriculum.
Activity C: Workshop II (2016)
Out of the eighteen participants, twelve filled out the questionnaire; responses are detailed in Supplementary File C. The participants emphasized the importance of gaining knowledge on the healthcare context, policy issues, regulations, and decision-theory, which could be classified under the broad heading of Introduction to HTA. In the opinion of the participants, skills such as those in scoping, evidence collection, and appraisal, management of projects, report writing, and communication were important. Attitude toward work, such as teamwork, open-mindedness, critical thinking, or system view on HTA were some of the soft skills.
Activity D: Workshop III (2017)
Twenty-four competencies related to knowledge, which are listed in Supplementary File D, were presented for scoring. It appeared that 25 percent of the competencies received a mean score of 8 (moderate relevance). Competencies related to knowledge were, for example, “know the basics of HTA and its particular methodologies” or “understand how HTA studies were initiated.” “Role of economic analysis in health care policymaking,” however, had the lowest mean score of 5 (relevant), and similarly, “understand the utility of decision analysis” got a score of 5.5.
With regard to skills, also twenty-four competencies were listed (Supplementary File D). The competency on the identification and location of sources of information needed for a HTA received the highest mean score of 10 (very highly relevant), whereas the competency on “how to improve a system to assess the performance of health care organizations” obtained a low score of 4 (slightly relevant), which can be interpreted as a competency not found to be necessary for the producers of HTA.
The mean scores for the nine listed competencies for attitude in Supplementary File D were quite homogenous, with the highest mean score of 9 (high relevance) and the lowest mean of 7 (mildly relevant). The four competencies that had a mean score of 9 covered topics like “critical approach to evidence” or “unbiased objective approach to all phases of HTA,” whereas “openness to patient/public involvement in HTA” had a mean score of 7.
Activity E: Questionnaire to Universities, Institutions, and HTA Agencies
This follow-up survey highlighted the discontinuity of many of the programs initially surveyed in 2016/2017; out of the eight graduate-level courses identified in the previous survey, two programs are currently active and one offers a doctoral program.
Out of the forty-one programs identified, nineteen (46.3%) are offered in Europe, two (4.9%) in Asia, ten (24.4%) in the Americas, four (9.8%) in Africa, two (4.9%) in Australia, and four are diverse. The type of programs offered are: face-to-face sessions, blended online programs, distance learning programs, and various short courses covering different competencies of HTA (e.g., principles and concept, report writing, HTA methodology, and economic evaluation). Most graduate-level HTA courses are modular in nature or are offered under a degree program such as Public Health or Healthcare management, Health economics or Epidemiology. These details can be found in Table 1 and Supplementary File E.
List of graduate and postgraduate HTA programs offered by universities
Seventeen out of fifty-two agencies responded to the survey targeted to HTA agencies, resulting in a response rate of 32.7 percent. Of the responding HTA agencies, 82.4 percent either offer in-house hands-on training or collaborate with universities. In-house trainings are offered either by senior staff or by external experts when junior staff does not have the necessary competencies or necessary experience (22;23). Most agencies offer training in HTA (100 percent), followed by literature searching (87.5 percent), systematic literature reviews (75 percent), health economics, or economic evaluation (75 percent), and training in ELSO issues (62.5 percent). In reference to the enquiry on collaboration, ten respondents mentioned that they collaborate in some manner, such as in joint staff programs with universities.
Following the documents' review, two workshops, and two surveys, a list of competencies for HTA was compiled. Tables 2 and 3 elaborate on each competency in relation to KSAs.
Competencies in HTA processes identified from handbooks/toolkits review, workshops, and questionnaires
K, knowledge; S, skills; A, attitude.
Competencies in HTA domains identified from handbooks/toolkits review, workshops, and questionnaires
K,knowledge; S, skills; A, attitude.
These set of competencies are grouped into HTA domains and steps (Reference Busse, Orvain, Drummond, Felix, Malone and Alric19): introduction; scoping; acquire; appraise and synthesizing. It appeared that the Master's degree courses and handbooks/toolkits that we reviewed did not cover all of the dimensions of HTA analysis in an equal and standardized way. The ELSO aspects and the more recently considered environmental aspects were lacking in some of the programs, whereas clinical effectiveness and economical evaluation were always included.
Discussion
This study involved several research activities over a period of 4 years.
A preceding workshop (Reference Mueller, Gutiérrez-Ibarluzea, Schuller, Chiumente, Ahn and Pichon-Riviere5) tried to classify the broad range of competencies needed to produce and use a HTA report. However, the resulting number of competencies identified was rather limited and did not cover the broad range of activities and processes of HTA. Having identified this important gap, this study attempted to systematically explore and develop an overview of the competencies necessary for the different steps in HTA, such as scoping of HTA topics, acquiring and appraising evidence, dissemination, or implementation of a HTA report. An initial effort was also undertaken to collate competencies relevant to the individual assessment domains (12).
Competencies differ depending on the job within the HTA ecosystem. This ecosystem consists of those who are using the HTA, producing HTA, and those with whom producers and users are collaborating in the field. In activity D, the competence on “mastering the concept of accurate clinical evaluation for understanding HTA” received a mean score of 7. This means that a user does not necessarily need to be competent in all aspects of a report, such as the design of individual studies. However, it would be important for the user to know and acquire skills to support the implementation of the recommendations and fully understand and communicate the desirable and undesirable consequences thereof. A knowledge broker in HTA would disseminate the recommendation(s) and the tradeoffs, whereas a clinician may still need to critically appraise individual studies. Skills in “clear communication of the purpose of HTA and use of methods” received a mean score of 5, which may be because a HTA doer is not in charge of communicating the project details externally.
The responses from the agencies show high frequencies of in-house training. The training often includes ELSO issues that have been sparsely covered by educational institutions. However, this can be mitigated by academia offering a core set of competencies aligned with the requisites of those conducting HTA. Those identified here could be suitable to inform the curricula for an introductory course in HTA-targeting stakeholders from a diverse background with a varied level of education. The programs can be tailored to the time availability of learners, their professional experience and background, and local learning needs. The order of competencies listed does not reflect the sequence of teaching or their order of importance. This flexibility in the combination of competencies has been observed in other fields, such as evidence-based practice for professionals (Reference Albarqouni, Hoffmann, Straus, Olsen, Young and Ilic9). Also, courses in public health, hospital management, or health sciences could act as a base for coordinated HTA activities, for example in raising awareness and advocacy for HTA (Reference Wild, Stricka and Patera6).
As HTA is a multidisciplinary field, individual contributors to, or users of, HTA are from a specific profession and do not need to have expertise in all areas or domains in HTA. Nevertheless, to be able to conduct, understand, and use HTA effectively, certain core competencies are recommended; it is especially desirable to cover the new definition of HTA that embraces all the required domains of HTA analysis. Furthermore, HTA training and educational programs are usually not targeted toward the need of users or toward mitigating their lack of certain competencies.
One workshop participant summarized the necessity of defining core and secondary competencies for HTA doers as follows: “All team members have an understanding of core concepts, but only certain individuals need to know how to perform certain tasks within a HTA. If the members align within a discipline, then they need a core competency in procedural aspect.” Additionally, we view these competencies and additional ones as a contemporary and dynamic set. As the HTA field evolves and matures, new competencies will need to be added to the set, as observed, for instance, in a recent paper by the members of the HTAi SDCB (Reference van der Wilt, Rüther and Trowman24) on the increasing need for (public) deliberation in HTA research or the need to include new domains to HTA analysis such as environmental aspects (Reference Polisena, De Angelis, Kaunelis and Gutierrez-Ibarluzea20).
Future work should focus on defining detailed core competencies needed for different training levels for different stakeholders in the field and comparing different approaches to teaching these competencies.
Limitations
The activities presented in this paper have some limitations. We are aware that limiting the source materials to general HTA handbooks/toolkits and educational courses may have prevented detailed insights into competencies necessary for specific domains and processes. However, by comparing multiple documents from various sources, shared knowledge and skills related to the different domains have been revealed.
The relatively small sample of participants attending the workshops and the number of respondents to the survey may have influenced the results. As such, the results of this paper should be seen as a first attempt toward the formulation of a set of HTA core competencies.
Conclusions
Based on the activities conducted, we have been able to identify an initial list of competencies for HTA. Further consensus among the HTA community is necessary to inform the curricula using a core set of competencies that address all components of HTA as reflected in the new definition. Additionally, as noted by the HTAi SDCB Committee in its paper (Reference Pichler, Oortwijn, Ruether and Trowman4), consideration of a broad range of capacity building activities in the field of HTA is required as well. Preparation of a checklist or a “menu” of options mirroring the diversity of HTA will ensure the coverage of specific needs of the various members of the HTA community. Collaboration with relevant groups, for example with relevant HTAi interest groups, could be enhanced to align the activities in this field. This would accelerate defining the core HTA competencies required for the different stakeholders involved in HTA.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462320001919.
Acknowledgments
The authors gratefully acknowledge the contribution of the HTAi members to the surveys and HTAi workshops in responding and engaging in robust discussion. We thank T. Schuller, IHE, Canada for her contribution and D. Panteli, Berlin University of Technology, Germany for reviewing and providing feedback.
Funding
This research received no specific grant from any funding agency, commercial or not-for-profit sectors.
