No CrossRef data available.
Article contents
Topic selection process in health technology assessment agencies around the world: a systematic review
Published online by Cambridge University Press: 07 February 2022
Abstract
Objective
The purpose of this study was to systematically review the process for topic selection by health technology assessment (HTA) agencies around the world to provide the knowledge base for the improvement of topic selection frameworks in HTA agencies.
Methods
A systematic search was conducted in PubMed and EMBASE to identify papers up to February 2019. Gray literature was identified by screening the Web sites of HTA agencies on the nonprofit member list of the International Network of Agencies for Health Technology Assessment (INAHTA). Data were extracted for each HTA agency and synthesized, with issues including general contextual information about each agency and the process of topic selection.
Results
Out of forty-nine nonprofit members of INAHTA, a total of seventeen HTA agencies with a framework for topic selection were identified from twenty-two included papers/documents. Multiple criteria were used for topic selection in all frameworks and agencies undertook multiple steps, which could include the specification of criteria for topic selection, identification of topics, short listing of potential topics, scoping of potential topics, scoring and ranking of potential topics, and deliberation and decision on final topics for HTA. Shortcomings were found in relation to methods of scoring and ranking as well as lack of monitoring and the evaluation of the process.
Conclusions
Our study provides insights into the current practice of topic selection in HTA agencies. Multiple criteria decision analysis methodology appears highly relevant to these processes. A consensus approach for the development of methods of topic selection would be valuable for the HTA community.
Information
- Type
- Article Commentary
- Information
- Copyright
- Copyright © The Author(s), 2022. Published by Cambridge University Press
References
Specchia, ML, Favale, M, Di Nardo, F, Rotundo, G, Favaretti, C, Ricciardi, W, et al. How to choose health technologies to be assessed by HTA? A review of criteria for priority setting. Epidemiol Prev. 2015;39:39–44.Google ScholarPubMed
Noorani, HZ, Husereau, DR, Boudreau, R, Skidmore, B. Priority setting for health technology assessments: A systematic review of current practical approaches. Int J Technol Assess Health Care. 2007;23:310–5.CrossRefGoogle ScholarPubMed
Kristensen, FB, Husereau, D, Huić, M, Drummond, M, Berger ML, Bond, K, et al. Identifying the need for good practices in health technology assessment: Summary of the ISPOR HTA council working group report on good practices in HTA. Value Health. 2019;22:13–20.CrossRefGoogle Scholar
Oortwijn, WP, Determann, DP, Schiffers, KM, Tan, SSP, van der Tuin, JM. Towards integrated health technology assessment for improving decision making in selected countries. Value Health. 2017;20:1121–30.CrossRefGoogle ScholarPubMed
Neumann, PJ, Drummond, MF, Jönsson, B, Luce, BR, Schwartz, JS, Siebert, U, et al. Are Key principles for improved health technology assessment supported and used by health technology assessment organizations? Int J Technol Assess Health Care. 2010;26:71–8.Google ScholarPubMed
Henshall, C, Oortwijn, W, Stevens, A, Granados, A, Banta, D. Priority setting for health technology assessment. Theoretical considerations and practical approaches. Priority setting subgroup of the EUR-ASSESS project. Int J Technol Assess Health Care. 1997;13:144–85.CrossRefGoogle ScholarPubMed
Castro, HEKR, Suharlim, C. A roadmap for systematic priority setting and health technology. Assessment (HTA). 2020. Available from: https://www.msh.org/resources/a-roadmap-for-systematic-priority-setting-and-health-technology-assessment-hta-a-practical [accessed Oct 1 2021].Google Scholar
Oortwijn, WJM, Baltussen, R. Evidence-informed deliberative processes: A practical guide for HTA bodies for legitimate benefit package design (Version 2.0). Nijmegen, 2021. Available from: https://www.radboudumc.nl/getmedia/17a96fdb-553b-4e68-81ab-4d8d9a7f9ff1/Guide-on-EDPs-2nd-version.aspx [accessed Oct 1 2021].Google Scholar
Drummond, MF, Schwartz, JS, Jonsson, B, Luce, BR, Neumann, PJ, Siebert, U, et al. Key principles for the improved conduct of health technology assessments for resource allocation decisions. Int J Technol Assess Health Care. 2008;24:244–58.CrossRefGoogle ScholarPubMed
Hagenfeldt, K, Asua, J, Bellucci, S, Jensen, MF, Morland, B, Oortwijn, W, et al. Systems for routine information sharing in HTA. Working Goup 2 Report. Int J Technol Assess Health Care. 2002;18:273–320.CrossRefGoogle Scholar
Garcia-Altes, A, Ondategui-Parra, S, Neumann, PJ. Cross-national comparison of technology assessment processes. Int J Technol Assess Health Care. 2004;20:300–10.CrossRefGoogle ScholarPubMed
Dehnavieh, R. Systematic review of prioritization criteria for topics of HTA projects suggestions for Iran. J Novel Appl Sci. 2015;4:940–46.Google Scholar
Husereau, D, Boucher, M, Noorani, H. Priority setting for health technology assessment at CADTH. Int J Technol Assess Health Care. 2010;26:341–7.CrossRefGoogle ScholarPubMed
Campbell, B, Morris, R, Mandava, L, Murthy, L, Gallo, H, Ong, KJ, et al. Identifying and selecting new procedures for health technology assessment: A decade of nice experience in the United Kingdom. Int J Technol Assess Health Care. 2014;30:454–60.CrossRefGoogle ScholarPubMed
Carlsson, P. Health technology assessment and priority setting for health policy in Sweden. Int J Technol Assess Health Care. 2004;20:44–54.CrossRefGoogle ScholarPubMed
Youngkong, S, Baltussen, R, Tantivess, S, Mohara, A, Teerawattananon, Y. Multicriteria decision analysis for including health interventions in the universal health coverage benefit package in Thailand. Value Health. 2012;15:961–70.CrossRefGoogle ScholarPubMed
Guest, G. Applied thematic analysis. Thousand Oaks, CA: SAGE Publications; 2012.CrossRefGoogle Scholar
Waithaka, D, Tsofa, B, Barasa, E. Evaluating healthcare priority setting at the meso level: A thematic review of empirical literature. Wellcome Open Res. 2018;3:2.CrossRefGoogle ScholarPubMed
Page, MJ, McKenzie, JE, Bossuyt, PM, Boutron, I, Hoffmann, TC, Mulrow, CD, et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ. 2021;372:n71.CrossRefGoogle ScholarPubMed
Oortwijn, WJ, Vondeling, H, Van Barneveld, T, Van Vugt, C, Bouter, LM. Priority setting for health technology assessment in The Netherlands: Principles and practice. Health Policy. 2002;62:227–42.CrossRefGoogle ScholarPubMed
Jankauskiene, D, Petronyte, G. A model for HTA priority setting: Experience in Lithuania. Int J Technol Assess Health Care. 2013;29:450–5.CrossRefGoogle Scholar
Kosherbayeva, L, Hailey, D, Kurakbaev, K, Tabarov, A, Kumar, A, Gutzskaya, G, et al. A process of prioritizing topics for health technology assessment in Kazakhstan. Int J Technol Assess Health Care. 2016;32:147–51.CrossRefGoogle ScholarPubMed
Shani, S, Siebzehner, MI, Luxenburg, O, Shemer, J. Setting priorities for the adoption of health technologies on a national level—The Israeli experience. Health Policy. 2000;54:169–85.CrossRefGoogle ScholarPubMed
Bastian, H, Scheibler, F, Knelangen, M, Zschorlich, B, Nasser, M, Waltering, A. Choosing health technology assessment and systematic review topics: The development of priority-setting criteria for patients’ and consumers’ interests. Int J Technol Assess Health Care. 2011;27:348–56.CrossRefGoogle ScholarPubMed
Canadian Agency for Drugs and Technologies in Health. Topic Identification and Prioritization Process. 2015. Available from: https://www.cadth.ca/ [accessed Jan 14 2019].Google Scholar
National Institute for Health and Care Excellence. Topic selection. 2018. Available from: https://www.nice.org.uk/about/what-we-do/our-programmes/topic-selection#ta-selection [accessed Jan 15 2019].Google Scholar
The Netherlands Organisation for Health Research and Development. Summary of assessment procedure. 2012. Available from: https://www.zonmw.nl/fileadmin/zonmw/documenten/Corporate/Engelse_website/ZonMw_summary_assessment_procedure.pdf [accessed Jan 16 2019].Google Scholar
Swedish Agency for Health Technology Assessment and Assessment of Social Services. Assessment of method in health care. 2018. Available from: https://www.sbu.se/contentassets/76adf07e270c48efaf67e3b560b7c59c/eng_metodboken.pdf [accessed Jan 16 2019].Google Scholar
The Institute for Quality and Efficiency in Health Care. General method (version 5.0). 2017. Available from: https://www.iqwig.de/en/methods/methods-paper.3020.html [accessed Jan 17 2019].Google Scholar
Health Technology Wales. Our appraisal process. 2019. Available from: http://www.healthtechnology.wales/about/our-appraisal-process/ [accessed Jan 17, 2019].Google Scholar
Health Information and Quality Autority. A guide to health technology assessment at HIQA. 2016. Available from: https://www.hiqa.ie/sites/default/files/2017-01/A-Guide-to-Health-Technology-Assessment.pdf [accessed Jan 17 2019].Google Scholar
Health Quality Ontario. Health technology assessment: Process and method guide. 2018. Available from: https://www.hqontario.ca/Portals/0/documents/evidence/reports/hta-methods-and-process-guide-en.pdf [accessed Jan 18 2019].Google Scholar
Belgian Health Care Knowledge Centre. Topic selection for the annual programme. 2019. Available from: https://kce.fgov.be/en/topic-selection-for-the-annual-programme [accessed Jan 17 2019].Google Scholar
Agency for Care Effectiveness.Medical technology evaluation methods and process guide. 2018. Available from: http://www.ace-hta.gov.sg/public-data/our-process-and-methods/ACE%20methods%20and%20process%20guide%20for%20drug%20evaluation%20(5%20Feb%202018).pdf [accessed Jan 18 2019].Google Scholar
The Malaysian Health Technology Assessment Section. Health technology assessment mannual. 2018. Available from: http://www.moh.gov.my/penerbitan/mymahtas/HTA_MANUAL_MAHTAS.pdf [accessed Jan 18 2019].Google Scholar
The Agency for Healthcare Research and Quality. EHC program topic nomination and selection. 2019. Available from: https://effectivehealthcare.ahrq.gov/about/epc/nomination/ [accessed January 18, 2019].Google Scholar
Institute for Clinical and Economic Review. Guide to understanding health technology assessment (HTA). 2018. Available from: http://icer-review.org/wp-content/uploads/2018/08/ICER-Guide-to-Understanding-Health-Technology-Assessment-6.19.18.pdf [accessed Jan 17 2019].Google Scholar
Health Intervention and Technology Assessment Program. Health technology assessment process guidelines. 2012. Available from: http://www.hitap.net/en/documents/18966 [accessed Jan 15 2019].Google Scholar
Migliore, A, Perrini, MR, Jefferson, T, Cerbo, M. Implementing a national early awareness and alert system for new and emerging health technologies in Italy: The COTE project. Int J Technol Assess Health Care. 2012;28:321–6.CrossRefGoogle ScholarPubMed
Douw, K, Vondeling, H, Oortwijn, W. Priority setting for horizon scanning of new health technologies in Denmark: Views of health care stakeholders and health economists. Health Policy. 2006;76:334–45.CrossRefGoogle ScholarPubMed
DiStefano, MJ, Krubiner, CB. Beyond the numbers: A critique of quantitative multi-criteria decision analysis. Int J Technol Assess Health Care. 2020;36:292–96.CrossRefGoogle Scholar
Liu, G, Wu, EQ, Ahn, J, Kamae, I, Xie, J, Yang, H. The development of health technology assessment in Asia: Current Status and future trends. Value Health Reg Issues. 2020;21:39–44.CrossRefGoogle ScholarPubMed
Sorenson, C, Chalkidou, K. Reflections on the evolution of health technology assessment in Europe. Health Econ Policy Law. 2012;7:25–45.CrossRefGoogle ScholarPubMed
Babigumira, JB, Jenny, AM, Bartlein, R, Stergachis, A, Garrison, LP Jr. Health technology assessment in low- and middle-income countries: A landscape assessment. J Pharm Health Serv Res. 2016;7:37–42.CrossRefGoogle Scholar
Oortwijn, WJ, Vondeling, H, Bouter, L. The use of societal criteria in priority setting for health technology assessment in The Netherlands: Initial experiences and future challenges. Int J Technol Assess Health Care. 1998;14:226–36.CrossRefGoogle ScholarPubMed
Marsh, K, Goetghebeur, M, Thokala, P, Baltussen, R. Multi-criteria decision analysis to support healthcare decisions; Springer International Publishing. 2017.CrossRefGoogle Scholar
Marsh, KD, Sculpher, M, Caro, JJ, Tervonen, T. The Use of MCDA in HTA: Great potential, but more effort needed. Value Health. 2018;21:394–97.CrossRefGoogle ScholarPubMed
Thokala, P, Devlin, N, Marsh, K, Baltussen, R, Boysen, M, Kalo, Z, et al. Multiple criteria decision analysis for health care decision making–An Introduction: Report 1 of the ISPOR MCDA emerging good practices task force. Value Health. 2016;19:1–13.CrossRefGoogle Scholar
Marsh, K, Lanitis, T, Neasham, D, Orfanos, P, Caro, J. Assessing the value of healthcare interventions using multi-criteria decision analysis: A review of the literature. Pharmacoeconomics. 2014;32:345–65.CrossRefGoogle ScholarPubMed
Facey, KM. Patient involvement in HTA: What added value? Pharm Policy Law. 2011;13:245–51.Google Scholar
Wale, JL, Scott, AM, Bertelsen, N, Meade, N. Strengthening international patient advocacy perspectives on patient involvement in HTA within the HTAi patient and citizen involvement interest group—Commentary. Res Involv Engagem. 2017;3:3.CrossRefGoogle Scholar
Facey, K, Boivin, A, Gracia, J, Hansen, HP, Lo Scalzo, A, Mossman, J, et al. Patients’ perspectives in health technology assessment: A route to robust evidence and fair deliberation. Int J Technol Assess Health Care. 2010;26:334–40.CrossRefGoogle ScholarPubMed
Abelson, J, Wagner, F, DeJean, D, Boesveld, S, Gauvin, FP, Bean, S, et al. Public and patient involvement in health technology assessment: A framework for action. Int J Technol Assess Health Care. 2016;32:256–64.CrossRefGoogle ScholarPubMed
Muhlbacher, AC, Kaczynski, A. Making good decisions in healthcare with multi-criteria decision analysis: The use, current research and future development of MCDA. Appl Health Econ Health Policy. 2016;14:29–40.CrossRefGoogle ScholarPubMed
Figure 1. PRISMA diagram. INAHTA, International Network of Agencies for Health Technology Assessment.
Table 1. General information of HTA agencies and literature included in analysis
Table 2. Description of six steps of topic selection
Table 3. Checklist of six stages of topic selection
Qiu et al. supplementary material
Qiu et al. supplementary material
File
347.6 KB
You have
Access
- 10
- Cited by
Introduction
Health technology assessment (HTA) is a useful priority setting tool to inform the allocation of limited health resources (Reference Specchia, Favale, Di Nardo, Rotundo, Favaretti and Ricciardi1;Reference Noorani, Husereau, Boudreau and Skidmore2). The HTA process covers the definition of the policy scope, prioritization of the technologies for assessment (which is also known as topic selection), assessment, and contextualization of findings to support decision making (Reference Kristensen, Husereau, Huić, Drummond and Bond3–Reference Neumann, Drummond, Jönsson, Luce, Schwartz and Siebert5).
The purpose of topic selection is to limit the number of topics that undergo a more comprehensive assessment, thereby allocating the limited resources available to HTA agencies in a more efficient way (Reference Kristensen, Husereau, Huić, Drummond and Bond3;Reference Oortwijn, Determann, Schiffers, Tan and van der Tuin4). Topic selection is not just limited to a process of deciding topics for assessment, but ideally involves multiple steps including the identification of possible technologies for assessment, prioritization, and decision on the possible assessments (Reference Henshall, Oortwijn, Stevens, Granados and Banta6–Reference Oortwijn and Baltussen8).
To select relevant topics through a structured and explicit system is fundamental for a good HTA process (Reference Drummond, Schwartz, Jonsson, Luce, Neumann and Siebert9). The Priority Setting Subgroup of the EUR-ASSESS project (Reference Henshall, Oortwijn, Stevens, Granados and Banta6) has provided some recommendations on topic selection for HTA and many European HTA agencies adopted the recommendations (Reference Hagenfeldt, Asua, Bellucci, Jensen, Morland and Oortwijn10). However, the practice of topic selection differs across HTA agencies.
In 2004, Garcia-Altes et al. (Reference Garcia-Altes, Ondategui-Parra and Neumann11) compared HTA processes among agencies in four different countries and found that there was a lack of explicit processes for topic selection. In 2007, Noorani et al. (Reference Noorani, Husereau, Boudreau and Skidmore2) reviewed twelve current priority-setting frameworks in eleven HTA agencies and identified differences across HTA agencies regarding the procedures used in topic selection, including categorizing, scoring, and weighing of policy criteria. More recently in 2015, two reviews analyzed the criteria used during topic selection across agencies and revealed differences between organizations in the number and nature of the criteria used (Reference Noorani, Husereau, Boudreau and Skidmore2;Reference Dehnavieh12), but did not consider the whole topic selection process.
Furthermore, more detailed information on the topic selection process has now been made available by many countries including Canada (Reference Husereau, Boucher and Noorani13), UK (Reference Campbell, Morris, Mandava, Murthy, Gallo and Ong14), Sweden (Reference Carlsson15), and Thailand (Reference Youngkong, Baltussen, Tantivess, Mohara and Teerawattananon16), but this new information is yet to be analyzed and compared systematically. Building on these previous studies, the purpose of this study was to systematically review the published papers and gray literature on the process for selecting HTA topics in different countries, to identify the steps involved, and to provide the knowledge base for the improvement of topic selection frameworks in HTA agencies.
Methods
Eligibility Criteria
Published papers and gray literature (e.g., documents, reports, process, and method guides from the HTA agency Web sites) in English that covered information on the process, pathway, framework, or method used to select topics within national or regional HTA agencies were included. Only those that provided a complete description of the whole selection process (i.e., from the identification of potential topic to decision on final topics) in each HTA framework were included.
Those that were not directly related to HTA topic selection (such as topic selection for clinical research and horizon scanning), merely mentioned the identification of topic without priority setting, just provided selection criteria without a description of other steps or only reported theoretical frameworks were excluded. Reviews and conference abstracts were also excluded.
Information Sources and Search Strategy
Based on three previously published systematic reviews on topic selection (Reference Specchia, Favale, Di Nardo, Rotundo, Favaretti and Ricciardi1;Reference Noorani, Husereau, Boudreau and Skidmore2;Reference Dehnavieh12), search strategies were developed with keywords including “health technology assessment,” “priority setting,” “topic selection,” and “topic identification” then applied in academic databases (PubMed and EMBASE) to identify relevant studies up to February 2019 (Supplementary Appendix Table 1).
Gray literature was identified by screening forty-nine Web sites of HTA agencies from the nonprofit member list of the International Network of Agencies for Health Technology Assessment (INAHTA) using “topic selection,” “priority setting,” “working process,” “process guide,” and “method guide” as keywords (Supplementary Appendix Table 2).
Reference and citation searching were performed on included studies to identify other relevant articles. Additionally, bibliographies of the three related reviews (Reference Specchia, Favale, Di Nardo, Rotundo, Favaretti and Ricciardi1;Reference Noorani, Husereau, Boudreau and Skidmore2;Reference Dehnavieh12) were checked manually for further potentially relevant publications.
Selection Process
After duplication, citations were assessed for relevance on the basis of title and abstract by two reviewers (YQ and YX). Citations that were considered relevant by at least one reviewer were retrieved for further full-text review. Eligible citations were included after assessment against selection criteria by both reviewers (YQ and YX).
Data Collection and Data Items
Data were extracted and synthesized by the HTA agency. A template with three sections was developed for data extraction and synthesis. Section one was a table for extracting general context information about the agency, including name of agency, country, and type of technology for topic selection. Section two was a table for extracting information on steps of topic selection in each agency, which was developed by a thematic analysis (Reference Guest17;Reference Waithaka, Tsofa and Barasa18) of reports from the HTA agencies that provided most information about the topic selection process. Section three was a table for extracting data on criteria for topic selection and stakeholders involved in deliberation, which was adapted from previous related reviews (Reference Specchia, Favale, Di Nardo, Rotundo, Favaretti and Ricciardi1;Reference Noorani, Husereau, Boudreau and Skidmore2;Reference Dehnavieh12;Reference Youngkong, Baltussen, Tantivess, Mohara and Teerawattananon16).
Two reviewers (QY and YX) collected data independently from the reports using the template for data extraction and synthesis. Missing data were marked as “not mentioned” or “unclear.” The two reviewers discussed the data after extraction and disagreements were resolved through consensus. As the reports included in this review were descriptive rather than quantitative, and because of the nature of the research question, no formal assessment of the quality or bias of included items was undertaken. The reporting of this review conforms with the PRISMA 2020 statement (Reference Page, McKenzie, Bossuyt, Boutron, Hoffmann and Mulrow19) (see Supplementary Appendix Tables 3–5 for results of data extraction).
Results
Selected Literature
As shown in the PRISMA diagram (Figure 1), after deduplication and exclusion of non-English articles and sifting based on the titles, forty-five articles remained from 788 potentially relevant articles yielded in PubMed and EMBASE. After examining abstracts, twenty-seven articles were excluded (twenty-five irrelevant papers and two reviews), and full texts of the remaining eighteen articles were retrieved and assessed for eligibility. After exclusion for limited information on the process of topic selection (n = 9) and repeated content (n = 1) and the addition of one article identified by manual screening of three reviews, nine published (Reference Husereau, Boucher and Noorani13–Reference Youngkong, Baltussen, Tantivess, Mohara and Teerawattananon16;Reference Oortwijn, Vondeling, Van Barneveld, Van Vugt and Bouter20–Reference Bastian, Scheibler, Knelangen, Zschorlich, Nasser and Waltering24) articles were selected for inclusion. In terms of gray literature, forty-nine Web sites of HTA agencies were screened and twenty-four potentially relevant resources were identified. After the exclusion of two non-English documents and nine documents with insufficient details, thirteen (25–36) were selected for inclusion. As such, a total of twenty-two papers/documents were included for analysis (see Supplementary Appendix Tables 6–8 for information of reports excluded).
PRISMA diagram. INAHTA, International Network of Agencies for Health Technology Assessment.
Overview of HTA Agencies Identified
Seventeen HTA agencies from fifteen countries were identified from twenty-two included papers/documents (Table 1). There were four from Asia including ACE (Singapore) (34), HITAP (Thailand) (Reference Youngkong, Baltussen, Tantivess, Mohara and Teerawattananon16;38), HTAD (Kazakhstan) (Reference Kosherbayeva, Hailey, Kurakbaev, Tabarov, Kumar and Gutzskaya22), and MaHTAS (Malaysia) (35); eight from Europe including HIQA (Ireland) (31), HTW (Wales, UK) (30), ICHTA (Israel) (Reference Shani, Siebzehner, Luxenburg and Shemer23), IQWiG (Germany) (Reference Bastian, Scheibler, Knelangen, Zschorlich, Nasser and Waltering24;29), KCE (Belgium) (33), MRU (Lithuania) (Reference Jankauskiene and Petronyte21), NICE (England, UK) (Reference Campbell, Morris, Mandava, Murthy, Gallo and Ong14;26), SBU (Sweden) (Reference Carlsson15;28), and ZonMW (Netherlands) (Reference Oortwijn, Vondeling, Van Barneveld, Van Vugt and Bouter20;27); and there were four from North America including AHRQ (USA) (36), CADTH (Canada) (Reference Husereau, Boucher and Noorani13;25), HQO (Canada) (32), and ICER (USA) (37).
General information of HTA agencies and literature included in analysis
a For HTA report program.
b For Technology Appraisal Program and Medical Technologies Evaluation Program.
Process of Topic Selection
Reports of CADTH (Canada) and SBU (Sweden), which provided the most comprehensive descriptions of the topic selection process, were reviewed thoroughly by both reviewers (YQ and YX) to identify the steps of topic selection. The key steps within CADTH and SBU were summarized by two reviewers independently, and a draft template of common steps along with their definitions was produced after discussion. These were piloted on other agencies and revised until saturation of information was reached (Reference Guest17). At which point the final template with six steps was produced (Table 2).
Description of six steps of topic selection
Not all six steps were included or described in detail in all frameworks (Table 3). The steps of criteria for topic selection, topic identification, and deliberation and decision on final topics for HTA were included in the framework of all HTA agencies. However, the other three stages (short listing of potential topics, scoping of potential topics, and scoring and ranking of potential topics) were not always included. There were nine frameworks (CADTH, NICE, IQWiG, ZonMW, SBU, HTW, KCE, ACE, and MaHTAS) that applied short listing to screen the potential topics; twelve frameworks (CADTH, HITAP, NICE, IQWiG, SBU, HTW, HIQA, HQO, KCE, ACE, MaHTAS, and AHRQ) included a scoping step; and eleven frameworks (MRU, CADTH, HITAP, ZonMW, HTAD, SBU, HIQA, HQO, KCE, ACE, MaHTAS, and AHRQ) used a quantitative method to score and rank potential topics before deliberation.
Checklist of six stages of topic selection
Notes: NM refers the stage was not mentioned in reports. NA refers the stage was not applicable in the framework.
ACE, Agency for Care Effectiveness; HITAP, Health Intervention and Technology Assessment Program; HTAD, Health Technology Assessment Department of Ministry of Health, Kazakhstan; MaHTAS, The Malaysian Health Technology Assessment Section; HIQA, Health Information and Quality Authority; HTW, Health Technology Wales; ICHTA, Israeli Center of HTA in health care; IQWiG, The Institute for Quality and Efficiency in Health Care; KCE, Belgian Health Care Knowledge Center; MRU, Mykolas Romeris University; NICE, National Institute for Health and Care Excellence; SBU, Swedish Agency for Health Technology Assessment and Assessment of Social Services; ZonMW, The Netherlands Organization for Health Research and Development; AHRQ, Agency for Healthcare Research and Quality; CADTH, Canadian Agency for Drugs and Technologies in Health; HQO, Health Quality Ontario; ICER, Institute for Clinical and Economic Review.
a Only definition of criteria for topic selection without description of the development process.
b Steps were described in sufficient details.
The six steps were not always applied sequentially in all frameworks. For example, IQWiG processed scoping of potential topics before short listing. KCE and HQO scored and ranked potential topics before short listing. Also, these steps were not always performed exclusively, that is, some frameworks included multiple steps that were performed together. For example, in four frameworks (MRU, HITAP, HTAD, and SBU), potential topics were scored and ranked during the deliberative meeting used for deciding the final topic list for HTA. The practices and approaches adopted in each stage across frameworks were also different, which are described in detail in the next subsections.
Specification of Criteria for Topic Selection
All frameworks described the criteria for topic selection. Out of the six frameworks with detailed descriptions of criteria selection, two (CADTH and MRU) identified criteria based on their own systematic reviews, whereas the other four (HITAP, ZonMW, MaHTAS, and HTAD) referred to published systematic reviews or related papers. Multiple criteria were used for topic selection in all frameworks. The three most common criteria used by agencies were burden of disease (n = 13), clinical/health impact (n = 12), and economic impact (n = 12).
Five frameworks developed weights for their chosen criteria for use in subsequent scoring and ranking. Delphi or expert consultation was used to select and weight criteria in four frameworks (MRU, ZonMW, HITAP, and HTAD), and the analytic hierarchy process was applied in CADTH.
Identification of Topics
In general, topics were (i) identified through horizon scanning by HTA agencies, (ii) requested directly by policy makers or an expert panel/committee, (iii) submitted by academic researchers, and (iv) nominated by the general public online. The sources of topic identification varied between agencies. For example, CADTH and NICE were identified from sources (i), (ii), and (iv) mentioned above, whereas HIQA only received topics requested by policy makers.
Only five published papers reported the number of potential candidate topics processed, with these showing a wide range in numbers across agencies: CADTH (n = 102 in 3 years), MRU (n = 26), HTAD (n = 41), NICE (n = 109 for the Interventional Procedures Programme), and ICHTA (n = over 100).
Shortlisting of Potential Topics
Nine agencies (CADTH, NICE, IQWiG, ZonMW, SBU, HTW, KCE, ACE, and MaHTAS) selected a short list of topics for further evaluation. Two types of short listing were identified.
The first type was to check the eligibility only. NICE, SBU, HTW, KCE, ACE, and MaHTAS applied explicit criteria of eligibility to filter potential topics. These criteria are different from those used for the prioritization of the short-listed topics. Duplication of topic and the remit of program were the most commonly used criteria.
The second type was to cut the number of potential topics using predefined criteria. In CADTH, topics are scored and ranked against duplication of effort, need, and stage of diffusion. ZonMW scored proposals against policy relevance and selected those with intermediate or high policy relevance for further scientific quality assessment.
In IQWiG, a selection committee nominated fifteen topics from a master list for further priority setting, and in SBU, a short list was produced by the SBU Board after internal and external scanning of fields of interest. However, little information on criteria used in short listing was reported by either IQWiG or SBU.
Scoping of Potential Topics
Different terms were used to describe this stage, including “scoping” (CADTH), “pre-scoping” (NICE), “topic processing” (IQWiG), “pilot study” (SBU), “exploratory study” (HTW), and “preliminary research” (MaHTAS).
In this stage, research questions were clarified and defined by contacting nominators or consulting experts if applicable. Evidence collection and processing was an important component at this stage, which was used for scoring (if applicable) and deliberation.
The amount and complexity of the work undertaken at this stage varied substantially between agencies. For example, HITAP and AHRQ mapped information in topic proposals against selection criteria then collected further data in order to capture important missed information, whereas IQWiG, CADTH, NICE, SBU, MaHTAS, and ACE simply referenced the conduct of bespoke literature reviews.
Scoring and Ranking of Potential Topics
In this stage, scoring was used to measure the performance or preference against selection criteria (to be combined with weights to build up the overall value).
Nine frameworks mentioned scoring and ranking potential topics against selection criteria (CADTH, HTAD, HQO, HITAP, KCE, ACE, ZonMW, and MaHTA). Referring to evidence against each criterion generated in scoping, proposed topics were scored and ranked either by researchers of agencies (CADTH, ZonMW, IHQA, and ACE) or an expert panel/committee (MRU, HITAP, HTAD, SBU, HQO, KCE, and MaHTAS). Direct rating techniques were used most commonly. CADTH, HTAD, HQO, HITAP, ZonMW, and KCE used a rating scale to score against each criterion, whereas ACE and MaHTA allocated points to each criterion. MRU provides little information about the technique of scoring.
The weighted sum model was widely used. CADTH, MRU, ZonMW, and MaHTAS applied different weighting across criteria to calculate the total score, whereas the others assumed equal weights (HITAP and HTAD) or did not mention weighting (HQO, KCE, and ACE). Ranking based on total scores was used to show the overall preference of potential topic. Only ZonMW used different weighting algorithms to explore the uncertainty of weighting on the total score. SBU and HIQA did not mention scoring, which implies that ranking was an output of a deliberative process.
Deliberation and Decision on Final Topics for HTA
In this step, an advisory body discussed the information collected on potential topics and advised or made a decision on final topics for HTA. Advisory bodies could include (i) an internal executive board of HTA agency, (ii) an internal working team of topic selection in HTA agency, or most commonly (iii) a special expert committee/panel set up for topic selection. The composition of the advisory bodies included a wide range of representatives, including policy makers, health professionals, academics, patients, and civil society. Only five frameworks (HITAP, IQWiG, ZonMw, KCE, and MaHTAS) had patient representatives, and no framework reported having representatives from the biotechnology industry or the general public.
Contents and details of evidence used during deliberation were also different among agencies. MRU mainly relied on expert opinions. HITAP, HTAD, ZonMW, ICHTA, and AHRQ mainly referred to information in proposals provided by nominators. IQWiG, CADTH, NICE, SBU, MaHTAS, and ACE referred to evidence based on original rapid reviews or additional pilot research. The ranks of potential topics were also provided or generated during deliberation in eleven agencies.
After the deliberation, a final list of topics for HTA were produced. However, the final result did not always agree with the ranks of potential topics. For example, in HITAP, nine out of thirteen potential topics were selected for HTA, eight of which were selected for best ranks. Interestingly, some agencies used this stage to triage topics to different forms of HTA. For example, MaHTAS highlighted that different types of HTA, including full HTA, mini-HTA, or evidence briefing, were recommended depending on the sufficiency of evidence. CADTH mentioned rejected topics being recycled or dealt with through less-extensive HTA methods following deliberation.
Discussion
Our study presents a comprehensive review of the topic selection process in HTA agencies around the world. Seventeen HTA agencies were identified with all of them using multiple criteria to evaluate topics. A framework for topic selection was outlined including six steps: specification of criteria for topic selection, topic identification, short listing of potential topics, scoping of potential topics, scoring and ranking of potential topics, and deliberation and decision on final topics for HTA. However, not all of these six steps were always included or described in detail in all frameworks. Our review, which provides detailed information on the practical process of topic selection in HTA agencies, is broader than similar previous studies that focused on criteria for topic selection (Reference Specchia, Favale, Di Nardo, Rotundo, Favaretti and Ricciardi1;Reference Dehnavieh12) or decision making on final topics (Reference Noorani, Husereau, Boudreau and Skidmore2).
Our review confirmed the diversity of practice observed by Noorani et al. (Reference Noorani, Husereau, Boudreau and Skidmore2). In particular, we found the number and type of criteria for topic selection varied between agencies, and the six steps were not always used nor applied in a consistent manner. Even for the same step, actors, stakeholders, and methods were not the same. This variability may be interpreted as reflecting differences in health priority, mandates and administration of programs, resources, and capacity in different contexts (Reference Henshall, Oortwijn, Stevens, Granados and Banta6). However, some of this variation could also indicate uncertainty relating to the best approach to take, which in turn raises the possibility that some of the adopted approaches are flawed. In general, we found that agencies in countries with well-established HTA systems (e.g., CADTH, SBU, and NICE) conducted topic selection processes more systematically.
Multiple criteria were used in all frameworks; however, the approaches used to select criteria were not always explicit. The most commonly used criteria identified in our review were burden of disease, economic impact, and clinical/health impact, which are similar to the findings in the two previous reviews (Reference Specchia, Favale, Di Nardo, Rotundo, Favaretti and Ricciardi1;Reference Dehnavieh12). However, the number and nature of criteria varied in different frameworks in our review, similar to previous studies (Reference Specchia, Favale, Di Nardo, Rotundo, Favaretti and Ricciardi1;Reference Dehnavieh12). These differences are likely to reflect policy and cultural differences; however, this cannot be ascertained as agencies seldom described how they selected their criteria for topic selection.
For a systematic approach to identifying topics, horizon scanning might be useful, which is aimed at identifying new and emerging health technologies that may have a relevant impact on the health system (Reference Migliore, Perrini, Jefferson and Cerbo39). However, only CADTH and NICE used formal horizon scanning systems to recognize possible technologies for assessment. (Reference Douw, Vondeling and Oortwijn40). Also, short listing of topics might be needed when the list of potential topics is long. In these circumstances, agencies can use short listing to reduce the workload of evidence collection and to obtain a manageable number of technologies for deliberation. Scoping of topics can also be used to assess the feasibility of a full HTA. Agencies such as NICE, HTW, and MaHTAS removed some technologies from the topic selection process based on stakeholder consultation and/or the nature of the evidence gathered during the scoping process.
In terms of the overall framework for topic selection, a common pattern was observed across agencies in our review, which was in contrast with the findings from previous studies. Garcia-Altes et al. (Reference Garcia-Altes, Ondategui-Parra and Neumann11), after comparison of four HTA agencies across Europe in 2004, found that organizations generally lacked explicit processes for prioritization for HTA. Noorani et al. (Reference Noorani, Husereau, Boudreau and Skidmore2) suggested there was not any particular pattern that emerged when comparing twelve HTA programs in 2007. However, we found that most agencies practiced MCDA in either a qualitative or quantitative way (Reference DiStefano and Krubiner41). This may reflect the developments in HTA across the world over the last decade (Reference Liu, Wu, Ahn, Kamae, Xie and Yang42–Reference Babigumira, Jenny, Bartlein, Stergachis and Garrison44), which could be supported by the change of the topic selection process in Canada (Reference Husereau, Boucher and Noorani13) and Netherlands (Reference Oortwijn, Vondeling and Bouter45).
Also, in contrast to the results produced by Noorani et al. (Reference Noorani, Husereau, Boudreau and Skidmore2), we found scoring and ranking were more commonly used to measure the relative priority of potential topics (eleven out of seventeen frameworks). These quantitative approaches tend to encourage participation and reflection as well as strengthen the level of transparency and accountability of the process (Reference Marsh, Goetghebeur, Thokala and Baltussen46), which may explain their widespread use among agencies. However, limitations were found in the methods used for scoring and ranking. First, equal weight or no weighting across criteria were noted in most agencies, which may ignore the different preferences between criteria. Second, the most prevalent model adopted for score aggregation is the additive model that is analytically simple but requires that the criteria do not overlap and are preferentially independent (Reference Marsh, Sculpher, Caro and Tervonen47). Third, uncertainty is not fully explored, yet issues such as choice of technique for scoring and aggregation, as well as heterogeneity in preferences, may contribute to the uncertainty of the results (Reference Thokala, Devlin, Marsh, Baltussen, Boysen and Kalo48;Reference Marsh, Lanitis, Neasham, Orfanos and Caro49).
We found no information in relation to monitoring and evaluation of the processes identified. Monitoring and evaluation is a critical link to control quality and develop a robust and consistent system of topic selection (Reference Henshall, Oortwijn, Stevens, Granados and Banta6). It seems likely that this occurs, albeit informally, in at least some of the agencies above as evidenced by the methods being updated. However, this information will only be captured through comprehensive engagement with each agency.
The involvement of patients or patient representatives in deliberation was not common among agencies. The involvement of patients may strengthen transparency, legitimacy, and fairness in decision making (Reference Facey50;Reference Wale, Scott, Bertelsen and Meade51), but the nature of patient participation was difficult to assess from the materials reviewed. Patients can contribute experiential knowledge about the technology under assessment, currently available technologies and other valuable information (Reference Facey, Boivin, Gracia, Hansen, Lo Scalzo and Mossman52), which may support deliberation. However, organizational and recruitment challenges maybe hinder HTA agencies from getting patient involvement in topic selection. Abelson et al. (Reference Abelson, Wagner, DeJean, Boesveld, Gauvin and Bean53) found key organizational challenges relating to the involvement of patients in HTA included time, financial resources, and expertise required to communicate with the patients. A difficulty in getting a “good representative” is another important recruitment obstacle (Reference Abelson, Wagner, DeJean, Boesveld, Gauvin and Bean53). Additionally, as topic selection is across many diseases/conditions, it is difficult to expect a single “patient representative” to provide insights on all topics.
There are limitations to our study. First, more databases could have been searched, and Web sites of HTA agencies that are not members of INAHTA could have been searched. However, our study aimed to provide an overview of the current practice of topic selection for HTA across agencies rather than reviewing the processes of all HTA agencies. Second, publication bias may exist in this study. The descriptions were restricted to the published articles and documents in English within the Web pages of the INAHTA members, which may not describe the current processes comprehensively. Ideally, our description of each agency's process would be validated by each agency and could have been supplemented by interviews or observation.
We believe the development of a good practice checklist for a topic selection process would help improve the quality of such processes because no guidelines exist specifically for topic selection as opposed to other aspects of HTA (Reference Kristensen, Husereau, Huić, Drummond and Bond3). The nature of topic selection for HTA seems ideally suited for MCDA, which includes defining the decision problem and identifying options, structuring criteria, collecting evidence and measuring performance, scoring and ranking (if applicable), then decision making (Reference Marsh, Goetghebeur, Thokala and Baltussen46;Reference Thokala, Devlin, Marsh, Baltussen, Boysen and Kalo48). Recent systematic reviews (Reference Marsh, Lanitis, Neasham, Orfanos and Caro49;Reference Muhlbacher and Kaczynski54) on the application of MCDA in health care show its potential to support transparent, consistency, and rigorous decision making in health care; therefore, it would appear worthwhile exploring the potential of using MCDA good practices to support the development of a checklist for topic selection.
Conclusions
Our study provides insights into current methods of topic selection and the variability of practice across HTA agencies. Several weaknesses were identified with the most important relating to the methods of scoring and ranking as well as lack of monitoring/evaluation of the process. The topic selection process has many parallels with MCDA and so a more rigorous adoption of MCDA methods could be useful for improving current topic selection processes. These findings may be useful for developing a consensus approach for good practice on topic selection.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462321001690.
Acknowledgment
Kun Zhao from the China National Health Development Research Center provides suggestions for the structure of the article.
Funding
This research received no specific grant from any funding agency, commercial, or not-for-profit sectors.
Conflict of Interest
There are no conflicts of interest.