Health technology assessment (HTA) is used to inform decisions relating to healthcare systems. The effectiveness of an HTA program will depend on its influence, the extent to which information provided in its publications has had an effect on decision makers, and in what ways. In this paper, HTA influence is considered to be any action or activity that can be credibly linked to information provided to a decision maker by an assessment (Reference Hailey, Macpherson, Aleman and Bakri1). HTA influence is used rather than “HTA impact,” as representing a more realistic indication of the place of HTA in decision making.
Information on the influence of HTA reports is a guide to the effectiveness of an assessment program. Such information is useful in reporting to funders of HTA programs, in quality assurance processes, and in contributing to global indications of HTA achievements. In principle, there will be interest in the influence of HTA on policy and administrative decisions, subsequent administrative action, delivery of health care, and on health status (Reference Hailey, Macpherson, Aleman and Bakri1). Much of the focus on HTA influence has been on the first of these. Subsequent administrative action is dependent on the availability of effective machinery and the willingness of the decision maker to make use of it. Influence of an HTA report on subsequent action and outcomes within a healthcare system depends on the actions of many individuals and organizations (Reference Hailey, Macpherson, Aleman and Bakri1).
There is still relatively little information available on the influence of HTA on healthcare decisions and their outcomes. Also, there is limited detail available on methods used to assess HTA influence and the experience of HTA programs in applying such approaches. A review by Gerhardus and Dintsios concluded there was little experience with study designs or methods that allow a valid assessment of the impact of HTA reports on the decision-making process. Only limited conclusions related to the impact of HTA reports could be drawn (Reference Gerhardus and Dintsios2).
A review of policies and processes for the introduction of new interventional procedures into clinical practice identified seven studies that described outcomes of policies (Reference Thavaneswaran, Spigelman, Baggoley, O'Connell and Maddern3). The results showed that while the safety, efficacy, and clinical and cost-effectiveness of new health technologies are important considerations in the decision-making process, several other factors also play an important role. Decisions were never based solely on the findings of HTAs. Niessen et al. (Reference Niessen, Bridges and Lau4) reported that thirty studies, including some on HTAs, found that use of economic evidence had a “substantial” impact on healthcare policy making; twenty-seven studies emphasized at least one other criterion. A further eleven studies found only a limited impact and two studies showed no impact.
The International Network of Agencies for Health Technology Assessment (INAHTA) had obtained information on HTA influence from its members but had not reviewed the available literature. The network decided in 2012 that a working group would undertake a systematic review of reports on HTA influence and its measurement. Five groups of either two or three reviewers were formed by working group members to share the tasks of abstract selection and data extraction. The Swedish Council on Technology Assessment in Health Care (SBU) undertook the literature review and provided project support.
A report on the review is available on the INAHTA Web site (5). For this article on the review, the literature search was updated and additional publications included. Further details were provided on the derivation of the categories used in reporting the review findings, and an additional presentation linking details of technologies that were assessed with indicators of HTA influence. Material in the tables was updated to reflect data from the additional studies and include percentages.
The objectives of the systematic review were to obtain information on the influence of HTAs on healthcare decisions and their outcomes, and on the methods used to measure such influence.
METHODS
A protocol for the review was prepared by members of the working group. Broad inclusion criteria were specified covering studies reporting consideration of HTA findings, and changes in policy, health technology use, health outcomes, or increased level of research.
Published literature was identified using PubMed, Embase, Cinahl, Cochrane Library, PsychInfo, NHSeed, HTA database, DARE, NHS Evidence, and the Swedish HTA database. The searches were supplemented by hand searching the bibliographies of selected papers and through contacts with HTA and other agencies. Publication dates were 2000 to August 2014, subsequently extended to November 2015. There were no language restrictions. Search terms included Technology Assessment, Biomedical, HTA, systematic review, evidence-informed, impact, influence, information dissemination, implement, policy making, health policy, and decision making. Further details of the literature search strategy are available in the INAHTA report on the project (5).
Inclusion criteria were studies that reliably reported consideration by decision makers of HTA findings and/or recommendations; comparative studies that included relevant measures related to use of a health technology before and after dissemination of an HTA; and studies that reported changes in one or more features that could be credibly linked to information provided by an HTA. Those features were policy related to a health technology, use of a health technology in a healthcare system, relevant health outcomes associated with use of a health technology, and an increased level of research or initiation of research. Expert opinion, correspondence, commentaries, and duplicate publications on the same study were excluded.
A data extraction form was developed, which included lists on approaches to assessing influence, types of decision, indications of HTA influence and opinions on influence used in previous INAHTA publications (Reference Hailey, Macpherson, Aleman and Bakri1;6). The form also included five indicators of study quality that were specified in the protocol. These had some relevance to quality but were also related to the scope of a study (Supplementary List 1). Quality ratings were given by the number of indicators that applied to each study, with scores from 1 to 5.
Each reviewer group was allocated a list of identified publications for initial screening using titles, abstracts and keywords. Any citations considered relevant or for which there was uncertainty were retained at that stage and the full papers obtained. The identified publications were considered independently by the reviewer groups and selected if they met the inclusion criteria. Differences between individuals within the reviewer groups were discussed and resolved by consensus.
Information extracted from the selected publications by the reviewer groups included the study setting, health technologies that were assessed, types of decision informed by the assessment, the approach used to assess HTA influence, main indications of influence, measures and/or opinion on influence, and non-HTA influences on outcomes. Any disagreements were resolved through discussion. For publications covering many HTA reports, the technologies were listed but other elements in the data extraction were based on the summary information that was provided, rather than considering each recommendation and its influence individually.
In some cases, the authors’ opinion on level of HTA influence was reported, or was apparent from details presented in the reviewed publication. For other publications, a judgement on the level of influence was made by the reviewers. Level of influence was recorded on a four point scale used in previous INAHTA projects (major influence on decisions, some input to decisions, some consideration of the assessment, minimal influence) (6).
RESULTS
After removal of duplicates, 4,767 publications were identified by the literature search. An overview of publication selection is shown in Supplementary Figure 1. Adjustments to initial selections were made through exclusion of earlier publications from series of reports on the same topic, papers that were not related to influence of HTA, and publications where there was insufficient information to provide a clear indication of influence. Fifty-five publications covering fifty-one studies were selected for review. Reports on measurement of HTA influence were obtained from nineteen countries: Argentina, Australia, Austria, Belgium, Canada, France, Finland, Germany, Italy, Malaysia, the Netherlands, the PRC, Poland, Spain, Sweden, Switzerland, Thailand, the United Kingdom, and the United States. Three publications provided information on more than one country—a report on EU countries, a survey with details from Australia, Brazil, Canada, Spain, and the United States, and a survey of Latin American and Caribbean countries. Study settings were national (Reference Carlsson24), regional (Reference Ballini, Minozzi, Negro, Pirini and Grilli12), both national and regional (Reference Thavaneswaran, Spigelman, Baggoley, O'Connell and Maddern3), multinational (Reference Thavaneswaran, Spigelman, Baggoley, O'Connell and Maddern3), and hospitals (Reference Schluessmann, Diel and Aghayev9).
Approaches taken to appraisal of HTA influence are shown in Table 1. Several studies used more than one approach. The most common was review of policy, and of acceptance of HTA recommendations (in 51 percent of studies). Eighteen studies (36 percent) used surveys or interviews with decision makers, thirteen (26 percent) reviewed or analyzed administrative data, and six (12 percent) considered the influence of primary studies.
Approaches Taken in Assessing HTA Influence
Note. The numbers following each item denote the references
Types of decisions informed by the HTAs are shown in Table 2. Decisions related to routine practice (in 67 percent of studies), coverage (63 percent), program operation (37 percent) and capital funding (35 percent) were the most common categories. Table 3 shows the indications of HTA influence that were noted during data extraction. Several studies showed more than one influence on decisions. Influence on decisions involving resource allocation was the most frequent indication (in 59 percent of studies). There were also several indications related to effects on practice (31 percent), and incorporation of HTA details into reference material such as guidelines and program management manuals (18 percent).
Types of Decisions Informed by HTA
a The numbers following each item denote the references
b Other decisions: Equipment sales (Reference Britton and Jonsson19), Legislation to regulate program (Reference Autti-Ramo and Makela23), Strategy planning process (Reference Bowen34, 35).
Indications of HTA Influence
aThe numbers following each item denote the references
Opinion from twenty-one (41 percent) of the studies was that HTA had had a major influence on decisions. In thirteen studies (25 percent), HTA had provided some input to decisions; with six (12 percent), there was some consideration of the assessment; and in three studies, there was minimal influence. In the other eight studies, details given indicated that HTA influence on decisions had varied for different technologies (major influence eighty, 54 percent), input to decisions twenty-four (16 percent), consideration of the assessment thirty-six (24 percent), minimal nine (6 percent). Some of the “minimal” ratings were associated with the early stages of HTA programs. Quality ratings were high (5 or 4) for twenty-seven studies (53 percent). Seventeen (33 percent) had a rating of 3, and seven had a rating of 2.
Brief details of material in the reviewed publications are shown in Table 4. Assessments on medical devices and surgical procedures informed decisions on coverage and conditions of use for technologies, with consequences for routine practice. The studies on screening technology point to the importance of HTA in providing input to government processes for the development and implementation of national screening programs. HTAs had a major influence on coverage decisions for pharmaceuticals; there were limitations in the influence of negative recommendations on the use of some drugs. The studies also indicated the place and value of rapid assessments and the success of hospital HTA programs in influencing local policy and administrative decisions. Further information is presented in Supplementary Tables 1 and 2.
Details of Technologies and HTA Influence
DISCUSSION
Both positive and negative HTA findings on health technologies were influential. Many related to relatively short-term targets (policy and administrative decisions). Some also covered subsequent administrative action and program planning issues. Gerhardus and Dintsios (Reference Gerhardus and Dintsios2) refer to use of interviews with decision makers, document analysis, surveys, and use of administrative data as methods in the evaluation of HTA influence. A similar mix of approaches was used by the studies included in this review, and also appraisal of the effects of primary studies. Approaches using review of decisions seemed useful. Some HTA programs had close contact with decision makers, giving opportunities for realistic appraisals.
The reviewed studies provide good examples of the place of HTA in healthcare decision making. Information on the influence of assessments can have a role in making a contribution to a broader, global perspective of HTA's achievements and usefulness (Reference Hailey, Macpherson, Aleman and Bakri1). There were clear indications of the benefits to both decision makers and HTA researchers from maintaining regular, appropriate contact at all stages of the HTA process (Supplementary Tables 1 and 2). For decision makers, there are examples of the sorts of decisions and programs that HTA informs. There are some insights into the decision-making process, and indications of savings, efficiencies, and practice changes linked to HTA findings. There are lessons to be learned from some limitations, both in consideration of HTA advice and in the implementation of decisions.
There were also examples that might be followed more widely on approaches to measuring influences. Valuable information on some longer term influences was obtained through monitoring administrative data and through survey approaches. Primary studies can form a useful part of the HTA process, as shown through indications of changes to practice patterns, relating to the delivery of health care (Reference Hailey, Macpherson, Aleman and Bakri1).
There was little indication of influence of HTA on the health status of patients, although this was captured to some extent by a review of SBU assessments. Two studies reported use of HTA linked to coverage with evidence development (CED) for consideration of policy decisions at regional or national level. A CED approach was also used with an HTA program that evaluated surgical technologies. Estimates of savings achieved through use of HTA were found in studies covering hospitals in Montréal and Buenos Aires, and in a study of HTA in Austria which noted contributions to disinvestment decisions.
The literature covering the matters addressed by this review is still quite limited. Few recent studies have considered the influence of HTA in any detail, and there is little on the longer term effects on clinical practice and health outcomes.
There is a progression of possible influence, from the decision-maker level with increased knowledge and awareness, to changes in policy and healthcare delivery, up to changes in patient outcomes. Changes to health care and improved health are dependent on the actions of many individuals and organizations. Measuring change across all stages of influence should be considered, as the contribution of the HTA to changes in patient outcomes is not going to occur if these earlier stages of influence have not first been realized.
Detailed appraisal of HTA influence, especially in the longer term, can become a significant research project. Such projects may be resource intensive, and be undertaken only occasionally. Availability of data may be an issue; access to records may need to be negotiated, or surveys of users of health technologies put in place. However, shorter-term evaluation of influence on decision makers may need only modest resources. Essentially it is a question of incorporating approaches to obtaining indications of influence into the routine management of an HTA program (Reference Hailey, Macpherson, Aleman and Bakri1).
Both assessors and decision makers have important roles and responsibilities in the HTA process. Future studies on HTA influence will be facilitated by good and continuing interaction between these parties. The HTA agency should aim to generate some reaction from decision makers to the material and advice that have been provided. There should be contact with the decision makers early in the assessment process and while the project is in progress. It is necessary to maintain a dialogue (Reference Hailey, Babidge, Cameron and Davignon62).
A monitoring system developed in Québec used telephone interviews with requesters and users to obtain reactions of decision makers to HTA reports and intended action on the technologies that had been assessed (Reference Hailey, Macpherson, Aleman and Bakri1). Perceived relevance of recommendations and intention to adopt recommendations can be used as a proxy for influence (Reference Hailey, Babidge, Cameron and Davignon62). The INAHTA impact framework provides a basic approach to collecting information on HTA influence (6). Approaches based in part on application of the framework have been able to capture details of at least short-term influence (Reference Hailey41). Resources for more detailed studies on longer term influence of HTA should preferably be a matter for negotiation between the assessment agency, its governance, and decision makers at an early stage.
Further studies would also be facilitated by encouragement for agencies and their clients to give details of both successes and failures of their HTAs in influencing decisions. “Failures” as well as successes need to be considered and acted on if influence measurement is to be useful for HTA program management, and as feedback to clients. Open provision and wide distribution by HTA programs of information on their influence has been recommended (Reference Hailey, Macpherson, Aleman and Bakri1).
This systematic review had several limitations. The estimated extent of influence is based on the authors’ findings and opinions, or on our to some extent subjective judgements, and should be critically reviewed. We may expect some degree of publication bias toward positive stories, but it is difficult to assess the extent of this. There were restrictions on the dates and scope of the literature search and on the approach taken to assessment of study quality, to take account of time and resources available to reviewers. On the other hand, the quality appraisal approach that was used touched on some things that are often not considered in a formal manner. The information presented is largely confined to summaries of details in the selected publications. There was not sufficient time to scan and follow up reference lists of included publications or to carry out citation searching. For example, an overview by Raftery and Powell of the UK Health Technology Assessment programme gives some indications of HTA influence that could be followed up (Reference Raftery and Powell63). Nor was it possible to systematically search the gray literature for evaluations of HTA programs and their influence.
POLICY IMPLICATIONS
While there is variation in the influence that HTA reports or programs might have, most of the examples considered here informed decisions on health technologies. All but three of the fifty-one studies identified successful HTA influence. HTA has been an important input to policy formulation and implementation in many settings. The publications covered by this review have much useful information related to HTA influence including approaches that might be used more often. Some of them had been produced as components of HTA program management (Reference Hailey, Macpherson, Aleman and Bakri1).
The limited number of studies following change in clinical practice and health outcomes indicates that these areas need much more attention in the future. Quality registers and clinical databases are growing rapidly around the world and they could be very useful tools for analyzing the influence of HTAs.
SUPPLEMENTARY MATERIAL
Supplementary List 1: https://doi.org/10.1017/S0266462316000611
Supplementary Figure 1: https://doi.org/10.1017/S0266462316000611
Supplementary Table 1: https://doi.org/10.1017/S0266462316000611
Supplementary Table 2: https://doi.org/10.1017/S0266462316000611
CONFLICTS OF INTEREST
David Hailey: Dr. Hailey has nothing to disclose. Sophie Werkö: Dr. Söderholm Werkö has nothing to disclose. Måns Rosén: Dr. Rosén has nothing to disclose. Karen Macpherson: Dr. Macpherson has nothing to disclose. Susan Myles: Dr. Myles has nothing to disclose. Verónica Gallegos Rivero: Dr. Gallegos-Rivero has nothing to disclose. Cecilia Hipólito-Olivares: Dr. Hipólito-Olivares has nothing to disclose. Sinikka Sihvo: Dr. Sihvo has nothing to disclose. Jasmine Pwu: Dr. Pwu has nothing to disclose. Wen-Wen Yang: Wen-Wen Yang has nothing to disclose. Yong-Chen Chen: Dr. Chen has nothing to disclose. Ana Perez Galán: Dr. Perez Galán has nothing to disclose. Alicia Aleman: Dr. Aleman has nothing to disclose. Elena Villamil: Dr. Villamil has nothing to disclose.
