In Austria, research in health technology assessment (HTA) has been conducted since the 1990s. It was initiated by the Institute of Technology Assessment (ITA) at the Austrian Academy of Sciences. In 2006, the Ludwig Boltzmann Institute for HTA (LBI-HTA)—a nonuniversity public research institute—was founded. It is primarily funded by public research funds and, to a smaller extent (40 percent), by so-called “partners” representing some of the core payers of the Austrian healthcare system (e.g., social insurances, Ministry of Health, hospital associations) and several academic institutions. The annual research program includes topics that are defined by the “partners” and those by the LBI-HTA itself, such as methodological research. In addition to HTA reports, the LBI-HTA program includes a monthly newsletter, a Web page, a series of lectures and conferences.
The LBI-HTA aims at producing independent scientific support for decision makers which should ultimately translate into an improved and efficient healthcare system and in better population health. However, neither the use of HTA reports in decision making nor the implementation of HTA recommendations is mandatory in Austria.
Because it is an inherent characteristic of HTA that its products are used in decision making, HTA itself needs to undergo an evaluation. Thus, a project was initiated to evaluate the impact of HTA research on the Austrian healthcare system. While a literature overview of existing methods of HTA impact evaluation and parts of the empirical results have been published elsewhere (1;Reference Zechmeister and Schumacher2), this study aims at presenting the overall methodological approach of the empirical study on impact and an overview of the results.
METHODS
Conceptual Framework
The framework according to which we structured the empirical analysis has been identified in the literature review (1;Reference Schumacher and Zechmeister3). The concept was adapted for the HTA impact evaluation in the Austrian context (Reference Schumacher and Zechmeister1;Reference Schumacher and Zechmeister3–Reference Gerhardus, Dorendorf, Rottingen, Velasco Garrido, Kristensen and Palmhoj Nielsen5). It is based on the theory that a linear influence of research into policy decision is scarce (Reference Gerhardus, Dorendorf, Rottingen, Velasco Garrido, Kristensen and Palmhoj Nielsen5;Reference Weiss6). We rather suggest a complex research reality where policy decisions are not directly affected by research results, but research concepts and theoretical perspectives influence policy making in a more indirect way according to the enlightenment model (Weiss, Reference Weiss1979) (Reference Weiss6). Thus, the framework is based on a multi-dimensional concept of impact and considers seven different categories of impact, as well as different target groups or system levels where effects may be identified (Table 1). Table 1 shows the impact categories we addressed and how we operationalized them.
Conceptual Framework
As indicators for “awareness,” we chose the frequency of downloads, knowledge about HTA reports amongst the target audience and the representation of HTA in the media. “Acceptance” is indicated if the assessments are perceived as helpful within the target groups, whereas the indicator for “(policy) process” was whether reports are factually used in the decision-making processes. If decisions are legitimized by HTA reports or HTA results are referred to by decision makers, impact was assigned to the category “(policy) decision.” Changes in clinical or refunding practice were defined as indicators for the category “practice.” The category “final outcomes” was limited to “economic impact,” while further issues of “final outcomes,” such as health outcomes, were not addressed due to methodological limitations. In terms of economic impact, the indicators were whether the reports resulted in rationalization (in cases of over-used technologies) and/or in explicit re-distribution of resources into evidence-based technologies or in withdrawing resources from technologies that lack evidence of effectiveness, safety or cost-effectiveness. Finally, impact in the category “enlightenment” was indicated if any evidence of an “HTA culture” could be identified in the decision-making processes (e.g., increasing transparency by introducing mandatory “conflict of interest statements”), in the research communities (e.g., improvements in study designs, increase in HTA research) and in the media (e.g., more objective media reports on health technologies).
The framework draws on the assumption that impact may happen at different levels of the healthcare system. We structured the healthcare system and the HTA target groups within it according to the micro, meso, and macro level. The micro level represents natural, single persons, such as researchers, patients or medical practitioners. At the meso level we allocated institutions, such as hospital associations, social insurances and professional associations. The macro level refers to political and governmental institutions at the federal and regional levels, and additionally includes the media and industry.
Empirical Analysis
In the qualitative empirical analysis we applied several empirical research methods to address impact from multiple perspectives and to thus present a rich picture of several dimensions of impact. While each single empirical method (e.g., interviews) would be insufficient to draw conclusions from, the aim was to construct a convergent “impact portrait” by contrasting different results, being aware, however, that the method does not allow for establishing causal inferences (Reference Schumacher and Zechmeister7). Table 2 demonstrates what type of impact categories and indicators were addressed by the different empirical methods. While we were able to address all impact categories in the framework, in terms of target groups we excluded patients/citizens and industry from the analysis, as they are beyond the LBI-HTA primary target groups.
Overview of Methods and Impact Categories/Indicators They Address
Interviews
We interviewed fifteen key stakeholders from the predefined system levels (Table 1), excluding representatives of patients/citizens and industry. Interview partners were selected according to the criterion of being a member of the LBI-HTA board or being engaged with HTA at the administrative level of the healthcare system. All selected interviewees agreed to the interview. We used a semi-structured questionnaire where (a) use and (b) perceived effects of HTA research products were addressed. The impact categories and indicators that have been addressed by the interviews are presented in Table 2.
At the micro level we interviewed two physicians, as well as a journalist, and evaluated their knowledge and use of HTA in their roles as individual professionals. At the meso level we interviewed seven representatives, all of whom were either working in hospital management, in an insurance company or in the audit court of an Austrian regional government. Finally, at the macro level, we selected five persons employed in decision-making bodies of the Austrian Ministry of Health.
We conducted a qualitative analysis of the transcribed interviews. According to the standard methods of qualitative interview analysis (Reference Gläser and Laudel8;Reference Mayring9), the text was structured into themes regarding the impact categories (e.g., knowledge about HTA products, changes in practice).
We structured similar text passages into paraphrases and arranged them with regard to positive/negative statements and the system level of the interviewee, respectively.
Download Analysis
The LBI-HTA provides free downloads of all reports and assessments that have been published online. In this context, we assessed the access rate of PDF downloads from 2006 to 2010 (excluding those from LBI employees). The analysis was carried out both in terms of report characteristics (subject matter, language) and in terms of the time lapse. We generated access statistics of every report published online. Although we acknowledge that the analysis is limited by the lack of a valid comparator, we expected to learn about trends in the development of awareness over time.
Questionnaire
We interviewed the LBI-HTA researchers using a questionnaire aimed at assessing the impact of their research on the scientific community both directly and by other means, such as capacity building. The survey covered three major communication channels: dissemination of research results (e.g., by means of publications, conference contributions), effects on future research and dissemination of HTA expertise (teaching, active memberships in committees), raising public awareness and change in the decision-making culture (e.g., “conflict of interest” statements).
Retrospective Routine Data Analysis
Administrative data were analyzed to gain information on changes in volumes and/or expenditure before and after publication of an HTA report. Administrative data included longitudinal information on quantities of technologies or services supplied and the tariffs that have been paid for these technologies. They come from different sources (hospital associations, single hospital units, Ministry of Health, social insurance bodies) and cover different time periods depending on the technology in question. Furthermore, these data showed whether new technologies were included into the publicly funded hospital or social insurance healthcare basket after our report had been published and upon what conditions inclusion was based. Details about the method and results of this analysis are presented in a separate study (Reference Zechmeister and Schumacher2).
Media Analysis
The media analysis first aimed at evaluating the LBI-HTA institutes’ press review in terms of quantitative media coverage, content, and type of media. Second, we evaluated how the mass media report about medical knowledge. In this context, we analyzed two Austrian national quality newspapers (Der Standard and Die Presse), covering a limited period from 01/2001 to 06/2010, for an existing “HTA culture” and for changes over time by applying the method of discourse analysis. The “health” and “science” sections of the two newspapers’ archives were searched. Four “key debates” were identified (drugs, infectious diseases, cancer and mental illness) and press articles relating to these topics were searched for in all sections of the two newspapers. In the articles selected, themes related to the “HTA culture” (see Table 2) were identified and described.
RESULTS
Table 3 summarizes the results of the different impact categories.
Summary of Impact Ordered by Impact Category and System Level
aHTA products: e.g., reports, newsletter, Web page.
-, low impact; +, medium/high impact; 0, no impact; ?, impact unknown; na, not analyzed
Awareness and Acceptance
Download analysis shows that the download per assessment increased steadily over time (Figure 1), yet single reports were found to be less frequently downloaded the longer they were online (after 7 months and 1 year, download frequencies are approximately one-third and a quarter of the first month, respectively). Whereas the number of reports online rose, the number of downloads per assessment increased disproportionally. “Top sellers” included reports covering subjects of high media interest, such as “HPV vaccination” or the so-called “swine flu.”
Average download per assessment per month.
Additionally, the media analysis shows that the number of press articles about the institute or its research results has risen continuously since 2006 (seven articles in 2006, in contrast to fifty-six in 2008). Most of these articles were found in print media (89 percent of all media types). Approximately half of the media reports were linked to specific HTA project reports. The other half of the articles dealt with HTA topics in general or the institute in particular.
Regarding the content of the articles, the institute is often portrayed as being a vehicle for cost containment in the healthcare system. Other characteristics attributed to the institute include being “investigative,” providing (comprehensive) facts and being the enemy of the pharmaceutical industry. Interestingly, in articles covering individual LBI-HTA research projects, results were often taken verbatim from the research report. In some newspaper interviews, decision makers legitimized their decision by referring to an individual assessment.
The interviewees, particularly those at the meso and macro levels, were aware of the HTA methodology and stressed its advantages, as, for example, the independency of HTA research, and its possibility to increase transparency and evidenced based decision making in health care. A member of the hospital management expressed it this way: “The aim is to make no more decisions about services and structures without using HTA research” (Interview 4). Nevertheless, interviewees remarked that HTA research in Austria has not arrived at all levels of decision making yet.
(Policy) Process, Decisions, and Changes of Clinical and Reimbursement Practice
At the micro level, general practitioners rarely use HTA assessments for individual treatment decisions: “Concerning my position as a doctor, it (HTA) is not relevant” (Interview 7). However, they use them for drafting reports or research protocols, as well as for gathering information in their roles as members of advisory committees.
In hospital associations, that is, at the meso level, HTA has increasingly been used for investment/reimbursement decisions and budget allocation, as well as for the preparation of negotiations: “HTA is helpful in deciding on new services or in long-term investment planning” (Interview 4).
In addition, it was indicated that the implementation of HTA recommendations led to improved organizational structures, such as more rigorous documentation of the administration of certain drugs in the case of “erythropoietin.” Due to new evidence provided by an HTA report, one hospital changed its existing treatment guidelines, which was also accepted and implemented by other hospitals: “The same new guidelines were implemented by the medical director in (city x)” (Interview 9).
Furthermore, interviews and administrative data indicated some changes in reimbursement and in clinical practice. For example, due to a lack of evidence concerning the effectiveness of certain cardiologic rehabilitation services, the social insurance decided to introduce conditional coverage for those services. Moreover, in five of six cases where the report identified an oversupply of technologies, a decrease in usage was identified. According to the interviews, this was primarily associated with the HTA report.
However, the inability to implement HTA recommendations was mentioned as well. This was mainly attributed to various pressure groups, such as the pharmaceutical industry and professionals’ associations. Further implementation barriers identified by the interviewees comprised the lack of acceptance of HTA research, as well as the lack of communication and liability. With the exception of the rapid technology assessment program for single hospital procedures, selective use of HTA reports, rather than standardized inclusion of HTA into the processes, was identified.
Economic Impact
Comprehensive results of economic dimensions are shown in a separate study (Reference Zechmeister and Schumacher2). In summary, the interview results as well as the administrative data showed that using HTA in decision making resulted in a variety of economic consequences. First, several technologies, which had been identified in the assessment as showing patterns of over-usage, were used more restrictively after the report had been published, leading to a decrease in expenditure: “One million Euros were, I would not say, saved but invested more reasonably” (Interview 4). This was particularly the case for hospital technologies. While overall quantification in monetary terms was not possible, interview and administrative data demonstrated that the expenditure decrease was of significant size and accounted for at least several million Euros for single hospital associations: “HTA has contributed to cost savings of a 7-digit magnitude simply by restricting indications for drug use” (Interview 4). Additionally, HTA has increasingly been used in prospective planning, hence before a reimbursement decision. This led to a more restrictive inclusion of new technologies into publicly funded health service baskets and may indicate a more evidence-based redistribution of resources.
Enlightenment
The interviewed journalist noticed at least selective changes in terms of discourse and reporting amongst medical journalists. In addition, a higher degree of “transparency for the citizen” (Interview 8), as well as more widespread use of HTA research in health policy planning were claimed, indicating some awareness of transparency issues and “HTA culture.” Furthermore, interviews demonstrated that—although still selectively—there is increasing use of evidence-based knowledge in the decision-making process, particularly at the hospital level. Moreover, conflict of interest declarations have been introduced in various advisory bodies at the Ministry of Health level, and 60 percent of the institute's researchers have been appointed to advisory boards by 2010, compared with 1 percent in 2006.
The evaluation of the researchers’ questionnaire showed that results are transported to the HTA community as well as to individuals (citizens, patients) through a variety of communication tools (e.g., journals, academic and public conferences) and that these have increased over time. Furthermore, regular teaching activities at universities have risen from three in 2006 to eight in 2009.
In contrast, the media analysis shows that the reporting on the benefits and harms of technologies, particularly in reports on cancer prevention and treatment, is characterized by an unbalanced presentation of technologies. Press articles are dominated by announcements of success and lack objective presentation of risks and adverse events. Only in single cases, elements of an “HTA culture” can be recognized in media reports, such as statements of conflict of interests or the reporting of different effect sizes to describe the effectiveness of a drug (e.g., absolute risk reduction versus relative risk reduction).
DISCUSSION
We identified evidence of impact within all of the predefined categories, but the extent differs depending on the category. Clearest evidence was available for the impact category “awareness,” while references regarding “acceptance” were rarely mentioned. Interestingly, the content of the press articles reflects images of the institute that are only partly consistent with the institute's own mission statement. The LBI-HTA seems to be more associated with being a vehicle for simple cost containment and rationing, rather than with supporting re-distribution of resources into evidence-based technologies or a more efficient use of resources (rationalization). This can be seen as an explanation for a lack of “acceptance”.
Furthermore, the results indicated an impact on decision-making processes and clinical and/or refunding practice, and they suggested considerable economic consequences from using HTA reports in decision making. Reports recommending a restricted use of technologies resulted in decreasing usage in the majority of cases analyzed. This was particularly true for hospital technologies. However, it became clear that changes in medical practice were not conclusively influenced by the physicians’ acceptance of an HTA recommendation, but were rather due to administrative measures.
In addition, both an increasing demand for HTA knowledge, as well as a rising HTA capacity building at various levels (e.g., through increasing teaching activities), became visible. Not least, from the enlightenment point of view, the appointment of HTA researchers to advisory boards and the successful implementation of conflict of interest declarations on those boards denote an increased rationality in decision-making processes. However, while there is evidence that the “HTA culture” has entered the decision-making processes in places, it seems to only be slowly finding its way into the media, and HTA as a tool for more rationality and transparency is still completely missing in several decision processes within the healthcare system.
Regarding the system levels and the related target groups, impact was mostly indicated at the meso and macro levels and less at the micro level, which is consistent with the primary target groups of the LBI-HTA.
Our approach of illustrating an “impact portrait” of an HTA institute's overall research program stands in contrast to recent impact research literature (Reference Schumacher and Zechmeister1;Reference Schumacher and Zechmeister3). So far, the methodical approach was either to compare the impact of HTA between countries (Reference Hailey10–Reference White12) or to evaluate the impact of a specific HTA report on a predefined system level (e.g., its influence on awareness amongst dentists on tobacco cessation (Reference Axelsson, Helgason and Lund13), changes in drug reimbursement) (Reference Dietrich14–Reference Sepulveda, Horowicz-Mehler and Doyle16).
Moreover, most of the studies identified were focused on a specific target group, like general practitioners (Reference Axelsson, Helgason and Lund13;Reference Alanen, Johannala-Kemppainen and Ijäs17–Reference Gagnon, Sanchez and Pons19), a single hospital (Reference Bodeau-Livinec, Simon and Montagnier-Petrissans20;Reference McGregor and Brophy21) or decision-making bodies (Reference Dietrich14–Reference Sepulveda, Horowicz-Mehler and Doyle16;Reference Blancquaert22–Reference Raftery, Hanney and Green24). The applied research method was, for the most part, questionnaires; a combination of two or more methods was rare.
The strength of our approach is that it takes into account that HTA research may affect all system levels in a multi-dimensional manner, rather than in a linear one. Thus, as some examples have shown, even if the reports are known and considered for decision making, the decision may not follow the evidence. Nevertheless, the awareness of HTA research, as well as the implementation into practice and political decision-making processes, is rising.
Recent HTA research results comparable to our findings showed that HTA reports do not necessarily result in policy decisions; however, an impact on agenda setting and awareness was evident in some cases (Reference Blancquaert22). While HTA research is widely known and accepted by decision makers, clinicians argue that HTA results would interfere with the doctor–patient relationship (Reference Rubinstein, Belizán and Discacciati25). Rapid assessments assigned by decision-making bodies have definitely had a high impact and were at least considered in decision making (Reference Hailey10).
Our aim was to construct a convergent “impact portrait” by contrasting different results. Therefore, we applied a mixture of research methods, both qualitative and quantitative in nature. Due to this complexity, we were not able to exploit each research method in depth. For example, we only interviewed fifteen individuals and were, therefore, not able to cover all target groups. Owing to the fact that being engaged with HTA was a criterion for inclusion, selection bias may have occurred.
Further limitations are a lack of benchmarks (e.g., download statistics from other institutions). Hence, the results are valid for an impact portrayal of HTA, but not for establishing causality. We tried to overcome this limitation by using a combination of different methods to obtain a variety of perspectives on impact from various empirical sources.
Some of the changes described in the context of “HTA culture,” such as the increasing demand for conflict of interest statements or increased transparency, may also have occurred without HTA research and some decisions need not necessarily be based on a specific HTA report. While this issue needs to be addressed in depth in further research, the empirical results from the different sources have indicated that HTA research(ers) have at least played some—and in specific cases even a prominent—role in triggering changes.
As a final limitation, the analysis was conducted by LBI researchers, as compared with independent external researchers. Hence, bias may be introduced in both directions—either over- or underreporting the impact. We tried to minimize bias by conducting the analysis with two researchers and double-checking the results for validity und consistence. Furthermore, we aimed at reporting the methods and the interpretation of results in a transparent manner. Finally, we performed both an internal and external review (including one non-HTA researcher).
CONCLUSION
The evaluation demonstrated an impact of HTA on all categories analyzed. Impact, however, varies and depends on the system level and its target groups. According to the results, HTA reports are primarily used by hospital management, social insurances, as well as the Austrian Ministry of Health. This is consistent with the primary target groups addressed by the HTA research of the LBI-HTA.
Not only has awareness of HTA increased, but the use of HTA recommendations in reimbursement negotiations or for decision making has also risen. This has resulted in rationalization and more restricted reimbursement of new technologies that enter the market, leading to reduced expenditure, on the one hand, and structural changes toward evidence-based reorganization, on the other.
Nevertheless, there is still potential to increase the impact within all categories. Moreover, some HTA reports or recommendations are broadly unknown or are, for some (uninfluenceable) reasons, not used for decision making.
To further increase the impact of HTA, research needs to be relevant for decision makers in terms of timeliness and topic addressed, which requires flexibility in the use of resources and, not least, the inclusion of HTA in decision-making processes needs to move from a voluntary basis to a mandatory one in Austria.
Further research should address the methodology on how to improve impact measuring, in particular the relationship between HTA and the overall improvement of health (care systems).
CONTACT INFORMATION
Ines Schumacher, MPH (ines.schumacher@hta.lbg.ac.at), and Ingrid Zechmeister, Dr. rer. soc. oec., MA, Ludwig Boltzmann Institute for Health Technology Assessment, Garnisongasse 7/20, 1090 Vienna, Austria
CONFLICTS OF INTEREST
The authors report no potential conflicts of interest.
