The Austrian Ludwig Boltzmann Institute for Health Technology Assessment (LBI-HTA) implemented an early awareness and alert system (EAAS) for oncology drugs in 2009. The so-called “horizon scanning in oncology” program (HSO) was developed based on the requests of regional hospital providers and the Austrian Ministry of Health (MoH). Ever-increasing expenditures and the early adoption of new oncology drugs in Austria necessitated an EAAS focusing specifically on anti-cancer drugs.
The purpose of this program is to identify new or emerging anti-cancer drugs for which a relevant therapeutic or financial impact can be expected, with the ultimate goals to provide further information about evidence-based decisions on the use of these therapies and to facilitate the estimation of budget implications. Accordingly, the main target groups are medical directors, heads of hospital pharmacies, representatives of the MoH, and members of pharmaceutical committees.
Briefly, this program consists of several steps inherent to an EAAS: after identification of new oncology drugs by searching nine sources including Web sites of regulatory bodies, peer-reviewed journals, and conference abstracts, the drugs are filtered every 3 months based on the criteria “availability of phase III results” and/or “submission of market authorization in Europe or in the United States” (1). The filtered drugs are then prioritized by an expert panel consisting of seven specialists, that is, hospital pharmacists and oncologists. The expert panel applies five criteria to identify drugs with a substantial impact on clinical outcomes and/or costs, for which concise assessment reports are published in English, consisting of chapters on drug description, indication, European Medicines Agency (EMA)/US Food and Drug Administration (FDA) licensing status, current treatment options, evidence (phase III/phase II studies), costs, ongoing trials, and a comprehensive commentary section. Relevant target groups are notified by e-mail when new reports are available. From approximately 120 identified anti-cancer drugs, approximately 12 are selected during each prioritization round, resulting in one to three reports every quarter. Currently, forty-eight assessments and three updates are available online (2).
Despite having defined target groups, our EAAS lacks customers actively requesting assessments on specific topics, which demonstrates that clear ties to informing decisions are missing. Even though financial and human resources needed for performing assessment of oncology drugs are minimal in comparison to the costs of these medicines, considerable resources are dedicated to the HSO program for regular monitoring of information sources to identify drugs, for preparing expert panel prioritizations, and for compiling the assessments. Under these circumstances, and particularly in times of economic uncertainty, the relevance of outputs for stakeholders is a prerequisite to justify the investment of scarce financial resources in research activities (3;4). Despite several countries having long-established EAASs, experience with evaluations is scarce (5;6). Packer et al. (5), for example, explored the accuracy of identification and filtration processes using end user and international databases as surrogates for significance to health services and patients. However, this shortcoming was acknowledged by the International Information Network on New and Emerging Health Technologies (EuroScan International Network), which emphasized the importance of EAAS evaluations by including a separate chapter on evaluation in the updated version of the EuroScan Methods toolkit (7).
Nonetheless, to identify whether maintaining our current EAAS processes and, ultimately, to determine whether the continuation of the system as a whole is feasible, we initiated a research project to formally evaluate the HSO project. Questions focused on whether our reports were used and if so by whom—and whether the topics prioritized for assessments and the content of the assessments were deemed relevant by our target groups.
Since the introduction of the HSO, the research and policy landscape has also evolved rapidly, aiming increasingly towards collaborative initiatives for the production of joint reports according to shared methodologies and standards. Such initiatives include the early advice program “Shaping European Early Dialogues for Health Technologies,” developed in connection with the European Network for Health Technology Assessment (EUnetHTA) Joint Action 1 and 2, and the European Commission Research Framework Programme research projects such as Advance-HTA or Integrate-HTA (8–10).
With the intention to adapt our HSO program to user demands and changing international developments, we aim to describe experiences garnered through evaluating the HSO program. We analyze the problems encountered and lessons learned for sustaining the relevance of outputs for decision makers. In addition, implications of contextual factors beyond those captured by evaluation methods will be described as necessary considerations to ensure the efficiency of research activities and outputs.
METHODS
To investigate the impact of the HSO program and to highlight areas where improvements could increase user satisfaction with our outputs, the evaluation aimed to ask questions designed to identify the actual readers of the HSO reports, and to determine whether the reports were used for decision making and, if so, the types of decisions involved. A further aim was to identify necessary adaptations of outputs, dissemination, and/or marketing strategies.
Based on a model developed by Gerhardus et al. (11), these questions were tied to four steps of a six-tiered hierarchical model: awareness, acceptance, policy process, and policy decisions. The two final steps, impact on clinical practice and impact on health and economic parameters, were not evaluated.
Four different methods were used to capture these dimensions (i) Download analysis: to capture the usage of HSO reports, the total number of site views, and the total and average number of downloads of all HSO reports published on the LBI-HTA Web site were analyzed. Average download numbers were calculated as the total number of downloads within a year divided by the number of months during which the report was available online. By using AWStats (Advanced Web Statistics 6.9), the respective numbers for twenty-four reports were computed for the time period October 2009 to February 2012.
(ii) Online survey: The usage, timeliness, relevance, quality, and format of our HSO reports were evaluated in spring 2012. The survey was sent to 130 individuals who receive the quarterly notifications on new reports and are thus considered to be the main target group of the HSO reports. Recipients of the survey consist of medical directors, heads of hospital pharmacies, representatives of the MoH, payers’ organizations and members of drug commissions. Members of this mailing list were initially identified by means of a Web site listing all Austrian hospitals with oncology departments. In addition, relevant MoH representatives known to our institute were added, and where public information on members of drug commissions was not available, hospitals were approached directly.
(iii) Environmental analysis: For the environmental analysis, other HTA agencies performing (early) assessments of anti-cancer drugs were identified by searching, in 2012, four databases (Centre for Reviews and Dissemination Health Technology Assessment [CRD HTA], International Network of Agencies for Health Technology Assessment [INAHTA], EuroScan, EUnetHTA's planned and ongoing [POP] database), as well as the Web sites of overarching HTA network organizations (INAHTA, EuroScan, EUnetHTA, and HTA International [HTAi]). To identify potential overlaps in topics assessed and similar publication dates, these sources were screened for the indications that had already been assessed within our HSO. Besides the publication dates and their relation to EMA licensing decisions, the content of assessments was analyzed. The aim was to identify redundancies in terms of HTA reports published on the same topics and at similar times to our HSO reports, which should allow the identification of necessary modifications of our HSO in terms of timing, topic, or content, or allow the initiation of collaboration.
(iv) Web site questionnaire: To further identify the professional and geographical background of the HSO users, downloaders had to indicate their country of origin and their profession by means of a pop-up questionnaire (data are presented from December 2012 to February 2014).
The results of these formal evaluation methods were then discussed with the expert panel involved in the prioritization of our HSO in March 2014. The main findings and open questions were presented to the panel and potential adaptations of our HSO program were discussed.
RESULTS
Impact Evaluation
Online Survey
With repeated reminders, a response rate of 28 percent (n = 36) for our online survey was achieved, which is slightly lower than average numbers reported (12) (see Table 1). Of those responding, 81 percent (n = 29) completed the survey. Despite the quarterly e-mail notifications, only approximately half of respondents were familiar with HSO products, thus only nineteen respondents were able to answer specific questions on the HSO program. Concerning satisfaction with HSO reports, overall, 94 to 100 percent of respondents were satisfied with the structure, breadth, content and quality, that is, the clarity, comprehensibility, and scientific quality of the reports (Table 2).
Online Survey Background
Online Survey Results
In terms of timing, 18 percent of respondents indicated that they were only fairly satisfied because they would have needed results at an earlier point in time. Nearly all respondents (94 percent) considered the assessed drugs as “relevant” and 71 percent answered that they used the reports for either clinical or reimbursement decisions. However, when asked whether they agreed with specific statements on the usefulness of the reports, more diverse answers were retrieved (Table 2).
Reasons for not using the reports as indicated by six respondents were lack of clear recommendations, that the publication language was English and not German, the publication date, or that reports were not or hardly relevant for their ongoing work.
Fourteen survey participants used the opportunity to provide free text feedback. The majority indicated that the reports were important, informative, offered impartial information, and gave a good and early overview on anti-cancer drugs. Two comments made by clinicians maintained that the reports were not important or that clinicians would be able to assess new drugs based on the licensing documents provided by the EMA alone. Six suggestions for improvements were: inclusion of a short summary in German, earlier assessments, inclusion of a cost-benefit analysis, a more pronounced display of side effects and quality-of-life aspects, as well as the implementation of comparative assessments of established anti-cancer treatments.
Download Analysis
The analysis of the twenty-four published HSO reports covering the period of October 2009 to February 2012 showed that they were downloaded approximately 7,000 times and viewed 14,000 times, thus ranking among the most often used products of the LBI-HTA (13). Average monthly download rates, that is, the total number of downloads within a year divided by the number of months the publications were available online, increased from 11.1 percent in 2009 to 42.3 percent in 2012. Concerning usage of individual reports, everolimus for the second-line therapy of advanced/metastatic kidney cancer was the report viewed most often (n = 1,752), whereas bendamustine for the treatment of chronic lymphocytic leukemia, non-Hodgkin lymphoma, and multiple myeloma was the report most often downloaded (n = 914). A clear association between active dissemination strategies, that is, e-mail notifications and the monthly LBI-HTA newsletter announcing new HSO reports, and an increase in downloads was not found.
Web Site Questionnaire
Due to the discrepancy between the rather high download rates and the low response rates to the online survey, the main users of our HSO reports could not be identified. Therefore, the Web site questionnaire was implemented in December 2012 to better identify user profiles. From December 2012 to February 2014, HSO reports were downloaded 1,696 times (see Table 3); the majority of downloaders were international users, and 34 percent came from Austria.
Results from the Website Questionnaire (December 2012–February 2014)
When user profiles were analyzed according to profession, the most prolific users of our reports were from industry, followed by HTA institutes, and clinicians or pharmacists (Table 3). Thirty-six percent of all downloads were attributed to the actual target groups, that is, clinicians/healthcare providers/pharmacists, as well as clinical and political decision makers; 3 percent of users were patients.
Due to the high number of international users, we further analyzed the data according to profession and country of origin (Table 3). Regardless of geographic area, industry still accounted for most frequent users. Slight differences were seen for clinicians, healthcare providers, and pharmacists with a higher percentage among Austrian users, whereas more non-Austrian health policy decision makers had downloaded the reports compared with Austrians. However, “other” professional background was indicated twice as often by Austrian users, without provision of further information.
Environmental Analysis
Overall, nine institutions that had also assessed oncology drugs (Supplementary Table 1) were identified. In terms of timing in relation to EMA licensing decisions, the HSO reports were published on average within four months after approval. Of note though, the filtration and identification criteria had been re-defined in the first 2 years of the HSO program to allow earlier identification of new drugs (nearer to approval by the EMA). In any case, besides the National Institute for Health Research (NIHR) Horizon Scanning Center, the HSO reports were among the first to be published. The vast majority of agencies published their assessments 6 months after EMA approval, and in some instances even after 32 months.
In addition, all but two drugs assessed within our HSO were also subject of assessments by other agencies; up to five further HTA institutes had assessed the same topics.
Expert Panel
The main findings of the four evaluation methods were presented to the expert panel involved in the prioritization of the anti-cancer drugs. Five of the seven experts attended a 1-day workshop in March 2013. Generally, high and increasing download rates of HSO reports supported the overall user satisfaction with quality, content, and indications assessed. However, main discussion points were:
-
• The timing of the reports: Because 18 percent of responders to the online survey had indicated that reports were published too late, the panel consisting of medical directors, clinicians, representatives of the social health insurance, and pharmacists was asked to share their experiences when they require information on new oncology drugs.
-
• Target audience: Because the main user of the HSO reports was industry, input was sought as to whether information requirements as well as timing of information differed between various occupational groups. In addition, panel members were asked to identify any further occupational groups involved in decision making that had not yet been included in the HSO mailing list.
-
• Sources of information: Because only approximately half of the e-mail alert recipients were familiar with the HSO reports, suggestions for a more refined dissemination strategy including linkage to other potential information platforms to announce new reports were requested; this aims to increase usage of our HSO reports in the intended target population—especially in Austria. In addition, panel members provided insights on their main information sources on new oncology drugs.
-
• Redundancies: Because of redundancies in topics assessed between HTA institutes, panel members’ views were obtained on adapting a new format for the HSO reports according to methods developed by EUnetHTA, a European HTA network.
-
• Relevance of topics: As mentioned by some participants in the online survey, the question arose as to whether comparative assessments on already established drugs, potentially including a cost-benefit analysis should become part of the HSO program.
After discussing these issues, the expert panel concluded that timing of the HSO reports had already been improved due to the change of identification and filtration criteria allowing a more timely publication of the reports. Assessment of already established drugs and cost-benefit analyses were considered to be outside the scope of the HSO program and data on costs were rarely available before EMA licensing.
However, several adaptations were suggested:
-
• An improved dissemination strategy was recommended by collaborating with and linking to Austrian specialist societies in the fields of oncology and pharmacology.
-
• Further relevant political decision makers such as hospital managers, heads of divisions, and financial controllers should be identified and included in the e-mail alerts.
-
• Contact should be established with members of the National Drug Commission for expensive drugs, which was established in 2014 in Austria. Even though it is not yet clear whether oncology treatments will fall within the remit of this Commission, it may act as an active customer in the near future with clear ties to decision making.
-
• Adapting the format of the HSO reports according to methods developed by international networks was acknowledged as an opportunity for increasing collaboration and reducing redundancies.
-
• In addition to the publication of the reports, experts indicated that the results of the prioritization of new oncology drugs may provide useful information for decision makers. Knowing which drugs are in the pipeline of manufacturers may specifically prove useful for hospital pharmacists, a group that is usually informed less, and later, than medical doctors, by pharmaceutical representatives
-
• Lastly, comparative effectiveness research in terms of considering all available treatment options for a specific indication and not only one comparator was discussed as an important approach, especially for the Austrian health insurance sector.
DISCUSSION
Impact evaluation has been recognized as a crucial step of HTA activities, not only to justify expenditures, but also for quality assurance (11). Even though consensus exists that a HTA should ultimately improve health or economic outcomes, a “gold standard” evaluation framework does not exist, with varying approaches described in the literature. Capturing the impact on actual policy decisions has been acknowledged as difficult and time-consuming, because several factors outside the influence of HTAs determine these outcomes: For example, the existence of and linkage to clearly defined policy processes influence health policy decisions, clinical practice, and, thus, on outcomes directly related to health or costs (11;14–17).
Our formal evaluation concentrated on proxies for assessing the impact of HTA, such as awareness and acceptance of HTA reports (15). Our experiences with the evaluation were manifold. First and most importantly, download rates, user satisfaction, and, to some extent, evidence for a change in the awareness about new oncology drugs confirmed the relevance of our outputs. Nonetheless, the importance of a refined dissemination and marketing strategy was highlighted to increase awareness and usage of the HSO reports in the intended target population. Of course, the fact that our impact evaluations were conducted in-house may have introduced bias and an external evaluation, as already acknowledged by Packer et al. (5), might have provided more neutral outcomes. In addition, even though the impact of nonresponse bias is discussed controversially, the online survey response rate of 28 percent may limit the validity of the findings on user satisfaction, because responders may have been more likely to be satisfied with the reports than nonresponders (18). On the other hand, quantitative methods, that is, download rates, are less prone to be influenced by the observers.
Second, by applying several methods in a continuous, multi-phase process, clear recommendations for further improvement were obtained. An additional product, that is, the publication of the prioritization results, has been introduced, potentially increasing the usefulness of the HSO program. Associations of relevant target groups (e.g., the Association of Hospital Pharmacists, the Society of Oncology Pharmacy and the Society of Hemato Oncologists) have been contacted and, as a result, newly available HSO reports will be announced on two of these Web sites to increase outreach. Because the Austrian health insurance is bound by law to formulate a decision on the acceptance of medications into the Reimbursement Codex within 180 days after an application for inclusion in this Codex has been requested by the manufacturer (19), it will also be tested whether comparative effectiveness research can be performed for new anti-cancer therapies within a reasonable period of time.
Third, our Web site questionnaire revealed that the most common users were not from Austria, but that 66 percent of HSO users had an international background. More specifically, user profiles based on profession and country of origin showed that a substantial number of international decision makers, clinicians, and HTA agencies also used our assessments. In addition, with the implementation of EUnetHTA's POP database which also contributed to the findings of the environmental analysis, it became apparent that oncology drugs were the topic of investigation in many agencies showing a considerable overlap. The collaboration of HTA bodies has been acknowledged as an important tool to reduce duplication and, thus, to increase efficiency (20). It is conceivable that a more comprehensive search may have yielded further HTA agencies that published assessments on oncology drugs, for example, those published in languages other than German or English, or those not listed in the relevant databases, but this would only emphasize the need for even further collaboration. Due to the high overlap in topics assessed, the LBI-HTA initiated a workshop on Collaboration in Oncology in October 2010 as an spin-off of the EUnetHTA initiative. Following this workshop, partners interested in conducting joint assessments on new oncology drugs have been sought out by sending “calls for collaboration”. Eleven calls have been sent out to date, resulting in thirteen collaboratively produced reports. However, the environmental analysis demonstrated further potential for collaboration with agencies not having participated in the workshop.
Furthermore, the international uptake by HTA agencies and the considerable overlap in topics highlights the importance of participation in and development of common tools and processes for HTAs. Collaborative efforts of individual HTA agencies are facilitated when the same methods are applied. Reductions in personnel needed, availability of expertise not covered by individual agencies and, therefore, potentially lower costs associated with producing HTAs also determine the efficiency of HTA programs. The re-use of reports produced according to a standard methodology will reduce time to publication and allow evidence-based coverage of an increasing number of technologies. As a consequence, HSO assessments will be produced as of 2015 according to the structure of the HTA Core Model for Rapid Relative Effectiveness developed by EUnetHTA (21). EuroScan International Network also offers a platform for exchanging methods used for specific stages of EAAS and experiences with impact evaluations providing the opportunity to identify areas for potential methodological and practical collaborations.
The increasing collaboration between HTA institutes and the usage of reports by international HTA agencies and other relevant target groups necessitate an even broader definition of impact (22). First and foremost, HTA agencies not embedded in legally binding decision-making processes should consider the usage of their products outside their intended target audience. Despite the intuitive validity of this claim, capturing this kind of impact is even more difficult, because frameworks are scarce, and our methods did not enable us to determine the underlying rational of international usage.
Nonetheless, we believe that participation in international networks, increasing collaboration of HTA agencies, the development of standard procedures, and the availability of more tools for exchanging information on planned or published reports will increasingly determine effectiveness and impact. Thus, in addition to the clearly defined adaptation of processes and methods based on findings from formal, program-specific evaluations, the consideration of international impact, and the responsiveness to external factors are crucial to guarantee the value for money of HTA agencies in the long run.
CONCLUSION
The evaluation of HTA activities by using a wide set of different methods does not only provide evidence for the improvement of research activities, but also yields clear recommendations to tailor outputs to the specific needs of the intended target population. This, however, cannot be seen as a standalone, one-time activity, but rather as a multi-phase process with feedback loops and the repeated monitoring of changes.
Even though evaluation frameworks focus on impact on the national or local level, a fast developing HTA community and increased international efforts for collaboration will gradually determine estimates of HTA impact. The evaluation of single programs focusing on the local or national target groups may, therefore, not provide satisfying results for keeping research activities up-to-date and for providing proof of return on investments. Refined methods and a broader definition of impact are thus needed to ultimately capture efficiency of national HTA programs.
SUPPLEMENTARY MATERIAL
Supplementary Table 1 http://dx.doi.org/10.1017/S0266462316000052
CONFLICTS OF INTEREST
All authors report having no conflict of interest.
