Currently, European countries assess and report on the relative effectiveness (RE) of pharmaceuticals as part of their reimbursement decision process according to their own methods and templates (Reference Kleijnen, Goettsch and d’Andon1). Harmonization of assessments and collaboration in the production across countries has the potential to avoid duplication of effort for both manufacturers and health technology assessment (HTA) organizations who are involved in preparing and reviewing submissions for innovative pharmaceuticals (Reference Hutton, Trueman and Facey2). The High Level Pharmaceutical Forum (HLPF), which was a European multi-stakeholder discussion platform between 2005 and 2008, included a specific working group on RE with the aim to support Member States in applying RE systems to allow containment of pharmaceutical costs as well as a fair reward for innovation (3). Based on the recommendations of the HLPF on relative effectiveness assessment (REA) a reporting model (the HTA Core Model® for Rapid REA of Pharmaceuticals, hereafter referred to as the REA model) and several methodological guidelines for rapid REA of pharmaceuticals were developed in work package 5 of the European Network for Health Technology Assessment (EUnetHTA) Joint Action 1 (WP5/JA1) (Reference Kleijnen, Goettsch and d’Andon1).
Assessing RE means presenting the clinical benefits and harms of the technology in comparison to other technologies. A “rapid” assessment refers to an assessment that should be conducted within a limited timeframe (Reference Kleijnen, George and Goulden4). Many European countries are legally restricted to a limited timeframe of 90 (pricing or reimbursement decision) or 180 days (pricing and reimbursement decision) to provide timely access to patients for newly launched pharmaceuticals (5).
The REA Model is a modification of the HTA Core Model®, which is a framework designed within EUnetHTA for production and reporting of shareable pieces of HTA information. The HTA Core Model® consists of a set of generic questions (i.e., assessment elements) grouped in nine domains (see Figure 1) (Reference Lampe, Mäkelä and Garrido6). Per assessment, assessors select generic questions from the model that they consider relevant for their topic and formulate them into research questions specific to their assessment. The Model provides methodological guidance for undertaking an assessment and a structure for reporting the results as question and answer pairs in the form of “result cards.” For national HTA report production it is considered potentially easier to adapt pieces of information from result cards than from a traditional foreign HTA report: the structured and detailed reporting in result cards facilitates users in finding the required information and in evaluating the quality of the information.
Tools used in pilot assessment
The cost and economic evaluation domain was excluded from the first version of the REA Model based on the recommendations of the HLPF, that health economic assessments should not be the primary focus of the European collaboration (3). In addition, some questions of the HTA Core Model® were excluded as they were not considered relevant for a REA of pharmaceuticals. The first version of the REA Model contained eight domains (see Figure 1) with seventy-eight generic questions. It was the first application of the HTA Core Model® that focuses on pharmaceuticals. In addition to the model, Guidelines for Rapid REA of Pharmaceuticals were developed for the methodological issues/challenges that assessors often face (see Figure 1).
A pilot assessment was conducted to test the first version of the HTA Core Model® and the draft Guidelines for Rapid REA of Pharmaceuticals, as well as the capacity for international collaboration required to produce an assessment. The aim of the present study was to describe and analyze the lessons learned from the pilot project based on input from three different perspectives: the assessors, the users (HTA organizations) and the marketing authorization holder (MAH). This article does not address the scientific quality of the pilot report.
METHODS
Organization of the Pilot
The outcomes of the pilot are listed below:
-
a) Primary outcome:
-
• Assessors’ perceptions of the 1st version of the model and draft guidelines
-
-
b) Secondary outcomes:
-
• Duration of the assessment
-
• Workload (in terms of person-days)
-
• Feasibility of international cooperation
-
• Users’ and MAH's perceptions about the format of the pilot report, its readability and adaptability of information for national purposes.
-
The pilot was led by a coordination team at the Dutch Health Care Institute (ZIN, formerly know as CVZ). For previous HTAs produced with the HTA Core Model®, many organizations and individuals were involved in the assessment to share the workload and maximize engagement as much as possible. Based on the same concept, eight domain teams were established, each including two to five authors. In total, fifty-four individuals representing twenty-two HTA organizations from sixteen European countries participated in the pilot (Table 1). Individuals were assigned as authors or reviewers to domain teams based on their expertise in specific HTA domains. For each domain, one participating HTA organization was appointed as the leader. Another expert team (synthesis team) was organized to prepare an overview of the relative effectiveness that combines the benefits (information from the clinical effectiveness domain) and harms (information from the safety domain) versus relevant comparators. The synthesis team consisted of project participants who had not been involved in the production of the safety or clinical effectiveness domain.
Domain Teams and Person-Days Used for the Pilot
AETSA = Andalusian Agency for Health Technology Assessment; A. Gemelli, University Hospital ‘A. Gemelli’; AIFA = Italian Medicines Agency; CAHIAQ = Catalan Agency for Health Information, Assessment and Quality; CAST = Danish Centre for Applied Health Services Research; CHE = Centre of Health Economics of Latvia; DPA = Directorate for Pharmaceutical Affairs, Ministry for Energy and Health Malta; ESKI, National Institute for Strategic Health Research; FIMEA = Finnish Medicines Agency; FOPH = Swiss Federal Office of Public Health; GÖG,The Health Austria GmbH; HAS = French National Authority for Health; HVB, Association of Austrian Social Insurance Institutions; IVZ-RS, Slovenian National Institute of Public Health; KCE, Belgian Health Care Knowledge Centre; MoH Spain, Spanish Ministry of Health; NICE = National Institute for Health and Clinical Excellence; NOKC = Norwegian Knowledge Centre for the Health Services; RIZIV, Belgian National Institute for Health and Disability Insurance; THL, Finnish National Institute for Health and Welfare; UETS, Health Technology Assessment Unit of the Autonomous Community of Madrid; ZIN, Dutch Health Care Institute.
Topic Selection
A list was produced of all pharmaceuticals that received market authorization between 1 June 2010 and 1 February 2011; excluding generic pharmaceuticals, orphan drugs, vaccines, extension of indications and medicinal product for diagnostic use only. From the list, pazopanib for the treatment of advanced renal cell carcinoma was chosen based on the preferences of WP5/JA1 member organizations and stakeholders (7).
Assessment Process
The assessment was divided into six different phases (Figure 2). In the first phase, coordinators provided the teams with a submission file, the European Public Assessment Report (EPAR) for pazopanib, and the results of a literature search performed by an information specialist in May 2011. The submission file was provided by the MAH and was based on the submission file that had been presented to the National Institute for Health and Care Excellence (NICE). The literature search included three reference databases (Medline, Embase, and CRD). Two coordinators independently screened titles and abstracts using defined exclusion criteria and selected articles which were potentially relevant to any domain, irrespective of study type. Full texts were retrieved for the included abstracts and placed in the electronic workroom available for all domain teams. Details of the literature search are presented in Supplementary Table 1, which can be viewed online at http://dx.doi.org/10.1017/S0266462314000622. In the second phase, the scope of the assessment was discussed. The scope was defined using a structured template with fields for the technology, indication, and comparison (TIC). Endpoints were not defined in the scoping phase; it was left for the domain teams to choose appropriate endpoints when formulating the research questions in the third phase. In the fourth phase, the domain teams planned the methodology to answer the research questions, including additional question-specific literature searches if necessary. In the fifth phase, the research questions were answered and reported. At the end of this phase, the RE section was produced by the synthesis team and the coordinators combined all domains into one report. The report was sent out for consultation (review) and finalized in the sixth phase. The report was reviewed by HTA organizations participating in WP5/JA1 and the MAH (November 2011–January 2012), the WP5/JA1 Stakeholder Advisory Group that consists of a group of content related experts from various stakeholders including European manufacturer associations of pharmaceuticals and medical devices (March–April 2012), and the public (June – July 2012).
Project phases and calendar days (CD) used for the assessment.
Evaluation of the Pilot
The coordinators collected feedback by email from authors throughout the project, including the number of days worked. Three different surveys were developed to obtain feedback from (a) pilot participants (authors, reviewers and members of the synthesis team) (b) HTA organizations participating in WP5/JA1 which we refer to as users as they should use the report, and (c) the MAH. The draft surveys were sent to WP5/JA1 organizations for feedback. The final surveys included, respectively, twenty-nine, twenty-six, and twenty-two mainly multiple-choice questions including space for comments and some open questions. The surveys were distributed by email along with the consultation of the first version of the assessment by HTA organizations participating in WP5 /JA1 and the MAH (November 2011–January 2012).
RESULTS
The first version of the assessment report, which was subject to consultation by HTA organizations participating in WP5/JA1 and the MAH, consisted of seventy-eight pages, with an Annex of 323 pages. Of the seventy-eight generic questions in the REA Model, two thirds were considered relevant for the assessment; twenty-eight questions were excluded as irrelevant (see Table 2). As five questions were further sub-divided into two separate research questions, the final number of question and answer pairs was fifty-five.
Selected Research Questions for Pazopanib Pilot and Grading of the Relevance of the Domains by Health Technology Assessment Organisations
*REA Model, HTA Core Model for rapid Relative Effectiveness Assessment of Pharmaceuticals.
** from 0–10 with 10 being very relevant.
Feedback Gathered Throughout the Process
Undertaking the Assessment
Only the effectiveness domain team based their assessment solely on the information provided by the coordinators. Six other domain teams performed additional searches for more than half of their research questions, and the legal domain team relied completely on their own searches. It was believed that the literature search (including a thorough and shared screening procedure for relevant articles) is not feasible in most real life rapid assessments because of the limited number of days available for a rapid assessment (90 or 180 days).
Using the scoping structure proposed by the HTA Core Model® (TIC) was not considered optimal for rapid REA of pharmaceuticals. As the main focus of a REA are the health benefits and harms, the authors wanted to include the outcomes in the scoping, and structure the scope according to PICO (Patient, Intervention, Comparator, Outcome) scheme. Due to diverging views, the number of comparators selected was discussed and questioned in several phases of the pilot.
The differing attitudes of participating countries toward the relevance and validity of main efficacy endpoints became apparent during the assessment, for example the importance of progression free survival and the validity of the overall survival data. Guidance was provided to all domain teams to present the endpoints neutrally and in a balanced way, leaving room for interpretation by individual countries. The aim was to provide results which would be as useful for as many countries as possible.
Resource Use and Duration of the Pilot
In total, 401 person-days were used for this assessment (see Table 1). Of these, 251 person-days were used by members of the domain teams. The input of the individuals varied between 4 and 30 person-days (mean 9 person-days; median 8.5 person-days). An additional 150 person-days were used for coordination and editing: this includes the time spent on identifying the topic, compiling the report and communicating the different phases with various stakeholders. The actual assessment by the authors (scoping, selecting research questions, planning domain methods and answering the questions) was performed within 3.5 months, from May to August 2011 (see Figure 2). An additional three months, from September to November 2011 was spent, mostly by coordinator and the synthesis team, in editing and collating the heterogeneous outputs of the eight domain teams and preparing the synthesis. In addition, the various rounds of consultation lengthened the process. So, the final report was published in December 2012.
Feedback Gathered Through the Structured Surveys
We received twenty-six responses from twenty-four organizations on the user's survey (HTA organizations). In the two cases of double response only the first was included, and one further response (not excluded) was partial. Hence, in total, twenty-four of thirty-three EUnetHTA organizations (73 percent) who participated in the development and testing of the REA Model and Guidelines responded to the survey. The response rate of the individual pilot participant's surveys was lower (33 percent, 16/49). In addition, the MAH provided a response, which was a consolidated view from a central project team and four different local operating companies (Finland, the Netherlands, Spain, and the UK).
Using the REA Model and Guidelines (Results from Participant's Survey)
Authors perceived the domain teams as a useful way of working as it allowed a division of interests and tasks across agencies and individuals. On the other hand, the parallel work of the domain teams and parallel work of authors within a domain resulted in duplication of work. The participants were mostly pleased with the collaboration within their domain team, but believed that there was too little information exchange across the domains. Informal ad hoc discussions did not occur as often as within an office, and this was mentioned as an apparent drawback of collaborating remotely.
Regarding the selection and formulation of relevant research questions, authors responded that specific research questions they considered relevant were missing and others were redundant. In addition, they reported that there seemed to be overlap between questions and that the wording of questions allowed for different interpretations. A common complaint of participants was that browsing between the numerous result cards (question and answer pairs) was cumbersome due to the lengthy documents and it was, therefore, difficult to get an overall picture from the separate cards.
The time frame was generally considered sufficient by the authors, although the pilot period included the summer holiday season. The work load was perceived by some authors as high. For example, authors of clinical effectiveness domain reported a heavy work load for the assessment phase because of the large number of comparators and the need to separate direct and indirect comparisons.
The draft methodological guidelines produced in WP5/JA1 were mainly used by clinical effectiveness and safety domain teams. The authors perceived the quality of the guidelines to be good, but commented that some guidelines were not targeted to HTA assessors. They were either too long or too complicated, and did not always offer a solution to the problem described. They lacked recommendations derived from real life assessment experience and recommendations on possible consequences for the assessment.
Readability of the Report and Adaptability of Information for National Purposes (Results from User's Survey & MAH)
A majority of the HTA organizations (17/24) and the MAH considered the approach of dividing HTA into domains useful. The following criticisms of the readability of the report were mentioned by HTA organizations and the MAH: the report is very long (too much information and a risk of the most relevant points becoming diluted), the structure was considered complex (it is cumbersome to find specific information and there is repetition of information) and the report was not directly suitable for decision makers. Recommendations for increasing the usability of the report were: further consolidation, including justification for the choice of comparator and outcomes, description of evidence gaps, and cross-referencing across sections.
The majority of the organizations (20/24) as well as the MAH believed that it is (partly) easy to use (extract or update) data from the result cards. However, it was mentioned that finding the specific result card that contains the information needed is cumbersome and that consistent and comprehensive labeling and annotation of supporting data tables would be helpful. The following suggestions were made for adding information to the synthesis section: more details regarding the indirect comparisons; numerical results for the main endpoint; cost-effectiveness information; limitations/uncertainties that remain in reviewing the evidence; different perspectives; and, a conclusion. The MAH emphasized that because of the international scope of a joint report it is important to avoid value judgments and to ensure that in cases where there is room for methodological debate within the international HTA community, different perspectives are presented and discussed in a balanced manner.
The HTA organizations were asked how relevant they found the domains for rapid REA of pharmaceuticals. The first four domains were graded as very relevant, whereas the last four were graded less relevant (Table 2). It was mentioned that the relevance of the last four domains depends on the pharmaceutical being assessed.
Seven of twenty-three HTA organizations responded that the report could be used as direct input for national report production, whereas 12/23 mentioned that the current report structure requires adaptation for direct use in national report production (e.g. adhering to local guidelines for assessment of pharmaceuticals, information regarding the local healthcare system, adding economic parameters). The majority of organizations (17/23) stated that the report would require adaptations to be used as a direct input for national decision making. The English language of the report was not a concern for most of the responding organizations (18/23).
Most WP5/JA1 organizations responding to the survey (19/23) thought it at least very likely that the credibility of the report is increased through broad participation of organizations/countries.
DISCUSSION
The pilot was very ambitious on different levels. First, while participation of twenty-two organizations in a single rapid REA shows commitment, it required intense coordination for organizing the input, and the output (producing one report out of many pieces of information). Additionally, the short timelines available for the scoping and assessment of a rapid REA were very challenging when working with so many organizations and so many authors on one assessment.
Some of the issues identified during this pilot were also identified in earlier pilot assessments performed with the HTA Core Model®, such as the challenges with communication in an international project and overlap and/or duplication of information (Reference Pasternack, Anttila and Mäkelä8). Overlap and/or duplication of information is less of a problem for undertaking a broad HTA without a limited timeframe as the basic idea is that the individual pieces (result cards) of information are the cornerstone for sharing work. However, for a rapid REA it is difficult to produce so much information within a limited timeframe. Moreover, for a rapid REA the most relevant end product may not be the result cards, but the synthesis (summary that combines all information about the RE).
It has already been suggested that for an assessment performed with the HTA Core Model®, duplication of efforts can be minimized by various domain teams working in sequence (Reference Pasternack, Anttila and Mäkelä8). Such an approach is, however, not possible for a rapid REA due to the limited time available. A reasonable alternative is a reduction in the number of authoring organizations and individuals participating in the assessment. With fewer authors, overlap of information in various parts of the assessment can be avoided more easily and the consistency throughout the assessment can be improved. In addition, efficiency in communication can be increased. The benefit of broad participation from various countries could still be ensured by involving several organizations, five, for example, in an in-depth review.
Most participating organizations considered the first four domains, which include the health problem and the current use, a description of the technology, clinical effectiveness and safety, as very relevant for a rapid REA but not all research questions in these domains. The lengthiness of the reports and the complex structure were considered cumbersome by both participants and users; it was not easy to find specific information. Therefore, we think that a selection of only the most essential information for a rapid REA, with a strong focus on the first four domains, would be preferable and improve readability and usability of the assessment. For the other domains, it could be considered during the scoping phase whether some aspects are particularly relevant for the topic that is assessed, and should be included in the assessment after all.
Using a submission file from the MAH as a relevant source is current practice in national rapid assessments of pharmaceuticals (Reference Kleijnen, Goettsch and d’Andon1), however, this was the first pilot with the HTA Core Model® to do so. In the absence of a template, a NICE submission file was used; the content of the file was not always easy to use for the assessors in the pilot. A template for such a submission file would be preferable to optimize the usefulness of the content and to provide guidance for the MAH. Based on the experience with this pilot we think that for future rapid REAs an extensive literature search at the beginning of an assessment is not feasible, due to the work and time involved. It would be more efficient if the MAH conducts the literature review and presents it as part of the submission file. To minimize the risk of working with biased or outdated results, the search results as presented by the MAH, should be checked for accuracy by the assessors.
Participating organizations believed that it was essential to have a comprehensive scoping exercise undertaken beforehand using PICO instead of TIC. To make the assessment accessible to a wide number of countries, a justification for the choice of comparators and the outcomes would be essential.
The discrepancy between the definition of clinical effectiveness in the HTA Core Model® and the definition of RE of the HLPF lead to several discussions during the assessment period. The first one describes the clinical benefits (9), whereas the second definition describes the benefit/harm ratio versus the comparator (10). Although an objective quantification of the RE would be valuable, currently there is no established method to calculate such a ratio. Therefore, it was decided to present the relevant outcomes from the clinical effectiveness domain and safety domain of pazopanib and the chosen comparators in the form of a table, together with the quality of the evidence available. This allows the different HTA organization to draw individual RE conclusions, based on local/national systems and values. Although it was perceived that the credibility of the report is increased through broad participation of organization/countries, it also calls for balanced and neutral description of topics where different countries may have different points of view (e.g., relevance of endpoints or methodological aspects). Further optimization of the synthesis should be possible in future assessments.
The resources used in this pilot were considerable. However, we do not believe that this pilot would be representative for future joint rapid assessments as it includes resources for a topic selection procedure, a literature search, setting up a process for doing a joint rapid assessment and collating pieces of information into one report. Nonetheless, in order for the timelines of a rapid REA (90 to 180 days) to be achievable, the number of authors should be reduced to a few (e.g., 2 to 4 persons) and the assessment should be generally limited to the first four domains of the HTA Core Model® for a rapid REA. Finally, for many participants the exercise was new, which means that work can also be done more efficiently after authors/reviewers have become more acquainted with the tools and the process.
One of the limitations of this study was the low response rate of the participants’ survey. The rate is partly attributable to confusion that was caused by circulating different surveys (participants’ and user's survey). However, the response rate was mainly low from the domain reviewers (less active participants) whereas most of the authors (more active participants) of the various domains did respond, resulting in detailed feedback from the experience. As this article describes the lessons learned from a single pilot it is uncertain whether these lessons learned would also be applicable to other assessment topics. Future pilots with other topics and consideration of the lessons learned would be valuable for confirmation.
CONCLUSIONS
In conclusion, our pilot shows that the objectives of undertaking joint REAs (to increase the overall quality of RE information and to reduce duplication of efforts and thus save resources) are shared by many organizations across Europe. To fulfill these objectives it is necessary to not only produce these joint assessments with increased efficiency, but also to make the assessments easily adaptable for national decision making. Between 2013 and 2015, ten more joint REA pilots will be performed to test the adaptability of the produced assessments for national/local use. Based on the experience described in this article, these pilots will focus on the first four HTA Core Model® domains and will involve only a few authoring organizations (e.g., two) per pilot. Involving several organizations (e.g., five) in an in-depth review should ensure the benefit of broad participation. The results of these pilots will define whether joint assessments will become common practice.
SUPPLEMENTARY MATERIAL
Supplementary Table 1 http://dx.doi.org/10.1017/S0266462314000622
CONTACT INFORMATION
Sarah Kleijnen, MSc, in Biology, MSc in Health Economics Policy and Law and Project Coordinator EUNetHTA WP5 (skleijnen@zinl.nl), Zorginstituut Nederland, PO Box 320, 1110 AH Diemen, The Netherlands & Pharmacoepidemiology and Clinical Pharmacology, Faculty of Science, Utrecht Institute for Pharmaceutical Sciences, P.O. Box 80082, 3508 TB, Utrecht The Netherlands
Iris Pasternack, MD, and HTA professional, Zorginstituut Nederland, PO Box 320, 1110 AH Diemen, The Netherlands & Summaryx Ltd. Untuvaisentie 2 A 19, FI-00820 Helsinki, Finland
Piia Rannanheimo, MPharm and Pharmacoeconomist, Finnish Medicines Agency, PO Box 55, 00034 Fimea, Finland
Jenni M. Vuola, PhD, GlaxoSmithKline, PL24, 02231 Espoo, Finland
Marc Van de Casteele, MD, PhD and co-ordinator experts pharmaceuticals, Rijksinstituut voor ziekte- en invaliditeitsverzekering, Directie Farmaceutisch Beleid, Tervurenlaan 211, B-1150, St. Pieters-Woluwe, Belgium and KU Leuven, Herestraat 49, 3000 Leuven
Anna Bucsics, Medical Doctor and (former) head of Department of Pharmaceutical Affairs, external lecturer, Hauptverband der Österreichischen Sozialversicherungsträger, Department of Pharmaceutical AffairsKundmanngasse 21, 1031 Vienna POB 600 Austria & University of Vienna, Department of Finance, Oskar Morgenstern-Platz 1, 1090 Vienna
Isabelle Zahra Pulis, B. Pharm(Hons), MSc and Director Pharmaceutical Affairs, Directorate for Pharmaceutical Affairs, Ministry for Energy and Health, St. Luke's Hospital, G’Mangia PTA 1012, Malta
Rossella Di Bidino, PhD in Management of Healthcare Organizations, MSc in In International Health Care Management Institutional, HTA Unit, “A. Gemelli” Teaching Hospital, Largo Gemelli 8, 00168 Roma, Italy
Dario Sacchini, MD, PhD, Lic Phil and Assistant Professor of Bioethics, Institute of Bioethics, “Agostino Gemelli” School of Medicine, Università Cattolica del Sacro Cuore, Largo F. Vito, 1 - 00177 Rome (Italy)
Simona Montilla, Pharmacologist, PhD and Master of Science in Health Technology Assessment, Italian Medicines Agency, Via del Tritone, 181--00187, Rome, Italy
Payam Abrishami, MD, MA and Advisor, Zorginstituut Nederland, PO Box 320, 1110 AH Diemen, The Netherlands
Sylvana Magrin Sammut, B.Pharm(Hons), MSc and Senior Pharmacist, Directorate for Pharmaceutical Affairs, Ministry for Energy and Health, St. Luke's Hospital, G’Mangia PTA 1012, Malta
Luisa Anna Adele Muscolo, Hospital Pharm D, Italian Medicines Agency, Via del Tritone, 181-00187, Rome, Italy
Pertti Happonen, MD, MPH and Director, Finnish Medicines Agency, PO Box 55, 00034 Fimea, Finland
Wim G. Goettsch, PhD and Project Leader EUnetHTA WP5, Zorginstituut Nederland, PO Box 320, 1110 AH Diemen, The Netherlands & Pharmacoepidemiology and Clinical Pharmacology, Faculty of Science, Utrecht Institute for Pharmaceutical Sciences, P.O. Box 80082, 3508 TB, Utrecht The Netherlands
CONFLICTS OF INTEREST
Sarah Kleijnen, Iris Pasternack, Piia Peura, Anna Bucsics, Isabelle Zahra Pulis, Payam Abrishami, Sylvana Magrin Sammut, Luisa Anna Adele Muscolo, Simona Montilla, Pertti Happonen and Wim Goettsch report grants from the European Commission. Jenni Vuola reports personal fees from GlaxoSmithKline. The other authors report they have nothing to disclose.
