Health technology assessment (HTA) was developed to inform health policy making in the 1970s, although it had been practiced in various forms for decades (1). In the 1990s, HTA began to be used for decisions on reimbursement, funding, and coverage of medicines and medical devices, which raised its profile (2). Since then, the number of agencies conducting HTA has been increasing worldwide (3;4). Such national HTA agencies can make recommendations to Ministries of Health or health insurance companies about the suitability of health technologies for reimbursement.
As individual national HTA programs often assess the same topics at similar times, international cooperation has been sought by HTA agencies (Reference Hailey3). Many believe that collaboration between agencies or programs is likely to reduce unnecessary duplication of activities by both agencies and health technology manufacturers, enable efficient sharing of expertise and information, lead to faster access for patients to new health technologies and advance the field of HTA (Reference Hailey3—Reference Kleijnen, George and Goulden8). This may also allow countries to more easily meet the Transparency Directive, which stipulates time limits for decisions on reimbursement for new technologies. However, there are also some disadvantages to such collaborations, including the potential lack of sensitivity to local context (Reference Hutton, Trueman and Facey2).
Despite a long history of enthusiasm for collaboration and publication of international HTA methodologies, practical examples of bringing together national HTA processes and methods are mostly recent (Reference Pasternack, Anttila and Mäkelä9;Reference Lampe, Mäkelä and Garrido10). Structured approaches to facilitate collaborative working have been pursued as part of the European Network for Health Technology Assessment (EUnetHTA) project and joint actions (11). EUnetHTA supports collaboration between European HTA organizations, and aims to facilitate an efficient use of resources available for HTA as well as mechanisms for knowledge sharing between organizations. To this end, EUnetHTA have developed the HTA Core model that includes a set of generic questions (referred to as issues or assessment elements) ordered within “topics” and “domains” that together define the contents of an HTA.
Many national processes for reimbursement or funding decisions include evidence submissions by the manufacturer of a technology, followed by an independent assessment of that evidence submission. Collaborative working between HTA organizations would be facilitated by a manufacturer's submission template that contains the evidence requirements of all European countries. Availability of such a template that is applicable to more than one country could also create efficiencies within industry.
Previous work by EUnetHTA explored the similarities and differences in the major methodological aspects of Relative Effectiveness Assessment (REA) in many European countries and was based on a survey with thirty-eight open or multiple-choice questions (Reference Kleijnen, George and Goulden8). The work presented in this study was undertaken as part of work package 7 in EUnetHTA Joint Action 2, which aims to develop a manufacturer submission template that brings together evidence requirements from all European countries. This study adds to the existing evidence base as it collects, analyses, and compares all of the specific evidence requirements for the HTA of pharmaceuticals from individual European national agencies responsible for reimbursement.
METHODS
Collection of Evidence Requirements from National Agencies
Evidence requirements were requested from national agencies making reimbursement decisions for pharmaceuticals (Supplementary Table 1). Some countries use the same evidence requirements for all reimbursement decisions. For this reason the term “health technology” is used throughout the results section. The documents used in the analysis included templates, application forms, and checklists of information to include in submissions. In instances where agencies had different templates and application forms for different types of pharmaceutical, for example new active substances versus agents of the same class and generic products, the documents for new active substances were used. Manuals outlining the process of HTA, reimbursement criteria or the general methods of completing an HTA were excluded from the analysis except where these documents were specifically associated with how to fulfil the evidence requirements.
Evidence requirements for HTA, or manufacturer's submission templates, were requested from the national agencies in English. Where English versions or translations were not available, these were generated by the authors of this article. Where possible, these generated English translations were cross-checked by the respective national agency, followed by an iterative process of clarification, until the best understanding of the information requested was available. If it was not feasible to produce English translations within the timeframe of the project or finalize the clarification process, translations of headings only were requested and used.
Data Extraction
Data were extracted by five EUnetHTA partners (Agenzia Italiana Del Farmaco [AIFA, Italy], College voor zorgverzekeringen [CVZ, The Netherlands—note that this organization will be called ZorgInstituut Nederland from April 2014]], National Institute for Quality and Organizational Development in Healthcare and Medicines [GYEMSZI, Hungary], Ministry of Health [Czech Republic], and the National Institute for Health and Care Excellence [NICE, United Kingdom]). A framework was developed to extract the information requested by the national HTA agencies into a platform that enabled comparison. This framework was based around the first four domains of the CORE HTA model: the description of the health condition, description of the technology, clinical effectiveness and safety (assessment elements version 1.1) (Reference Lampe, Mäkelä and Garrido10).
A sample of submission templates from England (NICE), France (Haute Autorite de Sante (HAS), Italy (AIFA), and the Netherlands (CVZ) was used to pilot extraction of evidence requirements using the CORE model topic and domain headings. Where necessary, the framework was amended to allow the inclusion of information requested by national agencies which were not covered in the CORE HTA model, or to merge aspects of the CORE model where distinctions between these aspects were not identified in the national submission templates. This amended framework was used to extract information from all available national evidence requirements.
Evidence requirements were extracted for each domain independently. Quality assurance was subsequently performed in which the information extracted for a country was checked against the respective submission template by at least one of the study authors who had not been involved in the original data extraction. Reconciliation was performed by two researchers, in which amendments made during quality assurance were checked and accepted or debated until consensus was reached.
Analysis
Evidence requirements were examined by domain and similarities and differences in the information requested were identified.
RESULTS
Evidence Requirements Received
Evidence requirements were requested from thirty-three countries in February 2013. After up to two follow-up requests, thirty-one countries responded and twenty-nine countries provided evidence requirements, usually in the form of a manufacturer submission template, by June 2013. The English versions from twenty-nine countries (88 percent of all countries contacted) were sufficiently finalized to form a basis for the analyses. Details of the evidence requirements examined are show in Supplementary Table 1.
General Findings
The amount and type of information provided for the analysis by the national agencies (hereafter referred to as countries) differed considerably. A crude indication of the difference is the number of pages of the template or application form received. These ranged from a one-page application form with enclosed documents (Denmark) to a 174 page submission template (Austria) (Supplementary Table 1).
Countries also provided variable amounts of guidance for completing their submission templates. Some countries did not provide any guidance for completing their templates (e.g., Malta, Luxembourg). Some countries included guidance within the submission template document (e.g., the Netherlands). Other countries provided guidance documents, containing information on how to complete the template, these documents ranged from six pages (Russia) to forty-eight pages (Scotland) (Supplementary Table 1).
The detail of information in the evidence requirements varies. Some countries ask general questions, with some giving an indication of the minimum information acceptable in the response, while others specify the exact information required from the manufacturer. The following provides an example using the topic of the disease targeted by the health technology under consideration:
France (general) “Describe in brief the disease targeted by the indications covered by the application”;
Belgium (specific) “Presentation of the pathology… the clinical context must clearly be detailed: -acute or chronic disease; seriousness of disease, in terms of mortality, morbidity and/or quality of life related to state of health. Epidemiologic data… The information with regard to prevalence and prevalence per subgroup, the impact and impact per subgroup… the prognostic factors depending on the various stages of the disease and the high-risk groups as well as the risk factors will be detailed”
Ireland (general + specific) “Provide an overview of the clinical condition. Include standard diagnostic criteria/testing devices where appropriate. Disease classification (define subclasses where necessary and relevant).”
Health Problem
All twenty-nine countries ask for some information on the health problem targeted by the health technology. As described above, some of this may be general questions, but where countries specify information to be included, these data were extracted.
Five of the thirteen information requirements on the topic of the health problem were generic and could apply across countries (Table 1). All countries requested that the target disease be identified in the application either by ICD-10 code, the indication as listed in regulatory information or chosen from a pre-specified list. Nine countries ask specifically about the natural course of the disease, symptoms or consequences for patients. Causes or risk factors for the target disease are requested by four countries.
Questions Included in European Manufacturer Submission Templates by Domain Classified as Relevant Internationally (Generic) or Specific to National Context
b This information may be included in an EMA, in which case this may be internationally relevant.
a Every country asked at least one question on each domain.
Seventeen countries are interested in the reimbursement status elsewhere. Seven ask about reimbursement status in the countries that make up the European Union specifically, while others ask about all countries in Europe or fewer specified countries. Turkey asks about the status across the OECD countries.
Some countries request nationally-specific information under this domain (Table 1). This includes target disease epidemiology (e.g., Estonia requests “number of patients in Estonia per year”). It also includes information on the diagnosis and current management of the target disease, where occasionally countries ask for generic information, but more often this is country specific. Examples include:
France (generic): “Describe the different treatments available. Also describe non-medicine based management techniques and their therapeutic use”
Croatia (nationally-specific): “Current clinical practice in the Republic of Croatia”
England (nationally-specific): “Please give details of any relevant NICE guidance or protocols for the condition for which the technology is being used.”
Other questions asked, for which national context is important, include defining a comparator to the technology being assessed, information on the target population (asked by twenty-six and twenty-five countries, respectively) and questions about the current use of the technology (asked by nine countries).
All countries include an explicit request for regulatory information, this may be in the form of a request for regulatory documents such as the Summary of Product Characteristics (SPC) and European Public Assessment Report (EPAR) to be appended, for information from the marketing authorization or for the reference code of the marketing authorization. Other regulatory information requested includes the US Food and Drug Administration assessment report, patent details, information on international status (including rejections) and information on co-marketing (i.e., where several pharmaceutical companies are selling medicinal products based on the same active ingredient, using different trade names).
Twelve countries request information which seems to be asking the applicant to describe unmet needs or to make a case for the need to introduce the technology, for example:
Denmark: “Overall health economic consequences of the disease”
Estonia: “Need for the service”
Finland: “A well-grounded proposal for the necessity of the medicinal product”;
France: “Describe the unmet needs”
Germany: “What therapeutic needs exist beyond the available treatment opportunities?”
Scotland: “The rationale for the development of the new product, indication or formulation, including perceived gaps in therapy”
Switzerland: “Expert summary of the importance of the disease area”
In summary, a total of five questions were identified for this domain for which answers are generic and applicable across borders, eight questions were identified that require national context (Table 1).
Description of the Technology
All twenty-nine countries ask for some information pertaining to description of the technology. Eighteen countries request the active ingredient (alternatively described as “active substance,” “drug substance,” or “therapeutic substance”). A range of codes are used to identify the substance including the Anatomical Therapeutic Chemical Classification System Code (ATC) (the most frequently requested, by nineteen countries), Nordic Article Number (VNR), State Institute for Drug Control number (SUKL), Pharma-Zentral-Nummer (PZN), and Codes Identifiants de Presentation (CIP). Other generic information requested includes the form of the pharmaceutical (described as “galenic form” “dosage form” “pharmaceutical form” and “administration mode”), its mechanism of action, packaging, dosing schedule (with eight countries specifically requesting DDD), pharmacokinetics, and just two countries, Austria and Croatia, ask for pharmacodynamics (Table 1).
Nine countries ask about how innovative the pharmaceutical is. Examples include:
France: “Mention special features of the application; where appropriate mention the originality of the medicine”
Malta: “Comparison of this medicinal product to other medicines on the Government Formulary List in terms of innovation”.
Three countries ask about the origin of the product. Luxembourg asks where the medicine comes from; Russia asks country of origin and production site; Turkey asks the place of manufacture and details about importation.
Although several countries request the SPC which may include requirements for use of the pharmaceutical, some countries also specifically request information on investments, training or additional resources that are necessary for use of the technology. Some of this may be generic information, for example: tests and investigations or additional therapies required by patients, or patient monitoring that must occur with use of the technology. However, some of the information requested may be dependent on national context.
Information that may be dependent on national context includes information on the place or context of use of the technology. Eleven countries ask whether the technology is only for use within the hospital setting; for example Belgium asks “reimbursement solely in hospital setting or also in an ambulatory setting”; Lithuania asks if treatment is in an inpatient setting or an outpatient setting. Other countries ask for other specific information on the place or context, examples include:
England: “Describe the location of care” and “Does the technology require additional infrastructure to be put in place”.
Estonia: “Location or space needed to provide the Service (ward, procedure room, operation room, other)” and “Need for additional infrastructure, specialized departments/service units, etc.”
Two countries ask about training and information needed to use the technology. Ten countries ask about who will apply the technology. The range of questions include: whether “a ‘prescription limitation’ restricting the right to prescribe the medicine only to a specialist (e.g., oncologist, endocrinologist, etc.)” is required (Czech Republic) and “describe staff usage” (England). Answers to such questions may be dependent on the structure of health services in a particular country.
In total, for this domain, nine questions were identified for which answers would be generically applicable across Europe and seven questions were identified for which national context is important.
Clinical Evidence of Efficacy and Effectiveness
All countries ask for some clinical information about the technology under review. However, the amount of information requested varies, for example, Luxembourg requests a comparative study (if available), Switzerland requests three studies, whereas Belgium requests a complete list of the evidence for the technology. Lithuania appears to request clinical effectiveness information in the context of pharmacoeconomic analyses only. Almost all of the information requested by countries could be considered generic (Table 1).
Study Identification
Of the twenty-eight countries asking for evidence on clinical effectiveness information, eleven request that the process of study identification is recorded, for one of these countries this is for the comparator only (Scotland), and for two of these countries it is only for the data informing the health economic analyses (Belgium, Denmark).
Nine countries request that the databases searched were recorded with six of these specifying at least some of the databases which must be searched. Belgium, England, Germany, and Norway request that the applicant searches Medline, Embase, and the Cochrane library. Croatia also requests that applicants search Medline and the Cochrane library, and also TRIP. Hungary requests that the applicant provides the references from a search of Medline. Only five countries explicitly request that a flow chart of study inclusion and exclusion be presented.
Eight countries ask for details of on-going studies, for example Scotland asks applicants to provide details of ongoing studies that would provide additional evidence within the next 6–12 months; France asks “Are there any work/studies in progress that are likely to give rise to an application or extension of indication in the next few years?”. Three countries specify that unpublished studies should be identified.
Study Description
The majority of countries ask for some descriptive information about each study considered in the submission, to be recorded in the submission. In the documents we reviewed, study level information was not requested by Hungary, the Czech Republic, Luxembourg, or Portugal.
Where countries ask for a description of identified studies the majority ask specific questions, however, Denmark, Finland, Hungary, and Sweden ask for a summary without any specific information requested. Only Finland asks about conflicts of interest for individual studies, although two countries (Italy and Poland) ask about source of funding of identified studies and Germany asks studies sponsored by the manufacturer to be listed which may highlight a conflict of interest.
Although fifteen countries ask for some form of critical appraisal of studies to be presented, the specific requirements vary considerably. Some require a full critical appraisal with methodology specified (e.g., Germany), while some make the general request to do a critical appraisal (e.g., Norway) or request that articles are ranked to highlight the most important which may involve some assessment of quality (e.g., Finland). There is also the request to rank studies in terms of “level of evidence” from several countries for example the Netherlands and France rather than an assessment of individual study quality.
Of the countries asking for individual level study results, Croatia also asks about the results of omitted studies, England asks if results are based on an intention to treat analysis (which is preferred) and Scotland suggests that graphical representations are particularly helpful.
Study Synthesis and Conclusions
Eleven countries ask for results of identified studies to be synthesized (for three of these countries this is in the context of data included in the health economic evaluation only), and a further nine countries ask for a clinical effectiveness summary which may include synthesis although this is not explicitly requested. The majority of the countries that ask for a synthesis request meta-analysis, if possible, or indirect or mixed treatment comparisons where direct comparisons are not available. Poland and England require a narrative review if studies are not suitable for meta-analysis. Nine countries ask explicitly for subgroup analyses.
Only four questions identified under this domain were considered to require a nationally-specific answer. These are a request for the external validity of the clinical effectiveness data presented (asked by twelve countries) with the other three questions relating to regulatory information. However, although most of the information presented in this section could be internationally relevant, study types that are acceptable to different countries vary (Table 2). For this reason, the results feeding into analyses of clinical effectiveness may not be the same for all countries.
Study Types Specified in National Submission Templates as Acceptable for Clinical Effectiveness and Safety Data
Evidence on Safety
Five countries specify that additional data sources could or should be used for consideration of safety data which may be different from the sources used for consideration of clinical effectiveness data (Table 2). These were: England, where further data from non-comparative trials, for example, post-marketing surveillance data can be used; Ireland, who allow case reports, observational studies or controlled trials; Italy, who ask for periodical reports (i.e., Periodic Safety Update Reports- PSURs), occasional ADRs reports or pharmacoepidemiological studies; Luxembourg who request a drug safety sheet; and Poland, who request case series analyses and patient registers.
Twenty-eight countries ask for patient safety data separate from pharmaco-economic analyses. Three countries ask about safety risk management and seven countries ask about any actions that have been taken as a result of regulatory safety updates. External validity of safety conclusions, safety risk management, and regulatory actions are considered to require national context, and the methodology used for answering the nationally relevant questions is currently different across countries.
DISCUSSION
We examined the evidence requirements for twenty-nine European agencies making national reimbursement decisions for pharmaceutical technologies. We found twenty-six evidence requirements for which generic answers could apply across borders and nineteen for which countries require nationally specific information.
The level of detail in the required information varies considerably across countries. Some countries ask specific questions while others request information under general headings. Some countries ask for all information in a single document, which may or may not include guidance for answering questions. Others provide specific guidance documents, or have methods and process manuals or handbooks that may be used to guide the development of the submission.
There were no evidence requirements which every country specifically requested in the same way. For example, all countries asked for regulatory information, but this was sometimes in the form of regulatory documents, extracts from these documents or for the reference numbers for the documents. In addition, there were twenty-one items of information which fewer than one-third of countries specifically requested. For some of these rarer items, it was not clear how the information would feed into decision making on reimbursement, for example “place of manufacture” or “pharmacokinetics.” On the other hand, some elements that might be considered good practice when conducting HTA, such as requesting information on conflicts of interest, and scale validation for any measures used in the studies presented were requested only by a few countries. Ireland is the only country to ask about publication bias to help interpret the results of study synthesis. Of course, these elements were only recorded where they were requested explicitly. Countries with a general question on study quality or who requested clinical study reports or published studies, may receive this kind of information or it may be implicitly expected. Other countries may use the manufacturer submission as the basis for their own independent assessment and, therefore, make such judgement of quality and bias themselves rather than requesting manufacturers do so.
An important finding was that most countries make no distinction between the evidence that provides data for clinical effectiveness and that which provides data for safety, and most use the same methods of identifying studies that will contribute information for clinical effectiveness and safety.
For some common questions, the context in which the questions are asked varies. For example, some countries ask as background information about the characteristics of the target population and its size. Other countries ask for information about target population size specifically in the context of calculating budget impact and a small number of countries either do not request information on target population size, or ask only a general question about incidence and prevalence of the disorder under consideration which may or may not fully reflect the target population under assessment. This could imply that this information is not necessary for decision making for reimbursement in some countries, and that even where such information appears to be necessary it may be used differently in decision making contexts. This will have implications for developing common submission template tools relevant to a range of countries.
The main strength of this study is that it included an almost complete set of the current European national evidence requirements for pharmaceutical reimbursement decision making. Limitations include difficulties in comprehension due to differences in terminology used in different countries or change in meaning in the translation to English, which may have affected the accuracy of some data extraction; inability to extract what is implicitly expected but not explicitly requested; and where there were several guidance documents, or particularly large guidance documents, it was sometimes unclear when to include information from these documents, and if so, how much.
The current analysis focused on evidence requirements related to relative effectiveness, and we have therefore not included items related to cost, cost-effectiveness or pharmaco-economic data from the national evidence requirements. The role that cost or cost-effectiveness related information plays in national decision making varies considerably across Europe. Some questions related to cost (e.g., on requirements for use, dosing or packaging) are being asked by countries that do not request formal cost-effectiveness or pharmaco-economic data, and may be used after the relative effectiveness assessment at the stage of pricing negotiations. Furthermore, due to concentrating on relative effectiveness assessment, we considered the evidence requirements for first evaluation of a pharmaceutical only. This means that in countries where evidence requirements for other types of product (e.g., new products of the same class or generic medicines) differed, these are not included in our analysis. Further analysis is needed, and planned within the EUnetHTA Joint Action 2, to compile these respective cost-related evidence requirements and explore if these can be brought together or otherwise configured in a common, potentially modular submission template.
The final limitation of the current work is that the authors' organizations were able to provide more information about their own evidence requirements which may have led to more complete interpretation of evidence requirements from the Czech Republic, England, Hungary, Italy, and the Netherlands.
Differences in terminology between submissions may reflect differences in evidence requirements, differences in translation, or nuances may have got lost translation. For example, terminology related to safety included the phrases “unintended effects,” “adverse effects,” “side effects,” “safety,” and “adverse drug reactions.” Terminology used for summarizing beneficial effects varied also, including “therapeutic value,” “medical benefit,” and “clinical effectiveness.” Several glossaries have been produced, and an agreement on preferred terminology, would help to provide mutual understanding when working across jurisdictions on HTA.
Kleijnen et al. (Reference Kleijnen, George and Goulden8) collected data from a similarly comprehensive number of European countries and focused on some methodological aspects of relative effectiveness assessment, such as the definition of comparator, and the preferred outcomes. The authors concluded that there are more similarities than differences between the major methodological aspects across countries, but pointed out that harmonization of methods and best practices for REA between jurisdictions is not identical to harmonization of market access in various jurisdictions. Furthermore, Kleijnen et al. (Reference Kleijnen, George and Goulden8) pointed out that this does not diminish the benefits of harmonizing REA because of the potential efficiency gains.
The work in this study suggests that it would be possible to put together a dataset of evidence requirements for REA to support reimbursement decisions across Europe. A range of benefits of collaboration in HTA processes internationally has been previously proposed, from more efficient use of analytical resources to faster and more appropriate reimbursement decision making. Benefits for manufacturers would be similar. An advantage of a common submission template that lays out clearly both the common evidence requirements and the ones requiring national contextualization is that it allows a reduction in duplication of effort while safeguarding the proper analysis of contextual factors necessary for national decision-making.
Some of the differences in evidence requirements between countries in Europe are justified as necessary for context-specific analysis or to address the criteria for decision making. Others may simply be the result of historical development of the health services, their politics and their funding. Justifiable differences that require adaptation to health care systems and settings, and methodological aspects independent of setting and health care system have been discussed before (Reference Mathes, Jacobs, Morfeld and Pieper4). We have identified that over half of the evidence requirements may be have answers which would be applicable to all countries in Europe, whereas a sizable minority of questions require nationally-specific answers.
The aim of this particular activity within the EUnetHTA Joint Action 2 is to develop a submission template that covers the evidence requirements across Europe, so it could be used to support national HTA processes in any European country, and also, where appropriate, joint assessments shared between several countries in the future. This piece of work, which collected, analyzed and compared the evidence requirements for pharmaceuticals in twenty-seven countries, was a first––and promising–step toward this aim.
Competing Interests Statement
A. Cangini and L.A.A. Muscolo work for a medicines regulatory agency (AIFA–Italy). The authors have declared that no other competing interests exist, and no specific funding was received for writing this article. The views expressed in this article are the personal views of the authors and may not be understood or quoted as being made on behalf of or reflecting the position of the regulatory agency or any of the committees or working parties of the regulatory agencies the authors work for.
SUPPLEMENTARY MATERIAL
Supplementary Table 1 http://dx.doi.org/10.1017/S0266462315000227
CONFLICTS OF INTEREST
Authors report nothing to disclose.
