The health technology assessment (HTA) Core Model® is a tool developed by the European network for Health Technology Assessment (EUnetHTA) to define and standardize elements of a Health Technology Assessment (HTA) within several domains (Reference Lampe, Mäkelä, Velasco Garrido, Anttila, Autti-Rämö and Hicks1). It aims to facilitate, internationally, the production and use of HTA information and to provide a solution to the variance in the extent and scope of analyses as well as for the differences in reporting results. Use of the HTA Core Model®, hereafter referred to as “the Core Model,” may lead to more effective collaboration and sharing of information between HTA agencies in Europe (Reference Lampe, Mäkelä, Velasco Garrido, Anttila, Autti-Rämö and Hicks1;2). As a methodological framework for the production of HTA information, the Core Model consists of a standardized set of HTA questions that allows users to define their research questions within a hierarchical structure. Furthermore, it provides methodological guidance for answering HTA questions and a common reporting structure for presenting findings in standardized format.
The Core Model differentiates between nine domains of an HTA report. Each domain is further divided into topics and each topic consists of several generic questions, defined as issues that should be addressed in the process of HTA. The combination of a domain, a topic, and an issue is defined as an assessment element (2). An HTA report based solely on the Core Model does not generate policy recommendations but does provide a broad overview of relevant aspects and the usefulness of new healthcare technologies or interventions (3).
Although the Core Model has been developed to promote standardized approaches across Europe, and joint production and sharing of HTA information, national guidelines are still widely applied and to date only a small number of HTA agencies have implemented the Core Model (e.g., 4–7). Nonetheless, use of the Core Model may be particularly valuable for countries without established HTA structures and few resources. It enables them to foster the exchange of information in areas of common interest between agencies, to adapt HTAs produced by others to their own national context, and to improve national awareness of HTA (8).
Comparing the Core Model with national HTA reports may help to identify whether the model can be applied as designed, and whether it is appreciated and suitable for national HTA production. A comparison between the Core Model and national HTA reports further enables the identification of the concordance and similarities between these approaches. An HTA agency can make optimal use of the HTA Core information if the content of the model is in concordance with the content of a national report (Reference Pasternack, de Groot, Kleijnen and Polman9). To our knowledge, only one European HTA agency, The National Health Care Institute of the Netherlands, performed a pilot case study in which the Core Model was evaluated by comparing it with a Dutch national report to identify its adaptability. This study revealed that the assessment elements (AEs) of the Core Model were comprehensive, covering most relevant questions of the national report, and serving as a checklist for identifying and clarifying which questions would be significant for a national report (Reference Pasternack, de Groot, Kleijnen and Polman9).
One of EUnetHTA's activities is to promote and encourage member countries to use their products, including the Core Model, for national uptake. National uptake is the general implementation of any EUnetHTA output (i.e. joint assessments, submission templates, guidelines, databases, HTA Core Model, etc.) in a local setting (2). Given the interest in standardizing HTA reports within Europe to foster collaboration and prevent duplication of work, the aim of this study was to explore to what extent the AEs of the Core Model can be used for the production of national HTA reports in a small country with few HTA resources, such as Luxembourg. For this purpose, two sub-studies were conducted. First, a case study aimed to identify the relevant AEs of the Core Model and to assess the concordance between a national HTA report and the Core Model. Second, semi-structured interviews were conducted with agents from several European HTA agencies to assess their experiences with the use and implementation of the Core Model.
MATERIALS AND METHODS
National HTA Production
In Luxembourg, the Cellule d'expertise médicale (CEM) was created in 2010 by a reform of the national healthcare system; it is attached administratively to the General Inspectorate of Social Security at the Ministry of Social Security. The CEM provides health technology advice and supports health sector decisions on medical devices, procedures, or counseling with analyses guided by scientific evidence. The Ministry of Social Security, the Ministry of Health, the nomenclature commission, and the national healthcare insurance fund in Luxembourg may request advice from the CEM. By making use of methods for HTA, reports typically consist of a narrative literature review and the assessment of twelve aspects that are relevant for the national classification and tariff system (10;11). These aspects assure the consistency of tariffs in the nomenclature of medical benefits and differ from the assessments of other European HTA agencies and the Core Model (2;Reference Stephens, Handke and Doshi12;Reference Schwarzer and Siebert13).
Case Study
The Genetic Counseling report of the CEM was used as a case example for comparing its concordance with the Core Model (14). The Genetic Counseling report was selected because it was the most recently published report by the CEM written in English with an appropriate amount of information. Elements from the Core Model that were (a) considered to be relevant for the national report, and (b) actually in concordance with the content of the national report were extracted. For the purpose of the comparison, the national report was divided into two parts: (i) results of the narrative literature review, and (ii) analyses according to the twelve assessment criterias. The relevance of the 144 unique AEs of the full Core Model Version 2.1, applicable for all four health applications including diagnostic technologies, medical and surgical interventions, pharmaceuticals, and screening technologies (15), were evaluated (i.e., by defining per AE if it could potentially be answered according to the subject of the national report) and the number of all potentially relevant AEs for the national report were added up to calculate a percentage according to the following equation: %=number of relevant AEs*100/total number of AEs.
Next, the Core Model and the national report were thoroughly reviewed regarding their concordance in terms of content and distribution of information, as suggested by Pasternack et al. (Reference Pasternack, de Groot, Kleijnen and Polman9). The element cards of the HTA Core Model® 2.1 (15) were analyzed to identify the parallels within the content of the national report. Therefore, the descriptions of the topics and element cards within each domain were read and compared with regard to content and objective to the Genetic Counseling report to identify to what extent the AEs in the Core Model overlapped with the information provided in the national report. The report was carefully inspected topic by topic to identify the concordance with the domains and extract parallel AEs of the Core Model. An overall concordance figure between the national report and the 144 AEs of the Core Model was calculated as %=number of parallel AEs*100/total number of AEs. The comparison was validated by a second researcher.
Semi-structured Interviews
Study Design and Participants
A qualitative cross-sectional approach was chosen to gather data about experiences with the use of the Core Model of European HTA agencies. The format of semi-structured interviews was used because it enables rich data collection and further discussion with the interviewees (Reference Murphy, Dingwall, Greatbatch, Parker and Watson16;Reference Harrell and Bradley17).
The participants were selected from among the members of Work Package 4 of EUnetHTA Joint Action 2 from different European countries. All of the agency participants (n = 26) of Work Package 4 were contacted by e-mail and invited to participate in the study. Potential participants were informed about the method and content of the qualitative measurements performed in the study, including the topics and length of the interview. Six agencies responded and agreed to participate (16 percent). The interviews were carried out with one representative per HTA agency. However, one agency could not be interviewed because the representative could not participate within the timeframe of the interview. Semi-structured interviews were conducted in May 2015.
Measurement Instrument and Data Collection Procedure
A standardized questionnaire was developed for the interviews. The selection of topics was based on providing relevant information about experiences with the use of the Core Model. In total, the interview consisted of twenty-five questions, which were divided into three topics: (i) current HTA methodology and national HTA report, (ii) feasibility of Core Model for HTA production, and (iii) implementation of the Core Model at the national level. As is characteristic of semi-structured interviews, the interview questions did not strictly follow the order of the structure because new insights came up during the interviews and some answers provided information relevant to successive questions (Reference Harrell and Bradley17). The questionnaire was developed by three researchers and reviewed by one additional expert from the CEM.
The interviews were conducted by telephone and were designed to last 30–45 minutes. Before the start of the interviews, the participants were informed that the information they provided would be handled anonymously and confidentially. Permission for recording the interviews was asked beforehand. Each participant agreed with the recording of the interviews, which increased the reliability of the study, because the recordings were listened to a second time (Reference Golafshani18). Data saturation was reached when no additional new information could be gained, and when further coding was no longer feasible (Reference Fusch and Ness19). The decision on data saturation was also based on the repetition of information.
Data Analysis
The information gathered was coded and clustered according to the topics, and the main ideas were highlighted. The results were aggregated into a summary table. Anonymous participant quotations were used to illustrate the findings. To increase the reliability of the results, the summary tables were reviewed by a second researcher (based on the recordings) and validated by the respondents of the HTA agencies. For that purpose, each respondent received an email with the respective summary table to validate his/her answers.
RESULTS
Comparison of the National Report and the HTA Core Model®
The results of the comparison between the Genetic Counseling report and the Core Model revealed that 72 (50 percent) of 144 AEs were considered to be relevant for the national report (Table 1). Of these 72 relevant AEs, 34 (47 percent) were (at least partly) covered in the national report. Altogether, eight domains of the Core Model were presented in the national report, and 34 AEs (24 percent) were in concordance with its content, considering the total number of 144 AEs (Table 2). The highest numbers of relevant AEs were included in the domains of ethical analysis (ETH) (68 percent), costs and economic evaluation (ECO) (63 percent), and safety (SAF) (57 percent) (Table 1). However, the SAF domain was not covered in the national report, revealing no concordance in content. Figure 1 illustrates the results of the comparison.
Relevant Assessment Elements per HTA Core Model® Domain
Note. *Only unique assessment elements counted in total figure since ten assessment elements are double throughout different domains.
Examples of Correspondence between the HTA Core Model® and the Genetic Counseling Report
Note. The full correspondence between the HTA Core Model® and the Genetic Counseling report can be found in Supplementary Table 1.
T = topic; AC = assessment criterion required for the national report.
Illustration of the concordance between the assessment elements of the health technology assessment (HTA) Core Model® and topics and twelve assessment criteria of the national genetic counseling report. Note. The abbreviations of the domains and assessment elements of the HTA Core Model® and assessment criteria of the national report are explained in the Supplementary Tables 2 and 3.
Four AEs of the health problem and current use of the technology (CUR) domain of the Core Model were recognized in the Genetic Counseling report (Table 2). The report provided information about the population, usage, and reimbursement of the technology. For example, an AE in concordance was “For which health conditions and populations and for what purposes is the technology used”. The overview about development, purpose, and outcomes of the technology, as included in the national report, had concordance with several AEs of the description and technical characteristics of technology (TEC) domain. For example, location of the service delivery, that is, the first criterion of the national report, had concordance with the first AE describing the general features of the technology.
With regard to the clinical effectiveness (EFF) domain, the AE “What is the effect of the technology on generic health-related quality of life” was moderately in concordance with the outcome assessment of genetic counseling as described in the national report. Nine AEs of the ETH domain were extracted that had contents similar to topics described in the national report. Moreover, one AE of the organizational aspects (ORG) and four AEs of the social aspects (SOC) domain revealed parallel content with the national report, respectively. An AE of the legal aspects (LEG) domain on providing appropriate information to the user was also retrieved in the national report, but classified there as ethical analyses. Other AEs of the LEG domain were also, at least partly, covered. The national report proposed a coefficient that reflected resource usage, including time, technical competence, and intellectual effort required for genetic counseling. Due to a lack of data, no further information on economic impact of genetic services was provided. Thus, concordance was found with only two AEs of the Core Models’ ECO domain.
Semi-structured Interviews
Of the six HTA agencies that replied, five participated in the qualitative study; of these, two originated from different HTA organizations in Austria (the Ludwig Boltzmann Institut for Health Technology Assessment [LBI-HTA] and the Hauptverband der österreichischen Sozialversicherungsträger [HBV-SV]), one was from Finland (the Finnish Office for Health Technology [FinOHTA]), one from Luxembourg (CEM), and one from Estonia (the Centre for Health Technology Assessment, Department of Public Health, University of Tartu). The five agencies reflected the wide geographical reach of HTA agencies, and revealed variability in the answers. As data saturation was reached, no further interviews were conducted. The main findings regarding production of the national HTA reports and experiences with the Core Model are presented in Table 3.
Main Findings of the Interviews on the Experiences of Representatives of Five Different European HTA Agencies with the HTA Core Model®
Note. Further statements of the representatives of five different European HTA agencies on their experiences with the HTA Core Model® can be found in Supplementary Table 4.
National HTA Production
Four of the five HTA agencies that were interviewed developed their methodological guidelines internally. The reports are publicly available and, although usually produced in the mother tongue of the national HTA agency, often English abstracts are provided. CEM Luxembourg received the assessment criteria from the nomenclature commission and does not publish their reports. The representative from LBI-HTA Austria revealed that as of 2015, HTA reports are produced in English, supplemented with comprehensive summaries written in German. All the agencies mentioned that they assess nonpharmaceutical interventions only. The HTA requests mainly originate from the Ministry of Health or other governmental, healthcare, or patient organizations.
The majority of the interviewees agreed that the relevant evaluation topics depend on the technology under assessment. However, the two Austrian HTA agencies and FinOHTA considered the EFF and SAF domains to be most important and included them in every report. In addition, the HVB Austria usually reports information about disease, epidemiology, use of technology, and the differences between comparators. The representatives from the Estonian and Luxembourgish HTA agency reported that the economic and budget impact analyses are the most relevant issues in healthcare decision making. Most respondents agreed that the main limitation of the national report is the comparability of information amongst countries.
Experiences with the HTA Core Model® for National HTA Production
The findings of the interviews on the experiences of different European HTA agencies with the HTA Core Model® are presented in Table 3. As the representative from HBV Austria stated “there are two ways for national HTA agencies to use the Core Model – first by adapting already existing EUnetHTA reports and pilots. The other way is to take the Core Model tool and use it for the production of national reports”. Four HTA agencies have used the Core Model to some extent for national HTA production. The Estonian HTA agency admitted that there have been no attempts to put the Core Model into practice due to the difference in reporting structure, but instead they have used the information of EUnetHTA pilot HTAs.
The LBI-HTA Austria has performed rapid assessments, while FinOHTA has produced full HTA assessments using the Core Model. The respondents from HBV Austria and from FinOHTA emphasized the flexibility of the Model and the fact that not all the AEs are needed. All of the respondents agreed on the well-established scientific background of the Core Model and its advantage with regard to comparability and providing a framework for reporting. The representative from the Estonian HTA agency confirmed that due to the standardized format of the Core Model, information between similar topics can be exchanged, which may reduce the workload.
Only a few limitations were noted by the respondents. Although the Core Model is very comprehensive, some respondents found the number of AEs too high, making its use time-consuming. The representative of FinOHTA mentioned that irrelevant AEs can be skipped. The representative from CEM Luxembourg suggested that only a few AEs should be mandatory. Other limitations identified related to language and lack of human resources in the agency to (re-)produce full Core Model reports (in English). Most agencies mentioned that they produce reports in their national language and have so-called “HTA house rules”.
Moreover, learning to use the Core Model takes time and it might be difficult to adapt a new working mechanism and layout. Finally, according to the representative of LBI-HTA Austria, implementation of the Core Model may be hindered by decision makers. The change in language and new outline and content of the report can be confusing and it is questionable how useful the Core Model is from the decision-makers’ perspective. The representative from FinOHTA mentioned that the relevance of HTA in the decision-making processes is not yet widely acknowledged.
The facilitating factors of implementation were indicated as well. Each agency representative agreed that HTA production should not be politically influenced. The Core Model is useful to agencies for informing decision making with the best available evidence; using the Core Model generates objective and transparent HTA information (Reference Lampe, Mäkelä, Velasco Garrido, Anttila, Autti-Rämö and Hicks1;2). Most HTA agencies indicated that they are autonomous bodies without strict guidelines and outcome requirements set by authorities. The implementation of a new methodology is feasible if no formal framework has to be followed. Other facilitators mentioned were the willingness to collaborate with other organizations, sharing the work, and contributing to the development of more valid and transparent HTAs.
DISCUSSION
The results of the case study showed that half of the AEs of the Core Model were considered to be relevant for the national Genetic Counseling report. Of these relevant AEs, 47 percent were covered, at least in part in the national report. Accordingly, more AEs could have been included in the national report. The results can be considered a reliable indication, because the CEM Luxembourg uses a standardized approach for producing this type of national report.
Taking the National Health Care Institute of the Netherlands as a case example, the study by Pasternack et al. (Reference Pasternack, de Groot, Kleijnen and Polman9) revealed that nearly half of all AEs of the Core Model were considered to be of interest for the national report. The Dutch report actually included 64 percent of the Core Model AEs. The CEM report revealed lower concordance with the Core Model, indicating a larger discrepancy between the national approach for producing reports and the Core Model.
HTA has been defined as a multidisciplinary activity considering all aspects that might be influenced by the technology under investigation and those factors influencing the technology. As a result, the number of relevant AEs depends upon the technology under assessment and other context-specific factors (Reference Busse, Orvain, Velasco, Gürtner, Jørgensen and Jovell20). Perhaps one may question if the Core Model is suitable for healthcare requests that do not represent classical health technologies, but concern health services demands or negotiations. Moreover, the results reveal that more established HTA agencies are likely to possess and use more comprehensive HTA approaches than smaller agencies.
A study investigating the methods used in HTA indicated that the most frequent attributes evaluated in HTA reports are effectiveness, cost-effectiveness, safety, and quality of life (Reference Stephens, Handke and Doshi12). This phenomenon was also reflected in the present study, because the domains of clinical effectiveness, costs and economic evaluation, safety, and ethics included the highest percentages of AEs considered to be relevant. Although the safety domain revealed no parallel AEs with the national report, safety is a prerequisite for the uptake of medical acts in the Luxembourgish schedule of medical benefits. A survey by Lavis et al. (Reference Lavis, Wilson, Grimshaw, Haynes, Ouimet and Raina21) demonstrated that only 5 percent of HTA reports evaluated were full HTAs and addressed social, ethical, and organizational aspects; this raises the question whether an HTA is what it claims to be (Reference Velasco Garrido, Gerhardus, Røttingen and Busse22).
The interviews gathered information about experiences with use of the Core Model from different European HTA agencies. As confirmed by the interviewees of this study, national HTA guidelines vary among countries, impeding the sharing of information and the possibility of comparing results, thereby limiting the transferability of HTA information. By pursuing the missions and working procedures validated at the national level, the CEM differed most remarkably from the other agencies regarding their assessment approach. According to Hailey (Reference Hailey23), a standardized methodology helps to improve the usefulness and generalizability of HTA reports. More standardized and comprehensive reports would also allow decision makers to be better informed about the short- and long-term consequences of the application of a specific health technology (Reference Wagener24).
The interview results confirm the conclusion of Lo Scalzo et al. (Reference Lo Scalzo, Vicari, Corio, Perrini, Jefferson and Gillespie25), noting that the advantages of using the Core Model are the standardized procedure, the completeness for HTA production, and the possibility of collaborating with other agencies. Smaller and less experienced agencies can benefit from collaborating with larger, often more experienced agencies (Reference Lo Scalzo, Vicari, Corio, Perrini, Jefferson and Gillespie25). Nevertheless, some difficulties may occur with using the Core Model, resulting from its workload and comprehensiveness. As noted by several interviewees, the Core Model should be used in a flexible way, selecting only relevant AEs. This approach cannot be considered to be novel, but is indeed a confirmation and successful implementation of the methodology of the Core Model as defined in the HTA Core Model User Guide (26).
A recent study by Kristensen et al. (Reference Kristensen, Lampe, Wild, Cerbo, Goettsch and Becla27) is in line with our findings, stating that the Core Model can be used in a local, national, or international context to produce structured HTA information that can be taken forward by users into their own frameworks, to suit their specific needs for informing decisions on a technology under consideration. Furthermore, the findings of a Dutch study (Reference Kleijnen, Toenders, de Groot, Huic, George and Wieseler28) support this statement, adding that use of the AEs should be selective and easy to tailor to the needs of the national context. In line with the findings of the current study, the Dutch study revealed that, other than time and human resources, legal regulations and rules can hinder the process of adapting the Core Model. Furthermore, the structure of the reports produced using the Core Model might not be accepted by decision makers due to their length and comprehensiveness, revealing an important barrier for implementation (Reference Kleijnen, Toenders, de Groot, Huic, George and Wieseler28). Nevertheless, a study of Decision-Oriented HTA recently introduced a new mechanism that may help to reduce the gap between the results of the Core Model and the final decision making (Reference Ritrovato, Faggiano, Tedesco and Derrico29).
Limitations and Recommendations for Future Research
The current study is subject to various limitations. First, no analytical tool was applied to identify the concordance between Core Model Version 2.1, applicable for all four health applications, and the national report, and only one case example was used for the comparison. Therefore, the generalizability of the results to other reports of the CEM is limited. To be able to make more valid statements, a validated analytical tool and second or third national report may be used for comparison.
Second, the Core Model for screening technologies may have been used as an alternative for the comparison with the genetic counseling report and potentially this may have led to somewhat different results. However, we believe that we did perform the most comprehensive comparison.
Third, the current study applies to the previous version of the Core Model 2.1, because the most recent version of the Core Model 3.0 was not published at the time the comparison between the national report and the Core Model was performed. Therefore, all results of the assessment elements and the Core Model refer to Core Model 2.1. Future studies should be sure to use the newest version of the Core Model for their comparisons.
Fourth, although the comparison was validated by another researcher, researcher's bias may have been introduced. Researcher's bias may also have been introduced in the national report, because each member of the CEM has a specific expertise. Depending upon which expert is working on the report, its focus may change. To reduce this bias, it would be valuable to systematically include several required AEs from each domain to balance the information.
Fifth, a limited number of agencies participated in the study due to the limited timeframe available, and selection bias may have occurred, because there was no randomization in the selection of agencies and their representatives. Nevertheless, the agencies participating in the interviews represent a wide geographic reach and experiences with HTA production. Moreover, the agencies’ representatives were considered able to provide reliable and valid information on their experiences with and opinions of the Core Model. However, to increase the generalizability of the study findings, future research may include more than one representative per agency and more countries.
Sixth, to date only one similar study has been conducted in the Netherlands. Therefore, the current study was limited in comparing its results to existing findings, though its novelty was revealing. More agencies should be stimulated to investigate the applicability of the Core Model in producing their national reports.
Finally, as the study found that a significant number of AEs were relevant but not included in the national report, future research should investigate to what extent the relevant decision makers judge these AEs to be important and useful for the national HTA report.
Conclusion
The results of the study should be interpreted with caution because only one HTA report was used in the comparison with the Core Model. Nevertheless, the current study presents a critical attitude in comparison with the European average, providing valuable insights regarding the usefulness and adaptability of the Core Model. Although the Core Model helped to identify relevant subjects for the national report, the results of this study revealed relatively low though acceptable concordance between the HTA Core Model and the national report selected as a case example, revealing the absence of several HTA aspects.
Perhaps the Core Model performs less well with regard to healthcare demands that do not concern classical health technologies. However, following the best practices of HTA, clinical effectiveness, safety, and cost-effectiveness should be assessed systematically. The interviews with HTA agencies revealed that the Core Model can be used flexibly, and it is not necessary to answer all the AEs. Only the most relevant questions, depending on the topic under assessment and national context, should be selected and answered in the report. This will be more feasible for smaller less experienced HTA agencies. Many agencies struggle with the production of transparent reports that are of use to other agencies, indicating the need for further research on the national uptake and feasibility of the Core Model.
SUPPLEMENTARY MATERIAL
Supplementary Table 1: https://doi.org/10.1017/S026646231700099X
Supplementary Table 2: https://doi.org/10.1017/S026646231700099X
Supplementary Table 3: https://doi.org/10.1017/S026646231700099X
Supplementary Table 4: https://doi.org/10.1017/S026646231700099X
CONFLICTS OF INTEREST
The authors have nothing to disclose.
