Many healthcare systems have Health Technology Assessment (HTA) processes to inform choices regarding which services and products to pay for from public resources. Drug reimbursement decisions, which usually combine clinical and economic evidence with value judgments, are extremely important, not only to patients but also to manufacturers and healthcare professionals. Drug reimbursement processes have attracted attention from several authors. In the literature, one can find comparative analyses, that is, describing a number of different national models (indicatively Reference Franken, Le Polain, Koopmanschap and Cleemput1–Reference Oortwijn, Determann, Schiffers, Tan and Van der Tuin4), descriptive and comparative studies analyzing reimbursement decisions (Reference Fischer5–Reference Allen, Walker, Liberti and Salek13), and empirical analyses on HTA decisions (Reference Dakin, Devlin and Odeyemi14–Reference Maynou and Cairns18).
Despite the efforts of the European Union (EU) to promote voluntary cooperation among HTA agencies, reimbursement decisions for the same drug differ across European countries (e.g., 6;8;11;18). Diverging HTA decisions mean that a particular treatment is covered in one country and not in another. While these decisions can be justified from a policy perspective (well-established procedures), often in neighboring countries, this may be difficult to comprehend for the public (Reference Nicod and Kanavos8), particularly because the clinical evidence reviewed is largely the same, and countries, while not of equal wealth, are of broadly comparable levels of economic development. As a result, one might expect broadly similar decisions (positive or negative) on drug reimbursement to be taken. However, this is not the case (e.g., Reference Allen, Liberti, Walker and Salek6;Reference Nicod and Kanavos8;Reference Maynou and Cairns11;Reference Maynou and Cairns18).
To find out why these countries reach different conclusions, in our latest work (Reference Maynou and Cairns18), we made a methodological contribution to determining the factors that might lead to different drug reimbursement decisions on cancer drugs. Results showed significant associations between reimbursement decisions and a drug's estimated cost-effectiveness, the existence of a financial managed entry agreement (MEA), a health system based on social health insurance, the condition's incidence rate, and the socioeconomic characteristics of the country.
Following on our previous research, our aim in this paper is to compare the agreement of cancer drug reimbursement decisions in a sample of European countries using inter-rater reliability measures. The current study is based on primary data on cancer drug reimbursement decisions from nine European countries from 2002 to 2014: Belgium, Germany, France, Spain, Sweden, Portugal, Poland, England, and Scotland. This data set was collected as part of the ADVANCE-HTA project (Reference Maynou and Cairns11;Reference Maynou and Cairns18). To achieve our goal, we use two inter-rater reliability measures to analyze agreement or congruency: (i) percentage of agreement and (ii) the κ score. The inter-rater reliability is the degree of agreement among raters. It is a score of how much homogeneity or consensus exists in the decision given by various parties. In this analysis, rater refers to the HTA agencies or systems.
The HTA system's characteristics are the drivers of our findings. Owing to the nature of their system (for a review, see (Reference Maynou and Cairns18)), France, Germany, and Spain tend to accept all new drugs into the national system. Other European countries, such as England, Scotland, Poland, and Belgium, have high rates of restriction and rejection. Portugal and Sweden have acceptance rates similar to Spain. However, the Spanish system is different from the Portuguese and Swedish because it does not formally allow for rejection and restriction at the national level, but the different regions enjoy some freedom of implementation. The Spanish system introduced the Informe de Posición Terapeutico (IPT), which allows a restricted or rejected decision, at the end of 2013. However, our data pre-date its implementation. In England, until 2016, when a cancer drug reimbursement decision was negative, there was still the option to get it funded through the Cancer Drug Fund (CDF). From 2016 onwards, the National Institute for Health and Care Excellence (NICE) can recommend entry to the CDF. However, the adoption by the CDF is outside of the HTA process which we are studying.
Previous literature using inter-rater reliability measures to examine agreement among decisions finds, in most cases, low agreement. However, these studies only consider a few countries and/or few drug-indications and/or a short time period (Reference Allen, Liberti, Walker and Salek6;Reference Nicod and Kanavos8;Reference Nicod12;Reference Habl, Laschkolnig, Habimana, Stürzlinger, Röhrling and Bobek19). In this paper, we look at nine European countries and we compare decisions for 161 cancer drug-indications from the period 2002 to 2014. By examining just one therapeutic area, we make the decisions more comparable across the setting. This paper contributes to the HTA literature, shedding light on the weak to medium agreement across cancer drug reimbursement decisions in Europe.
The remainder of the paper is structured as follows. Section “Methods” describes the methods, including the definition of the database and the applied statistics. The results are presented in section “Results”, and the main findings are discussed in the final section.
Methods
Database
The current research is based on primary data on cancer drug reimbursement decisions from nine European countries from 2002 to 2014: Belgium, Germany, France, Spain, Sweden, Portugal, Poland, England, and Scotland. The database was restricted to these countries to ensure a formal HTA system and public availability of information. The Scottish Medicines Consortium (SMC) was the starting point of our study because it appraises all drugs approved by either the Medicine and Healthcare Product Regulatory Agency (MHRA) or the European Medicines Agency (EMA). The drugs selected were classified under “malignant disease and immunosuppression” on the SMC Web site. The list of drugs was validated by reviewing NICE decisions for any additional observations (especially on further indications for included drugs). This process produced eighty-one drugs and 161 drug-indications (to account for multiple cancer indications of included drugs, as reimbursement decisions may vary for different indications of the same drug). Thus, 161 cancer drug-indications were considered for each country. Cancer was selected because of the high level of public interest in these reimbursement decisions, and because many cancer drugs have been appraised thus providing a rich data set. Data was collected during the EU-funded project Advance-HTA (7th Framework Programme Grant No. 305983). The creation of the data set is explained in detail in two previous studies (11;18). Supplementary Table 1 presents the data sources.
This database contains the outcome of the decision and the date when the decision was published. The decision outcome describes the final decision regarding the adoption of the technology. We define the outcome as Non-Favorable, Favorable with restrictions, and Favorable. These are the three main categories of the HTA decision. To distinguish between “Favorable with restrictions” and “Favorable,” the decision is considered to be restricted only when it differs from the indication detailed in the marketing authorization (e.g., when reimbursement is limited to a subpopulation of the patients for whom the drug has been authorized). The rate of acceptance, restriction, and rejection for each year in each country is generated from the individual cancer drug decisions collected for our database.
In order to capture all possible decisions, we included another category in the decision variable: Nonassessment. This category can be the result of different circumstances, such as the manufacturer deciding not to apply for reimbursement or the decision maker not requiring assessment of the drug by the HTA agency.
In order to get robust results, we have to consider drug-reimbursement decisions which are comparable across countries. In other words, the different HTA systems in Europe incorporate different HTA decisions' definitions. While for Belgium, England, Poland, Portugal, Scotland, and Sweden, the outcome of the decision comprises all aspects of the reimbursement and it is used to accept or reject the new technology; for Germany and France, it just indicates the added value of the drug and is used to decide on the type of price agreement. Moreover, for Spain, the national government tends to accept all drugs but then, the regions can decide on the level of access. As a result, in the analysis, we compare across the six comparable countries and then consider the other three (Germany, France, and Spain).
Applied Statistics
We use two inter-rater reliability measures, the percent agreement and the Cohen's κ score (Reference Cohen20). The percent “agreement among raters” is measured by the number of agreed decisions over the total decisions. It has been widely used when measuring inter-rater reliability as it is easy to calculate and directly interpretable (e.g., 6;21). However, the percent agreement statistic does not take account of the possibility of chance agreement. So, it may overestimate the true agreement among raters (Reference McHugh21). Cohen (Reference Cohen20) suggests the possibility of a false agreement if the agreement is by chance. For instance, this chance agreement can arise if the HTA system tends to accept/restrict/reject all drugs, does not perform an in-depth assessment (e.g., only clinical assessment), or assessments are not comparable. However, McHugh (Reference McHugh21) states that if the raters (HTA systems/agencies) are well trained, the percent agreement is a realistic measure to determine interrater reliability.
The Cohen's κ coefficient is a statistic which measures inter-rater agreement for qualitative (categorical) items. It ranges from −1 to +1, where 0 represents the amount of agreement that can be expected from random chance, and 1 represents perfect agreement between the raters. While κ values below 0 are possible, Cohen notes they are unlikely in practice. If the κ score is negative, it means that the two observers agreed less than would be expected just by chance. As with all correlation statistics, the κ is a standardized value and thus is interpreted the same across multiple studies (Reference Cohen20;Reference McHugh21).
$${\rm \kappa } = \displaystyle{{\Pr \lpar a\rpar -\Pr \lpar e\rpar } \over {1-\Pr \lpar e\rpar }}$$where Pr(a) represents the actual observed agreement, and Pr(e) is the proportion of units which would be expected to agree by chance. The denominator, 1−Pr(e), reflects the maximum agreement beyond chance that would have been possible given the marginal distributions. Note that the sample size consists of the number of observations made across which raters are compared. The statistical significance of the κ coefficient is based on the χ2 distribution.
Cohen suggests the following interpretation of the κ coefficient (Reference McHugh21): values ≤0 as indicating no agreement and .01–.20 as none to slight, .21–.40 as fair, .41–.60 as moderate, .61–.80 as substantial, and .81–1.00 as almost perfect agreement. However, for healthcare research, McHugh (Reference McHugh21) argued that the standard interpretation of the κ score was too mild. For example, 60 percent agreement implies 40 percent faulty evidence. For healthcare decisions, 40 percent of the sample evaluations disagreed might be considered a serious quality problem. As a result, the author framed a stricter interpretation of κ values for health care: κ = 0–.20 no agreement; κ = .21–.39 minimal agreement; κ = .40–.59 weak agreement; κ = .60–.79 moderate agreement; κ = .80–.90 strong agreement; κ > .90 almost perfect agreement. While McHugh's (Reference McHugh21) threshold was defined in a clinical context, it shows that a stricter approach to the evidence is required. This can also hold true for national healthcare decision making, such as HTA decisions. In the results, both thresholds are used for the interpretation.
However, κ scores have three limitations or paradoxes (Reference Feinstein and Cicchetti22–Reference Flight and Julious24): (i) for high values of concordance, low values of κ can be recorded; (ii) asymmetric, imperfectly imbalanced tables have a higher κ than perfectly imbalanced and symmetric tables; and (iii) if the sample of assessed drug-indications is different across countries, it finds values that often lead to underestimation of the actual concordance present in the data. The first two paradoxes will mainly affect countries that have high congruency (in our case those that tend to accept all drugs). Regarding the third paradox, to overcome the limitation, a sample of drug-indications was selected that is common to NICE and SMC, two of the agencies showing a higher agreement (Reference Habl, Laschkolnig, Habimana, Stürzlinger, Röhrling and Bobek19).
Results
Descriptive statistics are presented in Table 1 and Figure 1. This information characterizes countries as restrictive or not restrictive, for example, a country that is not restrictive about giving access (Germany, France, Spain) or, conversely, a country that tends to restrict and reject access (England, Scotland, Poland, and Belgium). The Supplementary Figure 1 reports similar information to Figure 1 but only for the assessed drug-indications (i.e., removing the nonassessed category).
Descriptive Statistics: Variables of Interest (2002–2014)
a Mean (standard deviation).
b (max, min).
Source: ADVANCE-HTA dataset, created by the author.
Acceptance, restriction, and rejection rate + nonassessed rate (2002–2014).
Another indicator for the type of HTA system is the nonassessment rate variable. The rate of nonassessment does not change over time because it is the percentage of nonassessed drug-indications out of the 161 drugs for the entire period. In line with the previous variables, Germany, France, and Spain have a very low rate, assessing nearly all new drugs. Scotland also joins this group; even though it has a high rate of restriction and rejection, it assesses nearly all drugs. Sweden, Portugal, Poland, and England, in contrast, all have a high rate of nonassessment.
The following tables (Tables 2 and 3) show the results of the analysis. First of all, we take into account the whole sample (Tables 2 and 3), while in the Supplementary Tables 2 and 3, we reduce the sample to the drug-indications assessed by NICE and SMC. In these tables, we first present the less restrictive countries (France, Germany, and Spain) and then the rest. Results are nearly the same for both samples.
Percentage of Agreement for 161 Cancer Drug-indications (2002–2014)
Kappa Score for 161 Cancer Drug-indications (2002–2014)
For the interpretation of the results of the percent agreement (Table 2), we follow Allen et al. (Reference Allen, Liberti, Walker and Salek6) thresholds. For the first six countries (i.e., Belgium, England, Poland, Scotland, and Sweden), results show in most of the cases a low congruence, apart from Belgium–Portugal, Scotland–England, Sweden–Belgium, and Portugal–Sweden where there is medium congruence. For France, Germany, and Spain, we can see that there is high congruence among them. Portugal shows high agreements with France and Germany, due to the high rate of acceptance in Portugal. However, while for France, the number of assessed drug-indications is high, for Germany, it is low (i.e., due to data availability).
We present the results of the κ score for the whole sample of 161 cancer drug-indications (Table 3) and for the subsample of 80 drug-indications assessed by NICE and SMC (Supplementary Table 3). To interpret the κ coefficient, the McHugh threshold is stricter than the Cohen one. Table 3 is highlighted in colors based on the McHugh threshold but the legend also reports the Cohen threshold. Results show in most of the cases, there is none (McHugh threshold) to slight (Cohen threshold) agreement between countries. The closest agreement that we find, even if it is still considered weak to moderate, is between England and Scotland. There is minimal to fair agreement for Portugal–Belgium, Sweden–Belgium, and England–Poland.
Discussion
The main aim of this research was to compare the agreement of cancer drug reimbursement decisions in a sample of nine European countries (2002–2014) using two inter-rater reliability measures, the percentage of agreement and the κ score.
Our findings present mixed results. While for percentage of agreement, the overall results are a medium agreement among cancer drug decisions in the nine European countries analyzed, the κ score shows weak agreement. The percent agreement is easily calculated and directly interpretable. However, it does not take account of the possibility of chance agreement. So, it may overestimate the true agreement among raters (Reference McHugh21). While the κ score takes it into account, it has some limitations or paradoxes which may lower the estimate of agreement excessively (Reference McHugh21). Overall, taking into account both measures, there is a weak to medium agreement among cancer drug decisions in the European countries analyzed.
England and Scotland show higher consistency between the two measures, showing medium agreement. This result was expected because both countries have a rather similar notion of cost-effectiveness and the incremental cost-effectiveness ratio (ICER) partly (and importantly) determines their HTA bodies' decision, the cost structure of the health services are similar, and the comparators are usually the same. England and Scotland are followed by Portugal–Belgium and Sweden–Belgium but with less agreement. This result might partly be explained by a similar proportion of accepted, restricted, and rejected decision among these countries.
For the rest of the countries, there are two main reasons to explain the low agreement. First of all, agreement will depend on the assessing rates of the countries. Some countries assess by law all new drugs (Scotland, Spain, Germany, and France) while others have a different system in place (i.e., regarding who initiates the reimbursement process) or a prioritization policy and, as a result, they have a lower assessing rate. Secondly, the nature of the HTA system matters in the final reimbursement decision, for instance, the system does not allow for restriction and rejection decisions or there is less availability of MEAs (Reference Maynou and Cairns18).
Comparing the percentage of agreement and the κ score, one can see for some countries (Spain, France, and Germany) a high congruence using the percentage of agreement (Table 2) but a nonagreement when using the κ score (Table 3). The explanation for these findings is that as France and Spain tend to assess all drugs and accept nearly everything, when the κ is computed with respect to another country that also has a high rate of acceptance, such as Portugal or Sweden, most of the κ scores imply false or chance agreement, due to the system characteristics and the calculation of the κ score. Moreover, for Germany, we only had access to few decisions (21) and in all cases, they were positive. For this reason, the κ score is 0 or impossible to calculate. For these three countries, the κ score has a limitation as has been shown in previous literature (paradox 1), that is, for high values of concordance, low values of κ are recorded (Reference Feinstein and Cicchetti22–Reference Flight and Julious24). However, as explained above, the results for France, Spain, and Germany must be treated cautiously as they are not directly comparable to the other six HTA systems.
Our findings are in line with previous literature looking at the congruency of HTA decisions (Reference Allen, Liberti, Walker and Salek6;Reference Nicod and Kanavos8;Reference Nicod12;Reference Habl, Laschkolnig, Habimana, Stürzlinger, Röhrling and Bobek19). Allen et al. (Reference Allen, Liberti, Walker and Salek6) used the percentage of agreement and a taxonomy of systems to assess HTA decisions. However, results demonstrate less alignment between HTA recommendations and HTA process taxonomic sets. Nicod and Kanavos (Reference Nicod and Kanavos8), using the κ score, found that 46 percent of the drug-indication pairs studied (287) received diverging recommendations across countries (five countries) and that the level of agreement between agencies was poor to moderate. Nicod (Reference Nicod12), while looking only at ten orphan drugs, found that six out of ten drugs received diverging HTA recommendations. Habl et al. (Reference Habl, Laschkolnig, Habimana, Stürzlinger, Röhrling and Bobek19) also used the κ score and showed a better level of agreement in assessing health technologies in a pharmaceutical sample (20 drugs) than in a medical devices and other technologies sample. So, while most of the previous studies found low to moderate agreement among HTA decisions, results differ depending on the drug sample size, the number of countries, and the therapeutic area. In our analysis, we focused on one therapeutic area (i.e., to make it comparable), on 161 cancer drug-indication pairs, nine countries (taking into account two main sets), and a longer period of analysis 2002–2014. This paper contributes to the HTA literature, by highlighting the extent of weak to medium agreement among cancer decisions in Europe.
During the recent years, there has been much effort at the European level to promote voluntary cooperation among the HTA agencies. In 2004, EUnetHTA was created and there is currently a new proposal for regulation of HTA being discussed in the EU Parliament and the Council (25). The proposed regulation aims to promote the convergence of methods, tools, assessments, and cooperation among HTA agencies based on a joint clinical assessment. The main objective is to reduce duplication and improve the efficiency of the HTA systems. However, even if the EU is trying to have binding decisions at the European level, the countries/regions are still concerned about their own competencies. So, despite substantial efforts at the EU level to promote voluntary cooperation and sharing of assessment, HTA decisions for cancer drugs differ across the EU countries analyzed. These diverging decisions can be justified from a policy perspective (as they are based on well-established procedures, from a patient or societal perspective); however, differential access to a particular treatment across neighboring countries may be difficult to understand for the public, particularly in countries with comparable levels of healthcare spending and similar levels of income (Reference Nicod and Kanavos8). However, further research is needed to determine the effect of this differential access on health outcomes.
Although the results are satisfactory, during this research, we encountered a number of challenges. One limitation is related to data collection and availability. For instance, for Germany, we only had access to few decisions. Moreover, to combine the data from each country in a single analysis, some assumptions were needed [i.e., details on the assumption can be found in (Reference Maynou and Cairns18)]. Regarding the methods, while the κ score is a standard inter-rater reliability measure, it has several limitations. According to McHugh (Reference McHugh21), the Cohen threshold for the κ score in health care is too mild. We overcome this limitation by using both thresholds in the results interpretation.
In this study, we have only looked at cancer drugs. Future research should also compare across therapeutic areas, in order to explore the level of agreement among decisions.
Acknowledgments
This paper was developed with data collected during the research project funded by the European Commission's Research Framework Programme (FP7), Advance-HTA (no305,983). The authors would like to thank the Advance-HTA Consortium members and the experts from the National HTA Agencies for their support on the data collection process. Moreover, the authors appreciate the good feedback received in the different conferences where this piece of research was presented (AES and Spanish HTA meeting). The authors would also like to thank you Dr. Carlos Campillo for his feedback on the paper.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S026646232000032X.
