Healthcare policy makers decide on the diffusion of healthcare interventions and reimbursement of their costs on behalf of society, often relying on the results of economic evaluation studies to do so. Health technology assessment (HTA) is a fairly young science and, while the methodology of economic evaluation studies has gradually matured in the past decade, the systematic incorporation of economic evaluation results into policy making is still too recent to fully understand how policy makers handle the multidimensional information they offer. Do decision makers take all dimensions reported in studies into account? Under what conditions do health gains or equity concerns outweigh other dimensions? What is the relative weight decision makers attach to productivity costs and uncertainty regarding the cost-effectiveness ratio? Answering such questions promotes transparency of the decision-making procedure and enables researchers to collect relevant decision-making data.
To our knowledge, only three articles have explored the explanation of past reimbursement decisions: two quantitative analyses focusing on guidance produced by National Institute for Health and Clinical Excellence (NICE) in the United Kingdom (Reference Dakin, Devlin and Odeyemi5;Reference Devlin and Parkin6) and one qualitative study of Dutch reimbursement decisions (Reference Pronk and Bonsel12). Each confirmed that cost-effectiveness is neither the only nor the dominant concern. Cost-effectiveness, clinical evidence, uncertainty, budget impact, the burden of disease, and technology type explained NICE's funding decisions better than cost-effectiveness alone (Reference Dakin, Devlin and Odeyemi5;Reference Devlin and Parkin6). Pronk and Bonsel studied Dutch drug reimbursement decisions during the precompulsory pharmacoeconomic dossier period of 1999–2002 and concluded that budget impact, therapeutic value, and burden of disease were prominent criteria (Reference Pronk and Bonsel12).
These studies are revealed-preference analyses, suggesting high validity. A limitation is that not all (potentially) influential decision-making factors can be studied because data available for current analysis derive from historical case characteristics. To enable prediction of future decisions, a wider range of concerns may need to be studied. This calls for stated-preference data.
Our study builds on existing preference studies among policy makers or the general public (Reference Al, Feenstra and Brouwer1;Reference Al, Feenstra and van Hout2;Reference Baltussen, Ten Asbroek and Koolman3;Reference Green and Gerard7;Reference Gyrd-Hansen8;Reference Ratcliffe, Bekker and Dolan14;Reference Strasmann15), exploring hypothetical decisions in a Dutch policy-making context, which differs from other countries in that, since 2001, policy documents about reimbursement decisions have promoted the idea of varying the cost-effectiveness threshold with disease severity (4). A definition of disease severity was put forward to foster clarity (Reference Stolk, van Donselaar and Brouwer16). This policy model responded to the observation that policy makers were unable to fix a CE threshold. It would be interesting to see if this model held in practice and to identify the criteria that lay next to cost-effectiveness and disease severity.
We shed light on this matter by asking policy makers and other healthcare professionals to make choices between healthcare programs with divergent economic evaluation results. Our research questions are: (i) What criteria are important for respondents in healthcare priority setting? (ii) To what extent do respondents make tradeoffs between these criteria? (iii) Do our respondents take a societal perspective, as advocated in the literature? For example, economic theory would exclude budget impact as a decision criterion (a dollar is a dollar).
We sought preliminary answers using a mixed method design. First, we performed a discrete choice experiment (DCE) for Dutch healthcare professionals (policy makers, HTA experts, advanced HTA students) faced with priority setting on the national level. For the policy makers, we subsequently held a focus group discussion on the experiment. A DCE allows the study of preferences while mimicking the type of trade-offs policy makers make when choosing between interventions. A DCE is, furthermore, efficient due to its experimental design and small respondent group, allowing the study of preferences of small groups of decision makers. Our results illuminate the decision-making process and serve to facilitate further research.
METHODS AND DATA
Choice Situation
In twenty-seven choice sets, we asked respondents to elect reimbursement of one of two different healthcare interventions, A or B, which represented unlabeled, curative treatments (e.g., a new medicine or type of surgery). (Table 2 presents an example of the choice sets.) Both treatments were incrementally compared with usual care. The results of the interventions were normal outputs of economic evaluation studies with a societal perspective. Healthcare costs, other costs, and health effects in both the short and long runs were taken into account in the cost-effectiveness ratio.
The hypothetical treatments targeted patient groups most common to receiving treatments: males and females aged 50 to 75. This was motivated by our limited sample size and the desire to produce results that were meaningful in practice. We used a forced-choice design and, because respondents could not opt out, it provided information on many choice situations.
Attributes and Levels
A large number of possible attributes of interventions are mentioned in the literature and may be relevant to decision making (Reference Al, Feenstra and Brouwer1;Reference Baltussen, Ten Asbroek and Koolman3;Reference Strasmann15). We discussed many of them with five experienced HTA researchers at length before selecting the seven shown in Table 1.
Attributes and Levels of the DCE
Note. DCE, discrete choice experiment; QALY, quality-adjusted life-year; QoL, quality of life.
Attributes such as the amount of health gained and cost-effectiveness are key ingredients of any economic evaluation study. Disease severity (linked with equity concerns) has often been demonstrated to be a major consideration and was important to test in the Dutch context (Reference Baltussen, Ten Asbroek and Koolman3;Reference Gyrd-Hansen8;Reference Stolk, van Donselaar and Brouwer16). Savings in productivity costs were included, because Dutch guidelines for economic evaluation studies of new drugs require them. Budget impact is not standard output of economic evaluation studies, but it appears that policy makers often consider it (Reference Niezen, De Bont and Busschbach11). Including the composition of health gained (extension and/or improved quality of life [QoL]) was an outcome of the HTA researchers’ focus group. We also included uncertainty regarding cost-effectiveness. Uncertainty measures are increasingly standard research outputs, but we lack firm evidence on their use and whether they are appreciated by policy makers. With respect to health, Al et al. have suggested considering decision-makers’ risk attitudes (Reference Al, Feenstra and van Hout2).
The attributes were of such nature that we could freely decide on their intensity and range of values. We chose to use three levels per attribute: low, intermediate, and high. We used more than two levels to allow for identification of nonlinear attribute weights, and fewer than four levels to remain within the study constraints of a small sample. We decided on the attributes’ range of values pursuant to communications with and documents of relevant parties. The budget impact attribute (or additional annual medical costs) was given a minimum of €10 million (instead of 0 million), as most interventions incur additional costs. The top level, €50 million, is considered a very serious budget impact by Dutch policy makers. The range for the incremental cost-effectiveness ratio was inspired by the recent Dutch Health Council report that suggests €80,000 as a maximum paid for an additional quality-adjusted life-year (QALY) in case of an intervention for patients with a high disease severity (4). The disease severity of the patient before treatment was explained using the following examples: a low disease severity is, e.g., eczema or non chronic, mild low back pain (with an estimated QoL score of 0.94 on a scale of 0–1); a moderate disease severity is, e.g., heart failure or moderate rheumatoid arthritis (estimated QoL score 0.65); a high disease severity is, for example, progressive multiple sclerosis (estimated QoL score 0.33).
We carried out a pilot study with ten HTA researchers, which resulted in editing the DCE layout and changing attribute levels for productivity savings and the cost-effectiveness ratio. For an example of a complete choice set, see Table 2.
Example of a Choice Set
I prefer:
○ Treatment A
○ Treatment B
Note. QALY, quality-adjusted life-year.
Experimental Design and Regression Model
Our design of twenty-seven pair wise comparisons of two hypothetical treatments was 94 percent efficient (Reference Kocur, Adler and Hyman9;17). Each choice alternative was paired to an alternative in such a way that overlap of attribute levels was minimal. We adapted the choice set in one respect: we disallowed the seemingly implausible combination of a low disease severity (pretreatment) with the highest number of QALYs (4) gained per patient. For more information regarding the design, see the supplementary material. Respondent choices were analyzed using a multinomial logistic regression model. To show the impact of the decision attributes, we will present marginal effects of a unit change in each attribute.
Risk Attitude
We were interested in the respondents’ risk attitude toward health affects and healthcare costs. Hence, for one specific choice situation we asked whether the respondents preferred the situation in which (i) all uncertainty is related to health effects, (ii) all uncertainty results from costs, or (iii) they were indifferent.
Follow-up Focus Group Discussion
Upon completion of the questionnaire, we discussed the exercise with policy makers (members of the Ministry of Health, Department of Pharmaceutical Care; the Health Care Insurance Board; the Council for Public Health and Health Care). Issues included the exercise as a whole, attributes and levels, possible choice strategies, and the most difficult choice situations.
Respondents
The DCE questionnaire was administered to sixty-six respondents during November 2007 through March 2008. All respondents were familiar with economic evaluation studies; all were given oral and written questionnaire instructions. Approximately 40 percent were policy makers, deciding or advising on health insurance packages, pharmaceutical reimbursement, or general matters concerning allocation of healthcare funds and governance of health care (Table 2). Because the group of people directly involved in the decision-making process is small in practice, 60 percent of our sample consisted of other people having relevant knowledge of the decision-making process. One-third comprised master students in health economics who had finished a course in HTA studies. Approximately one-fifth of the respondents was engaged in performing HTA studies (see Table 3). We tested for heterogeneity in preferences between subgroups of respondents by including interaction terms between the subgroup and each of the attributes.
Characteristics of DCE Respondents
Note. HTA, health technology assessment.
RESULTS
DCE Questionnaire
For one of the twenty-seven choice sets, program A dominated program B with respect to all but one of the seven attributes. The exception was composition of the health gain, whose preferred level cannot be indicated beforehand. All sixty-six respondents preferred program A, which adds to the validity of the study.
Regression Results
The regression model's explanatory power is quite satisfactory for a DCE (r-square = 0.307) (Supplementary Table 1, which is available at www.journals.cambridge.org/thc2010017). The observed sign for each coefficient was as expected: an intervention with a higher budget impact, higher cost per QALY, and more uncertainty was less likely to be preferred, whereas disease severity and the amount of individual health gain were positively linked to the probability of choosing the intervention. All attributes were significant (p < .01), except for productivity savings (p = .10). The positive sign for the composition of health gain indicates that respondents preferred health gains that include quality of life improvements over extension of life without improved quality of life.
The only interaction term that was nearly statistically significant (p = .06) was that between budget impact and cost-effectiveness. It was positive but small, thus tapering the negative effect of budget impact and the cost-effectiveness ratio.
The relative importance of each attribute in decision making varies. For example, if an intervention costs an extra €10 million per year, its average probability of being preferred decreases by 16.6 percent (see the incremental effects in Supplementary Table 1). The results indicate that a one-level change in disease severity or cost-effectiveness has a very serious impact, changing the probability of choosing the program by 40 percent. The change in the amount of health gained per person was 20 percent. The impact of the other four attributes was smaller but not negligible. For example, a 10 percent rise in uncertainty resulted in a 9.8 percent decrease of choice probability.
Table 4 displays the effects of the observed attribute weights on attractiveness of a variety of hypothetical treatments. We compiled six scenarios representing archetypal cases. Scenario 1 set attributes at their most preferred levels; scenario 2 had “middle of the road” levels; scenario 3 was a typical worst case. Scenarios 4, 5, and 6 were added to illustrate the effects of budget impact and disease severity, where only one of each scenario's attributes was not set at mid-level. Scenario 4 had a lowest-cost level; scenarios 5 and 6 had low and high disease severities, respectively. Table 4 shows the probability of one scenario being chosen over other another as predicted by the regression model, given the set of alternatives presented to the respondents. The scenarios’ attractiveness clearly varied. Disease severity affected the acceptance rate enormously (scenarios 1, 5, 6). The budget impact was also essential (scenario 4).
Predicted Probability of Acceptance of Choice Situations (Scenarios)
alow, low disease severity; high, high disease severity
QALY, quality-adjusted life-year; QoL, quality of life.
It was surprising that respondents (especially HTA students) preferred health gains that were accompanied with quality of life improvements over extension of life. Does this finding threaten the standard QALY model? Perhaps not. By definition, extension of life occurs as a health benefit in the distant future (especially for younger students), which might be discounted more than a quality of life gain occurring in the near future.
Subgroups Compared
We also tested for significant differences in the weights of the attributes between respondent subgroups. These subgroup models had approximately the same explanatory power as the overall model. The selection of statistically significant attributes was identical for all subgroups. However, for some attributes, subgroups appeared to have somewhat different weights, (see also Supplementary Table 2, which is available at www.journals.cambridge.org/thc2010017).
The policy makers attached relatively more weight to disease severity than others. HTA students showed a weaker, but still substantial, preference for disease severity. Policy makers considered uncertainty related to cost-effectiveness, but less than others. As expected, for HTA experts uncertainty was relatively more important, as they have put it on the agenda. The HTA experts also attached somewhat more weight to cost-effectiveness but less to the magnitude of individual health gains. To our surprise, HTA experts weighted productivity savings less than the other respondents, but it should be noted that this attribute is the only non significant one for the total group of respondents.
HTA students attached relatively more weight to individual health gains and, as previously mentioned, they preferred (more than average) health gains that included quality of life improvements as opposed to a longer life without quality improvement.
Risk Attitude
As a final question, we presented an intervention that cost €45,000 per QALY compared with usual care with a 20 percent probability that the costs per QALY were at least double (i.e., €90,000). We asked if the respondents preferred the situation in which all uncertainty was related to health effects, all uncertainty was in costs, or they were indifferent. Only 11 percent preferred uncertainty from health effects, 56 percent from costs; 27 percent were indifferent; 6 percent was missing. The results indicated that respondents are generally more risk averse toward health effects than costs. Remarkably, 23 percent of the students preferred uncertainty related to costs, whereas no HTA expert had that preference.
Focus Group Discussion
The DCE was designed to explore trade-offs between the various attributes. Some policy makers indeed tried to find an optimal balance on all criteria; some used simpler decision rules. All policy makers believed that the DCE was a valuable exercise, but added that cases in reality are often more complex and multidimensional. This confirms Lancsar's findings (Reference Lancsar and Louviere10).
Many policy makers believed that disease severity was their primary concern. Willingness to pay for treatments for minor ailments was rather limited. For some, disease severity and QALY gain dominated their decisions. Many believed that the most difficult choice sets were between moderate and high disease severity. Two stressed the importance of the composition of the health gain.
The policy makers also made statements about information they believed was missing: (i) the number of patients covered by the proposed healthcare interventions, (ii) the healthcare costs per person, (iii) the distribution of health gains by socioeconomic status, (iv) the budget impact of the intervention vis-à-vis the total healthcare budget, and (v) information on risk behavior related to disease.
Finally, respondents commented on the quality of the questionnaire, and whether it triggered them to express clear preferences. Regarding budget impact and productivity savings, some respondents believed they needed a larger range of values to clearly demonstrate their preferences. Regarding uncertainty, some believed that it was only considered in relation to high cost per QALY or with a substantial budget impact. A few respondents stated that 0.5 QALY per person is only a modest health gain. It should be noted, however, that pharmacoeconomic dossiers often show health gains of a new drug smaller than 0.5 QALY.
Scenario Description/Labels
Some policy makers were comfortable with the nonspecific scenarios with respect to diseases and treatments; they believed that more specification would have biased their judgment. Others, however, stated they needed more specification to form an opinion. One participant wondered if the QALY concept was reliable enough as health information in the scenarios and speculated that more disease-specific information would be necessary for an informed decision.
DISCUSSION
The analysis in this study clearly revealed what criteria stemming from HTA studies are most important for reimbursement decisions of medical treatments and to what extent tradeoffs are made. Severity of disease, costs per QALY gained, individual health gain, and the budget impact were the most decisive attributes. A program targeting more severe diseases increased the probability of reimbursement dramatically. Uncertainty related to cost-effectiveness was also important. Respondents preferred health gains that include quality of life improvements over extension of life without improved quality of life. For policy makers and the other health professionals, the same set of attributes was important, but policy makers attached somewhat more weight to disease severity and less to uncertainty.
According to the results, the respondents’ underlying optimization function seemed to contain elements that were both utilitarian (preferring more health effects and less costs) and egalitarian (preferring a more equitable distribution of health) (Reference Rabinowicz13).
Our DCE had a comparatively small sample because we were limited to Dutch healthcare respondents familiar with the results of HTA studies. Our objective, however, was not to collect detailed choices for allocation decisions of specific health services, but rather to glean the relevance of attributes in the decision-making process and the tradeoffs made by respondents. Further study with a larger, preferably international, sample and further variation in attributes and levels should be conducted to verify the robustness of our results. In addition, comparison of the results with new material on actual reimbursement decisions would be worthwhile.
Several other studies have been done following a stated-preference design, polling the decisions of policy makers and/or the public (Reference Al, Feenstra and Brouwer1;Reference Al, Feenstra and van Hout2;Reference Baltussen, Ten Asbroek and Koolman3;Reference Green and Gerard7;Reference Gyrd-Hansen8;Reference Ratcliffe, Bekker and Dolan14;Reference Strasmann15). Five studies used a conjoint analysis or discrete choice experiment (DCE) framework. The Baltussen et al. (Reference Baltussen, Ten Asbroek and Koolman3) DCE involved Nepalese policy-makers’ and health professionals’ choices for a public health priority setting. He found that the Nepalese preferred interventions that (i) targeted large sets of middle-aged groups and severe diseases; (ii) had significant individual health benefits; (iii) led to poverty reduction; and (iv) were cost-effective.
Schwappach and Strasmann (Reference Strasmann15) investigated the reliability of an internet-based survey to elicit preferences for priority setting of hypothetical treatment programs. They reported a preference for programs that targeted younger age groups and common diseases, had significant individual health benefits, and were below average in cost.
Green and Gerard (Reference Green and Gerard7) analyzed health program choices of the UK general public. Using less complex qualitative attributes (e.g., very good, fairly good, fairly poor, very poor cost-effectiveness), he showed that DCE was feasible and valid for the general public and that all four attributes—health improvement, cost-effectiveness, disease severity, and the availability of other treatments—were important.
Gyrd-Hansen (Reference Gyrd-Hansen8) investigated the Danish public's view on the tradeoff between the amount and the distribution of health gains secondary to interventions: the Danish public gave priority to those in a more severe health state.
Ratcliffe et al. (Reference Ratcliffe, Bekker and Dolan14) studied the views of UK National Health Service decision makers and care providers using four attributes: health benefits (in QALYS), the share of QALYs gained for the worst off, waiting time for treatment, and travel distance to care facilities. All were important, although health benefits dominated. Because costs and cost-effectiveness were not used as attributes, it is difficult to draw direct conclusions for allocation decisions.
Although the results of studies vary with context and research objects, one could say that our study largely confirms these results, but shows a larger set of important (quantitative) decision criteria, including their relative importance.
Do our Dutch respondents take a societal perspective, as advocated? Economic theory supports as attributes: cost-effectiveness, individual health gain, disease severity, and productivity savings. All these criteria were significant, except for productivity savings. Economic theory would exclude budget impact (a dollar is a dollar) as a criterion. In our study, budget impact is important, which corresponds with recent research (Reference Niezen, De Bont and Busschbach11). It may reflect that the normative content of economic theory is not upheld after all, or that decision makers face particular problems in its application.
An implication of this study is that we can derive information about CE threshold variance across severity levels. The marginal rate of substitution for disease severity versus cost-effectiveness was €-27,995. This suggests that the acceptable cost-effectiveness ratio in case of high disease severity is €56,000 (2 × €27,995) higher than for low disease severity. Examples of a low and high disease severities given to respondents were, respectively, mild eczema with a QoL index of 0.99 and progressive multiple sclerosis with a QoL of 0.33. Combining the willingness to pay €56,000 for a QoL-gain of 0.66 (0.99–0.33) suggests a QALY value of approximately €93,000 (€56,000/0.66). This meshes nicely with the recommendations of the Dutch Health Care Council suggesting a maximum cost effectiveness ratio of €80,000 per QALY gained for a very severe disease and approximately €10–15,000 for low disease severity (4).
CONCLUSION
We performed a discrete choice experiment (and focus group discussion) among Dutch policy makers and other health professionals to identify the criteria stemming from HTA studies that are most relevant in reimbursement decisions of medical interventions and what trade-offs emerge between them. The analysis revealed that all attributes were very significant except for the savings in productivity costs. Severity of disease, costs per QALY gained, individual health gain, and the budget impact were the most decisive attributes. A program targeting more severe diseases increased the probability of reimbursement dramatically. For policy makers and the other health professionals, the same set of attributes was important, but policy makers attached somewhat more weight to disease severity and less to uncertainty. Our study largely confirms results of other studies, but shows a larger set of important (quantitative) decision criteria, including their relative importance.
To conclude: Have (Dutch) policy makers made up their mind? We think they seem to have reasonably well-articulated preferences: six of seven attributes were clearly significant. Further study with a larger sample and further variation in attributes and levels might verify the robustness of our results.
SUPPLEMENTARY MATERIAL
Supplementary Table 1
Supplementary Table 2
CONTACT INFORMATION
Marc A. Koopmanschap, PhD (koopmanschap@bmg.eur.nl), Associate Professor, Elly A. Stolk, PhD (stolk@bmg.eur.nl), Senior Research Fellow, Department of Health Policy and Management, Erasmus University Rotterdam, P.O. Box 1738, 3000 DR Rotterdam, The Netherlands
Xander Koolman, PhD (a.h.e.koolman@tudelft.nl), Associate Professor, Faculty Technology, Policy and Management, Delft University of Technology, Jaffalaan 5, 2628 BX Delft, The Netherlands
