4.1 Budget allocation

The amounts allocated to groups A, B and C did not differ significantly between version 1 and version 2. This indicates that the order in which the questions were introduced had little impact on participant’s priority-setting choices. In brief, the order effect was effectively eliminated.

In each of the budget allocation exercises more was allocated to group A than groups B or C. This confirms that the size of the health improvement achieved by different patient groups is important (Nord et al., Reference Nord, Richardson, Street, Kuhse and Singer1995a; Abellan-Perpiñán and Prades, Reference Abellan-Perpiñán and Prades1999; Roberts et al., Reference Roberts, Bryan, Heginbotham and McCallum1999). However, if participants had sought to maximise health group A would have received 100% of the budget in each exercise. This was not the case. Even when the QALY gain to group B was 80% lower than group A (11 vs 55 QALYs in question 1, exercise 4), over a third of the budget (37.5%) was allocated to them, and when the QALY gain to group B was 45% lower than group A (30 vs 55 QALYs in question 1, exercise 1), nearly half of the budget (46.7%) was allocated to them (Figure 2). This confirms the findings of previous studies that the societal value of an intervention is not proportional to the size of the health improvement (Dolan and Cookson, Reference Dolan and Cookson2000; Nord et al., Reference Nord, Undrum Enge and Gundersen2010).

Also, the amounts allocated to groups B and C did not differ significantly in any of the exercises (Supplementary Table 1). This was so even in exercise 4 when there was an 80% difference in the pre-treatment health states of B and C. This is consistent with the results of the Likert questions, which showed stronger support for an equal division of the budget than for giving group B priority. It is possible the repetition of the exercise may have made respondents want to appear consistent, and thus biased them towards giving the same answer across the two questions within each budget allocation exercise. However, there are good reasons for treating groups B and C differently: for example, it may be considered fair to give more of the budget to group B to compensate them for their past health losses [argument (g)], or to equalise the lifetime health of the two groups [argument (h)] or because group B are members of the disability community [argument (i)]. There was no evidence from the discussions that respondents thought these arguments were inconsistent, or thought that support for them would be perceived by others as inconsistent. It is also worth noting that written answers to the budget allocation exercises were confidential. Here, also, respondents (on average) allocated similar amounts to groups B and C, although the question of how this would appear to others was not an issue.

The results show that past losses in QoL are of relatively little importance for priority setting compared with potential future losses. It cannot be concluded from this that past life-years – i.e. the age of a person, is of little importance, or that life-expectancy is of little importance. It follows that nothing about the life-year version of the fair-innings argument – the idea that people are entitled to a fair minimum length of life – can be inferred from the results of the present study. However, they do pose a challenge to recent studies reporting support for the generalised fair innings argument, which maintains that people are entitled to a fair lifetime allocation of QALYs (Stolk et al., Reference Stolk, Pickee, Ament and Busschbach2005; Dolan and Tsuchiya, Reference Dolan and Tsuchiya2012; Olsen, Reference Olsen2013; Ottersen et al., Reference Ottersen, Maestad and Norheim2014). The lifetime aspect of this view captures the relevance of the past, and the QALY component secures the relevance of QoL. It implies that past losses in QoL are no less important than future losses for the purposes of setting priorities, which is contrary to the findings of the present study. To be consistent with the results presented here, the generalised fair-innings argument would have to countenance applying different weights to past and future losses in QoL. The generalised fair-innings argument is captured in argument (h), which received the lowest ‘persuasive’ score in version 1 and the equal third lowest ‘persuasive’ score in version 2 (Table 3).

4.2 The Likert arguments

Some of the Likert arguments in support of group B received a higher rating in version 1 than version 2. This could be due to the order in which the arguments were presented, or to variations in the amount of time devoted to the different arguments in the different groups. On the other hand, there were also significant similarities between the two sub-samples. For example, in both version 1 and version 2 respondents found the arguments in favour of an equal division of the budget more persuasive than those in favour of group B. This was so both in scenario 1 and scenario 2 (Table 3). Also, both sub-samples gave arguments (s) and (t), which focused on the present and future suffering of groups B and C, their highest ‘persuasive’ scores (91%). Finally – and of most importance from a policy point of view – variations in the weight attached to some of the individual arguments did not translate into significant differences in how the budget was allocated.

Arguments (s) and (t), which achieved the highest Likert scores, support an equal division of the budget on the grounds that the current and prospective health status of groups B and C is the same with and without treatment (s), and that the past cannot be changed (t). The following quotes from the discussions, when respondents were directly comparing groups B and C (scenario 2), are typical of this view:

Argument (i) in Table 3 states: ‘The people in group B are members of the disability community and deserve special consideration’. The inclusion of this argument was prompted by Ubel et al.’s observation that “the pre-existing paraplegia scenario makes many people think about the importance of avoiding discrimination against people in the disability community”. Participants accepted that group B are members of the disability community on the basis that they have a permanent health problem that adversely affects their QoL. In more detail, the argument is that the disabled often have a harder life – they come up against significant barriers while trying to do the things others take for granted – both due to their health problems and to systematic social discrimination, and for this reason deserve special consideration. Opinions may differ about whether this is a good argument or not: ~30% of our participants found the argument that ‘the people in group B are members of the disability community and deserve special consideration’ persuasive.

Argument (i) differs from arguments (g) and (h) in the following way. In the discussions of arguments (g) and (h) the focus was on the effect of the disabilities on the individuals in group B. In the discussions of argument (i), which appeals to membership of the disability community, the focus broadened to include society’s response to those with disabilities, not just the effect of the disabilities themselves. The results show that respondents did not consider argument (i) identical to either argument (g) or (h): for example, 11% of version 1 respondents found argument (h) persuasive, while 23% found argument (i) persuasive, and 25% of version 2 respondents found argument (g) persuasive, while 37% found argument (i) persuasive.

4.3 Methodology

If a respondent who received version 1 of the questions allocated 70% of the budget to group A and 30% to group B, and they thought that group C should be a higher priority for treatment than group B, then they should allocate at least 30% to group C in the A vs C comparison. This is exactly what we observe from the budget allocation results. Transitivity of preference is a tricky notion [cf. Arrow’s impossibility theorem (Anand, Reference Anand1993)] and if respondents had violated transitivity it would not necessarily mean they were irrational: we would look for an explanation. But the fact that they did allocate this way strongly suggests that they understood the comparison of A vs C even though it was not directly discussed in version 1. They understood it and allocated the way they did because they had discussed the A vs B and B vs C comparisons and the arguments in Table 3. This is why only version 2 received scenario A vs C. It was included in version 2 because one of the sub-samples had to receive the onset scenario first in order to test for the order effect, but it was not necessary that version 1 receive it.

A unique feature of the present study was the use of group discussions to achieve deliberation with respect to the study questions. This sets the approach apart from self-administered postal surveys, web-based surveys and one-on-one interviews. Deliberation has the potential for producing more rational and informed choices, “because free discussion and open debate allow relevant information to be distributed, mistaken reasoning to be exposed, and all the reasons for and against [views] to be debated and considered” (Freeman, Reference Freeman2000: 383). Additionally, it encourages participants not just to reflect upon alternatives, but to reflect critically upon their own preferences and evaluate them as well-supported by the evidence, fair, and consistent with their other beliefs (Rawls, Reference Rawls1971). The public nature of the discussion groups encouraged participants to find solutions that are in the interests of everyone since justifications that serve narrow self-interest are challenged by others in the group (Jönsson, Reference Jönsson2009; Hakli et al., Reference Hakli, Miller and Tuomela2010). This is consistent with adopting a social rather than a self-interested decision-making perspective.

It is possible that participants who are more confident, or have a higher perceived social status, could have an undue influence on other participants (Morgan, Reference Morgan1997). Several measures were taken to minimise this. For example, the groups were divided into four age strata, which contributes to group cohesiveness and reduces the possibility of imbalances in power or social status that can inhibit discussion (Finch and Lewis, Reference Finch and Lewis2003; Hesse-Biber and Leavy, Reference Hesse-Biber and Leavy2006). One potential downside, however, is that younger participants do not hear the views of older participants, and vice versa. Also, participants were never asked to arrive at a consensus. This avoided creating a competitive situation where one view must prevail, and where dominant individuals may have an undue influence. Finally, the demeanour of the moderator is important in setting the tone and expectations of the discussions. If a respectful, supportive atmosphere is achieved participants are less afraid to participate, and less likely to be swayed by dominant individuals (Morgan and Spanish, Reference Morgan and Spanish1984; Stewart and Shamdasani, Reference Stewart and Shamdasani1990).

Another unique feature of the study was the presentation of the arguments in Table 3. It is possible that this could inhibit the free expression of ideas or bias the direction of discussion. However, it was important that participants were aware of the main arguments in support of different allocation principles and their implications if their choices were to be informed. To minimise bias the arguments were presented (both verbally and in writing) in objective language, the moderator was careful not to convey a personal viewpoint regarding them, and arguments for and against each option were impartially explored. Also, during the initial discussions participants had the opportunity to raise their own considerations before being presented with the arguments in Table 3 to avoid a potential anchoring effect. The aim was to ensure participants were informed about all of the major ethical considerations, while at the same time minimising the possibility of bias.

Some participants in the study may not have agreed that the health state descriptions used in the figures correspond accurately with the percentages. This is an implication of the fact that there is significant heterogeneity in individual preferences for health states (Roberts and Dolan, Reference Roberts and Dolan2004). However, participants were asked to judge from a social rather than a self-interested perspective, so it is appropriate they were given an indication of what the public associates with 70% of full health, 55% of full health, etc. The aim was to achieve an acceptable balance: descriptions were sought (1) that matched up reasonable well, though perhaps short of an exact mapping, with the percentages, and (2) that were relatively easy to remember, reduced the cognitive burden on participants, and provided a common focus for respondents that would lessen differences in interpretation. The EQ5D-5L was chosen because it met these criteria better than the alternatives. Had a more complex set of descriptions been used (e.g. from the SF-6D, HUI 3, 15D or AQoL) this would have increased the cognitive burden on participants. If no descriptions at all were used participants would have lacked a shared understanding of what the community (on average) associates with 70%, 55%, etc. of full health. The discussions confirm that participants accepted that the percentages represent the average social valuation of the health states irrespective of their personal valuation.

4.4 Elimination of the order effect

Three possible explanations for the order effect were considered previously: depending on the order in which the scenarios are received, participants may or may not think about the importance of avoiding discrimination, exaggerate the negative impact of new onset of paraplegia, or focus on the ineffective treatment. While differing in their details, all three explanations appeal to a form of anchoring: respondents have an immediate reaction to the first question (either the pre-existing or onset scenario) which significantly influences their answer to the second. In the present study participants engaged in a process of critical self-examination that allowed them to move beyond their first reactions. This was facilitated by the direct comparison of the pre-existing and onset scenarios, the opportunity for deliberation with others, and presentation and discussion of the Likert arguments. It would be possible to measure the relative influence of these factors by varying one while holding the others constant. However, the purpose of the present study was not to investigate the precise causes of the order effect, but to cast light on respondent’s consistent, underlying preferences and the reasons for them.

The thought might occur that the differences between the results in the present study and those in the earlier studies could be due to the choice of health states. If they are less severe than paraplegia – the health state used in the Ubel et al. and Damschroder et al. studies – this could explain the absence of an order effect. However, in BA 4 both patient groups would remain in, or be left in, severe pain or discomfort and severe anxiety or depression without treatment. This rates 0.20 on the EQ5D-5L and a number of other scales (Patrick et al., Reference Patrick, Starks, Cain, Uhlmann and Pearlman1994), whereas paraplegia rates about 0.48 (Bahena-Salgado and Bernal-Márquez, Reference Bahena-Salgado and Bernal-Márquez2007). The latter study measured self-reported QoL among 23 paraplegic patients using the Functional Independence Measure scale and the EQ-5D. It is unlikely then that the differences between the results are attributable to the choice of health states. In fact, including health states both worse than and better than those used in the previous studies provided a test of whether the order effect is a function of the level of severity.

The possibility cannot be ruled out that for some unknown reason, or purely by chance, the recruitment of subjects lead to atypical groups that were resistant to order effects from the outset. However, we know from the previous studies that people are susceptible to the order effect, and there is nothing to suggest our participants were different in this regard. Participants were recruited by a professional recruitment company to achieve gender balance and a range of age, income and educational levels. There was an element of self-selection since all subjects volunteered to register with the recruitment company. But there is no obvious connection between this and lack of susceptibility to framing effects. Further, the opening questions revealed no significant differences between our sample and the earlier studies on this topic. While it remains a possibility that our sample was atypical of the general public and resistant to the order effect, we think the more plausible explanation is that the opportunity for deliberation, the direct comparison of the pre-existing and onset cases, and exposure to the Likert arguments eliminated inconsistency.

4.5 Study limitations

The study was subject to a number of limitations and caveats. The first was the small sample size. There were practical reasons for this. Group discussions like those described here (as well as one-on-one interviews) are expensive to run on a large scale. Postal or online surveys are relatively cheap by comparison, but provide limited opportunities for respondents to consider problems in depth or to benefit from deliberation. From a practical point of view the best approach to the investigation of preferences on conceptually and ethically challenging issues is a mixture of large-scale postal or online surveys in combination with a smaller number of more intensive studies where greater efforts can be taken to ensure that participants understand the questions and have to time reflect, deliberate and seek clarification of the task.

How the results of studies employing different data collection methods are to be combined is a challenging question. On the basis of ‘eyeballing’ the results of a number of preference studies on potential to benefit and severity of illness, Nord formulated ‘rules of thumb’ to guide decision-makers. He acknowledges that the evidence is “scattered and heterogeneous” (Nord, Reference Nord1999: 37), that the rules encapsulate social preferences “to a reasonable degree” (p. 43), and that quantitative models for estimating the societal value of health care activities should – ‘at least roughly’ (p. 43) – reflect the structure of concern encapsulated in the rules. Although this approach lacks a degree of precision in the early stages, as new data is accumulated the ‘rules of thumb’ can be modified, re-tested and gradually made more precise.

Despite the relatively small number of participants, the opening questions revealed no significant differences between our sample and the earlier studies. While this does not demonstrate that our sample was fully representative of the broader population, it does provide evidence that our participants were not highly atypical in their opinions regarding this topic. Nevertheless, caution must be exercised in drawing general conclusions when sample sizes are small, and the present study should be considered exploratory in nature.

Other limitations arose from the need to reduce the cognitive burden on participants. For example, the result of non-treatment was always immediate death. This context was chosen to preserve comparability with the Ubel et al. and Damschroder et al. studies. Also, the need for treatment always occurred at age 25, which was chosen due to the importance of the social and economic contribution of people at this age. Another limitation was the small number of generic health states used. It is possible, for example, that people may be more willing to compensate for past pain than for past problems with mobility or self-care. It was beyond the scope of the present study to investigate these issues. Nevertheless, if these or other features of the scenarios were different results could potentially also differ.