Charting a different course

Smith and Sach argue that there are fundamental questions that need to be answered in CV research – whether there is a role for CV in health economics, and also how it might reach its desired potential. In the descriptive assessment of studies included in their database,Footnote 1 they come to a pessimistic conclusion. However, even just looking at the same information, one could note: there has been an increasing number of CV studies published in health, applied in a growing number of countries, in an ever broadening number of clinical areas. The number of studies where health is not the focus has increased; thus the method is being used to go beyond health outcomes. An increasing number of studies has examined health status, probabilities or both, from which one could conclude that CV studies are looking at valuing elements of the quality-adjusted life year (QALY) where the QALY model itself is quite restrictive. Their own evidence answers the question of whether there is a role for CV in health economics. Our conclusion from all of this would be considerably more optimistic than the authors – and to quote Smith and Sach ‘the popularity of CV…shows no sign of abatement’. The second question they pose, we would argue, is not possible to answer with the types of descriptive information they present and without more explicitly addressing ‘potential for what’ and a more in-depth look at the objectives of the studies.

If one were to look more purposefully at the map of CV in health, a clearer picture than that drawn by Smith and Sach would emerge and show that indeed CV in health has progressed and developed over time as a ship without a sail would not. For example, it could be argued that the re-emergence of CV in health in the early 1990s, generally recognised as being signified by the papers by Donaldson (Reference Donaldson1990), Gafni (Reference Gafni1991), Johannesson and Jonsson (Reference Johannesson and Jonsson1991) and Johannesson et al. (Reference Johannesson, Jonsson and Borgquist1991), was aimed at improving decision-making. This has been structured at three levels (initially two) of decision-making as portrayed in Shackley and Donaldson (Reference Shackley and Donaldson2000) and Donaldson et al. (Reference Donaldson, Mason and Shackley2006a). The first was the development and use of CV to aid decision-making at the level of close substitutes, largely in helping to choose between therapies for the same group of patients. A more complete set of references to this work, and summary of progress, can be found in Donaldson (Reference Donaldson2001). The second level is contributing to allocative decisions in choosing between patient groups or across health programmes, an area of application started by Olsen and Donaldson (1998) and which led to the subsequent EuroWill project (O’Shea et al., Reference O’Shea, Stewart, Donaldson and Shackley2002; Shackley and Donaldson, Reference Shackley and Donaldson2002; Stewart et al., Reference Stewart, O’Shea, Donaldson and Shackley2002; Olsen et al., Reference Olsen, Donaldson and Periera2004a, Reference Olsen, Kidholm, Donaldson and Shackley2004b; Protière et al., Reference Protière, Donaldson, Luchini, Moatti and Shackley2004; Ryan et al., Reference Ryan, Scott and Donaldson2004; Olsen et al., Reference Olsen, Donaldson and Shackley2005). Other (non-health) areas have paralleled this area of research (Hsee et al., Reference Hsee, Loewenstein, Blount and Bazerman1999). The third level of decision-making, that of national health technology assessment, is discussed later in this paper.

Within these first two levels of decision-making, the pursuit has been relatively conventional and systematic – trying to establish methods to achieve greater validity and reliability. This includes work on different payment vehicles (which started earlier in health than indicated by Smith and Sach (Donaldson et al., Reference Donaldson, Thomas and Torgerson1998)). It could be argued that more success has been achieved in establishing greater validity in the first level (the valuation of close substitutes) than in the second level, where basic problems still exist in terms of scope effects (Olsen et al., Reference Olsen, Donaldson and Periera2004a) and the ability of CV to produce the same orderings of health programmes as a simple ranking even within the same individual (Olsen et al., Reference Olsen, Donaldson and Shackley2005). Each of these challenges has stimulated further research to investigate the potential for improvement (Currie et al., Reference Currie, Donaldson and Lee2005; Donaldson et al., Reference Donaldson, Shackley and Currie2006b).

With reference to the above programmes of work, some of Smith and Sach’s criticisms now seem odd. Their point about ex ante and ex post perspectives and the misspecification of CV studies is less relevant if one considers CV questions in terms of these three levels of decision-making; the key being to decide whether a study informs choice between interventions for given patient groups or between programmes (and patients), from which the rest, in terms of appropriate scenario descriptions etc., would follow. Related to this, the original Olsen and Donaldson paper and the subsequent EuroWill enterprise were established precisely to introduce notions of option value, externalities and non-health outcomes into the willingness-to-pay (WTP) metric, whilst accounting for the fact that that metric would have to be applied in a public sector environment. Much of the conceptual basis for this was laid out around the time of commencement of EuroWill (O’Brien and Gafni, Reference O’Brien and Gafni1996) which led to different contextual approaches, based on insurance or taxation, being tested to examine the impact of such notions on WTP (Olsen et al., Reference Olsen, Kidholm, Donaldson and Shackley2004b). Likewise, at the level of decision-making for close substitutes, the issue of process utility has not been ignored (Donaldson and Shackley, Reference Donaldson and Shackley1997).Footnote 2

The criticism that very few CV studies make use of their advantage to compare benefits with costs also seems odd to us. For example, the Olsen and Donaldson approach was devised to account for the fact that, in a publicly funded health care system, even if the benefits of one programme are shown in a study to be greater than its costs, the opportunity cost of implementing that programme will be the benefits sacrificed by not funding care for some other group of patients. Hence the comparison of helicopters with hearts and hips! The theoretical arguments for this approach, and a demonstration that cost–benefit analysis in health is not as simple as checking if benefits are greater or less than costs, are outlined in detail in Shackley and Donaldson (Reference Shackley and Donaldson2000).

The ‘non-adoption’ of monetary valuation of health benefits

Smith and Sach repeatedly mention the lack of impact of CV on health care decision-making, which they attribute to the ‘ad-hoc and uncoordinated manner’ in which WTP research has been conducted. Do the apparent methodological meanderings of WTP researchers really constitute the central reason for preventing the wider uptake of CV as a tool for decision makers? We suggest not.

The development of valuation methods in health economics, after the earliest applications of cost–benefit analysis in the 1970s (Acton, Reference Acton1976), followed a path initially focussed on clinical decision-making and priority setting. In the 1970s, health economics focussed largely on bridging the disciplinary worlds of economics and medicine, and incorporating examinations of costs alongside clinical measure of outcomes. The next phase of development in valuation methods was of the QALY as a measure of health benefit to improve upon the limitations of looking at clinical measures alone. In the 1990s, the development of CV (as well as discrete choice stated preference methods) came into the picture alongside continued application and development of the QALY. From this larger perspective, it seems very alarmist of Smith and Sach to suggest jumping ship at a time when the valuation of health benefits fleet has been built to allow for greater choice in best meeting the purpose of any given research or policy journey.

Donaldson and Shackley (Reference Donaldson and Shackley2003) document the use of WTP in influencing decision-making in health authorities, in several clinical areas and, of course, in environmental and safety policy areas. Still, there is no doubt that CV does suffer from an image problem which hinders its use in health. This is, in part, due to the fact that WTP is positively associated with ability to pay, the argument then being that this is not a good basis for resource allocation in publicly funded health care systems. Advocates of the QALY as the appropriate benefit metric have frequently rehearsed the ‘ability-to-pay’ argument as a reason for supporting cost utility analysis based on QALYs to the exclusion of other approaches. Again, however, the ability-to-pay argument is more complex than is usually portrayed, and it has been shown in the literature that WTP can indeed be defended on distributional grounds (Donaldson, Reference Donaldson1999). In addition, it has been demonstrated that QALYs suffer from the same distributional challenges (Donaldson et al., Reference Donaldson, Birch and Gafni2002). This lesson will come up again later in the paper.

A more positive development regarding the policy impact of CV is, in fact, alluded to by Smith and Sach. They clearly document increased use of CV in health economics, yet have a persistent concern about its lack of policy impact. Despite historical arguments that have persisted against WTP-based measures, they are now on the map of policy-makers through the commissioned work that has recently taken place in the UK on the social value of a QALY (the third level of decision making referred to in our introduction), and which is taking place internationally too (e.g. Gyrd-Hansen Reference Gyrd-Hansen2003, Reference Gyrd-Hansen2004; Brouwer and Rutten, Reference Brouwer and Rutten2005). Although we might continue to debate the wisdom of trying to estimate WTP for a QALY, two methods previously construed as rivals have been brought together in this enterprise. We would contend that such policy-related use of WTP in health is in large part due to the contributions of the earlier literature referred to above and in Smith and Sach’s paper.

The role of guidelines

Now to the role of guidelines. As we have said in the introduction to this commentary, it is not clear whether Smith and Sach are advocating guidelines for reporting of studies or guidelines for their conduct. The former always falls foul of journal space for reporting on a list which itself is already one page long in Smith and Sach’s paper! We would agree that authors should always be able to make such details available, which would indeed be good practice.

We would not agree that the solution for CV in health is guidelines, whether about reporting or conducting applied studies. But, even if guidelines were the way forward, we are astonished that Smith and Sach have settled on the National Oceanic and Atmospheric Administration (NOAA) Panel guidelines as the simplest starting point. For a period in the mid-1990s, one of us (C.D.) experienced review processes that were guideline driven, resulting in papers being rejected because they did not comply with these guidelines. We would contend, but would be happy to submit to Smith and Sach’s judgement, that these papers subsequently made contributions to the literature. The NOAA guidelines were intended for a specific purpose – standardising environmental CV studies for the purposes of damages assessment in the legal system. They are now 15-years old and even within the environmental economics literature are not now adhered to strictly. Within health, it also can be argued that key aspects of the NOAA guidelines have subsequently been discredited. The closed-ended or referendum approach, was recommended by the Panel as the payment vehicle in CV because it would lead to more conservative WTP values. However, several papers since then have shown this judgement to be wrong (Frew et al., Reference Frew, Whynes and Wolstenholme2003; Ryan et al., Reference Ryan, Scott and Donaldson2004). Arguments have also been presented that other approaches, to use Smith and Sach’s words, ‘resemble consumer choice in an actual market situation’ just as well (Donaldson et al., Reference Donaldson, Thomas and Torgerson1998).

A major concern is with the unspoken aspects of what exactly is being proposed here. Would papers be prevented from being published if such guidelines were not adhered to? What would this do to innovation? Especially, as suggested by Smith and Sach, if we are even to be guided on ‘how to deviate from the guidelines appropriately’. Given that the Nobel Laureates on the NOAA Panel could not get this right, it raises the question of whether customers for CV research would rather have it conducted by experts in the field or someone who has ticked all the boxes in the guideline list. We would go for the former and would query the added value of the latter, especially given its potential to stifle innovation. There are excellent manuals written by such experts which would provide much better guidance and not have the same stifling effect (Bateman et al., Reference Bateman, Carson, Day, Hanemann, Hanley, Hett, Jones-Lee, Loomes, Mourato, Özdemiroglu, Pearce, Sugden and Swanson2002).

The stifling effect of guidelines is apparent in that section of the health economics literature in which cost–benefit analysis and WTP are often disregarded simultaneously, on the basis of simple, superficial arguments that do not stand up in the face of more considered analyses. In the introduction to their book, Cost-Effectiveness in Health and Medicine, Gold et al. (Reference Gold, Siegel, Russell and Weinstein1996) state that ‘(CBA) presumes to put a dollar figure on the value of human life and uses controversial methods to do so’, going on to argue that, ‘cost–benefit analysis’s primary valuation method is WTP, an approach whose difficulty lies in its intrinsic favoring of the programs and diseases of the affluent over those of the poor’ (Gold et al., Reference Gold, Siegel, Russell and Weinstein1996: xxii). It seems that national level bodies undertaking health technology assessments committed to a journey of cost–utility analysis have been influenced by such texts and their authors in the hope of arriving at a distribution-free utopia which does not exist, a trap that Smith and Sach may also be falling into when they state that the National Institute for Health and Clinical Excellence (NICE) may think that ‘UK society does not share the values of CV’. Guideline bodies composed of a broader range of economists tend to be more eclectic in the evaluation approaches recommended (Drummond and Jefferson, Reference Drummond and Jefferson1997).

Some concluding remarks

Without exploring the studies in the CV literature fully, Smith and Sach have settled on some rather alarmist conclusions about potentially needing to abandon the CV ship in health, and they have also settled on some rather alarming solutions. We cannot help but worry that if one were to turn the same attention to the other valuation methods available to us in health economics, in particular the QALY, the lessons from the distributional concerns in CV will re-emerge. Further lessons may also emerge from the scrutiny Smith and Sach give to scenario specification. In the UK, NICE uses a health-related quality-of-life tariff which one only has to eyeball to realise that the numbers do not make sense, such numbers being based on a time trade-off approach which uses EuroQol (EQ-5D) health states scenarios with 10-year time horizons. Generally, what we are saying is that applying the arguments against CV to other valuation approaches would require us to also abandon the QALY and leave us shipwrecked.

In the words of Charles Bower:

Having taken a different view of the descriptive information Smith and Sach displayed regarding CV in health, and taken a more focussed look at some of the literature with explicit attention to the research purpose, we would draw a much more positive conclusion of the health of CV, and have considerable optimism for its future direction. Over time, CV in health has been increasingly applied with ‘no sign of abatement’ in an ever expanding number of countries and contexts (clinical and beyond), significant methodological work has been undertaken, new lines of enquiry are explicitly linking CV and QALY methods, and CV methods are now being sought out at a high policy level, nationally in the UK and internationally. We believe we are embarking on a period of heightened innovation in CV methods in the upcoming years, which will only spur on innovation in QALY methodology enhancing the range of options available to health economists in the valuation of health and health care.