Introduction
Economic evaluation has been defined as ‘ensuring the value of what is gained from an activity outweighs the value of what is sacrificed’ (Williams, Reference Williams and Teeling Smith1983).
There is a considerable literature on how to use economic evaluation methods to inform health care decision makers (e.g., Gold et al., Reference Gold, Siegel, Russell and Weinstein1996, Drummond et al., Reference Drummond, Sculpher, Torrance, O’Brien and Stoddart2005). The adoption of economic evaluation guidelines by decision-making bodies in several countries has seen the institutionalization of economics as a basis for decision making aimed explicitly at getting maximum output from whatever resources are committed to health care. The National Institute of Clinical Excellence (NICE) in the UK is perhaps the most widely discussed example of this approach. Williams (Reference Williams2004) describes NICE as ‘the closest anyone has come to fulfilling the economists dream’, while Smith (Reference Smith2004) argues that NICE may prove to be ‘one of Britain’s greatest cultural exports, along with Shakespeare, Newtonian physics, the Beatles, Harry Potter and the Teletubbies’.
Major limitations in the original NICE guidelines (or NICE 1) were identified indicating that the guidelines were unable to achieve the maximization of health gains from NHS resources and hence failed to meet the needs of NHS decision makers (Birch and Gafni, Reference Birch and Gafni2002; Gafni and Birch, Reference Gafni and Birch2003). More recently it has been argued that such guidelines are associated less with getting ‘the biggest bang for the bucks’ and more with getting ‘the biggest bucks for a bang’ (Birch and Gafni, Reference Birch and Gafni2006), generally a nightmare for both economists and decision makers. For example, an estimated £575 million additional NHS funding was absorbed by interventions recommended under the first 2.5 years of NICE 1 (Mayor, Reference Mayor2002). This is consistent with the consequences of the use of similar economic evaluation guidelines in other jurisdictions prior to the formation of NICE (e.g., Australia (Zinn, Reference Zinn2002) and Ontario, Canada (Laupacis, Reference Laupacis2002)). These cost escalations had been anticipated given the implicit assumptions on which the economic evaluation guidelines were based (Gafni and Birch, Reference Gafni and Birch1993; Birch and Gafni, Reference Birch and Gafni1993). Interestingly, attempts to highlight the implications of these implicit assumptions for health care decision making generally were dismissed (Johannesson and Weinstein, Reference Johannesson and Weinstein1993), while the identified limitations of the development and consequences of the application of NICE 1 guidelines for the NHS (i.e., Birch and Gafni, Reference Birch and Gafni2002) were heavily criticized by one group of eminent researchers with affiliations to NICE.Footnote 1
Following formal reviews of the NICE guidelines by the House of Commons’ Select Committee (2002) and the World Health Organization (2003), a revised set of guidelines was developed (referred to as NICE 2) using a ‘reference case’ intended to provide the template for submissions to NICE (National Institute for Clinical Excellence, 2004). In this paper, we consider whether NICE 2 can help turn the economist’s nightmare provided by NICE 1 into the economist’s dream.
Cost-effectiveness analysis and the elusive threshold
NICE 2 adopts a ‘reference case’ that specifies the methods considered ‘to be the most appropriate for the Appraisal Committee’s purpose and consistent with an NHS objective of maximizing health gain from limited resources’ (National Institute for Clinical Excellence, 2004, emphasis added). This constrained maximization focus is further justified as a consequence of NICE not being involved in setting the NHS budget and hence having to work within the policy context of available NHS resources.
Despite this policy context, NICE 2 adopts cost-effectiveness analysis ‘as a means to establish whether differences in costs between options can be justified in terms of changes in health effects’ with incremental cost effectiveness ratios (ICERs) providing the analytical tool on which recommendations are to be based. Affordability of the new technology is beyond the remit of NICE 2 (as it was with NICE 1). Although estimates of the net impact on NHS costs are required, this information is not used in the development of recommendations but is important ‘to allow effective national and local financial planning’. Given the mandatory requirement that health authorities make funds available from their existing budgets for implementation of NICE recommendations within three months of the recommendation’s publication, this simply provides a ‘heads up’ for authorities to search for cutbacks in their other activities in order to pay for new ‘cost-effective’ interventions.
The formal reviews of NICE 1 questioned the absence of any explicit cost-effectiveness threshold against which the estimated ICERs of new interventions could be compared. Although empirical evidence indicated an implicit threshold of £30,000 per QALY was being used (Towse and Pritchard, Reference Towse, Pritchard, Towse, Pritchard and Devlin2003), this was consistently denied by NICE (World Health Organization, 2002; Devlin et al., Reference Devlin, Parkin and Gold2003; Rawlins and Culyer, Reference Rawlins and Culyer2004). In contrast, NICE 2 addresses the issue of the threshold explicitly. It notes that under a fixed budget the threshold value consistent with maximizing health gains from available resources is the opportunity cost of the programmes displaced by the new more costly technology or the shadow price of the NHS budget constraint (Claxton et al., Reference Claxton, Sculpher, Culyer, McCabe, Briggs, Akehurst, Buxton and Brazier2006). Because (1) there is insufficient information to determine this shadow price, and (2) the shadow price is dynamic (problems previously identified by Devlin, Reference Devlin2002; Gafni and Birch, Reference Gafni and Birch2003), NICE 2 reverts to considering ICER values of other programmes that are currently funded as ‘a legitimate reference’. The implicit assumption is that something that is currently funded must be an efficient use of NHS resources (so past decisions were based on, or by chance resulted in, the efficient use of health care resources), hence anything that is relatively more cost effective (i.e., has a lower ICER) must also represent an efficient use of the same NHS resources and hence should be implemented though not necessarily instead of current programmes with the higher ICERs. NICE offers no advice about how recommended programmes should be implemented (i.e., the sources of the additional resource requirements), this being left to local decision-makers’ discretion.
Despite considering the use of a threshold to be ‘inappropriate’, NICE 2 goes on to specify two different thresholds, a lower ‘most plausible’ threshold of £20,000 per QALY and a higher (and presumably less plausible) threshold of £30,000 per QALY as a means of making judgements about whether the intervention represents ‘an effective use of NHS resources’ (emphasis added).Footnote 2 The NICE 2 decision criterion is as follows:
Interventions with ICERs below the lower threshold are judged primarily on their ICER value (implying they are recommended for adoption)
Interventions with ICERs between the two thresholds are judged not exclusively on their ICER value but also on other factors relating to uncertainty associated with the estimated ICER, the innovative nature of the intervention, particular features of the condition and populations being treated and the wider societal costs and benefits. In other words, they are unlikely to be recommended for adoption unless some of these features are judged to warrant paying more per unit health gain than the lower threshold.
Interventions with ICERs above the higher threshold require ‘increasingly strong’ justification beyond the ICER value. In other words, it is unlikely that even with these features the intervention would be recommended.
There is no justification regarding how the use of this decision criterion is related to the stated NICE 2 objectives. But the efficiency of use of available resources is not a matter of ‘judgement’, whether that be applied in the absence of explicit thresholds (as in NICE 1) or in the context of NICE 2 specific thresholds. As Cookson et al. (Reference Cookson, McDaid and Maynard2001) note:
In absence of any information about opportunity cost, however, [decision-makers] cannot attempt to achieve the efficient use of resources. (p. 743)
A positive ICER value in most cases means the resources currently used to support the current intervention for a particular condition or patient group are insufficient to support the new intervention. The efficiency of adopting the new intervention depends crucially on where the additional resources required to support the new intervention are to be taken from and at what opportunity cost. Simple threshold ICER values are insufficient to address the question of efficient resource use because they assume inter alia a stream of unlimited resources with constant opportunity cost available for additional investment in health care (Birch and Gafni, Reference Birch and Gafni1993, 2004, Reference Birch and Gafni2006; Gafni and Birch, Reference Gafni and Birch1993, Reference Gafni and Birch2006; Sendi and Briggs, Reference Sendi and Briggs2001; Sendi et al., Reference Sendi, Gafni and Birch2002).
This issue has been dismissed as one of affordability, something that is not considered by NICE when making judgements about cost effectiveness. Instead it is NICE’s job ‘to judge whether something ought to be purchased from within the resources made available to the NHS’ (Rawlins and Culyer, Reference Rawlins and Culyer2004). But that requires explicit consideration of the opportunity cost of the additional resource requirements for a new intervention – something the NICE 2 approach avoids. No substantive economics training is required to understand that matters of efficiency cannot be separated from matters of affordability. The dream home we have aspired to for so long might provide a higher benefit per unit cost than our current accommodation but if it costs significantly more than our current accommodation budget than that’s where it is likely to stay – in our dreams. As Williams (Reference Williams2004) notes, if affordability could be separated from efficiency there would be no need for a threshold.
The pursuit of equity
The National Health Service owes much of its existence to concern with equity in access to services among the UK population (A National Health Service, 1944). Hence, it is appropriate that guidelines aimed at informing NHS decisions about new health care interventions incorporate aspects of equity relevant to these founding principles. NICE 1 was not limited to ‘maximizing the health gain from the use of NHS resources’ but also involved ‘removing unfairness in the availability of technologies in different localities’ (National Institute for Clinical Excellence, 2001). However, NICE 1 did not discuss the implications of this equity consideration for the evaluation of interventions. Instead it appears that NICE 1 was concerned with the determination of efficiency (what services are to be funded to be efficient in the deployment of NHS resources), while equity was to involve determining the resource requirements to implement ‘efficient’ programmes in an equitable way (what resources are required to be equitable).
If providing a service represents an efficient use of available resources in each locality, then equity concerns become subsumed by the ‘efficiency’ question of maximizing health gains. If, however, the service represents an efficient use of resources in some localities but not in others, the decision-making problem is extended to one of maximizing health gain subject to multiple constraints; the total resources committed to NHS services and the requirement that, if a service is provided in one location, it must also be provided in all locations. NICE 1 did not address the (opportunity) cost of equity in terms of the reduction in total health gain associated with providing the service in all locations (Birch and Gafni, Reference Taylor, Hutton and Culyer2002, Reference Birch and Gafni2003). Instead, equity was to be pursued, whatever the cost, whether that involved increased NHS expenditures by diverting resources from other non-NHS uses, or by taking resources from elsewhere in the NHS.
Although the aims of NICE 1 restricted equity concerns to matters of geographical access to services, NICE 1 guidelines also incorporated equity considerations as they pertain to the variations in the costs of intervention by social group (e.g., where hospital stays vary in accordance with the availability of social support between social groups). In this case, NICE 1 required calculation of the weighting of benefits in the higher cost group needed to produce the same ICER as in the rest of the population. The appropriateness of the intervention was then based on a judgement whether this additional benefit weighting is warranted. This post hoc judgement involves no consideration of opportunity cost. Instead, services that might represent an inefficient use of resources are to be funded where decision makers judge them to be warranted on equity considerations. Additional resources, beyond those forming the resource constraint for the analysis of efficiency, are to be found from somewhere to support providing these services – irrespective of opportunity cost considerations. With no constraint on equity ‘weights’, those making the judgements are not operating within the constrained resources of the decision-maker’s problem.
NICE 2 has much less emphasis on equity considerations. The reference case pays no attention to promoting geographical equity in access to services, although the NICE citizens’ council has decided that local variations in cost ‘ought not result in variations in availability of health care’ (Rawlins and Culyer, Reference Rawlins and Culyer2004), thus maintaining the principle of ‘equity at any opportunity cost’ implied under NICE 1. Moreover, the reference case is based on the equity principle of a unit of outcome is of equal value, regardless of any other characteristics of the individuals benefiting from the intervention. This ‘neutral’ position on benefit valuation is based on ‘the absence of consensus on … differential weights being attached to QALYs’ and is subject to review following the results of ‘current research’. It also appears to recognize that equity and efficiency are not separable but part of the same analytical exercise, where the maximization of benefits is subject to any differential weighting in the valuation of those benefits.
However, the threshold-based decision criteria discussed above provide a potential route through which additional equity considerations are being accommodated. The other factors required to support adoption of interventions with ICERs greater than £20,000 per QALY include the particular features of the condition and the population receiving the intervention. So an intervention aimed at treating conditions primarily associated with poor circumstances but with an ICER of say £28,000 per QALY might be supported, whereas an intervention aimed at treating conditions primarily associated with a more affluent population but with the same ICER might be rejected based on the notion that it is in some way ‘fairer’ to prioritize interventions with a higher ICER in favour of poorer groups. But this implies weighting benefits to the poor greater than benefits to the affluent – at least for this particular intervention, even though the NICE 2 reference case rules out such differential weighting. Moreover, this post hoc use of equity as a decision criterion involves no consideration of opportunity costs and hence lies beyond the constrained maximization problem. The resources available to the NHS have no influence on these equity considerations and hence the decision outcomes bear no relationship to the decision context as defined by available resources and other opportunities for NHS resource use.
While it is beyond our remit to say whether NICE 2 should or should not use differential weightings for outcomes, it is important to emphasize that, if differential weightings are to be used, they must be used systematically through the specification of equity weights for different social groups within the objective function. As Williams and Cookson (Reference Williams, Cookson, Culyer and Newhouse2000) argue:
If the nature and implications of … equity principles are to be clarified in a policy relevant way, it is necessary to quantify what might otherwise merely remain vaguely appealing but ambiguous slogans. Only with some quantification will it be possible to convert them into criteria that can be applied in a consistent manner, and with a reasonable chance of checking on performance (i.e., holding people accountable).
In this way, if policy makers decide that the benefits to the poor warrant an equity weighting of 1.25, then this weight must be applied to the benefits accruing to the poor in all interventions, and the maximization of health gains from NHS resources be determined in the context of this weighting. Given that the additional resources to support new programmes (and hence the opportunity costs of those programmes) are determined by local decision makers, the policy makers equity weights must also be adopted in making these decisions. Only in this way do we ensure that the chosen equity criteria are applied consistently to the valuation of both benefits and opportunity costs.
Discussion
Any attempt to promote improvements in the efficiency of NHS resource use must be based on the explicit use of opportunity cost. Given the absence of opportunity cost considerations under NICE 2, the recommendations emerging from the application of NICE 2 guidelines are essentially value judgements about whether the estimated effects of an intervention are worth the estimated resource costs, in the opinion of those making NICE recommendations. But ignoring the underlying principles of economics undermines the economics basis of the recommendations which has led Sculpher et al. (Reference Sculpher, Claxton and Akehurst2004) to suggest that this research be seen less as economic evaluation and more as ‘just evaluation for decision-making’. Dropping the economic basis of the research has important implications for the outcome of the research. As noted above, the use of NICE guidelines (as well as similar guidelines in other jurisdictions) has been associated with a substantial increase in expenditures on interventions without proof of any net health gain. According to Evans’ (Reference Evans1984) notion of the Health Care ‘Income-Expenditure’ identity, it implies a substantial increase in the incomes of organizations that produce those interventions. Is this a dream or a nightmare? This might depend on where one stands in the health care system. For example, Morgan et al. (Reference Morgan, Barer and Evans2000) argued that manufacturers, given the opportunity ‘will influence what questions are asked and how’ and noted the increasing prevalence of ‘private for profit companies (many with high profile academic researchers as principals or sitting on Boards of Directors)’ that are specializing in serving the evaluation needs of manufacturers. They go on to note that ‘as economic models involve numerous assumptions that are subjective and often subtle, they are particularly susceptible to bias driven by conflicts of interest’. Certainly the absence of consideration of the opportunity costs of increasing NHS resources devoted to these interventions seems to be consistent with the interests of the ‘producers’. Williams (Reference Williams2004) noted that, ‘a high (ICER) threshold makes NICE more popular with the “sellers” (of interventions)’, while Cookson et al. (Reference Cookson, McDaid and Maynard2001) argue that ‘NICE has effectively become an advocacy mechanism by which lobbies of specialists and their supporters in the pharmaceutical industry extract more public money from the NHS. Instead of challenging the pharmaceutical industry to show value for money, NICE has become their “golden goose”.’ Even within the context of a fixed resource budget, NICE 2 involves the reallocation of resources towards the producers of those programmes recommended by NICE and away from other unidentified programmes.
Abraham (2002) has called for a pharmaceutical regulatory system that is capable of delivering ‘publicly defensible assessments which are uncompromisingly in the interests of public health’. He notes that following the Thalidomide incident in the UK, a report on the safety of drugs recommended that the expert committee charged with advising the UK regulatory agency on new drugs should be ‘entirely independent of industry’. He goes on to note that
As early as 1970, the Department of Health invited the ABPI to consider a policy whereby people holding consultancies in industry would not be appointed (to the expert committee) … nevertheless, the ABPI refused to lend support to the policy proposal and it was never implemented.
Abraham goes on to argue that expert advisors to regulatory agencies ‘should be required to suspend all conflicts of interest during their time in office’.
In response to concerns with potential conflicts of interest that we raised in our earlier critique of the NICE 1 guidelines (Birch and Gafni, Reference Birch and Gafni2002), some health economists have accused us of making inappropriate allegations about the motives and integrity of those involved in developing NICE's methodological guidance. It is worth noting that only three of the 14 members of the original NICE Guidance Steering Committee and none of the Methodology Working Party for NICE 2 listed any relationships with the pharmaceutical industry in the published guidelines documentation. For example, one of the members of the original Steering Committee (Taylor et al., Reference Taylor, Hutton and Culyer2002) did not list his affiliation at that time with the Office of Health Economics (Chairman of the Policy Board), an organization ‘supported by an annual grant from the Association of the British Pharmaceutical Industry (ABPI)’ (Towse et al., 2002).
Doubilet et al. (Reference Doubilet, Weinstein and McNeil1986) noted ‘there is no theoretical justification for asserting that the strategy with the lowest cost-effectiveness ratio is the most desirable one’. Torrance et al. (Reference Torrance, Thomas and Sackett1972), Drummond (Reference Drummond1980), and Drummond et al. (Reference Drummond, Torrance and Mason1993) noted that the efficient allocation of a fixed budget among competing programmes cannot be achieved using information on ICERs but requires the use of mathematical programming techniques. If NICE is to be the dream for economists and health care decision makers by helping to improve the efficiency of use of NHS resources, the economic methods used to appraise interventions must incorporate explicit recognition of the opportunity costs of new interventions (Birch and Gafni, Reference Birch and Gafni1992; Sendi et al, Reference Sendi, Gafni and Birch2002; Oliver, Reference Oliver2003). To do otherwise is to be dishonest about our discipline. But as Smith (2000) has noted:
Failures with honesty may lead to the ultimate failure of NICE, which could be the inability to say no except in obvious cases.
