2.1 Ex-ante evaluation

Just as a trial and/or simulation model would be undertaken before introducing a new treatment to the NHS, ex-ante modelling can be used to estimate the likely impact of the planned service change. This was the purpose of the impact assessments previously performed by the Department of Health. Ex-ante analysis should be used to select from potential service changes to be pursued, revealing those that could not possibly be cost-effective even under the most optimistic assumptions.

Ex-ante modelling can range from a relatively simplistic exercise estimating the maximum potential benefits of the proposed change in terms of QALYs to a detailed simulation model, and can serve a number of purposes. Such detailed microsimulation models are employed by the American Congressional Budget Office (2007) to examine the potential impacts of proposals to increase health insurance coverage.

Ex-ante modelling should form the initial stage of any planned service change, and could highlight how sparse the evidence base for many proposed programmes is. If the service intervention is targeted at patients with a particular condition, estimates of the current disease burden could be utilised to obtain estimates of the potential health gain. Evidence on the incidence of the condition or event in question could give potential effect size estimates, and be used to calculate the possible resource use implications.

If the budget set to be allocated to the programme is known, ex-ante modelling could be used to estimate the threshold of improvement needed for it to be cost-effective. Available literature on the expected effect size could then be compared to this to assess whether the service intervention is likely to produce benefits of that magnitude.

A recent ex-ante analysis of plans to increase weekend hospital services in England provides an example (Box 1). The service change was proposed in response to findings that the risk of mortality is higher amongst patients admitted to hospital at the weekend compared to those admitted during the week. Yet closer examination of the evidence revealed that even if the mortality rate experienced by patients admitted at weekends was reduced to that of patients admitted during the week, the costs of the service extensions would exceed the maximum NICE would recommend the NHS should pay to achieve a health gain of this size.

A recent ex-ante analysis of plans to increase weekend hospital services in England provides an example (Box 1). The service change was proposed in response to findings that the risk of mortality is higher amongst patients admitted to hospital at the weekend compared to those admitted during the week. Yet closer examination of the evidence revealed that even if the mortality rate experienced by patients admitted at weekends was reduced to that of patients admitted during the week, the costs of the service extensions would exceed the maximum NICE would recommend the NHS should pay to achieve a health gain of this size.

Yao et al. (Reference Yao, Novielli, Manaseki-Holland, Chen, Klink, van der, Barach, Chilton and Lilford2012) provide a framework for ex-ante evaluation of generic service delivery interventions. Their nine-step guide includes the identification of suitable end points, the use of expert elicitation in the absence of evidence on effectiveness, methods for deriving and synthesising data to input into an economic model, and calculations of cost-effectiveness. Notably, they suggest presenting the cost-effectiveness estimates in a number of different ways to aid decision-makers and reflect the uncertainty surrounding the estimates.

2.2 Evaluating the impact in terms of QALYs

The primary outcomes assessed in service evaluations are often intermediate process, clinical or patient safety measures. Re-admissions, length of stay and mortality rates are also commonly evaluated. Whilst these are important indicators of a programme’s success, the impact should also be measured in terms of QALYs. This allows decisions to be made over whether the service intervention should be funded based on comparisons on the same terms as other claims on health care resources.

Service evaluations commonly rely on administrative data, which often do not contain the necessary preference-based health-related quality-of-life measures needed to estimate QALYs. An exception is the English patient-reported outcome measures (PROMs) programme (Department of Health, 2008; Gomes et al., Reference Gomes, Gutacker, Bojke and Street2016). EQ-5D data are collected before and after surgery, enabling direct calculation of QALYs which could then be used in an evaluation of a service intervention targeting any of the four conditions covered.

In the absence of such data, we developed a method to estimate the impact of service interventions on QALYs indirectly using administrative data (Box 2). This method involved estimating the impact of the programme on mortality, and applying a discounted and quality-adjusted life expectancy (DANQALE) tariff to these mortality changes. Further developments have since been made to this method using survival analysis techniques (Meacock et al., Reference Meacock, Sutton, Kristensen and Harrison2016).

It is also possible to supplement administrative data analysis with economic modelling techniques, as demonstrated in an evaluation of stroke services centralisation (Hunter et al., Reference Hunter, Davie, Rudd, Thompson, Walker, Thomson, Mountford, Schwamm, Deanfield, Thompson, Dewan, Mistry, Quoraishi and Morris2013). Hunter et al. utilised audit data containing a condition-specific health measure, and derived utility values for health states based on this information using a mapping algorithm. They then built a Markov model to estimate the incremental cost-effectiveness of the service reorganisation using a combination of information available in administrative data and stroke audits.

These examples demonstrate that it is possible to obtain estimates of a programme’s impact in terms of QALYs in the absence of primary data collection, and similar methods should be utilised to a much greater extent. Further improvement in the methods to estimate QALYs in the absence of primary data collection would make a valuable contribution to this field.

2.3 Assessing costs and opportunity costs

As described by Maynard, although effectiveness measurement is essential, “it is like a cart without a horse if it is not matched up with cost data, which demonstrates how much care is given up when a procedure is adopted” (Reference Maynard2012: 10). Costing consists of three elements: identification, measurement, and valuation of the resources associated with each programme under consideration (Drummond et al., Reference Drummond, Sculpher, Claxton, Stoddart and Torrance2015). Economic evaluations conducted alongside trials can collect primary data on patient-level resource use. Identification and measurement of the relevant resources is often more complex when examining service interventions. The interlinked nature of health service organisation makes it difficult to assign resources to particular services, with many resources not being fully divisible. In addition, patient-level resource utilisation may not be available, with evaluators forced to rely on programme-level cost estimates. Top-down costing may be the only option, with micro-costing often unfeasible.

For the valuation stage, evaluations of service interventions can use national tariffs. Such tariffs represent average treatment costs across all patients and providers, with the marginal costs likely to differ from the provider’s perspective. However, from the commissioner’s perspective, these are the actual prices paid, and so represent the appropriate cost figures from this evaluation perspective. Administrative data can also be utilised to estimate impacts on costs through examining intermediate outcomes such as length of stay and re-admissions.

Drummond et al. (Reference Drummond, Sculpher, Claxton, Stoddart and Torrance2015) conclude that whilst cost analysis is a central feature of economic evaluation, it has received relatively little attention from analysts to date. Focus has been on developing appropriate statistical methods to deal with the distributional characteristics of cost data, with a neglect to the costing process itself (Graves et al., Reference Graves, Walker, Raine, Hutchings and Roberts2002). This is all the more true in service evaluation, where costs are often not considered at all.

As with any economic evaluation, the objective of placing a monetary value on the resources utilised is to obtain an estimate of their opportunity costs. In the case of health care spending, this represents the possible health gains foregone through not providing alternative treatments.

Opportunity costs are considered in economic evaluation using the cost-effectiveness threshold, which is taken to represent the health forgone elsewhere as other NHS activities are displaced (Claxton et al., Reference Claxton, Martin, Soares, Rice, Spackman, Hinde, Devlin, Smith and Sculpher2015). However, the current threshold of £20,000–£30,000 per QALY employed in England was founded on values implied by past decisions rather than evidence of the true opportunity costs experienced in the system. Recent work by Claxton et al. produced the first empirical estimates of the true scale of opportunity costs faced by the NHS in England when additional costs are imposed (Box 3). They estimated the cost-effectiveness threshold to be £13,936 per QALY, meaning that funding new interventions with an incremental cost-effectiveness ratio higher than this figure imposes a net loss on the system.

It is unclear what impact this work will have on the threshold used by NICE. Regardless, the research provides a methodology for estimating the true opportunity costs associated with funding decisions. Employment of this methodology in service evaluations would enable quantification of the magnitude of health forgone and an indication of where this opportunity cost is likely to fall, making the abstract notion of opportunity costs somewhat more tangible to decision-makers.

2.4 Accounting for spillover effects

Service interventions can have consequences beyond their intended effects. A change in one area may lead to unintended effort diversion away from other areas of care not covered by the initiative, having a negative impact. Alternatively, changes aimed at improving one area of a service may lead to improvements in other areas, having a positive impact. These additional effects are termed spillovers, defined as impacts on costs and/or outcomes other than those explicitly targeted by the intervention. Spillovers can occur within patients targeted by the intervention onto other untargeted areas of that patient’s care, across patients not targeted but treated within the same organisation, or across organisations. In contrast to opportunity costs, which represent the indirect impact on unidentified patients as a result of possible health care foregone elsewhere in the system, spillovers are direct impacts that can be identified by widening the evaluation’s scope.

These spillover effects introduce two additional complexities into evaluations of service interventions. First, it may be necessary to look beyond the intended outcomes of a programme to assess the full impact of the service change. Spillovers could alter the cost-effectiveness decision if additional benefit or cost consequences occur beyond the intended scope. Second, the potential for spillovers requires consideration when selecting an appropriate control group in a non-experimental study as it is vital that this group is truly unaffected by the intervention.

The Quality and Outcomes Framework (QOF) introduced financial incentives for areas of care delivered by GPs, providing an example of where spillovers may be relevant. GPs received incentives for recording risk factors including smoking status for patients with chronic diseases. Sutton et al. found evidence of substantial positive spillovers, with recording of clinically effective but un-incentivised risk factors including alcohol consumption increasing for patients who were targeted by the QOF (Box 4). As the QOF paid GP practices for the recording of risk factors, the inclusion of these positive spillover effects reduced the cost to commissioners per risk factor recorded by a factor of two.

Leaving spillovers out of economic evaluations could distort conclusions about a service intervention’s cost-effectiveness, leading to allocative inefficiency. Further research is required to develop a systematic approach to the identification and measurement of spillovers in service intervention evaluations (Kristensen et al., Reference Kristensen, Meacock and Sutton2015).

2.5 Generalisability

Whilst it may be of some value to know whether a particular service intervention represented a cost-effective use of resources in the local setting within which it was implemented, once the change has been made there may be little ability to reverse it. Furthermore, the effectiveness of a service is largely determined by contextual factors, which will differ between organisations (Turner et al., Reference Turner, Goulding, Denis, McDonald and Fulop2016). Of greater use to decision-makers is generalisable evidence on service changes and the potential impacts of scaling and spreading interventions found to be cost-effective. This poses a question different from that addressed in a typical HTA, of whether option A is cost-effective compared to option B.

Whether a service intervention is cost-effective is the simpler question to answer, as this requires no understanding of the production process, but simply an estimate of the treatment effect and costs associated with the programme (Shiell et al., Reference Shiell, Hawe and Gold2008). However, understanding which elements of a programme generate positive effects, and how these could be implemented elsewhere, is far more informative (Lamont et al., Reference Lamont, Barber, Pury, de, Fulop, Garfield-Birkbeck, Lilford, Mear, Raine and Fitzpatrick2016). This requires an understanding of a programme’s underlying causal mechanism, or ‘active ingredients’ (Medical Research Council, 2008).

It may not be possible to unpick the causal mechanisms behind an intervention using quantitative analysis alone. A mixed methods approach is likely necessary, combining quantitative measurement of costs and benefits with a qualitative exploration of how and why impacts are generated (Turner et al., Reference Turner, Goulding, Denis, McDonald and Fulop2016). A recent paper proposes a means by which economic evaluations could be combined with theory-driven realist evaluations to better explain how service interventions work, who they work for, in what circumstances, and why (Box 5).