In most developed healthcare systems, the public sector devotes significant resources both to healthcare services and to health services research. Within the United Kingdom for example, in 2000, expenditure on the National Health Service (NHS) was £69.2 billion, 81% of which was public money (31). In the United States, public expenditure on Medicare and Medicaid amounted to $259.1 billion in 2002 (6). Public funds are also spent on research and development as opposed to health care per se. For example, in 2000, the UK Department of Health spent £500 million on research (22). In the United States, $18.8 billion was spent on research by the federal government (42).
Given that an important objective for any healthcare system is to maximize some measure of health gain from available resources, the benefits generated by all resources, including those that are earmarked for research, have to be maximized. The objective of achieving efficiency in health-related research, as well as the provision of services, will need to address issues such as: Which clinical areas should receive research resources? Which type of research should be undertaken (e.g., randomized trials or observational studies)? What end points should be measured? What are the appropriate sample sizes for studies? This question of allocation between provision of services and research and development clearly is relevant from a public sector or healthcare system perspective, but it is also important if a broader societal view is taken that would include the costs and benefits to all sectors of the economy in providing health care and conducting research.
Methods to establish the value for money of alternative health care technologies are well established. Cost-effectiveness analysis is the most widely used form of evaluation (37). The expected additional cost of one technology over another is compared with expected additional health outcomes, typically expressed in terms of quality-adjusted life years (QALYs) (i.e., the incremental cost-effectiveness ratio [ICER]) (28). If the resultant ratio falls below a prespecified threshold value (which may be an administrative “ceiling ratio,” an empirically based measure of society's maximum willingness to pay for additional health gains or an explicit shadow price of a budget constraint [46]), then the technology can be regarded as cost-effective, contingent on the information currently available.
Analysis such as this feeds into the decision-making process regarding which healthcare activities should be provided by the healthcare system. For example, in the United Kingdom, the National Institute for Health and Clinical Excellence (NICE) carries out technology appraisals of specific healthcare interventions, medicines, devices, and procedures to establish their clinical and cost-effectiveness. On the basis of these appraisals, NICE issues guidance to the NHS on the most appropriate use of resources within a defined scope (i.e., disease area, group of patients, etc.). Similar agencies exist to make technology coverage decisions in other jurisdictions (25).
The appropriate methods to inform the allocation of resources for research are less clear. Several methods have been proposed and some have been used successfully to identify priority areas for research. These methods include measures of the burden of disease or the technology (30;45), measures of the expected “payback” from research (5;15;35), and estimates of the welfare losses due to variations in clinical practice (39). However, each of these approaches has methodological problems. First, they view research as a means of changing clinical practice and not as a means of providing additional information to reduce the uncertainty about what is appropriate clinical practice. Indeed, measures of “payback” or welfare losses due to variations in clinical practice require the analysis to identify “appropriate utilization” or which technology should be adopted a priori. Therefore, these methods implicitly assume that there is no uncertainty surrounding the decision that the proposed research is supposed to inform. Second, these approaches, particularly measures of disease burden, attempt to identify research priorities using aggregate measures across broad clinical areas. However, the information generated by evaluative research is only valuable if it informs specific clinical decisions for specific groups of patients. The value of research in a clinical area is simply made up of the value of research about each of the constituent clinical decision problems faced within that area. Therefore, if aggregate measures such as burden of disease suggest a clinical area is a “high” priority, it does not necessarily mean that specific evaluative research relating to any one clinical decision problem will be valuable. Similarly, proposed research to inform a particular decision in a “low” priority disease area may be very valuable. For this reason, attempts to identify research priorities across broach clinical areas using aggregate indicators may be erroneous.
To inform research priority setting, a measure of the societal value of resolving a particular research question is required. This finding can inform specific clinical decisions for defined groups of patients. An appropriate methodological framework should consider the uncertainty surrounding the adoption of a health technology in terms of the likelihood of making a wrong decision. It should also view the value of research as the extent to which further information will reduce that decision uncertainty. Given that funding for research and healthcare provision ultimately comes from the same budget, the opportunity cost of spending on research can be seen in terms of forgone funding for health care per se. This means that decisions regarding research must be addressed in a manner that is consistent with the way in which decisions are made regarding healthcare provision. An appropriate framework, therefore, should value the additional information generated by research in a way that is consistent with the objectives and the resource constraints of healthcare provision.
Value of information (VOI) analysis offers a methodological framework that explicitly considers the uncertainty surrounding the decision by a healthcare system to adopt a health technology (11). This framework values the additional information, which may be generated by further research, in a way that is consistent with the objectives and the resource constraints of heathcare provision (the cost-effectiveness threshold). This consistency allows a comparison of the potential benefits of further research with the costs of further investigation. If the costs of investigation exceed the benefits, then the proposed research will not be cost-effective.
Within this study, we present a practical application of VOI analysis and show how it offers a powerful tool to guide research decisions within a given area. We apply the methodology to a specific decision problem: the use of glycoprotein IIb/IIIa antagonists (GPAs) for the treatment of patients with non-ST elevation acute coronary syndrome (ACS). This treatment is an area that twice has been appraised by NICE (21;29;40), who have accordingly offered specific guidance to the UK NHS and have made recommendations for further research (33).
VALUE OF INFORMATION ANALYSIS
VOI analysis is founded in statistical decision theory (36;38;39) and has been successfully used in other areas of research such as engineering and environmental risk analysis (24;26;44). Although its use in health care has been set out formally by several researchers (for example, see references 1;7;8;10;11;17;20), currently, there are few practical applications of the technique. In a recent comprehensive review of the use of the VOI technique in health risk management, Yakota and Thompson (47) identified a total of forty-four applications, eighteen of which were within the field of medical care.
The decision to adopt a technology based on current evidence is conceptually separate from the decision to acquire more information by conducting further research. The adoption decision can be made using methods analogous to those described above and by converting incremental ratios into net benefits (NB) (10,43). This is done by valuing health gains in money terms using the threshold for cost-effectiveness. The technology offering the highest expected net benefit on the basis of information currently available offers the best value for money and should be regarded as cost-effective (20).
The decision to acquire more information should be based on the consequences of the uncertainty surrounding a decision to adopt a technology given current information. It is inevitable that the information used to calculate costs, outcomes, and net benefits associated with alternative healthcare technologies will be subject to uncertainty. There will be a chance that a decision based on current information will be “wrong,” and there will be opportunity costs in the form of resources and health gains forgone. These expected costs of uncertainty can be reduced by acquiring further information through research. It is this reduction in the cost of uncertainty that is the value of information.
Probabilistic sensitivity analysis can be undertaken to establish the uncertainty associated with the decision to adopt a technology. This requires assigning a distribution to each parameter within the model to reflect the uncertainty associated with its mean value and using Monte Carlo simulation to propagate this uncertainty through the model. This approach provides a large number of simulations of cost, effect, and net benefit for each technology under evaluation (3).
The expected value of perfect information (EVPI) can be calculated directly from the simulation output. The opportunity loss is the difference between the NB of the technology chosen to be cost-effective on the basis of expected net benefit across all iterations and the NB of the technology that offers maximum net benefit for that particular iteration or realization of uncertainty. Averaging these opportunity losses across all iterations gives the expected cost associated with existing decision uncertainty. This cost is the maximum payoff for a single patient from any amount of information and, therefore, provides an estimate of the EVPI (formal notation of the EVPI calculations can be found at http://www.york.ac.uk/inst/che/pdf/priority.pdf and references 1;12).
However, information is a public good (i.e., if it is available to inform the management of one patient, it is available for that purpose in all patients). By multiplying the individual EVPI by the expected population of patients who will benefit from the information, the maximum value of information derived from future research can be quantified. This requires an estimate of the period of time over which the information would be beneficial, the number of patients affected within this period, and discounting to present values (10).
The EVPI provides a maximum value against which the costs of research may be compared. If the costs of the proposed research exceed the EVPI, the proposed research cannot be regarded as cost-effective. Moreover, the EVPI associated with particular parameters or groups of parameters such as utility estimates, treatment effects or costs, can also be calculated. This EVPI for parameters (EVPPI) (1) provides some indication of the type of future research that would be most beneficial and could be considered cost-effective. For example, if the EVPI for parameters subject to selection bias such as relative treatment effect is high, a randomized controlled trial may be required. However, if EVPI is associated with other parameters relevant to natural history, costs or quality of life, then other research designs may be more appropriate. The methods for calculating EVPPI are analogous to calculating EVPI (formal notation is provided at http://www.york.ac.uk/inst/che/pdf/priority.pdf and references 1;12).
THE CASE STUDY
The Disease and Interventions
The case study relates to the use of GPAs, which are a class of drug used to prevent platelet aggregation in the acute treatment of patients with non-ST elevation ACS. The aim of these drugs is to reduce the risk of cardiac death and acute myocardial infarction (MI). Within the UK, two broad groups are licensed: abciximab (ReoPro, Eli Lilly) is a monoclonal antibody targeted at the receptor; whereas eptifibatide (Integrilin, Schering Plough) and tirofiban (Aggrastat, MSD) are more conventional pharmacological receptor antagonists. GPAs are used in two ways to manage ACS patients: either as an adjunct to percutaneous coronary interventions (PCI; e.g., angioplasty) or as part of medical management regardless of whether they go onto to have a PCI. Abciximab is licensed currently as an adjunct to PCI, whereas tirofiban and eptifibatide are licensed for use only in medical management. Further details of the disease can be found at http://www.york.ac.uk/inst/che/pdf/priority.pdf.
The Model
Full details of the model has been published elsewhere (34,41) and summary details of the model structure and input parameters are provided at http://www.york.ac.uk/inst/che/pdf/priority.pdf. A decision-analytic model was developed to synthesize the available evidence regarding the effectiveness and costs of GPAs in comparison with usual care. The purpose of the model was to assess the cost-effectiveness of the use of these drugs in the UK in patients with non-ST elevation ACS and to establish the value of further research in this area. A lifetime perspective was adopted, benefits were measured in terms of QALYs, and costing was carried out from the perspective of the NHS. Three GPA-based strategies in comparison with usual care were evaluated to represent the full range of possible approaches to using GPAs in the United Kingdom: GPAs as part of initial medical management (Strategy 1); GPAs in patients with planned PCI, where GPAs are started once a decision to undertake PCI has been made (Strategy 2); GPAs as an adjunct to PCI, where the agent is used at the time of PCI or is started up to 1 hour before the procedure (Strategy 3); and no use of GPA (usual care; Strategy 4).
Probabilistic sensitivity analysis was used to evaluate the impact of parameter uncertainty on the adoption decision. In addition, several scenario analyses were undertaken to evaluate uncertainty associated with several structural and data assumptions. These included variations in the sources of data used to populate the model (e.g., alternative relative risk estimates, non-UK sources for baseline risk), the inclusion of an additional strategy to reflect a potentially relevant comparator to the GPAs and a risk-based subgroup analysis. Full details of the results of the sensitivity and scenario analyses are published elsewhere (41). For this study, we focus on two of these analyses for the purpose of illustration. We examine, first, the effect on EVPI of evaluating an additional management strategy—clopidogrel, which has been shown to be an effective treatment for these groups of patients (14), in addition to the three GPA-based strategies. Second, we perform a subgroup analysis and evaluate the impact of treating patients with GPAs who are defined, a priori, as being at either high- or low-risk of future cardiac events. Data for each of the sensitivity analyses were taken from published sources (2;4;14;23).
VOI analysis was undertaken for the base-case model and for the two sensitivity analyses described above. In each case, the EVPI was estimated for the full model and for groups of parameters within it (EVPPI). The parameters were grouped according to the specific type of study that would be required to obtain further data on them. A range of scenarios is presented to reflect different assumptions related to the period of time over which the information would be beneficial (between 5 and 15 years) and for different threshold values for the ICER (between £10,000 and £50,000 per QALY). Population level EVPI values were estimated using an estimated annual incidence of 59,756 (32) and a 6% rate of discount (16).
RESEARCH RECOMMENDATIONS FOR THE USE OF GPAS FOR PATIENTS WITH NON-ST ELEVATION ACS
Adoption Decision
The cost-effectiveness of the alternative strategies is compared using standard decision rules (27). Table 1 details the expected cost and QALYs and ICERs for the base-case model and for the two scenarios modeled under sensitivity analyses. For the base-case analysis and clopidogrel sensitivity analysis, Strategy 2 is not cost-effective because it is more costly and less effective than Strategy 3 (i.e., dominated) and Strategy 3 is not cost-effective because of extended dominance (28). Moreover, in the clopidogrel sensitivity analysis, clopidogrel is also not cost-effective due to extended dominance. Therefore, under both of these scenarios, Strategy 1 (GPAs used as part of medical management) results in an ICER of around £5,700 per additional QALY in comparison with standard care (no GPA).
The results differ in the scenario analysis involving risk subgroups. Here, Strategy 1 remains cost-effective for those defined as high risk (Strategy 2 and 3 are dominated), with an ICER of £3,890 when compared with Strategy 4. However, for the low-risk group, Strategy 1 is dominated and the ICER of Strategy 3 (GPA in conjunction with PCI) compared with Strategy 4 is £800,000, which suggests that Strategy 4 (no use of GPAs) is likely to be considered the most cost-effective form of management.
VOI Analysis
Population EVPI
For the base-case model, the population EVPI ranges from £11.46 million (over 5 years assuming a threshold value of cost-effectiveness of £30K) to £35.56 million (15 years, threshold = £50K) depending on assumptions regarding the lifespan of the technology and the value of the threshold. If the lifespan of the technology is 10 years and the threshold for cost-effectiveness is £30,000, the EVPI is £20.032 million (EVPI per episode at this threshold is £43; Table 2). In both sensitivity analyses, the EVPI values change considerably. Figure 1 presents the population EVPI values under base case assumptions and for the two sensitivity analyses assuming a 10-year lifespan for the technologies. For the base case, clopidogrel and high-risk subgroup analyses, EVPI increases up to a local maximum at the point where the threshold value (λ) is equal to the value of the ICER for the adoption decision of Strategy 1 relative to Strategy 4 (e.g., £5,738 per additional QALY for the base case). Up to this value, EVPI is increasing. This is because the uncertainty surrounding the adoption decision is increasing (error probability increasing), as is the value applied to the consequences of making an incorrect decision (i.e., λ). After this point, the uncertainty in the adoption decision decreases. Whereas uncertainty surrounding the adoption decision beyond this point begins to fall, the consequences associated with making an incorrect decision continue to rise. The overall effect on the EVPI depends on the interaction between these terms. For the base-case model, as the threshold value approaches £18,000 the EVPI falls, implying that the probability of an incorrect decision is reducing at a rate that is sufficient to offset the increasing costs of making an incorrect decision. After this point, the EVPI increases as the threshold increases, demonstrating that, although the error probability is still falling, this change is now being outweighed by the costs of making an incorrect decision.
Population expected value of perfect information (EVPI) for the base-case and sensitivity analyses models.
The pattern of EVPI is different for the low-risk subgroup analysis. Under this scenario, population EVPI values are negligible at ceiling ratios below £6,000; at these low threshold values, the level of decision uncertainty and the consequences of that uncertainty are both low. After this point, EVPI continues to increase. This finding is because, although there is a small probability that each of Strategies 1–3 are cost-effective, this probability increases at higher values of λ (i.e., there is nonnegligible error probability), and the consequences of the decision uncertainty are increasing.
Population EVPPI
Table 2 details the population EVPPI values for groups of parameters, under an assumed lifespan for the technologies of 10 years. Under base-case assumptions, it is clear that all of the uncertainty is encapsulated within the relative risks associated with Strategy 1. This finding suggests that further research would be most beneficial if it were directed toward obtaining better estimates of the relative treatment effects of GPAs used as medical management. A similar pattern is observed in the high-risk sensitivity analysis. When clopidogrel is added as a further management strategy, there are positive and similar EVPI values for both the relative risks associated with Strategy 1 and the relative effects of clopidogrel. Further investigation of the individual parameters making up each relative risk group identified that, in each case, the relative risk of death in patients not undergoing an acute PCI accounts for all decision uncertainty.
The low-risk sensitivity analysis yields interesting results as it is in this group that the adoption decision changes from the base-case analysis. Although, with current information, the optimum decision is to continue with usual care (Strategy 4), the VOI analysis suggests that, for a threshold value of £10,000 per additional QALY, there is value in obtaining further information on baseline risk and long-term outcomes, in addition to further information regarding the relative benefits of GPAs used as medical management.
Research Recommendations
The EVPI for the base-case model indicates there may be considerable value in commissioning further research to reduce the uncertainty associated with the adoption decision. Using base-case assumptions, EVPI is between £47.71 and £57.81 per patient for threshold values between £10,000 and £50,000. Translating this finding to a population figure, the EVPI is between £20 million and £26.9 million, assuming the lifespan of the technology is 10 years. This value represents an upper limit on the costs associated with the decision uncertainty; thus, any costs associated with any proposed further research should not exceed this amount if this research is to be considered efficient.
EVPI is driven exclusively by the relative risk of death in patients not undergoing an initial PCI in Strategy 1. This would suggest that future research should be directed toward reducing the uncertainty associated with the relative risk of death in ACS patients who are prescribed GPAs and who do not undergo a PCI procedure in the acute phase. This is not entirely surprising because, at the time of the analysis, approximately 95% of patients did not receive a PCI during their initial episode.
When clopidogrel is included as a treatment option, Strategy 1 remains the optimal decision but EVPI increases. This finding is because there are only small differences in cost and outcome between the clopidogrel option and Strategy 1. Although Strategy 1 remains the optimal adoption decision, the inclusion of clopidogrel results in a significant increase in the level of uncertainty surrounding the adoption decision itself, because there is a .33 probability that clopidogrel is cost-effective. This additional uncertainty surrounding the adoption decision explains the high EVPI. The results of this scenario suggest that further research to identify the relative benefits of clopidogrel and GPAs as part of medical management, and compared with the current service provisions, would be of benefit.
When the model is run for separate subgroups, we see a change in the adoption decision depending on risk. For high-risk patients, the optimal decision remains Strategy 1 and the pattern of EVPI results mirror the base case model, although the magnitude of the values per patient increase. For low-risk patients, although the adoption decision changes, the decision uncertainty remains. It is clear that, for low-risk patients, there is value in obtaining further information regarding both their baseline risks of death and nonfatal MI and the relative effects of treatment.
DISCUSSION
As the demand for health care increases in developed countries, so too does expenditure on research and development on new technologies to satisfy this demand. After 25 years of active research into the methods and application of cost-effectiveness in health care, it is now widely accepted that these methods should be used to inform decisions about which technologies should be reimbursed within collectively funded systems. Given the volume of resources at stake, the complexities and uncertainties involved and the inevitable need to set priorities, it is now essential to use similar analytic frameworks to identify the most cost-effective areas for research. Value of information analysis provides a set of methods for research prioritization, which are consistent with the methods of cost-effectiveness analysis for technologies and which have a firm theoretical underpinning in statistical decision theory (36;38;39).
An important feature of value of information analysis is that the potential value of additional research is assessed assuming that clinicians will undertake practice that research indicates is the most cost-effective. In other words, there is no assumed gap between practice identified as optimal and actual practice. The rationale for this assumption is that the issues of what is appropriate (i.e., cost-effective) clinical practice and how such practice is implemented are quite separable. Value of information methods are focused on the potential value of additional information in reducing decision uncertainty; implementation is concerned with the polices used to get practitioners to undertake practice that research identifies as optimal. Research currently is under way to develop methods that formally consider the value of additional information through research and the potential value of implementation interventions within the same analytical framework (18;19).
The glycoprotein model used in the case study was developed to inform the NICE appraisal of those therapies. In September 2002, NICE issued guidance that, among other things, GPAs should be considered as part of medical management for patients with unstable angina or non-ST segment elevation MI (33). The guidance notes recognized that there was considerable uncertainty associated with the evidence base in this area and made some recommendations for further research. These recommendations included research to evaluate the effects of GPAs in current UK practice in non-ST segment elevation ACS patients who are not scheduled for PCI, and the efficacy of GPAs in subgroups such as women. There were also recommendations for research to establish the relative roles of GPAs and clopidogrel in the short-term management of ACS patients and into clinical risk factors that could be used in treatment allocation.
These research recommendations, as for all other NICE technology appraisals, were not based on formal VOI analysis (the VOI analysis in the case study was undertaken after the appraisal was completed). Rather, they are based on the Appraisal Committee's understanding of the major gaps in the evidence. Comparing NICE's research recommendations with those suggested by the VOI analysis presented here indicates that they are broadly consistent. However, the VOI analysis overall and for groups of parameters has the potential to inform much more detailed recommendations. For example, the case study suggests that the key area of uncertainty relates to the relative effectiveness of GPA versus standard medical management, and also in comparison with clopidogrel.
Two general issues are highlighted by this analysis. These issues relate to the impact of scope and patient heterogeneity. Whereas the impact of broadening the scope of the analysis for the case study presented here did not change the initial adoption decision, it had marked impact on the potential value of future research. Moreover, taking account of the heterogeneity between patient groups in this case study showed clearly how both the adoption decision and the type of future research required can differ between groups. If the VOI approach is going to be used in research prioritization then it is essential that the scope of the analysis is sufficiently broad to encompass all relevant treatment alternatives and that the impact of heterogeneity is reflected in both the adoption decision and the decision to conduct further research.
It is important to emphasize that the EVPI represents a maximum value of additional research. As such, it represents a necessary, rather than a sufficient, condition for future research. As long as the cost of a given research project is less than the EVPI, there is at least a potential for it to represent an efficient use of resources. To establish a sufficient condition and decide if further research will be worthwhile and identify efficient research design, we need to consider the marginal benefits and marginal cost of sample information. The same framework of value of information analysis can be extended to establish the expected value of sample information for particular research designs and to compare these expected benefits of research to the expected costs (1). This type of analysis provides a societal payoff to alternative designs and can be used to establish optimal sample size, optimal allocation of patients within a clinical trial, appropriate follow-up, and which end points should be included in the design. Indeed this framework can be used to identify a portfolio of different types of studies that may be required to provide evidence sufficient to support the use of a healthcare technology (9;13).
POLICY IMPLICATIONS
This study has presented the application of value of information methods to a policy relevant decision problem recently faced by the UK's NICE. What is the policy relevance of this work? The characteristics of the GPA decision problem are consistent with most of those faced by health systems internationally: limited efficacy (as opposed to effectiveness) data from short-term trials largely undertaken to license pharmaceuticals provide a highly uncertain basis to determine the cost-effectiveness of the product and, hence, its appropriateness for funding/reimbursement. The use of decision modeling and value of information analysis allows health systems to establish (i) whether a technology is cost-effective, given existing (often limited) evidence; and (ii) the type and extent of additional evidence that needs to be gathered to support potential revisions of the decision in the future. As such, the methods provide a coherent framework within which to address the question—When do we have sufficient evidence to support the use of a new technology?
CONTACT INFORMATION
Zoë Philips, PhD (Zoe.Philips@nottingham.ac.uk), Lecturer, School of Economics, University of Nottingham, Sir Clive Granger Building, University Park, Nottingham NG7 2RD, UK
Karl Philip Claxton, DPhil (kpc1@york.ac.uk), Senior Lecturer, Department of Economics and Related Studies, University of York, Heslington, York YO10 5DD, UK
Stephen Palmer, MSc (sjp21@york.ac.uk), Senior Research Fellow, Laura Bojke, MSc (lg116@york.ac.uk), Research Fellow, Mark John Sculpher, PhD (mjs23@york.ac.uk), Professor, Centre for Health Economics, University of York, Alcuin A block, Heslington, York YO10 5DD, UK
