The rapidly increasing range and expense of new medical devices has created a growing need to demonstrate that a new product is superior to an existing one in terms of “value for money.” Although, formally, demonstrating safety and performance of a new device is sufficient to receive CE-marking (European Conformity), companies are under rising pressure to articulate the value of their products in terms of their incremental cost-effectiveness. However, as conducting health economic evaluations is not a core activity of most medical device companies, especially not the smaller ones, such an evaluation, if undertaken at all, is usually done as a one-off exercise at the late stage of development of new technologies. Nevertheless, the importance of beginning to estimate cost-effectiveness at an early stage of technology development, and to iterate this evaluation as the development progresses, is increasingly acknowledged, and recent examples of this are reported in the literature (see, e.g., Reference Dong and Buxton4;Reference Girling, Freeman and Gordon6;Reference van Til, Renzenbrink, Groothuis and Ijzerman18).
The benefits of an iterative estimation of the cost-effectiveness of new technologies have previously been described by Sculpher et al. (Reference Sculpher, Drummond and Buxton12), suggesting a four-stage economic evaluation that starts with an early assessment of the technology, and as the development and testing progresses, further analysis is undertaken that allows more confidence to be attached to the cost-effectiveness estimates. Warburton (Reference Warburton19) later proposed what she called “the economic evaluation loop,” including three steps, where the first one is an initial estimate of costs and benefits of each strategy; the second step explores the potential impact of uncertainty to target future primary research which will be conducted in step three. This process is repeated until the conclusions are considered to be consistent under all plausible assumptions.
More recently, the use of Bayesian methods in health technology assessment (HTA) has been reviewed for the NHS HTA program (Reference Spiegelhalter, Myles and Jones14). Bayesian methods allow for existing evidence, knowledge or beliefs about a parameter, formally expressed as a probability distribution, to be updated by new information as it becomes available from further studies, making explicit and quantitative use of all information available at that point. The NHS-HTA report has emphasized the use of a Bayesian framework for making predictions, synthesizing evidence, designing trials, and evaluating the value of further information in a mainly theoretical way. The value and practicality of the implementation of Bayesian methods within the dynamic framework of HTA has been illustrated by Fenwick et al. (Reference Fenwick, Palmer and Claxton5), evaluating the cost-effectiveness of different preoperative patient management strategies before major elective surgery. As such, their example related to a situation where the innovation is fully developed for use in practice and updating of evidence involves the accumulation of hard trial data
There might be, however, a more commercial application of an iterative Bayesian approach for medical devices. Given that these are often fast-changing technologies (i.e., the life cycle of a specific type or variation of a device is often as short as 18–24 months, considerably less than that of pharmaceuticals for example) and their development is characterized by a constant flow of incremental product improvements, making early and rapid assessments of their likely cost-effectiveness is of particular importance. Not only to help companies reduce their failure rates, but also to help ensure that patients and other users of medical devices gain access to the most beneficial technologies as fast as possible. The iterative Bayesian approach might support this in three ways: (i) by allowing the estimation of potential cost-effectiveness to be part of the investment decision process and to avoid investing in a technology that could never be cost-effective; (ii) by supporting companies to prioritize between several competing possibly cost-effective concepts or prototypes; (iii) by identifying from early stages of development those parameters that have the largest impact on the likely cost-effectiveness of the product to direct scarce research resources. Whether or not a company should fund additional research to reduce uncertainty relating to the reimbursement decision can be informed by means of value of information (VOI) analysis, but this needs to be adapted to inform the value of that further information to the company, rather than the standard value to society of reducing uncertainty.
The objective of this study is to show how this iterative Bayesian approach can be used to inform early decisions regarding medical devices in situations where both the availability of data as well as time to perform the analyses may be relatively scarce (e.g., as compared to the study of Fenwick et al.). Although limited data and scarce analytical resources may impose constraints to the level of sophistication of the analyses to be performed, the applicability of the underlying theory stills holds and we suggest to start with relatively simple health economic analysis from the very early stages, gaining greater depth of analysis in later stages as more information becomes available.
APPLICATION OF A COMBINED ITERATIVE AND BAYESIAN FRAMEWORK TO THE MEDICAL DEVICE INDUSTRY
Medical device manufacturers generally start off the device development process with a concept or prototype product and then make a ‘business case’ for it, which includes costs of development, resource requirements and justification for the project. Because the route to commercialization of a device is complicated by regulatory and reimbursement approval requirements, most medical device companies operate some form of staged decision-making development process that is regularly reviewed and decisions are taken as to whether and how to proceed. At each stage, the decisions are based on a mix of objective and subjective information regarding costs and potential future benefits of the device. Whereas attempts are made to remain as objective as possible, much early decision making is based on beliefs, and in the vast majority of cases no formal economic evaluation takes place until the late phase in the development of the technology.
The proposed methodology fits into such a gate-staged decision process. In this study, a simplified three-stage process is considered to assess the potential cost-effectiveness of the new technology. We take a collective view to the economic evaluation where costs are those incurred in the healthcare system and benefits include all health effects on the individual (8), as this is likely to be the perspective which will inform the eventual reimbusment decision. In the early phase, most of the analysis is based on elicited beliefs and plausible assumptions about the effects and costs of the new technology; in the mid-phase some evidence would be available that will update our prior beliefs from the early stage and can be incorporated into a decision model; in the final phase, all available evidence can be formally synthesized to inform a decision that will not necessarily ignore previous evidence.
Early Stage
An early stage is usually characterized by limited availability of clinical and economic data and, therefore challenges the applicability of standard health economic methods to inform the decisions at hand. At this stage, the company also typically faces having to choose between a possibly large number of alternative directions to take forward the development of a particular technology, that is, the application of a device to different types of interventions, different clinical settings, or different clinical indications. In the absence of data regarding the new technology, the analysis should be based on the available evidence concerning the current technology that the new device aims to substitute or will compete with, and expert opinion and/or assumptions regarding the likely impact on cost and effectiveness of the new device. Then, the expected incremental effectiveness of the new technology can be estimated at a given willingness to pay per effectiveness unit. For example, using the effectiveness measure of quality-adjusted life-years (QALYs), the established willingness to pay per QALY lies between £20 and 30k in the UK as proposed by NICE (8). This approach has been described in the literature as the “effectiveness gap” (Reference Sculpher, Drummond and Buxton12) or “headroom method” (Reference Cosh, Girling and Lilford3). When the company has worked out the potential cost of the new technology, this can be compared with the results of this headroom analysis that would place a bound on the maximum reimbursable price.
An example of the latter is the exploration of the likely cost-effectiveness of tissue engineered bladder substitute as an alternative to the use of bowel in substitution cystoplasty after resection of cancer (Reference Cosh, Girling and Lilford3). In the absence of QALY values data, an estimate of the disutility of current practice was made by means of a survey of urologists. A median utility value of 0.95 after cystoplasty was found versus an estimated utility of 1.0 after having a tissue engineered bladder (which represents the most optimistic scenario). Assuming a mean of 10 years of life expectancy among the group that present this condition, this provides an estimate of the incremental QALY of: 10 years * (1–0.95) = 0.5; which considering the £30,000 maximum potential threshold is translated into: £30,000 * 0.5 = £15,000. Moreover, considering the savings by avoiding bowel surgery of £1,000, the headroom is suggested to be £16,000 under the most favorable circumstances. This sort of analysis would inform a company in an early stage of development that the use of tissue engineered bladder (as compared to cystoplasty after resection for cancer) would not be a cost-effective drawn on scarce resources for the healthcare system, and hence unlikely to be reimbursed, if could not be produced in such a way that its eventual market price is considerably less than £16,000.
The main limitations of this stage are the difficulties of eliciting prior beliefs about the variables for which no evidence yet exists. There is a growing body of research into how to elicit expert's knowledge accurately and reliably (Reference O'Hagan, Buck and Daneshkhah9). The elicitation methods can be made more or less formal, but the use of plausible assumptions from expert opinion will fit better in the time frame of this early assessment than more sophisticated and time-consuming techniques. In the context of medical devices, relatively good prior information on the performance of the device may be available because their physical mechanism makes their effectiveness generally robust to small changes. To this extent, data from earlier studies on previous generations of the device, or studies from different locations can provide relevant information (Reference Campbell1). Although this analysis can only be considered as indicative, it can still provide some evidence to prioritize medical devices to avoid investing in devices which could never be cost effective.
Mid Stage
At a mid-stage, typically, observational studies would provide some clinical evidence of the effectiveness of the new technology, and some initial cost estimates would be available. Under these conditions, decision-analytic modeling techniques can be applied and the Bayesian approach would provide a means of combining the new, but limited, data with the beliefs previously identified. The prior beliefs about the new technology, ideally elicited from a group of experts, would be used as the prior distribution of the variable of interest, which can now be updated with the evidence that has become available in this stage. In this way, the Bayesian approach still allows for the expert opinion based on extensive experience to influence the estimated outcome. This is of particular interest when the newly available evidence comes from uncontrolled and small trials that may report extreme values that should not be taken at face value (Reference Spiegelhalter, Myles, Jones and Abrams15). Also, very critical at this stage is the need to know what extra information would be most valuable for the company. Simple sensitivity analyses can help to identify which are the key parameters affecting the cost-effectiveness of their product. These analyses are relatively easy to undertake and to understand, and will help the company to find out whether a change in these parameters is likely to make a meaningful difference to the model outcomes and potentially decision making as based upon this. However, the cost in terms of time and resources of collecting this extra information is typically high, and companies will face the decision of whether or not they should fund additional research to reduce uncertainty. Value of information analysis could provide insights into the value of carrying out further research, and highlight the variables in the model for which additional information would be most valuable, if formal analysis were to be undertaken from a societal perspective. This part of the Bayesian approach is based on the idea that information is valuable because it reduces the expected costs of uncertainty surrounding a decision. The expected costs of uncertainty are determined by the probability that a decision, based on existing (prior) information, will be wrong and by the consequences of that potentially wrong decision to the society (Reference Claxton, Sculpher and Drummond2). The expected value of perfect information thus reflects the discrepancy between the current information position and a position of perfect information (no uncertainty). When this analysis is taken one step further to estimate the so called “partial value of information,” it informs us for which specific consequences of the technology (e.g., impact on utilities, costs, or health status) more information is needed to make a less uncertain decision in the future. However, as standard VOI is based on the value to society, and not on the value to a company of further information, its relevance in a commercial context is less clear. Ideally in this context, the methodology of VOI analysis would be adapted to inform the value to a company of conducting further research. The probability of making the wrong decision, in this case to abandon (proceed with) the device when the eventual purchase decision is (not) to reimburse the product, has a potential cost to a company that should be compared with the cost of collecting further information internally to reduce uncertainty.
Identifying the magnitude of the uncertainty and the parameters with the biggest impact on it is a very important contribution at this stage when there is still scope for further research before a final commercial “go ahead” decision needs to be made. An example of cost-effectiveness assessment at this stage, albeit to make a decision as to whether a publicly funded trial was justified, is given by Girling et al. (Reference Girling, Freeman and Gordon6). They focus on an early assessment of second generation left ventricular assist devices (LVAD) as compared to optimal medical care for patients with heart failure but not eligible for heart transplantation. Using data from a randomized controlled trial (RCT) of first generation LVADs, and prior distributions elicited from a group of leading clinicians regarding the effect of second generation LVADs, they concluded that the technology is extremely unlikely to be cost-effective if the device costs as much as £60,000. They also undertook a Bayesian VOI analysis to assess the value of discovering the actual size of the survival benefit of the device (a key parameter in the model). The prediction was that the public cost of a future LVAD trial would not be justified over any reasonable period unless the expectation was that the cost of the device would in future be substantially less than £60,000.
Given the increasing complexity of the techniques, this mid-stage becomes more demanding in terms of data inputs, for which robust evidence may or may not yet be available. A substantial effort may be required to elicit and apply plausible assumptions and distributions to parameters for which no evidence is yet available. The approach also requires some experience with building decision-analytic models. Nevertheless, such models provide an ideal framework to update the results when new evidence becomes available. This stage can if needed be refined at several points of time and it enables aggregating evidence from a wide range of sources. In each stage, the “posterior” estimate of a parameter value that emerges from a refinement becomes the “prior” estimate that goes into the next stage of refinement.
Late Stage
The health economic analyses undertaken in the late stage are typically designed to inform external decision makers (e.g., health service payers) about the expected cost-effectiveness of the new technology, and so to make the case for reimbursement of the product. A reimbursement decision would preferably be based on a model regarding incremental effectiveness and costs underpinned by evidence provided by large RCTs rather than by speculative evidence (Reference Sculpher, Claxton, Drummond and McCabe11). Ideally, early assessment would have identified the parameters that have a relatively large impact on the product's cost-effectiveness and for which additional and better information is thus most valuable. This would subsequently be incorporated into the RCT design so that it provides the key data in the most suitable way, for example in terms of sample size and length of follow-up and so makes most efficient use of research resources. As previously noted, Bayesian methods provide the possibility of accumulating the previously collected evidence with the new data. However, because prior beliefs and observational data may well be affected by bias, the techniques that are used to combine randomized and observational evidence should take into account the different nature of data sources. Meta-analyses that simply pool data from all sources together are not likely to be appropriate because they fail to recognize the different types of evidence being synthesized. Several methods can be used to overcome this problem, and those vary by the sophistication of the technique. The most straightforward approach is to use observational data as the prior distribution for the information provided by the RCT. If we have assessed the observational data to be of sufficiently high quality, the result of a meta-analysis that combines these data can form the prior distributions. Otherwise, the observational evidence can be down-weighted using methods that introduce some caution around the potentially biased estimates (for instance, the priors could be centered around the observational pooled estimate but a much larger variance can be used). This methodology was explored for the assessment of the efficacy of electronic fetal heart rate monitoring (EFM) (Reference Sutton and Abrams16). The authors compared results of meta-analyses that include and exclude the observational data in addition to the randomized studies, and illustrated different ways of combining the different types of evidence. The inclusion of observational data in general favored EFM. More sophisticated methods involve the use of a hierarchical model to pool data from different sources simultaneously. Those methods accommodate qualitative and quantitative heterogeneity in terms of the size of the effect of a technology, and study design of the evidence. Prevost, Abrams, and Jones (Reference Prevost, Abrams and Jones10) applied a three level (i.e. study level; study type level; and population level) Bayesian model to estimate the relative risk of breast cancer mortality in studies of breast cancer screening, and combined evidence from both randomized and nonrandomized trials. They found that this sophisticated approach gave an almost identical estimate to that obtained by performing a meta-analysis ignoring study type, but the three level model allowed for greater uncertainty and hence produced a wider 95 percent confidence interval.
RCTs are considered to be the most powerful research design for establishing whether an intervention is effective, because they most successfully eliminate bias as compared to other research design (Reference Torgerson and Torgerson17). Different types of studies have been hierarchically ordered as regards their methodological quality as level 1 (RCTs), level 2 (controlled observational studies), level 3 (observational studies without control groups), and level 4 (expert opinion). It has been argued that the elicited views of informed experts, or the observational evidence collected in previous stages, become irrelevant in the presence of large RCTs. Especially as the empirical evidence has accumulated, the need for subjective opinion disappears, and the estimates are dominated by the evidence provided by the data. Others however, have recognized that although studies lower in the hierarchy are more prone to bias, RCT data are often limited to selected populations, short time spans and selected comparator treatments, and thus, the value of evidence from anywhere in the hierarchy will depend on its quality and relevance (8). Moreover, RCTs are usually designed to detect specific outcomes but they may not be powered to measure other outcomes of interest, such as a low probability for a severe adverse event. Therefore, prior information based on vast expert knowledge or observational evidence can enhance the evidence from RCTs, and Bayesian modeling does provide ways of combining the evidence from this variety of disparate sources.
Discussion
In this study, we have set out a methodology for integrating health economic modeling in early, mid and late development stages of medical devices, and for updating this analysis in an iterative process using a Bayesian approach.
Whereas others have previously demonstrated the applicability of this approach for supporting reimbursement decisions for innovations (i.e., mainly pharmaceuticals) for which relatively hard data may be available, and the resources and time available to perform the analyses is generally longer, we sought to adapt these methods as to reliably support medical device development cycles without unduly delaying these.
The immediate advantage of an early analysis of the potential cost-effectiveness of a new technology is to form a basis for prioritizing between several potential device developments in the face of scarce developmental resources. By focusing on those devices most likely to be cost-effective, the failure rate at each stage of the development process should be reduced, as should be the development costs. If further investment on research is driven by identifying the parameters for which more information is most valuable, this is also likely to enhance efficient use of research and development resources. For users of medical devices, this might translate in earlier access to the most beneficial new technologies.
Notwithstanding its likely appeal, there are a couple of potential caveats to the described approach that need consideration. First, because many of the techniques described in this study are not simple and need a substantial level of expertise on economic evaluation and evidence synthesis methods, the cost and time involved in developing these analyses may act as a barrier to their integration into the development cycle of a new product. We aimed to address this issue by varying the sophistication of the models from stage to stage, allowing for simple health economic models at the start, and proceed to greater depth of analysis at the later stages. The critical element remains that at any given stage, decisions are taken based on analyses that contain the best knowledge that is available about the product but are also feasible within the data and time restrictions. Second, it has to be acknowledged that there might be limitations to the extent to which meaningful interpretations can be derived from early stage evaluation due to both the learning curve phenomenon and the fact that the process of innovation in medical devices is one of continuous incremental improvements in close interaction with the users of the technology (Reference Siebert, Clauss and Carlisle13). Both factors might be difficult to estimate at this point in the development, even more so when the intended users are not involved. Interaction between all stakeholders as early as possible in the development cycle is therefore strongly recommended (Reference Martin, Murphy, Crowe and Norris7). Finally, the focus in this study has been on developing a methodology for assessing the economic value of technologies to inform reimbursement decisions, which are taken from a societal or a health system perspective. Although adopting such a wider perspective from the early start is likely to be the best approach for anticipating the reimbursement decision, in addition there should be a more commercial perspective that can inform a firm's investment decision. The models could inform an early prediction of firm's potential revenues and the optimum price based on the estimated likely cost-effectiveness in daily practice. The commercial analysis would thus build on the features of the cost-effectiveness analysis, but would require additional information on production cost, scale, and so on. Further research, however, is needed to investigate the practicality and likely value of the approach outline in this study to medical device companies and the wider healthcare society, as well as to explore a methodology that appropriately address the value of information analysis applied to a commercial perspective.
CONTACT INFORMATION
Laura Vallejo-Torres, MSc (Laura.Vallejo@Brunel.ac.uk), Research Fellow, Lotte M. G. Steuten, PhD (Lotte.Steuten@Brunel.ac.uk), Research Fellow, Health Economics Research Group (HERG) & Multidisciplinary Assessment of Technology Centre for Healthcare (MATCH), Brunel University, Uxbridge, Middlesex, UB8 3PH, UK
Martin J. Buxton, BA (Soc Sci) (Martin.Buxton@Brunel.ac.uk), Professor, Professor of Health Economics, Health Economics Research Group (HERG) & Multidisciplinary Assessment of Technology Centre for Healthcare (MATCH), Brunel University, Uxbridge, Middlesex, UB8 3PH, UK; Director, Health Economics Research Group (HERG), Brunel University, Uxbridge, Middlesex, UB8 3PH, UK
Alan J. Girling, MSc (a.j.girling@bham.ac.uk), Senior Research Fellow, Richard J. Lilford MA (r.j.lilford@bham.ac.uk), Professor, Professor of Clinical Epidemiology, Department of Public Health and Epidemiology & Multidisciplinary Assessment of Technology Centre for Healthcare (MATCH), University of Birmingham, Birmingham, B15 2TT, UK
Terry Young, PhD, BSc (Terry.Young@Brunel.ac.uk), Professor of Healthcare Systems, School of Information Systems, Computing and Mathematics & Multidisciplinary Assessment of Technology Centre for Healthcare (MATCH), Brunel University, Uxbridge, Middlesex, UB8 3PH, UK; Director, Multidisciplinary Assessment of Technology Centre for Healthcare (MATCH), Brunel University, Uxbridge, Middlesex, UB8 3PH, UK
