Agencies such as the National Institute for Health and Clinical Excellence (NICE) need to both synthesize and summarize the available evidence, and assess the relative cost-effectiveness of competing clinical interventions. They also need to ensure that those conducting the necessary analyses have made use of the most appropriate, often increasingly sophisticated, quantitative methods; it turns out that many new statistical approaches are implemented under the Bayesian modeling paradigm due to the flexibility of the framework. See elsewhere (Reference Ashby1) for a recent review.
The requirement for nonstandard analytic methods arises from two main issues. First, evidence synthesis and cost-effectiveness analysis is a complex process that generally requires construction of a “decision model” (Reference Buxton, Drummond and Van Hout2–Reference Briggs and Sculpher4) which is a formal representation of a disease process in a heterogeneous population. The available evidence (accurately reflecting uncertainty) then needs to be used to assess how this process will be affected by different treatment options. Second, limitations in evidence due to a paucity of relevant high-quality studies means that there are inevitable gaps in the quantities necessary to populate the model, and these have to be filled with an element of judgment. It turns out that both of these issues can be addressed using Bayesian ideas. Bayesian methods were reviewed in 2000 (Reference Spiegelhalter, Myles, Jones and Abrams5;Reference Spiegelhalter, Myles, Jones and Abrams6) and a primary motivation for this current review is to establish the degree of penetration of such methods over the past decade. Before progressing to the details of our review, we provide a concise description of the Bayesian paradigm and its applied use in health technology assessment (HTA).
THE LEGACY OF BAYES
When Reverend Thomas Bayes, a nonconformist minister of Tunbridge Wells, died in 1761 he left behind a manuscript that when published posthumously (Reference Bayes7) contained two fundamental and revolutionary ideas. He is primarily remembered for Bayes theorem, a formal law of probability that tells us how to learn from experience: we initially express our uncertainty as a prior probability distribution, which on the basis of observed evidence is revised to a posterior distribution using Bayes theorem (this is in contrast to the Classical approach to statistical analysis which only makes use of the observed evidence). However, his other insight precedes this mathematical rule and is far more fundamental. This is the idea that probability distributions are not just applicable to predicting “chance” phenomena such as dice and cards, but can also be placed over unknown states of the world (i.e., to represent prior opinion about proportions, event rates and other unknown quantities).
This apparently esoteric idea has immediate and important application in health technology assessment. Any assessment of the impact of an intervention requires assumptions about unknown parameters such as the average effect on a defined population, the period of effectiveness, compliance rates and so on. There is epistemic uncertainty about these parameters that can, taking a Bayesian approach, be expressed as a probability distribution. This permits “probabilistic sensitivity analysis,” in which the influence of the uncertainty about the parameters is propagated through a model to qualify any claims about the eventual cost-effectiveness of the intervention. These techniques require a Bayesian interpretation of the parameter uncertainty.
Implicit and Explicit Bayes
In the context of evaluating healthcare interventions, the Bayesian approach has been defined as “the explicit quantitative use of external evidence in the design, monitoring, analysis, interpretation and reporting of a healthcare evaluation”(Reference Spiegelhalter, Abrams and Myles8). Within HTA both “implicit” and “explicit” Bayesian methods may be used (Reference Ades, Sculpher and Sutton9). Implicit Bayes refers to any analysis in which a distribution is placed on a parameter but without overtly referring to Bayesian ideas. For example, probabilistic sensitivity analysis (Reference Briggs10) where probability distributions are placed on imprecisely-known quantities in the cost-effectiveness model, plausible values simulated from the distributions, and the resulting expected costs and effectiveness of the treatments calculated. Repeating this analysis many times (each time sampling values) allows the uncertainty about the overall cost-effectiveness to be communicated—this is known as a Monte Carlo analysis. It is then possible to use these results to perform a value of information analysis (Reference Claxton11) to determine the expected costs of decision uncertainty predicted by the cost-effectiveness model and the maximum value that can be placed on additional research aimed at reducing this uncertainty.
Explicit Bayes refers to analyses that actually use Bayes theorem, whether the prior distributions are “informative,” in the sense of expressing substantive opinion, or “noninformative,” in the sense of trying to have as little influence as possible (Reference Gelman, Carlin, Stern and Rubin12). For example, when comparing the effectiveness of more than two treatments, or where no head-to-head trials exist, mixed treatment comparison methods (also known as network meta-analysis) may be applied (Reference Lumley13–Reference Cooper, Sutton and Lai15); such complex nonstandard statistical methods require a flexible modeling framework and therefore are most often fitted within a Bayesian framework using “noninformative” prior distributions (Reference Spiegelhalter16). Explicit Bayesian methods using “informative” prior distributions may also be applied in evidence synthesis; for example, where the overall aim is to include all the evidence, while allowing for different degrees of uncertainty (due to potential biases, or generalizability) associated with different studies (Reference Turner, Spiegelhalter, Smith and Thompson17).
In overview, within HTA the main advantages of Bayesian analysis, compared with the Classical approach to statistical analysis, include the more efficient use of all available data, more flexible framework to adapt to nonstandard situations, and more interpretable probability results directly regarding the quantities of interest (Reference Spiegelhalter, Abrams and Myles8;Reference Sutton, Abrams, Jones, Sheldon and Song18). Barriers to the use of Bayesian methods include the use of prior distributions which may be seen as subjective (although “noninformative” prior distributions may be defined—see above), nontrivial elicitation of prior beliefs, and computationally complex, and therefore time consuming, to implement (although this has become less of an issue with the development of freely available specialist software such as WinBUGS) (Reference Spiegelhalter, Thomas, Best and Lunn19).
OBJECTIVES
In this study, we aim to examine the use of implicit and explicit Bayesian methods in HTA and to identify whether this has changed over time. A case study is presented in the Appendix, selected from the HTA reports reviewed, to demonstrate the extend to which Bayesian methods may be used to aid the HTA process.
METHODS
All UK NIHR Health HTA Programme reports listed on their Web site (http://www.hta.ac.uk/) as published between 1997 and 2011 inclusively were selected for review (a subsample of these reports also informed the NICE appraisals). We decided to focus our review on these HTA reports because they provide in-depth accounts of the methods applied both within the systematic review and the economic analysis, due to no explicit word limit restrictions as imposed by many journals; thus, providing an excellent sample to explore the use of Bayesian methods in HTA. HTA reports were excluded if they were primary research (e.g., randomized controlled trials), or focused on a particular methodological issue (e.g., errors in HTA models, feasibility of value of information). The main focus of the review was to identify secondary research reports which had used Bayesian methods in their evaluation(s). Bayesian methods were classified as either implicit or explicit using the definitions specified above. In addition, data were also extracted on the software used to undertake the Bayesian analysis.
RESULTS
Figure 1 shows a flow diagram of the identification of NIHR HTA reports for inclusion in this review. Of the 608 HTA reports published between 1997 and 2011, 375 were identified as relevant for this review. Of these, 155 (41 percent) contained an implicit and/or explicit Bayesian analysis; of which 128 (83 percent) HTA reports contained an implicit Bayesian analysis alone, 3 (2 percent) explicit Bayesian analysis alone and 24 (15 percent) both implicit and explicit Bayesian analyses. Seventy-six of these 155 (49 percent) reports identified for inclusion in our review also informed a NICE appraisal, and of these sixty-two (82 percent) contained an implicit Bayesian analysis alone and fourteen (18 percent) contained both implicit and explicit Bayesian analyses.
Flow diagram of identification of UK National Institute of Health Research (NIHR) health technology assessment (HTA) reports for inclusion. PSA, probabilistic sensitivity analysis; VOI, value of information; NICE, National Institute for Health and Clinical Excellence.
Overall, of the 155 HTAs containing a Bayesian analysis, 154 (99 percent) developed economic decision models (i.e., 58 developed a decision tree model, 68 a Markov model, 15 a Discrete event simulation (DES) model, 6 a Decision tree and DES, and 6 a Markov model and DES), of which 152 (99 percent) applied probabilistic sensitivity analysis and 18 (12 percent) performed value of information analysis.
Twenty-seven HTAs explicitly used Bayes theorem, of which only six specified informative prior distributions mostly in the evidence synthesis models. Twenty of the twenty-two (91 percent) HTAs that specified “noninformative” prior distributions (including two that specified both “noninformative” and “informative”) used the Bayesian framework to undertake indirect/mixed treatment comparisons meta-analysis (Reference Lumley13;Reference Caldwell, Ades and Higgins14). The remaining two performed Bayesian pairwise meta-analyses.
Figure 2 depicts how Bayesian methods have been used in the HTA reports reviewed over time. Overall, there has been an increase in the use of Bayesian (both implicit and explicit) methods (as indicated by the total height of each bar), although there is an unexplained drop in 2008 and 2010. The solid line shows the percentage of HTAs reviewed that also informed a NICE appraisal and the dotted line shows the percentage of these that applied Bayesian methods. After 2004, it can be observed that the number of HTAs informing NICE appraisals decreased due to the introduction of Single Technology Appraisals (20;Reference Wailoo and Pickstone21) whereby pharmaceutical companies submit their own HTAs for review by NICE. However, of those HTAs that did inform NICE appraisals, a large percentage applied Bayesian methods.
Stacked bar chart showing the percentage of health technology assessment (HTA) reports reviewed containing different types of Bayesian analysis
In Figure 2, the bars are subdivided into the types of Bayesian analyses used over time. Overall, implicit Bayesian methods were used more often than explicit methods, and always included a probabilistic sensitivity analysis for the cost-effectiveness evaluation. It can be observed from Figure 2 that before 2004, less than 20 percent of HTAs per year applied Bayesian methods and all of these applied implicit methods in the form of probabilistic sensitivity analysis; that is none of the HTAs used any other form of Bayesian analysis. In April 2004, the first NICE guide to the methods of technology appraisal (22) was published. Although the term Bayesian analysis did not appear in the guidance, it did state that “Probabilistic sensitivity analysis should be conducted on models to reflect the combined implications of uncertainty in parameters” (Section 5.8.1), “formal value of information methods are available” (Section 5.11.2) and “Where no head-to-head trials are available, consideration is given to indirect comparisons, subject to careful and fully described analysis and interpretation” (Section 3.2.2.2). Figure 2 shows the likely impact this guidance had on the uptake of both implicit and explicit Bayesian methods. Focusing on the three components outlined in the NICE guidance, Table 1 shows a significant increase in the use of mixed treatment comparison (2005 to 2008: 11 percent) and value information (2005 to 2008: 8 percent) methods in HTAs post 2004. Updated guidance was issued by NICE in June 2008 (23). Again, the term Bayesian analysis did not appear in the document. Although the guidance on probabilistic sensitivity analysis and value of information remained largely the same, the guidance on the use of mixed treatment comparisons was expanded stating that “When head-to-head RCTs exist, evidence from mixed treatment comparison analyses may be presented if it is considered to add information that is not available from the head-to-head comparison” (section 5.3.13). Table 1 shows an increase in the number of HTAs applying both mixed treatment comparison (2009 to 2011: 19 percent) and value of information (2009 to 2011: 21 percent) methods.
Number of HTA Reports Containing Bayesian Methods That Used Mixed Treatment Comparisons, Probabilistic Sensitivity Analysis and Value of Information in 3 Time Periods
HTA, health technology assessment.
Where stated, explicit Bayesian analyses were conducted using freely available software WinBUGS (http://www.mrc-bsu.cam.ac.uk/bugs/winbugs/contents.shtml) and/or R (http://www.r-project.org/), whereas implicit Bayesian analyses were conducted using a variety of commercially available decision modeling and spreadsheet packages including Simul8 (http://www.simul8.com/), Data TreeAge (http://www.treeage.com/) and Microsoft Excel (http://office.microsoft.com/en-gb/excel/).
A case study, selected from the HTA reports reviewed to demonstrate the extent to which Bayesian methods may be used to aid the HTA process, is presented in the Appendix.
DISCUSSION
This review has assessed the uptake of Bayesian methods to inform HTAs published by the NIHR HTA Program between 1997 and 2011 inclusively. The use of both implicit and explicit Bayesian methods has increased over the time period studied. This is partly due to the publication of the method guides by NICE (22;23) which promotes the use of relevant methods but also due to the development of freely available, more user-friendly, Bayesian specialist software packages such as WinBUGS (Reference Spiegelhalter, Thomas, Best and Lunn19) which have aided the analysis of more complex evidence synthesis structures (e.g., mixed treatment comparisons). For example, the original HTA of neuraminidase inhibitors for the treatment of influenza published in 2003 (Reference Turner, Wailoo, Nicholson, Cooper, Sutton and Abrams24) presented separate meta-analyses for the two active treatments (zanamivir and oseltamivir) under review compared with placebo as no head-to-head trials of the two active treatments existed. However, the updated review, published in 2009 (Reference Burch, Paulden and Conti25), applied explicit Bayesian methods to obtain an indirect estimate of the two treatments compared with one another as well as placebo. Similarly, the recently published review of obesity treatments (Reference Ara, Blake and Gray26) collated and updated the previous two HTAs (evaluating the effectiveness and cost-effectiveness of Orlistat (Reference O'Meara, Riemsma, Shirran, Mather and Riet27) and Sibutramine (Reference O'Meara, Riemsma, Shirran, Mather and Riet28) separately) using explicit Bayesian mixed treatment comparison methods to bring the evidence together within a single analysis. The above analyses may have been possible to conduct using Frequentist statistical methods; however, the main advantages of using a Bayesian approach include the flexibility of WinBUGS to fit complex nonstandard statistical models and the ability to make direct probability statements such as the probability each treatment is the “best.”
Despite an observed increase in the use of Bayesian methods over time in the HTAs reviewed here, a comprehensive review of over fifty health technology assessment (HTA) and pharmacoeconomic guidelines from thirty-eight countries revealed that only twelve HTA organizations (Reference Dequen29) worldwide explicitly discuss the use of Bayesian methods. These include the Canadian Agency for Drugs and Technologies in Health guidelines (30), which state that Bayesian approaches are particularly “well suited” for healthcare assessments, identifying the most important sources of uncertainty and providing more accurate estimates. Also the Haute Autorité de la Santé (HAS) guidelines (31) that refer to the use of Bayesian methods to perform network meta-analysis to allow the complete hierarchy of evidence within a therapeutic area to be drawn upon. Health Austria (Reference Fröschl, Brunner-Ziegler and Conrads-Frank32) also presents the advantages and growing popularity of Bayesian methods in solving complex models and the Agency for Health Research and Quality in the United States (33) supports “the use of Bayesian methods with vague priors in CERs [comparative effectiveness reviews].” Several guidelines that do not explicitly use the term Bayesian, such as those published by NICE (23), Scottish Medicines Consortium (SMC) (34), and the Pharmaceutical Benefits Advisory Committee (PBAC) in Australia (35), do implicitly endorse their use by positively advocating the use of methods such as probabilistic sensitivity analysis and mixed treatment comparisons.
The HTA process has developed as a procedure of two halves (Reference Drummond, Iglesias and Cooper36). This review has identified how explicit Bayesian methods are mostly used by statisticians to assess clinical effectiveness by means of evidence synthesis (e.g., mixed treatment comparisons, generalized evidence synthesis) whereas implicit Bayesian methods are mostly used by health economists/decision modelers in the economic evaluation (e.g., probabilistic sensitivity analysis, value of information). This traditional professional split is reflected in the structure of the HTA reports and can prove problematic; for example, the format of the pooled clinical outcome may not “match” the data requirement for the economic evaluation (Reference Drummond, Iglesias and Cooper36;Reference Novielli, Cooper, Sutton and Abrams37) (e.g., for the clinical review the most appropriate summary measure may be median survival time whereas the economic evaluation requires mean survival time), and/or the uncertainty associated with a particular outcome may not be appropriately specified when input into the decision model.
There are increasing attempts to integrate the two components of HTA, both to ensure that the results of the evidence synthesis carry through accurately and consistently into the economic model, and to allow a unified approach to sensitivity analysis (Reference Spiegelhalter and Best38;Reference Cooper, Sutton, Abrams, Turner and Wailoo39). Specifically, it is an advantage to be able to integrate probabilistic sensitivity analysis to unknown quantities, with deterministic sensitivity analysis to different assumptions about the structure of models and which data to be used. The aim being to identify and communicate to decision makers what are the pieces of evidence and assumptions that are driving the preference for one treatment over another, so that these can be subject to particular scrutiny and possible refinement. The Transparent Interactive Decision Interrogator (TIDI), developed by Bujkiewicz et al. (Reference Bujkiewicz, Jones and Lai40) and applied in the 2011 published HTA of treatments for psoriatic arthritis (Reference Rodgers, Epstein and Bojke41), enables the two components of the HTA process (i.e., the systematic review of effectiveness and the economic evaluation) to be combined within a single coherent framework by linking different software packages (e.g., WinBUGS for evidence synthesis, Excel for decision modelling and R for graphics) together through an Excel frontend. All results from the analyses (e.g., evidence synthesis and cost-effectiveness) are clearly returned to Excel for clear presentation. The TIDI concept also facilitates more formal critique of decision models by decision makers (such as members of appraisal committees of the National Institute for Health and Clinical Excellence in the UK) by allowing advanced statistical models under different scenarios to be run in real time (including the incorporation of decision makers own beliefs about, for example, study inclusion/quality weightings, etc.), thus making the decision process more efficient and transparent. For a more detailed description of TIDI see Bujkiewicz et al. (Reference Bujkiewicz, Jones and Lai40).
Overall, we have shown that the use of Bayesian methods in HTAs has increased over time despite not explicitly being endorsed in the guidelines published by many of the main international HTA agencies. We envisage that this increase in the uptake will be sustained into the future because, as HTA questions become more complex and demanding, and methodology evolves in response to this, the flexibility of Bayesian methods seem best suited to implement and address nonstandard, often complex, approaches. For example, recent methodological developments where there is potential for Bayesian methods to make an even greater impact on healthcare evaluations in the future include (i) assessing and adjusting for the relevance and rigor of evidence used in both the evidence syntheses (Reference Turner, Spiegelhalter, Smith and Thompson17); (ii) addressing structural uncertainty in the economic decision model (Reference Jackson, Thompson and Sharples42); (iii) assessing model fit in both the evidence syntheses and economic decision model (Reference Spiegelhalter, Best, Carlin and van der Linde43); and (iv) incorporating beliefs of decision makers.
SUPPLEMENTARY MATERIAL
Appendix: Case study: www.journals.cambridge.org/thc2013123
CONTACT INFORMATION
Nicola J Cooper, PhD; Alex Sutton, PhD; Pascale Dequen, M.Sc; Sylwia Bujkiewicz, PhD Department of Health Sciences, University of Leicester, Leicester, UK
David Spiegelhalter, PhD Centre for Mathematical Sciences, University of Cambridge, Cambridge UK
CONFLICTS OF INTEREST
Nicola Cooper, Sylwia Bujkiewicz, Alex Sutton, and David Spiegelhalter have received a grant to their institution from Medical Research Council (MRC), UK. Pascale Dequen's institution has received an educational grant from Pfizer Ltd.
