Decision-analytic models of cost-effectiveness are widely used in health technology assessment (HTA) to address the complexity and uncertainty of policy decision making (Reference Brennan and Akehurst2;Reference Buxton, Drummond and VanHout3). Models typically do not restrict the definition of relevant evidence by PICO (Patients, Intervention, Comparator, Outcome) but assess evidence within an analytical framework that reflects the complexity associated with the context of the decision.
The broad range of evidence required to support this approach is widely acknowledged and the role of evidence in populating model parameters is largely understood (Reference Cooper, Coyle, Abrams, Mugford and Sutton6;Reference Golder, Glanville and Ginnelly11;12;Reference Weinstein, O'Brien and Hornberger15). Search methods aimed at retrieving this range of evidence however have not been developed. Moreover, the role of evidence in informing the development of a relevant model framework and the underlying information-seeking processes that support this process remain poorly understood (Reference Cooper, Sutton, Ades, Paisley and Jones5).
This study aims to assess systematically the scope of evidence and the purposes for which evidence is used in decision-analytic models of cost-effectiveness and to consider this scope in terms of implications for searching. This is done through a content analysis of the evidence cited in two published cost-effectiveness models and through the generation of a classification of evidence relevant to both models.
METHODS
The models analyzed were used to inform reimbursement decisions of the National Institute for Health and Clinical Excellence (NICE) in England and Wales. One assessed two agents in the treatment of early (stage III) colon cancer (Reference Pandor, Eggington, Paisley, Tappenden and Sutcliffe14). The other assessed a drug in the management of hypercholesterolemia (HCE) (Reference Ara, Tumur and Pandor1). The colon cancer (CC) model considered cost-effectiveness in terms of a cost per quality-adjusted life-year (QALY) and life-years gained (LYG). It extrapolated over a life time horizon 5-year empirical evidence of overall survival from randomized controlled trials (RCTs). The HCE model considered cost-effectiveness in terms of a cost per QALY. It translated over a life time horizon short-term surrogate evidence from RCTs of changes in cholesterol level into changes in risk of cardiovascular events.
All text reporting the cost-effectiveness models was analyzed. The bibliographic references of cited sources were extracted and the study design or format of the cited information was established. The extract of text supported by each citation was used to form a description of why the source was being cited.
These data were used to categorize each source according to three dimensions; the type of information (e.g., costs), the type of evidence (e.g., RCT), the modeling activity for which the evidence was being used (e.g., populate model parameter). These categories were used to develop a three-dimensional classification of the scope and uses of evidence in both models.
The classification of evidence aimed to generate a description of the breadth and diversity of evidence cited. However, consideration of each citation in isolation cannot capture fully the complexity of the evidence base used. To address this, further analysis to identify relationships within and between the dimensions of the classification was undertaken and the classification was reconsidered in the context of the reported text.
The content analysis was reflected upon to consider implications for search methods.
RESULTS
In total 130 unique sources of evidence were cited (CC model = 41, HCE model = 89). In some cases, multiple sources of evidence were used for the same reason. In other cases, an individual source was cited several times for different purposes.
The three-dimensional classification of evidence defined fourteen types of information, seven types of sources of evidence and five uses of evidence. The classification of evidence is in Table 1.
Classification of Evidence in Decision-Analytic Models of Cost-Effectiveness
RCT, randomized controlled trial.
Types of Information
Of the fourteen categories, five related to the costs and effects of interventions including clinical outcomes, adverse effects, utilities, unit costs, and resource use. Three provided information on the condition of interest. These were natural history, epidemiology, and prognosis. Two related to disease management including current practice and prescribing rates. Two categories, compliance and patient preferences, identified factors affecting treatment uptake. Two final categories provided analytical rather than clinically related information. These included information to inform modeling methods and the methods and results of other models in similar disease areas.
Types of Sources
The seven categories defined can be divided into research and non-research sources. Research-based sources included evidence syntheses including meta-analyses, systematic reviews and cost-effectiveness models, RCTs, including one economic evaluation alongside an RCT, observational studies defined mainly as prospective cross-sectional surveys or longitudinal cohort studies and methodological studies.
Three categories are described as non-research based. The first is expert judgment as published opinion and, more frequently, personal communications. “Reference sources” is used to describe standard, accepted sources, often providing generic information including drug formularies or disease classifications, or sources that have some inferred authority due to consensus on their reliability or relevance to the context of the decision. Examples of the latter include policy guidance and practice guidelines. Lastly “routine data sources” is used to describe information collected for routine or administrative purposes including national disease registers, life tables, and health service activity data.
Uses of Evidence
This dimension focuses not on the evidence but on defining modeling activities supported by evidence. The first category is the population of model parameters where evidence provided a broad range of data to support estimates for incorporation in the models including costs, effects, baseline population characteristics, and baseline risk of events.
The second is the use of evidence in the design and specification as opposed to the population of the model framework. It includes the use of evidence to gain an understanding of the decision problem and to translate this into an analytical framework. Evidence was used in both models to inform and justify a broad range of judgments and specifications including the disease pathway and corresponding model structure, the model perspective and time horizon, the specification of treatment scenarios, and the definition of cost and resource groups.
Remaining categories cover the use of evidence in informing analytical activities. Model validation included the use of expert advice and between model comparisons. Sensitivity and uncertainty analysis included the citation of alternative sources to explore heterogeneity in the data. Informing the analytic modeling approach was an important category particularly in the HCE model which cited twenty-seven sources to inform the method of extrapolation of surrogate evidence into long-term clinical endpoints.
Further Analysis
Several relationships were observed between the three dimensions. Evidence used to populate the models was grouped largely in the epidemiology, clinical outcomes, unit costs, and utilities categorizations. These are defined elsewhere as important data inputs (Reference Cooper, Coyle, Abrams, Mugford and Sutton6;Reference Coyle, Lee, Donaldson, Mugford and Vale7;Reference Golder, Glanville and Ginnelly11). With the exception of unit costs, this information was associated with expected types of evidence. Clinical outcomes came mainly from RCTs or meta-analyses, epidemiological information from longitudinal studies and utilities from cross-sectional studies. Unit costs drew on six of the seven types of evidence suggesting, within the data observed, no relationship between unit cost information and specific types of evidence.
Evidence used in the design and specification of the models suggested more complex relationships with multiple sources being brought to bear on individual information needs. This perhaps reflects a greater complexity in the information needs associated with model development. For example information used to establish relevant treatment scenarios included current practice information, prescribing rates, clinical outcomes, resource use, adverse effects, and patient preferences. This was drawn from a range of sources including reference sources, routine data, RCTs, and experts.
An important relationship was observed in the use of expert judgment, used mainly in the population of model parameters and in the design and specification of the models. Its use appeared to be less important in populating the model (16 percent of citations associated with model population) compared with its use in supporting the design and specification of the model (31 percent of citations). In addition, in populating the model its role appeared to be associated with interpreting evidence rather than providing data. This reflects the conflicting roles of expert judgment in modeling where data derived from experts is placed at the bottom of evidence hierarchies (Reference Coyle, Lee, Donaldson, Mugford and Vale7) but where the concurrence of experts on the authenticity of the model is a recognized form of model validation (Reference Eddy8).
Reconsideration of the classification in the context of the published reports revealed a relationship between the model time horizons and the types of information. Evidence relating to populations and interventions other than those specified in the PICO-focused question was frequently cited. For example, the HCE model cited a wide range of utility estimates relating to several cardiovascular events. The CC model cited effectiveness, costs, resources use, utilities, and adverse effect evidence in relation to advanced colon cancer in addition to early stage III cancer. The disease pathway within a model time horizon is commonly defined by various health states. The characteristics of the model population and of the management of that population typically changes from state to state. Information including but by no means restricted to costs, effects, utilities, and epidemiology is needed to populate these states. The scope of information required therefore is defined not by the PICO question but by all the health states in the model.
DISCUSSION
The study provides an in depth analysis of the scope of evidence used to inform typical models of cost-effectiveness. The aim of the study was to generate a classification that characterized the evidence in terms of the types of information, the types of sources of evidence and the purposes for which the evidence was used. A further aim was to consider the implications of the classification for searching for evidence. These implications are summarized in Table 2.
Implications for the Development of Search Methods
The models had multiple information needs as observed in previous studies (Reference Golder, Glanville and Ginnelly11). The scope of the evidence base could not have been retrieved by a single search strategy replicated across several sources. Whereas an exhaustive search is not required for every information need, a range of searches is required to identify an evidence base of adequate scope to inform a decision-analytic model (Reference Glanville, Paisley, Shemilt, Mugford, Vale, Marsh and Donaldson10).
In addition, the scope of information needs relevant to a model is not expressed explicitly in the definition of the PICO question and the PICO question is not a sufficient tool for the planning of search strategies. Alternative mechanisms for the systematic translation of information needs into search strategies are required.
The analysis identified a broad range of information beyond costs and effects. Important categories defined elsewhere were identified in the current study (Reference Cooper, Coyle, Abrams, Mugford and Sutton6). In addition, the use of evidence to inform more complex information needs such as establishing a picture of current practice and disease management and to establish and justify the analytical modeling approach was observed. The retrieval of this range of information requires an understanding of information needs and of the sources from which evidence might be drawn. Although this level of understanding exists for the retrieval of effectiveness evidence, it does not currently exist to the same degree for other types of information. The development of a conceptual understanding of the modeling process is required to better characterize the diversity and complexity of information needs associated with model development.
The analysis observed routine data sources, reference sources and expert judgment as important categories of evidence in addition to research-based sources. Although the use of these sources in modeling is recognized (Reference Eddy8;Reference Garrison, Neumann, Erickson, Marshall and Mullins9), it is not clear how they fit within evidence hierarchies. It is important that the role and status of these types of evidence is established and that knowledge on how to identify and interrogate these sources is generated to develop a systematic and strategic approach to retrieving information beyond the searching of bibliographic databases. A characteristic of routine data and reference sources is that their relevance is often specific to the context of the decision making, be that geographical or a particular reimbursement process. The development of resource lists relevant to local decision-making processes would provide useful guides to the search process.
Evidence was used in both models for purposes other than the population of model parameters. Evidence had a contextual role where it was used to support the understanding of the decision problem and inform the judgments underpinning the model framework. It also had an analytical role whereby it was used to inform and justify the methodological approach and analytical processes of the model. These activities and the information-seeking processes associated with them are largely implicit and are not acknowledged to any great degree in the modeling literature (Reference Chilcott, Tappenden and Rawdin4;Reference Paisley13). The credibility of models relies on an evidence-based approach to all aspects of model development. An understanding of the modeling process is required to retrieve the full range of evidence relevant to model development.
CONCLUSION
Effective searching demands an understanding of the scope and characteristics of the information required and of the purposes for which the information is used. The classification generated in this analysis is based on two case study models, both of which are typical models of cost-effectiveness. It is the first attempt to define comprehensively the role of evidence in the development of decision-analytic models. The classification of evidence provides a useful tool which can be developed further in order that the broad range of information required for decision-analytic models is understood and systematically identified.
ACKNOWLEDGMENTS
Julie Glanville and Jim Chilcott advised on the design and commented on the results of the study. The content analysis forms part of a larger study funded by a Department of Health Fellowship in England and Wales.
CONTACT INFORMATION
Suzy Paisley, MA (s.paisley@sheffield.ac.uk), Senior Research Fellow, School of Health and Related Research (ScHARR), University of Sheffield, Regent Court, 30 Regent Street, Sheffield S1 4DA, United Kingdom
CONFLICT OF INTEREST
S. Paisley received a Department of Health Fellowship for this work.
