Keeping evidence reviews up-to-date with changing evidence is difficult (Reference Bastian, Glasziou and Chalmers1). Current practice in updating such reviews varies greatly; despite continual changes in the evidence most evidence reviews are updated infrequently or not at all (Reference Garritty, Tsertsvadze, Tricco, Sampson and Moher7). Approximately one in four Cochrane reviews may be out-of-date within 2 years of publication, whilst approximately one in fifteen may be out-of-date by the time they are published (Reference Shojania, Sampson and Ansari17). When the purpose of a review is to inform clinical practice recommendations, there are important health service implications if they become outdated. Most clinical guidelines are out-of-date within 6 years, and that they should be considered for update after 3 years (Reference Shekelle, Ortiz and Rhodes16).
Different approaches exist for updating systematic reviews of the evidence and some have been evaluated empirically in the context of updating Cochrane reviews (Table 1). An ideal approach would be to be updating reviews supporting clinical practice recommendations continuously (i.e., using a “near-real-time” approach). However, such an approach would require very substantial resources. Routinely updating all clinical practice recommendations at a pre-set time point after original publication (i.e., using a “time-based” approach) might seem logical, but the speed and extent to which evidence evolves over time varies considerably for different interventions. This means that an approach purely based on time-based criteria does not offer the most efficient use of resources: indeed, most clinical practice recommendations remain unchanged when reviews of evidence are updated every 2 years (Reference French, McDonald, McKenzie and Green6). Prioritizing recommendations for updating when it is judged that the evidence base has changed sufficiently (i.e. using a “priority-setting” approach) is more efficient than using “time-based” criteria (Reference French, McDonald, McKenzie and Green6;Reference Hopewell, Loudon, Clarke, Moher, French, Scholten and Eisinga8), but requires proactive surveillance of the literature (Reference Sampson, Shojania and McGowan14), and elicitation of advice from key informers (Reference Hopewell, Loudon, Clarke, Moher, French, Scholten and Eisinga8;Reference Shekelle, Newberry and Maglione15). Furthermore, judgments about which clinical practice recommendations ought to be prioritized for updating (because it is thought likely that they will change) are complex and subjective.
Different conceptual approaches for keeping clinical practice recommendations up-to-date
The Interventional Procedures Program (“the Program” hereafter) of the National Institute for Health and Clinical Excellence (NICE), produces clinical practice recommendations for the whole United Kingdom on the use of new interventional procedures, based on evidence about their safety and efficacy. The Program's capacity is fixed, and since its inception in 2002 it has produced approximately forty items of guidance on different procedures each year. The Program has developed a process for updating recommendations on interventional procedures using a hybrid approach (i.e., involving aspects of both “time-based” and “priority-setting” components). This process focuses resources on interventions for which the original evidence base was limited and which, therefore, were issued with cautious recommendations for use in clinical practice (Table 2). We describe the process for updating recommendations for procedures with limited evidence and evaluate the Program's experience with an initial cohort of procedures with recently updated recommendations. Our aim is to assist reviewers and policy makers who face similar challenges in keeping recommendations up-to-date.
METHODS
Context:
New interventional procedures are defined as those involving incision, puncture, or entry into a body cavity or orifice, or the delivery of electromagnetic, ionizing or acoustic energy: they typically encompass surgical, endoscopic, endovascular, laser, and ablative treatments. Between 2002 and June 2010 NICE's Interventional Procedures Program has produced 400 clinical practice recommendations documents on 352 different procedures (13). Draft recommendations are produced by an independent Advisory Committee (the “Committee” hereafter) based on overviews of the available published evidence (“overviews” hereafter), advice from clinical experts, and patient commentaries. Following public consultation on the draft guidance, and after consideration by the Committee of consultation comments received, final guidance is published by NICE (12;13).
The published evidence on which the Committee concentrates its attention is selected by the Program team and presented in a detailed table of the overview document following a thorough literature search. Typically, evidence is presented in this table from all randomized controlled trials (RCTs) and systematic reviews (when those are available), together with evidence from the highest quality, largest and most relevant non-randomized comparative and case-series studies. Normally this table contains evidence from a maximum of eight studies and all other published reports are presented in less detail in an appendix. The evidence on new interventional procedures is typically quite limited and normally all or most studies of high quality can be accommodated. Hereafter and unless otherwise noted, we use the term “evidence” to refer to studies presented in this detailed table: that is the main evidence used by the Committee in their evaluations.
Procedures considered by the Program are typically given one of four main types of recommendation, based on judgments on the quality, quantity, and consistency of the available evidence (Table 2) (Reference Campbell, Barnes, Kirby, Willett, Wortley and Lyratzopoulos2;10).
Process for Updating Recommendations:
During 2002–8, recommendations on a small number of procedures were updated in response to advice from the clinical community, usually because of major advances in the evidence base or safety concerns. Examples included endovascular stent grafting of abdominal aortic aneurysms—initial guidance produced in 2003, updated in 2006, after the publication of the pivotal “EVAR” trials (4;5); and foam sclerotherapy for varicose veins—initial guidance produced in 2006, updated in 2009 because of publications raising concerns about the risk of gas embolism (Reference Ceulen, Sommer and Vernooy3). It was recognized that “proactive” (as opposed to simply “reactive”) updating of guidance would need to be addressed and in 2008 a system for doing this was developed, focusing on procedures with “special arrangements” recommendations. Such procedures were considered for updating 3 years after initial publication of recommendations. At that time, among other considerations, advice is sought from expert clinical advisers about any changes in the context that the procedure is being used (for example, because of clinician preferences or service changes) and about any new anecdotal information on its efficacy and safety, and a new literature search may be done.
More specifically, the following steps are used in considering whether “special arrangements” recommendations ought to be updated. Step 1. Clinical experts comment on whether there have been substantial changes in the evidence base since publication of original guidance. Contextual information on potential changes in technique, frequency of use, and reported outcomes is also collected. Step 2. If clinical experts indicate that updating of the guidance may be appropriate, an updated literature search is carried out, and a draft “updated” scoping document is prepared. Step 3. Expert advice and literature search information are discussed by the Program team and a decision to update (or not) the guidance is taken. Considerations that may influence this decision include: The nature and volume of new evidence (e.g., study design, patient numbers, duration of follow-up, new outcomes); whether the new evidence provides data about specific uncertainties expressed in the original guidance; and any other pertinent factors provided by the clinical experts.
Updating of recommendations involves a full new search of the literature and new specialist advice and patient commentary, new production of draft recommendations and new public consultation (i.e., the methods and process for updating guidance are identical to those used for producing new guidance). This system also proactively considers the updating of “research only” guidance, although a decision to update such recommendations could also be triggered reactively, by the publication of new evidence. However, no “research only” guidance was updated during the study period. Proactively considering “normal arrangements” considerations for updating was judged of uncertain benefit and would have required much greater resources, although such updating may be prompted in response to new evidence questioning a procedure's efficacy or safety.
Evaluating Initial Experience with Updating of Recommendations:
For procedures with “special arrangements” recommendations which were updated, we examined associations between (i) recommendations in the original and the updated guidance (specifically whether they remained as “special arrangements” or changed to “normal arrangements”) and (ii) the evidence presented to the Committee (in the detailed overview table described above). In assessing the evidence we specifically considered the number of patients treated by the procedure in all studies, and by each type of study (e.g., RCT, non-randomized comparative study, case series); and the “mean follow-up” duration of patients in all studies, and by each type of study (calculated assuming that all patients in each study were followed up for the mean or median follow-up period which was reported). We examined the significance of differences in the number of additional patients and additional person-years of follow-up between procedures with changed and unchanged recommendations using binary logistic regression.
RESULTS
Between April 2008 and February 2010, eleven procedures, selected on the criteria described above, were re-appraised using the standard processes of the Program and issued with updated recommendations (this is a continuous sample and in that respect these “updates” are typical). The median interval between publication of the initial guidance and the updated guidance was 5.3 years (range, 3.3 to 6.5 years) (Supplementary Table 1, which can be viewed online at www.journals.cambridge.org/thc2012004). Recommendations were changed from “special arrangements” to “normal arrangements” for six and remained as “special arrangements” for the other five.
Most of the additional evidence related to studies other than RCTs (i.e., non-randomized comparative studies, and case series): for procedures with “changed” and “unchanged” recommendations respectively, only 3 percent and 13 percent of additional patients were included in RCTs, whilst 39 percent and 41 percent were included in non-randomized comparative studies and 58 percent and 45 percent in observational (case series) studies.
For the six procedures with changed recommendations, evidence was presented on 8,745 patients (mean of 1458 additional patients per procedure)—an increase of 231 percent compared with the initial appraisal (Table 3). For the five procedures with unchanged “special arrangements” recommendations, evidence was presented on 2,893 patients (a mean of 579 additional patients per procedure)—an increase of 200 percent. The difference between the two groups of procedures (changed / unchanged recommendations) in the mean number of additional patients was not significant (p = .294).
Number of Additional Patients* (Compared to Those Included in Initial Appraisal) Included in Evidence Overviews Presented to Committee Members at the Time of “Updating” Recommendations, by Study Type and Procedure
ESWT, extracorporeal shock-wave therapy; RFA, radio-frequency ablation; IDET, intradiscal electrothermal therapy.
* Negative values denote occasions where the number of patients in the updated evidence overview document (see Methods, Context) included studies with fewer patients compared with the original overview. This has occurred for different reasons, including the “splitting” of an original single item of guidance covering more than one indication to two items of guidance (as for ESWT), restricting the update during the study period of an original guidance covering more than one indication to a single indication (allogeneic pancreatic islet transplantation for type 1 diabetes mellitus) and differences in the selection of evidence.
For the six procedures with changed recommendations, an additional 31,010 patient-years of follow-up were available (representing a 663 percent increase from baseline) whereas an additional 4,863 patient-years of follow up were available for the five procedures with unchanged recommendations (representing a 341 percent increase from baseline, Table 4). The difference between the two groups of procedures (changed / unchanged recommendations) in the mean number of additional years of follow-up was not significant (p = .190).
Additional* Patient-Years of Follow-up Reported in Evidence Overviews at the Time of the Updated Appraisal (Compared to Initial)
ESWT, extracorporeal shock-wave therapy; RFA, radio-frequency ablation; IDET, intradiscal electrothermal therapy.
* See footnote in Table 3 for explanation of negative values.
DISCUSSION
We evaluated the initial experience of a system that was conceived to target resources available for updating clinical practice recommendations for new procedures toward those with inadequate initial evidence, which had resulted in “special arrangements” recommendations. Our findings indicate that this system does have a useful function because half of procedures that were re-appraised had these recommendations changed. While the nature of additional evidence for all procedures was similar, procedures with changed recommendations had greater number of additional patients, and those patients were followed-up for longer periods. This observation shows the importance of these features of the evidence to the Committee in making its recommendations, despite the fact that the data were, by and large, not from RCTs. The evidence from RCTs was limited in quantity overall and there was little difference in the amount of additional RCT evidence between procedures with “changed” and “unchanged” recommendations.
Re-issuing updated yet unchanged “special arrangements” recommendations may be as useful to health services as publishing recommendations which have been changed: reiterating such recommendations on the basis of updated evidence signals ongoing uncertainty about aspects of the efficacy and safety of a procedure—this is informative to patients, clinicians and purchasers of health care, and also to researchers and research-funding bodies who may prioritize investment in further studies about such procedures. The Program produces approximately forty items of guidance each year and has a fixed capacity. Therefore, updating the recommendations of the procedures examined here competed out the production of an equal number of “new” items of guidance (on novel procedures) during the reviewed period. This opportunity cost has to be considered in relation to the perceived benefit of updating. The value that users of clinical practice recommendations put on them being “updated” could be explored further by qualitative research.
Precisely how long an interval to leave before routinely considering an update of recommendations remains a matter for debate, although common practice seems to indicate some consensus for approximately three years (Reference Bastian, Glasziou and Chalmers1;5;Reference Shekelle, Ortiz and Rhodes16). In our experience the cut-off period of three years used by our Program appears to be appropriate in the context of updating recommendations for interventional procedures, based on our findings. It should be noted that although we initiate consideration of updating 3 years after the publication of the original “special arrangements” guidance, the actual production of the final updated recommendations takes longer (typically another year) because of the time intervals required for literature and other evidence to be prepared and presented to the Committee, and for our processes of public consultation and final guidance production.
Our decision to concentrate resources on reviewing guidance with “special” rather than “normal arrangements” recommendations was made pragmatically but seems to be supported by our experience, because occurrences where the efficacy and safety of procedures previously issued with “normal arrangements” guidance has been questioned by subsequent evidence are relatively rare. One example is “Endoscopic saphenous vein harvest for coronary artery bypass grafting,” originally issued with normal arrangements guidance in December 2007. This guidance was subsequently updated in May 2010, in response to publication of evidence which increased the uncertainty about longer term efficacy and safety of the procedure (Reference Lopes, Hafley and Allen9). The main recommendations changed from “normal” to “special” arrangements (11).
Previous studies have used different approaches to judge whether and when evidence reviews or guidelines could be considered to be “out-of-date” (Reference Bastian, Glasziou and Chalmers1;Reference Shekelle, Ortiz and Rhodes16;Reference Shojania, Sampson and Ansari17). In this study a change in clinical practice recommendation (“special-to-normal”) could be considered as a de facto hard outcome that the original recommendation was “out-of-date.” Unlike previous studies (Reference Bastian, Glasziou and Chalmers1;Reference Shekelle, Ortiz and Rhodes16;Reference Shojania, Sampson and Ansari17), our updating of recommendations was done within a “high-stake” context, that is, the updating of clinical practice recommendations for the UK national health services. This contrasts with updating reviews of evidence in the context of academic research, which may not be directly related to changes in clinical practice or service delivery. Another special feature of our study is its focus on guidance for new interventional procedures—an area of clinical practice and medical innovation for which early evidence is typically limited both in quality and quantity (Reference Campbell, Barnes, Kirby, Willett, Wortley and Lyratzopoulos2).
A potential limitation of our study is the type and amount of evidence selected for presentation to the advisory Committee, which was chosen from an overview of the most relevant and valid studies, as opposed to a fuller systematic review typically encompassing all literature on a topic (12). However, this approach has been applied consistently for all procedures since the inception of the Program, and to procedures with both “changed” and “unchanged” recommendations, so it cannot be reasonably expected to have biased our findings. In addition, the Committee members make decisions based on the evidence overviews presented to them, and therefore their use has “face validity.”
The overall association between the amount of additional evidence and probability of changed recommendations is intuitive. This association was generally borne out by our findings but was not uniform for all the procedures, suggesting that other (contextual) factors also influence the Committee's decision-making, in addition to the amount of additional evidence. It is possible that procedures which remained with cautious guidance may be either on a slow upward trajectory of evidence development or on a downward trajectory (with new / additional evidence further building up a picture of unproven efficacy and/or safety).
The fact that the amount of additional evidence from RCTs (rather than non-randomized comparative studies and case series) was not clearly associated with the probability of changed recommendation is worthy of reflection, and several potential explanations can be considered. First, this observation may be a chance finding—particularly if one considers the very small amount of RCT evidence available for many procedures. Second, new RCTs are sometimes unable to resolve evidential uncertainty, and/or had negative or conflicting findings. Third, expert clinical advisers or patient commentators provide cogent views about current practice which may appropriately influence the Committee's judgment—the RCT evidence notwithstanding. Fourth, factors relating to the indications for which procedures are used (such as the prognosis of the underlying condition, or the availability or non-availability of alternative treatment options) may influence the Committee's decision-making, either “raising” or “lowering” the threshold of what it considered to be adequate evidence. Finally, it is possible that large volumes of additional observational evidence are given relatively more weight by the Committee than smaller amounts of additional evidence from RCTs, for example because it may be believed that observational evidence may reflect current, and “real-world” practice more accurately. The limited sample size of our study does not allow these possibilities to be analyzed in more detail, but they may inform future research.
In conclusion, our experience supports the notion that keeping clinical practice recommendations up-to-date with evidence requires dedicated resources and an explicit system to prioritize updating recommendations for interventions with initially limited evidence. Even after prioritization, the outcomes of updates cannot be assumed a priori—nearly half of the interventional procedures in our study were re-issued with recommendations for clinical practice that were similar to the initial ones. Further studies are needed about the value placed on “updated” recommendations by clinicians, policy-makers, and patients. We still need to know more about the opportunity costs that society is prepared to pay for updating recommendations for clinical practice at the expense of opportunities for producing recommendations on newer technologies, when resources are limited.
SUPPLEMENTARY MATERIAL
Supplementary Table 1 www.journals.cambridge.org/thc2012004
CONFLICT OF INTEREST
All authors report having no potential conflicts of interest.
CONTACT INFORMATION
Georgios Lyratzopoulos, MD, FFPH, FRCP, MPH, DTM&H, Consultant Clinical Advisor, Interventional Procedures Program, National Institute for Health and Clinical Excellence (NICE), London, United Kingdom; Cambridge Centre for Health Services Research, Department of Public Health and Primary Care, University of Cambridge, Cambridge, United Kingdom
Steven Barnes, MA, Technical Analyst, Heather Stegenga, MSc, BHSc(Hons), Technical Analyst, Suzi Peden, MSc, BSc(Hons), Interventional Procedures Program, National Institute for Health and Clinical Excellence (NICE), London, United Kingdom
Bruce Campbell, FRCS, FRCP, MS, Chair, Interventional Procedures Program, National Institute for Health and Clinical Excellence (NICE), London, United Kingdom; Peninsula College of Medicine and Dentistry, Exeter, United Kingdom
