Interventional procedures of all kinds have traditionally been introduced into clinical practice in a variable and often haphazard way. Many represent the evolution of existing methods for treatment or diagnosis, but even quite novel techniques have often been pioneered by enthusiasts and have then diffused into practice without any organized evaluation or monitoring and with often sparse evidence and of limited quality (1;2;6;7;10;13) . This finding has also applied to the use of innovative medical devices, which in Europe, for example, simply require a “CE mark,” representing a manufacturer's declaration that the device conforms to all applicable European Union legislation and that the appropriate assessment procedures have been completed (4). Such devices can then be used without further monitoring. All this is in stark contrast to pharmaceutical products, which have been subject to strict regulation and monitoring for many years throughout the world.
Recent methodological research has provided empirical evidence about the potential for biased (“optimistic”) assessment of the efficacy and safety of new healthcare technologies early after their emergence (5;9). New interventional procedures may, therefore, be at particular risk of both imperfect and limited evidence, and of being subject to “optimism” bias.
In recent years, many countries have started to consider the challenges of evaluating new treatments, and new interventional procedures in particular, and of monitoring their introduction into healthcare systems. The International Association for Health Technology Assessment (INAHTA) currently has member organizations from twenty-two different countries (8). The prime concerns are patient safety and delivery of the most efficacious treatments. Selecting appropriate patients, informing them well, and delivering care by appropriately trained clinicians and teams are all important related issues. Costs are also relevant—new interventions need to be reasonable value for money and not to be targets for frequent litigation.
The Department of Health directed the National Institute for Health and Clinical Excellence (NICE) to address the safety and efficacy of new interventional procedures in 2001. The Institute established its Interventional Procedures Advisory Committee (“the Committee”), which met for the first time in October 2002 and which has considered the safety and efficacy of procedures at monthly meetings ever since. The Committee drafts guidance, which it reconsiders after a month of public consultation. Guidance is then ratified by the NICE Guidance Executive and is published for the National Health Services in England, Wales, Scotland, and Northern Ireland (14). Private medical insurers and private hospitals in the United Kingdom also have an expectation that this guidance will be observed by clinicians.
This study concerns the evidence used by the Committee to evaluate interventional procedures and to produce guidance on their safety and efficacy. The aim of the study was to review the type and amount of published evidence for procedures that received recommendations for “normal arrangements” compared with the evidence for procedures with recommendations for “special arrangements” for clinical governance, consent, and audit (“cautionary guidance”).
METHODS
Procedures, Evidence, and Type of Recommendations
Procedures reviewed in this study comprise an initial list of procedures originally compiled by the preceding Safety and Efficacy Register of New Interventional Procedures (SERNIP), before the Programme's initiation in October 2002 (3), and a second group of procedures, received by prospective direct notification post-October 2002 (14;15). For most procedures, relevant evidence was identified through in-house rapid literature searches, although for some of the initial procedures, external reviews or systematic reviews were also used. The Committee's recommendations fell into two broad categories: guidance either stated that evidence on safety and efficacy was adequate for the procedure to be used with normal arrangements for governance, consent, and audit (“normal arrangements”); or that the evidence was inadequate or uncertain, so the procedure should only be used with special arrangements for governance, consent, and audit (“cautionary guidance”). Among procedures with “cautionary guidance” there was sometimes adequate evidence to support the safety of the procedure but not its efficacy and visa versa. Additional recommendations were often made about training, multidisciplinary teams, contribution to registries, and research, but these were not the focus of the present study.
Definition and Measurement of Independent Variables
Three independent variables were defined—“study type,” “sample size,” and “mean follow-up length”: their associations were examined with type of recommendation issued—“normal arrangements” or “cautionary guidance.”
A “study type” category was ascribed to each individual published study presented to the Committee as part of the evidence relating to individual procedures as follows:
- studies providing randomized controlled evidence (either from systematic reviews or randomized controlled trials (RCTs) or primary RCTs);
- studies providing nonrandomized comparative evidence; and
- studies providing evidence without comparator information (case series or case reports).
Each procedure was ascribed a category of evidence, based on the “highest category” (“a” being highest, and “c” lowest) available for it in any published study. For example, if there was at least one study providing randomized controlled evidence then the procedure was judged to be supported by randomized controlled evidence, independently of the fact that additional evidence from other study types (such as case series) might have also been included in the literature overview presented to the Committee.
“Sample size” was defined as the total number of patients reported in the studies who had been treated with the procedure.
“Mean follow-up length” was the sum of the product of the mean follow-up length and study sample size for each of the studies relating to a particular procedure, divided by the total sample size of all studies relating to the procedure.
Statistical Analysis
Associations between study type and type of recommendation were examined by cross-tabulating type of recommendation with study type, treating the latter as a dichotomous variable of randomized controlled (defined as category “a” above) versus all other studies (categories “b” + “c”). For the group of procedures not supported by randomized controlled studies, a further cross-tabulation was performed by subdividing “study type” into studies providing nonrandomized comparative evidence (category “b” above) and case series/case reports (category “c” above). Statistical significance was examined using the chi-squared test.
The median sample size and median mean follow-up length of procedures with “normal arrangements” and “cautionary guidance” were calculated. The significance of associations of sample size and mean follow-up length with type of recommendation was examined by the Mann–Whitney U-test.
Lastly, in multivariable analysis, a multiple logistic regression model was used to examine the simultaneous effects of study type, sample size, and follow-up length on the probability of a “normal arrangements” recommendation being issued, adjusting for notification era (SERNIP or prospective notification). The reported odds ratios (ORs) for sample size and mean follow-up length denote the change in the probability of a “normal arrangements” recommendation being issued by one-patient increase in sample size and 1-month increase in mean follow-up length, respectively. For type of evidence, the OR values denote the change in probability of a “normal arrangements” recommendation associated with the availability of type “a” (versus type “b + c”) evidence, or the availability of type “b” (versus type “c”) evidence, as applicable.
RESULTS
Guidance was issued on 130 interventional procedures during the study period, from the inception of the Programme in October 2002 to the end of September 2005. For seventy (54 percent) procedures, “normal arrangements” recommendations were issued, whereas sixty (46 percent) procedures were issued with “cautionary guidance.”
The overviews presented to the Committee on the 130 procedures included information on a total of 736 studies in detailed tables (a median of five studies per procedure). These comprised 85 (12 percent) category “a” studies of randomized controlled evidence; 135 (18 percent) category “b” evidence, of nonrandomized controlled studies; and 516 (70 percent) category “c” studies, containing evidence from case series or case reports.
In total, the 736 studies included information on 183,729 patients treated with the relevant procedures (range, 1–86,418; median, 394). The study with a sample size of 86,416 patients was much larger than any other, accounting for 47 percent of all the patients reported in all 736 studies (it was a study of laser surgery to the cornea for refractive errors). Excluding this single study, the distribution of sample size by procedure was approximately normal (right skewed; Figure 1). The subset of controlled (randomized or nonrandomized) studies also included information on 23,672 patients treated with comparator procedures.
Sample size (number of patients treated) by procedure (n=129, excluding a single study with very large sample size; see text).
For 106 of the 130 procedures, the reported data allowed calculation of mean follow-up length across all studies. The distribution of follow-up length is described in Figure 2.
Distribution of mean follow-up length (in months) by procedure (n=106).
There was randomized controlled evidence (i.e., at least one randomized study) for forty-four (34 percent) procedures. Of the remaining eighty-six procedures, thirty-four (26 percent) had evidence from at least one nonrandomized controlled trial and fifty-two (40 percent) had evidence only from case series/reports. Procedures with randomized controlled evidence were more likely to be issued with a “normal arrangements” recommendation than with “cautionary guidance” (66 percent versus 48 percent, respectively; relative risk, 1.38; p = .063; Table 1). No procedure with “cautionary guidance” was supported by a systematic review of RCTs.
Among the eighty-six procedures without randomized controlled evidence, those with evidence from nonrandomized controlled trials (category “b”) were significantly more likely to be issued with a “normal arrangements” recommendation than those with only case series/reports (category “c” evidence; 62 percent versus 38 percent; relative risk, 1.61; p=.047; Table 2).
Excluding the single study accounting for 47 percent of all treated patients, the median sample size of studies relating to procedures for which “normal arrangements” recommendations were issued was significantly larger (n=605; range, 26–6,842) compared with the median sample size of studies relating to procedures for which “cautionary guidance” was issued (n=240; range, 1–3,261; p<.001).
Mean follow-up length was longer for procedures that received recommendations for “normal arrangements” (median, 16.7 months; range, 0–84 months) than for those that received “cautionary guidance” (median, 14.6 months; range, 0–67 months), although this difference did not reach statistical significance (p=.160).
Finally, in multivariable analysis, randomized controlled evidence, larger sample size and longer mean follow-up length were all associated with greater probability of a “normal arrangements” recommendation, although the association was only significant for presence of randomized controlled evidence and sample size (Table 3). When restricting the analysis to those procedures without randomized controlled evidence (n=68), the same associations were observed, although none of them was significant (Table 4).
DISCUSSION
The differences observed in type and amount of evidence leading to decisions for “normal arrangements” or “cautionary guidance” are, in many ways, unsurprising. However, these decisions were made individually on each procedure at different times over a period of 3.5 years; and they include the “learning curve” of the Interventional Procedures Advisory Committee at NICE. This review of our experience gives some reassurance that factors which ought logically to have influenced NICE's recommendations generally did so—the category of evidence available, the numbers of patients treated, and follow-up length.
A potential criticism of the Interventional Procedures Programme method of working is that literature reviews, although thorough, were not “systematic” and it is, therefore, possible that relevant studies might have been missed. This is unlikely, because any important studies that were missing would probably have been proffered during public consultation: consultees frequently cited studies that had either been included in the overviews already or that had not been published (but not studies that had been “missed”). Moreover, empirical evidence suggests that nonsystematic, rapid literature review protocols may be as good as systematic reviews and they are more cost-effective (12). Rapid production of guidance is a key feature of the Interventional Procedures Programme.
The Committee regularly debates the process of decision making and the kinds of recommendations that it produces, so these will continue to evolve. A list of factors has been developed to guide decision making, but the great diversity of procedures and indications for their use makes this challenging (16). There is a tendency for the Committee to regard evidence that is limited in quality or volume (or both) as adequate for uncommon, life-threatening conditions, especially if no good alternative treatments exist. By contrast, even small numbers of serious side effects may raise important safety concerns for procedures intended to treat non–life-threatening conditions, such as those that cause only “cosmetic” problems. It is worth noting that the amount and quality of published evidence about a procedure has the potential to influence recommendations in different ways. For example, on occasions a large volume of high quality evidence might show a procedure to lack efficacy or to be relatively unsafe.
One of the striking observations of this study was the relatively small number of procedures that had been investigated by RCTs, reaffirming the previously described paucity of such trials (1;7). There are many factors that mitigate against RCTs for new procedures—not least the increasing bureaucratic hurdles to be negotiated in setting up research. Procedures often disseminate rapidly, led by patient demand, financial motives, and of clinicians' convictions that “they work.” Especially when a new procedure is an alternative to a well-established and apparently efficacious one, a large trial might well be required even to demonstrate equivalent efficacy. When procedures are used in the treatment of potentially fatal diseases, there may be ethical considerations that make RCTs difficult or impossible to conduct.
Both convention and hierarchical categorizations of evidence regard RCTs as the “gold standard.” However, it should be noted that the data from early RCTs on interventions (particularly small RCTs) may subsequently be refuted (9). In addition, RCTs are not necessarily the best means of obtaining safety information—especially data on uncommon but serious side effects. Although data from nonrandomized trials may not always be as comprehensive as conventionally thought (11), safety information may be best provided by large case series, registries, or even case reports.
Length of follow-up is often limited: this finding is to be expected when dealing with procedures that are relatively new. Some procedures for treatment of cancer posed special challenges in this respect, for example, those used for malignancies with long survival times, such as prostate cancer. A frequent dilemma was how much reliance to place on surrogate outcomes such as scans or levels of bloodstream markers of malignancy, rather than on survival or quality of life. There was generally a strong preference for clinical rather than surrogate outcomes, such as results of imaging or laboratory tests.
This study has reported that rather more procedures were recommended for “normal arrangements” than for “cautionary guidance.” The findings suggest that, to an extent, this is because the Programme inherited many of its initial procedures from a pre-existing register: some had been on the register for several years and had become established, supported by good or moderate evidence.
The acid test for draft guidance is the 4-week period of public consultation on the NICE Web site, when any interested individual can submit comments (14;17). There have been few occasions on which the main recommendation (i.e., “normal arrangements” or “cautionary guidance”) has been criticized. Most comments are about matters of detail and not about the Committee's overall appraisal of the evidence. While this is generally reassuring, there have been occasions on which cogent comments by consultees have prompted major revision of the guidance with important changes to the recommendations, or even a request for a full systematic review before reconsidering the procedure. The process of public consultation is a vital “safety net,” especially when the Committee is uncertain or divided.
Implementation of Interventional Procedures Guidance is an issue. The well-developed systems for clinical governance and for inspection of hospitals in the United Kingdom underpin the adoption of NICE guidance both in the National Health Service and in the private healthcare sector. We believe that working closely with professional organizations (in particular medical specialist societies) is fundamental and is one important route to the hearts and minds of practicing clinicians. Our guidance to them when the evidence for a procedure is uncertain or inadequate simply spells out: “Tell your hospital; tell your patients; and audit your results”—these are no more than principles of good clinical practice. An important message to individual clinicians and commercial developers of new interventional procedures is to produce better evidence about safety and efficacy, such as that provided by RCTs with large sample sizes and long-term follow-up, which are more likely to influence decision makers toward positive decisions.
CONTACT INFORMATION
William B. Campbell, MS, FRCP, FRCS (bruce.campbell@nice.org.uk), Chair, Interventional Procedures Programme, National Institute for Health and Clinical Excellence, 71 High Holborn, London WC1V 6NA; Consultant Vascular Surgeon, Royal Devon and Exeter Hospital, Barrack Road, Exeter, Devon EX2 5DW, UK
Steven J. Barnes, MA (steven.barnes@nice.org.uk), Interventional Procedures Analyst, Centre for Health Technology Evaluation, Rebecca A. Kirby, BSc (rebecca.kirby@ nice.org.uk), Former Interventional Procedures Coordinator, Sarah L. Willett, BA (sarah.willitte@nice.org.uk), Project Manager, Interventional Procedures Programme, Sally Wortley, MPH (sally.wortley@nice.org.uk), Interventional Procedures Analyst, Centre for Health Technology Assessment, Georgios Lyratzopoulos, FFPH, MRCP, MPH, DTM&H (georgios.lyratzopoulos@nice.org.uk), Consultant Clinical Advisor, Interventional Procedures Programme, National Institute for Health and Clinical Excellence, MidCity Place, 71 High Holborn, London WC1V 6NA, UK
We acknowledge NICE's Information Services Department for conducting the initial literature searches in relation to the interventional procedures examined in this study. The authors have not received any funding from external sources for the conduct of this study. Ethical approval was not required. All authors declare that they have no conflict of interest. B.C. conceived and supervised the study, with contribution from all other authors, and was the principal author. S.B. collected data and performed analyses. R.K. S.Wi., and S.Wo. collected and collated data. G.L. performed data analysis and is the main coauthor. All authors contributed to the interpretation of the study findings. All the authors contributed to preparation of the manuscript. B.C. is guarantor.
