Healthcare policy makers and providers have an increasing need for independent and objective advice about the usefulness (or lack of usefulness) of healthcare technologies, particularly those that are new. Unlike pharmaceutical products, new interventional procedures may start to disseminate with little published evidence (Reference Campbell, Barnes, Kirby, Willett, Wortley and Lyratzopoulos1). Additional information and opinions from clinicians with specialist knowledge may, therefore, be both desirable and influential when producing advice about their use.
Previous research indicates that the conclusions of both meta-analyses and randomized trial publications may be favorably influenced by the organizational and financial affiliations of the authors (Reference Jorgensen, Hilden and Gotzsche4;Reference Kjaergard and Als-Nielsen5). Financial interests may also result in a relatively more favorable assessment of the cost-effectiveness of interventions (Reference Chauhan, Miners and Fischer3). Although an experimental study has suggested that clinicians' professional backgrounds may influence the content of clinical guidelines (Reference Carpenter, Hutchings, Raine and Sanderson2), little is known about how expert clinical adviser background and experience may influence opinions about the clinical utility of interventional procedures.
The Interventional Procedures Programme of the UK's National Institute for Health and Clinical Excellence (NICE) has produced guidance since 2002, based on reviews of published evidence and opinions of expert clinical advisers (Reference Campbell, Barnes, Kirby, Willett, Wortley and Lyratzopoulos1;6;7). The Program's objective is to appraise the efficacy and safety of emerging interventional procedures, notified by clinicians, patients, manufacturers, or other interested people. Interventional procedures are defined as those involving an incision, puncture, or entry into a body cavity, or the use of ionizing, electromagnetic, or acoustic energy. Such procedures range from radiofrequency ablation of hepatocellular carcinoma to laser lumbar discectomy; and from vagal nerve stimulation for epilepsy to uterine artery embolization of fibroids. The process and methods used have been previously described in detail (6;7). Briefly, overviews of published evidence and commentary from expert clinical advisers is presented to an independent Advisory Committee (the Interventional Procedures Advisory Committee). The Committee drafts guidance, which it reconsiders after a month of public consultation. Guidance is then ratified by NICE's Guidance Executive and published. We used NICE's Interventional Procedures Programme as a “real-world” setting to examine whether the opinion of expert clinical advisers about whether an interventional procedure is established, efficacious, and safe is influenced by their professional experience and background.
METHODS
Expert clinical advisers to NICE's Interventional Procedures Programme are nominated by UK specialist societies or associations, in response to requests for advice about specific procedures relevant to their specialties. They provide advice in writing, by completing a purpose-made structured questionnaire. For this study, information was extracted from relevant fields of the questionnaires. Three domains relating to (self-declared) professional experience and background were defined (“operator” [carries out, or has carried out the procedure], “researcher” [has undertaken research relating to the procedure], and “conflict of interest”) and responses dichotomized into “yes” or “no” categories (see Supplementary Materials File 1: Methods, which can be viewed online at http://www.journals.cambridge.org/jid_thc). The three categories were not exclusive of each other, that is, an expert clinical adviser could be both “operator” and “researcher” and also declare a conflict of interest. In addition, three domains were used for expert clinical advisers' opinions about the clinical utility of procedures (whether they were established [as opposed to novel], efficacious, and safe) and, similarly, information was dichotomized. Data extraction was carried out by two independent researchers, using a preconstructed and piloted data extraction manual including precise description of data fields and coding definitions. A third researcher verified the appropriateness of coding, and a fourth, more senior researcher was used as the final arbiter in cases of uncertainty.
Separate cross-tabulations between each of operator, researcher, and conflict of interest status and opinion on whether a procedure is established, efficacious, or safe were carried out, and the significance of any possible associations was assessed with the χ2 test (univariate analysis). Subsequently, three separate multiple logistic regression models were used, by treating each of the clinical utility domains (i.e., whether a procedure was established, efficacious, and safe) as the binomial outcome of interest, and adjusting for all expert clinical adviser characteristics (operator, researcher, and conflict of interest status; multivariate analysis). These three models provide information about the direction and degree of any association between a given individual characteristic (say, whether an adviser has operative experience of the procedure of interest) and the probability of a favorable opinion being expressed in relation to whether a procedure is established, efficacious, or safe, adjusting for all other individual characteristics (say, researcher and conflict of interest status).
RESULTS
Information was available on 673 different expert clinical adviser questionnaires relating to 183 different interventional procedures. Seventy-five questionnaires were excluded: fifty-three because they related to procedures for which no final guidance was issued, and twenty-eight because expert clinical advisers stated that they did not have adequate knowledge of the procedure (six of those twenty-eight questionnaires also fell into the “no related piece of interventional procedures guidance issued” category). Following these exclusions, there were 598 expert clinical adviser questionnaires relating to 182 different interventional procedures—on average, 3.3 clinical expert adviser questionnaires per procedure.
Ascertainment completeness was 75 percent, 74 percent, and 78 percent for operator, researcher, and conflict of interest status, respectively. For the clinical utility domains, ascertainment completeness was 93 percent, 94 percent, and 95 percent for opinions about whether a procedure was established, efficacious, and safe, respectively. Excluding records with missing data, 54 percent of all questionnaires were completed by expert clinical advisers who were operators, 28 percent by advisers who were researchers, and 25 percent by advisers with a conflict of interest (an expert clinical adviser could have replied as “yes” in more than one of the above domains).
Excluding from the denominators those questionnaires in which relevant information was missing, in nearly half the questionnaires the procedure was judged as established (49 percent; 265/547) and safe (54 percent; 306/569); in nearly one quarter of the questionnaires it was judged to be efficacious (24 percent; 136/564).
In multivariate analysis (i.e., adjusting for each other of the professional experience and background characteristics), operators were significantly more likely to consider a procedure as established (odds ratio [OR] 3.93; 95 percent confidence interval [CI], 2.43 to 6.36; p < .001), efficacious (OR 1.76; 95 percent CI, 1.00 to 3.08; p = .049), and safe (OR 2.28; 95 percent CI, 1.43 to 3.65; p = .001).
In multivariate analysis, no significant associations between researcher status and opinions about whether a procedure is established, efficacious, and safe were observed (OR values: 1.49 [95 percent CI, 0.87 to 2.56], 1.20 [95 percent CI, 0.67 to 2.16], and 1.52 [95 percent CI, 0.91 to 2.52], respectively), despite the univariate analysis being suggestive that researchers were more likely to express such favorable opinions (see Supplementary Materials File 2: Results, which can be viewed online at http://www.journals.cambridge.org/jid_thc). Similarly, in multivariate analysis there were no significant associations between conflict of interest status and opinions about whether a procedure is established, efficacious, and safe (OR values: 0.80 [95 percent CI, 0.44 to 1.46], 1.21 [95 percent CI, 0.64 to 2.27], and 1.30 (95 percent CI, 0.75 to 2.27], respectively), despite the univariate analysis being suggestive that advisers who declared a conflict of interest were more likely to consider a procedure as safe (see Supplementary Materials File 2: Results, which can be viewed online at http://www.journals.cambridge.org/jid_thc).
DISCUSSION
This study provides empirical evidence about associations between the professional background and experience of expert clinical advisers and their judgments on the utility of new interventional procedures, within the context of NICE's Interventional Procedures Programme. The findings suggest that operative experience for a given procedure is a strong and independent predictor of a favorable clinical opinion about whether that procedure can be considered established, efficacious, and safe.
Conversely, research experience and self-declared conflict of interest status relating to a procedure were not independently associated with a favorable opinion of a procedure's clinical utility, despite the univariate analysis being indicative of some such associations. Specialist advisers who are operators may also be researchers and/or also have a conflict of interest—and these three different characteristics can confound each other. Adjustment for each other of the different expert clinical adviser characteristics, as carried out in this study, is, therefore, advisable.
Previous research has suggested a significant association between financial affiliations of authors and favorable conclusions in research papers or health technology appraisal submissions (3–5). There is also evidence that different medical specialties may reach different conclusions about a given healthcare technology (2). Our study used a “real-world” as opposed to an experimental research setting: we believe this “naturalistic” setting is an important asset.
That the study used “routinely collected” data, originally collected for service (as opposed to research) purposes, could be regarded as a limitation. Categorization of opinions about clinical utility was post hoc, and based on a posteriori judgment by the researchers. Nevertheless, in most instances the advice provided was “categorical” in itself, and interpretation of opinions was only required in a minority of questionnaires (less than 15 percent). Misclassification of “true opinions” about a procedure's clinical utility is unlikely to have been systematic (i.e., consistently in favor of either a “positive” or a “negative” opinion). Therefore, such potential misclassification would have made any potential associations between professional experience and clinical utility domains more difficult to detect. This means that, although it is possible that true and significant associations between researcher and conflict of interest status and opinions about clinical utility might have been “missed” by this study, the observed associations relating to operator status are valid and, perhaps, conservative.
The findings of the study, including the observed favorable influence exerted by operative experience on the clinical utility of procedures, should not be interpreted as a type of “biased” behavior by expert clinicians. Bias assumes the systematic and consistent deviation from the “truth” or from an objective correct assessment. However, there is no universally accepted “true” opinion or “correct” assessment when dealing with the efficacy and safety of these procedures: there are commonly no randomized studies, and the total amount of published evidence is often very limited, both in terms of patient numbers and length of follow-up (Reference Campbell, Barnes, Kirby, Willett, Wortley and Lyratzopoulos1).
Direct operative experience with a given interventional procedure was a strong predictor of a favorable opinion about it. This is not surprising, because operators carrying out a procedure outside a research context should be sufficiently convinced about its safety and efficacy to have it offered to their patients. They also have first-hand experience of the effect of the procedure on patients, and this experience is likely to have been favorable if they are continuing to use it. Occasionally, expert clinicians reported that they had used a procedure but had abandoned it because of concerns about efficacy or safety—but such reports were un-common.
Interpretation is more difficult for the observed lack of association between researcher status and opinion about whether a procedure is established, efficacious, or safe. Experts may be actively involved in research either because they are in equipoise about a procedure (for example, in the context of controlled studies), or they may already have a preconceived favorable opinion of a procedure, and are carrying out “research” to help further introduce or develop it. Over time, experts who in the past were involved in research about a procedure are likely to form a diverse group who may subsequently have either adopted the procedure in their practice, or who no longer use it. The observed absence of associations between researcher and conflict of interest status and the clinical utility of a related procedure may reflect a true lack of association, or may be because of chance (small sample size). Our study, because of its “real-world” nature, did not include an a priori calculation of a likely effect size and sample size: we analyzed all information that was available to us. Conflicts of interest are a major concern to organizations such as NICE, which publish guidance that may influence clinical practice on a wide scale, with potentially important commercial consequences. It was, therefore, somewhat reassuring that self-declared conflict of interest was not associated with “positive” opinions about procedure, although it is possible that incomplete information might be responsible for the observed lack of association. The study did not examine specific types of conflict of interest separately (for example, distinguishing between material conflicts of interest and intellectual ones).
The opinions of expert clinical advisers are an important source of information for decision makers producing guidance about the use of interventional procedures, especially when published evidence is sparse or of poor quality. These experts are bound to have a variety of conflicts of interest, as well as a variety of previous operator and researcher experience that could, at least theoretically, influence their opinions. This study has shed some light on how their previous professional experience and background may influence their advice, but further “real-world” studies on this subject would be useful. In any such future work, reference to some kind of universally accepted “correct” assessment about the clinical utility of interventional procedures would be an advantage, although this is likely to prove elusive. Meanwhile, obtaining the views of several different clinical experts, with varying backgrounds (in particular of clinical experts both with and without operative experience of a procedure), seems a reasonable way of supplementing published evidence about procedures. That is the method we continue to advocate.
CONTACT INFORMATION
Georgios Lyratzopoulos, MD, FFPH, MRCP, MPH, DTM&H (georgios.lyratzopoulos@nice.org.uk), Consultant Clinical Adviser, Andrew R. Hoy, MA(Hons) (andrew.hoy@nice.org.uk), Research and Development Analyst, National Institute for Health and Clinical Excellence, MidCity Place, 71 High Holborn, London, WC1V 6NA, UK
Darmarajah Veeramootoo, MBBS, MRCS (docrajv@hotmail.com), Trust Registrar—General Surgery, Nava V. Shanmuganathan, MBBS, BSc (vithya@doctors.org.uk), ST1 Vocational Training Scheme, Royal Devon and Exeter NHS Foundation Trust, Barrack Road, Exeter, EX2 4DN, UK
Bruce Campbell, MB, MS, FRCP, FRCS (bruce.campbell@nice.org.uk), Chair, Interventional Procedures Advisory Committee, National Institute for Health and Clinical Excellence, MidCity Place, 71 High Holborn, London, WC1V 6NA, UK
