INTRODUCTION
A learning curve can be defined as an improvement in performance over time. This improvement tends to be most rapid at first and then tails off over time. Three main features of a learning curve can be recognized. An initial or starting level defines where the performance level begins. The rate of learning measures how quickly a particular level of performance is reached. Last, the asymptote or expert level is the level at which performance stabilizes (see Figure 1). Learning curves have been observed for many health technologies (such as minimal access technologies) but rarely quantified (Reference Ramsay, Grant and Wallace15).
Features of a learning curve.
Poor quantification of learning curves can complicate the design of randomized trials of nondrug technologies. Trials are often designed with limited evidence of learning curve features available, which leaves the trial open to criticism that insufficient account of learning had been taken. Concerns over the presence of a learning curve have particularly hindered surgical trials (Reference McCulloch, Taylor, Sasako, Lovett and Griffin11). Current design approaches to overcome learning curves, such as operators performing a fixed number of procedures before being eligible to participate or “expertise” trials (Reference Devereaux, Bhandari and Clarke4), are often based upon poor evidence and do not necessarily protect studies from criticism (Reference Cook, Ramsay and Fayers1;Reference Douek, Smith, Oshowo, Stoker and Wellwood5). One possible approach to increase understanding of the learning curve for a specific technology is to review the technology's literature, abstracting details of the features of the learning curve. In this study, we illustrate how a literature review of the learning curve can yield information about the nature of learning of a specific technology. Limitations in reporting which hamper this approach are highlighted, and guidance for future research in this area is given.
Example Technology—Fiberoptic (TRACHEAL) Intubation
Fiberoptic tracheal intubation is a technique for the management of the airway and is used for many patients who present difficulties with conventional intubation. Fiberoptic intubation is substantially different from the conventional technique, requiring new practical skills to be learned (Reference Dashfield and Smith2). Several studies have shown that fiberoptic intubation takes longer than conventional intubation (Reference Smith, Mackenzie and Scott-Knight19), and the most common cause for a failed fiberoptic intubation has been identified as a lack of training or experience (Reference Morris13). Lack of expertise has been suggested as the main reason for the current underutilization of the technology in first world countries (Reference Marsland, Robinson, Chitty and Guy10). Despite improved training programs, there is evidence that operators are still increasing their proficiency after initial training (Reference Dashfield and Smith2;Reference Osborn, Jackson and Smith14;Reference Smith, Jackson, Hurdley and Clifton17;Reference Smith and Jackson18). The time taken to perform intubation, the procedure time, is an important outcome measure, because long procedure times are acknowledged to be associated with increased risk of morbidity (Reference Vaughan20).
METHODS
Search Strategy
A search strategy was developed in Medline (up to December week 4 2000) and adapted for use in Embase and Science Citation index. The Cochrane Library (2001 issue 1) was checked for relevant reviews. Several terms were identified for selecting papers that had reported procedure times, especially those that had evaluated operator experience. Differences between British and American spellings were taken into account. Language restrictions were not made. The abstracts of potential papers generated by the search were assessed to identify suitable studies. If the abstract established the relevance of the paper or there was a high likelihood of this, the full paper was acquired. The full papers were checked, and approximately 5 percent were assessed by a second reviewer. The bibliographies of included papers were scanned for additional papers for inclusion.
Inclusion Criteria
Randomized controlled trials (RCTs), crossover trials, controlled clinical trials, cohort studies, and case–series studies were included. All studies that used fiberoptic (nasal or oral) tracheal intubation on adult patients (16+ years of age) and reported a procedure time were included. Studies were not excluded for use of unorthodox equipment as long as this equipment was not deemed to alter the technique substantially.
Assessing Learning
The primary outcome was the procedure time in seconds. Data were abstracted on the equipment used, timing definition, the experience of the operators and any information relating to the starting level, rate of learning, and asymptotic (expert) level. A consultant anesthetist identified groups of studies that used similar equipment and timing definitions. Abstracted data were recorded on a specially developed form. Where possible, starting level, rate of learning, and expert level were combined across studies. As some heterogeneity between study estimates was expected, the DerSimonian and Laird random effects method was used (Reference DerSimonian and Laird3). Pooled estimates along with 95 percent confidence intervals (CIs) were calculated using STATA software release 9.2.
RESULTS
Description of Studies
The database searches produced 499 references: 338 from Medline, a further 134 from Embase, and 27 from Science Citation. No relevant reviews were found in the Cochrane Library. The number of references from each database reflected the order in which they were searched. Of the 499 references, 89 were identified for further investigation and full papers were retrieved. After assessment, thirty-nine studies were included in the review. An additional seven papers were identified from references as possibly being relevant, and of these, four were included, giving a total of forty-three included studies (published on the Web site).
The majority of the included studies (60 percent) were European, with eleven from North America, five in Asia, and one from Australia. Of the included studies, there were twenty-two RCTs, eleven case–series/cohort studies, nine controlled trials, and one crossover trial. Fiberoptic intubation was compared with at least one type of conventional intubation in sixteen studies. A single fiberoptic intervention group was investigated in twenty-nine studies; two or more fiberoptic intervention groups were compared in fourteen studies.
Procedure Time
All forty-three papers reported the time taken to perform fiberoptic intubation. A definition of how the time was recorded was given in forty-one papers with thirty-two different definitions stated (published on the Web site). A variety of different equipment was used between studies. The studies were grouped according to whether they had performed nasotracheal or orotracheal fiberoptic intubation.
Operator Experience
Of the forty-three studies, twenty-one gave explicit information concerning the previous fiberoptic experience of the operator(s). Nine reported no prior experience, seven commented on experience (for example “skilled” or “relative novice”), four gave the number of procedures previously performed, and one reported the experience in terms of the number of years undertaking procedures.
Three studies commented on experience of nonfiberoptic intubation techniques but did not explicitly state no prior experience of fiberoptic intubation. In nine studies, only the professional status of the operator was given. The use of the term “anesthetist” was assumed to suggest that the person was experienced with conventional intubation. Ten studies did not give any explicit information about the operator(s) experience level or professional status. It was likely that the respective authors, who were anesthetists, performed the intubations and were, therefore, experienced in conventional intubation.
Combining Features of the Learning Curve
Given the differences between equipment used and the variation in definitions of timing, only two subsets of the studies were considered sufficiently homogeneous by a consultant anesthetist for grouping together.
Nasotracheal Fiberoptic Intubation
The times of four studies (all case–series) were considered suitable for combining (Reference Dashfield and Smith2;Reference Osborn, Jackson and Smith14;Reference Smith, Jackson, Hurdley and Clifton17;Reference Smith and Jackson18). These studies, all with the same principal author, used the same equipment to perform nasal fiberoptic intubations in at least one intervention group. All operators had a similar level of prior experience and had not performed a “real” fiberoptic intubation before. The definition of procedure time was also consistent between studies. The four series of intubations are shown in Figure 2. All four series suggested that there was a reduction in the procedure time as the experience of the operator increased.
Four nasotracheal fiberoptic tracheal intubation case–series.
There was variation in the initial time point between studies (ranging from 112 to 178). Pooled mean procedure times for the initial level was 133 seconds (95 percent CI, 113–153). The number of intubations performed in each study was low and variable, making it unlikely that the expert level was attained. Pooled mean procedure times for the 10th intubation was 71 seconds (95 percent CI, 62–79), although the final data point in Smith et al. (Reference Smith, Jackson, Hurdley and Clifton17) suggests that a lower level was achievable. This paper gave an estimate of 35 seconds for the asymptote in one paper. Three studies stated a value of 45, for the number of intubations required to reach the asymptote level (Reference Dashfield and Smith2;Reference Smith, Jackson, Hurdley and Clifton17;Reference Smith and Jackson18), two referencing the third (Reference Smith, Jackson, Hurdley and Clifton17). Two papers gave a half-life of nine intubations (Reference Dashfield and Smith2;Reference Smith, Jackson, Hurdley and Clifton17).
Orotracheal Fiberoptic Intubation
The times of three case–series studies were considered suitable for combining (Reference Johnson and Roberts9;Reference Schaefer, Marsch and Keller16;Reference Smith and Jackson18). A further study, compared experienced (consultant) versus inexperienced operators (trainee) (Reference Hartley, Morris and Vaughan8). Although standardized within study, the level of training varied substantially across studies. None of the operators in the case–series studies had previously performed fiberoptic intubation. The procedure time was recorded in a standardized way. The three case–series of intubations are shown in Figure 3 and suggested that there was a reduction in the procedure time as the experience of the operator increased.
Three orotracheal fiberoptic tracheal intubation case–series.
There was large variation in the initial level between studies (ranging from 88 to 240 seconds). Pooled mean procedure times from the three case–series studies for the initial level was 81 seconds (95 percent CI, 49–112). A measure of the rate of decrease was given as a straight line with a slope of -6.0 seconds per intubation (intercept of 106 seconds) over the first 10 procedures (Reference Schaefer, Marsch and Keller16). One study quoted forty-five as the approximate number of intubations required to reach the asymptote level; however, that estimate was quoted from a nasotracheal intubation study (Reference Gravenstein, Melker and Lampotang7). The number of intubations performed in each case–series study was again low, making it unlikely that the expert level was attained. Pooled mean procedure times for the 10th intubation was 51 seconds (95 percent CI, 37–66). The data from the comparative study demonstrated a mean expert level of 33 seconds, and the final data points from the case–series suggest that a level around 40 seconds was achievable.
DISCUSSION
We have demonstrated that it is possible to quantify a learning curve for a health technology using a systematic literature review approach. This approach could inform the design of randomized trials where an operator's “prior expertise” is a necessary design feature (Reference Devereaux, Bhandari and Clarke4). Substantial learning was observed for fiberoptic intubation, which has implications for the design and analysis of clinical trials in this area. Similar magnitude effects may well occur in other technologies. For some techniques, such as novel surgical procedures, learning curves may be particularly important. Clinical trials that fail to quantify and report the learning curves expose themselves to criticism, and their results will as a consequence be less convincing. This criticism also applies to cost-effectiveness analysis where learning could impact upon estimates of cost as well as effectiveness. The net effect of learning on cost is uncertain as, although we may, a priori, expect more experienced operators to have higher effectiveness (for example, shorter procedure times and length of hospitalization), more experienced operators will be more costly.
Expertise trials (Reference Devereaux, Bhandari and Clarke4) can be criticized because “expertise” is often poorly defined. The use of a fixed number of procedures before operator participation can be similarly criticized. The approach outlined here provides an alternative evidence-based method to incorporate learning curve features in trial design. The estimates produced by this approach provide an average measure of learning that can be used to define the required level of expertise. Individual operator differences will persist in the trial, which should be accounted for by an appropriate statistical method (Reference Cook, Ramsay and Fayers1).
The existence of operator learning has been widely reported for fiberoptic intubation, but few papers gave any details on the prior experience of the operators. Even when some information was given, it was often unclear exactly what the level of experience actually was. Only one study that reported an operator as “experienced” quantified the statement by stating the number of intubations previously performed (thirty intubations) (Reference Fridrich, Frass and Krenn6). One study recommended just ten intubations for a operator to achieve “an acceptable level of technical expertise” (Reference Johnson and Roberts9). We recommend that the level of experience should be reported as fully as possible, in lieu of a better measure, the number of procedures previously performed by each operator should be stated along with details of any prior training received.
The procedure time is an important measure of the value of an intubation technique, not only in terms of the use of medical staff time but also with respect to the potential harm to the patient. Prolonged procedure times have increased risk of morbidity, and in rare cases, even death can result. The lack of consistency in the procedure time is a barrier to adopting the review approach to assessment of learning. The number of different approaches to fiberoptic intubation and the variety of equipment available exacerbated this problem. Ideally, the process and timing of fiberoptic intubation should be standardized using a definition that is suitable for different approaches to fiberoptic intubation.
Learning Curve for Fiberoptic Intubation
The pooled estimates from the review suggested that performing ten intubations (oral or nasal) probably accounted for a large part of the learning curve, but little information was given in the literature concerning the asymptotic level of performance or the rate of progression. The handful of case–series that were reported were too short to be conclusive, but were suggestive that times could be improved by a further 20 seconds.
CONCLUSIONS
Learning curves can have a major impact on trial design and on the subsequent reporting and interpretation of the results. The review approach allowed learning to be quantified for our example technology, which in turn informs trial design and cost-effectiveness analysis. However, poor and insufficient reporting constrained formal statistical estimation. Standardized reporting of nondrug technologies—such as an extension of CONSORT statement (Reference Moher, Schulz and Altman12)—should be developed to improve the overall reporting and level of consistency. This strategy could incorporate aspects relevant to the ascertainment of a learning curve such as procedure methodology, inconsistency in timing, and the reporting of operator experience.
CONTACT INFORMATION
Jonathan A. Cook, BSc, PhD (j.a.cook@abdn.ac.uk), Statistician, Craig R. Ramsay, BSc, PGDip, PhD (c.r.ramsay@abdn.ac.uk), Senior Statistician, Health Services Research Unit, University of Aberdeen, Health Sciences Building, Foresterhill, Aberdeen, Aberdeenshire, AB25 2ZD, Scotland
Peter Fayers, PhD (p.fayers@abdn.ac.uk), Professor of Medical Statistics, Department of Public Health, University of Aberdeen, Polwarth Building, Foresterhill, Aberdeen, Aberdeenshire, AB25 2ZD, Scotland; Professor of Medical Statistics, Cancer Research and Molecular Medicine, Norwegian University of Technology and Science, Faculty of Medicine, Trondheim N-7006, Norway
