Introduction
Temporal bone surgery is a key skill of the otorhinolaryngology resident. A shortage of cadaveric temporal bones makes virtual reality simulation training a promising alternative to traditional cadaveric dissection for mastoidectomy training.Reference George and De1, Reference Mills and Lee2 In addition, self-directed training using simulator-integrated tutoring may reduce the need for dedicated expert instructors, whose availability is often limited by their busy schedules and clinical duties. Residents could acquire some basic competencies in the virtual environment, thus saving cadavers and costly instructional resources for subsequent, more advanced training. Tutoring in medical technical skills training is diverse and has been demonstrated to be effective in different settings.Reference Wiet, Stredney, Sessanna, Bryan, Welling and Schmalbrock3–Reference Ungi, Sargent, Moult, Lasso, Pinter and McGraw5 The role of simulator-integrated tutoring in facilitating mastoidectomy skills is largely unexplored.Reference Sewell, Morris, Blevins, Dutta, Agrawal and Barbagli6
Performance assessment is essential for monitoring and tracking the progress of a trainee's technical skills and for establishing the effect of learning interventions such as virtual reality simulation training. Several assessment tools for evaluating mastoidectomy performance have been developed.Reference Zirkle, Taplin, Anthony and Dubrowski7–Reference Butler and Wiet9 These are based either on the process, such as global rating scales which require live or recorded observation of the performance or on rating the final product.Reference Martin, Regehr, Reznick, Macrae, Murnaghan and Hutchinson10, Reference Szalay, MacRae, Regehr and Reznick11 Final-product analysis does not consider how the goal is reached and is normally used for performance assessment after the procedure is finished. Nevertheless, final product like items were found to be relevant in assessing mastoidectomy competency.Reference Andersen, Cayé-Thomasen and Sølvsten12, Reference Wan, Wiet, Welling, Kerwin and Stredney13 Final-product analysis may have limitations if used to monitor progress and technical skills development in novices: if the analysis is not applied at the optimal time point, the score may not reflect the peak performance but instead a lower performance. This could occur because the novice either was not allowed adequate time to perform the procedure or lacks knowledge on when to stop and proceeds to damage vital structures. In addition to the peak effect, final-product analysis could be limited by a ceiling effect, in which a performance plateau is reached because the scale lacks discriminatory power,Reference Munz, Moorthy, Bann, Shah, Ivanova and Darzi14, Reference Moorthy, Munz, Adams, Pandey and Darzi15 or simply because self-directed training alone is insufficient for novices to progress beyond a certain level.
This study aimed to establish whether ceiling and peak effects limit final-product analysis of novice mastoidectomy performance in a virtual reality temporal bone simulator. It also investigates whether additional simulator-integrated tutoring can counter these effects.
Materials and methods
Twenty-two medical students from the University of Copenhagen, Denmark, volunteered to participate in the study and provided written informed consent. They were all novices with no previous exposure to or experience with temporal bone surgery.
All participants received a one-hour introductory lecture on temporal bone anatomy, the mastoidectomy procedure and the virtual reality simulator. They were then asked to perform a complete mastoidectomy with posterior tympanotomy in the Visible Ear Simulator. The Visible Ear Simulator is a freeware real-time three-dimensional (3D) virtual temporal bone simulator that can be downloaded from the Internet.16 The software runs on a standard personal computer with a Nvidia Geforce GTX™ graphics card and supports the Geomagic Touch (3D Systems, Rock Hill, South Carolina, USA) haptic device for force-feedback and intuitive drilling.Reference Sorensen, Mosegaard and Trier17, Reference Trier, Noe, Sørensen and Mosegaard18 The Visible Ear Simulator provides a step-by-step tutorial of the surgical procedure, along with text and illustrations from the simulator and the option of enabling an accompanying integrated tutor function with volumetric green-lighting of the volume to be drilled in each step of the tutorial (Figure 1). An experimental version of the simulator was developed to autosave successive copies of the virtual temporal bones at pre-defined intervals during the procedure.
Image showing a screenshot of the simulator with the on-screen tutorial (left panel) and the corresponding green highlighted volume to be drilled in the surgical step (main image).
Participants were allowed 60 minutes to perform the procedure. They were self-directed and provided with the on-screen tutorial. Participants were randomized to have the on-screen tutorial supplemented by the simulator-integrated tutoring (volumetric green-lighting). Participants were quasi-randomized by the day of training using computer-based assignment.
The simulator autosaved the virtual temporal bones at 10-minute intervals between 20 and 60 minutes. The final products were later analysed by two senior otologists blinded to the participant, the time of saving and the use of tutoring. The saved files were opened in the simulator and rated using a modified Welling Scale with 26 binary items for assessing the mastoidectomy performance.Reference Andersen, Cayé-Thomasen and Sølvsten12
Data were analysed using IBM SPSS Statistics software version 22.0 (Armonk, New York, USA) for MacOS X. Parametric statistics were used because data were normally distributed according to the Shapiro–Wilks test. When comparing continuous data, dependent or independent sample t-tests were performed as appropriate. Fisher's exact test was used to compare categorical data between groups. Statistical significance was set at a p value less than 0.05.
Results
Baseline characteristics such as age, sex, years of medical study, and simulation and computer-related experience and interests are shown in Table I. There were no significant differences in baseline characteristics between the tutored and non-tutored groups.
Baseline characteristics
*On a five-item Likert-like scale. IT = information technology
The mean final-product score for all participants was 17.3 at 60 minutes (the maximum time allowed) (Table II). Consistent with a peak effect, the highest score at any time point during the procedure (i.e. the peak score) was higher (18.0; p < 0.01). None of the participants achieved the maximum score of 26 and only 3 reached a final-product score of more than 20 at some point during the procedure, reflecting a ceiling effect.
Mean final-product scores
*Significant difference between the ‘early peakers’ peak and 60-minute scores (p < 0.01). CI = confidence interval
For further analysis, participants were divided into two groups according to whether their scores peaked during the procedure and then deteriorated (‘early peakers’) or continued to increase throughout the procedure (‘late peakers’; Table II). Ten of the 22 participants (45.5 per cent) achieved their highest (i.e. peak) score before 60 minutes and their final-product performance progressively deteriorated thereafter. The mean peak score was significantly higher in the early peaker group than in the late-peaker group (18.9 vs 17.2; p < 0.01; Table II). As a group, the early peakers consistently outperformed the late peakers throughout the procedure, except at the 60-minute point when both groups had equal scores (Figure 2).
Graph showing the performance of the early and late peakers. FPA = final-product analysis
Participants randomised to simulator-integrated tutoring with volumetric green-lighting had a similar performance level to the non-tutored participants during the first 40 minutes. However, after this point the tutored participants outperformed the non-tutored participants (Figure 3). This was reflected in significantly higher mean peak and 60-minute scores for the tutored group (Table II). Although simulator-integrated tutoring led to a better performance, it did not eliminate the peak and ceiling effects: Fisher's exact test showed a similar distribution of tutored and non-tutored participants for the early- and late-peaker groups (p = 0.69), the top 15 per cent scores were achieved by a 2:1 ratio of tutored to non-tutored participants, and the top 25 per cent highest scores were achieved by equal proportions of tutored to non-tutored participants.
Graph showing the performance of the tutored and non-tutored groups. FPA = final-product analysis
Discussion
This prospective study into the limitations of final-product analysis of mastoidectomy performance used a virtual reality simulator with 10-minute interval autosave to map the final-product progress. It found both peak and ceiling effects. Additional simulator-integrated tutoring with volumetric green-lighting did not eliminate these effects.
Little is known about peak and ceiling effects in final-product performance assessment. In most studies on mastoidectomy, novices are given a certain timeframe within to finish the procedure and final-product analysis is applied to assess performance after the given time. This timeframe is assumed to be sufficient to finish the procedure for all novices without continuing and possibly damaging the final product. However, this requires novices to know the procedure in depth beforehand. Little consideration has been given to the fact that not all novices possess this knowledge and might use any additional time to explore the temporal bone further. This behaviour would be natural and acceptable in a learning context but can be problematic in an assessment situation.
This study monitored the students' progress by final-product analysis at several time points. When studying the performance of complete novices in self-directed learning, a peak effect was found: almost half of the participants had a peak final-product score before the end of the allocated period (60 minutes) and then started making mistakes, thereby decreasing their score. This was exemplified by a participant who achieved a score of 20 out of the maximum 26 points at 40 minutes but finished with a score of 16 at the end of the session when the final-product assessment would usually be carried out. It is unknown whether the performance of remaining participants (designated ‘late peakers’) would have peaked or reached a plateau if they were given more time. Nevertheless, time and performance are dependent factors and this should be considered when using final-product analysis to assess mastoidectomy performance.
The performance of the early peakers demonstrates that novices can lack the knowledge of when to stop. In this study, participants were self-directed, with some having access only to the on-screen step-by-step tutorial whereas others also had additional volumetric green-lighting of the volume to be drilled in each step. In contrast, it has been established that feedback and directed goal setting are key elements in surgical technical skills acquisition.Reference Bech, Lönn, Falkenberg, Bartholdy, Räder and Schroeder19–Reference Kruglikova, Grantcharov, Drewes and Funch-Jensen22 A study into virtual reality thoracoscopy simulation training found that an educator-guided group performed significantly better than a self-directed group.Reference Bjurström, Konge, Lehnert, Krogh, Hansen and Petersen21 Similar results were reported for virtual reality colonoscopy simulation training.Reference Kruglikova, Grantcharov, Drewes and Funch-Jensen22 Consistent with these data, directed self-regulated learning is reported to facilitate long-term learning by providing an activated, mentally involved trainee with limited autonomous control within a structured setting.Reference Kruglikova, Grantcharov, Drewes and Funch-Jensen22–Reference Brydges, Nair, Ma, Shanks and Hatala24
A common limitation of technical skills assessment is a ceiling effect that prevents the assessment tool from distinguishing between novices and experts. Another ceiling effect that results in blocking further progress relates to the simulation design and degree of difficulty.Reference Bech, Lönn, Falkenberg, Bartholdy, Räder and Schroeder19, Reference Fann, Caffarelli, Georgette, Howard, Gaba and Youngblood25 The latter ceiling effect was found in this study: none of the participants achieved the maximum score, and only a few achieved a score above 20 points. Whether this is a true ceiling or could be overcome by repeated practice alone warrants further investigation. Simulator fidelity and design may also contribute to the ceiling effect. Nevertheless, both the relative low performance in this self-directed setting and previous reports of directed self-regulated learning suggest that additional instruction, guidance and feedback are needed to improve performance beyond this ceiling. Further analyses of the assessment scale items where novices fail could improve future instructions and thus simulator-integrated tutoring.
The addition of simulator-integrated tutoring with volumetric green-lighting to indicate volume to be removed at each procedural step significantly increased both the peak and final performances. The tutored group performed better than the non-tutored group, especially in the later, more difficult stages of the procedure. However, simulator-integrated tutoring can be a double-edged sword: on the one hand ‘perfect is the enemy of good’, meaning that the tutor could induce or force risky behaviour, for example an instruction to ‘expose more of the facial nerve’ could result in a reduced final-product performance. On the other hand, in a learning situation, this behaviour could facilitate skills acquisition because novices are encouraged to perform difficult tasks and can thus learn from their mistakes. Although simulator-integrated tutoring had a significant effect on performance it did not eliminate the peak or ceiling effects.
• Final-product analysis was investigated as a progress-monitoring tool
• Peak and ceiling effects are possible limitations of final-product analysis
• If adequately addressed, final-product analysis may be used as an advanced feedback tool
• Timing of assessment and clear instruction are important when assessing novice mastoidectomy performance
These findings are consistent with reports on the limitations of self-directed virtual reality simulation training of novices. For novices to acquire and improve surgical technical skills such as those required to perform mastoidectomy, simply getting a virtual reality simulator is not enough: repeated virtual practice, additional instruction and tutoring, and supplementation with other training modalities, are needed as well to reach the appropriate level of surgical competency. Final product based assessment can be used to monitor a trainee's progress and may have a further role if its limitations can be adequately addressed. Automated simulator-based assessment is a promising method for combining multiple assessments.Reference Wan, Wiet, Welling, Kerwin and Stredney13, Reference Kerwin, Wiet, Stredney and Shen26 Combined with other simulator-gathered metrics, continuous final-product analysis could be used to monitor real-time progress during virtual practice and provide on-going feedback to guide the novice to attain an optimal performance. Individualised feedback and simulator-integrated tutoring could enhance learning in self-directed training, but a clearer definition of goals and progress monitoring is needed. Limited surgical training resources could motivate these developments.
Conclusion
The timing of assessment and provision of adequate instruction is important when using final-product analysis to assess the mastoidectomy performance of novices. Peak and ceiling effects could limit the final-product score, and this should also be considered in conventional applications of final-product analysis. The current simulator-integrated tutor function did not eliminate these effects. However, future developments and improvements in on-going feedback and directed goal setting could potentially address these limitations.
Acknowledgements
We acknowledge the computer graphics engineer P T Mikkelsen, Computer Graphics Laboratory, Alexandra Institute, Aarhus, Denmark, for developing the Visible Ear Simulator, including the experimental version used in this study. We also thank A Kverneland, Department of Otorhinolarynogology – Head & Neck Surgery, Rigshospitalet, Copenhagen, Denmark, for providing statistical support.
