Introduction
Temporal bone surgery is complex because it requires navigation around critical structures such as dura of the middle and posterior fossa, sigmoid sinus, carotid artery, facial nerve and the labyrinthine.Reference Zagzoog and Yang1 The ‘gold standard’ of temporal bone surgery, as in any other surgical specialty, is cadaveric dissection.Reference Hu, Wattchow and de Fontgalland2 Cadaveric dissection provides a profound visual and tactile experience that mimics surgery on a living human body.Reference Hu, Wattchow and de Fontgalland2 However, the sole use of this method of surgical training has become increasingly difficult as a result of issues such as scarcity of cadavers, high costs, limited availability of equipped dissection laboratories and possibility of exposure to biological risks.Reference Iannella, Pace, Mucchino, Greco, DeVirgilio and Lechien3, Reference Frithioff, Sørensen and Andersen4 Furthermore, exposure to paediatric and rare pathologies, as well as a range of anatomical variation, is not always possible with cadaveric dissection.
Supplementary methods of surgical training have been actively investigated to ensure trainees have achieved a sufficient level of skills before handling scarce resources. Leading among these supplementary methods are computer-based simulation such as virtual reality and the use of additive manufacturing, or three-dimensional (3D) printed, specimens. While virtual reality (VR) offers a repeatable, low-cost experience, it is not able to sufficiently simulate the surgical experience of a real dissection. On the other hand, additive manufacturing specimens provide a more realistic dissection experience along with repeatability. However, the long-term cost of additive manufacturing surgical training is higher because the specimens are not reusable. Both forms of simulation offer the ability to create standardised surgical curricula with exposure to different anatomies, rare pathologies and paediatric specimens.
Various studies spanning over two decades have established the face and content validity of virtual reality simulation and its effectiveness in improving surgical skills in temporal bone surgery.Reference O’Leary, Tykocinski, Dahm, Pyman, Hutchins and Stevenson5–Reference Copson, Wijewickrema, Zhou, Piromchai, Briggs and Bailey11 However, as the use of additive manufacturing models is a more recent advance, not as many studies exist that investigate its effectiveness in this context.
Previous work has investigated the creation of additive manufacturing models that realistically mimic the appearance and tactility of a cadaveric temporal bone.Reference Iannella, Pace, Mucchino, Greco, DeVirgilio and Lechien3, Reference Da Cruz and Francis12–Reference Gadaleta, Huang, Rankin, Hsue, Sakkal and Bovenzi16 For example, Mowry et al.Reference Mowry, Jabbour, Rose, Wiet, Svrakic and Zopf17 and Frithioff et al.Reference Frithioff, Frendo, Pedersen, Sorensen and Wuyts Andersen18 conducted reviews of 3D printed temporal bone models, comparing their software applications, 3D printers and material. They found that stereolithography printers with powder and resin as the printing materials provided the best results. Validation of additive manufacturing temporal bone models was typically performed using face and content validity questionnaires.Reference Iannella, Pace, Mucchino, Greco, DeVirgilio and Lechien3, Reference Da Cruz and Francis12–Reference Gadaleta, Huang, Rankin, Hsue, Sakkal and Bovenzi16 These qualitative evaluations largely used the opinion ofsurgical registrars, with a few employing expert surgeons.Reference Frithioff, Frendo, Pedersen, Sorensen and Wuyts Andersen18 In a recent systematic review, Frithioff et al.Reference Frithioff, Frendo, Pedersen, Sorensen and Wuyts Andersen18 concluded that although most studies reported positive attitudes toward the models and their potential for training, the educational quality of such validations was low (i.e. Kirkpatrick level 1).Reference Cook and Reed19 Stronger evidence is therefore required as to the effectiveness of additive manufacturing in the training of temporal bone surgery.
The qualitative validation of additive manufacturing specimens cannot be dismissed out of hand, however, because it paves the way to investigations on their training efficacy and subsequent integration into temporal bone surgery curricula. Also, as additive manufacturing technology evolves rapidly, it is important that the additive manufacturing models are continuously validated as to their ability to faithfully reproduce the anatomical structures of the temporal bone and overall drilling experience.Reference Frithioff, Frendo, Pedersen, Sorensen and Wuyts Andersen18 As such, in this study, we investigated the face and content validity of new additive manufacturing temporal bone specimens developed and manufactured by Fusetec (Adelaide, South Australia). In contrast to previous work that only investigated healthy normal temporal bone specimens, we validated three different cases: healthy normal, healthy highly pneumatised and healthy sclerotic bones. To obtain more reliable feedback, and minimise individual bias, we employed three senior consulting otologists who are involved in surgical training for the validation.
Materials and methods
Additive manufacturing temporal bone models
The additive manufacturing temporal bone models obtained from Fusetec were advanced manufactured based on high-resolution axial computed tomography (CT) scans of cadaveric temporal bones. They were segmented on Materialise Mimics software, then converted to STL files in Netfabb for mesh cleaning and customising specific engineering features. Materials were selected under surgical guidance from previously document dog bone tensile tests. The bone and anatomical structures were produced using multiple materials using 0.0125-mm slices with a proprietary voxel-based software integration. All soft tissue and bony structures of the temporal bone were represented: inner and outer auditory canal, labyrinthine, tympanic membrane, ossicles, facial nerve, chorda tympani, sigmoid sinus, carotid artery and dura mater. To facilitate realistic haptic feedback on bone and soft tissue, the models were constructed with Shore hardness of 83–86 and 28–23. The three cases of temporal bone models used in the study were produced with different air cell composition to replicate normal, highly pneumatised and sclerotic bones. Figure 1 shows axial cross-sections of the three temporal bone models used.
Axial cross-sections through the cochlea of the three cases of additive manufacturing temporal bone: healthy normal bone (a), healthy highly pneumatised bone (b) and healthy sclerotic bone (c). The cutting plane is in pink, while the structures cochlea, facial nerve and sigmoid sinus are coloured in light blue, purple and dark blue, respectively. Note the difference in pneumatisation of the three specimens.
Study design
Ethics approval was obtained from the Human Research Ethics Committee of the Royal Victorian Eye and Ear Hospital (HREC number 24/1599HL). All participants provided written consent.
Three senior otologists who are involved in teaching temporal bone surgery were recruited. The surgeons’ experience in training varied: 6, 5 and 5 years for raters 1, 2 and 3, respectively. Each surgeon performed temporal bone dissections to evaluate the face and content validities of the three cases of the additive manufacturing temporal bone specimens. The dissections were conducted at the Royal Victorian Eye and Ear Hospital Temporal Bone Laboratory using standard operating theatre equipment (microscope, micro-drills and irrigation-suction systems) and all necessary personal protective equipment (gowns, gloves, masks and eye protection).
After performing each dissection, each surgeon completed a questionnaire assessing their experience with the specimen. The questionnaire developed by Da Cruz and FrancisReference Da Cruz and Francis12 was used for this purpose, selected by an expert otologist because of its ability to assess anatomical and/or drilling realism and elements addressing temporal bone dissection. This questionnaire consists of 23 questions in 4 categories (anatomical realism, usefulness as a training tool, task-based usefulness and overall reactions), based on a 5-point Likert scale (1 = strongly disagree to 5 = strongly agree). In addition, the surgeons were asked to provide any feedback and/or comments outside the questionnaire.
The questionnaire responses were tabulated in Microsoft Excel, and average scores and standard deviations for all specimens, as well as those for each specimen, were calculated. The intraclass correlation coefficient between the three raters was calculated using SPSS version 29 (IBM, Chicago, IL, USA) to test for inter-rater reliability. In addition, the questionnaire scores were compared with the scores of existing 3D-printed temporal bone specimens. To enable comparison across different validation scales, five categories of evaluation used in the literature for additive manufacturing validation were identified: anatomical realism, drilling realism, basic surgical skills, mastoid surgery skills and skill transfer to real patients. For each study, the items belonging to each category were identified from the scoring scale used and the average scores were calculated based on the number of items and the number of participants. The scores were then normalised so that each category represented a score out of 100.
Results
Overall results
Figure 2 illustrates the three drilled additive manufacturing specimens. Table 1 shows the responses for all three specimens from the three raters. The intraclass correlation coefficient of the overall specimens for the three raters was 0.745. The Bland–Altman plots in Figure 3 illustrate the agreement between each pair of raters. Bland–Altman plots, also known as difference plots, are a convenient way to assess the agreement between two sets of measurements.Reference Giavarina20 The y axis shows the difference between the two paired measurements and the x axis represents the average of these measures. Mean and 95 per cent confidence intervals of the differences are also plotted (upper and lower limits in Figure 3). An ideal agreement is zero difference.Reference Doğan21 Each bubble represents an instance of the difference in a pair of ratings against their average.
The three cases of additive manufacturing temporal bones after drilling: healthy normal bone (a), healthy highly pneumatised bone (b) and healthy sclerotic bone (c).
Bland–Altman plots of the overall specimen results of the three raters: (a) rater 1 and rater 2, (b) rater 1 and rater 3, and (c) rater 2 and rater 3. Numbers in the bubbles represent the number of repetitions of agreement between each pair of raters at that point (the number of instances a given difference in the ratings with respect to their mean occurred).
Face and content validity questionnaire responses of overall specimens from the three raters
3D = three-dimensional
Results for the three individual specimens
The score of the healthy normal bone specimen from the three raters (mean ± standard deviation) was 4.20 ± 0.77, while that of healthy highly pneumatised bone specimen was 4.33 ± 0.66. The healthy sclerotic bone specimen received a score of 4.23 ± 0.73. The intraclass correlation coefficients of the three raters for the healthy normal, highly pneumatised and sclerotic bone specimens were 0.820, 0.652 and 0.747, respectively. Figures 4, 5 and 6 illustrate the results of the face and content validity questionnaire from the three raters for the above specimens.
Face and content validity questionnaire responses for healthy normal bone specimens from the three raters: (a) anatomical realism rating, (b) usefulness as a training tool rating and (c) overall reaction rating.
Face and content validity questionnaire responses of healthy highly pneumatised bone specimens from the three raters: (a) anatomical realism rating, (b) usefulness as a training tool rating and (c) overall reaction rating.
Face and content validity questionnaire responses of healthy sclerotic bone specimens from the three raters: (a) anatomical realism rating, (b) usefulness as a training tool rating and (c) overall reaction rating.
Overall, the raters commented that the bone specimens had a very realistic drilling experience, including drilling pitch and residue. The raters’ opinion was that the colour of the specimens could be whiter to allow the structures to be pinker rather than yellow. The healthy normal bone specimen was identified as a good specimen for advanced cases because the cortical mastoidectomy was very easy, but the facial recess was very tight. Meanwhile, raters commented that the healthy sclerotic bone specimen was a good representation for difficult anatomy, specifically the tegmen and sigmoid sinus. This bone specimen also produced a higher drilling pitch compared with the other two specimens.
Comparisons with other existing three-dimensional temporal bone models
Table 2 compares the questionnaire scores of this study and others that used the same validation scale.Reference Da Cruz and Francis12 Table 3 shows a category-wise comparison of existing additive manufacturing temporal bone evaluations across different validation scales, normalised as discussed above.
Comparison of the face and content validity questionnaire scores of Da Cruz and Francis,Reference Da Cruz and Francis12 Chien et al.Reference Chien, da Cruz and Francis14 and this study, which used the same questionnaire
* Scores are out of 5. 3D = three-dimensional
Comparison of the face and validity scores between existing additive manufacturing temporal bone specimens that used various questionnaires*
* The scores are all averages and have been adjusted to be out of 100%. N/A = not available
Discussion
Cadaveric dissection remains the gold standard in surgical training, including in otology, despite issues such as disease transmission, maintenance cost and limited availability.Reference Iannella, Pace, Mucchino, Greco, DeVirgilio and Lechien3 Alternatives such as virtual reality simulation and additive manufacturing models have been actively investigated to address the drawbacks of the sole use of cadaveric dissection in training temporal bone surgery.
This study evaluated the face and content validity of three different cases of 3D printed temporal bone models manufactured by Fusetec using the questionnaire developed by Da Cruz and Francis.Reference Da Cruz and Francis12 We used senior ENT surgeons experienced in training registrars for the validation of the specimens rather than ENT trainees used in some previous studies. This ensured that the results were of a higher quality as their knowledge of training requirements and what is required of a training specimen is more reliable.
Overall performance
These models received highly positive responses from the three raters, with an average of 4.26 ± 0.72 (out of 5), indicating that experts were of the opinion that they would offer a similar surgical training experience as cadaveric dissection. These models rated highest for their usefulness for teaching cortical mastoidectomy (4.89 out of 5). On the other hand, item colour contrast was rated the lowest (3.22 out of 5). The raters suggested the bone should be whiter to allow the structures to be pinker rather than yellow. Raters also highlighted that the models produced very realistic residue and drilling pitch.
Case-wise performance
There were three different cases used in this study: normal, pneumatised and sclerotic temporal bone specimens. None had pathologies. Each questionnaire item of each specimen received high responses from the three raters (3 and above), with scores of 4.20 ± 0.72, 4.33 ± 0.66 and 4.23 ± 0.73 for the three specimens respectively. In addition, the agreement levels of the three raters for each specimen were also high, Raters commented that the high drilling pitch on the sclerotic temporal bone specimen was quite realistic.
Inter-rater reliability
The agreement level among the three raters was high (overall intraclass correlation coefficient = 0.745), with the lowest agreement being for the highly pneumatised specimen (intraclass correlation coefficient = 0.652) and the normal and sclerotic specimens receiving intraclass correlation coefficient scores of 0.820 and 0.747, respectively. Additionally, according to the Bland–Altman plots, rater 1 was observed to be consistently most lenient and rater 3 was the strictest in their rating. As the three raters were of similar experience levels, this difference in scoring is likely due to individual differences and standards.
Comparison with existing literature
When comparing the face and content validity results of Da Cruz and Francis,Reference Da Cruz and Francis12 Chien et al.Reference Chien, da Cruz and Francis14 and this study (Table 2), which utilised the same questionnaire, this study achieved the highest average score, followed closely by Da Cruz and FrancisReference Da Cruz and Francis12 and, lastly, Chien et al.Reference Chien, da Cruz and Francis14 From Table 3, which compares the face and content validity of existing additive manufacturing temporal bone specimens using various questionnaires, it can be seen that the scores for anatomical realism decreased from 2015 to 2024, while those of the other categories either increased or remained the same. This seems to contradict the fact that with the advancement of additive manufacturing technology, better additive manufacturing models with higher resolution are being produced. This decrease could be because of higher expectations on the part of the participants because they have been increasingly exposed to improved additive manufacturing specimens through the years.
Comparison of recent additive manufacturing models and printing technologies
The recent studies conducted by Mowryet al.Reference Mowry, Jabbour, Rose, Wiet, Svrakic and Zopf17 and Iannella et al.Reference Iannella, Pace, Mucchino, Greco, DeVirgilio and Lechien3 are the most comparable to the current study because they used more advanced technology as well as the more reliable assessments of experts. According to Mowry et al.,Reference Mowry, Jabbour, Rose, Wiet, Svrakic and Zopf17 the highest scoring models were produced with FormLabs Form 2 and Zcorp 650 stereolithography printers, which offer a range of resin material but are limited to printing one material at a time. Likewise, Iannella et al.Reference Iannella, Pace, Mucchino, Greco, DeVirgilio and Lechien3 used a Photon mono × 4k stereolithography printer to create their temporal bone models. Additionally, these machines are designed for engineering hard plastic prototypes, and some require extensive post-printing treatments. On the other hand, Fusetec additive manufacturing technology allows printing of multiple materials and adjustment of Shore hardness to 70A. Different materials can be selected and allocated in almost any proportion digitally prior to printing. As such, a large range of Shore values, texture, colour and density can be produced that simulate the characteristics of different tissue types. Moreover, post-printing treatment is only required to wash out support material. As such, Fusetec additive manufacturing models offer a high level of flexibility in defining colours, textures and haptic properties.
Limitations and future work
The comparisons in Table 3 have been divided into categories and normalised so that the scores of different studies that used different validation questionnaires could be compared. This, in addition to the differences in rater experience, as well as the rater numbers, adds bias to the comparison results.
As observed in Frithioff et al.,Reference Frithioff, Frendo, Pedersen, Sorensen and Wuyts Andersen18 qualitative studies, such as the one discussed here, are not sufficient by themselves to validate the educational quality of additive manufacturing models. Nevertheless, they are still important as they pave the way for higher quality validations. Indeed, the next step is to investigate the effectiveness of these additive manufacturing models in training temporal bone surgery skills. Once this has been established, a simulation-based curriculum that incorporates virtual reality and additive manufacturing training should be designed and validated to take advantage of the benefits offered by both technologies.
• The sole use of cadavers in temporal bone surgery training is impractical because of resources scarcity and high costs
• The proposed additive manufacturing temporal bone specimens were able to faithfully reproduce a training experience similar to that on cadaveric temporal bone
• Further studies regarding the effectiveness of these specimens in training surgical skills are needed
Conclusion
This study successfully established the face and content validity of three different additive manufacturing temporal bone specimens manufactured by Fusetec. Further studies regarding the effectiveness of these models in improving trainees’ temporal bone surgical skills are needed so that these specimens can be integrated into surgical curricula.
Competing interests
None declared.
Acknowledgements
The authors thank Fusetec (Adelaide, South Australia) for providing the additive manufacturing temporal bone specimens for this study.
