INTRODUCTION
Radiotherapy is increasingly a three-dimensional (3D) volumetric process. Advances in computed tomography (CT) scanning have for some time allowed generation of high-resolution volumetric patient data for localisation and simulation, and dose–volume histogram-based plan evaluation.Reference Chen 1 More recently the use of volumetric critical structure dose constraints or objectives,Reference Marks, Yorke and Jackson 2 and the move towards volumetric prescription 3 have ensured a 3D approach is increasingly adopted by treatment planners and clinicians. Technological advances such as volumetric arc therapy and cone-beam CT localisation have built on this new paradigm and increased the use of 3D volumetric approaches. Target and organ at risk outlining is the last of the radiotherapy procedures to be conducted largely in two dimensions (2D) with most centres maintaining a sequential axial slice-by-slice approach. When combined with the move to thin-slice CT this can be a slow and laborious task and this has understandably inspired much research into automation of this process.
Current research aiming to speed up the time-consuming structure segmentation process is focussed firmly on automatic boundary detection algorithms. Although these offer significant time saving within a range of different anatomical sites,Reference Anders, Stieler, Siebenlist, Schäfer, Lohr and Wenz 4 – Reference Simmat, Georg, Georg, Birkfellner, Goldner and Stock 6 they have limited value for target segmentation. Use of atlas-based models or ‘smart’ learning algorithms are proving to be valuable for rapid normal tissue segmentation but rarely fit to tumour volumes due to their inherent variability. Any auto-segmentation tool invariably requires substantial user input in terms of contour editing,Reference Voet, Dirkx, Teguh, Hoogeman, Levendag and Heijmen 7 although sites with clearly defined anatomical boundaries and limited patient variability present greater rates of fully automated success. A recent comprehensive review of auto-segmentationReference Sharp, Fritscher and Pekar 8 clearly identifies the valuable role of these tools while acknowledging that they provide a ‘starting point’ for the clinician to review and edit. This need to maintain input and decision making is also clearly desired by cliniciansReference Whitfield, Price, Price and Moore 9 who are understandably reluctant to place this crucial step in the hands of algorithms.
A parallel avenue of research is aiming to devise tools to enable radiotherapy clinicians to manually segment and edit volumes rapidly. These tools would facilitate both target volume segmentation and rapid editing of auto-segmentation volumes. One such approach aims to generate volumes from a small number of points. Previous workReference McBain, Moore and Green 10 has reported useful time saving of a tool that utilises a small number of points on axial images to generate a 3D volume rapidly. A more recently developed software tool 3D-RIOTReference Bridge, Fielding, Pullar and Rowntree 11 takes advantage of emerging visualisation and modelling techniques in order to facilitate 3D point placement on a range of orthogonal planes. In addition to the minimal point paradigm, the software also provides the user with simple intuitive 3D animation modelling tools to allow volumetric editing of structures across multiple slices concurrently. The process has been described previously.Reference Bridge, Fielding, Pullar and Rowntree 11 The combination of minimal point segmentation, use of multiple planes and volumetric multi-slice editing is designed to speed up outlining while retaining full manual control of the process.
The previous qualitative phase of evaluation for the 3D-RIOT toolReference Bridge, Fielding, Rowntree and Pullar 12 highlighted the potential value of the software as well as the steep training curve and difficulties adjusting to the new 3D minimal point paradigm. Specific feedback suggested that there are difficulties with experienced radiotherapy consultants adapting to the new paradigm and that the software tool should be subjected to further evaluation with a cohort of less time-pressured and less experienced outliners. The rationale underpinning this feedback was that it would reduce the impact of preconceptions regarding outlining methods and allow users to develop their own 3D outlining process. It was decided to test the software on radiotherapy students at the start of their planning learning. This cohort was selected as they would have minimal experience of axial outlining yet have some understanding of bladder anatomy and the purpose of outlining. Accordingly a simple volume (the male bladder) formed the focus for the testing to reduce the reliance on clinical experience. This paper presents the results of a bladder outlining study that aimed to compare outlining times and dice similarity coefficients (DSC) for the new paradigm with those of a conventional manual outlining tool.
METHODS AND MATERIALS
All students undertaking their first planning module in a pre-registration University radiotherapy course (n=52) were approached via e-mail and offered participation in a study and evaluation day. These students had previously attended a single short clinical placement block but as yet had not performed their own radiotherapy planning; they were unfamiliar with using treatment planning software. All students were provided with an overview of the project and guidance for general bladder outlining. After this they were randomised by self-selection of sealed envelopes into two subgroups; 2D (using ECLIPSE) and 3D (using 3D-RIOT). Each subgroup received software-specific training in regard to bladder outlining and the opportunity for practice with at-elbow support from a tutor using a male bladder patient dataset.
To account for their lack of clinical experience and remove any bias arising from poor clinical judgement, students were also provided with printed images illustrating the gold standard contours on the CT images for reference during training. The volume had previously been rated as ‘easy’ to contour by radiation oncologists although there was a sizeable prostate ‘indentation’ that required careful shaping.
Following training and when each student felt ready, each student outlined the same bladder dataset as during training while timing the process. Students were timed from the first point placement to confirmation of the outline by a tutor and paused the timer when they felt confident that their outline was correct. At that point the outline was checked by a tutor using a visual comparison with the gold standard and if acceptable the time was noted. If incorrect the student was advised about necessary corrections and the timer was restarted. The resulting outlines were saved for comparative analysis.
The previous studyReference Bridge, Fielding, Rowntree and Pullar 12 identified a steep learning curve for the software and it was deemed prudent to investigate the value and efficacy of training provided for this cohort. Students were asked to complete a survey comprising Likert-style questions (as seen in Table 1) about the training as well as their preferences for outlining tool.
User feedback concerning relative challenges of training and outlining processes
Abbreviations: 2D, conventional outlining (ECLIPSE); 3D, 3D-RIOT volumetric outlining tool.
All student contours were not only checked visually during data collection but were also subjected to volumetric comparison. This was performed within ECLIPSE by importing anonymised radiation therapy Structure Digital Imaging and Communication in Medicine files for each contour and combining all the contour sets with the original CT dataset and gold standard contour set. This then enabled calculation of DSCReference Zou, Warfield and Bharatha 13 for each outline using the intersection of the two volumes (overlap) and the sum of the volumes (combined):
$${\rm DSC}{\bf{{\equals}}} \left( {2{\times} \lsquo {\rm Overlap} \rsquo } \right){\bf{ \,/\,}} \lsquo {\rm Combined} \rsquo $$
Ethical approval for the project was provided by the University Research Ethics Committee. Independent observers were appointed to oversee the data collection in order to minimise the risk of coercion arising from potential power relationshipsReference Foucault and Gordon 14 between the researcher and students. Participation was voluntary and unrelated to academic work; students were provided with a sandwich lunch as a participation incentive.
RESULTS
A total of 11 students volunteered to participate, leading to completion of ten datasets. One user encountered difficulties with the 3D software during the timed outlining which led to them failing to create a satisfactory outline. Their data was accordingly excluded from the timing evaluation.
Students outlined faster using the 3D segmentation tool than the conventional manual outlining software with a mean completion time of 11·9 minutes compared with 19·2 minutes (p=0·03). Figure 1 illustrates the comparative outlining times for each of the ten users. It can be seen from the plot that there was more variability in the time taken for users to complete a satisfactory outline with the conventional outlining tool (standard deviation=6·75 minutes) than with the new 3D tool (standard deviation=3·16 minutes).
Comparative times for manual outlining per user with conventional (ECLIPSE) and new volumetric (3D-RIOT) segmentation tools.
Analysis of the survey data showed that the users expressed a preference for the new tool (eight users) over the conventional outlining software (two users) as seen in Table 1. This was interesting when compared with the challenges of training using the 3D tool. There was a slight difference in perception of efficacy of training between ECLIPSE and 3D-RIOT with all ten students feeling ‘well prepared’ for using ECLIPSE and six for ‘3D-RIOT’. Six students also reported that the 3D-RIOT tool took the longest to become familiar with.
Outlines generated by both methods compared well with the gold standard provided and were independent of outlining tool. Typical volumes generated with both tools are depicted in Figure 2. The mean DSC for both outline sets was identical with 0·965 for ECLIPSE and 0·964 for 3D-RIOT. Standard deviation of DSC was also similar at 0·009 (ECLIPSE) and 0·011 (3D-RIOT).
Typical bladder contour generated by conventional software (left) and 3D-RIOT (right).
DISCUSSION
The volumetric segmentation tool resulted in significant time saving for manual generation of bladder outlines that were identical to a gold standard. The new paradigm relied on minimal point mesh generation utilising multiple orthogonal planes followed by unique multi-slice editing based on a 3D animation tool.
Time saving
Although based on a small cohort of novice outliners, the statistically significant gains in time strongly suggest that minimal point mesh generation and volumetric editing are capable of speeding up manual segmentation of target volumes. Although this study was based on a relatively simple volume, the tool’s 3D editing abilities provide the flexibility required to sculpt more complicated structures with improved efficiency. The results suggest a mean time saving of just over 7 minutes which compares well with previous workReference McBain, Moore and Green 10 using an axial-only based minimal point system. This study gathered data from unbiased novice outliners who were inexperienced with the tool as opposed to data collection by the principal investigator; it is likely that additional experience would elicit further time saving.
Volumetric analysis
It was evident that both tools enabled the users to re-create the gold standard outline with no significant difference in DSC for ECLIPSE and 3D-RIOT. This finding also corroborates well with those of a previous studyReference McBain, Moore and Green 10 into a minimal point system. It should be stated that in order to overcome lack of clinical experience with segmentation, the users in this study were provided with a hard copy of the gold standard and had outlines checked by tutors before final submission. Furthermore, the volume selected was not particularly challenging; again to overcome the lack of clinical experience of the cohort. Thus it is unsurprising that both 2D and 3D contour sets had low inter-user variability and consistent high DSC scores.
Training
Although this cohort received intensive training with both software packages, it was clear that the 3D paradigm required more substantial training and practice than the conventional software. Point placement, in particular, requires great care in order to avoid creation of mesh ‘holes’ in the volume which then take time to correct. This typically arises when too many points are utilised too close together which can confuse the mesh growing algorithm; deletion of the errant point invariably amends the issue. Unfortunately for one user this issue escalated during the timed data collection and resulted in a failure to complete an outline, despite previous success during training. It is clear that additional training should be provided with regard to avoidance and management of these ‘holes’.
Minimal point multi-planar segmentation
Feedback from the previous studyReference Bridge, Fielding, Rowntree and Pullar 12 and existing evidenceReference Giezen, Kouwenhoven and Scholten 15 highlighted the value of using information from non-axial imaging planes for segmentation. The use of orthogonal planes for point placement enables clear visualisation of superior and inferior borders which present a ‘capping’ challengeReference McBain, Moore and Green 10 to axial-based software. Our previous study’s findingsReference Bridge, Fielding, Rowntree and Pullar 12 also highlighted the significant challenge that this presented to clinicians who valued the orthogonal plane data but struggled with orientation and 3D navigation. The use of a small number of key points to create an accurate volume also presented a challenge to outliners entrenched in axial outlining where more points correlate to increased accuracy. This group of novice outliners had no preconceptions with regard to segmentation procedures and readily adopted an efficient approach to volumetric outlining using a small range of well selected points. Each user adopted a different approach to the volume with some placing points mainly on axial slices and others using the full range of orthogonal images. Additional work to develop point placement protocols for different sites is ongoing.
Multi-slice volumetric editing
Feedback from the previous study and also observations during this phase highlighted the significant training requirement needed with regard to the 3D sculpting tools in order to maximise their effectiveness and efficiency. Users needed to grasp the concept that superior and inferior slices were being edited concurrently; while this speeds up the contour editing process it requires the user to select the most appropriate slice for editing with care. User feedback, however, suggested that they were able to gain proficiency with these tools with relative ease; this was in contrast to previous findings with experienced clinicians who struggled with the 3D nature of the tools. The new paradigm offers some useful visualisation and time-saving tools while facilitating full manual user control over boundary positions. The ability of the tools to edit multiple slices rapidly suggests future value in facilitating rapid editing of auto-generated or atlas-derived volumes, particularly those utilising large numbers of thin-slice CT images. Ongoing work is seeking to determine the potential impact the sculpting tools can have on these datasets.
Limitations
There was poorer recruitment than expected to the study with ˂20% of potential participants engaging. Although the results were statistically significant it would have been useful to have gathered data from a wider range of users. The use of novice outliners potentially presents a threat to external validity but this cohort was selected due to their lack of preconceptions with axial outlining in response to feedbackReference Bridge, Fielding, Rowntree and Pullar 12 from experienced radiation oncology consultants. This cohort also required the software to be tested with a simple volume which may not represent the ideal indication for use of the new paradigm. Ongoing work is testing the software with more challenging volumes including prostate, and head and neck. The software is still in development and lacks functionality in relation to use of magnetic resonance data and handling of multiple structures. Future work aims to build on these promising findings and further develop the tool into a more clinically viable solution.
CONCLUSIONS
A minimal point 3D volumetric manual outlining tool utilising orthogonal CT planes demonstrated significant time saving for bladder segmentation compared with axial-based outlining within a group of novice outliners. The study confirmed feedback from radiotherapy consultants who suggested the novel paradigm and 3D orientation challenges would be more readily adopted by novice outliners. Future work aims to establish the role of the 3D multi-slice sculpting tools in editing of auto-segmentation derived contour sets.
Acknowledgements
This project was supported by a scholarship from Queensland University of Technology. The authors are grateful to End to End Visuals for their invaluable input to the software development. The support of Radiation Oncology Mater Centre in Brisbane is also gratefully acknowledged.
