INTRODUCTION
Successful radiotherapy depends on the accurate delivery of radiation to target anatomical structures with millimetre precision. The spatial relationship between different organs as well as the geometric positioning of the equipment, the patient and the radiation dose distribution requires therapeutic radiographers to not only use the equipment competently but also have a sound understanding of three-dimensional (3D) geometry. Training in radiotherapy combines academic teaching with placement in clinical departments; these are typically pressured environments with high patient throughput and a low tolerance for error. As with medical education in many other professions,Reference Lammers, Byrwa, Fales and Hale 1 – Reference Wang, Liu and Wang 7 a range of simulation resources have been enthusiastically adopted to facilitate training away from the clinical area. The Virtual Environment for Radiotherapy Training (VERT) is a high-fidelity simulation hardware and software resource that places the user in a 3D immersive virtual radiotherapy bunker using clinical anonymous patient data. VERT was introduced in 2007Reference Bridge, Appleyard, Ward, Philips and Beavis 8 , Reference Phillips, Ward and Page 9 and is currently installed in academicReference Kirby, Pennington and Al-Samarraie 10 and clinical departmentsReference James and Dumbleton 11 across the world. VERT provides opportunities for pre-clinical skills acquisition,Reference Boejen and Grau 12 technique learning,Reference Nisbet and Matthews 13 , Reference Green and Appleyard 14 physics and dosimetry teaching,Reference Beavis and Ward 15 – Reference Kirby 17 plan evaluationReference Bridge, Crowe, Gibson, Ellemor, Hargrave and Carmichael 18 and visualisation of anatomy. VERT has also been used to help prepare experienced clinical staff with implementation of new modalities.Reference Boejen, Vestergaard and Hoffmann 19 The resource provides a useful array of tools to help close the gap between theory and practice. In particular, the use of clinical control systems within the platform allows students to gain confidence with clinical techniques without interfering with the daily workflow. Students are able to learn from mistakes without harming patients and enter the clinical workplace with some technical skills, allowing them to focus more on patient care and communication than the controls of the equipment. VERT, along with a dedicated patient information module known as Patient Education and Radiotherapy Learning (PEARL), has also found value for patient educationReference Sulé-Suso, Finney and Bisson 20 – Reference Hansen, Nielsen, Boejen and Vestergaard 23 where it can help to prepare patients for what is an unfamiliar and potentially daunting environment.
Recent presentations and discussion at VERT users group and research meetings have indicated that there is a wide variety of resource development and implementation internationally. New users, however, are commonly finding that initial resource development is time consuming and that opportunities for collaboration are restricted. The evidence base underpinning VERT pedagogy is still small and suffers from low cohort numbersReference Bridge, Appleyard, Ward, Philips and Beavis 8 , Reference Green and Appleyard 14 , Reference Kirby 17 and disparate outcome measures. A more collaborative approach to resource development and underpinning evaluation may enable VERT to be used more productively and lead to an evidence-based approach to ongoing development.
The aim of this study was to collect data and user feedback in order to establish the current and future role of VERT and explore the potential for collaborative resource development and research. This data will help to define best practice guidelines as well as inform development of the VERT user group’s strategy regarding curriculum and resource development and research.
METHODS AND MATERIALS
Data collection
No appropriate pre-existing data collection tool was available so a short, 18-item, questionnaire was developed in order to answer the specific research questions developed during discussion within a multi-national research team. The survey tool comprised a mixture of multiple option and short-answer open questions arranged into three sections that gathered data on the software and hardware used in the VERT facility, current usage of VERT and suggestions for future practice and resource development. The survey was piloted by an independent researcher in order to check reliability and validity. Ethical approval for the project was granted by the University of Liverpool Research Ethics Committee. A link to the survey was distributed to all 52 VERT users worldwide, as identified using Vertual Ltd customer database, in February 2016. Users were asked to identify a single person at their institute who was not on the research team to complete and submit the date. A reminder to complete the survey was distributed 1 month later. Additional input from Vertual Ltd on the design and content of the survey was not utilised in order to minimise potential bias. The survey was deployed by the University of South Australia via SurveyMonkeyTM to enable anonymous online completion.
Analysis
Quantitative data were collated and analysed using the descriptive tools within SurveyMonkeyTM. Responses to the open short-answer questions were subjected to qualitative thematic analysis using a conceptual approach. This identified key themes that were sorted under the following categories: VERT use, pedagogical practice, resource development, collaborative approaches/research and barriers/challenges. Validation of identified themes was conducted by additional members of the research team and qualitative data were triangulated against the quantitative findings.
RESULTS
Demographics and VERT use
There was a 90% response rate (47 out of 52) and compliance was generally high, although some questions (such as use of specific physics and patient information ʻPEARLʼ modules) were omitted, limiting the value of the data. Responses to 11 of the additional short-answer ʻclarificationʼ questions provided a total of 105 comments for analysis. Most users were using the latest version of the software with only two having failed to upgrade in the last 2 years. Stereoscopic function was used by 42 users (69%), although only 20 (43%) used the position tracking function. VERT was most commonly used with pre-registration and prospective students as seen in Figure 1 but was also used with post-registration students, students on other courses and visitors. When asked to estimate total annual hours of use, 37 participants (78·5%) reported using 1 day/week or less for a typical 30-week academic year. A rough total estimate across the participants was 8,000 hours annually worldwide.
Frequency of use with different groups. Abbreviation: VERT, Virtual Environment for Radiotherapy Training.
Barriers to VERT use
Participants were asked to identify barriers to use of VERT; the most common of these were related to the availability and location of the facility as well as technical support. Overall 15 responses (32%) indicated that there was a perception in their institution that VERT was not useful. It was not clear from where this perception had arisen, although one respondent did comment that lack of use had resulted in low staff confidence when using it. Another suggested that VERT was being seen as the answer to everything which was problematic and students would be no worse off without it, they would just be taught differently.
Pedagogical practice
Most VERT activities (93%) were conducted in small group seminars or with students working individually or in pairs. In centres indicating the use of more than one licence, these were utilised on laptops or desktop personal computers primarily for teaching preparation using the virtual presenter function, educational and promotional events and to support the delivery of VERT-based teaching in clinical sites without permanent VERT facilities using a mix of 2D and 3D modalities. A wide range of activities were supported well by VERT as seen in Table 1. It was clear that most participants used VERT to help students to either gain understanding of concepts and techniques or to gain practice with techniques and practical skills. The system was also commonly used to teach understanding of anatomy. Less commonly used but popular activities included: plan evaluation, learning and practice of physics and quality assurance, as well as supporting provision of multi-disciplinary teaching and staff continuing professional development (CPD). In addition, benefit was perceived by users in relation to student assessment, research, enhancing retention and as an additional placement as seen in Table 2. This table highlights the extent to which users thought each potential aspect of VERT was useful, whether this has been implemented locally (as seen in Table 1) or not.
Activities supported by Virtual Environment for Radiotherapy Training
Abbreviations: RT, radiotherapy training; QA, quality assurance; CPD, continuing professional development.
Perceived value of Virtual Environment for Radiotherapy Training (VERT)
Abbreviations: QA, quality assurance; CPD, continuing professional development.
Future collaborative resource development
Participants were asked about the level of support for collaborative resource development and sharing at institutional level and it was encouraging to see 30 users (64%) in favour of this. Concerns relating to intellectual property rights were highlighted by six users (13%) but overall there seemed to be support for taking this forward. One of the resource areas identified as having potential for collaboration was the development of additional patient datasets with a variety of body sizes, supported by additional workbooks for both students and staff CPD activities. This may be beneficial for those users without access to a treatment planning software or permission to use clinical datasets. In addition, there was a substantial recurring theme in free text responses of users being open to collaboration or an online community of practice. Table 3 summarises these responses.
Comments relating to collaborative working
Future collaborative research
One of the key themes for collaborative research identified the need to determine the impact that VERT has on learning and assessment using both formative and summative methods. There was also a perceived need to support this with both quantitative and qualitative evidence. Other respondents expressed a need to determine VERT’s role in peer-assisted learning, the impact for final year radiotherapy and other healthcare students and, in patient education, the role in supporting patient decision making. Its value in supporting the development of 3D understanding of geometry was also identified as an area for further study together with its possible contribution to reducing the amount of time required for clinical placement. Those who identified themselves as new users reported that they would be interested in collaboration and would welcome support from other more experienced users and researchers in getting started:
I am going to start using it for patient education soon and will wish to evaluate it so would be interested in collaborating with others using it this way
We are planning to use VERT for research; maybe it’s good to have any idea how to start?
Discussion
The aim of this study was to establish the current and future role of VERT and explore the potential for collaborative resource development and research. The findings have revealed a difference between current use and perceived value which may be linked to challenges associated with implementing VERT resources. The barriers facing users have been identified from this study and users have provided feedback relating to the potential value of collaboration to help with this. The following discussion expands on the findings and explores the possible role of collaboration with resource development, deployment and evaluation.
Implications for VERT pedagogy
The difference between current use of VERT and the perceived value for different activities suggests that most users have a development need and would value assistance with implementing some VERT activities. Given the strong support for collaborative development and research it may be sensible to consider a consistent approach to VERT pedagogy that provides VERT users with a basis on which to develop and share approaches that reflect best practice and work in local contexts. Nisbet and MatthewsReference Nisbet and Matthews 13 highlighted the value of a generic workbook with clear alignment with Bloom’s taxonomy of learning for supporting VERT seminars and practical sessions. This survey has shown that radiotherapy concepts and techniques are commonly taught using VERT and a collaborative approach to developing and sharing resources may not only help support new users but also extend the potential datasets available for comparative evaluation of the impact of the VERT resources.
Barriers facing VERT users
The barriers and challenges encountered by users fell into four main categories. The first was the issue concerning the availability of staff time although it was not clear whether this related to the development of resources or the time available to actually engage with the platform and the ability to engage new staff. The second related to the use of VERT in its 3D mode with comments relating to students finding the 3D glasses cumbersome and therefore being reluctant to use them. There was also a perception that aspects of the software were out of date especially in relation to image guided radiotherapy functions. However, no further explanation was offered and therefore it is not clear if this comment relates to older versions of the software. The final theme related to the financial constraints that some users encounter and how this impacts on their ability to introduce new tools particularly in the field of patient information.
Although this question has illustrated a range of challenges facing users with VERT implementation, more work is needed to identify which of these issues can be addressed by the manufacturers and which are related to logistical and financial problems. In the latter cases it may be the case that collaborative resource development and sharing may decrease the impact of time and financial constraints and help support wider use of simulation.
Collaborative resource development and sharing
A funded project in 2011 established a collaborative model across a consortium of six Australian Universities in 2011. This development was based on a Community of Practice, defined as ʻa group of individuals who share their interests with a specific topic, gaining greater knowledge of and expertise through interacting regularlyʼ.Reference Wenger, McDermott and Snyder 24 Members of the resulting VERT Academic Community of Practice successfully collaborated to develop and share VERT teaching resources that were then embedded in programs nationally to improve the effectiveness of existing practices.Reference Probst and Borzillo 25 This 2.5-year project saw the creation of multiple pedagogically based clinically relevant training resources accessed by all universities through shared cloud-based technology. Evaluation of the impact of this initiative is as yet unpublished but the initiative has certainly contributed to the findings of this audit by expanding the extent of user engagement in Australia.
The high response rate of this survey suggests that VERT users are a particularly engaged academic community internationally. This is also demonstrated by the success of the ongoing VERT Users Group meetings. Based on this, the enthusiastic support for collaborative resource sharing in the VERT community demonstrated through this survey and the successful introduction of a Community of Practice in Australia, it is suggested that the establishment of a wider Academic Community of Practice be considered. It would, however, be interesting to see whether a model similar to that adopted in Australia can be adopted internationally in the absence of project ʻseed fundingʼ. In terms of wider health education, this would seem to be a useful model for increasing use of health simulation resources and further collaboration with other disciplines should prove to be valuable.
Future collaborative research
It was interesting to see enthusiasm for developing additional evidence relating to current and future VERT use. In particular, the data revealed a need for ongoing large-scale and collaborative projects to measure the specific pedagogical impact of VERT. It was clear that users wished for data to help direct their use of VERT as not only a learning and teaching tool but also within an assessment role. These new research questions urgently need addressing via both quantitative and qualitative methods to capture the impact of VERT on both academic outcomes and the student experience. It is hoped that future collaboration is aimed at designing and implementing high-quality research across a range of users to ensure high levels of validity.
Limitations of study
The study had a high response rate but it was not clear from the data which individuals had completed the survey tool; ideally this would have been the institutional VERT lead. A limitation of this study was that respondents were asked to make rough estimations of usage. Although this is less accurate than a prospective timing measurement it should provide a reasonable approximation. There is also likely to be a slight respondent bias as individuals in a VERT role are likely to perceive VERT more favourably. Despite this, it was clear that some negative comments had been included in the feedback, which suggests a more balanced approach was adopted. The support of the VERT manufacturers in facilitating contact with users may have influenced response and potentially introduced recruitment bias; however, the high response rate suggests that this impact is minimal. Participants were advised that manufacturer input did not extend to survey tool construction, deployment or data analysis phases; this combined with the anonymity of response should have further ameliorated introduction of manufacturer bias.
A further source of potential bias arises from the relative distribution of users internationally with over-representation from some countries. This in turn may have led to biased data arising from different learning programmes and cultural influences. The anonymous nature of the survey, coupled with some countries having low number of users means that comparison of trends within different countries was impossible to determine.
Future development
Several respondents offered suggestions for new tools related to problem solving activities, for example, how changes in various parameters including focus to skin distance and gantry angle effect dose distribution, improved image matching functions and the development of the virtual presenter facility to include the addition of text. From an anatomical perspective, the development of an anatomical atlas to include lymph nodes was also suggested. In addition to these themes, the value of combining VERT with other technology enhanced learning platforms was also identified as a research need. Ongoing collaborative evaluation from users will continue to provide suggestions for ongoing development and ensure that software development matches user requirements. This survey has demonstrated the support for collaboration between users and involvement of a wider user community of practice with simulation software manufacturers may ensure that development continues to match user expectations.
CONCLUSIONS
The results of this audit indicate that VERT is used for around 8,000 hours annually (less than 1 day/week in most centres) mainly to support small group teaching of radiotherapy concepts and provide practice of techniques. Advanced technical features are not used fully with many users and there are resource issues impacting on users’ ability to use the software to its full capability. Despite this, the simulation package is perceived as valuable for radiotherapy education across a wide range of topics and learning activities. It is possible that many of the implementation barriers facing VERT users could be addressed via a collaborative approach to resource development and sharing; it is clear from the findings of this audit that there is widespread support for such an initiative.
This audit has also demonstrated enthusiasm for ongoing collaborative research among the radiotherapy simulation community. This is vital if implementation of VERT-based resources is to be supported by evidence-based findings. It is hoped that a collaborative approach will help to address the current limitations of the evidence base arising from small-scale local projects based on student satisfaction and opinion data. Adoption of an international Academic Community of Practice for collaborative simulation resource development, deployment and evaluation may be of value and is worthy of further discussion and consideration.
Acknowledgement
The authors wish to extend their thanks to Vertual Ltd for their assistance with approaching potential participants for recruitment.
