Introduction
Background
Disaster management is a core component of patient and public safety and has been receiving increasing priority as each new natural (eg, Hurricane Katrina) or manmade event (eg, the terrorist attacks on September 11, 2001) highlights the impact of large numbers of ill or injured people on the limited capabilities of the health care system.Reference Perry and Lindell1 The recent US National Strategy for Public Health and Medical Preparedness further emphasizes the need for an enhancement of national emergency preparedness by encouraging coordination and leadership among federal departments to effectively provide emergency management education and training.2 This also has been incorporated into health accreditation agencies, such as The Joint Commission, through new emergency management requirements for health care systems to perform emergency response exercises at least twice each year.3 As a result, there is an impetus to develop more emergency preparedness training programs for healthcare professionals.
While several US agencies (eg, Federal Emergency Management Agency (FEMA), Health and Human Services, National Fire Academy, and the Centers for Disease Control and Prevention (CDC)) have developed emergency preparedness education programs to meet these needs; these programs typically utilize first-generation Internet technologies to provide single-student, one-way dissemination of knowledge through online courses or scripted lectures.4-7 However, new, online technologies can support large collaborative and interactive networks of students who can gather in virtual teams and individually become active participants who construct, rather than reproduce, their knowledge through interactions with peers, instructors, and the immersive environment.
A real time, three-dimensional (3D) online world called Second Life is available that allows users to navigate and interact with a multi-user, virtual environment (MUVE) using an “avatar,” which is a computerized character that represents the participants in the virtual world.8 Each individual controls his/her avatar via a computer keyboard and mouse. Avatars are able to freely navigate (walk, run, and fly) in the 3D environment and are capable of complex interactions (eg, touch, pick, pull, or push) with objects in the virtual world. Participants can use the avatar to communicate with other participants’ avatars via 3D spatial audio and direct text chat. Users also can communicate nonverbally via simple gestures (eg, waving, clapping, and nodding). Within these MUVEs, lighting can be adjusted to represent different times of day. Weather (eg, rain, snow) can be simulated or based on local Really Simple Syndication (RSS) feeds, and a full range of special effects (eg, smoke, fire) also can be simulated. Since the online environment is run on a server cloud, it enables multiple users to simultaneously log onto a shared online virtual space, irrespective of geographical location.
Role playing exercises have been used for teaching and studying collaborative Command and Control for emergency management,Reference Woltjer, Trnka, Lundberg and Johansson9, Reference Trnka and Jenvald10 and tactical decision-making games have been used for training in non-technical skills (such as critical decision making) for incident management,Reference Crichton, Flin and Rattray11 but these typically have taken the form of tabletop exercises using written descriptions of the scenarios with interjections and group discussions of responses by the players.
In health care, in particular, simulation-based training with high-fidelity medical simulations has been developed to teach and study team collaboration and decision making in emergencies.Reference Weaver, Salas and Lyons12, Reference Issenberg, McGaghie and Petrusa13 However, these simulations tend to be patient centered, with a focus on team management of a cardiac arrest mannequin, or practice in health care provider-patient interactions, diagnosis, and management,Reference Flanagan, Nestel and Joseph14, Reference Cook, Erwin and Triola15 rather than process-centered simulations with a focus on providing the experience of working together and making decisions in a disaster situation. This experience is further enhanced by the playback abilities of the MUVE, which allows for reflective practiceReference Schon16 so that students can critically reflect and evaluate their own experiences.
Unlike previous Internet-based health professions education,Reference Cook, Levinson and Garside17 a MUVE allows for multiple participants to experience the same dynamic situation. It is suited particularly to experiential learning that requires an environmental context (eg, the ability to rapidly alter and reassess a dynamic situation), allowing for practiced critical decision making and rapid problem solving in ambiguous and stressful situations. Learning is enhanced by the interaction of participants who have varying degrees of educational and life experiences. Consequently, the use of MUVEs as educational tools in disaster preparedness training is an area of growing interest.Reference Ramloll and Beedasy18, Reference Chen, Li and Kotz19 Preliminary findings from similar projects are positive and provide some evidence to support the role of simulation-based training in enhancing collaborative team work, allowing for rehearsals to reflect and improve performance and an immersive ability to reproduce real-life emotions.Reference Shilling, Zyda and Wardynski20-Reference Huang and Alessi23
The primary goal of this study was to pilot a team-based disaster management curriculum in a MUVE. The educational goals were to develop and evaluate a novel virtual learning curriculum for: (1) training new users to function in a MUVE; (2) providing a basic Healthcare Incident Command System (HICS),24 Emergency Severity Index (ESI),Reference Tanabe, Travers and Gilboy25 and Simple Triage and Rapid Treatment (START)Reference Super, Groth and Cleary26 triage education in a MUVE; (3) performing two functional emergency preparedness drills in a MUVE; and (4) providing students the opportunity to watch and critique their performances through edited “machinimas” (videos made of the virtual world) played back after the functional drills.
Methods
Study Design
This was a descriptive feasibility study with pilot qualitative and survey data designed to provide proof of concept for an innovative emergency preparedness curriculum delivered in a MUVE. As this was a pilot project, the study evaluation framework included both pedagogic and technologic performance measures. Observations made by the study team and students were collected using semi-structured debriefing. Student program evaluations, as well as self-efficacy assessments specific to the targeted audience, were completed during the final lesson. Local Institutional Review Boards approved the study, and all students signed informed consent forms prior to participation.
Setting and Selection of Participants
Study participants were emergency department nurses and Command Post (CP) personnel (hospital administrators) recruited from the Veterans Affairs San Diego Healthcare System (VASDHS) Disaster Management Committee, and the Emergency Department's Emergency Preparedness Committee. For the first course, seven students, all of whom were full time registered nurses, consented to participate. These nurses function as charge and triage nurses, and are trained in the Emergency Severity Index (ESI) five-level triage system.Reference Tanabe, Travers and Gilboy25 For the second course, seven students (all full-time staff who would be expected to assume HICS CP positions during an emergency) consented to participate. In all situations, students were assigned avatars based on their expected, real-life roles during a disaster.
MUVE Educational Technology
The online MUVE used for this project, Play2Train, was developed by a collaborative team of engineers and programmers from Idaho Bioterrorism Awareness and Preparedness Program (IBAPP) at Idaho State University, and was enhanced and prepared for this course in collaboration with the authors at the University of California, San Diego, California USA, and at the VASDHS, San Diego, California USA.
The Play2Train environment, a virtual space accessed through the free online environment, Second Life (Version 1.23.4 with the Second Life grid viewer, Linden Research Inc., San Francisco, California USA) consisted of three private interconnected islands or “sims” (short for simulations) that comprised a scale representation of a small community-based hospital and a surrounding neighborhood.
The Play2Train virtual hospital utilized in this course consisted of an emergency department with a patient waiting area, virtual treatment rooms, common staff areas, and a large field and parking lot to allow for temporary tents (both triage and treatment) to be set up. In addition, there were virtual conference and meeting rooms with virtual video screens used to show multimedia content (video and slides) in the virtual environment. Students were provided with individual 17-inch gaming laptops with dedicated video processors. Voice communication was enabled by use of biaural microphone headsets. Internet access was provided using University of California, San Diego's gigabit Ethernet local area network.
Avatars were developed to match participants’ roles during the exercises, and appropriate uniforms and assigned HICS role vests were used. In order to simulate a patient surge, “bots” (short for robots) were developed and programmed to play the role of 30 unique patients. In this instance, the bots deployed were a type of weak artificial intelligence robot that stored a prerecorded audio (one to two sentences of symptoms or complaints), vital signs (though a pop-up text), and the ability to don a respiratory mask. Each bot's audio recording, pop-up vital signs, and mask could be activated by a student avatar by touching a clearly marked tag on the bot's torso or face. In addition to these bots, eight project team members logged in as patient avatars and were able to perform more complex role play (eg, a patient with evolving chest pain requiring emergency care). All patients (bots and avatars) wore interactive ESI and START Triage Tags. Based on ESI and START protocol, students were expected to triage and tag patients, and transport them to appropriate treatment areas either in the hospital parking lot or provided treatment areas within the virtual hospital. The students were expected to plan the allocation of treatment areas to patients based on ESI and START triage categories, as well as in accordance with appropriate infection control practices (eg, respiratory isolation for patients with influenza-like illness). Command Post personnel were provided with evolving resource count boards and communication information for dissemination as per the principles of HICS.
Curriculum
The multidisciplinary facilitator team comprised emergency preparedness, emergency medicine, public health subject matter experts, medical education experts (including a medical anthropologist for qualitative evaluation), and information technology engineers and programmers. The emergency department course was administered in August-September 2009 with six, 90-minute sessions in the computer laboratory. The CP course was administered over three, three-hour sessions (to accommodate student work schedules). Both groups started with the same scenario (Figure 1) and had similar learning objectives tailored to job class. Both groups were expected to demonstrate their expected emergency roles in response to a pandemic influenza; the emergency department nurses were expected to demonstrate their ability to appropriately utilize the triage methods available to them, and the CP personnel were expected to appropriately manage hospital resources using HICS principles.
Course Details and Starting Scenario for Patient Surge Training in a Virtual World
Overall lesson progression was focused on reiterating and reinforcing knowledge and skills learned in a previous lesson, before introducing a new concept (Table 1). Thus, students learned to function in the MUVE first (walk, talk, and interact with objects), and then, learned about HICS, ESI, and START triage within the virtual environment. This culminated with two consecutive functional team drills in which students had to demonstrate their new skills and use critical reasoning to work together and manage a large patient surge. The final lesson was used for reflective learning, self-evaluation, course evaluation, and a brief lecture about recognizing and dealing with personal and co-worker stress during a disaster. Students also received paper handouts at each lesson to be incorporated into a course notebook for reference as needed.
Lesson Plan and Evaluation Methods for Patient Surge Training in a Virtual World
The first lessons followed a blended-learning format in which students met in a real-life computer classroom, and had a brief introduction to the expectations and skills of the lesson (10 to 15 minutes) before logging into the MUVE for the actual learning and experience. The virtual-learning experiences varied from virtual didactic lessons with an instructor utilizing slides or a movie, to hands-on experiences and tours with a facilitator (all of which were real world). At the completion of the lesson, student avatars gathered in the MUVE for a quick “hot wash” (ie, rapid debriefing) as part of the drill. At the completion of the hot wash, students logged off from the MUVE and conducted a semi-structured debriefing back in the physical classroom (10-15 minutes). Facilitators also debriefed separately (within the MUVE) after each lesson. The final lesson was held in a small physical classroom with a round table to encourage discussion (rather than the forward-facing computer laboratory). During this lesson, select video clips from the machinima of the simulated exercise were reviewed to stimulate recall and generate discussion. A collage of photos taken during the drill is shown in (online only).
Methods of Measurements and Analysis
Participant feedback was elicited through a series of open-ended questions designed for each exercise to encourage participants to speak freely about technical challenges they encountered, comment on course content, or simply express their reactions to their immersion in a virtual environment. Videotapes of all of the real-life focus groups were transcribed and reviewed using an iterative approach to pursue significant themes or dominant issues that emerged from the interview data.
Final written course evaluation scores (12 questions) completed by each participant were based on a five-point Likert scale (1 = Strongly Disagree, 5 = Strongly Agree) and were reported as class means; standard deviations were not included as the results were not normally distributed. Students’ self-reported changes in knowledge pre- and post-course (eight questions for the emergency department course, and six questions for the CP course) also utilized a five-point Likert scale (1 = Strongly Disagree, 5 = Strongly Agree). Individual differences were averaged over all students for each question pre- and post-intervention. The class knowledge shift (mean delta) was averaged for each question as well as overall for each course (emergency department Triage and CP).
Results
Fourteen students completed the course, the final written evaluations and the focus group; seven students completed the emergency department triage course, and seven students completed the CP course. None of the students had prior MUVE experience. All students reported positive experiences in written course evaluations (Table 2) as well as in real-life debriefing sessions. Although the numbers were small, all students had improved post-course disaster preparedness knowledge scores from a minimum increase of 0.3 to a maximum increase of 2.3 on the self-reported, five-point Likert scale. Post-intervention knowledge score changes for emergency department triage showed improvements ranging from .3-1.4 on the 5-point scale (Table 3), while the CP course knowledge score improvements ranged from .7-2.3 (Table 4). Weekly and final student focus groups, as well as instructor debriefings and reviews of drill video records (machinimas) revealed several pedagogic and technologic themes.
Student Evaluations of the vTRAIN Program
Emergency Department Triage Self-Reported Knowledge Levels Using a 1-5 Likert Scale
Command Post Self-Reported Knowledge Levels using a 1-5 Likert Scale
Team Communication
Students were able to interact with each other in the virtual world via their avatars and voice plus text chat. The multi-user/multiplayer interactions allowed students to practice team communication during an event. In each of the patient surge drills, the students were able to establish a response team with a command and control structure. They demonstrated effective communication among triage nurses and with the nurse supervisor as well as between the nursing supervisor and the CP personnel. The students pointed out that pre-existing relationships and familiarity with each other were immensely important in facilitating the team interactions in the virtual world.
Within the CP drill, students quickly realized the need to focus on their assigned HICS role when resource management and communication became problematic. Although they had been trained in HICS, it wasn't until several students began taking calls from different sources (eg, the County Health Department, press, other hospitals, reports from the emergency department) that the students appreciated the need for a clear method (and chain of command) for communication.
Team Planning
Students quickly realized that there were no assigned locations for treatment areas for different triage categories. This created a patient backup when the chosen triage tent was not large enough to accommodate the number of patients in that category, and a less utilized tent had to be converted. Students mentioned this during the beginning of the second drill and actually did an avatar team “huddle” at the beginning to decide on threshold numbers for re-evaluation.
Students in the CP drill commented on the opportunity to explore and observe the main operations area (emergency department triage). They were allowed to wander and listen in on the cacophony of a large patient surge. Although this typically cannot be done in a real-life situation, the CP students felt that seeing how a patient surge drill might be carried out would help them to plan better locations (eg, critical care tent proximity), and play a supportive role in providing resources.
Team Decision Making
Students were given the opportunity to attempt problem solving without interference from the facilitators. In fact, when the nurse monitoring the tent with the least acute problems was needed to help with the overwhelming numbers of patients presenting to triage, she actually asked for volunteers with health care experience from the patients (role played by the facilitator group) and assigned the most qualified to take her place.
Within the CP, students had to manage reassignment of HICS roles when one of the students had to leave the drill to attend to real-life duties. Although this was an unexpected drill interjection, the students later mentioned that it was a good example of the need to cross-train in HICS roles to be more prepared for such events in real life.
Visual Debriefing
Students positively assessed the ability to reflect on and re-evaluate their individual and team performance. Several students noted improvements that could be made only once they had experienced the machinima of their drill. In fact, after completing the drill in a confined hospital space, many students began to wonder how space would be best utilized during a surge in their emergency department. While in the CP drill, students who were confronted with a resource shortage in the virtual drill began to reflect on the need for better resource tracking (particularly locations of rapidly-needed items such as N95 masks and air filters) in real life.
Students in both classes mentioned how having to function in an unfamiliar environment (the MUVE) was similar to the types of skills needed to manage during a disaster situation requiring rapid decision making and incomplete information. That is, the unfamiliarity of the virtual world, and overcoming it as a team through obstacle courses, scavenger hunts, and a virtual functional drill, in itself, was a real-life, relationship-building experience.
Discussion
This study was an innovative attempt to combine comprehensive reflective practice theory and evaluation using a team-driven learning approach in a virtual learning environment (MUVE). Although it was a pilot study, several interlocking themes emerged: (1) technology; (2) teaching and learning; (3) emergency preparedness; and (4) how all three components influence the students (Figure 2).
Dominant themes from patient surge training in a virtual world technology (and its associated user interface). Teaching and learning (objectives and evaluation) and Emergency Preparedness (authenticity) are interlocked to provide a unique learning experience for students.
Technology
One of the biggest obstacles was how to engage “non-gaming” students quickly in a MUVE. This was accomplished by designing and using virtual orientation stations, and by using brief, real-life, instructor-led sessions that demonstrated basic skills. How to optimize usability for novice students was an ongoing challenge. Surprisingly, students quickly discovered skills not taught to them (eg, flying, private texting), and quickly became immersed in the environment. Within a few lessons, they had shifted from talking about their interaction within the virtual environment (eg, which button to press to talk, or how to turn left) to focusing on evaluating the course content that was presented in the MUVE. They began speaking about their avatars in the first person, and referring to locations in the virtual hospital by pointing in real life (eg, “The resource board is over to the left.”).
Teaching and Learning
Although several different teaching styles and techniques were attempted, in the end, a blended learning style, in which students participated in both virtual and real-life learning environments during each session (each session began and ended in a physical classroom) was used. Within the MUVE, students were provided with individual learning opportunities (examining virtual instructional poster boards), traditional PowerPoint formats (eg, HICS training), and open-ended team functional exercises. In general, students reported feeling the most engaged during the active team exercises within the MUVE. By the end of the lesson, students already had begun to verbalize further virtual scenarios that they wanted to explore as a group. Other network-based epidemic simulations have had similar results. Perhaps this is because the simulations often are open-ended and less defined than traditional classroom or text-book based projects; therefore, the students benefit from the ability to operationalize their problems, observe their own actions, and construct their own knowledge.Reference Huang, Tsai and Wen27
Emergency Preparedness
Chaos and ambiguity are not uncommon during a disaster, and the ability to predict or recreate an exact situation is impossible. However, the virtual environment allows for controlled variables and the ability to easily track numbers (eg, number of patients triaged in a period of time) during repeat performances. The MUVE also allows for realistic interjections (eg, rain, closed passageways, decreased resources), rapid decision making with limited information, and the ability to experience the consequences of those decisions. Furthermore, a recent study suggested that previous disasters tend to catalyze emergency preparedness efforts facility-wide.28 As national emergency preparedness competencies are developed, virtual scenarios can be created rapidly to specifically test them.
Students
Students responded positively to this new curriculum. They reported feeling energized and immersed in the virtual training world. They quickly absorbed the HICS, ESI, and START triage teaching material, but seemed most motivated by an active, team-based learning style in the functional drills. Many students felt that their virtual experience would carry over to real-life experiences (eg, the need to team “huddle” early in a patient surge situation). In fact, other research of network-based simulations in epidemiology and public health has noted a similar response by engaging students in group problem-solving and flexible interactivity.Reference Huang, Tsai and Wen27 The proposed study would continue to explore this new teaching medium, and in particular, to compare this methodology with the costs, efficacy, and impact of usual disaster preparedness education.
Costs
Although the online virtual world, Second Life, was free to access, it did require some additional hardware to maximize the virtual world experience. This included new laptops that could support gaming graphics cards (not typical of a standard hospital computer) and special headsets to experience the spatial sounds and improve communication through individual microphones. Although this was an initial investment, the same hardware can be used for subsequent drills, thereby decreasing the overall costs each time it was reused. This also was true of the time spent by the technical team preparing the virtual environments and tools for each drill.
Another surprising lesson from this pilot study was the students’ views of what was considered “non-productive times” that often are incorporated into a formal cost-benefit analysis. Most commonly, this would occur in the early stages of a novice virtual reality user. For example, if one student was having difficulty navigating in the virtual environment, the other students often took this opportunity to further explore their virtual surroundings or discover other interactive objects that were set aside from previous drills. They also seemed to enjoy teaching each other when someone was struggling (eg, “Walk down the hall towards my voice. You're almost there!” or “Try right-click on the wheelchair.”). During debriefing sessions, when the instructors would start to fast-forward the videos through the early learning times, the students requested the instructor to pause the video; they equated these early virtual reality experiences to the early phase of an actual disaster when environments and protocols have not been established. They reflected on the need to try other communication methods and ask for help in a new environment (the virtual reality), and then pointed out why this would be a good skill to have during a real-life disaster.
Limitations
This was a test of a concept for a new and innovative emergency preparedness curriculum. The sample size was small, and not all students were able to attend all of the lessons due to conflicting clinical work schedules. Due to these small numbers, there was limited quantitative analysis and even less inference can be drawn from this data, other than a trend toward positive experiences of the students and the need for further study, including knowledge retention over time and comparative studies with standard disaster preparedness training.
Although none of the students had prior experience in a MUVE, all had experience using electronic medical records on a day-to-day basis. Real-life technical help also was provided in the physical classroom, which a new student alone with his or her computer would not have available. Although no students dropped out of the study, it is possible that, in future studies, some new students may feel overwhelmed by the use of this new technology in a stressful emergency management context, thus limiting their willingness and ability to participate.
Finally, the curriculum was tested on a sample of emergency department nurses and hospital administrators (all of whom had previous clinical experience and currently work together). It is possible that other job classes or students without previous working relationships may respond differently to a similar curriculum or a different use of the teaching medium.
Conclusions
Team-based, reflective learning in a MUVE is particularly suited to disaster preparedness training because it encourages active student participation and problem solving, and has the potential for multi-disciplinary, long-distance interactions. It is novel in that it is a process-driven simulation that utilizes blended virtual and real-world teaching and learning. It can provide experiential learning without putting students in danger, and can support a student-centered environment that allows for playback and reflection.
Acknowledgements
The authors thank the VA San Diego Healthcare System Emergency Department faculty and staff, as well as the Emergency Management Committee for their invaluable help and support on this project. The authors also acknowledge Barbara Gray for her editorial help in the preparation of this manuscript.
