Introduction
The lack of emergency preparedness training (EPT) for patient care providers—including clinicians, hospital workers, mental health providers, public safety and law enforcement officials, community volunteers, EMS, HazMat and Fire personnel—poses significant risks to both patients and providers. During the 1995 Tokyo sarin gas attacks, for example, most patients bypassed first responders and reported directly to hospitals, where staff was exposed to sarin due to inadequate personal protective equipment (PPE) and training.Reference Tokuda, Kikuchi, Takahashi and Stein1 Furthermore, during Hurricane Katrina, EPT deficits were cited as significant factors contributing to adverse patient outcomes.Reference Barkemeyer2-Reference Leder and Rivera7
Providing comprehensive EPT for medical trainees is important to the future success of emergency preparedness operations, yet few medical schools have defined and implemented EPT core competencies for health professionals.Reference Ciraulo, Frykberg and Feliciano8-Reference Chen, Wilkinson, Richardson and Waruszynski24 Recent reviews suggest that health care worker EPT programs lack clarity, objectivity, competency-driven goals, scientific rigor, prospective validation, and consistency across medical specialties.Reference Subbarao, Lyznicki and Hsu25-Reference Hsu, Jenckes and Catlett28
A prior study revealed how medical students can value and adopt core EPT elements via a novel addition to an existing curriculum.Reference Scott, Carson and Greenwell19, Reference Scott29 A significant limitation of the study was the relatively simplistic measurement of EPT performance. A new course from the Center for Health Professional Training and Emergency Response (CHPTER) was developed to create loud and chaotic, multi-actor, clinical disaster scenes (combining more than 10 human actors with human simulators at one time) that would enhance provider knowledge, skills, and comfort necessary to save lives during a simulated disaster. This study presents the performance data of trainees participating in the CHPTER EPT course in 2011 and 2012, including trainee self-assessment, course assessment, precognitive and postcognitive performance, small-group scenario performance, and clinical mass-casualty scenario team performance.
Methods
Curriculum Development—Organization
In 2009, CHPTER was formed as South Carolina's first collaborative EPT center for care providers (www.musc.edu/chpter). Over the course of several months, a CHPTER task force established learning objectives for a 1-day EPT course utilizing existing evaluative frameworks, coursework, and data.Reference Scott, Ross, Schnellmann and Wahlquist30-Reference Bloom, Engelhart, Furst, Hill and Krathwohl39 Using a modified Delphi process, dozens of competencies were consolidated into 18, and subsequently assigned to five competency domains (Table 1). The task force then developed performance objectives to match competency objectives, and these guided content development for the didactic, small group, and simulation course components. The project was approved by the Medical University of South Carolina Institutional Review Board (IRB).
Course Learning Objectives, Competency Domains, and Performance Objective Categories
aDiscrete Knowledge/Cognitive Objective.
bAttitudinal/Affective Objective.
cPerformance/Skill Objective.
dReferences are to learning objectives, above.
Small-Group Exercise Development
Didactic Module 1 was developed as an introductory lecture to the training course. Didactic modules 2, 3, and 4 were developed to include a small group exercise designed to prepare students for the afternoon disaster scenario. Following a short slide-based lecture, trainees were randomly assigned to teams of four to six and were assessed by CHPTER instructors who served as small group facilitators.
The team building exercise (module 2) consisted of seven performance components and four fictional disaster scenarios printed on 34-piece puzzles. The unsolved puzzles were presented to teams in large envelopes, and teams were instructed to complete at least one disaster scenario. Instructors evaluated the teams based on their ability to complete the puzzle and verbalize appropriate responses to threats presented. The communications exercise (module 3) consisted of three scenarios (bus crash, factory explosion, and chemical leak) presented to team members on preprinted handouts, and teams were instructed to complete at least one scenario. Scenarios were discussed in small groups, guided by a facilitator who measured group effectiveness in communicating clinical disaster information. The triage exercise (module 4) consisted of a tabletop triage exercise using 60 small toys; each toy represented a simulated patient and was imprinted with clinical information. Groups were asked to classify the 60 simulated patients according to Simple Triage and Rapid Treatment (START). Facilitators evaluated trainees based on their ability to work as a team to rapidly assess and accurately triage several patients at once.
Human Simulation and Actor Scenario Development
Over six months, CHPTER worked with a university-based human simulation center to develop a series of fictional clinical disasters that combined up to six SimMan high-fidelity and low-fidelity patient simulators (Laerdal, Wappinger Falls, New York, USA) and up to 15 trained live “actors” to simulate a clinical disaster. One of the center's larger observation rooms resembled a small emergency waiting room, and a door to the outside of the simulation center simulated an emergency department (ED) entrance. Storyboards and a stage map (Figure 1) were developed for the simulated exercise. On the stage map, three zones were identified, with an observer assigned to each zone (O1, O2, O3). The letter A designated a human actor and the letter S designated a human simulator. On the color stage map (available online), blue indicated an observer, while green, red, yellow and black indicated a patient's START triage classification. If a chair appeared empty, with no patient (no letters, no color), then it was simply an empty chair.
Simulator and Actor Staging in Human Performance Lab Blue, observer trainer; green, triage green; yellow, triage yellow; red, triage red; black, triage black A, actor; S, human simulator; O, observer trainer; 1, zone 1; 2, zone 2; 3, zone 3 Example: 2S1 (red) is a high-fidelity human simulator patient who is triage red in zone 2
Members of the CHPTER task force were trained to operate simulation center equipment and software. Observers were assigned to different geographical areas (zones 1, 2 and 3) of the scene so that they could more easily focus on performance objectives during the chaotic movement of patients and actors during the scenario.
Two months prior to training, 15 actors were recruited—including trained patient actors from the medical university—and provided pre-scripted roles. To enhance reliability and validity of performance objectives expected to be achieved by trainees, actors were taught how to follow specific behaviors when confronted with trainees in a chaotic environment.
Multi-Actor Clinical Disaster Scenario: “Influenza-like Illness”
The clinical casualty scenario developed by the curriculum task force involved the acute presentation of cruise-line tourists complaining of cough and shortness of breath. During the 6-minute scenario (“Influenza-like Illness”) both actors and simulators presented with various levels of respiratory complaints, several of them outside awaiting access to the ED waiting room. Working in teams of four to six, trainees were told they were health care providers from different areas of the hospital called down to mitigate the complex and chaotic scene using skills they had learned during didactic and small group lessons. The responders were not aware that some of patients were suffering from inhalation anthrax. They also were unaware that one of the patients was carrying two bags of powder, presumed to be anthrax.
Of the four human simulators used in this scenario, two were unstable and required active airway maneuvers (i.e., simple jaw thrusts) in order to open their airways. If teams recognized patient acuity and acted within four minutes, the simulators’ vital signs normalized. If not, CHPTER observers allowed the physiologic parameters of the simulators to deteriorate irreversibly, resulting in cardiac arrest.
Trainees were also confronted with several actors demanding care who were trained to escalate their behavior during the scenario unless appropriate performance measures were met. Once escorted to the Green Triage (this coincides with the “Team Staging Area and BLS/PPE Equipment” area in Figure 1) waiting area, patients with minor complaints changed costumes and presented to the ED entrance as new patients. One actor (representing the terrorist disguised as a tourist) utilized a distraction caused by other actors to bypass security and enter the ED waiting room. If within four minutes the teams recognized that their resources were outstripped and closed the ED door, the patient surge (and terrorist threat) was averted. If not, the actor entered the ED and held up two bags of powder, exposing all patients and all providers to a simulated lethal dose of anthrax.
An educational intervention was developed for all teams participating in the mass-casualty scenario. Immediately after completing their first attempt at the scenario, team members participated in a facilitator-led debriefing using digital video (and split- screen technology) from four different camera viewpoints. Center for Health Professional Training and Emergency Response instructors reinforced key components of the learning objectives and competencies for the course without prompting trainees with specifics about how to remedy the disaster scene. Per study design, teams repeated the 6-minute scenario after the debriefing. Team membership and scenario content did not change.
Research Metrics
The task force developed an online pre-test and two post-tests for trainees utilizing a pool of questions developed by the task force to meet the learning objectives and competencies of the course. The pre-test consisted of two parts. In the first part, Likert-scale self-assessment questions measured trainees’ sense of personal capability and comfort level in handling a disaster. The second part included 23 discrete multiple-choice questions.
The post-test consisted of three parts. The first part was identical to the first part of the pre-test (Likert-scale self-assessment questions). The second part of the post-test contained 23 discrete multiple-choice questions from the question pool. In some cases, slightly modified questions from the question pool were selected. For example, one post-test question described the same patient triage scenario in the pre-test but assigned different vital signs to the patient so the triage answer was “Red” instead of “Yellow”. The third part of the post-test consisted of Likert-scale and open-ended questions for trainees to evaluate course content and instructors. Trainees were asked to complete the same post-test four to six months following the training.
Performance objectives during the mass-casualty scenario were measured and recorded by three CHPTER observers (zones 1, 2 and 3 observers). Observers recorded whether performance objectives were “met” based on their observations of the team during the scenario. Observers developed a single list of “met” performance objectives for each team and, with the help of digital video playback, were able to complete any part of the checklist they did not have time to finish. Any data missing from an observer's checklist was considered an “unmet” performance objective.
Descriptive statistics (means, standard deviations, and medians) were used to describe elements of the EPT training, as appropriate. Wilcoxon rank sum tests were used to compare the pre- and post-test values within the cohorts of medical students and VHA trainees. Because many of the pre- and post-tests were completed by the same individuals, and because the test was completed anonymously (as required by the IRB and, unfortunately, making it impossible to conduct paired testing), the P values obtained from the Wilcoxon rank sum tests are conservative estimates. In other words, if the pre- and post-test scores were linked and a paired analysis performed, the resulting P values would have been smaller than what was observed in the independent testing done by the Wilcoxon rank sum test.Reference Diehr, Martin, Koepsell and Cheadle40
Results
Participants
On February 28, 2011, 10 fourth-year medical students received approximately nine hours of training at the university simulation center. The students volunteered for the course after receiving e-mail from CHPTER and the College of Medicine. On March 13, 2011, 17 participants from the Veterans Health Administration (VHA) Strategic Health Care Group (EMSHG) were provided the same 9-hour course. The VHA clinicians volunteered for the course after receiving e-mail from CHPTER and the local VHA medical center. In the second year of the course, 14 fourth-year medical students were trained on April 2, 2012. With the exception of a small percentage of actors and course instructors, recruitment methods and course content did not change in the second year of the course. Trainee descriptions are shown in Table 2, along with medical student and VHA demographic data. Trainees (N = 39) included 24 (62%) medical students; seven (18%) physicians; seven (18%) nurses; and one (3%) emergency manager. Of the VHA providers, seven (47%) reported greater than 16 annual hours of disaster training while 63% (n = 15) of the medical students reported no annual disaster training. One medical student erroneously self-identified as a mental health provider. After confirming that this was an error, the student's responses were included in the analysis. In addition, two of the VHA clinicians completed the course but did not complete all components of the online assessments, including the demographic data survey. The responses from these clinicians were excluded from the analysis.
Trainee Demographics and Hours of Disaster Training Required and Performed per Year
Abbreviation: VHA, Veterans Health Administration.
Discrete Knowledge: Before and After EPT
Pre- and post-test data for medical students and VHA clinicians are depicted in Table 3 for before and after EPT as well as at a point four to six months after EPT.
Trainee Cognitive Results Before and After Training and Self Assessments, and Course Assessment Results
Self-Assessment of EPT Knowledge and Skills
Both medical students and VHA clinicians were asked to assess their EPT skills and comfort levels with performance during a clinical disaster utilizing a 1 to 100 analog scale with higher scores indicating increased comfort levels. Data for medical student skill ratings are depicted in Table 3.
EPT Course Assessment
Trainees were asked to rate the course using an analog scale from 1 (did not meet expectations) to 100 (exceeded expectations). Higher scores indicated positive responses for the evaluation of the training components. These data are presented in Table 3.
Small-Group Exercise Performance
Results of the small-group exercise were as follows:
1. Team Building: Module 2—For the seven performance components in the team building exercise, 89% (8 of 9) of the teams “identified a team leader” and 78% (7 of 9) “pre-huddled, team leader assigned roles, and team members understood roles” whereas 69% (6 of 9) of the teams “recognized and discussed the threat.” All teams “completed at least one image puzzle, accurately described scene” and reported a positive learning experience with the team building exercise.
2. Communication: Module 3—Eight of nine teams completed all performance components (discussion of chain of command; discussion of relevant verbal communication; and verbalize an accurate, concise and clear report). Teams’ average rating of the “bus crash” scenario during the small group exercise using an analog scale, from 1 (did not meet expectations) to 5 (exceeded expectations), was 4.4 out of 5.
3. Triage: Module 4—For the first five performance components (developed team, communicated appropriately with other medical personnel, constructed appropriate scene layout, classified patients using START triage system, and reassessed patients already triaged), 8 of 9 groups completed all. For the last performance component (classified all victims within the time limit), 3 of 9 groups were not able to finish on time. The average number of patients reported by each triage category relative to accurate triage count was red (20.3 vs 21), yellow (17.0 vs 19), green (12.2 vs 12) and black (8.1 vs 8).
Clinical Mass-Casualty Scenario Team Performance
During the 6-minute scenario, nine teams of four to five trainees were asked to mitigate a clinical disaster scene that included three simulated patients and 11 actors. Ten of the actors were patients and one of the actors served as the ED security guard. The zone 1 observer was responsible for rating team performance for all teams (Table 4).
Performance Objectives Completed by Nine Teams During Clinical Mass-Casualty Scenario, by Zone
Abbreviations: ER, emergency room; EOC, emergency operations center; PPE, personal protective equipment.
Likert Scale Assessment of Team Performance
There were nine categories of team performance (recognition of disaster, external communication, internal communication, utilization of resources, patient triage and care, patient counseling, personal and staff safety, and facility safety and security), and their assessments are depicted in Table 5. For all categories, teams improved their scores after the debriefing (Table 5).
Clinical Mass-Casualty Scenario Team Assessment
1-10 Scale; 1 = did not meet expectations and 10 = exceeded expectations.
Disaster Scenario Team Performance
Of the 34 performance objectives during the 6-minute scenario, 23 were completed by at least half of the teams during their first attempt. Thirty-one were completed by at least half of the teams during their second attempt. Of the 13 patients in this disaster scene, two simulated patient (2S1 and 3S2) were suffering from occluded airways and could be saved if teams intervened clinically with an airway maneuver prior to the 4-minute mark. Four of nine teams in zones 2 and 3 were able to resuscitate 2S1 during their first attempt; 5 of 9 teams were able to resuscitate 3S2 during their first attempt. All but one of the teams resuscitated 2S1 and 3S2 after the debriefing. In zone 1, 3 of 9 were able to secure the ED door and thwart an exposure to anthrax within four minutes during their first attempt and all teams were able to secure the ED within four minutes during their second attempt. Team performance (ie, teams’ ability to resuscitate 2S1 and 3S2 and prevent anthrax exposure) as a function of declining patient stability and scene safety are presented in Figures 2, 3, and 4.
Median Time to Meet Performance Objectives in Zone 1, by Team Attempt and Safety of Disaster Team
Median Time to Meet Performance Objectives in Zone 2, by Team Attempt and Stability of Patient 2S1
Median Time to Meet Performance Objectives in Zone 3, by Team Attempt and Stability of Patient 3S2
Discussion
Efforts to foster EPT for medical trainees in a simulated environment date back to the 1950s when researchers assessed the use of movies to augment students’ training for military and disaster events.Reference Ruhe and Byfield41 The Institute of Medicine's 1999 report, To Err is Human: Building a Safer Health System, spurred new development of the use of human patient simulators to train medical providers.Reference Kohn, Corrigan and Donaldson42 In the 1990s, lessons learned from the aviation industry's Cockpit Resource Management (CRM) led a group of anesthesiologists to pioneer the Acute Crisis Resource Management (ACRM) training module that uses human patient simulators to help reduce error during emergent clinical events.Reference Howard, Gaba, Fish, Yang and Sarnquist43-Reference Murray, Boulet, Kras, Woodhouse, Cox and McAllister45 Subsequent research has suggested a potential benefit of human simulators to train care providers for clinical emergencies.Reference Wayne, Didwania, Feinglass, Fudala, Barsuk and McGaghie46-Reference Wong, Jenkins, Yao and Irwin56
Early applications of simulation to emergency medicine include its efficacy to provide emergency medicine team training,Reference Small, Wuerz, Simon, Shapiro, Conn and Setnik57 a pilot study of simulation to provide Emergency Medicine Crisis Resource Management (EMCRM) trainingReference Reznek, Smith-Coggins and Howard58 and the theoretical use of multiple patients during simulation to replicate the chaotic environment in the Emergency Department.Reference Kobayashi, Shapiro, Gutman and Jay59 Other descriptions of simulation in emergency and military medicine are available.Reference Bartley, Stella and Walsh9, Reference Scott, Miller and Issenberg13, Reference Hsu, Jenckes and Catlett28, Reference Small, Wuerz, Simon, Shapiro, Conn and Setnik57, Reference Andreatta, Maslowski and Petty60-Reference Schumacher, Runte, Brinker, Prior, Heringlake and Eichler67 Recently a comprehensive list of disaster core competencies for acute care providers was published, but it provides little guidance as to how to incorporate simulation into the disaster classroom.Reference Schultz, Koenig, Whiteside and Murray68
A literature search in the PubMed database was performed. Additional online searches were completed to capture government reports from, for example, AHRQ (US Agency for Healthcare Research and Quality, www.ahrq.gov). The search yielded approximately 350 articles and reports. Several evidence-based studies were found that established the efficacy of human simulation to train care providers for a disaster.Reference Miller, Scott and Issenberg12, Reference Subbarao, Bond, Johnson, Hsu and Wasser33, Reference Vincent, Burgess, Berg and Connolly63, Reference Kyle, Via, Lowy, Madsen, Marty and Mongan69-Reference Gillett, Peckler and Sinert72 Only one study utilizing a multi-actor clinical disaster scenario to measure the lifesaving performance of medical trainees was found. Wallace quantified the disparity between the times required to resuscitate simulators and actors during a simulated disaster drill. Six simulators and six actors were presented to triage teams in waves of three patients at a time.Reference Wallace, Gillett, Wright, Stetz and Arquilla73 Because the maximum number of patients during an individual triage encounter was limited in the Wallace study, it is not clear whether the tested curricula would have an impact on trainees confronted with several patients at once.
The Center for Health Professional Training and Emergency Response's 1-day EPT course for care providers recreated a loud and chaotic clinical disaster scene (more than 10 human actors with human simulators at one time) and enhanced care provider knowledge, skills, and comfort level necessary to save lives during a simulated disaster. Discrete knowledge of trainees improved significantly across the board after taking the course in 2011 and 2012. Statistically significant improvements were noteworthy for both the medical students and the experienced disaster medical providers from the VHA, suggesting that this course could be suitable to both novice and experienced audiences. Trainee self-assessment of their EPT knowledge and skill also dramatically improved after taking the course. The overall rating for the course was 96 out of 100, which represents one of the highest course evaluations completed at the medical university.
The human performance measured during the three small group exercises (team building, communication and triage) demonstrated that nearly all of the groups were able to complete performance components accurately and on time. During the disaster scenario in the afternoon session, trainees were faced with a multi-actor simulated disaster. Overall, teams performed well; two out of three performance objectives were met during the first attempt and teams improved their performance after participating in a short debriefing. Teams successfully identified and performed rescue airway maneuvers to resuscitate two simulated patients who otherwise would have expired. They also identified security threats and implemented measures to prevent exposure to anthrax.
While none of the components of the curriculum are individually unique or novel, the curriculum is competency-based with measurable performance outcomes and potentially applicable to a wide range of medical providers. With the exception of Wallace, this study is unique in that it demonstrates medical trainees’ ability to recognize and resuscitate human simulators during a multi-actor simulated clinical disaster. Based on a review of the literature, this is the first published curriculum to use high-fidelity simulation and multi-actor scenarios to measure the lifesaving performance of care providers confronted with more than 10 patients at once.
Limitations
The power assessment of this study (N = 39) is low and the external validity of this curriculum has not been established. A larger-scale study, or preferably a multi-center trial, would allow further analysis of the impact and validity of the curriculum. The use of trained professional actors (who played key patients) as well as trained volunteers (who played patients in the crowd) was utilized to minimize variations in human behavior, improving but not perfecting the study's reliability.
The study sought to measure the impact of EPT on discrete knowledge both immediately after and four to six months after the training day. Unfortunately, these results were not statistically significant, largely because several trainees were lost to follow-up during the first year of the course. A more robust predesign and postdesign with appropriate follow-up could sufficiently power this comparison. In addition, trainees were not tested prior to the disaster scenario (before the training day), primarily because of time and financial constraints. This limits the study's ability to ascertain a performance baseline for the trainees.
Future iterations of CHPTER curricula will need to include a cost assessment to demonstrate the feasibility of this curriculum for target audiences lacking human simulation capability. A version of the curriculum that uses low-cost, low-fidelity simulators and actors is currently under production by CHPTER.
Conclusion
This one-day EPT course improved care provider knowledge, skill, and comfort necessary to save lives during a simulated disaster. Trainees were able to resuscitate two unstable simulated patients and prevent anthrax exposure to the hospital during a chaotic and loud disaster scenario. A larger-scale study, or preferably a multi-center trial, is needed to further study the impact of this curriculum and its potential to protect provider and patient lives.
