Introduction
Disaster Medicine addresses the critical field of preparing responses to both natural and manmade disasters. When a sudden, mass-casualty incident, such as the terrorist attack on the World Trade Center Complex in New York City (New York USA) in 2001, challenges a community’s demand on services, the importance of disaster preparedness is re-enforced. This incident’s impact on responder safety and health, along with the large number of incident victims, highlighted the need for future preparation to mitigate both victim and responder harm during a disaster.Reference Daniel, Gist, Grock, Kohlhoff, Roblin and Arquilla 1 Disaster preparedness provides for improved response to an incident and reduction in preventable harm during disaster management.Reference Reissman and Howard 2
In 2013, United States Federal Emergency Management Agency (FEMA; Washington, DC USA), the primary government agency for coordination of disaster response, declared 67 major disasters ranging from floods to storms to explosions.Reference Newman 3 This trend has resulted in Disaster Medicine being integrated in medical school training, residency curriculums, and the establishment of Disaster Medicine fellowships.
The growing field of Disaster Medicine is recognized by The Joint Commission (TJC; Oakbrook Terrace, Illinois USA), a United States private organization in charge of health center accreditation which requires annual exercises assessing health care organizations on their emergency preparedness plans. 4 The incorporation of disaster preparedness into resident education is now mandated by the United States Accreditation Council for Graduate Medical Education (ACGME; Chicago, Illinois USA) program requirements for Graduate Medical Education in Emergency Medical Services. 5 Despite this requirement, the effectiveness of both hospital-wide drills and various modes of resident education are not well established. In reviewing the use of surveys, direct observation, and video analysis to measure hospital disaster drills performance, the results are disparate. 6 Research shows a lack of standardization in disaster training methods with specific deficiencies at the levels of command, control, and communication.Reference Kaji, Langford and Lewis 7 Given the varied forms of disaster training and multiple agencies involved in response to a disaster, research is needed in the field of quantifying disaster training and its subsequent efficacy in disaster response by the participants. With enough data, a standardized approach to disaster preparedness can be developed.
This study evaluated the effectiveness of a disaster drill modeled as an Olympic Games challenge for resident education on disaster management. It was hypothesized that the Emergency Medicine (EM) residents participating in a disaster drill would have increased knowledge on disaster management after the Olympic Games style drill. Given the added benefit of simultaneously practicing a hospital’s emergency plan, this study further aimed to demonstrate that hospital disaster protocols can be evaluated objectively with the model of a Disaster Olympics drill.
Methods
Study Design
This was an observational study using a pre-test and a post-test. The study was approved by the Institutional Review Board of the State University of New York (SUNY) Downstate Medical Center (Study number: 501662-1; Brooklyn, New York USA).
Study Setting
The full-scale exercise (FSE) was conducted at SUNY Downstate and Kings County Hospital Center (KCHC; Brooklyn, New York USA). University Hospital of Brooklyn is the teaching Hospital of SUNY Downstate Medical Center. University Hospital is comprised of 376 beds. The KCHC is a Level 1 trauma center with 627 beds. The Emergency Residency program is a four-year training program which includes 60 EM residents, EM/Internal Medicine (IM) residents, and multiple EM fellowships. This FSE was a complex event that required detailed planning from staff of numerous departments within the hospital, New York City Medical Reserve Corps (MRC) members, New York City Fire Department volunteers, and other non-related outside institutions.
Study Protocol
In this study, participants completed a test prior to and after participating in a FSE. A pre-test was given online one week prior to the drill to a convenience sample of 30 participants from the total population of eligible emergency department (ED) residents. An identical post-test was given online three months after the drill to a second convenience sample of 30 ED residents. Assessment groups were not mutually inclusive; however, residents could be sampled at both pre- and post-test intervention points. The learning objectives of the FSE, pre-test, and post-test focused on the curriculum of the US Department of Health and Human Services (HHS; Washington, DC USA) competencies for health care providers: 4
-
1. Understand the process for setting up and running decontamination;
-
2. Understand the process for setting up a registration system;
-
3. Understand the process for setting up a point-of-distribution (POD) site;
-
4. Understand how to safely don and doff (putting on and removing a decontamination suit) hospital staff and decontaminate a victim;
-
5. Understand the management system (Hospital Incident Command System) positions involved with managing a hospital emergency;
-
6. Participate in and understand tasks related to crisis decision making, coordination, and communications for disaster operations; and
-
7. Educate hospital staff about their roles in the emergency response.
The drill was conducted on July 10, 2013 during a five-hour period. The residents/players exercised the hospital’s established emergency response plans, policies, and procedures as they pertained to a disaster significant enough to warrant the activation of the hospital’s Emergency Operations Center.
The Olympic Games design of the disaster drill divided emergency residents into six Olympic teams, with each team completing one of the following five challenges: donn and doff (one team at SUNY); decontamination tent setup (two teams, one at SUNY Downstate and one at KCHC); registration/triage set up (one team at SUNY); infection control (one team at SUNY); and POD set up (one team SUNY). The teams were provided with written instructions outlining the tasks for each of their respective challenges. Tables 1 is an example of the instruction sheet for the donn and doff team (see the Appendix [available online only] for the instructions given to the other teams). Furthermore, a background scenario was applied to the triage, infection control, and POD teams, and these teams were encouraged to work together, although they each had discrete tasks (see the Appendix [available online only] for the background scenario).
Donning/Doffing Group Instructions
Abbreviation: EM, Emergency Medicine.
Measures
Multiple tools for evaluation were utilized with the goal to identify retention of knowledge by ED residents and to monitor real-time performance of the Olympic teams at both hospitals.
Pre- and post-test scores were used to assess attainment of knowledge for the ED residents participating in the educational intervention of the drill.
During the drill, objectives were assessed with a set of 24 binary questions specific to each Olympic team. These real-time assessments were given to each teams’ evaluator(s). The questions assessed the completion of critical actions required in a disaster response for each Olympic team challenge. An overview of the Olympic teams’ critical actions assessed is listed in Table 2. The evaluation forms for each Olympic challenge are in the Appendix (available online only). The performance of each Olympic team was evaluated based on the percentage of critical actions the team met. The overall percentage of achieved critical actions shared by the teams was also assessed.
Critical Actions Assessment Form
Abbreviations: KCHC, Kings County Hospital Center; MRC, Medical Reserve Corps; POD, point-of-distribution; PPE, personal protective equipment; UHB, University Hospital Brooklyn.
The hospitals were evaluated on their ability to support the participants’ completion of the tasks. Evaluators were given binary questions specific to the hospitals’ response to the Olympic challenges, and each hospital was graded based on the percentage of objectives met. This assessment methodology allowed for deficiencies in both trainee education and hospital response to be identified objectively.
After the drill, the Olympic teams prepared a Power Point (Microsoft Corp., Redmond, Washington USA) presentation on their task and a one-page algorithm on how to initiate the emergency operation plan specific to the hospital. This presentation was assessed by a panel of judges and Gold/Silver/Bronze awards were awarded to the Olympic teams.
A hot wash, or after action review, was done with ED residents after the drill. The review incorporated the evaluator (using data from the real-time assessment forms filled out during the drill) and the MRC comments (using data from a questionnaire given after the drill). The objectives of the drill were reviewed with the ED residents and an assessment to whether the objectives were accomplished was made by them.
Data Analysis
Data analysis was performed using STATA version 10.0 (Stata Corp.; College Station, Texas USA). Descriptive analyses were undertaken for the overall study population. For categorical variables, differences were assessed using Pearson X2 tests. For continuous variables for pre- and post-test scores, differences were compared using independent sample t-tests. A significance level of P <.05 was used in all analyses. Data were stratified for hospital site performance of critical actions reporting frequencies for objectives met.
Results
Resident Pre-/Post-test Scores
Table 3 shows the demographics of the residents/players in the drill. The mean resident post-test scores were higher than pre-test scores to a degree that was statistically significant (62% versus 53%; P =.002).
Participant Characteristics
Abbreviations: EM, Emergency Medicine; IM, Internal Medicine.
Olympic Team Critical Actions
The residents’ performance ranged from 48% to 63% of objectives completed correctly met. Figure 1 shows the percentage of objectives met for each individual Olympic team.
Percent Hospital Tasks.Abbreviations: KC, Kings County Hospital Center; POD, point-of-distribution; UHB, University Hospital Brooklyn.
Hospital Critical Actions
The hospitals met 50%-100% of their objectives. University Hospital Brooklyn met 67% and KCHC met 50% (see the Appendix [available online only] for the critical actions measured for the hospitals).
Discussion
Disaster preparedness is nationally mandated for health care systems; however, the method of this preparation is subject to diverse strategies. In designing disaster education, it is important to identify the knowledge to be taught, format the learning for the participants, objectively evaluate the knowledge attained by the learners, and show utility of the skills attained.
This study evaluated the effectiveness of a Disaster Olympics drill to educate physicians in training. The Disaster Olympics combined games as an engaging learning tool and TJC-mandated disaster drills to evaluate resident and hospital response to various disaster scenarios. Through the use of pre- and post-test comparisons, this drill showed a statistically significant benefit of a FSE in the form of Disaster Olympics to educate on the core concepts of disaster management.
During the drill, the residents and hospital staff were evaluated in real time, with objective measurements tracked by evaluators. Through analysis of critical actions completed during the drill, weaknesses were identified and addressed through changes in disaster protocol and re-training of staff. The greatest weaknesses identified for the residents’ critical actions were in safety and communication. Specific communication deficiencies included communicating with the public and use of translation services to communicate with victims. During the after action review, the residents were debriefed on the safety and communication errors during the drill. Safety protocols noted to be deficient during the drill were reviewed and the residents re-educated. The need for communication strategies in mass-casualty incidents with foreign, deaf, and minors as victims was identified.
The hospitals’ performance evaluations revealed institutional deficiencies and errors in protocols that subsequently were amended as a result of the drill. For example, during the drill, it was identified that the devices for electronic patient information did not receive a signal in an area designated for patient overflow according the emergency operations plan; this error was addressed and fixed in real time. Missing critical actions by the hospitals included providing adequate space, safety, supportive staff, communication, and access to the incident command center. The methodology of this drill provided a mechanism to inform hospital systems of objective deficiencies in current protocols so that they can be addressed.
In the context of the current consensus regarding the necessity for standardized disaster preparedness education and the need for effective methods to assess this education, this study demonstrated an effective approach to physician education. The Disaster Olympics model is a platform that is adaptable to multiple scenarios, health care facilities of all sizes, and participants with varied levels of disaster preparedness knowledge. The use of similar drills with assessment tools in the form of pre- and post-test comparisons, real-time critical actions checklists, and post drill reviews can be applied ubiquitously. The validity of this educational model can be assessed in future studies and the creation of a standardized method of delivering and evaluating disaster education eventually can be achieved.
Limitations
This study did not have a comparison group to the participants of the Olympic Games educational model. Although the study shows an improvement in the subjects’ knowledge after participation, the study does not prove this method is more or less effective than other educational models. Future studies using a comparison group can evaluate the efficacy of the Olympic Games model compared to other modalities of disaster education.
The study has minimal objective data showing improvement in the core curriculum of the residents. The pre-/post-test scores are one example of these data. A future project in the planning stages is to take the same group of residents and have them re-do parts of the drill and assess for improvement. Specifically, the avian flu scenario will be re-tested and a comparison of triage and treatment accuracy and implementation times can be analyzed. Furthermore, the hospital performance can be measured over a series of drills using this pilot study as reference for the hospitals’ emergency operation plans efficiency in execution.
Conclusions
Using the model for a disaster drill designed as Olympic Games, teams of residents were challenged to perform critical actions for various components of a disaster scenario. The shared knowledge gained was quantified through a pre- and a post-test, which showed statistically significant higher post-test scores. The percentages of critical actions met were used objectively to identify gaps in resident knowledge of disaster preparedness and the participating hospitals’ emergency operations plans. Future studies applying this model can demonstrate its effectiveness in disaster preparedness with the goal to develop a standardized approach.
Supplementary Material
To view supplementary material for this article, please visit http://dx.doi./org/10.1017/S1049023X16000212
