Introduction
Disaster Medicine is an increasingly important part of medicine, especially Emergency Medicine. The sub-specialty of Disaster Medicine is still in its infancy, and teaching of Disaster Medicine often is scant or even non-existent in medical school curricula.Reference Cummings and Della Corte1 Furthermore, despite the fact that emergency physicians often are required to play key roles in disaster management, many Emergency Medicine residency programs contain little or no formal training in Disaster Medicine. There is a need for the development of a simple Disaster Medicine training tool for Emergency Medicine residents.
For a teaching tool to be useful in an Emergency Medicine residency, it must fulfill three criteria: (1) the tool must not utilize a disproportionate amount of time during the residency program; (2) since residents likely will not be excused from their clinical duties, the curriculum must be implementable on a longitudinal basis; and (3) the curriculum should include academic knowledge as well as practical management skill development.
The Disastermed.Ca simulation program is a software project developed by the principal investigator.2 It includes a collection of simulated patients representing a large disaster, along with a simulated hospital tracking system. Participants manage patients using the simulator as they would in real-life situations, including triage, investigations, procedures, and disposition decisions in real-time. The simulator has been shown to be an effective method of simulating Emergency Department responses to a major disaster.Reference Franc-Law, Bullard and Della Corte3 In addition, the software has been used as a teaching tool for the European Master in Disaster Medicine program at l’Università degli Studi del Piemonte Orientale “Amedeo Avogadro” in Novara, Italy.
The study aimed to evaluate the efficacy of an expedited Disaster Medicine lecture series in conjunction with the Disastermed.Ca patient simulator as a resident education tool. It was hypothesized that the curriculum would result in improved performance on the test instrument, and increase confidence in management of disasters among the participants.
Methods
Study Design
The study was a prospective, observational cohort study.
Study Setting
All residents enrolled in an Emergency Medicine residency program at the University of Alberta were invited to participate in the study. There are two streams to Emergency Medicine certification at this institution: a five-year residency through the Royal College of Physicians and Surgeons of Canada, and a one-year residency after a two-year Family Medicine program through the College of Family Physicians of Canada. Residents were made aware that participation in the study was voluntary and would not affect their residency evaluation. Participating residents provided informed written consent.
Study Protocol
Residents were assigned a random four-digit code number at the beginning of the course for identification on the pre- and post-test: none of the researchers were able to interpret the code.
A pre-test was performed to evaluate initial resident knowledge. Since there is no standardized test for Disaster Medicine knowledge, it was necessary to develop a test specifically for this purpose. The test included three parts. Part I was a collection of 35 multiple choice questions covering various topics of general knowledge in Disaster Medicine. Nineteen of the questions were derived from a questionnaire previously used to evaluate the efficacy of a 24-hour, optional course for medical students.Reference Ingrassia, Geddo and Lombardi4 These questions were translated from Italian to English by the principal investigator. The remaining questions were developed by the principal investigator in accordance with the published guidelines for Canadian medical schools.Reference Cummings and Della Corte1 Part II of the examination consisted of 12 triage questions, again based on English translations of an established Italian tool.5 Unfortunately, since the specific study population of residents was very small, the same population could not be used to both validate the test and partake in the study; thus the pre- and post- test remains a pilot instrument that has yet to be validated. Part III of the test consisted of a series of questions focusing on the attitudes and sense of preparedness of the participants, and is based on previous work of Bartley et al. used to evaluate the efficacy of a disaster exercise scored on a modified 10-point Likert scale (see Supplementary Material online).Reference Bartley, Stella and Walsh6
The teaching curriculum consisted of five one-hour academic sessions. Academic sessions were didactic lessons given by the principle investigator (JF). A slide presentation was given during each session; however, the lecturer attempted to ensure that session remained interactive. During each session, the residents participated in an exercise focused on the development of a hospital disaster plan for a simulated hospital, followed by a short tutorial on a single aspect of the use of the simulator software. Where possible, simulator topics were similar to the curriculum topic (Table 1). Curriculum design was centered around published guidelines for Canadian medical schools.Reference Cummings and Della Corte1 As suggested by the guidelines, topics for residency training included definition of terms, philosophy of Disaster Medicine, description of disaster management, history of Canadian disasters, risk analysis, Emergency Medical Services, hospital disaster planning, medical management of disasters, and psychosocial aspects of Disaster Medicine. However, since all of the residents were in Emergency Medicine programs, the curriculum focused mostly on hospital disaster management. The overall goal was to have the participants themselves develop a disaster plan for a simulated hospital, using facilitated discussions regarding hazard vulnerability analysis, Command and Control, and triage. The participants then used this hospital disaster plan during the subsequent simulation. In addition, although the residents were given a preliminary plan for the Emergency Department of a large urban hospital, they were given the opportunity to modify the layout of the hospital if they felt this to be necessary.
Curriculum summary (h = hours).
Following the five academic sessions, the residents participated in the Disastermed.Ca Emergency Department disaster simulation. The simulation hospital was configured to reflect the changes to hospital layout chosen by the residents. Residents connected to the mobile simulation server using their own laptop computers and web browsers. During the simulation, disaster patients arrived at the simulated hospital; the patient distribution of acuity and time was based on the ergometric work of de Boer et al. (7) Residents were asked to manage the patients, including triage, physician assignment, laboratory testing, radiographic studies, and disposition, as they would if the situation were a true event. All patient management maneuvers were to take place in real-time, with delay times for procedures and investigations determined electronically by the Disastermed.Ca database to represent realistic real-time delays.Reference Franc-Law, Bullard and Della Corte3 All management, investigations, and procedures took place directly on the participants’ laptop computers. Residents were able to request assistance from management staff, who used the printed Disastermed.Ca protocols based on established guidelines for use in disaster exercises.Reference Glooven, Van Laar and Garbett8
Although no individual markers of physician performance were obtained, aggregate markers of Emergency Department management milestones, including average time for patients to reach certain markers of care (triage, bed assignment, physician assignment, and disposition), were obtained from the simulation software. In addition, the number of patients to reach each of the above milestones was obtained. Accuracy of triage, and compliance with standards of patient care were obtained, but only in aggregate with no markers of individual performance.
Immediately following the simulation, residents completed the post-test, which contained the same questions as the pre-test.
The study protocol was approved by the University of Alberta Health Research Ethics Board.
Measurements
The primary outcome for the study was the difference between the pre- and post-test scores. An additional outcome was the ratings of confidence in disaster management among the participants. Performance indicators for Command and Control also were obtained by the faculty using a previously published instrument.Reference Ruter, Ortenwall and Wikstrom9 Lastly, triage accuracy was assessed by comparing the participant’s triage coding to the standard codes from the Disastermed.Ca database that had been assigned using the computerized Canadian Triage and Acuity Score (CTAS) software Version 9.30.008.04.1002 ER1 ( iSoft, Banbury, UK ).10
Data Analysis
Statistical analysis was performed using the “R” statistics package Version 2.10.1 (The R Foundation, Vienna, Austria) for SuSe Linux Version 10.2 (Novell,Waltham, Massachusetts, USA). The null hypothesis of no change between pre- and post- test scores was assessed using a two-sided, paired t-test. Comparison of pre- and post-test ratings given on the Likert scale were made using a two-sided Wilcoxon signed rank test with continuity correction; the null hypothesis being no change between pre- and post-test attitude scoring. A p-value < 0.05 was considered statistically significant for all tests.
Results
Twenty-five residents agreed to take part in the training curriculum. Attendance at the five pre-simulation sessions (pre-test plus four teaching sessions) was unmonitored; to maintain anonymity, no attendance records were taken.
Seventeen medical residents (68%) participated in the simulation, which took place over a three-hour period. During the simulation, 185 patients registered into the simulated Emergency Department. This included 121 victims of the simulated disaster and 64 non-disaster patients. Of the 121 disaster patients, 81 were triaged, 55 were assigned to rooms, and 54 were assigned to a physician.
Resource use during the three-hour simulation included 81 venous blood samples, 24 arterial blood gases, 44 chest x-rays (including 32 portable films), 24 head computed tomography (CT) scans, seven body CT scans (chest, abdomen, or pelvis), and nine focused abdominal ultrasounds. Procedures were performed on 66 patients, including 53 intravenous lines, nine intubations, seven chest tube placements, three blood transfusions, and one central line. Disposition included 19 consultations, 34 admissions, 48 discharges, six operating room admissions, and three deaths.
Fourteen residents completing both a pre- and post-test represent the study group (Figure 1). Mean pre-test score was 29.5/48 (range = 23–39). Mean post-test score was 29.0/48 (range = 20 – 40). Mean difference between post and pre-test scores was 0.35/48 (range= -8–6). There was no statistically significant difference between the post and pre-test scores. (95% CI = -2.2–3.0; p = 0.77). The analysis was repeated by dividing the tests into two sections: general knowledge and triage. On the first test section (35 general knowledge questions), average score increased by 1/35 (3%), although the result was not statistically significant (95% CI = -0.5–3.2; p = 0.13). For the triage questions, average score decreased by 0.8/13 (6%). This result was not statistically significant (95% CI = -2.1–0.6; p = 0.24).
Comparison of pre-test and post-test scores (maximum score = 48)
The pre- and post-tests were also used to evaluate attitudes towards disaster planning. When “I am well prepared for a disaster” was rated on a 10-point modified Likert scale, the median pretest score was 2 (range = 1–7). The post-test median was significantly higher at 6 (range = 2–7; p = 0.003). Participants also were asked to rate on the modified Likert scale whether they felt disaster preparedness was a high priority at the University of Alberta Hospital. Median pretest score was 5.5 (range = 2–7) while median post-test score was 7 (range = 3–9; p = 0.1). Lastly, the participants were asked to rate how well the department was prepared for a disaster. Pre-test score median was 5 (range = 2–8). Conversely, post-test median was 7 (range = 4–9), representing a statistically significant change (p = 0.03).
All seventeen residents who were involved in the simulation returned the post-simulation evaluation; however not all residents answered every question. Median rating when asked to rate the exercise overall was 8/10 (range = 6– 9). Median rating when asked how well the scenario tested the readiness and capability to implement the disaster plan was 8/10 (range = 5–10). All (17/17) residents indicated that they found the simulation to be a valuable learning experience. Of the 14 residents who answered the question about preferences, 100% (14/14) felt that a simulation-based training module was preferable to a lecture-based training program. Of those who voiced a preference for session length, 79% (11/14) felt the training module was of appropriate length, while 21% (3/14) felt that the training module was too long, usually citing that too many lectures were provided. Residents gave a high average rating to the quality of the exercise management staff (median = 10/10; range = 8–10).
The accuracy of patient triage was compared to established standards from the Disastermed.Ca database (Figure 2). The command and control markers during the study, as scored by the exercise management staff, are shown in Table 2.
Accuracy of triage scoring (negative numbers indicate under-triage, i.e., patient was triaged to a less acute triage score than ideal; positive numbers indicate over-triage, i.e., patient was triaged to a more acute triage score than ideal)
Command and Control markers (min = minutes; Command and Control markers were assessed using nine key benchmarks; time to complete the benchmarks for the study group (“Completed Time”) was compared to the standards of Ruter et al. (“Goal”).(9)).
Discussion
Following the course, there was a significant increase in the residents’ perception that they were well prepared for a disaster, indicating that the course had increased confidence among the participants. In addition, the course increased confidence among participants that the Emergency Department was prepared for a disaster.
When the initial testing phase is complete, the curriculum may be further shortened by eliminating the pre- and post-test, and reducing the number of teaching modules to four and total curriculum time to approximately eight hours. In addition, although data such as triage accuracy and markers of Command and Control are difficult to interpret in isolation, benchmark standards for the future could be created through repeated use of the same exercise.
Beyond resident education, the tool also may be considered for education at higher levels of responsibility (staff physicians). The possibility that the abbreviated course could be compressed into approximately eight hours of dedicated time may make it ideal for application in a one- or two-day conference.
Limitations
There are several potential reasons why a significant increase in test scores was not found. First, class attendance was erratic, and no inventory of participant attendance was performed. Nearly all of the test questions were addressed directly during the course lectures; however, answers to these question frequently were wrong in the post-test. It is likely that the erratic lecture attendance was a major contributor. Secondly, since there are no validated instruments to assess knowledge of Disaster Medicine, it may be that the test simply is not valid for assessing knowledge. The triage questions in the post-test examination were based on the Simple Triage and Rapid Treatment (START) criteria.10 The START criteria were taught to the residents during the lectures, as START represents the most commonly used form of disaster triage. However, the participants were given the opportunity to choose a triage method, and elected to use the CTAS system during the simulation.11 The students did not use the computerized CTAS system when assigning the triage score; rather, residents assigned the CTAS category by unstructured observation. It is possible that increased familiarity with the CTAS system may have made triage by the START criteria less familiar. Lastly, it may be that the skills learned in the Disaster Medicine simulation are simply not amenable to assessment by multiple choice testing.
The first step in investigating the lack of increase in participants’ scores on the knowledge and triage testing would be to repeat the same study protocol with a larger and more captive group of participants, perhaps during a one- or two-day conference in which attendance is mandatory and convenient. It is likely that the increased attendance at learning activities would greatly increase the differences between the post and pre-test scores. If a repeated study fails to find a difference between post- and pre-test scores, a second step would be to evaluate the test instrument itself for validity and reliability. The testing tool used in this study was designed primarily to assess knowledge acquisition. However, no separate tool was employed to test the broad cognitive skills required to successfully manage a disaster: command, leadership, communication, situational awareness, workload management, resource management and decision-making.
The study population was small, limited by the number of residents in the training program, and heterogeneous (residents were enrolled in one of two training programs and were at various stages of their training).
Conclusions
The abbreviated simulation-based training module in Disaster Medicine was well received by Emergency Medicine residents. All residents agreed that the course was a valuable experience, and felt that the simulation-based learning module was preferable to a module based solely on didactic lectures. Most felt that the curriculum was of appropriate length at approximately eight hours. Although there was no statistical improvement in test scoring, following the course, residents felt more confident in the ability to manage a disaster, both on an individual and departmental basis.
Abbreviations:
CT = computed tomography
CTAS = Canadian Triage and Acuity Score
START = Simple Triage and Rapid Treatment
