Introduction
Disasters are critical events for health workers, and responses to disasters have not always been successful. Poor communication, confused management, and inadequate patient flow usually characterize health medical services, and when resources are limited, medical care is inadequate.Reference Collander, Green, Millo, Shamloo, Donnellan and DeAtley 1 In Asia, gaps and challenges still exist in national health surge capacity, including preparedness, staff education, and training.Reference Algaali, Djalali, Della Corte, Ismail and Ingrassia 2
The Kingdom of Saudi Arabia is the largest Arab state in Western Asia. It is divided into 13 provinces and has a population of 28 million (including expatriates). 3 It is a high-income country with a gross domestic product of US$ 746.2 billion in 2014. Floods have been by far the most important natural disasters in Saudi Arabia, but the mass gatherings during Ramadan and for Hajj are important vulnerable events that create strains on public emergency and transportation services, infectious disease control, and food security.Reference Alamri 4 Saudi Arabia also faces the threats of unique pathogens, such as the Middle East respiratory syndrome (MERS).Reference Epstein and Olival 5 Moreover, political unrest in the Middle East heightens the possibility of catastrophe due to violent conflicts and terrorist attacks. Case studies of disaster responses in Saudi Arabia have noted a reactive approach and a greater need for the involvement of health professionals, including nurses and physicians, in disaster management.Reference Alamri 4 , Reference Aldamegh 6
Analysis of case studies of different disasters shows that the learning process plays a significant role in the efficacy of crisis management. Many negative effects on health care systems are the result of inadequate or ineffective learning.Reference Raich, Adler, Stuhlinger, Lorenzoni and Duscheck 7 Current education and training of health workers do not adequately address the unique and complex situations that occur in disasters. Most health workers are not familiar with topics such as disaster triage, the incident command system (ICS), and the special needs of patients exposed to catastrophic events.Reference Schultz, Koenig, Whiteside and Murray 8 Educational efforts are being invested in emergency medicine and family medicine to further develop capacities in disaster medicine,Reference Franc-Law, Ingrassia, Ragazzoni and Della Corte 9 but knowledge of disaster medicine among health professionals remains insufficient.Reference Ingrassia, Foletti and Djalali 10 , Reference Usuzawa, E and Kawano 11
No published reports on the evaluation and assessments of post-graduate training programs in disaster medicine in Saudi Arabia were found. This course was proposed mainly by the disaster medicine unit at Mohammed Bin Naif Medical Center, Riyadh, Saudi Arabia, with the collaboration of three experts from the medical services of the Ministry of Interior (Riyadh, Saudi Arabia), and the Simulation and Skills Development Center of King Abdullah Bin Abdulaziz University Hospital (Riyadh, Saudi Arabia). The main objectives were to introduce an interactive practical training course for health care providers covering knowledge and skills on how to respond appropriately to prehospital major incidents and disasters according to profession-specific roles, and to support and train the health care community in disaster risk reduction approaches in collaboration with other sectors, such as civil defense, safety, and security.
Methods
Overview of the Course
This interactive course consists of 32 hours over four days and focuses on general principles in prehospital major incident management. It was prepared by the disaster medicine unit of Mohammed Bin Naif Medical Center in collaboration with 10 external instructors. It was piloted at the Officers Club of the Minister of Interior and sponsored by Mohammed Bin Naif Medical Center, King Fahd Security College in Riyadh, Saudi Arabia. The course was started on March 13, 2017. During the first three days, morning lectures covered the main core of disaster management. In the afternoon, simulation sessions and workshops were held. The last day was a full-scale simulation using the mass-causality simulation system (Major Accident Card Simulation; MACSIM). 12 The course included classroom lectures, disaster exercises, skills sessions, and a tabletop session.
Development of the Course
The course was designed according to the local context and was based on international standards proposed by the World Association for Disaster and Emergency Medicine (WADEM; Madison, Wisconsin USA).Reference Murray, Clifford, Seynaeve and Fisher 13 With focus on prehospital aspects, it includes three cores of primary disciplines, clinical and psychosocial aspects, emergency and risk management, and public health. The level of education was set at the second level of WADEM’s multi-levels scale for education and training.Reference Murray, Clifford, Seynaeve and Fisher 13 A comprehensive literature review was conducted to identify competence-based disaster medicine programs.Reference Schultz, Koenig, Whiteside and Murray 8 , Reference Smith, Levy, Hsu and Lee Levy 14 , Reference Walsh, Subbarao and Gebbie 15 Four domains were determined and six levels of proficiency (Bloom’s education taxonomy)Reference Adams 16 were used to established core competences (Table 1). A blended learning approach was used by introducing the domains in lectures and using group discussions, experiential learning, and simulation exercises (MACSIM).
Domains, Competencies, and Educational Activities in the Course on Major Incident Management Piloted in Saudi Arabia, 2017
Abbreviations: ICS, incident command system; PPE, personal protective equipment.
Domain 1: General Concepts in Disaster Medicine
This domain had 11 competences and was covered by eight lectures. Some of the instructors used two videos after the lectures to present case studies (a Hajj stampede and the 2009 Jeddah flood) for group discussion. Others used PowerPoint (Microsoft Corp.; Redmond, Washington USA) presentations. In the exercise on the Jeddah flood, the participants were divided in three groups, each of which was assigned to Emergency Medical Services (EMS), public health, or the hospital sector. The exercise was divided into three phases. In the pre-flood phase, each group had to identify activities designed to reduce or eliminate the risks to persons or property or lessen the potential consequences. In the second phase, each group had to identify the immediate response activities needed to save lives during the flood, protect the environment, and meet basic human needs. For the post-impact phase, participants had to identify activities and programs needed to return to acceptable conditions. For each phase, interactive discussions were held with the participants to compare their strategies with the actual activities that took place. The exercise ended with recommendations for future disasters.
For the tabletop exercise, two instructors developed a scenario about a student housing collapse with 40 victims. The main objectives of the exercise were to practice and apply the concept of medical ICS and its functional components, and to explain how ICS elements can be modified to promote flexible adaptation of the response to the scale and type of incident. The exercise covered two phases: the first two hours of the disaster, and the following ten hours. The participants were divided in two groups, given a worksheet to organize resources (ie, personnel and ambulances), and asked to assign roles for each activated position. The instructors directed the discussion with specific questions for each group and gave them a timeline to answer the questions using the worksheet. Each group had to answer the questions, distribute the resources provided, and write the assigned role for each function position for the two above-mentioned phases. During the debriefing, the standard operation procedures and strategies for coordination with other involved agencies, such as fire brigades, search and rescue teams (under civil defense), and police, were discussed with the participants. Post-incident analysis was done at the end of the exercise.
Domain 2: Safety and Protection
This domain, which had four competences, was covered by three activities. The lecture covered strategies to prevent and mitigate risks to self and others. An active shooter incident was presented as an example to illustrate the standard operation procedures required from health care providers in such situations. The basic principles of selection and use of appropriate personal protective equipment (PPE) during major incidents (mainly chemical) were introduced as a three-part workshop. The first part was a lecture about the disaster-scene exclusion zone, indications for using PPE, and the levels required for PPE in chemical, biological, and radiological threats. This session was followed by a video provided by safety departments of the Ministry of Health. The video shows the steps for donning and doffing of Level C PPE and the process of decontaminating people and equipment. In the second part, the participants were trained in donning and doffing PPE Level C. In the third part, the participants practiced the decontamination process on mock victims in a large decontamination vehicle.
Domain 3: Major Incidents Management in the Scene
This domain had six competences. Two lectures introducing major incidents and common triage were followed by a simulation exercise about a road traffic accident between two school buses with 25 injured victims. The participants were divided into five groups, and each group was provided with a white board. Each board had 25 magnetic cards for victims. The front of each card provided information about age and sex, position of the victim on the scene, visible injuries if present, and the mental state of the victim. On the back of the card was information about the primary assessment findings. The participants were given time to categorize the victims with triage color tags and stickers representing primary life-saving procedures and interventions. The algorithm for primary sieve triage with revised trauma score for secondary triage (used in training for the first time in Saudi Arabia) was displayed on a board for everyone to review during the exercise. The exercise ended with a debriefing in which each group discussed the categorization and triage of the victims with their instructors. The session ended with a self-assessment of the sieve triage.
Domain 4: Communication
The domain of communication during major incidents and disasters was introduced as lectures followed by experiential activities on devices used for communication among rescue teams, fire brigades, EMS, hospitals, and the regional command center.
Major Accident Card Simulation Model (MACSIM)
The MACSIM system was used in this study for first time in Saudi Arabia and the Gulf Region.
The MACSIM is used for education and training on responses to major incidents, mainly for different health care disciplines (ie, paramedics, nurses, and physicians), but it can also be used by other agencies such as rescue service, fire brigade, and police. It can test the plans, preparedness, and performance of responders (ie, prehospital, regional command center, and hospital). Training on all components of the response chain were given simultaneously, including communication and coordination between different response agencies. The system was validated for evaluation of the effect of training on knowledge and skills in medical disciplines.Reference Montan, Ortenwall and Lennquist 17
The exercise scenario was about a suicidal bomb explosion at a convention center during an international disaster day when people were having lunch, with more than 80 victims. On the first day of the course, meetings were held with a rescue team leader and police officers to explain the organization and coordination procedures during the exercise to be conducted on the last day.
On the third day, the instructors delivered lectures on the MACSIM system components, including patient cards, treatment symbols, priority markers, staff symbols, and transport symbols. The participants were divided into groups for explanation of the patient card symbols. The information sheets that were distributed included signs for communication/inspection, palpation/auscultation (found on patient cards), and treatment symbols. As the virtual city in the simulation was originally in a European context, it was adapted by introducing minimal modifications in its demography, geography, health facilities resources, and layout of the area where the exercise was provided. A video was used to explain how the MACSIM exercise was performed.
On the last day, the participants arrived two hours before the exercise for briefing about the MACSIM system. The participants were taken to see the stations for scene, triage, treatment, transportation, ambulance dispatch, and hospital. Each white board represented one of these stations. In the patients’ cards, there were vital signs that were changed by the instructor (to show deterioration) if the participant did not provide the proper treatment.
After the exercise (three hours), the participants were debriefed about performance and decision making. The discussion was mostly on how to avoid certain patient deaths on the scene and after arrival of the medical staff. The participants received a certificate of completion at the end of the exercise.
Instructors
Three of the instructors were from the Medical Services of the Ministry of Interior. The others were from the Ministry of Health hospitals (n=2), King Abdullah Bin Abdulaziz University Hospital (n=2), Safety and Security Departments of the Ministry of Health (n=1), civil defense (n=1), and King Khalid University Hospital (n=1). The eight medical instructors were specialized in disaster medicine and management, emergency and disaster medicine, emergency medicine, and simulation. Non-medical instructors (n=2) were a safety engineer and a crisis consultant.
Participants
Twenty-nine health care providers (four females, 25 males) were recruited from main government health facilities in Riyadh (ie, hospitals and polyclinics). Invitation letters were sent to the facilities, and the participants were nominated by the chairperson of the emergency medicine department or the disaster committee at these facilities. Recruitment targeted health care providers, but one health care facility sent a safety engineer. Eleven participants were students in Master’s programs of Public Health in Disaster Management and Epidemiology, two held Master’s degrees in Disaster Management and Crisis, and one had a European Master’s degree in Disaster Medicine. All the participants consented to participation before the course, and the study was approved by the institutional review board (No. H-01-R-012).
Evaluation of the Course
On the first day, a pre-test of knowledge was conducted, as well as a pre-self-assessment survey specific for the MACSIM simulation exercise. The same test and survey were also given on the last day (post).
Assessment Tools
Pre- and Post-Test
Each instructor was asked to prepare questions relevant to their material. The test, consisting of 25 multiple-choice questions with only one correct answer, was to be completed within 30 minutes.
Self-Assessment Survey
The course had two different self-assessment surveys, one for sieve triage knowledge given after the triage tabletop exercise. Sieve triage is not widely used in Saudi Arabia, and none of the attendees had prior knowledge or experience in it. This survey was prepared by three experts and was approved by the MACSIM course director in Sweden (Prof. Sten Lennquist). The second survey was done for the MACSIM simulation exercise for prehospital components.
Feedback
After completion of the course, a Likert scale-based survey was used to obtain feedback from the participants on their overall evaluation of the course. The survey took about eight minutes to complete.
Eight months after the course, a follow-up survey was sent to the participants. The questions included the following: (1) Was this the first course you attended in this topic? (2) Was this course beneficial to you? (3) Did you obtain the knowledge and skills needed from this course? and (4) Have you applied any principles learned from this course since taking it?
Statistical Analysis
The data were analyzed using Statistical Package for Social Science (IBM-SPSS; Armonk, New York USA) version 22 for Windows (Microsoft Corp.; Redmond, Washington USA). The reliability for the knowledge and practice items was analyzed by calculating the test-retest reliability coefficient. The data are presented as means and standard deviations (SD) and as medians and interquartile ranges (IQR). Descriptive analysis was used to present an overview of the findings. The non-parametric Wilcoxon signed rank test was used to compare pre-test and post-test scores, and Kruskal Wallis test was used to compare the scores by type of profession. The level of statistical significance was set at P ≤.05.
Results
Assessment of test-retest reliability for the knowledge and practice items gave a coefficient of stability of 0.8-0.7, indicating acceptable reliability.
The study recruited 29 participants (25 males; four females) but eight did not do the pre-test. Of the 21 who took the pre-test (19 males; two females), there were nine paramedics (42.9%), seven physicians (33.3%), two nurses (9.5%), two technicians (9.5%), and one pharmacist (4.8%). Of these 21 individuals, 17 (80.9%) did the post-test. Descriptive statistical analysis (mean [standard deviation]) was done for all available pre-test and post-test data.
The overall score on the pre-test was 55.1% and it increased on the post-test to 68.4%. To assess the gain in knowledge from the course, pre-test and post-test scores were compared by Wilcoxon signed ranks test for the 17 participants who took both tests (Table 2). The overall post-test scores (Mean 17.1 [SD=2.8]) were significantly higher (P=.003) than the pre-test scores (Mean 13.7 [SD=2.8]).
Pre-Test and Post-Test Scores of the Course on Major Incident Management Piloted in Saudi Arabia, 2017
Three out of the four domains had significant difference between pre- and post-test results (Table 3). Given that the participants were from different health care specialties (ie, physicians, paramedics, nurses, technicians, and pharmacists), there was no significant difference between them in the pre- and post-tests (P >.05).
Pre- and Post-Test Knowledge About Disaster Health Management Course for health Care Providers
a Statistically significant (P ≤.05).
The total average improvement in the self-assessment survey of knowledge of prehospital components (MACSIM) was 80.0% (P<.05), and the post-test scores were significantly higher than the pre-test scores for each question (P=.037-.001; Table 4). The participants (n=23) were also asked to evaluate their confidence in using sieve triage on a Likert scale. The mean score for the overall questions was (8.6 [SD=1.2]; Table 5).
Pre- and Post- MACSIM Simulation Exercise Self-Assessment for Prehospital Components in the Course on Major Incident Management Piloted in Saudi Arabia, 2017
Note: Wilcoxon signed rank test; P <.05 is considered significant.
Abbreviation: MACSIM, Major Accident Card Simulation.
Post-Sieve Triage Intervention Survey of Participants in the Course on Major Incident Management Piloted in Saudi Arabia, 2017
Note: n=23.
Only eight participants (27.6% of all course attendees) answered the overall post-course survey (Table 6). Defining agreement or strong agreement with the given statement as a positive evaluation, 87.5% of the respondents found that the course was appropriate and informative, 75.0% that it was scheduled at a suitable time of year, 87.5% that the facilities and location were appropriate and satisfactory, and 75.0% that the course material was clear and organized. Moreover, 87.5% found that the instructors were effective and responded to questions in an informative, appropriate, and satisfactory manner, and 75.0% that the handouts for discussion groups and case studies were clear and useful. Finally, 75.0% of the participants found that the workshop was valuable, 87.5% that the workshop lived up to their expectations, and 87.5% that the content was relevant to their job.
Overall Participant Evaluation of the Course on Major Incident Management Piloted in Saudi Arabia, 2017
Note: n=8.
a Sum of agree and strongly agree.
Eight months after the course, a follow-up survey was sent to the participants to assess their satisfaction with the course and with retention of the knowledge and skills they gained. Fifteen of the 29 course participants (51.7%) responded, of whom six (40.0%) indicated that it was the first time they attended such a course; all of them agreed that it was beneficial and that they still retained the knowledge and skills eight months after the course. Almost all of the respondents indicated that they applied the principles learned from the course since taking it, especially in drills and daily emergencies, but none of them dealt with real major incidents.
Discussion
Disaster preparedness and training have become basic requirements for The Central Board of Accreditation for Healthcare Institutions in Saudi Arabia (Riyadh, Saudi Arabia). 18 The quality of disaster training programs has an impact on self-perceived disaster readiness of first responders.Reference Stevens, Jones and Smith 19 For that reason, this pilot course is a competences-based, inter-professional course using blended learning teaching methods and involved not only medical and paramedical instructors, but also those in civil defense, safety, and security, all of whom are involved during the response phase.
Competences-based programs have been shown to improve the knowledge and skills of health care professionals during major incidents responses. Several authors have recommended such programs for education and training in the disaster fieldReference Schultz, Koenig, Whiteside and Murray 8 , Reference Walsh, Subbarao and Gebbie 15 because they deliver educational results that can be measured and used as quality improvement guides.Reference Schultz, Koenig, Whiteside and Murray 8 In medical education, the effectiveness of training courses in enhancing knowledge and skills can be evaluated by comparing the clinical work before and after the training; for example, in trauma management training programs. 20 However, in disaster medicine education and training, the situation is different due to the wide variation in the patterns of disasters and the factors that affect the response outcome.Reference Montan, Ortenwall and Lennquist 17 Consequently, no single method can assess the value of training programs, and more than one method is used for the assessment of disaster medicine training programs.Reference Montan, Ortenwall and Lennquist 17 , Reference Amrein-Beardsley and Narnett 21 The course was evaluated by different tools for multiple components: pre- and post-tests of knowledge and skills, a post-test on intervention for sieve triage, and pre- and post-self-assessment of perceived knowledge and skills related to performance objectives of prehospital components of a MACSIM simulation exercise.
Assessment of the gain in knowledge from the course by comparing pre- and post-test results showed a statistically significant enhancement in knowledge (P=.003). Assessment of perceived knowledge and skills related to performance objectives of prehospital components in MACSIM showed an overall average increase of 80.0% (P <.05).
In the triage education session, lectures on all types of triage were presented, with more focus on sieve triage, followed by a simulation exercise of sieve triage on a road traffic accident scenario. The participants were divided into five groups, and each group was asked to triage 25 cases. After the subsequent debriefing session, the participants filled out a self-assessment survey of their perceived competence. The highest score was obtained on the question: “How capable would you be to perform primary (first) triage of injured causalities on-scene in a major incident today using sieve triage system?” The mean score was (8.9 [SD=1.3]). The next highest score (Mean 8.8 [SD=1.4]) was for “How well do you know sieve triage system?” The results of this survey showed that the participants gained considerable knowledge and skill in performances of sieve triage through their first exposure to this procedure (Table 5).
In this course, the instructors from civil defense, safety and security, and police were involved to make the participants aware of the roles of other agencies, and to show them how to coordinate and communicate with them by unifying the approach and language among all responders. This setting and the group discussions also gave the participants the opportunity to network and start using a unified language and understanding the standard approach, which is essential for communication, organization, and selection of the resources required during major incident responses.Reference Schultz, Koenig, Whiteside and Murray 8 Understanding the roles of the different responding professionals and learning how to work together are key aspects of a successful disaster response.Reference Willems, Waxman, Bacon, Smith and Kitto 22
The pre-test results did not differ between the different professions, and neither did the post-test results (P >.05). This is likely due to basing the course on the level-two education described by WADEM (basic knowledge for all health care providers in different professions). Inter-professional learning with colleagues improves the quality of care and assistance provided during a disaster.Reference Atack, Parker, Rocchi, Maher and Dryden 23 For example, involvement of medical students with other health care students (ie, nurses and technicians) can help them to understand the roles and responsibilities of other health care providers, and at the same time, integrate their understanding of teamwork and communication skills in complex situations of patient care.Reference Kaplan, Connor, Ferranti, Holmes and Spencer 24 - Reference Zapko, Ferranto and Brady 26 The same thing can be applied to post-graduate practicing personnel, as in this course. A blended learning approach by combining lectures, group discussions, experiential learning, and simulation exercises was used in order to increase the knowledge and satisfaction of the participants.Reference Collander, Green, Millo, Shamloo, Donnellan and DeAtley 1 , Reference Khorram-Manesh, Ashkenazi and Djalali 27 , Reference Lateef 28 The benefit was observed in the results of the assessments. Simulation-based learning is becoming widely recognized in medical education because it offers clear, recognizable benefits for new trainees to learn invasive procedural skills, especially in environments outside hospitals and clinics, such as disaster scenes.Reference Khorram-Manesh, Ashkenazi and Djalali 27 , Reference Kneebone 29
Only the prehospital components were included in the simulation exercise because this is the first experience with MACSIM in Saudi Arabia, and because the course capacity and logistics were limited to prehospital components. The performance objectives for the prehospital components included organization on the scene, communication, coordination, principles of management of victims, and incident command structures. The overall improvement in self-assessment score was 80.0%, which was significant at P <.05. This improvement is comparable to a reported improvement of 74.0% (P <.001).Reference Montan, Ortenwall and Lennquist 17
When the participants were debriefed after the MACSIM exercise, one major deficiency was found in over-triage and over-use of resources. At the same time, there was under-triage, with green area victims being left for a long time without re-triage, which can affect the prognosis of the victims and lead to development of complications.
It is important that participants retain the knowledge and skills they gain in a course for a period of time. Participants in courses using blended learning maintain enhanced confidence in their knowledge for at least six months.Reference Bajow, Djalali and Ingrassia 30 , Reference Golden and Karpur 31 In agreement, a survey of the participants eight months after the course showed that they were satisfied with their retention. Importantly, they had the opportunity to apply what they learned in the course to drills and daily emergencies.
Limitations
This study has two limitations. First, the targeted health care providers were those without any training or experience in disaster preparedness, but some of the invited hospitals sent health care providers with previous disaster experience, which may have affected the expected improvement percentage. Second, the MACSIM instructors faced difficulty in testing the complete chain of disaster management, such as hospital components, because there are many. To maximize the benefit from MACSIM, in the future, the full chain of management will be tested, including hospital departments, providing the trainers with only related lectures of MACSIM, and allowing for two simulation exercises on two separate days.
Conclusion
These results show that inter-disciplinary education programs in disaster medicine for health care providers may improve their response to major incidence. Such programs can help them to respond appropriately according to their roles. Conducting competences-based, disaster medicine programs can augment disaster preparedness in Saudi Arabia.
