Nursing as the largest health care discipline plays a key role in disaster response. A disaster-ready nursing workforce is critical in each phase of the disaster cycle, including preparedness, response, and recovery, to respond to health care needs. 1 To date, there has been no standardization of training focusing on promoting disaster practice and leadership related to nursing. To meet this need, the National Disaster Health Consortium (NDHC) has been developed by Wright State University, College of Nursing and Health, in collaboration with the National Center for Medical Readiness with input from the American Red Cross, Medical Reserve Corps, and military personnel. The program goal is education of nurses in an interprofessional setting to provide both leadership and care in disasters. As a blended learning product, the NDHC program consists of online, classroom, and field components designed to provide nurses (military, public health, acute care, and community-based) and other providers standardized training in disaster preparedness, response, and recovery.
The NDHC curriculum is based on competencies from the Hierarchical Learning Framework of Competency Sets in Disaster Medicine and Public HealthReference Walsh, Subbarao and Gebbie 2 and incorporates elements of interdisciplinary nursing care across the disaster cycle to ensure the delivery of effective and efficient care to individuals and communities. Disaster care is practiced by nurses in a variety of roles, including immediate response, public health, and population-based practice. Training needs to encompass all phases of the disaster cycle to ensure nurses are ready to meet health care needs.
Evaluation of learning outcomes and practice capability is critical to the success of the NDHC program. The purpose of this pilot research was to measure self-efficacy and exercise performance outcomes of the NDHC training program.
Specific Aims
The goal of the pilot study was to measure the effectiveness of disaster training through a systematic analysis of learning outcomes for program improvement.
The specific aims were as follows:
-
1. To assess participants’ performance in a disaster response and recovery exercise after NDHC training. It was hypothesized that participants would meet performance standards in functional exercises after training.
-
2. To determine the differences in participants’ perception of self-efficacy across the disaster cycle before and after program training. It was hypothesized that participants would have higher perceived self-efficacy following NDHC training.
Conceptual Framework and Review of Literature
According to the Association of Public Health Nurses, 3 “Preparing for, responding to, and recovering from disasters is a public health priority that increasingly must be addressed alongside day-to-day public health practice in an environment of constrained resources” (p 5). Leaders within the nursing profession have identified a need for disaster training of nurses, yet there is no recommended specific content for this training. The National Center for Disaster Medicine and Public Health (NCDMPH) was established by Presidential Directive to develop a joint academic program between disaster medicine and public health. The NCDMPH has developed a set of competencies for health care workers who respond to disasters. 4 , 5 The competencies are set forth in the Hierarchical Learning Framework of Competency Sets in Disaster Medicine and Public Health.Reference Walsh, Subbarao and Gebbie 2 The base of the framework is supported by both public health and disaster medicine (Figure 1). The NDHC objectives and content are derived from identified competencies and subcompetencies within the model as well as role function category-specific competencies. As a newly developed hierarchical learning framework, this research addresses a gap in the literature by determining the effectiveness and refining training curricula based upon the framework developed by the NCDMPH.
Learning Framework. From Walsh L, Subbarao I, Gebbie K, et al.² [Reprinted with permission. Copyright: Society for Disaster Medicine and Public Health (2012).]
METHODS
Design
This pilot study used a pre-test, post-test design to collect data on study participants’ perceived self-efficacy before and after the NDHC training. Performance during summative functional exercises was assessed by using a researcher-developed rubric after training. The proposal for this project was exempted by the institutional review board. Informed consent was obtained online before student participation. Results of assessments were confidential. No identifiers were attached to any stored data.
Sample
The target population was participants enrolled in the NDHC training courses from October 2014 through May 2015. A convenience sample was drawn from the target population. The participants were predominantly nurses (96%), although some of these nurses had dual training as paramedics or emergency medical technicians (23%). The participants were diverse in their current employment, representing hospitals, community health agencies, administration, academia, and humanitarian (Red Cross) and military sectors. A power analysis identified that, assuming a medium effect size (d=0.5), and alpha of 0.05, and a power of 0.8, at least 32 pairs of observations were needed. To account for possible attrition from the study, the desired sample was increased 20% for a total of 38. Inclusion criteria were 18 years of age or older and enrolled in the NDHC program. The study exceeded the suggested sample size.
Program Description
The NDHC program is designed for health professionals who may not typically work in the disaster area but who will be called on in both disaster response and recovery as health care providers. To optimize outcomes, the whole health care community should have a unified response during a disaster. The target audience for NDHC includes health workers such as nurses, social workers, pharmacists, physicians, mental health professionals, allied health professionals, civilian and military workers, and first responders. The mission of the NDHC is to enhance medical readiness and resiliency through the standardization of disaster health preparedness, response, and recovery training among all health care professionals in civilian, military, and humanitarian sectors. This competency-based training program prepares health professionals and first responders by using the Hierarchical FrameworkReference Walsh, Subbarao and Gebbie 2 and providing comprehensive knowledge of community preparedness, response, and recovery for disasters. The NDHC program immerses participants in a one-of-a-kind interactive curriculum that combines online, on-site, virtual, and hands-on simulation in real-world disaster scenarios designed to develop decision-making, leadership, and disaster response skills under stress. Incorporating active learning enables participants to obtain a better working knowledge of community preparedness, response, and recovery in disasters. The NDHC is designed to ensure that interprofessional, competency-based training pulls all health professionals into compatible training, taking the health care community a significant step closer to a nationwide standardized response to disasters. The NDHC is a model course with the ultimate goal of leading the way for standardization in disaster training.
Setting and Program Description
Prior to taking part in the course, prerequisite activities included completing 4 Federal Emergency Management Administration (FEMA) courses as well as basic disaster life support. Students were then asked to complete online and interactive unfolding case studies as well as self-efficacy assessments that were delivered through the NDHC program’s learning management platform (a web-based system delivering course content). Students then completed the 4 online course modules that included a variety of topics including History of Disasters, One Health All Hazards, Emergency Management in Healthcare Organizations, and Crisis Leadership During Disasters. Online training was followed by a 4–5-day on-site training that took place at the National Center for Medical Readiness (NCMR). This state-of-the-art, collaborative training and research facility is located on 52 acres in Fairborn, Ohio. Post-assessment of self-efficacy and performance was conducted at NCMR. Refer to Table 1 for a summary of the NDHC learning activities.
Teaching Methods
Measures
The dependent variables included (1) perception of self-efficacy defined as a score on the Emergency Preparedness Information Questionnaire (EPIQ) and (2) performance during the functional exercise defined as the score on the performance rubric. The independent variable was the NDHC training.
Self-Efficacy Questionnaire
The EPIQ measures self-reported familiarity with emergency preparedness competency dimensions. Developed in 2003 by Wisniewski, Dennik-Champion, and Peltier,Reference Wisniewski, Dennik-Champion and Peltier 6 the tool was modified in 2008 by Garbutt, Peltier, and Fitzpatrick.Reference Garbutt, Peltier and Fitzpatrick 7 Previous use of the EPIQ documents its psychometric properties in measuring nurses’ self-reported familiarity with emergency preparedness competency dimensions (Cronbach alpha=0.734, F=264, P<0.001). The instrument includes 45 items from 8 subscales (Incident Command System; triage; communication and connectivity; psychological issues and special populations; isolation, decontamination, and quarantine; epidemiology and clinical decision-making; reporting and accessing critical resources; and biological agents) with one item assessing overall familiarity with response activities.
Performance-Based Rubric (Functional Exercise)
Under the direction of an exercise development consultant and experts from NCMR and from the NDHC advisory committee, a rubric was formulated that included the key components of response for the triage, re-triage/transport, surge, and sheltering scenarios. The rubrics were reviewed for face validity by the disaster experts. Interrater reliability to assess evaluator agreement was assessed after the first on-site exercise by using interclass correlation. The Cohen’s Kappa’s statistics ranged from 0.72 to 0.88 on each of the 4 rubrics. The interrater reliability was assessed as satisfactory. Minor changes were made to the tools following the first cohort to improve the rater documentation of performance. For example, one target capability stated, “Did the student properly position the victim?” This was revised to state, “Did the student properly position the victim (sidelying/recovery)?” An overall reliability statistic was not calculated. The authors were not trying to develop an internally consistent instrument. The goal was to develop a useful instrument for evaluating each necessary part of the exercises developed from the critical capabilities.
The rubric assessed target capabilities in each of the scenarios. In the triage and re-triage/transport exercises, content was linked to the Department of Homeland Security 8 target capabilities of “Emergency Triage and Pre-Hospital Treatment” and critical tasks were derived from the FEMA-identified critical tasks “Res.B3a 4.4.3 Conduct triage upon identification to determine type of assistance required.” For the triage exercise, the assessment included 25 items: (1) triage, sample item: Did the student identify the correct category for the victim’s injury?; (2) scene safety, sample item: Did the student verbally identify potential hazards?; (3) principles of extrication (21 items), sample item: Did the student use proper safety measures in breaking the window? Within the surge exercise, one of the critical tasks identified was, “Res.C1b 1.15.5 Coordinate with community healthcare systems when developing surge capacity plans for Acute Care hospital” and “Res.C1b 1.14.2 Ensure facility based evacuation plans include identification of receiving facilities and transportation assets. Transportation assets should be coordinated and planned out with response partners.” 8 There were 10 items in this category. Sample assessment items included, Did the student identify the correct surge level? Assessment of the sheltering exercise was based upon the critical task, “Res.C3a 2.1.1 Develop and implement training programs for mass care personnel to include sheltering, feeding and bulk distribution for general population.” 8 There were 13 items; sample item: Complete shelter intake registration form (at least 2).
Procedures
Solicitation to participate in the study was offered electronically via e-mail before beginning the online course. Those students who agreed to participate in the study indicated consent electronically in a method approved by the institutional review board. Study subjects were asked to complete the online self-efficacy assessment before beginning the online portion of the course. Following completion of the on-site courses, study participants were again asked to complete the same self-efficacy assessment.
The evaluators for the exercise consisted of a variety of personnel including those with health care expertise (eg, medical students) and those with disaster training (eg, disaster medical assistance team members). Exercise evaluators were provided a situation manual that included an overview of the exercise scenario. Training on the use of the evaluation tool was provided at a pre-exercise meeting by the exercise coordinator. During the exercise, performance was measured once, after training at NCMR.
RESULTS
Data Analysis
Specific Aim 1: On the final day at NCMR, students participated in a daylong disaster simulation. Performance was assessed by using the researcher-developed rubrics. Participants were evaluated in the areas of triage, re-triage, surge response, and sheltering. Ratings were assigned as met (no assistance), partially met (minor assistance), or not met (major assistance). Overall, greater than 90% of all rubric criteria were either met or partially met. The most often missed criteria were related to tourniquet application. The findings reported are from the second and third cohorts (the tool was revised after the first pilot cohort) (Table 2).
Rubric Performance Results from Cohorts 2 and 3Footnote a
a Yes=met/no assistance; partial=partially met/minor assistance; not met=not met/major assistance.
Specific Aim 2: Results were analyzed by using statistical software (SPSS Version 22; IBM Corp, Armonk, NY). Descriptive statistics were obtained for both instruments. To determine the efficacy of the training, a paired t-test was done to determine whether there was a significant difference between pre- and post-training assessments. A total of 64 participants consented to be part of the study and completed the EPIQ pre/post training. The EPIQ is reversed scored with lower scores indicating higher levels of disaster knowledge confidence. A mean pre-training score of 154 was obtained with a mean post-training score of 81. Based on results of a paired t-test (df=63), significant improvement of EPIQ scores (t=-15.69, P<0.001) was noted following NDHC training.
DISCUSSION
Nurses are critical personnel in providing disaster response. Thus, nurses must be prepared to respond across the disaster spectrum; this preparation requires education and training to promote safety and prevent harm to those affected by disaster.Reference Evans and Baumberger-Henry 9 The Hierarchical Learning Framework provided core competencies and an outline of role/function and category-specific competencies for health care workers. The NDHC program was designed as a training mechanism for health care workers to obtain these core competencies and develop select role/function-specific competencies. The NDHC training program provided training that included not only a didactic classroom component, but also hands-on implementation in realistic environments.
Previous studies examining disaster education for nurses have been limited by a lack of standardized tools for measuring outcome.Reference Jose and Dufrene 10 This pilot study identified positive outcomes in terms of self-efficacy (t=-15.69, P<0.001) on a pre/post examination of the learning outcomes. The participants’ mean self-efficacy score (EPIQ) all items after training was 1.23, demonstrating participant support of a high degree of familiarity with items (a score of 1=very familiar and 5=not familiar). The concepts measured included emergency preparedness terms and activities, Incident Command, ethical issues in triage, epidemiology and surveillance, isolation/quarantine, psychological issues, special populations, and critical resources, which closely align with the core competencies of disaster medicine and public health. Mastery of these concepts is the foundation on which the hierarchical pyramid of competencies are built.
The success of the course may be attributed to the variety of teaching methods used for providing education in both online and on-site activities that included readings, lectures, discussions, simulations, and drills. These findings are similar to previous studies examining the use of multi-method training opportunities for nurses related to disaster response.Reference Evans and Baumberger-Henry 9 The actual performance of skills in the daylong exercises indicated mastery of both core competencies and role-specific competencies as identified with the hierarchical pyramid. The lowest scoring area was related to the application of tourniquets (participant did not recognize need for tourniquet). It is thought the low score in this area was due to the accuracy of the moulage application. In future exercises, efforts will be made to improve in this area. Overall, feedback from participants regarding the on-site experience and exercises were very positive. The following are sample comments from the anonymous participant evaluations of the experiences. “This program made me more aware of all the pieces/players involved in a disaster. Also made me think more about preparation from a personal & profession and community level and my involvement in all of it.” “I feel that I learn best through practice. I enjoyed the opportunity to participate at a facility that is capable of designing an exercise that mocks a disaster scene. The course inspires me to continue disaster preparedness education to play a role during actual disasters.”
Limitations were also identified in this pilot study. Despite the interprofessional nature of disaster response, study participants were primarily nurses. While funding was secured to provide education for nurses, scholarships were not available to all health disciplines, limiting interdisciplinary participants. This limitation was attributed to the cost of the program and time commitment required as these barriers have been previously identified as barriers to disaster education.Reference Stanley and Wolanski 11 Additionally, the week-long commitment was difficult for many professionals to commit to for the on-site training. These barriers must be addressed in future studies to ensure access to standardized training to develop published competencies and the disaster learning framework for all health care providers. One final limitation of this study was that it measured only perceived performance and performance in a simulated setting. To accurately assess competencies addressed by the National Center for Disaster Medicine and Public Health, further research also needs to address outcomes of those receiving training in actual disaster settings. These complex translation to practice studies will require that interprofessional teams not only train together, but work together to determine how to best measure outcomes of actual response for making any needed program improvements.
CONCLUSIONS
Disasters are increasing in both number and intensity across the globe. Disaster response requires that nurses function as members of interprofessional teams when disaster strikes. While increasing efforts are being made to implement interprofessional educational opportunities, disaster education often still occurs within silos with each discipline training separately. To truly prepare health care providers based on the disaster learning framework, training must include not only the core competencies of the model as addressed in this program, but also the discipline competencies and how disciplines interact to promote safe, quality care in a disaster situation. Educational projects, like the NDHC program, offer the potential to address the need for a standardized, interprofessional disaster training curriculum. Continued efforts to address barriers and identify best practices for preparing health care providers are needed to ensure preparation when disaster strikes.
Acknowledgment
This research was funded by a grant from the Association of Community Health Nursing Educators. The authors acknowledge the work of Jack Smith, Daniel Kirkpatrick, Jim Gruenberg, and Dr. Sharon Stanley for their contributions to the development and implementation of the NDHC program.
From Walsh L, Subbarao I, Gebbie K, et al.2 [Reprinted with permission. Copyright: Society for Disaster Medicine and Public Health (2012).]
