Introduction
A public health disaster is an event generating a casualty profile which threatens to overwhelm existing public health and medical resources.Reference Obaid, Bailey, Wheeler, Meyers, Medcalf, Hansen, Sanger and Lowe 1 Since 2001, medical and public health organizations have put much effort into planning and preparing for public health disasters.Reference Bartley, Stella and Walsh 2 The increase in disaster planning efforts has been funded largely by mechanisms established through the Public Health Security and Bioterrorism Preparedness and Response Act of 2002 and the 2006 Bioterrorism Act Reauthorization.Reference Thomas, Hsu, Kim, Colli, Arana and Green 3 These Acts established annual funding for hospitals and public health disaster preparedness, contingent on achieving specific benchmarks. Requirements for hospital and public health preparedness funding prompted organizations to create emergency operations plans, acquire information-sharing technology, hire emergency response personnel, and establish preparedness training and exercise programs. Reference Autrey and Moss 4 To improve multiple agency disaster response in the US, a national response framework and National Incident Management System were established in 2008.Reference Biddinger, Savoia, Massin-Short, Preston and Stoto 5 The framework utilized the Incident Command (IC) system to provide a standardized management model for coordinated and collaborative disaster response at any level: single institution, single jurisdiction, or national. The IC system has been adopted by health care, public health departments, and numerous disaster response agencies as a primary component of disaster planning and management.Reference Bartley, Stella and Walsh 2 , Reference Thomas, Hsu, Kim, Colli, Arana and Green 3 , 6 , Reference Buck, Trainor and Aguirre 7 Despite establishment of disaster planning and training improvements among health care facilities and public health, it remains difficult to assess the preparedness of an agency as health disasters rarely occur and limited assessment standards have been developed.Reference Buck, Trainor and Aguirre 7 - Reference Gretenkort, Harke, Blazejak, Pache and Leledakis 13 Disaster exercises provide a unique environment in which response of health agencies to simulated disasters can be evaluated.
The need for models to assess the training and planning of an organization is highlighted by recent disasters. Urban hospitals and public health with substantial resources and manpower have been caught ill-prepared, resulting in serious consequences. Urban hospitals and public health overwhelmed by Hurricane Katrina (2005; Gulf Coast, USA) identified key points of failure in emergency operations planning, coordination between community health agencies, and staff knowledge of emergency operation plans.Reference Klein, Brandenburg, Atas and Maher 14 - Reference Franco, Toner, Waldhorn, Maldin, O’Toole and Inglesby 18 Insufficient planning and development of disaster infrastructure led to critical failures in some health agencies that resulted in the death of patients and significant post-disaster litigation.Reference Rodriguez and Aguirre 19 Sufficient disaster planning in other agencies was undermined by limited staff knowledge of planning and lack of regional coordination.Reference deBoisblanc 17 , Reference Hodge and Brown 20 Overall, the inability to adequately test and evaluate emergency operations planning before a disaster occurs ultimately challenges the effectiveness of plans.Reference Autrey and Moss 4 , Reference Berger 21
From 2010 to 2013, the Center for Preparedness Education (CPE) at the University of Nebraska Medical Center (Omaha, Nebraska USA) developed six tabletop exercises and six functional exercises to test disaster planning in rural Nebraska health care coalitions. The purpose of these exercises was to simultaneously test IC knowledge and coordination of disaster response in multiple rural organizations. The CPE is a multi-agency consortium with a 10-year history of outreach to the public health and health care communities. The CPE was established with a mission to enhance community resilience through affordable preparedness education.
Methods
Exercises
Three Medical Response Systems (MRS) utilized the CPE exercise design team from 2010-2013 to develop and coordinate functional exercises. A total of six exercises were developed and conducted for three MRS: Rural Region One MRS (2011); Tri-Cities MRS (2010); and West Central MRS (2010-2013). A total of 667 participants and 83 command structures were tested and evaluated at health care and public health agencies. Additionally, other response agencies participated, including Emergency Medical Services (EMS), Fire, Emergency Management, and county officials. A convenience sampling of health care and public health agency command staff performance during functional exercises was conducted over these six functional exercises from 2010 to 2013.
The West Central MRS 2010 functional exercise tested and evaluated seven hospitals, three public health departments, and one MRS. The Tri-Cities MRS 2010 functional exercise engaged and evaluated IC structures at 12 hospitals, four public health departments, and one MRS. The West Central MRS 2011 functional exercise evaluated ICs at seven hospitals, three public health departments, one MRS, and the Nebraska Public Health Laboratory (Lincoln, Nebraska USA). The Rural Region One MRS 2011 functional exercise assessed ICs at 16 hospitals, three clinics, five public health departments, and one MRS. West Central MRS 2012 functional exercise tested and evaluated seven hospitals, three public health departments, and one MRS. West Central MRS 2013 functional exercise tested and evaluated seven hospitals, three public health departments, and one MRS. The IC staff and evaluators received no compensation for participating in the exercise beyond normal wages as an employee.
Exercise Planning
Homeland Security Exercise Evaluation Program (HSEEP; Washington, DC USA) exercise planning guidelines were used for exercise development by the CPE exercise design team. Exercises were developed and implemented by the exercise design team as well as MRS steering committees. Exercise planning committees were established for MRS regions, consisting of representatives from hospitals, local public health departments, MRS coordinators, and members of the CPE exercise team. Teleconference, Telehealth, webinar, and in-person meetings were used for the initial planning conference, mid-term planning conference, and final planning conference of each exercise. Planning committees developed exercise objectives for all participating agencies and objectives specific to MRS, hospital, and public health players. The CPE exercise design team provided exercise documentation and exercise briefings. Exercise documentation generated for each exercise included: Exercise Plan, Master Scenario Events List, Player Communications Plan, Controller Communications Plan, Participant Feedback Form, Exercise Evaluation Guides, After Action Report, and Improvement Plans. Planners conducted three, 1-hour briefings for each exercise by Telehealth or webinar: controller briefing on functional exercises, player briefing, and controller/evaluator briefing. Each exercise was hotwashed in every facility following the exercise and after action conferences were conducted by the CPE.
Exercise objectives were developed for evaluation of all exercise participants, public health agencies only, and medical agencies only. Based on exercise objectives and the type of agency (hospital, public health, clinic, or MRS), each participating agency was given specific events to drive IC activation and performance in the exercise. The IC staffs were expected to use standard procedures and emergency operation plans to respond accordingly to exercise events. Patient admission information, community events, press requests, and intra-agency communications were simulated to engage IC staff in disaster-response decision making. All participant objectives for the exercises included testing regional bed tracking and mass-fatality response, as well as medical asset request plans and procedures. Hospital-specific exercise objectives included testing multiple modes of communication, requesting additional staff through the volunteer system, and evaluating hospital plans to transfer patients to alternate facilities. The objectives of public health exercises are to test initiation of IC and unified command, evaluate plans to inform and assist vulnerable and hard to reach populations, and to assess the need for pharmaceutical and non-pharmaceutical disease containment measures.
Exercise Conduct
Injects were developed by the CPE and reviewed by an exercise planner at each facility. Facility exercise planners established expected outcomes for injects that were included in the exercise. Injects were delivered (Figure 1A) to IC centers or Emergency Operation Centers (EOCs) electronically using LiveProcess (LiveProcess; Chelmsford, Massachusetts USA; used West Central MRS 2010) or email (used Tri-Cities MRS 2010, Rural Region One MRS 2011, and West Central MRS 2011-2013). Players communicated directly with all participating agencies listed on the communications plan, including hospitals, public health departments, MRS coordinators, emergency managers, and the Nebraska Public Health Laboratory (Figure 1B). The simulation cell operated at the CPE simulated all communication with agencies not participating in the exercise. The simulation cell is a virtual operations center equipped with a desktop computer, smartboard, whiteboards, projector, 15 phone lines for voice and fax, polycom, wireless internet, and is equipped to facilitate a simulation team of 15 to 20 individuals using laptop computers. The CPE maintains a bank of 15 laptop computers for exercise simulation. This simulation facility is utilized by the CPE’s staff and network of trained exercise simulators. Trained exercise simulators include members of the Center’s exercise design team, representatives from state and local public health, the Nebraska Public Health Laboratory, as well as physicians and nurses. The exercise simulation team acts as the driver for regional disaster exercises by delivering information to players, as well as receiving communications (phone calls, fax, email, or text messages) from exercise command staff participants and on-site controllers. The simulation team captures a great deal of exercise data on individual agency activities as well as regional collaboration. The simulation process was the same for each facility and had been pre-scripted to make sure that all participants had a similar experience; however, the simulation was based on exercise objectives and was able to be altered based on interaction between agency IC and the simulation cell. Exercise protocols were developed to allow participants to terminate exercise participation to respond to any real-world emergencies. The CPE simulation cell conducted the exercise by delivering disaster information or injects to MRS, hospitals, and public health command centers. The command centers then reacted to the information as if in an actual disaster. Examples of disaster information delivered to command centers include: (1) patient information; (2) press reports; (3) information provided or requested by state and federal agencies; and (4) instructions for participants limiting their capabilities, such as loss of phone communications. The majority of simulated information is determined prior to the exercise and detailed as “injects” in the Master Scenario Events List. The CPE develops injects that are delivered to specific agencies at set times using specified methods of communication. Contingency injects also are developed to drive specific player action if actions are not achieved when desired. Exercise controllers at each participating agency also communicate with the simulation cell during the exercise to augment the flow of information or request simulation to be created as needed throughout the exercise. The simulation cell also simulates all activity that would be generated by agencies involved in disaster response that are not participating in the exercise such as the Centers for Disease Control and Prevention (Atlanta, Georgia USA), state health department, law enforcement, local morticians, or EMS providers. To fulfill the simulation component, exercise planners used law enforcement, EMS, and public health experts to represent agencies not participating in the exercise to provide additional realism to exercises. Hospitals, public health departments, and other participating agencies in the exercise respond to information delivered from the simulation cell with internal and external communications to coordinate internally, directly with other participating agencies, or with simulated non-participating agencies through the simulation cell (Figure 1B).
Flow of Exercise Information among Participating Organizations. A. Simulation Center Delivery of Exercise Injects. B. Participating Organization Actions and Communications. Abbreviation: EMS, Emergency Medical Services.
Exercise Evaluation
Each facility EOC had at least one evaluator and controller on-site to monitor and assess participant actions. Facility exercise controllers also had the ability to communicate directly with the simulation cell to adjust the flow of information to the IC or alter upcoming exercise events. Post-exercise participant feedback was gathered using HSEEP participant feedback forms. On-site exercise controllers de-briefed participants and conducted the post-exercise hotwash. Participant feedback forms were collected. Exercise evaluators assessed IC actions using an IC-specific exercise evaluation guide. The IC staff also maintained logs of actions, which were collected by the research team.
The IC logs, participant feedback forms, hotwash feedback forms, and any other documentation generated by exercise play was gathered by the exercise senior controller and analyzed. Results of the documentation analysis were presented by the CPE staff in the facilitated after action conference. The analysis results were used to guide discussion and consensus building with representatives from all participating agencies to develop Improvement Plans.
Exercise Debriefing
A debriefing was conducted immediately following each exercise at participating hospitals, public health departments, and the simulation cell. The debriefing was a facilitated discussion between IC staff, exercise controller, and the exercise evaluator at each agency. Debriefing facilitation consisted of a series of open-ended questions prompting a discussion guided by a trained facilitator to encourage reflection on the exercise experience.Reference Courtney, Toner and Waldhorn 22 Standard debrief questions include: (1) What went well? (2) What did not go well? and (3) What could be improved based on your performance today? Standard practice for exercise debriefing is to elicit three answers for each of the three debriefing questions from the entire group of participants. The IC staff and evaluators were asked to share any additional debriefing comments on standardized HSEEP participant feedback forms to provide an additional mechanism for feedback. The debriefing and participant feedback form captured participant input on both the structure of the exercise itself, as well as IC performance within the exercise.
After Action Conference
Regional after action conferences were conducted after each functional exercise. The after action conferences afforded participating agencies a forum to discuss lessons learned through the exercise, including: individual agency improvement needs, regional coordination improvements, and training needs. The regional after action conference was crucial to improving regional coordination, not only for disaster response, but also for training and planning. Each conference used a consensus-building methodology to develop a regional Improvement Plan. The after action conferences produced an After Action Report as well as an Improvement Plan. The After Action Report detailed who participated in the process, the key strengths of the exercise participants, and areas of improvement. The Improvement Plan detailed necessary changes in the areas of planning, operations, equipment, training, and exercises. The Improvement Plan also contained agencies, an individual point-of-contact, and an expected completion date for every improvement item. The After Action Report and Improvement Plan served to guide individual agencies and the collective region in completing key items identified to improve disaster response capabilities.
Results
All scenarios were based on infectious diseases requiring medical care and public health investigation to drive interagency coordination. The IC activities were as follows:
-
∙ Activation of IC;
-
∙ Assess and continually monitor disaster indicators and information;
-
∙ Assess and continually monitor agency supplies and resources;
-
∙ Coordinate response efforts with other agencies;
-
∙ Report disease information to local and state public health;
-
∙ Prepare public messages;
-
∙ Request medical assets and staff through appropriate systems;
-
∙ Utilize secondary communications;
-
∙ Coordinate patient transfers; and
-
∙ Utilize regional bed tracking and mass-fatality response systems.
The observation tool was based on a combination of HSEEP, Hospital IC System, and peer-reviewed guidelines.Reference Bartley, Stella and Walsh 2 , Reference Gretenkort, Harke, Blazejak, Pache and Leledakis 13 , Reference Dreifuerst 23 , Reference Kaji and Lewis 24 The members of the exercise planning team reviewed all agency assessments. Reviewers had the expertise to completely evaluate each agency’s IC actions (Table 1). Exercise actions were evaluated in two categories: (1) emergency operation plan adherence and (2) intra-agency coordination.
Observed Implementation of Key Emergency Operation and Regional Coordination Activities
By looking at the data from Table 1, a number of issues were detected in these categories. Some issues were not surprising. This exercise design team had commonly observed issues in IC practice, such as difficulties with backup communications, lack of knowledge regarding request of additional assets, as well as lack of knowledge of how to request additional medical staff. More surprising issues observed included: not activating IC, lack of knowledge on whom to contact for patient transfer requests, and agencies not fulfilling requests for incident actions plans.
West Central Medical Response System
Members of the West Central MRS found these exercises to be invaluable to many aspects of the regional planning and preparedness. The process allowed the numerous regional partners to be involved in the simulated response, reaching beyond local jurisdictions. The regional exercise model provided the participants the experience of working with other agencies not normally involved in planning or training, and the simulation cell enabled each agency to test collaboration with state and federal agencies, albeit simulated collaboration.
A critical element of these exercises on a local level was the development of an Improvement Plan, which identified planning gaps at both the regional and individual jurisdiction levels. The stakeholders assisted in developing the improvement plan and identified a point-of-contact from each agency to be responsible for following up with individual improvement items. The point-of-contact was then asked to report at monthly steering committee meetings on the status of their improvement efforts. This was helpful to keep the critical function of implementing corrective actions at the forefront and not left behind with ongoing, day-to-day activities and duties. Most of the objectives and improvement items followed closely to the Public Health and Healthcare Preparedness Capabilities developed by the Office of the Assistant Secretary for Preparedness and Response (Washington, DC USA) and the Centers for Disease Control and Prevention. During the four years of regional exercising, public health departments in the West Central MRS region had addressed 12 out of the 15 preparedness capabilities and the hospitals had addressed six out of the eight health care capabilities, making needed improvements in each area.
Tri-Cities Medical Response System
Tri-Cities MRS had found that the work that goes into planning and implementation was well worth the effort, given the value of the lessons learned. Successfully employed, the functional exercise could simulate the stressful, rushed, and uncertain atmosphere that those in IC would face in an actual disaster. The MRSs and educational institutions like the CPE could help rural and urban areas overcome staffing, time, experience, and expense challenges inherent in developing and carrying out exercises. Tri-Cities MRS had found that once those barriers were reduced, health care facilities were more likely to participate and embrace the lessons learned. Without these systems in place, many could not, or would choose not to, take on the larger, multi-agency exercises.
Rural Region One Medical Response System
Within the Rural Region One MRS area, there were hospitals eight to 275 miles far. In order to test the capability of the hospitals to work together, the different types of electronic capabilities that the CPE provided were used during this functional exercise. The large area that Rural Region One MRS covered made it impossible for hospitals to conduct a functional exercise otherwise. Hospitals use of electronic capabilities for the functional exercise proved effective in testing its practicality and usefulness; plans were made to use Blackboard Connect (Blackboard Inc.; Washington, DC USA), already used as the staff notification system, for the notification of all the hospitals Rural Region One MRS representatives. The representative would be notified in case of a real-life incident and of the conference call that would be held to update the health care coalition members; this would provide situational awareness and allow the hospitals involved to make a request for resources to hospitals. Rural Region One MRS work with the CPE in this functional exercise and the electronic capabilities were essential in forming appropriate plans.
Discussion
Rural health organizations have the added challenge of addressing disaster preparedness with limited resources. Rural health agencies have difficulty not only in carrying out disaster preparedness training and planning under normal operation, but also in responding to real disasters.Reference Ablah, Molgaard, Fredrickson, Wetta-Hall and Cook 25 Rural health and first responder agencies are geographically dispersed, as are the populations they serve. The distance between rural health agencies increases the need for greater regional coordination in overwhelming scenarios.Reference Ablah, Molgaard, Fredrickson, Wetta-Hall and Cook 25 , Reference Jan 26 Due to staffing shortages, rural agencies have been shown to have difficulty carrying out training and practicing disaster response and coordinated regional trainings.Reference Ablah, Molgaard, Fredrickson, Wetta-Hall and Cook 25 , 27 - 30 Evidence-based selection of disaster training programs is lacking, yet essential to rural areas with scarce resources for planning, training, and responding to disasters.
Disaster exercises remain the primary tools to test disaster planning before a disaster occurs. The scalability of the functional exercise is a strength of this study as it enables the assessment of coordination among multiple responding organizations. Large functional exercises enable examination of coordination at various response agencies involved in disaster response. A functional exercise does not involve deployment of resources. Instead of deploying assets and personnel, functional exercises employ a high degree of simulation to drive decision making by agency command staff.
Development of quantitative methods to assess disaster preparedness have been identified as crucial for progression of agency readiness, yet to date, preparedness literature is dominated by anecdotal and qualitative evaluations of disaster preparedness.Reference Berman and Lazar 11 , Reference Courtney, Toner and Waldhorn 22 , Reference Ablah, Tinius, Horn, Williams and Gebbie 31 , Reference Yeager, Menachemi, McCormick and Ginter 32 Assessment of health agency preparedness through disaster exercises remains largely qualitative and quantitative methods have not clearly been established.Reference Courtney, Toner and Waldhorn 22 , Reference Ablah, Tinius, Horn, Williams and Gebbie 31 Only two quantitative evaluations of disaster exercises have been performed. 33 , Reference McDonald 34 Ablah et al. evaluated disaster response competencies and training learning objectives with a hybrid cross-sectional and cohort follow-up design; the study evaluated improvement of self-reported competencies between a pre-test and post-test using training as the intervention.Reference Kaji and Lewis 24 Hawley et al. evaluated self-reported knowledge of mental health preparedness in conjunction with the delivery of mental health preparedness training for allied and public health professionals in Kansas (USA).Reference Martin 35 Both quantitative studies utilized self-reporting of individual’s knowledge on questionnaires.Reference Kaji and Lewis 24 , Reference Martin 35
Quantitative evaluation of an agency’s preparedness using a disaster exercise should include assessment of both exercise design and agency response. Exercise design establishes the criteria for what the exercise is architected to accomplish and what actions the exercise is crafted to drive. These criteria are established by specific exercise elements: objectives, format, participating agencies, and scenario. Quantization of agency response requires evaluation of participant actions by a trained evaluator as well as self-assessment of participants and further analysis of any discrepancies. Quantitative response evaluation using outcome measures indicating information sharing, adherence to protocol, and quantifying response actions.
Measurement of health agency disaster preparedness has been crude; relying on grant deliverables, isolated exercise evaluation, broad self-assessment, and accreditation requirements. These current measurements assess elements of disaster preparedness and do not establish quality of planning. Additionally, planning done for a self-assessment or accreditation requirement may represent planning that exists only on paper, is not known by staff, and will be easy to abandon in a disaster. Assessment of disaster preparedness typically focuses on one agency or a single exercise with individuals representing multiple agencies participating in a discussion-based exercise. While these practices have merit for developing the capacities needed to respond to disasters, they have limited ability to test the operational aspects of a response requiring actions of more than one agency.
Conclusions
Functional exercises utilize a high degree of simulation to test IC staff at one or more agencies in an operational environment. The flow of information and communications between participating agencies using standard means of communication is a strength of the functional exercise in testing regional disaster response coordination. The real exchange of information between participating agencies represents the value of a functional exercise over other types of exercises.
