Situational awareness, the gathering and analysis of operational information, is a central concept to all emergency preparedness and response. 1 Having accurate and timely situational awareness to inform decision-making in a large-scale emergency is one of the more important and most difficult tasks of emergency response. In health care emergency response, appropriate situational awareness information is important for assessing the current and future state of the health care system and informs decision-making about the operations, resources, and nature of care provided by the health care system.
The 2009 Institute of Medicines’ (IOM) landmark report Guidance for Establishing Crisis Standards of Care for Use in Disaster Situations: A Letter Report, 2 and several recent national and international disasters have brought to light the need to create comprehensive planning for situations that may require the altering of standard health care and delivery practices, also known as crisis standards of care. The 2009 IOM report outlined a framework to define surge capacity within the health care system as a continuum: conventional (the spaces, staff, and supplies used are consistent with daily practices within the institution), contingency (the spaces, staff, and supplies used are not consistent with daily practices but provide care that is functionally equivalent to usual patient care practices), and crisis (adaptive spaces, staff, and supplies are not consistent with usual standards of care but provide sufficiency of care in the setting of a catastrophic disaster). 2
In 2013, the IOM published a second report outlining a toolkit for jurisdictions and response organizations to develop indicators and triggers for crisis standards of care decision-making. 3 Per the IOM, an indicator is defined as “a measurement, event, or other data that is a predictor of change in demand for healthcare services delivery or availability of resources. This may warrant further monitoring, analysis, information sharing and/or select implementation of emergency response system actions.” 3 The IOM also defines a trigger as “a decision point that is based on changes in the availability of resources that requires adaptations to healthcare services delivery along the care continuum”; these can be either scripted or unscripted. 3 Identifying appropriate triggers and indicators is vital to provide a framework for regional decision-making to support health care response during an emergency.
Unfortunately, no literature is available on the best practices or techniques used by communities to identify the triggers and indicators for decision-making in a disaster. The use of the Delphi method, a consensus-building survey technique, has become more prevalent in the development of emergency response planning and response activites.Reference Carley, Mackway-Jones and Donnan 4 - Reference Lerner, McKee and Cady 9 A recent consensus statement by the American College of Chest Physicians employed a modified Delphi model to produce consensus recommendations for the care of the critically ill and injured during pandemics and disasters.Reference Christian, Sprung and King 10 Additionally, the Delphi method was recently used in the United Kingdom to identify critical resources and quantities required to support care in a mass casualty incident.Reference Duncan, Colver and Dougall 11
The Northwest Healthcare Response Network, an independent 501(c)(3) organization serving as the health care coalition (“a collaborative network of healthcare organizations and their respective public and private sector response partners within a defined region” 1 ) in King and Pierce Counties, in collaboration with the Washington State Department of Health, worked to create tools and protocols to support the collection, analysis, and presentation of essential elements of information from the health care sectors during a disaster. Our goal was to identify key decisions along the continuum of care (conventional, contingency, and crisis), 2 and a consensus set of critical triggers and data elements used to inform those decisions, at the regional and state levels concerning health care and public health response during an emergency.
METHODS
The project was divided into 2 distinct phases to achieve the defined goal.
-
∙ Phase 1: Clinical Survey. A group of clinical panelists identified a set of consensus indicators and triggers to support regional decision-making.
-
∙ Phase 2: State Triggers and Indicators. A group of stakeholders, including the Washington State Department of Health, the Northwest Healthcare Response Network, and public health and health care coalition leads around the state, developed state-level triggers and indicators to support state-level decision-making.
Regional and state-level triggers and indicators were identified through 2 different successive processes to allow the results from the clinical survey to inform the creation and results of the state-level triggers and indicators. Additionally, having a dual-phased approach allowed us to identify panelists with expertise in each level, regional-level vs state-level, of the decision-making process.
Clinical Survey Methods
To achieve the defined goals in Phase 1, we employed a classic Delphi method, a consensus-building survey technique, to create a consensus list of triggers and data points to inform key decisions to support health care during an emergency. The Delphi method is a multistage survey often used to develop consensus or collective agreement among a group of informed individuals.Reference Keeney, Hasson and McKenna 12 We employed the classic Delphi technique to ensure we gathered a wide range of brainstormed triggers and indicators. Key decisions points were identified through interviews with local public health emergency preparedness and health care emergency managers. In the first round, panelists were provided a description and questions associated with 10 key decisions (see the online data supplement) to support health care and public health during an emergency and were asked to brainstorm indicators and triggers they thought would be important to inform those decisions. In the second and third rounds, panelists were asked to rank the importance of the proposed triggers or indictors for each decision. For this survey, a 5-point Likert scale was used (1, very unimportant; 2, unimportant; 3, neutral; 4, important; 5, very important).
Sample
The panelists included individuals from a heterogeneous background that represented voices and opinions from many facets of health care emergency preparedness and response. For this survey, we employed purposive and snowball sampling and ensured that all panelists met the defined criteria: (1) works in a specialty or field that would be called upon to respond during health care emergencies, (2) has 8 to 10 years of work experience in their field, (3) holds sufficient experience within their specialty to be able to provide insight on departmental decision-making, and (4) has the capacity, willingness, and time to participate in all rounds of the process.
There is no clear guidance in the literature about the necessary number or criteria for identifying panelists required for a Delphi.Reference Keeney, Hasson and McKenna 12 To obtain informed and accurate triggers and indicators, participants needed to have both experience and expertise in their field. Participants were identified through the project manager’s knowledge base and recruitment with health care coalition leads around the state of Washington. This project falls under public health practice and was not subject to institutional review board approval. 13
Consensus
Items were deemed to have reached consensus if they were rated consistently by 70% of the panelists. Due to the low number of panelists, some rankings were combined to provide a clearer picture of consensus among the panelists. The rankings of “important” and “very important” were combined to determine if the item reached 70% consensus; likewise, rankings of “unimportant” and “very unimportant” were combined to determine if the item had reached 70% consensus (Figure 1).Reference Duncan, Colver and Dougall 11 Items that reached consensus were removed from the subsequent round of the survey. Following the second round of the clinical survey, individuals were provided feedback on the group ranking (median) as well as their own previous ranking of each proposed trigger or indicator. This allowed panelists to reevaluate their answers, if appropriate, with the group median ranking.
Consensus Determination Flow Diagram.
Ethical Considerations
Answers and comments by panelists did not connect names or individuals to any answers provided in the feedback round or final reports. Because response rate is a concern in the Delphi method, all panelists were provided full information regarding the conduct, process, expectations, timeline, and procedures involved in the project. The project manager remained as objective as possible throughout the process by not making judgement or substantial changes to the participant’s comments and by maintaining confidentiality.Reference Keeney, Hasson and McKenna 12
Data Collection and Analysis
All data were collected via surveys by use of SurveyMonkey (San Mateo, CA). In round 1, proposed indicators and triggers were qualitatively analyzed from the free response comments of participants, preserving the original language as much as possible. If multiple participants proposed the same trigger or indicator, answers were combined to streamline the second-round survey process.
In rounds 2 and 3, detailed statistics of the rankings of all panelists, comments, and whether the items reached consensus were extracted. Individual reports were created for each panelist outlining the group median, whether an item reached consensus, the previous ranking, and any additional comments for each proposed trigger or indicator. All statistical calculations of the median, frequencies, and whether items reached consensus were done by using STATA version 12 (StataCorp, College Station, TX).
State-Level Trigger and Indicator Methods
To achieve the goals outlined in Phase 2, we employed a modified Delphi technique that consisted of a workshop to identify proposed triggers and indicators followed by a survey designed to gather rankings to reach consensus. Fourteen key state-level decisions (see the online data supplement) that may be made in a disaster to support state-wide health care and public health were identified through interviews with the Washington State Department of Health. Some of these key decisions were similar to those identified in Phase 1, but some were distinct to the role or authorities of the Washington State Department of Health and the State Health Officer. A workshop was held to identify proposed triggers and indicators through a consensus-building procedure, similar to that of the clinical survey from Phase 1. Participants of the workshop were briefed on the outcomes from the Phase 1 survey and were provided a copy of the consensus triggers and indicators. The triggers and indicators from Phase 1 were evaluated during the Phase 2 workshop to directly inform the creation of the state-level triggers and indicators. In the workshop, participants were asked to individually brainstorm proposed triggers and indicators for each state-level decision point on a white board. Next, individuals were asked to rank each of the items brainstormed by all participants on the basis of their importance in the decision-making process: important, neutral, and unimportant. Following ranking, the group discussed any comments or questions that remained and came to consensus on each trigger and indicator. This process was repeated for each of the key discussions identified.
Following the workshop, a single-round survey was conducted with all public health emergency preparedness and health care coalition leads from around Washington State to gather their feedback on the proposed triggers and indicators. Similar to the survey in Phase 1, panelists were asked to rank the importance of each proposed trigger or indicator for the identified decision. For this survey a 5-point Likert scale was used (1, very unimportant; 2, unimportant; 3, neutral; 4, important; 5, very important). Similar to the clinical survey, items were deemed to have reached consensus if they were rated consistently by 70% of the panelists. As in the clinical survey, some rankings were combined to provide a clearer picture of consensus among the panelists (Figure 1).
All survey data were collected by using the SurveyMonkey survey platform. Detailed statistics of the rankings of all participants, comments, and whether the items reached consensus were extracted. All statistical calculations of the median, frequencies, and whether items reached consensus were done by using STATA version 12.
RESULTS
Clinical Survey
Of the 21 individuals recruited, 2 dropped out of the project before the first round. A total of 17 of the remaining 19 participants completed the first 2 rounds of the clinical survey. A total of 14 of the 17 panelists completed the third round of the clinical survey. The participating clinical panelists represented 6 counties (King, Pierce, Thurston, Skagit, Snohomish, and Spokane), 5 public health regions (regions 1, 3, 5, 6, and 9) and 5 metropolitan regions (Seattle, Tacoma, Spokane, Olympia, and Everett) around the state. Three panelists had 8 to 11 years of experience in their field of practice, 5 had 12 to 15 years, 1 had 16 to 19 years, and 8 had 20 or more years of experience. Panelists represented a diverse background of clinical specialties (Table 1).
Fields of Expertise of the Clinical Survey Panelists
A total of 213 triggers and indicators were identified through the first round of the clinical Delphi survey. Following round two, 92 proposed triggers and indicators (43.2%) reached consensus. The remaining 121 items and 10 additional items identified in round two were presented back to the panel for a second evaluation in round three. A total of 30 (22.9%) additional items reached consensus. In all, 122 (54.7%) reached consensus and all were deemed important in the decision-making process (Table 2 and supplementary material A in the online data supplement).
Overview of Indicators and Triggers Identified Through Clinical Survey
State-Level Survey
Participants in the brainstorming workshop included emergency preparedness leads from the Washington State Department of Health and the Northwest Healthcare Response Network. A total of 18 participants completed the state-level survey. Participants included public health emergency preparedness and health care coalition leads and represented all 9 public health regions within Washington State and all major metropolitan areas.
A total of 140 triggers and indicators were identified through the brainstorming workshop and were presented to all state-level survey panelists. A total of 110 (79%) of the items reached consensus and all were deemed important in the decision-making process (Table 3 and supplementary material B in the online data supplement).
Overview of Indicators and Triggers Identified Through State-Level Survey
DISCUSSION AND CONCLUSIONS
This project developed a set of regional and state-level consensus triggers and indicators that build the foundation for health care and public health decision-making during an emergency response. The recruitment of diverse panelists from around the state and various levels of response (clinicians, public health emergency preparedness, health care coalitions, state department of health) and a solid methodological framework allowed for the implementation of a rigorous process for planning. Developing the project in 2 phases, the clinical survey and the state survey, allowed the results from the clinical survey to inform the state-level survey and connect the concepts and ideas in a cohesive approach. The triggers and indicators identified are vital to providing the basis for situational awareness information gathering and decision-making concerning health care during an emergency in our region.
A major strength of this project was the use of rigorous methodological techniques to brainstorm, evaluate, and validate conclusions in emergency preparedness and response, a field that does not always employ such techniques. The panelists in both surveys represented heterogeneous geographical and specialty backgrounds lending validity to the triggers and indicators identified. Our engagement and input from various levels of emergency preparedness and response (clinical, local, regional, and state) led strength to our ability to define and connect triggers and indicators at the regional and state level. Current literature utilizing consensus-building techniques in emergency preparedness planning, along with our project, provide a foundation for the use of the Delphi method in emergency response planning.
Nevertheless, there were also limitations. One criticism of the Delphi method is in the choice of panelists.Reference Keeney, Hasson and McKenna 12 If one were to conduct a similar process with an entirely different group of panelists, one could draw different conclusions on identified triggers and indicators. To mitigate this, we were able to implement sampling that ensured all panelists were experienced in their field to participate. All triggers and indicators identified through this process will continue to be validated and modified as necessary through future emergency planning and response. A limitation to the clinical survey was panelist attrition and the amount of time needed to complete the survey: 17 of the original 21 panelists (80.95%) completed the first 2 rounds, and only 14 of the original 21 recruited panelists (66.67%) completed all 3 rounds of the surveys. Additionally, we combined the original 5-point Likert scale into a 3-point scale to ease the calculation and interpretation of consensus, which may mask some of the nuances in the relative importance or unimportance of some identified triggers and indicators.
Further investigation should be conducted to validate (through modeling or real-world events) these triggers and indicators to identify their usefulness in decision-making during a disaster. Additionally, it would be interesting to see if panelists from other expertise and geographic areas around the country agree with the relative importance of these triggers and indicators to support decision-making for health care and public health in an emergency.
The Northwest Healthcare Response Network has begun using these consensus triggers and indicators to identify the data needed to inform decision-making. Although some of the triggers and indicators appear self-evident, the actual data required to inform them must be realistically obtained during a crisis; only then will the indicators and triggers serve the purpose for which they were created. These data will then inform the appropriate questions to ask health care organizations during a response and provide the right information to inform decision-making in an emergency response to support health care and public health. The identification of consensus triggers and indicators for health care emergency response is crucial in supporting informed and comprehensive health care situational awareness processes. These triggers and indicators can inform the creation of standardized questions to ask health care, public health, and other partners in a response to inform regional situational awareness and subsequently inform decision-making to support the health care system during a response.
Funding
Funding for this project was provided through support from the US Department of Health & Human Services, Assistant Secretary for Preparedness and Response, Hospital Preparedness Program, through a contract with the Washington State Department of Health.
Author Contributions
Rebecca Lis and Dr. Vicki Sakata had full access to all of the data and take responsibility for the integrity of the data and the accuracy of the data analysis. Conception and design: Lis, Sakata, Lien. Analysis and interpretation of the data: Lis, Sakata. Drafting of the article: Lis, Sakata. Critical revision of the article for important intellectual content: Lis, Sakata, Lien. Final approval of the article: Lis, Sakata, Lien. Statistical expertise: Lis. Obtaining of funding: Lien. Collection and assembly of data: Lis. Other: Discussions with Kathy Lofy, Michael Loehr, Erika Henry, Carina Elsenboss, Ashley Kelmore, Cindy Miron, Scott Foster, Nigel Turner, Danica Little, Barbara Jensen, Mary King, Grant Tietje, Aaron Resnick, Cory Fairbanks, and Jennifer Lord informed this project.
SUPPLEMENTARY MATERIAL
To view supplementary material for this article, please visit https:/doi.org/10.1017/dmp.2016.174
