Criteria or Metrics Related to Organizational Characteristics

Nine PERRC studies examined the functions, activities, infrastructure, or other organizational and system features of PHEPR organizations/entities and the relationships of these characteristics to preparedness capacities and capabilities. Six studies assessed the characteristics of local health department (LHD) programs, while the other 3 examined the relationship between health departments and other PHEPR system entities (eg, health care delivery and community-based organizations).

Savoia and colleagues used data from the 2005 National Association of County and City Health Officials national survey of LHDs and applied 21 indicators to assess the association between health department characteristics and emergency preparedness (Table 1).Reference Savoia, Rodday and Stoto ^{7 ,} Reference Leep ⁸ Investigators found that LHDs serving larger populations are more likely to have staff, capacities, and to have conducted activities that support emergency response. For example, 60.5% of the LHDs serving the largest communities reported having a public information specialist, compared with only 3.2% of the smallest communities. After controlling for the size of the population served, health departments with a board of health had higher emergency preparedness capability based on 1 or more measures of emergency preparedness staffing, capacities, activities, and performance. LHDs having a board of health were more likely to have ensured review of their legal authorities, written or updated their emergency plans, conducted drills and exercises, or implemented PHEPR workforce training. The investigators suggest the potential benefit of merging small health departments into coalitions, as such coalitions, collectively, may be more likely to mobilize sufficient trained staff and capacities to ensure a successful emergency response.

Davis et al. developed and tested the Local Health Department Preparedness Capacities Assessment Survey (PCAS).Reference Davis, Mays and Bellamy ⁹ The survey instrument was fielded with the 85 North Carolina LHDs and 247 comparison LHDs. Interviews conducted as part of instrument development and validation “emphasized that emergency preparedness is a team effort in which different health department personnel have a different working knowledge of facets of preparedness.” Therefore, the authors recommended that the PCAS should be completed by multiple health department staff to increase the reliability of assessment findings.

Using the PCAS, Davis et al and Bevc et al. found decreases in surveillance and investigation and legal preparedness capacities over the 3 survey years, which they attributed to multiple years of funding cuts and job losses specifically for preparedness.Reference Davis, Bevc and Schenck ^{10 ,} Reference Bevc, Davis and Schenck ¹¹ Davis et al. found that LHDs that had participated in a performance improvement effort reported significantly greater surveillance and investigation, workforce and volunteer, communication, legal preparedness, exercise and emergency event, and corrective action capacities than comparison LHDs that had not participated in such performance improvement programs.Reference Davis, Bevc and Schenck ¹² The authors concluded that the PCAS can be useful to assess how well LHDs are performing preparedness capabilities and identify opportunities for program improvement.

Dalnoki-Veress and colleagues developed and applied an Analytical Hierarchical Process tool to identify potential gaps in LHD emergency preparedness planning during a radiological emergency scenario.Reference Dalnoki-Veress, McKallagat and Klebesadal ¹³ The tool allowed LHD officials to characterize the perceived relative importance of and challenges related to elements of preparedness. The interexchange of information between LHDs, the public, and state and federal partners, and ensuring the clarity of the information shared were public health officials’ dominant concerns when responding to the radiological emergency. The investigators suggest that this internet-based tool can be used to identify and rank gaps in emergency preparedness capacity and capability at the local level across a range of emergency scenarios.

Hunter et al. assessed the capabilities and functions activated by LHDs, emergency medical services, and hospitals during response to a statewide simulated improvised explosive device emergency.Reference Hunter, Yang and Petrie ¹⁴ Investigators identified 5 core capabilities commonly activated across all agencies, as well as 1 to 4 activities frequently activated by specific agencies. The majority of respondents reported that interorganizational information sharing was hampered by the complete or partial failure of communications equipment or systems. Investigators recommend further use of the novel epidemiological-concept-based exercise model for conducting exercise-based evaluation and illustrated, through its use, the need for early discussion among emergency response partners about expected and actual organizational roles, responsibilities, and resource capacities within the PHEPR system.

Agboola et al. developed an online toolkit for measuring performance in emergency response exercises using performance measures tested in over 60 emergency preparedness exercises.Reference Agboola, Bernard and Savoia ¹⁵ The toolkit was pilot tested in collaboration with 10 public health agencies and 4 health care agencies from 8 states. Thirteen (93%) exercise planners reported that the performance measures generated from the toolkit were appropriate for the creation of exercise evaluation forms to collect data useful for evaluating their organization’s performance during the exercise and preferred the toolkit to the Homeland Security Exercise Evaluation Program exercise evaluation guides. The investigators suggest that use of this toolkit could improve the quality, standardization, and comparability of the performance data collected during simulated emergencies.

Glik and colleagues examined health department partnerships with community-based organizations (CBOs) and faith-based organizations (FBOs), and developed and evaluated a tool LHDs can use to assess the frequency and nature of their collaborative activities with CBOs/FBOs.Reference Glik, Eisenman and Donatello ¹⁶ The Assessment for Disaster Engagement with Partners Tool (ADEPT) evaluates collaboration across 4 domains: communication outreach and disaster coordination (eg, if LHD participated in emergency response at CBO/FBO locations), resource mobilization for disasters (eg, if CBOs/FBOs provided services during a disaster), organizational capacity–building for disasters (eg, if LHD had trained CBO/FBO staff in emergency response), and partnership development and maintenance for disasters (eg, if LHD and CBOs/FBOs developed a community-wide disaster preparedness plan that had defined CBO/FBO responsibilities and roles). Investigators suggest that organizations with higher ADEPT scores have more active relationships with CBOs/FBOs, which may enhance community resilience. The use of this assessment tool may allow LHDs to assess how effective these partnerships are, identify areas for improvement, and examine response performance over time.

Criteria and Metrics Related to Emergency Response Performance

Twelve PERRC studies examined public health preparedness system data related to actual event responses, providing a view of response similarities and differences across events, both within and across emergency types. The majority of these investigations focused on either infectious disease events or natural disasters. Investigators employed both quantitative and qualitative methods to examine factors associated with response performance with the aim of improving future performance.

Hunter and colleagues evaluated the local public health system emergency response to the 2011 tsunami threat in California by surveying local public health, emergency medical services, and emergency management agencies using CDC PHEP capabilities framework.Reference Hunter, Crawley and Petrie ¹⁷ The distribution of roles most likely to be performed by emergency management agencies (eg, assuming lead incident management role to guide evacuation or community recovery) and public health agencies (surveillance and epidemiology, environmental health, and mental health/psychological support) in response to the tsunami were identified and can inform preparedness planning.

Using CDC’s 15 PHEP capabilities as a framework, Hunter and colleagues compared public health agencies response activities for 120 emergency incidents (event types: infectious diseases, severe weather, chemical, bioagent, radiation, mass casualty) to identify commonalities in the response patterns.Reference Hunter, Yang and Crawley ^{18 ,} Reference Hunter, Hunter and Yang ¹⁹ Their analyses revealed the ways in which the system’s partner organizations are adaptive to the nature of the threat resulting in differential activation of functions and partners based on the type of incident. For example, important response activities engaged in by public health departments during infectious disease events included public health surveillance and epidemiology, public health laboratory testing, nonpharmaceutical interventions, and information sharing. Responses vary greatly in duration (eg, a 5-hour response to a white powder incident versus the much longer response to the Deepwater Horizon oil spill). The authors suggest that use of this framework to identify and compare response activities by system partner and type of hazard may be useful to predict the resources and capabilities required to respond to future acute emergency events.

Stoto employed an evidence-based logic model of the PHEPR system to frame a case study of the public health response to the 2009 H1N1 pandemic (pH1N1), arguing that capability-based (as opposed to capacity-based) preparedness measures assessed by actual observation of a PHEP system in action are likely to be better indicators of how that system will perform in the future.Reference Stoto ²⁰ During the pH1N1, it was not simply laboratory capacity and availability of surveillance systems, but the actual performance of the laboratories and systems when called upon to detect and characterize a new pathogen and provide accurate situational awareness that demonstrates response effectiveness. Similarly, it is not simply possession of a vaccine or medication dispensing plan but health department collaboration with key public and private organizations in their community to successfully dispense vaccines or medications that is critical to timely prevention of diseases. The investigators suggest that qualitative assessment of the system capabilities of PHEP systems can be more useful than quantitative measures of static capacities, when the focus is real-world performance and quality improvement.

Zhang and colleagues studied performance of disease surveillance and notification systems in Mexico, the United States, Canada, and several other countries during the 2009 pH1N1.Reference Zhang, Lopez-Gatell and Alpuche-Aranda ²¹ Using a “critical events” approach, they developed a timeline of important epidemiological and public health response events during the disease outbreak, identified critical events that affected the timing of the response, and determined the factors that influenced the timing of the critical events. Critical events considered in the pH1N1 outbreak included timing of the detection of increases in number of cases associated with the disease outbreak, characterization of cases (eg, demographics and disease severity), identification and characterization of the causative agent, and the issuance of notifications about the outbreak. The authors noted that the global pH1N1 response illustrated that enhanced laboratory-based surveillance systems and improved global notification systems both contribute to earlier detection and characterization of emerging and reemerging diseases.

In another study that assessed surveillance systems in the context of the 2009 pH1N1 outbreak, Stoto reviewed the evidence regarding differential age-specific risks associated with pH1N1 infection and surveillance systems’ ability to accurately monitor H1N1 cases over time.Reference Stoto ²² The author noted that while surveillance and response associated with the H1N1 outbreak was acceptable, case finding may have varied by age and case reporting may have been biased due to changes in health-care seeking behaviors resulting from public awareness and concern. The author recommends the use of new surveillance methods in future health emergencies, including conducting telephone surveys of representative populations and seroprevalence surveys in well-defined population cohorts, to supplement surveillance data sources that may be influenced by healthcare-seeking behaviors.

To identify practices associated with more successful implementation of H1N1 vaccination in school-based and public clinics in 2009 by LHDs, Klaiman and colleagues used a “positive deviance” approach based upon the principle of identifying and learning from top performers.Reference Klaiman, O’Connell and Stoto ^{23 ,} Reference Klaiman, O’Connell and Stoto ²⁴ The investigators developed a process map to measure success, outlining important steps in implementing vaccination, and obtained opinions from LHD staff peers about performance among an initial sample of LHDs. Having an established LHD relationship with local school authorities, communicating effectively with parents, and ensuring clinic logistics allow for an easy flow of students were practices associated with more successful school-based vaccination campaigns. For public clinic-based vaccination, the findings highlight the importance of defining priority groups to receive vaccination, communicating with the public, maintaining adequate staffing, establishing community partnerships, and maintaining flexibility in implementation.

Piltch-Loeb and colleagues introduced the use of a “critical incident registry” (CIR) for PHEPR system evaluation to learn from actual emergency responses and identified some key characteristics for a CIR to be feasible and useful.Reference Piltch-Loeb, Kraemer and Nelson ²⁵ Establishing and using a CIR was considered helpful to identify and critically analyze rare events and responses to them and to drive learning and quality improvement. The investigators also developed a peer assessment approach in which health departments engage peers to analyze individual critical incidents and report on areas identified for improvement within and across PHEPR systems.Reference Piltch-Loeb, Nelson and Kraemer ²⁶ Based on the field tests and the views of the health professionals who participated in them, the authors suggest that this approach is feasible and leads to a more in-depth analysis of response activities than standard methods. Peer assessment and use of a CIR could be an alternative to standard emergency response quality improvement approaches such as use of After Action Reports (AARs) and improvement plans (IPs).

Following the 2009 pH1N1, Stoto and colleagues conducted a workshop to identify lessons about the PHEP system response performance gleaned from AARs and IPs.Reference Stoto, Nelson and Higdon ²⁷ The workshop participants included state health department and LHD personnel who had prepared the AARs and IPs for CDC review. The participants discussed the various barriers in the response activity, revealing potential lessons concerning situational awareness, resource mobilization, and communications. Many health departments identified the need to strengthen partnerships with health care providers, health care systems, pharmacies, schools, community organizations, insurers, and others to plan for and improve vaccine distribution.

To understand how lessons learned from the response to real incidents may be used to maximize knowledge management and quality improvement, Savoia et al. reviewed the national repository of AARs at the Lessons Learned Information Sharing website describing the public health system response to the pH1N1 and 3 hurricanes.Reference Savoia, Agboola and Biddinger ²⁸ During these responses, public health systems experienced challenges while implementing 3 PHEP capabilities: emergency public information and warning, information sharing, and emergency operations coordination. Recurring challenges were reported by multiple states and local public health agencies in response to these 2 types of incidents: difficulty in sharing information with external partners, obstacles to timely information release, confusion in roles and responsibilities, and lack of incident command system use. The findings from this study could be helpful in improving practices for measuring system level preparedness and improvement efforts.

Criteria and Metrics Related to Workforce Capacity or Capability

Eight PERRC studies focused on factors affecting emergency response workforce capacity, capability, and performance, ranging from psychosocial attributes to occupational and personnel-related resource considerations.

Barnett et al.⁽²⁰⁰⁹⁾ used the Extended Parallel Process Model (EPPM) to develop the Johns Hopkins~Public Health Infrastructure Response Survey Tool (JH~PHIRST) to assess perceptions and attitudes of public health workers in the context of various possible public health emergency scenarios.Reference Barnett, Balicer and Thompson ^{29 ,} Reference Witte ³⁰ Based on the responses to open-ended questions that assessed the perception of the threat and efficacy, respondents were grouped into 4 categories. Analysis of survey responses indicated that individuals reporting higher threat and efficacy levels were significantly associated with a positive willingness to respond (WTR). The JH~PHIRST tool may help public health agencies customize training programs to optimize emergency response attitudes in health departments.

In 3 other studies, Barnett et al. and Errett et al. applied the original or a modified EPPM-based JH~PHIRST questionnaire to examine predictors of response willingness among public health workers, evaluate a training curriculum to improve risk- and efficacy-related perceptions and WTR, and examine how psychological preparedness may be associated with WTR.Reference Barnett, Thompson and Errett ^{31 –} Reference Errett, Barnett and Thompson ³³ WTR was greater for naturally occurring emergency scenarios (weather-related and pandemic influenza) than for terrorism-related emergency scenarios (radiological “dirty” bomb and anthrax bioterrorism). These studies indicated that LHD workers’ WTR was scenario-specific, training may be needed to boost WTR, and those who perceived themselves as psychologically prepared for response were more WTR than others. The authors suggest that training and periodic assessment of WTR metrics could play an important role in improving PHEP program workforce capabilities.

To examine response workforce participation and performance, Savoia and colleagues developed and tested 2 questionnaires for Medical Reserve Corps (MRC) volunteers: one to assess performance and attitudes of volunteers, and another to assess barriers that prevented some volunteers from participating in influenza clinics.Reference Savoia, Massin‐Short and Higdon ³⁴ Factor analysis indicated that 20 items in the Volunteer Self-Assessment Questionnaire could be grouped into 5 factors. Results concerning volunteers’ performance were consistent with observations from LHD staff working with the volunteers and external evaluators observing the flu clinics’ activities. The investigators suggest that the toolkit can be used by MRC coordinators to monitor and modify their effectiveness in engaging volunteers in public health activities.

Horney and colleagues investigated the human, fiscal, informational, physical, and organizational resource capacities of Public Health Regional Surveillance Teams (PHRSTs) colocated in 7 LHDs in North Carolina and evaluated how PHRST capacities may be associated with LHD services they provided.Reference Horney, Markiewicz and Meyer ³⁵ The investigators found that most variation was seen in human resources capacity; variation in team composition was associated with differences in the support and services that PHRSTs provide to LHDs. For example, teams that had a doctor or an epidemiologist had larger budgets and provided more support and services, and teams with a pharmacist reported more partners.

Schuh et al. and Potter et al. studied how responding to an emergency affects the routine functions of health departments.Reference Potter, Schuh and Pomer ³⁷ The investigators developed and pilot tested the Adaptive Response Metric tool. The tool focuses on areas in need of continuity of operations planning, investments in technology, or exercising and training. The tool consists of 5 gradient stages reflecting the amount of change in health department functions in response to an emergency. It includes a weighting scheme based on a function’s respective proportion of the overall budget, enabling development of a weighted metric of overall impact on the health department. The metric was applied to the experience of 4 LHDs in California in the context of their response to the 2009 pH1N1 outbreak. The investigators were able to demonstrate, for example, how different functions of a health department are impacted at different times and to different degrees during an emergency response. The authors suggest that the Adaptive Response Metric protocol could provide information about patterns of variable response burden among LHD subunits that would be useful for improving preparedness planning and response management.