Introduction
Extreme weather events are expected to affect the health status of millions of people.1 Small islands (like Puerto Rico) are especially vulnerable to these effects of climate change.1 It has been posited that climate change adaptation needs to become part and parcel of comprehensive disaster risk management.Reference O’Brien, O’Keefe, Rose and Wisner2,Reference Thomalla, Downing, Spanger-Siegfried, Han and Rockstrom3 Reducing disaster risk requires long-term engagement and is largely a task for local actors (with support from national organizations).Reference O’Brien, O’Keefe, Rose and Wisner2,Reference Schipper and Pelling4 The National Academy of Sciences (Washington, DC USA) has recognized community engagement as a critical component of disaster-related decision making, planning, and risk reduction measures that will promote healthy outcomes.5,6
However, for communities to protect themselves from disaster-related hazards, they must first be able to assess their own risks, plan their own interventions, and then measure their own effectiveness. The most effective disease interventions are known to prevent human exposure to the health hazard. Population protection measures (PPMs; ie, shelter-in-place and evacuation) are an effective means for preventing exposure (and therefore adverse health effects) due to environmental hazards. But (as the COVID19 crisis has exemplified), in order to become applicable in the real-world, PPM interventions that restrict or direct population movements must also be developed through community participation.Reference Wilson7 Public health has a long history of community-based participatory research and interventions for managing health risks due to a range of hazardous exposures.Reference Wilson7
However, a research-practice gap exists across all fields of public health and medical practice, including disaster-related health science:Reference Brownson, Eyler, Harris, Moore and Tabak8,Reference Kirsch and Keim9 “Our inability or unwillingness to apply what is known to improve health results in significant health deficits and persistent inequalities.”Reference Brownson, Eyler, Harris, Moore and Tabak8 In fact, there is no evidence of significant change in US disaster-related mortality rates over the past 50 years (1969-2018) despite billions of dollars in public outlays.Reference Keim, Kirsch and Alleyne10 Health inequity has persisted for decades among US minorities affected by disasters.Reference Environmental Disasters and Frumkin11 Recent events have raised public concern regarding systemic inequality during public health emergencies, such as Hurricane Maria (2017) and the COVID19 pandemic.Reference Kishore, Mahmud and Kiang12,Reference Webb Hooper, Napoles and Pérez-Stable13
Public health has moved forward in recent years to bridge the research-practice gap. Evidence-based public health calls for knowledge of the determinants and consequences of disease, as well as the efficacy, effectiveness, and costs of interventions.Reference Vanagas, Bala and Lhachimi14,Reference Brownson, Fielding and Maylahn15 Many health departments are now pioneering a new “Public Health 3.0” model in which leaders partner across multiple sectors and are leveraging actionable data and clear metrics to address the social, environmental, and economic determinants of health and inequity.Reference DeSalvo, Wang, Harris, Auerbach, Koo and O’Carroll16
And while effectiveness research is commonplace in other areas of public health, there have been few studies of intervention effectiveness related to disasters (eg, hurricanes). Despite repeated urging of public health leadership, disaster epidemiology remains chiefly concerned with etiological, rather than evaluative, hypotheses.Reference Lurie, Manolio, Patterson, Collins and Frieden17,18
This study applied a mixed methodology to test the efficacy of an innovative public health intervention for engaging high-risk island communities to rapidly write their own population protection plans for hurricanes. Efficacy is here measured in terms of the international standard for “usability,” defined as “the extent to which a system can be used by specified users to achieve specified goals with effectiveness, efficiency, satisfaction, and freedom from risk.”Reference Ågerfalk and Eriksson19,20 Efficacy is a measure of the performance of an intervention under controlled circumstances (as compared to effectiveness, which is a measure of performance under “real-world” conditions).Reference Victora, Habicht and Bryce21 Measures of efficacy and effectiveness describe the quality of outcomes, as compared to efficiency that describes the quality of performance (usually as a rate).
Satisfaction is a user-focused measure of quality. In simple terms, it involves “ensuring value” for the project participant. Participant value is a function of the relative risk of engagement (ie, economic, social, environmental, and health) compared to the benefit of engagement.Reference Ågerfalk and Eriksson19 Community engagement is therefore influenced by perceptions of risk associated with the intervention.
Methods
Description of the Planning Process
The planning method used for this intervention was based upon a previously described process developed at the Centers for Disease Control and Prevention (CDC; Atlanta, Georgia USA) that applies an Operational, Objective-based, Consensus-based, Capability-based, and Compliant, or “O2C3,” approach for plan writing and a Strategy, Objective, Activity, Responsibility, or “SOAR,” structure for organizing plan content.Reference Keim22,Reference Keim23 This same methodology has demonstrated plausibility and reports of cross-cultural transferability among academic and governmental settings in over 200 jurisdictions world-wide.Reference Bazeyo, Mayega and Orach24–Reference Yi, Za and Fan26
The “O2C3” planning is a facilitated process of group plan-writing that is: objective-based (O); written at an operational level of detail (O); consensus-based (C); capability-based (C); and compliant (C) with local and national cultural norms, policies, and regulations.Reference Keim22,Reference Keim23
The “SOAR” acronym is used to describe the organizational structure (ie, data schema) for information stored in the plan. The achievement of each protection plan capability is described in a cascading level of detail starting from the (S) strategic goal (S); to the operational objectives (O) that accomplish that goal; to the activities that accomplish each objective (A); and parties responsible for performance of each activity (R).Reference Keim22
The population protection plan is organized in a hierarchical fashion starting with 12 core capabilities. Table 1 lists these core capabilities that were identified for PPMs based upon the US Department of Homeland Security (Washington, DC USA) “hub and spoke model” for evacuation.27
Core Capabilities Included in the Population Protection Plan
For each of the 12 capabilities, each element of SOAR was proposed, read aloud, discussed, and then decided by group consensus. The SOAR elements were transcribed into spreadsheet format using Excel (Microsoft 365; Microsoft Corp.; Redmond, Washington USA). The spreadsheets were created in both English and Spanish, and then both versions were simultaneously shared on a large projection screen for the entire group to view together.
This process was facilitated in a plenary setting using a consensus-based approach for decision making.
Project Design
During a one-year term, the authors implemented a pilot project in two municipalities of Puerto Rico to test an innovative, community-based approach for disaster risk reduction. Municipality A (census = 50,000) is located on the coastline where it is at high risk for hurricane-related wind, coastal storm surge, and riverine flooding. Municipality B (census = 25,000) is located in the mountains, and is at high risk for hurricane-related wind, landslides, and riverine flooding.
The intervention is designed to prevent human exposures to hurricane hazards (eg, wind, landslides, floods, and storm surge) BEFORE they occur. The project, known as “Plan to Protect,” focused on early advance evacuation of families with members that were children (<5 years old) and/or elderly (>65 years old). Table 2 provides a logic model for the project.
Logic Model for “Plan to Protect” Pilot Project
Abbreviations: PPP, population protection plan; O2C3, Operational, Objective-based, Consensus-based, Capability-based, Compliant; TTX, tabletop exercise.
To initiate the project, implementing team members convened in Puerto Rico to develop 13 project objectives that would achieve the five project goals stated in the award. Using a facilitated, consensus-based approach (based upon the O2C3 method), the team then drafted 72 activities that would accomplish these 13 objectives.Reference Keim22,Reference Keim23 The group also assigned primary responsibility and a deadline for completion for each activity.
The strategy included four community convenings. During the first convening, the “Risk Communication Meeting” (RCM) participants learned about hurricane-related health risks specific to their community. Next, the team used the O2C3 planning method to facilitate a community-level “Strategic Planning Workshop” (SPW) to generate a strategic-level plan (ie, one containing only S - strategic goals and O - operational objectives).
Once a strategic plan was developed, the study then compared two separate methods (eg, exercise and plan-writing) to add tactical-level of detail (ie, A - activities and R - responsible parties) to the plan content. This was compared for two separate methods: (1) a modified “Tabletop Exercise” (TTX); and (2) a “Tactical Planning Workshop” (TPW). All workshops were assisted by simultaneous, English-Spanish translation. All written materials were provided in both English and Spanish versions.
Planning workshops (SPW and TPW) used a standardized international O2C3 model for capability-based planning according to the SOAR organizational format developed by the CDC.Reference Keim22,Reference Keim23 The TTX used a standardized US national model, America’s PrepareAthon Facilitators and Evaluators Handbook for Whole Community Tabletop Exercises: Hurricane, developed by the US Federal Emergency Management Agency (FEMA; Washington, DC USA).Reference Emergency Management Agency28
Study Design
The study measured the efficacy of a community-based planning intervention for achieving the intended short-term outputs and outcomes. Efficacy was measured in terms of ISO 9241-11, an international standard for usability that includes effectiveness, efficiency, user satisfaction, and “freedom from risk” among users.20 Table 3 provides a detailed listing of indicators used for measuring efficacy in terms of effectiveness, efficiency, satisfaction, and degree of engagement (used here as an indicator of “freedom from risk”).
Indicators Used for Measuring Intervention Usability, in Terms of Effectiveness, Efficiency, Satisfaction, and Community Engagement
Data related to the following sources were entered into and analyzed using Excel.
Review of Work Plan Documentation—All project activities were documented according to a time-bound workplan developed by team consensus. Times and dates of actual completion were documented for all objectives and activities and then compared to deadlines.
Review of Workshop Outputs—Workshop outputs and timelines were reviewed for effectiveness (in terms of completeness and accuracy in details) and efficiency (in terms of outputs/hour). Community plan outputs from the three interactive workshops (SPW, TTX, and TPW) were analyzed.
The number of plan objectives written and number of hours worked were documented for the SPW. Completeness was calculated as the percentage of plan objectives (or activities) written by the community out of all objectives (or activities) that were planned to be completed during that specific workshop. Presence or absence of a designated responsible party was also recorded for each activity of the plan. Completeness was calculated as a function of the percentage of entries that included designation of party responsible for completion of each activity.
Efficiency rates for strategic-level planning were calculated for SPW outputs as the number of strategic and operational-level objectives completed per hour. Efficiency rates for the TTX and TPW were calculated as the number of tactical-level activities completed per hour. Responses were also categorized and scored according to the level of detail provided in the description of each activity. Four team members served as independent reviewers. Blank entries were scored as zero; one-word entries were scored as one; unintelligible phrases were scored as two; intelligible phrases were scored as three; and intelligible full sentences were scored as four. Descriptive statistics including means and confidence intervals were calculated for the reviewers’ mean scores of tactical-level detail. A threshold for “adequacy” of tactical-level detail was defined as any score ≥3.0 where on the whole, entries are, at minimum, deemed “intelligible” by four separate reviewers. Plan activities were compared for TTX and TPWs plans developed in both communities. The same reviewers provided scoring for the same sets of variables. All reviewers were blinded to the type of workshop (eg, TTX or TPW) and the community of origin for the data they reviewed. In addition, the number of plan activities without a corresponding assignment of responsible party was tallied from protection plans that resulted from both TTXs and the TPWs. Percentages were calculated for any plan activities lacking an assignment of responsibility out of all activities listed during each TTX and TPW.
Survey of Workshop Participants—Participants were surveyed anonymously by paper questionnaire. At the end of each of four community convenings, respondents were asked to use a five-point Likert scale for assessing their own knowledge, skills, abilities (KSAs), attitudes, and beliefs regarding the project, in general, and that specific workshop (eg, RCM, SPW, TTX, or TPW). Percentages of participant agreement with a five-point Likert scale were used to assess qualitative statements about the four community convenings. Responses to qualitative statements were classified according to the following three categories: “agree,” “neutral,” and “disagree.” Participant selections of “strongly disagree” and “disagree” on the Likert scale were combined into one category entitled “disagree.” Participant selections of “strongly agree” and “agree” on the Likert scale were combined into one category entitled “agree.” No changes were made to selections for “neutral.”
Review of Workshop Participant Roster—A roster of workshop participants was maintained to include participant’s name and affiliation. Attendance was recorded for each of the workshops, according to one of the following affiliations: federal, territorial, municipal, nongovernmental organizations (NGO), and members of the general community.
Focus Group Interviews—An independent evaluator (not involved in the planning or exercises) conducted a one-hour-long, in-person focus group interview with six self-selected individuals from each target community. Community members were asked a series of open-ended questions related to their KSAs, attitudes, and beliefs related to the workshops. Participants received a $25 gift card as incentive.
The same evaluator also conducted one telephone focus group interview with local project staff members. Three individuals participated. Locally based project staff members were asked open-ended questions related to their KSAs, attitudes, and beliefs related to the workshops.
Ethics
This study was ethically reviewed, approved, and performed under a sole source contract funded by the CDC, administered by the US Association of State and Territorial Health Officials (ASTHO). The CDC reviewed and approved the work plan and publication for this project prior to implementation.
Results
Results from Review of Project Work Plan
A comparison of the final project documentation according to the workplan timeline revealed the successful on-time completion of 13 (100%) of the 13 project objectives and 71 (99%) of the 72 project activities within 12 months, as intended.
Results from Review of Project Outputs and Outcomes
Both communities (comprising an average of 35 persons representing 19 agencies and the public) demonstrated the ability to draft and exercise their own hurricane population protection plan (averaging 60 pages long), within a duration of 14.5 total hours from start to finish.
Table 4 provides a summary of the results from a review of outputs and timelines (in terms of the degree of plan completeness; rate of plan-writing; and degree of tactical detail in the plan) for each of the three workshops held in each of two target communities.
Outputs and Outcomes of the Six Project Workshops that Performed Planning
Abbreviations: SPW, Strategic Planning Workshop; TPW, Tactical Planning Workshop; TTX, tabletop exercise.
Effectiveness—The SPWs were 100% effective in both communities for the writing of all objectives identified (by both communities) as necessary for effective implementation of the 12 core capabilities included in each plan.
The TPWs were 100% effective in both communities for the writing of all activities identified (by both communities) as necessary for effective implementation of the 75+ objectives included in each plan.
During the TTXs, Communities A and B effectively completed 86% and 81% (respectively) of the activities associated with objectives previously developed in the SPWs.
The overall mean reviewer scores of tactical-level detail in plans resulting from TTXs were nearly identical at an estimated 2.51 (CI, 2.40-2.61) and 2.50 (CI, 2.38-2.61) for Community A and B, respectively. Reviewers’ estimates of tactical-level detail were also notably similar for plans resulting from the TPWs (3.56 [CI, 3.51-3.59] and 3.59 [CI, 3.54-3.63]) for Community A and B, respectively. This similarity may imply a certain degree of reproducibility. In addition, on face value, TPW levels of tactical detail appeared one-third higher as compared to that generated in a TTX. However, the probability distribution of this relatively small data set did appear to have a slightly negative skew (range -1.1 to +0.5) with a kurtosis of (-0.9 - +0.7), thus hampering accurate parametric comparison between TTX and TPWs.
Finally, the mean percentage of plan activities that included no assignment of tactical-level responsibility (scored yes or no by one blinded reviewer) was identified as 78% and 80% for Communities A and B, respectively, during TTXs. In comparison, the mean percentage of plan activities that included any assignment of tactical-level responsibility (scored yes or no by the same reviewer) was estimated as 0% for both communities during TPWs. This similarity may again imply a certain degree of reproducibility and efficiency in the TPW model as compared to TTX (especially since this involved a simple binary [present/absent] tally of the entire dataset, as compared to a subjectively-scored sample).
Efficiency—Overall, the time-efficiency of strategic planning (objectives only) also appeared reproducible and nearly the same for SPWs in both communities (14.0 objectives/hour and 14.4 objectives/hour, respectively). On average, there were 21% less plan activities written during the TTX as compared to the TPW. However, the duration of the TTXs was also 20% less than the TPWs. Thus, the time-efficiency (in terms of number of activities completed per hour) was nearly equal for TTX and TPW in both communities. The TTX participants appeared able to write plan activities at the same rate as they did during the TPW, but with less detail. However, the non-normal sample distribution did not allow for an accurate comparison of this difference.
Results of Survey Related to Workshop Participant KSAs and Attitudes
Table 5 lists the percentages for participant agreement (using Likert scales) to assess qualitative statements about the four community convenings (n = 198, response rate = 71% of all attendees). Nearly all (>90%) of participants agreed that: (1) workshops met their intended objectives; (2) the content was relevant to work, community, and family; (3) the material was clearly presented and easy to understand; (4) the exercise was well organized; (5) participant comments were welcomed; and (6) participants had a good understanding and were supportive of the project. Eighty percent of community members agreed that they had demonstrated the ability to implement a health risk reduction plan with community consensus.
Percentage of Participant Agreement Using Likert Scales to Assess Qualitative Statements about the Four Community Convenings
Results from Review of Participant Roster
Table 6 provides a summary of participation in the four community convenings, according to affiliation of the stakeholder.
Participation in Workshops (A and B), According to Stakeholder Type
Abbreviation: NGO, nongovernmental organization.
Convenience Sample Data from Focus Group Interviews, Summarized by Indicators of Usability
Effectiveness—Strengths: Participants mentioned the benefit of participating in a rapid, collaborative planning process and were eager to try the approach in their work. Participants appreciated that the planning process was broken down into clear steps. Weaknesses: Participants requested an additional community meeting to decide how to take action based upon the risks. Implementing staff expressed a desire for more training and practice before initiating the project.
Efficiency—Strengths: Participants appreciated the benefit of having everyone in one place at one time to “speak the same language” about disasters. Participants liked when the facilitator had time to explain concepts, but also appreciated when they prevented them from dwelling on any one topic for too long. Weaknesses: While simultaneous translation was readily available, it was suggested that use of bilingual co-facilitators would encourage people to speak up more. Participants also suggested providing for better control of off-topic conversations.
Community Satisfaction—Strengths: The participants in both communities found the workshops valuable. Participants mentioned the benefit of creating new and strengthening existing partnerships within the community. Weaknesses: Participants expressed doubt that the plans would be approved and adopted by the government and worried that elected and appointed leaders were too transient to be relied upon for continuity. They expressed a desire for stronger community leadership.
Community Engagement—Strengths: Participants expressed an appreciation to engage with community residents and learn more about their needs and concerns. Likewise, some felt community engagement provided an opportunity for agencies to demonstrate to the public that they are working to protect them from harm. Weaknesses: At times, participants recognized that important agencies and organizations were missing from the conversation. Participants also voiced the need for more engagement from municipal, central, and federal agencies.
Discussion
Population Protection Measures
Population protection measures have become more important to emergency management operations in recent decades.27 In March 2020, 42 US states were under shelter-in-place orders: a total of 308 million people, or 94% of the US population.29 Recent events further underscore the need for community engagement in support of public health interventions that involve PPMs. And yet, there is little scientific evidence available related to the effectiveness or efficiency of public health interventions that would assist communities to perform these PPMs.
The challenge is to tailor the PPM to best address a variety of factors, including a community’s demographics, location, infrastructure, resources, authorities, and decision-making processes.27 For this to occur, the respective roles and responsibilities of individuals, families, governments, NGOs, and the private sector must be negotiated and mutually agreed upon in advance. Population protection plans must also be written at an operational level of detail to capture the unique demographics, capabilities, and risks of the community. Public health is uniquely positioned at the local level to facilitate this process with its long history of “town-hall style” community engagement, especially related to reducing hazardous exposures.Reference Wilson7,29
Community Engagement
Community engagement is a useful approach for obtaining public input about policy decisions that require difficult choices among competing values (eg, PPMs). According to the National Academies of Science, “Although average citizens may lack the expertise to comment on technical issues, they are very capable of deliberating on the values underlying public policy decisions in crisis situations.”6
Besides effectiveness and efficiency, the usability of any public health intervention (including PPMs) is also influenced by the degree of satisfaction and “freedom from risk” perceived among participants.Reference Ågerfalk and Eriksson19,20 To be usable, public health interventions related to PPMs must not only be effective and efficient, they must also be perceived by the community to have a value that outweighs potential social, economic, environmental, or health risk.
While the risk of engagement was indeed negligible related to economic, health, or environmental threats, some participants in this study (particularly those in positions of authority or responsibility) may have perceived a social risk related to their participation during the workshops. Social risks may include the potential to appear that one (or one’s agency) is: (1) poorly informed; (2) poorly suited to perform a task for which one is responsible; (3) overly cooperative with a rival group; or (4) unwilling to commit responsibility for activities identified in the plan.
It appears that the most usable PPMs engender a high degree of community satisfaction with the engagement process, as well as the intervention content. Satisfaction is a participant-focused measure of value. Participant value is a function of the relative risk compared to the benefit of engagement in the workshops.Reference Ågerfalk and Eriksson19,20 Thus, the usability of an intervention is also dependent upon a perceived “freedom from risk” on behalf of the participants. The most usable PPMs take into consideration economic, social, environmental, and health risks that may influence the full range of public and private community members (perhaps a good lesson for the current challenges of COVID19 public health interventions related to PPM [ie, shelter-in-place]).
Finally, to be a fair broker of the public trust, community-based interventions must encourage the collaboration of professional partners while remaining committed to a “partnership of equals” and producing outcomes of value to the entire community (not just the “official” organizations).Reference Wilson7,Reference Cobb30 The use of consensus-based decision making during the planning process allowed for equitable partnerships during this process – a key element of community satisfaction and risk perception.
Consensus aims to be inclusive, participatory, cooperative, and egalitarian.Reference Keim23 Consensus-based decision making not only seeks agreement, but also to resolve or mitigate the objections of the minority to achieve the most agreeable decision. Consensus-based decision making serves to incorporate the socio-economic and cultural input of community in all aspects of the process, encouraging stakeholdership and commitment. The process results in equitable partnerships that require the sharing of power, resources, credit, results, and knowledge, as well as a reciprocal appreciation of each partner’s knowledge and skills (as was noted in the results of the focus group analysis).
Limitations of Study
Descriptive studies typically lack causality and are prone to bias. However, the validity of such findings can be greatly enhanced by studies such as this that suggest the intervention is having an important effect and perhaps warranting of more detailed observational trials.Reference Victora, Habicht and Bryce21,Reference Rychetnik, Frommer, Hawe and Shiell31
This study was designed to measure efficacy which, by definition, occurs under well-controlled conditions. And while documentation of the efficacy, reproducibility, and transferability of this process are important prerequisites for implementation and dissemination of this intervention, it will be necessary to study this intervention in a larger number of applications over time in order to ascertain its effectiveness in terms of public health practice and health outcomes.
Conclusions
Frontline communities have successfully demonstrated their ability to understand the environmental health hazards in their own community, rapidly write consensus-based plans for PPMs, participate in an objective-based TTX, and perform these activities in a bi-lingual setting. This intervention appears to be efficacious for public use in the rapid development of community-based PPMs. More study is needed to ascertain impact on practice and health outcomes over the medium and long term.
Conflicts of interest/funding/disclaimer
This study was performed under a grant from the Centers for Disease Control and Prevention (CDC; Atlanta, Georgia USA), administered by the Association of State and Territorial Health Officers. The CDC reviewed and approved the work plan and publication for this project prior to implementation. The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the CDC. The authors declare no conflicts of interest.
