Literature Search

In 2016, Heagele documented a lack of a valid and reliable HEP instrument in the public health and emergency management research community. ^{Reference Heagele1} For the current study, a review of the existing literature was conducted again to determine if a valid and reliable instrument emerged since that publication. The search was delimited to peer-reviewed academic journal articles published between January 2015 and January 2019, with abstracts available, and in the English language. Simultaneous searches of the MEDLINE Complete (EBSCO, Ipswich, MA), CINAHL Complete (EBSCO, Ipswich, MA), APA PsycINFO (APA, Washington, DC), Social Sciences Full Text (EBSCO, Ipswich, MA), and Health and Psychosocial Instruments (EBSCO, Ipswich, MA), databases were conducted in December 2018. Eligibility criteria included original research studies measuring HEP of individuals or households (i.e., not communities, responders, or health care organizations) with a survey, tool, scale, instrument, or checklist. The following key word combinations were used: “household OR citizen OR emergency OR disaster AND preparedness,” “individual AND preparedness,” “individual AND disaster OR emergency AND preparedness OR planning,” “disaster OR emergency AND supplies OR kit,” and “hazard AND preparedness OR readiness.” After removing duplicates, 38 articles met eligibility criteria from the 1587 initial results. Another 14 articles were identified either through citation trails or the social networking website for researchers, ResearchGate.

Of the 52 articles suggested, only 22 articles provided brief instrument question development information. ^{Reference Bagwell, Liggin, Thompson, Lyle, Anthony and Baltz2-Reference Nakai, Tsukasaki and Kyota4,Reference McNeill, Alfred, Mastel-Smith, Fountain and MacClements7,Reference Bodas, Siman-Tov, Kreitler and Peleg8,Reference Wirtz and Rohrbeck15-Reference Gargano, Caramanica, Sisco, Brackbill and Stellman17,Reference McNeill, Killian, Moon, Way and Garrison19-Reference Tam, Huang and Chan32} However, none of these articles specifically focused on instrument development. A few researchers analyzed data from nationally representative surveys that included questions on HEP, such as the Behavioral Risk Factor Surveillance System Questionnaire, ^{Reference Smith and Notaro6} Health Retirement Survey, ^{Reference Cox and Kim5,Reference Killian, Moon, McNeill, Garrison and Moxley12} General Social Survey, ^{Reference Nukpezah and Soujaa11} and the Public Readiness Index’s Readiness Quotient. ^{Reference Najafi, Ardalan, Akbarisari, Noorbala and Elmi33,Reference Ranjbar, Soleimani, Sedghpour, Shahboulaghi, Paton and Noroozi34} For the researchers who created their own instruments, the Federal Emergency Management Agency (FEMA) recommendations were the most popular definition of HEP used to inspire the instrument questions. The authors of 21 articles provided their instrument questions in their articles. ^{Reference Bagwell, Liggin, Thompson, Lyle, Anthony and Baltz2,Reference Cox and Kim5,Reference McNeill, Alfred, Mastel-Smith, Fountain and MacClements7,Reference Killian, Moon, McNeill, Garrison and Moxley12,Reference Howe14-Reference Gargano, Caramanica, Sisco, Brackbill and Stellman17,Reference McNeill, Killian, Moon, Way and Garrison19,Reference Jackson20,Reference Murakami, Siktel, Lucido, Winchester and Harbord24,Reference Stewart, Grahmann, Fillmore and Benson25,Reference Thomas, Sobelson and Wigington28,Reference Toor, Burke and Demeter29,Reference Najafi, Ardalan, Akbarisari, Noorbala and Elmi33,Reference Smith and Notaro35-Reference Nash40} Some authors asked broad questions about HEP, such as “did you assemble a disaster supply kit?,” whereas others asked about the presence of specific supply kit items such as flashlights, radios, food, and water. Only 8 articles contained information on pilot-testing of the instrument. ^{Reference Bagwell, Liggin, Thompson, Lyle, Anthony and Baltz2,Reference Nakai, Tsukasaki and Kyota4,Reference Bodas, Siman-Tov, Kreitler and Peleg8,Reference Annis, Jacoby and DeMers22,Reference Tam, Huang and Chan32,Reference Ranjbar, Soleimani, Sedghpour, Shahboulaghi, Paton and Noroozi34,Reference Adame and Miller41,Reference Phibbs, Good, Severinsen, Woodbury and Williamson42} A total of 15 articles included instrument reliability data, ^{Reference Cox and Kim5,Reference Bodas, Siman-Tov, Kreitler and Peleg8,Reference Nukpezah and Soujaa11,Reference Killian, Moon, McNeill, Garrison and Moxley12,Reference Kim and Zakour16,Reference Jackson20,Reference Kleier, Krause and Ogilby23,Reference Yong, Lemyre, Pinsent and Krewski26,Reference Chilton and Alfred27,Reference Ranjbar, Soleimani, Sedghpour, Shahboulaghi, Paton and Noroozi34,Reference Wei and Lindell36-Reference Welton-Mitchell, James, Khanal and James38,Reference Adame and Miller41,Reference Najafi, Ardalan, Akbarisari, Noorbala and Elmi43} and 4 articles contained validity data. ^{Reference Bodas, Siman-Tov, Kreitler and Peleg8,Reference Killian, Moon, McNeill, Garrison and Moxley12,Reference Wei and Lindell36,Reference Notebaert, Clarke and MacLeod44} The remainder of the articles provided no instrument development data or reliability and validity information.

Researchers accrue evidence for validity by examining the scores resulting from an instrument that is used for a specific purpose, with a specified group of respondents, under a certain set of conditions. ^{Reference Waltz, Strickland and Lenz18} Validity must be assessed each time an instrument is used in order to determine if validity generalization can be made for various populations under study. ^{Reference Waltz, Strickland and Lenz18}

Rigorous research designs need to start with instruments that are psychometrically sound; information about the psychometric properties should be obtained and evaluated before an instrument is selected for use. ^{Reference Waltz, Strickland and Lenz18,Reference Devon, Block and Moyle-Wright45} Without access to publications focusing on instrument development, it is impossible for researchers to evaluate the psychometric properties of current HEP instruments. Unfortunately, no official publications detailing original instrument development of any HEP instruments were available for review. As such, it was impossible to discern the conceptual basis that guided the instrument development, critique the methods used to generate questions, and compare the questions to an original concept definition.

After the literature search and review of the evidence, the investigators agreed that a valid and reliable HEP instrument had not emerged. The decision to proceed with the development of a HEP instrument via a Delphi study was made.

Existing Instruments: Criterion Validity

Criterion validity is established when a newly developed instrument has an empirical association with a commonly used, benchmark instrument. ^{Reference DeVellis46} Lacking a benchmark instrument resulted in lack of criterion validity for HEP measurement, but many of the instruments were inspired by the FEMA recommendations. This lends support for criterion validity of these HEP instruments. Data were collected for 2 studies with iterative versions of the Preparedness Assessment (PA), an instrument that was based on the “Ready” ⁴⁷ and the Texas “Ready or Not?” ⁴⁸ campaign materials. ^{Reference McNeill, Alfred, Mastel-Smith, Fountain and MacClements7,Reference McNeill, Killian, Moon, Way and Garrison19} The initial PA was a dichotomous survey where respondents were asked to answer ‘yes’ if they had an item, or ‘no’ if they did not. Examples of questions posed to participants include, ‘‘Have you ever had any emergency preparedness education?’’ and ‘‘Do you have a first aid kit?’’ Participants reporting they possessed the item scored 1 (yes); while those reporting they did not scored 0 (no). These question response scores were summed to create a preparedness index with a possible range of 0 to 28. ^{Reference McNeill, Killian, Moon, Way and Garrison19}

The PA questions were compared to questions of 22 other instruments from the literature search; the similarities and differences between what was included on the instruments and how questions were worded were examined. The investigators, including the developers of the PA, agreed to adapt the PA for the Delphi study. This new instrument was named the Household Emergency Preparedness Instrument (HEPI). The goal was to create an all-hazards, comprehensive, easily understandable HEP instrument to present to the disaster research community.

Expert Panels: The Delphi Technique

The Delphi technique was used to establish face and content validity and evaluate cultural bias of the HEPI. The online Delphi technique is a widely accepted survey method used to generate group consensus and develop measurement instruments from geographically dispersed expert participants spanning a wide range of disciplines and roles. ^{Reference Waltz, Strickland and Lenz18,Reference Hsu and Sandford49,Reference Parks, d’Angelo and Gunashekar50}

There is no consensus on what constitutes an expert, but Delphi participants should be primary stakeholders with various interests related to the target issue, have somewhat related backgrounds and experiences, and include representation from all relevant social and cultural groups. ^{Reference Waltz, Strickland and Lenz18,Reference Hsu and Sandford49,Reference Parks, d’Angelo and Gunashekar50} While there is also no general agreement on the number of participants required for a consensus study, Delphi studies are commonly completed with under 100 participants. ^{Reference Hsu and Sandford49,Reference Parks, d’Angelo and Gunashekar50} The Delphi technique is recommended when: (a) the participants do not have a history of adequate communication; (b) input is needed from more individuals than can effectively interact in a face-to-face exchange; (c) time and cost make frequent group meetings infeasible, and; (d) participant anonymity is needed to reduce the effects of dominant individuals (i.e., the bandwagon effect). ^{Reference Waltz, Strickland and Lenz18,Reference Hsu and Sandford49,Reference Parks, d’Angelo and Gunashekar50} In addition to being fast, inexpensive, and versatile, the strengths of the Delphi technique include: opinions of many experts can be condensed into a few precise statements; participants can respond at their own convenience; and participant anonymity limits the potential influence of other experts to conform to social norms, organizational culture, or standing within a profession. ^{Reference Waltz, Strickland and Lenz18,Reference Hsu and Sandford49,Reference Parks, d’Angelo and Gunashekar50}

Recruitment

Delphi participants are purposefully selected for their expertise. ^{Reference Hsu and Sandford49} Interdisciplinary colleagues of the investigators with disaster response or disaster research experience, and the corresponding authors of the articles found in the literature search were e-mailed an IRB-approved recruitment and consent script, along with the link to the survey. The snowball sampling technique where participants suggest other potential participants was also utilized. In addition, the World Association of Disaster and Emergency Medicine (WADEM) sent the recruitment message via e-mail to members on the investigators’ behalf. Inclusion criteria were English-reading adults aged 21 years or older, reflecting the highest age of consent worldwide. ⁵¹

Participants came from 36 low, middle, and high income countries on 5 different continents (Figure 1) and represented disaster response experts from the disciplines of public health, emergency management, medicine, nursing, pharmacology, firefighting, emergency medical services, social work, psychology, sociology, epidemiology, bio-ethics, hospitality, national security, environmental management, geography, public administration, humanitarian relief, education, and business. Participants were asked to self-identify as either an expert or a community stakeholder. Table 1 describes the sample demographic characteristics.

FIGURE 1

TABLE 1

Sample Demographics

Data Collection

While there are no universal guidelines for how to carry out a Delphi study, ^{Reference Parks, d’Angelo and Gunashekar50} common data collection procedures include asking participants to complete multiple iterations of a structured, formal, electronic questionnaire designed to elicit opinions and exchange feedback. All rounds of the Delphi were exchanged via Qualtrics online survey software, version August 2019 (Qualtrics, Provo, UT, USA).

Typically 3 rounds are needed to reach a consensus. ^{Reference Hsu and Sandford49} Through their endorsement of the degree to which specific HEP actions and items are essential, participants compared their concept definition of HEP to the HEPI questions. They also evaluated question clarity and conciseness and pointed out ways of measuring the phenomenon that may have been excluded. ^{Reference DeVellis46} Qualitative and statistical analyses were used by the investigators to modify subsequent iterations of the questionnaire until consensus was reached. ^{Reference Waltz, Strickland and Lenz18,Reference Hsu and Sandford49} Once the responses were tabulated and summarized, they were returned to the participants. Additional feedback was sought on the questions that did not achieve consensus. Participants were given 2 weeks to participate in each round. Prospective rounds were limited to the group of participants who responded to the questionnaire in the first round.

Data Analysis

Descriptive statistics were used to summarize the collective judgements of the participants (using the Statistical Package for the Social Sciences for Windows, version 25.0 software; International Business Machines Corporation, Armonk, NY, USA). ^{Reference Waltz, Strickland and Lenz18,Reference Hsu and Sandford49} In the first round, consensus on a HEPI question was achieved when 80% of the participants’ votes fell within 2 categories based off a 5 point scale. ^{Reference Hsu and Sandford49} A HEPI question was kept when 80% or more of the participants marked the question as important or essential. Likewise, questions were discarded if 80% of the responses fell in 2 categories on the lower end of the rating scale.

The open-ended responses were analyzed with qualitative content analysis via NVivo software for Mac version 11.4.1 (QSR International Pty Ltd, Doncaster, Victoria, Australia). Content analysis aims to attain a condensed description of a phenomenon by categorizing written data so that it can be counted. ^{Reference Elo and Kyngäs52} Participants’ responses were coded and placed into categories. These data were primarily used to edit existing HEPI questions.