In April 2015 an earthquake rocked Nepal, resulting in over 8500 fatalities and more than 17,000 injuries. 1 In the aftermath of such disasters, questions surrounding the quality of relief efforts arise and point to the need for performance measures that can provide relief workers and agencies with an evidence-based approach to ongoing and future responses.
Clear, standardized metrics that comprehensively assess the quality of disaster response can help to identify successes and challenges from past incidents and streamline future responses with the potential of improving outcomes for impacted communities. The United Nations has recognized the need for such performance measures. 2 Existing measures are typically agency specific, however, and are not standardized for application across responding entities. For example, the United Nations Office for Coordination of Humanitarian Affairs conducts several performance evaluations with the goal of “promoting transparency, accountability and learning through systematic and objective judgments about the relevance, efficiency, effectiveness and impact of humanitarian interventions.” 2 The US government’s National Response Framework mandates evaluation, generally in the form of after-action reports, with 2 separate tools available to evaluate US response: the Continuity Evaluation Tool, to assess federal planning, and the Continuity Assistance Tool, for nonfederal entities. 3 However, tools such as these are designed for interagency use rather than for use in guiding disaster response more widely. The World Health Organization recently released guidelines to establish minimum standards for expatriate teams and a registration process for responding teams.Reference Norton, von Schreeb and Aitken 4
Previous work has primarily focused on identifying disaster preparedness metrics through expert opinion. Existing preparedness guidelines largely consist of infrastructure and resource requirements for relief delivery, such as medical station capacity, equipment inventories, and triage guidelines.Reference Adini, Verbeek and Trapp 5 Past efforts toward developing response performance measures have focused on processes and outcomes, such as quality of resource coordination or post-disaster measures of morbidity, mortality, and quality of life.Reference Rådestad, Jirwe and Castrén 6 A concerted effort needs to be made to expand on existing measures and to develop standardized performance measures that can be used across responding groups and agencies to systematically evaluate the quality of disaster relief.
The evaluation of response is often limited by the lack of standardized indicators for measuring performance.Reference Agboola, Bernard and Savoia 7 Toolkits have been developed to address this but are often tailored for measuring performance in response exercises or infectious disease outbreaks.Reference Agboola, Bernard and Savoia 7 , Reference Belfroid, Hautvast and Hilbink 8 Without performance measures commonly agreed upon and used, responders risk repeating mistakes of the past: limited coordination, suboptimal utilization of aid, and limited access to lifesaving care.Reference Daftary, Cruz and Reaves 9
Review of past disaster responses can inform the construction of performance measures that ultimately will improve disaster relief.Reference Kun, Rose and Morris 10 With the practical need for standardized performance measures in mind, we sought to identify factors critical to response that may serve as the underpinnings of a disaster response performance measurement framework and the basis for developing such measures. To do this, we conducted a comprehensive review of published literature on 3 disasters: the 2003 Bam, Iran, earthquake; the 2004 Indian Ocean tsunami; and the 2010 Haiti earthquake. We chose these events because they garnered a rapid and significant response from national and international sources.Reference McCunn, Ashburn and Floyd 11 - Reference Bridgewater, Aspinall and Booth 13 Also, these events occurred within the last 15 years and thus reflect current disaster response capabilities and challenges. The prolonged responses to these events and resulting complexity revealed items essential to address in future disasters.
METHODS
We conducted a review of articles published on the 3 disasters. The review focused on any literature published from the time of each incident through 2015. Inclusion criteria developed a priori were (1) English-language articles and (2) pertaining to health-related disaster response. Reviews, original research, and practice pieces were included. We performed the review in May 2015 by use of the PubMed (National Library of Medicine, Bethesda, MD) search engine with separate searches for each event with use of the terms “disaster response” or “disaster relief” or “humanitarian response.” We excluded articles not related to these events. All retrieved titles and abstracts were reviewed by at least 1 of 3 authors (NP, SAB, MA), followed by a full-text review by 1 of 3 authors (NP, MA, OA). Data extraction was completed by one author (NP) and reviewed by an additional author for agreement (MA). We collected information on author, year, study title, journal, disaster type and date, population, methodology, aims of study, and summary of findings. Table 1 summarizes the results from our search strategy.
Search Strategy
RESULTS
We initially identified 710 articles and retained 496 after removing duplicates. Of these, 146 articles met the inclusion and exclusion criteria based on review of titles. We then reviewed the abstracts for these and excluded 22 additional articles, resulting in a final sample of 124 articles. Seventy-two articles pertained to the Haiti earthquake, 38 to the Indian Ocean tsunami, and 14 to the Bam earthquake (Table 1).
Based on a careful review of the 124 papers, we identified 5 categories of activities or themes, each with a number of key subcategories and specific types of activity. These categories were informed by recurrent barriers to disaster response and factors critical to success identified in the peer-reviewed literature related to the incidents under study. This article describes these findings and creates an organizing framework that can serve as the basis for development of standardized performance measures for disaster relief (Table 2).
Framework for Developing Performance Measures for Disaster Relief
a Pre-deployment in this context refers to screening before sending personnel or goods.
Personnel
All 3 disasters had reports of varying levels of preparation by teams. In multiple reports, team members did not receive physical or mental health screenings or screenings of their capabilities.Reference Bridgewater, Aspinall and Booth 13 - Reference Aitken, Leggat and Robertson 15 Few organizations had a ready list of volunteers with information regarding their medical or disaster training and their passport status. Even disaster-response-oriented nongovernmental organizations reported difficulty in deploying a sufficient number of well-trained staff.Reference Telford and Cosgrave 16 In Haiti, trauma surgeons, orthopedists, and neurosurgeons were deployed.Reference Millard 17 Further out from the event, generalists handled postoperative care and the population’s overall health care needs.Reference Jaffer, Campo and Gaski 18 As the weeks progressed, many obstetric and gynecologic issues were encountered.Reference Millard 17 In all 3 incidents, the need for general, primary care and nursing providers was greater than the need for specialty care, especially over the long term.
Ultra-specialized forces such as search and rescue teams were deployed in each disaster. At one point during the Haiti response, there were close to 2000 international search and rescue professionals on the ground saving 132 people.Reference Hopmeier, Pape and Paulison 19 Some of these teams did not arrive until several days after the quake, by which time their particular skill set was less useful than in the hours immediately after the disaster.Reference Peleg, Kreiss and Ash 20
The efficacy of responders was inhibited by language and cultural barriers. Translators were identified as a rate-limiting step in relief operations.Reference Lane 21
Supplies and Equipment
Issues surrounding supplies and equipment constituted a notable theme across incidents. Difficulty with tracking supplies and deploying unneeded or insufficient supplies were noted repeatedly. The majority of medical supplies brought were for the treatment of acute illness or injury. Nations were burdened by “the receipt of vast quantities of un-inventoried supplies, ranging from high to very low priority items, which created bottlenecks at the airport.”Reference Smith, Fink and Hansch 22 Medications became useless due to prescriptions being written in foreign languages or being expired.Reference Smith, Fink and Hansch 22 , Reference Roy 23 Screening methods before supplies were accepted would have allowed for the refusal of unusable items.Reference Bayard 24 When appropriate supplies were brought, they were quickly depleted across all incidents. The need for tetanus immunoglobulin was underestimated in both Haiti and the tsunamiReference Bridgewater, Aspinall and Booth 13 and providers did not have appropriate storage equipment.Reference Auerbach, Norris and Menon 25 Antibiotics were in short supply, whereas less useful medications such as amiodarone were at hand.Reference Bayard 24 In Haiti, most food and water supplies for patients were provided by family members.Reference Rosborough 26 An adequate assessment of appropriate food supplies should include consideration of the need for cooking, refrigeration, or disinfection. Consideration should be given to locally available foods.Reference Aitken, Leggat and Robertson 15 Supply rations such as the US military’s MREs (meals ready to eat) proved somewhat useful.Reference Aitken, Leggat and Robertson 15 However, in one example, meals containing pork were dropped into an area where religious beliefs prohibited their use. During the tsunami response, some areas received an overabundance of food with much wasted, whereas other regions went without.Reference Roy 23 , Reference Yamada, Gunatilake and Roytman 27 Across all 3 disasters, the needs of special populations were not adequately accounted for.
Transportation
Articles on all 3 disasters noted issues related to both the transport system and arrival hubs. Damage to infrastructure made existing roads impassable.Reference Yamada, Gunatilake and Roytman 27 The overwhelming burden of personnel and equipment arriving at international ports congested transport hubs and occupied ground transport. For example, the Haitian airport lacked coordination and staffing and could not handle the large amounts of incoming personnel and supplies.Reference Hopmeier, Pape and Paulison 19 , Reference Birnbaum 28 - Reference Bateman 30 Also, tsunami response accounts describe how equipment and supplies were held at ports of entry as the result of customs regulations that were not adapted to facilitate post-disaster receipt of goods.Reference De Silva, Chikersal and Snoad 31
The coordination of military forces, both international and local, was useful in distributing supplies.Reference Kapila, McGarry and Emerson 32 One report of the Indian Ocean tsunami response noted that US Navy Blackhawk helicopters cooperated with the US Agency for International Development in Indonesia to deliver supplies to devastated areas.Reference Joyce 33
Timeliness and Efficiency
In the first hours and days after the Haiti earthquake the greatest numbers of life-threatening injuries were encountered. International responders with the capabilities to care for these conditions mostly arrived after this 72-hour time period.Reference Millard 17 , Reference Lee, Low, Ng and Teo 34 Local residents recovered the greatest number of survivors and transported the majority of patients.Reference Telford and Cosgrave 16 In Bam, 85% to 95% of survivors were extricated within 24 hours.Reference Abolghasemi, Radfar and Khatami 12 The Iranian military was operational within hours of the event.Reference Abolghasemi, Radfar and Khatami 12 The majority of search and rescue teams arrived in Bam after most rescues had already taken place. In Haiti, as the response shifted from rescue to recovery, the initial responders found their capabilities insufficient to address current needs. Nations including the United States had deployed disaster medical assistance teams whose skills could address acute injury or illness or even exacerbations of chronic illness but did not contribute to infrastructure recovery efforts. This demonstrates the importance of coordinating the timing and nature of resource deployment with the post-disaster phase.
Interagency Coordination
Interagency coordination was problematic in all 3 responses. A significant number of the international responders did not obtain an accurate assessment of the needed resources in advance. Many relief organizations operated independently and conducted their own needs assessment. There was limited to no interagency agreement on which locales and populations would most benefit from aid or what type of aid was appropriate.Reference Zoraster 14 , Reference Telford and Cosgrave 16 , Reference Smith, Fink and Hansch 22 , Reference Auerbach, Norris and Menon 25 Moreover, inadequately prepared responders at times placed a burden on local resources.Reference Smith, Fink and Hansch 22 In one example, a province in Thailand reported having to coordinate the activities of 14 volunteer organizations. 35
RESEARCH AND POLICY IMPLICATIONS
The review presented here focused on observations from responses to 3 disasters that gave rise to a notional framework for developing standardized performance measures for disaster relief. Through a careful review of the peer-reviewed literature related to these incidents, we identified key barriers to disaster relief and factors that were noted to be important to successful response. We identified 5 categories of themes or activities that we considered to be key to these responses: (1) personnel, (2) equipment and supplies, (3) transportation, (4) timeliness and efficiency, and (5) interagency coordination. Under each one, again informed by our literature review, we identified key subcategories and specific activities that collectively can serve as a basis for developing standardized performance measures. The validity and comprehensiveness of these categories and items can be tested by applying them in a checklist fashion to recent or future disaster responses, after which standardized performance measures can be developed through a consensus process. While these may not be exhaustive, the framework proposed here lays the groundwork for further development of standardized performance measures for disaster response. Future work should apply a formal consensus process such as a Delphi process with a number of experts to determine which measures are best suited for standardization, which in turn can inform policy and practice guidelines for disaster relief.
CONCLUSIONS
Development and application of performance measures to evaluate past and future disaster response efforts can improve the quality and timeliness of such efforts. A comprehensive review of 3 disasters helped identify 5 key categories, subcategories, and specific activities that can serve as an organizational framework for developing standardized performance measures through a future consensus process.
Acknowledgments
Part of Dr. Puppala’s and Dr. Awad’s contributions to this work were conducted while they were Fellows in Emergency Medical Services and Disaster Medicine at the Ronald Regan Institute of Emergency Medicine at the George Washington University in Washington, DC.
