Introduction
Background
Natural hazards (NHs) are an integral part of nature. They have existed from the inception of creation and have affected human beings adversely since the dawn of their existence.Reference Babaie, Ardalan, Vatandoost, Goya and Akbarisari 1 Recently, their frequency has increased due to factors such as climate change.Reference Coppola 2 In the last decade (2003-2012), over 2,130 NH-triggered disasters occurred across the world with approximately 3,867 million victims.Reference Smith, Wasiak, Sen, Archer and Burkle 3 In addition, NHs are expected to increase in frequency and intensity in the next decades.Reference Guha-Sapir, Hoyois and Below 4 , Reference Phalkey, Dash, Mukhopadhyay, Runge-Ranzinger and Marx 5
These catastrophic events destroy infrastructures and lead to uncountable deaths and injuries, damage to livelihoods, collapsing health facilities, disabling services, and displacement of people.Reference Guha-Sapir, Vos, Below and Ponserre 6 , Reference Ardalan, Mowafi and Khoshsabeghe 7 In such conditions, the risk factors for adverse health impacts are increased.Reference Scnall, Wolkin and Noe 8 One of the most concerning issues in disasters is the outbreak (OB) of communicable diseases (CDs).Reference Isidore, Kouadio, Aljunid, Kamigaki, Karen and Oshitani 9 Although it is usually over estimated,Reference Scnall, Wolkin and Noe 8 , Reference Isidore, Kouadio, Aljunid, Kamigaki, Karen and Oshitani 9 this concern is reasonable because many OBs and epidemics have been reported following past disasters.Reference Scnall, Wolkin and Noe 8 , 10 , Reference Qadri, Khan and Faruque 11
Thus, an integral part of every public health response to disasters is communicable disease management (CDM). 12 - Reference Yan and Mei 14 Management of CDs in disasters, like other components of public health response, requires a large number of resources such as manpower, equipment, facilities, and vehicles. Performance assessment (PA) of every organization and program is a critical managerial principal. It is important to stakeholders such as service providers, fund raisers, donors, and decision makers.Reference Tohma, Suzuki and Otani 15 , Reference Ten Asbroek, Arahi, Geelhoed, Custers, Delnoij and Klazinga 16 In addition, measuring and improving the performance of the programs is required by international organizations.Reference Tohma, Suzuki and Otani 15 , Reference Ten Asbroek, Arahi, Geelhoed, Custers, Delnoij and Klazinga 16 A lack of PA is a significant weak point in every program and organization.Reference Groene, Klazinga, Kazandjian, Lombrail and Bartels 17
However, there is a vast amount of literature on PAs, and it has received significant attention in the last decades. Many PA frameworks are developed to assess the performance of organizations and health systems,Reference Sheikhzadeh 18 and some researchers have tried to assess the performance of health systems in the disaster response.Reference Christopher, Murray and Frenk 19 , 20 There is still considerable disagreement with regard to the application of PA frameworks in disasters. A key problem with most of the literature on PA frameworks is that they do not consider the characteristics of the disasters. According to this review, there is a lack of knowledge and methodology in PAs in the disaster management field.Reference Lennquist and Hodgetts 21 Thus, it is emphasized by many researchers and research centers.Reference Babaie, Ardalan, Vatandoost, Goya and Akbarisari 22 , Reference Altevogt, Pope, Hill and Shine 23 Regarding the mentioned reasons, the objective of this study was to develop a PA framework for CDM in NH-induced disasters.
Definitions of Used Key Terms
Smith et al define the term “performance assessment” as a “systematic process that seeks to monitor, evaluate, and communicate the extent to which various aspects of a system meet its key objectives.”Reference Lazar, Caglious and Gebbie 24
In the literature, a “communicable disease” is defined “as an illness that arises from transmission of an infectious agent or its toxic product from an infected person, animal, or reservoir to a susceptible host.”Reference Smith, Mossialos and Papanicolas 25
“Communicable diseases management” is the process that relies on prevention, timely detection, treatment, and monitoring of its progress in an accepted scientific protocol.
Methods
This study was designed in order to develop a PA framework and indicators for assessing the performance of CDM in response to natural disasters. The study was conducted in the Islamic Republic of Iran in 2014. Iran is a disaster-prone country 26 , Reference Ardalan, Kandi and Talebian 27 that experiences disasters every year and has always been affected by their adverse health effects. This study was conducted in four stages.
Systematic Literature Review
In order to identify the existing CDM PA frameworks in disasters and the related indicators, a comprehensive systematic literature review (SLR) was conducted. A SLR is a logical stage in such studies. The full methodology of the conducted SLR has been discussed previously in another paper by the authors of this study.Reference Lennquist and Hodgetts 21
Developing a PA Framework for CDM in Disasters and its Indicators
In the second stage, qualitative approach and grounded theory were used as the methodology for developing the most appropriate PA framework and its indicators. Focus group discussions (FGDs) and semi-structured, deep interviews were organized with participation of CDM office experts. These participants purposively were selected from different offices of the Centers for Disease Control and Prevention (CDC) of the Iranian Ministry of Health (MOH; Qods, Iran) who were experienced in response to disasters.
The interviews took 45-60 minutes for each person and FGDs took 1.0-1.5 hours. The interviews and FGDs were done in the participants’ offices. Verbal consent was obtained, and all participants were informed that they could refuse to participate or withdraw from the interviews and FGDs at any time. The data collection process continued until data saturation was met. All interviews and FGDs were recorded and transcribed by investigators. The interviews were analyzed using the Colaizzi’s descriptive method. Then, meaning units, codes, categories, and concepts were extracted.Reference Khankeh, Fallahi, Ranjbar and Ahmadi 28 A PA framework was developed through this process. Some techniques, including member check, peer check, researcher capability, and persistent observation, were used for conformability of the data and expert check was used for trustworthiness of the study (investigator triangulation).Reference Khankeh, Fallahi, Ranjbar and Ahmadi 28
Selection of Appropriate Indicators
After the PA framework was developed and its indicators were extracted, a survey was conducted for selecting the most appropriate indices. A questionnaire using the Likert scale was designed and its validity and reliability were checked. To assess the validity, the questionnaires were forwarded to nine experts and their comments were included. For assessing the reliability of the questionnaires, they were forwarded twice, at an interval of two weeks, to 15 experts; then, they were collected and its validity was calculated. The researcher-developed questionnaires were distributed in medical universities, managers of CDC offices, and surveillance experts (all of them) who participated in a 3-day national workshop. After two days, the completed questionnaires were collected and their data were entered into SPSS20 (IBM Corp.; Armonk, New York USA) after coding. If more than 70% of the study participants marked the appropriateness of indicators lower than average, they were excluded from the study. The remaining indicators were included in the next stage.
Indicators Weighting
In this stage, an analytic hierarchy process (AHP) was used for weighting of the developed indicators. An AHP is a multi-criteria decision making method developed by T. SaatyReference Khankeh, Mohamadi and Ahmadi 29 that has been used in many environments and numerous studiesReference Saaty 30 - Reference Chung, Chen, Chang, Chang and Lai 32 when researchers are faced with many criteria.Reference Peng, Li and Zhang 33 The CDC workers of Tabriz University of Medical Sciences (Tabriz, Iran) were selected at random for conducting this process. For this purpose, a questionnaire was designed and its application was explained to the participants. The completed questionnaires were included and analyzed using the Expert Choice Software11 (Expert Choice Inc.; Arlington, Virginia USA) which is used for decision making based on AHP.
Results
Results of SLR
The SLR results are published in another article written by the authors of this study,Reference Lennquist and Hodgetts 21 so they are not described in this report entirely. According to the results of SLR, the researchers concluded there was a lack of knowledge in this area. There was no specific PA framework for CDM PA in response to disasters. However, the CDC's updated guideline for the Public Health Surveillance System (SS) is the most frequently used guideline in this area.Reference Ping, Tan and Yang 34
Results for Developing a PA Framework and its Indicators in CDM Disaster Response
Two FGDs and 14 semi-structured and deep interviews were conducted with purposively selected experts. In all, 22 people participated in FGDs and interviews. Fifty percent of them were women, six (27.7%) were medical specialists in the field of infectious diseases, 10 (45.45%) were general physicians, and six (27.27%) were officers in CDM. Two of them (9.09%) had 5-10 years, three (13.63%) had 10-15 years, eight (36.36%) had 15-20 years, five (22.7%) had 20-25 years, and four (18.18%) had more than 25 years of experience in the field of CDM in disasters.
The FGDs and interviews were analyzed in two steps. In the first step, they were analyzed for developing a PA framework. The analysis of the transcripts of the interviews and FGDs resulted in 634 codes. The similar codes were identified, and the repeated ones were then removed. Finally, the similar codes were categorized into 33 subcategories. The similar subcategories were merged and four main categories were finally emerged, including input, process, product, and outcomes (IPPO).
Table 1 demonstrates the detailed results of the coding and categorizing process. Thus, the main framework for PA in CDM in disasters is found within the IPPO framework.
Results of Coding Process and Categorizing of Codes
Abbreviations: CD, communicable disease; IPPO, inputs, process, products, and outcomes; SS, surveillance system.
Input—The participants of FGDs and interviews believed that input plays an important role in CDM. Therefore, in any PA framework, the input shall be considered the first step. For this category, nine subcategories were extracted from interviews, including manpower, equipment and supplies, guidelines, protocols and their applications, transportation, medications and biologic products, providing services location, financing, and preparedness plans.
Process—Process was the second category extracted from coding of the interviews. It encompassed 11 subcategories: importance of process, participation of non-health sectors, rapid health needs assessment, environmental health, developing SS, immunization, vector control, CD control, outbreak management, risk communication, and coordination. The product could be reached through the process of using the input. The process should be designed and implemented well.
Product —Another part of the PA framework that emerged from the analyzing of participants’ interviews was product. This category included seven subcategories: immunization coverage, CD control, SS, outbreak management, environmental health, incidence and prevalence rate, and risk communication.
Outcomes—The last component of the developed PA framework for CDM in disasters was the outcomes component. It included five subcategories: the satisfaction rate of the community and of the stakeholders, decreased rates in morbidity hospitalizations, disability, and mortality.
In the second step of this stage, which was similar to the first step, the interviews were analyzed for CDM PA indicators. The analysis of the transcripts of the interviews and FGDs resulted in 753 codes. The similar codes were identified and then the repeated ones were removed. Finally, the similar codes were categorized into 28 subcategories. Similar subcategories were merged and the PA indicators of each of the four main categories (IPPO) finally were developed.
The appropriateness of the developed indicators was evaluated with a valid and reliable researcher-developed questionnaire in a Likert scale. The participants of this stage were heads of the CD departments of medical universities from all over Iran, as well as the CDC of MOH officers. Seventy-six questionnaires were distributed and 49 were returned (response rate=64.47%).
Of 49 people who returned the questionnaires, one was less than 30 years old (2.2%), 12 were between 30-40 years (24.5%), 20 were between 40-50 years (40.8%), 13 were more than 50 years old (26.53%), and three did not respond to these items. Thirty-four were male (73.9%). Regarding job experience in the field of CDM, eight out of the 49 (10.2%) had 5-10 years, six (12.44%) had 10-15 years, five (10.2%) had 15-20 years, 18 (36.73%) had 20-25 years, and nine (18.36%) had more than 25 years. Their education levels were fellowship (n=1; 2.2%), general practitioner (n=10; 20.4%), graduate student (n=11; 22.44%), Bachelor of Science (n=19; 38.77%), and Associate of Science (n=5; 10.2%); three persons did not specify their educational level.
Of 167 indicators that were proposed to be appropriate evaluations, 49 could not get the agreement of 70% of participants and were deleted. Finally, 118 indicators were selected to be used in PAs. Because there were many PA indicators and their assessment was difficult, an expert panel was conducted to select the most appropriate ones. Fifteen experts purposively were invited to the expert panel, of whom 11 (participation rate=73.33%) participated in the panel. According to the panel, eight indicators were deleted and 75 indicators were integrated with others; finally, 40 indicators were selected for the PA of CDM in disasters. These indicators are presented in Table 2.
(continued). Communicable Diseases Management Performance Assessment Indicators
Abbreviations: CDM, communicable disease management; ICS, incident command system; PPV, positive predictive value; RC, risk communication; RHNA, rapid health needs assessment; SS, surveillance system.
Weighting of Indicators
In the final stage of this study, the AHP approach was used for weighting the developed indicators. The East Azerbaijan Health Center (East Azerbaijan, Iran) CDM department experts were selected randomly for weighting. The Expert Choice Software was used for weighting.
Eight out of 11 experts (72.72%) participated in the weighting process. The weighting process was done in two stages. First, five criteria that an indicator should have, including clear, relevant, adequate, economic, and monitorable, were weighted (Figure 1). Then, the sub-indicators (alternatives) were weighted according to the mentioned five criteria. The detailed results of the weighting process are presented in Table 3.
Priority of Selected Criteria with Respect to the Goal (Weighting of CDM PA Criteria in Disasters). Abbreviations: CDM, communicable disease management; PA, performance assessment.
Final Weights of CDM PA Indicators in Response to Disasters
Abbreviations: CDM, communicable disease management; ICS, incident command system; PA, performance assessment; PPV, positive predictive value; RHNA, rapid health needs assessment; SS, surveillance system.
Discussion
In this study, the authors attempted to develop a PA framework and its indicators for the PA of CDM in response to disasters. The PA in response to disasters has been a research priority in recent years.Reference Babaie, Ardalan, Vatandoost, Goya and Akbarisari 22 , Reference Altevogt, Pope, Hill and Shine 23 A collection of methods, including SLR, qualitative approach, expert panel, and AHP process, have been used to achieve the study objective. Finally, the IPPO framework and 40 PA indicators have been developed for the PA of CDM in response to disasters.
The lack of such a framework and its indicators has been emphasized in previous studies and it has been a research priority for many related research centers.
The methodology was consistent with previously published articles individually.Reference Ten Asbroek, Arahi, Geelhoed, Custers, Delnoij and Klazinga 16 , 35 However, in this study, all of them were used integrally in a logical process.
In this study, the IPPO PA framework was extracted from interviews. Products, outcomes, and impacts are usually assessed in the PA, but there is also evidence for using “inputs and process” as parts of PA; for example, in the National Health Service (NHS; United Kingdom) PA framework, “process” is one of the dimensions.Reference Tashobya, Silveria, Ssengooba and Nabyonga-Orem 36 In the Performance Assessment Tool for Hospitals (PATH) project, “input” is a sub-dimension of “efficiency.” There is another dimension entitled “staff orientation,” and its definition is related to “input.” In addition, “safety” is another criterion that can be considered as an input.Reference Phalkey, Dash, Mukhopadhyay, Runge-Ranzinger and Marx 5 In the World Health Organization (WHO; Geneva, Switzerland) framework for assessing the performance of health systems, “resource generation” (human, physical, and knowledge) is one of the most important factors that influence the performance of health systems.Reference Chang, Lin and Northcott 37 In Ghana's holistic assessment of the health sector, “inputs” are more highlighted and some indicators are considered for measuring it.Reference Veillard, Champagne, Klazinga, Kazan djian, Arah and Guisset 38 In the PA framework that has been developed for Dutch Health System (Netherlands), “finances” (input), and “delivery of high quality care” are the two important components.Reference Tohma, Suzuki and Otani 15 In the WHO field guide for malaria epidemic assessment and reporting, “process indicators” are an important section of PA. 39
Some of developed indicators in this study have been used in other PA frameworks. Of 40 indicators that were developed in this study, nine (22.5%) are in common with CDC updated guidelinesReference Ping, Tan and Yang 34 and 13 (32.5%) are similar to the WHO. As mentioned earlier, the CDC guideline has been developed for Public Health SS and WHO indicators have been developed for monitoring and evaluating CD SS. Only 21 indicators (52.5%) are different and are specific for disasters.
Weighting of PA indicators was done for the first time in the current study in two stages. In the first stages, the criteria of clear, relevant, economic, adequate, and monitored were prioritized, and in the second stage, the alternatives were weighted. The WHO emphasizes that an indicator should be clear and consistent. The criteria from the first stage are considered in result-based management handbooks. 40
The weight of each indicator shows its importance in the PA. Economy had the highest weighted score among the criteria. As for alternative indicators, the inputs indicators had the highest weight (0.444) followed by process (0.390), products (0.091), and outcomes (0.075). With the use of the weighted scores, the performance of CDM can be evaluated in different disasters.
Limitations
Limitations of the SLR were discussed in another article written by the authors of this study.Reference Lennquist and Hodgetts 21 Another limitation was setting the agreement of 70% of participants for selecting appropriate indicators may have caused deletion of some indicators. However, a threshold had to be selected, which can per se cause limitation.
Because a high number of indices were finally selected, weighting of them was a boring task. The authors tried to reduce its effect by giving more time to the weighting process, but it was not clear if it had affected the decision.
Conclusion
Despite the importance of PA in any program, there is a lack of any existing PA framework for CDM PA in disasters. Thus, in this study, a PA framework (IPPO framework) and the related indicators were developed and weighted for the PA of CDM in disasters through a series of methods in four stages. According to the study participants, it is an appropriate framework and applicable in disasters. Although it was developed and utilized for the first time, the authors believe this framework and its indicators are applicable in the field.
Acknowledgements
The authors wish to acknowledge the grammatical revision of the manuscript in English facilitated by the Consultation Unit, Office of Publications and Scientometrics, Tehran University of Medical Sciences (Tehran, Iran).
