When hazards occur in vulnerable communities, they destroy infrastructure, 1 which leads to the loss of healthcare facilities and structuresReference Ardalan, Mowafi and Khoshgsabeghe 2 or disruption of their performance. 3 Disruption in the health system breaks down the health conditions in the disaster-affected areas. In addition to causing mortality and injuries, disasters also disrupt access to health services.Reference Djalali, Hosseinijenab and Hasani 4 As a result, these events pave the way for epidemics and outbreaks of communicable diseases.Reference Thomas, Hsu and Kim 5 - Reference Tohma, Suzuki and Otani 7 Many people and media believe that there is a high risk for an epidemic of communicable diseases after disasters,Reference Yan and Mei 8 , Reference Schneider, Tirado and Rereddy 9 although credible sources do not support this claim.Reference Yan and Mei 8 However, rumors and stories about outbreaks of communicable diseases in disasters horrify the disaster-affected people. The cholera epidemic after the 2010 Haiti earthquake, which affected more than 604 634 people and led to the hospitalization of 329 697 and the death of 7463, is evidence in this regard.Reference Brazilay, Schaad and Magloire 10
Therefore, it is critical that health systems set up a communicable diseases surveillance system (CDSS) immediately after a disaster for an appropriate and effective response.Reference Nelli, Kakar and Rahim Khan 11 - Reference Polonsky, Luquero and Francois 14 "A surveillance system is a systematic process for collecting, summarizing, analyzing and publishing data, and the results of its findings to stakeholders for the development of interventions." 15
It is so important that the Centers for Disease Control and Prevention (CDC) in the United States formed a disaster surveillance work group for this purpose in 2006.Reference Scnall, Wolkin and Noe 16 The Texas Department of State Health Services also developed a surveillance system (SS) for this purpose, named the Disaster-Related Mortality Surveillance, which was activated for the first time during Hurricane Ike.Reference Choudhary, Zane and Beasley 17
Like all programs, the performance of these systems should be assessed by experts. Their strengths and weaknesses should be extracted for the continuous development of the system. Performance assessment (PA) is a systematic process that monitors and assesses the achievement of goals of different parts of an organization or program and provides stakeholders with results.Reference Smith, Mossialos and Papanicolas 18 , Reference Tashobya, Da Silveira and Ssengooba 19 However, despite numerous experiences in setting up a SS for communicable diseases after disasters,Reference Magloire, Mung and Harris 20 , Reference Sabatinalli, Kakar and Rahim Khan 21 there are not any PA indicators. There is just a PA framework in which the indicators are not covered and only the features of a PA have been discussed. Although the CDC guideline is designed for health care systems in general and not specifically for communicable diseases and disasters, it is mainly used in the similar studies. 15
Therefore, considering the importance of PA for all programs, the problems caused by lack of PAs, and the knowledge gap in this field, this study was conducted to design PA indicators in four areas—input, process, product, and outcome—for CDSSs in disasters.
METHODS
A 3-phase mixed-method (quantitative/qualitative) design was used for the development of PA indicators of SSs in response to disasters. Of course, before starting this study, the researchers conducted a systematic literature review to identify the existing PA indicators of SSs.Reference Babaie, Ardalan and Vatandoost 22 Then a qualitative approach was used for selecting indicators and developing potentially related indicators. A semistructured questionnaire was used to conduct focus group discussions and interviews. A purposeful sampling method was used to select individuals to participate in the group discussions and interviews. Selection criteria for interviews were the experience of individuals in the Communicable Diseases Management program in previous disasters and their desire to participate in the interviews. They were experts in the field of health in disasters and emergencies, health services managers, and officials of Communicable Diseases Management Departments of the Islamic Republic of Iran Ministry of Health. Oral consent was obtained from the participants. The focus group discussions and interviews started with the main question, “What indicators do you think can be used to assess the performance of a CDSS in a disaster?” Other questions based on the given answers were asked to obtain more detailed and richer data. The discussions of each session were immediately extracted and analyzed. Gaps in the data from the group discussion sessions and interviews were addressed in subsequent meetings. The meetings continued until data saturation was achieved.
The content of the discussions was analyzed manually using the Strauss and Corbin model (1998)Reference Strauss and Corbin 23 because this study was a part of another study that was conducted by using the grounded theory method. (First, the texts of the interviews were divided into meaning units. Then the codes were extracted from the meaning units. Similar codes were put together and subcategories were extracted. Finally, by putting the codes together, the main categories were ultimately extracted.) The outcome of this phase was the development of SS PA indicators.
After the indicators were developed, their appropriateness was investigated. A Likert scale questionnaire was designed and its validity and reliability were tested. Then, each of the indicators was measured on the Likert scale from 1 to 5 (ie, completely appropriate to completely inappropriate). The questionnaire was distributed among the heads of Communicable Disease Management Departments in medical universities throughout a country-level seminar. The completed questionnaires were collected, and the results were analyzed using Statistical Package for the Social Sciences 20 (SPSS20). The indicators that were considered appropriate by more than 70% of the participants were retained, and the other indicators were excluded from the study.
Next, a panel of experts from the Ministry of Health and health care administration experts screened and finalized the set of PA indicators. Nine interested individuals were invited to form the panel, of whom 6 were present at the meeting (participation rate, 66.66%).
The Communicable Diseases Management Center officers in the East Azerbaijan Province of Iran were randomly selected to determine the importance of the indicators and to weight them. They were asked to compare elements in the model to assess the performance of CDSSs in response to disasters using the analytical hierarchy process introduced by Professor Thomas L. Saaty. The analytical hierarchy process, one of the best-known and most widely used multivariate decision-making techniques, uses paired comparisons with several options and criteria.Reference Giri and Nejadhashem 24 Weighting was done in 2 phases. In the first phase, 5 criteria that had to have an indicator, such as clarity, relevance, economic impact, adequacy, and ability to monitor, were weighted.
In the second phase, participants used the criteria to compare all the indicators in pairs and weight them. A questionnaire was designed with the software and distributed among the participants. The participants were taught how to complete the questionnaires and were asked to complete them slowly and patiently. Finally, the questionnaires were completed and collected within 3 weeks, and the data were entered into the software.
Results were analyzed using the Expert Choice software 11, which was designed for fuzzy computing and multiple-criteria techniques. Its indicators were designed for the analytic hierarchy process.
RESULTS
A total of 21 people were interviewed for extracting PA indicators of CDSSs, of whom 10 (47.62%) were male and 11 (52.38%) were female. Five (23.8%) of the participants of interviews were specialists or subspecialists, 10 (47.62%) were general practitioners, and 6 (28.57%) were experts or top experts. Of these, 2 (9.52%) had less than 10 years of job experience, 4 (19.05%) had between 10 and 15 years, 9 (42.86%) had between 16 and 20 years, 3 (14.29%) had 21 to 25 years, and 3 (14.29%) had a history of over 25 years. The mean length of the interviews was 53 minutes.
Results of Interview Analysis
A total of 363 codes were extracted, of which 91 repeated codes were excluded. The remaining codes were classified into 40 subcategories and 4 main categories: input, process, product, and outcome. Of the 40 subcategories, 13 (32.5%) were inputs, 11 (27.5%) were processes, 8 (20%) were products, and 8 (20%) were outcomes.
Results of the Second Phase
This phase was carried out with the participation of 49 heads of communicable disease management departments at the medical universities and schools throughout the country.
A total of 67 questionnaires were distributed, of which 39 were completed and returned (response rate, 58.21%). In addition, 19 questionnaires were distributed among officers in CDM, of which 10 completed questionnaires were returned to the researchers after frequent follow-up (response rate 52.63%). Of the 49 people who participated in this phase of the study, 1 was less than 30 years old (2.04%), 12 were between 30 and 40 (24.49%), 20 were between 40 and 50 (40.82%), 13 were older than 50 (26.53%), and 3 (6.12%) did not fill out this section so their ages remained unknown. Thirty-four (69.39%) of the participants were male and the rest were female.
In terms of job experience, 8 (16.33%) of the participants had between 5 and 10 years, 6 (12.24%) had between 10 and 15 years, 5 (10.2%) had between 15 and 20 years, 18 (36.73%) had between 20 and 25 years, and 9 (18.37%) had over 25 years. The job experience of 3 participants was unknown.
Of the 111 proposed indicators in this phase, 33 indicators (29.73%) did not meet the desired criteria, from the participants' viewpoints, and were excluded from the study. Finally, 77 of the indicators won the approval of at least 70% of the participants. Because of the high number of indicators and the difficulty of assessing them, the proposed indicators were revised again by an expert panel of researchers and specialists. Finally, 26 indicators (33.77%) were excluded and the remaining 51 entered the next phase (weighting). Of the final 51 indicators, 10 were related to input (19.61%), 17 to process (33.33%), 13 to product (25.49%), and 11 to outcome (21.57%).
Determining the Criteria for Weighting
The criteria were compared two by two to determine the weight. For example, the “economic” criterion was compared to the “clear” criterion to determine which was more important.
The calculation results are shown in Table 1.
Importance of the Criteria
Inconsistency: 0.03
It should be noted that, in cases where the inconsistency was higher than the standard (0.1), the participants were asked to revise their scoring. This continued until the acceptable inconsistency was achieved.
Weighting of the Indicators
At this phase, the priority of each indicator was judged with the criteria. The judgment was based on Saaty's 9-point scale. The results of this calculation were registered after the paired comparison matrix with the criteria. Inconsistency of items was calculated through normalization of row and column averages. The question answered in this section was, “Among the criteria of clarity, relevance, economic impact, adequacy, and ability to monitor, which one is preferred and to what extent?”
Determining the Final Weights of the Indicators
At the final stage of the weighting process, the results of the 2 previous phases were integrated and the final weights of items were calculated. The detailed results were presented in Table 2. The highest weight score, which was 49.1 out of 100, belonged to the indicators in input, and the scores of processes, products, and outcomes were 31.4, 12.7, and 6.8, respectively.
Weights of Indicators
Finally, the developed indicators for PA of CDSSs in disasters and their definitions were shown in Table 3.
Final List of Performance Assessment Indicators for Communicable Disease Surveillance Systems Designed for Use in Natural Disasters
DISCUSSION
The present study was designed and conducted to develop PA indicators based on the input, processes, products, and outcomes framework for CDSSs in response to natural disasters. The researchers interviewed 21 experts in related fields in 2 phases and extracted 51 indicators in the 4 areas of input, processes, products, and outcomes. The indicators were weighted using the analytical hierarchy process approach.
The absence of such indicators was noted in previous years by many experts.Reference Kouadio, Koffi and Attoh-Toure 25 - Reference Osman, Berbary and Sidani 27 The frameworks and indicators proposed and applied in previously published work had been designed for usual conditions (not disasters). It is obvious that the conditions following a disaster are different from the normal ones. So, the designed CDSS and associated processes will be different from those designed for normal conditions. 28 Thus, PA indicators of such systems should also be different from those that are appropriate under usual conditions. Although some of the indicators in previous studies and developed frameworks have been used in this study, these are the first indicators that have been developed specifically for the PA of CDSSs in disasters.
For example, all 9 attributes of the CDC guidelines, 15 31 of the World Health Organization (WHO)-proposed indicators in Communicable Diseases Surveillance and Response: A Guide to Monitoring and Evaluating, 28 and 7 of the proposed indicators in the “Surveillance System” chapter in Communicable Diseases Control in Emergencies: A Field Manual 29 were included in the PA indicators list.
In this study, the performance indicators for each of the 4 areas—input, processes, products, and outcomes—are presented. In Control of Infectious Diseases in Emergencies, CDSS includes 6 processes: diagnosis, reporting, examining, verification, analysis, and feedback. Although some indicators have been formulated for each of the 6 processes, these indicators are not used without supporting activities, such as input, education, communication, and supervision. Essentially, any shortcomings in this area will affect the performance. Thus, these indicators are included in the present PA.
However, these areas has been considered in previous studies of PA (in fields other than disaster). In various studies, some indicators have been developed for the PA of these areas and have been used practically.Reference Veillard, Champagne and Klazinga 30 - 32 The 4 indicators provided by WHO for assessment have been noted. 29
Other important points in this study are the weights of indicators and their importance in the PA. These indicators can be used not only to assess the performance of designed SSs in response to disasters, but also to rank them through the obtained indicator scores. In this study, the highest weight (49.1 out of 100) is devoted to the input indicators. Although they are not among the highest indicators proposed by the WHO, 41 out of the 95 indicators are in the input areas. 29 The results are similar in the outcomes. Outcome indicators in this study have the lowest role in PA (8.6 out of 100). Among the indicators proposed by the WHO, only one indicator is devoted to outcome as well. The indicators help identify the strengths and weaknesses of CDSSs in disasters. Obviously, the extraction of these cases is the responsibility of every manager and could possibly improve and enhance CDSSs in the future.
CONCLUSION
Natural hazards and the disasters that they create in communities have always been an inevitable part of human life. They will also continue to occur in the future. The common effects of these disasters are the destruction of infrastructure including health facilities. This paves the way for the occurrence and prevalence of communicable diseases that may intensify the side effects of the disasters as they themselves become the secondary disasters. To manage these conditions, the first and the most important step is to design and set up a CDSS. The performance of a SS, like that of any other program, should be assessed for efficiency and effectiveness. In the existing literature, attention is paid to the lack of such indicators, but few practical actions have been taken. To overcome this deficiency, this study recommends 51 indicators for PA of CDSS. The researchers believe that these indicators will be effective and useful in the PA of SSs. Although there may be shortcomings and problems with these indicators, it is hoped that researchers around the world will overcome these weaknesses by testing the indicators in the field.
Acknowledgements
This study was part of a PhD thesis/dissertation supported by Tehran University of Medical Sciences.
Funding
This study has been funded and supported by I.R. Iran's National Institute of Health Research (NIHR), Tehran University of Medical Sciences.
