Introduction
Interventions are designed to change an element of the current status of a community, or a Societal System or its components. All evaluations have an ultimate goal of decreasing the risk that a hazard will morph into an event that results in a disaster. Interventions are specific to the needs generated by actual or anticipated changes in level(s) of function(s). Each intervention provided before, during, or following a disaster or other emergency should be evaluated for both its value and the process involved in its implementation. However, there is a paucity of evaluations of disaster-related health interventions in the peer-reviewed literature. Of the more than 1,000 references reviewed in a comprehensive review of the health aspects of the 2004 Indian Ocean Earthquake and Tsunami, it was not possible to identify the interventions provided, much less their outcomes and impacts on the affected populations.Reference Birnbaum, Daily, O’Rourke and Kushner 1 In a review of 701 papers published from 2009-2014 in the journals Prehospital and Disaster Medicine and Disaster Medicine and Public Health Preparedness, only 20.3% were related to interventions; of these, 23.8% were related to relief responses, 74.2% to risk-reduction interventions, and two papers (2.0%) described recovery responses. 2 Of those papers related to risk-reduction interventions, most described educational programs without documenting their impact on risk reduction. The few reports that do exist in the scientific, peer-reviewed literature lack any consistent common structure, and therefore, are difficult to compare and catalogue.
Evaluations of disaster-related activities are limited in that it currently is impossible or difficult to conduct randomized, controlled trials (RCTs); most evaluations of interventions are limited to before-after designs, case-controlled studies, and surveys. Although conducting controlled studies (experimental, controlled) is possible for evaluating some recovery and risk-reduction interventions, it is unlikely that randomization studies will be possible. However, the scientific study of all disaster-related interventions is essential to the development of standards and best practices. Repeated confirmation of findings provides increasingly relevant evidence with good external validity.
The recent actions by the Inter-Agency Standing Committee of the United Nations (Geneva, Switzerland; Transformative Agenda and the Sendai Framework of 2015) stress the need for accountability for disaster-related interventions.Reference Adibhatia, Dudek, Ramsel-Miller and Birnbaum 3 , 4 Accountability is the responsibility to provide evidence to stakeholders and funders about the effectiveness and efficiency of programs. 5
Interventional Disaster Research
In relation to disasters and other emergencies, as well as to risk-reduction and capacity building, interventions are undertaken for the purpose of changing the current status (levels of functions) of a community or components of a community. Thus, the over-riding goal of any intervention is to provide the goods, services, and other resources that are required to meet the needs of a Societal System in order to change the current status in a positive direction. In order to develop standards that apply to disasters and emergencies, and to the best practices for a compromised function in a specific setting that are based on these standards, the interventions provided and what difference(s) they made in the pre-intervention functional status must be identified.
Interventional disaster research consists of evaluations of the interventions provided during the relief or recovery phase of a disaster or for reducing the risk that a hazard may produce a disaster (risk reduction; development). To evaluate means to assess or appraise; to ascertain or fix a value to; to examine and judge carefully. 6 Thus, evaluation is the process used to place a value on something. Interventional disaster studies consist of evaluations that aim to determine the effects, outcomes, costs, impacts (including benefits), and processes of interventions provided during any phase of a disaster, or as part of development (risk-reduction), and to determine the value of the intervention for the community.
The evaluation of any specific disaster-related intervention is not part of epidemiological disaster research. Although findings from epidemiological disaster studies may establish the context and needs for future interventions or detect the impact of risk-reduction interventions implemented prior to the onset of the event, epidemiological disaster studies do not include the evaluation of specific interventions. Epidemiological studies may be used to validate the impact of interventions provided as risk reduction prior to the onset of the precipitating event. In the context of disasters, evaluation is used to: (1) identify the effects, efficacy, and effectiveness in attaining the objectives, costs, efficiencies, outcomes, and impacts of interventions; (2) place a value of the intervention for the community; and/or (3) identify the strengths and weaknesses of the process involved in providing disaster-related interventions in order to improve the effectiveness and benefits at a lower cost the next time the intervention is considered for use.
Ineffective or inappropriate disaster-related interventions not only waste valuable resources but also may have negative effects on the stricken population.Reference Pickett 7 Even with the best of intentions, not all interventions have a positive impact. However, as disaster-related interventions rarely are evaluated and reported, 2 many such responses are repeated, and will continue to be repeated. The greatest need in Disaster Health is evidence to develop and support standards and best practices. Such evidence only can come from publication of structured and systematic studies and evaluating the interventions provided before, during, or following a disaster or emergency.
The purposes of interventional disaster evaluations include the: (1) determination of the worth/value of specific interventions and demonstration of accountability including needs-effectiveness; (2) development of evidence required to establish standards, and from these standards, to define best practices; (3) identification of interventions that produce a positive or negative impact in a given or similar setting; (4) development of methods and the validation of indicators that can be used to evaluate interventions; (5) definition of competencies and the education and training required to achieve them; (6) improvement of the processes in terms of costs, effectiveness, and efficiency for subsequent uses of the intervention; and (7) provision of information for use in obtaining the resources required to provide further interventions. 8
As with epidemiological disaster studies, evaluations of specific disaster-related interventions should attempt to answer: (1) What was done?; (2) What happened?; (3) Where was it done?; (4) When was it done?; (5) Why was it done?; (6) How was it done?; (7) Who did it and who was affected?; and (8) What were the outcomes and other effects? The final question is an important aspect of any evaluation and essentially defines the impact and what, if any, difference the intervention made for the targeted population. 9 - Reference Pfefferbaum, Noffinger, Sherrib and Norris 11
Types of Evaluations
According to Øvretveit, there are four types of evaluations that are useful in health care: (1) experimental; (2) developmental; (3) economic; and (4) managerial. 9
Experimental Evaluations
Experimental evaluations are conducted using RCTs. Experimental evaluations, when well executed, have the highest internal validity and are used to prove cause-effect relationships associated with the intervention. During any phase of a disaster or emergency, it is difficult, if not impossible, to establish a randomized control and experimental group required to conduct RCTs. Experimental evaluations (using a control group) may have a place in evaluating risk-reduction interventions, and may have applications in recovery interventions, although use of randomization will be unlikely.
Developmental Evaluations
Developmental evaluations consist of three types: (1) active; (2) pragmatic experimental; and (3) social research.
Active and Pragmatic Experimental Evaluations
Active and pragmatic experimental evaluations are used to provide feedback during the conduct of the intervention and may have some applications in the evaluation of disaster management (managerial) and of risk-reduction and recovery interventions.
Social Developmental Evaluations
Social developmental evaluations have long been used in social science studies and for the evaluation of educational programs, as well as in evaluations of disaster-related interventions. Developmental evaluations utilize before-after designs (quasi-experimental; longitudinal). Although they have been used in the evaluation of health services, policies, and procedures (process), generally, they have not been applied to evaluations of treatment regimens. Developmental evaluation studies describe the interventions and the users of the intervention and facilitate the development of knowledge by building on other studies. Developmental evaluations focus on methods (process), produce facts, and include many assumptions. Generally, they have not focused on the consequences (effects or costs) of the intervention being evaluated. However, these variables should be added to any evaluation of disaster-related interventions. While developmental evaluations allow one to compare similar interventions in similar settings, they require tightly structured reporting. Without a rigorous structure, comparisons become difficult and may not contribute to the development of evidence. Useful designs for use of developmental evaluations are discussed later in this paper.
Economic Evaluations
Economic evaluations examine the costs associated with an intervention and the consequences of implementing the intervention. In addition to the direct financial costs, the opportunity, environmental, utilization of other resources including goods and services consumed, and administrative costs are accounted. The effectiveness of the intervention in attaining the objective is examined per unit of cost (resources consumed by the process). Use of any resource has an opportunity cost in that the resource could have been used for another purpose. The costs accounted may be direct (used by the intervention) or indirect (associated with the intervention).
The purpose of economic evaluations is to identify the intervention that has the lowest cost for the same effects and benefits. 9 This requires that comparisons be made between interventions. Several derived variables (computed from assessment data) that assist in the evaluation include: (1) effectiveness; (2) efficacy; (3) efficiency; (4) needs-effectiveness; (5) needs-efficacy; (6) cost-effectiveness; (7) cost-utility; and (8) cost-benefit. The effectiveness of an intervention is whether the intervention achieved all or part of its objective(s). The efficacy of an intervention is whether it contributed to achieving the established goal. The efficiency of an intervention relates to the lowest cost in achieving the desired outcome. Need-effectiveness compares the effectiveness with the ability of the intervention to meet the needs for which it was implemented—was the selected intervention the best to meet the identified needs? Needs-efficacy is whether the consequences of the intervention met all or part of the needs required to reach the overarching goal for which the intervention was selected. Cost-effectiveness consists of the resources consumed to achieve a degree of effectiveness; cost-utility relates to the well-being of the affected community in relation to their costs; and cost-benefit consists of the resources consumed to obtain a benefit to the community (eg, the amount of money required to achieve a given benefit). Cost-utility and cost-benefit are related closely. While all of these economic descriptions relate to the effects of the intervention, the indicators and units for these derived variables have not been codified. However, using these derived variables facilitates comparing interventions in a given setting in order to be able to select the best intervention for a given situation with the lowest possible costs.
It is suggested that economic evaluations be part of any developmental evaluation. Some evaluations may focus entirely on the economics associated with the evaluations while others may focus on the impacts of the intervention. Combining these two evaluations in one study likely will increase the costs of the study. Much of the data obtained in such assessments and derivations will be qualitative in nature and will require scaling for comparisons.
Managerial Evaluations
Managerial evaluations are done for the purposes of: (1) routine monitoring; (2) special monitoring; (3) enhancing/optimizing performance; and (4) needs-effectiveness. Managerial evaluations focus on accountability: (a) how and why the resources consumed were spent; (b) protection from unsafe services; (c) ensuring that standards are met; (d) enhancing performance; (e) improving implementation procedures and policies; and (f) assisting in creating and testing a system. They seek to improve the use of resources and decrease the risks to the population affected or at risk for an event.
A special form of managerial evaluations is used to examine the performance of individuals or organizations at specific levels of operations. Performance is the execution of an action; something accomplished.Reference North and Norris 12 Generally, it relates to the effects achieved per unit of input (efficiency: output/input). Performance evaluations seek to determine the lowest costs and highest productivity and how well a given service achieves desired results. Satisfaction with the services/intervention provided also is a measure of performance. 9 Performance evaluations are used to evaluate personnel and operations during disaster exercises as well as during actual disasters and other emergencies.
Given that, generally, it is difficult, if not impossible, to accomplish RCTs in the settings surrounding a potential or actual disaster or other emergencies, most disaster-related evaluations are limited to before and after constructs using any one or combination of the types of evaluations described. Therefore, the objectives and purpose of any evaluation must be described in detail. Possible study designs are outlined later in this paper.
Research efforts in Disaster Health aim to evaluate an intervention directed at changing the current status of the community affected or the communities-at-risk for a similar event or, in some cases, for any event. The term “impact” often is confused with the terms “output,” “outcome,” and/or “effect.” Frequently, these four terms have been used interchangeably. Likewise, the terms “goals” and “objectives” relative to an intervention/response have been used interchangeably. The inappropriate use of each of these terms tends to confound the science.
In order to sort out the use of these terms, it is helpful to view all interventions/responses as production functions (transformation processes) by which the current status is changed by the implementation (application) of the process/intervention. This process has been translated from industry and education to the social sciences, and is called a “Logic Model.” 13 - Reference Bickman 19
Logic Models
Considering all interventions as production functions (Figure VI-1) provides the structure needed to analyze the processes used and aids in evaluating interventions that are used prior to, during, or following a disaster or other emergency. A logic model is a graphic representation of a program [project] that shows the intended relationships between investments and results and provides a framework for describing relationships between investment, activities, and results. 14 Logic models are a derivative of production functions (transformation processes). 13 - Reference Cobb and Douglas 15 The concept of a logic model was introduced by Bennett in the Seven Levels of Evidence,Reference Chambers 16 while the term “logic model” was first used by Wholey in 1979.Reference Bennett 17 Since the 1970s, logic models have been used for quality management and planning by private, public, and non-profit organizations in the domestic and international arenas (including by some donors).Reference Wholey 18 - Reference Rogers 30
Classic Production Function. The transformation process converts the resources into outputs/effects that change the current status.
Logic models are used to describe a program and its theory of change. “A theory of change is a description of how and why a set of activities—be they part of a highly focused program or a comprehensive initiative—are expected to lead to early, intermediate, and/or long-term outcomes over a specified period.”Reference Suchman 31 Logic models are useful in planning, implementation, evaluation, and communication.Reference Anderson 32 , Reference Taylor-Powell 33
The University of Wisconsin-Extension (UWEx; Madison, Wisconsin USA) has promulgated a logic model for the design and evaluation of educational programs.Reference Taylor-Powell, Jones and Henert 34 This logic model defines logical relationships between inputs, outputs (activities and participation), outcomes, and impacts over the short-, medium-, and long-term. The model depicted in Figure VI-2 is an example from the UWEx for Teaching and Training Guide Developing Logic Models.Reference Taylor-Powell, Jones and Henert 34 (p10) The principal purpose of the logic model is to evaluate the change (improvement) in the current situation. The process is driven by, or in response to, defined needs derived from assessments of the current situation. It focuses on the process(es) (interventions) used to create the change(s). A logic model is based, in part, on a set of assumptions and recognizes that the effects, outcomes, and impacts associated with the change may be influenced by factors other than the intervention (ie, external factors). Importantly, the model distinguishes the outcome from the impacts and provides the structure necessary to be able to compare interventions. A logic model provided by the Kellogg Foundation (East Battle Creek, Michigan USA) further clarifies the differences between resources, inputs, activities, outputs, outcomes, and impacts. 35
University of Wisconsin Logic Model. This logic model has been used for the evaluation of educational programsReference Pfefferbaum, Noffinger, Sherrib and Norris 11 , Reference Taylor-Powell, Jones and Henert 34 (p10) to analyze the changes in the current situation produced by the transformation process (intervention) that result in outcomes and impacts. The evaluations examine the essential assessments from which needs are determined, and the process, outcomes, and impacts are identified. The model recognizes that assumptions and factors that are external to the process may alter the outputs, outcomes, and impacts of the intervention. Reproduced with permission from the University of Wisconsin-Extension.
The Disaster Logic Model
To date, none of the logic models identified can be applied precisely to the evaluation of interventions provided before, during, or following disasters and other emergencies. Disaster-related interventions seek to change the current level(s) of function(s) of one or more Societal Systems (or components) affected by, or at-risk for, a catastrophic event. Thus, the effects (outputs) from the transformation process should consist of a positive change(s) in the functional status (levels of function(s)) of the community or a System within a community (may prevent/minimize further deterioration on level of function). Also, existing logic models do not require a definition of the goals or objectives of the interventions, nor do they relate the needs, effects (outputs), outcomes, and impacts to the goals and objectives for which the intervention was selected. Therefore, for the purposes of studying disaster-related interventions, the logic model has been revised to include relating the goals and objectives of the intervention with the identified need(s), the effects of the intervention with the goal and objectives, the outcomes with the objectives of the intervention, and the impacts with the overarching goal of the interventions. Separating the effects (outputs), outcomes, and the impacts is essential for the useful evaluation of a disaster intervention. This version of the logic model is referred to as the “Disaster Logic Model” (DLM; Figure VI-3).
The Disaster Logic Model (DLM). A logic model modified for use in evaluating interventions provided before, during, or after a disaster. This DLM relates the outcome to the objectives and the impacts to the overarching goal. The effects can be tempered by factors that are external to the transformation process. *Requires assumptions. **May be achievement of a number of actions in which case it is not an effect of the intervention.
The DLM outlines a production function designed to transform the current status into a new (and hopefully, better) state. In order to achieve the new state, an intervention is implemented and resources are consumed. 36 , Reference Birnbaum, O’Rourke, Daily and Loretti 37 The resources consumed are the costs (human, equipment, supplies, environmental, economic, political, and/or opportunity). The outputs are the products (effects) of the transformation process (the intervention provided). 36 , Reference Birnbaum, O’Rourke, Daily and Loretti 37
The DLM consists of a series of consecutive steps (stages) beginning with the initial assessments of current levels of functioning of a Societal System and/or its components through the determination of the value of a specific intervention to the affected community or a community-at-risk. The outputs may have various effects on the status of the populations, infrastructure, the environment, and/or the economy for which the intervention/response was initiated—there may be more than one effect produced by the intervention. These effects generate the impacts. The impacts are the “so what” of the intervention 8 , 9 , Reference Birnbaum, O’Rourke and Daily 38 (ie, what difference(s) did the intervention make for/on the community affected or at-risk?). The impact(s) produced by the intervention may be positive or negative. Positive impacts improve the pre-intervention status and are called “benefits.”Reference Pickett 7 Other effects produced by the intervention may have further compromised the pre-intervention status (negative impact). The negative impacts may have greater significance than the positive impacts. The DLM can be applied to evaluating the results of any intervention, regardless of the setting or phase in which it was provided.
All responses (interventions) must be directed towards meeting a defined need or set of needs of the person, System, or community affected, or likely to be affected, by an event. Needs are the difference(s) between the available goods, services, and other resources required to sustain or achieve a given level of function, and those goods and services needed to meet that requirement. The identified need(s) is (are) synthesized into a Strategic Plan that defines the goal(s) that should be accomplished in order to meet the defined need(s). Interventions are selected that are likely to contribute to achieving the defined goal. The objective(s) of the interventions selected should move, or contribute to moving, the status of the affected community or community-at-risk towards the defined goal. The outcome(s) of an intervention is the achievement of the level of function stated in the objective(s) of the intervention. 36 , Reference Birnbaum, O’Rourke, Daily and Loretti 37
For example, if assessments indicate that 3,000 internally displaced persons in a camp are at a high risk for an outbreak of cholera, the goal would be to prevent the development of a cholera outbreak/epidemic in the camp. An intervention would be selected that has as its objective to immunize 3,000 camp residents. If 3,000 residents subsequently received cholera immunization, the objective of the intervention was achieved (ie, the outcome met the objective for which the intervention was designed). However, if, despite the successful completion of the immunization campaign, an outbreak of cholera occurred within the camp, the goal of preventing a cholera outbreak/epidemic was not achieved. And, if the immunization process resulted in any deaths, the status of the population actually deteriorated (had a negative impact) although the objective of the intervention was achieved. The other effects of the immunization process may have outweighed the probable benefits (positive impact) of the immunization intervention. Thus, merely achieving the desired outcome of the intervention does not reveal the impact of the intervention.
Assumptions are the beliefs about the program [project], people involved, and how the program [project] will operate.Reference Birnbaum 39 The synthesis of assessments and other information is an integrative process and the results include assumptions based on the synthesis of information available and used. Assumptions are a part of all logic models. Documentation of assumptions is an important aspect of the use of logic models.
Use of the DLM provides the structure needed for identifying the value of any disaster intervention or project. It can be used to identify the effects (outputs); the outcome(s) (effects related to the objective(s) of the intervention); the impacts on the affected population or the population-at-risk; as well as the resources (human, materials, financial, environmental, opportunity, and political costs) consumed by the transformation process. The efficacy, efficiency, cost-benefit, and cost-effectiveness in achieving the objectives can be derived from the assessment data collected and can be compared with those from other interventions by the same or different providers in similar settings. The DLM also provides the structure for the steps outlined in the Relief/Recovery and Risk-Reduction Frameworks. 36 , Reference Birnbaum, O’Rourke, Daily and Loretti 37 Each of the stages in the DLM is described briefly below.
Current Status
The first task outlined in the DLM is to assess the current status (level(s) of function(s)) of the Societal System(s) or its components under consideration. An assessment is the product obtained from assessing; the collection of relevant information that may be relied on for making decisions. 40 , Reference Thompson 41 Assessments are processes by which facts are gathered that reflect one or more elements of the current functional status of one or more of the Societal Systems of the affected community or the community-at-risk. This requires that the community be considered in terms of its functional Societal Systems.
The current status includes the levels of function (including ongoing changes in the rate of change occurring) of the Societal System or a component(s) of the System being considered, as well as those aspects of other Societal Systems that may impact or be affected by the intervention being evaluated. Assessing the current status requires the use of a set of indicators of function according to the Societal System being assessed. Structural damage related to the changes in function also may be included, as some interventions may be directed towards repair, rehabilitation, or replacement of damaged structures. Thus, the current status is synthesized from the results of the assessments of function and damage of one or more Societal Systems. The process of synthesizing assessment data is integrative and requires inputs from experts or persons experienced with the process. There always is a risk of faulty assessments and/or misinterpreting the data and information used.
As all interventions aim to change the current functional status of a population or System, the following assessment information is essential: (1) the assessed pre-event and pre-interventional functional status of the community and/or the specific Societal System being studied; (2) when the assessment was conducted; (3) what indicators of function were used in conducting the assessment; (4) who performed the assessment; and (5) who synthesized the information.
Needs
Needs are synthesized from multiple assessments (including levels of function, damage, the setting, available goods, services, and other resources), and are expressed as goods, services, and other resources required (including personnel, infrastructure, and process). Therefore, the determination of needs requires the transformation of levels of function into the goods, services, other resources, and infrastructure required to halt or minimize further damage, fill gaps in essential services, return levels of function to their respective pre-event levels, augment resilience, and/or prevent/mitigate events and damages in the future (Figure VI-3). The needs identified are assumptions of what goods and services must be provided to correct actual or anticipated deficits in functions. There always is a risk that these assumptions are incorrect. The identified need(s) is (are) synthesized into a Strategic Plan that defines the goal(s) that should be accomplished in order to meet the defined need(s).
All interventions are intended to meet one or more defined need of one or more Societal Systems. Interventions are selected that are likely to contribute to meeting the need and achieving the defined goal. The objective(s) of the intervention(s) should be selected on the basis of the likelihood of its ability to move the status of the affected community or community-at-risk towards the defined goal with optimal efficiency.
Strategic Planning
Each intervention provided during any phase of a disaster must be in response to an identified need that is synthesized into the goals outlined in a Strategic Plan. Strategic planning is:
[A]n organizational management activity that is used to set priorities, focus energy and resources, strengthen operations, ensure that employees and other stakeholders are working toward common goals, establish agreement around intended outcomes/results, and assess and adjust the organization's direction in response to a changing environment. It is a disciplined effort that produces fundamental decisions and actions that shape and guide what an organization is, who it serves, what it does, and why it does it, with a focus on the future. Effective strategic planning articulates not only where an organization is going and the actions needed to make progress, but also how it will know if it is successful.Reference Pickett 42
Strategic planning includes all measures taken to achieve a specific goal, and in which order (priority). Strategic planning requires consideration of all of the needs defined by each of the Societal Systems, and often is based on previous experience and expertise; “A strategic plan is a document used to communicate with the organization, the organization's goals, the actions needed to achieve those goals and all of the other critical elements developed during the planning exercise.”Reference Pickett 42 Strategic Plans result from a planning process that integrates defined needs with available resources. The Strategic Plans that result from the planning process must include strategic goals and the objectives of suggested interventions. Disaster Response Plans are Strategic Plans. Some Strategic Plans also may contain generic outlines of Operational Plans.
A goal is an aim; 43 the result or achievement toward which an intervention (transformation process) is directed. 43 - Reference Smith 47 Goals typically are expressed in broad terms of what is to be accomplished through some project or program. Goals are broad, overarching, general, and often abstract (non-specific).Reference Schuman, Lewis and Ritchie 48 The overall goal of any intervention is to change the current status of a Societal System or any of its components in a direction that is positive for the community affected or at-risk.
An objective is something worked toward or striven for; something sought or aimed at.Reference Smith 47 - Reference Lewis 49 The specific objective(s) of possible interventions should define how the intervention will contribute to achieving the overarching strategic goal. An objective is a milestone that reaches towards the goal. In contrast to goals, objectives are narrow, precise, tangible, and concrete.Reference Smith 47 - Reference Lewis 49 Objectives may be written to meet the so-called “SMART” rules: Specific, Measurable, Achievable, Realistic, and Time defined.Reference Schuman, Lewis and Ritchie 48 Each intervention has its specific objective(s). Many different objectives achieved by several different interventions may be needed to contribute to achieving the overall strategic goal. A possible objective could be performing a specific number of procedures such as immunizations (achievement index). An essential element of any Strategic Plan is the provision of appropriate timelines for achievement of the goal and for selected interventions.
Strategic planning for disasters and other emergencies is a responsibility of Coordination and Control entity. Strategic Plans are developed by the most appropriate Coordination and Control agency (Cluster, Unified Command). Some components of the Strategic Plan may be delegated to the most appropriate Societal System, but the overall responsibility rests with the most appropriate level of Coordination and Control.
Selection of an Intervention
Interventions are selected by Coordination and Control based on their likelihood of achieving the objective that supports the goal stated in the Strategic Plan. Attaining a goal could require multiple interventions. The process for selecting a specific intervention also consists of assumptions that the intervention will be successful in meeting the defined objective and will contribute to achieving the overarching strategic goal. A discussion of a selected intervention should include descriptions of the assumptions upon which the selection is based. 36 , Reference Birnbaum, O’Rourke, Daily and Loretti 37
Operational Plan
Operational planning is the process of linking strategic goals and objectives to tactical/operational goals and objectives.Reference Pickett 50 It describes milestones, conditions for success, and explains how, or what portion of, a Strategic Plan will be put into operation during a given operational period.Reference Pickett 42 An Operational Plan is the plan that a manager uses to accomplish his/her job responsibilities: “The purpose of the operational plan is to provide organization personnel with a clear picture of the tasks and responsibilities in line with the goals and objectives contained within the strategic plan.”Reference Pickett 50 The differences between a Strategic Plan and an Operational Plan are outlined in Table VI-1.
Differences between a Strategic Plan and an Operational Plan
“An operational plan is concerned with what the lower level units within each division must do, how they must do it, and who is in charge at each level;”Reference Pickett 50 Operational Plans are the means needed to activate an intervention and make it work. An Operational Plan must be in place before any intervention is initiated. Operational Plans must be developed by the organization planning to deliver the intervention. Operational Plans are approved by Coordination and Control. Some organizations possess generic Operational Plans (ie, Disaster Response Plans) for interventions. These generic plans must be adjusted according to the current situation/circumstances.
Providing (Execution of) the Intervention (Transformation Process)
The selected intervention is implemented after approval from Coordination and Control, which is responsible for coordinating the intervention with other interventions currently being provided or that are anticipated. Coordination and Control determines where the intervention will be provided, and establishes and endorses timelines, including beginning, termination, and reporting processes and requirements. It also provides linkages to other interventions that may be seeking the same or a similar goal, or that may have an impact on the intervention being implemented.
Implementation of an intervention is the transformation process that utilizes resources to produce outputs. Each intervention must be described in detail, including: (1) the specific objective of the intervention as well as how and when the needs for this intervention were determined (ie, assessment indicators used); (2) who provided the intervention and how and where it was provided; (3) who received the intervention; (4) the duration of the intervention; and (5) the costs of the intervention (human, opportunity, financial, environmental, supplies, equipment, or political).
Results of the Intervention
All interventions are selected because it is assumed that they will change the current status (level of function) of a Societal System or component(s) of a System in a positive direction (benefit) for the affected community or a community-at-risk.
The results of the intervention are assessed using the same indicators of function that were used to define the pre-intervention level(s) of function(s) upon which the needs were identified. Changes in levels of function are detected, measured, and reported. In scientific papers, these findings are described in the “Results” section of the paper.
Outputs, Effects, and Outcomes
There is substantial confusion in the use of the terms “output,” and “effects,” and “outcome.” Outputs are the products of transformation processes (parts of Production Functions); 51 the direct product of an activity [intervention] including types, levels, and targets of services delivered; the activities, products, and interpretation generated through investments of resources.Reference Thompson 52 , Reference Heinrich, Molenda, Russell and Smaldino 53
Effects are the result or consequence of an intervention.Reference Williams 54 , Reference Thompson 55 Effects must be judged in relation to the objective(s) for which the action(s) was implemented. The outputs may include various effects on the status of the populations, infrastructure, the environment, and/or the economy for which the intervention/response was initiated; there may be more than one effect produced by the intervention. However, not all of the effects of an intervention may relate to the objective; other expected or unanticipated effects, or under- or over-estimated magnitude of the effects, may occur. These effects also must be included in any evaluation of an intervention. Some of the effects may have greater significance than the one for which the intervention was intended (ie, its objective). In the DLM, the terms effects and outputs essentially are synonymous. To avoid confusion, use of the term “effects” is recommended in describing the results of an intervention.
All effects of an intervention must be documented, including: (1) the effect(s) relative to the objective of the intervention (ie, the outcome), including how and when it was measured (indicators used); and (2) all other effects (positive and negative), including how and when they were assessed.
Outcomes are the changes or results that a specific intervention was selected to achieve; the results of an intervention relative to the objectives of the intervention.Reference Heinrich, Molenda, Russell and Smaldino 53 , Reference Pickett 56 The outcome of an intervention refers to the effect(s) that is/are related specifically to the stated objective of the intervention (signified in the diagram in Figure VI-3 by reversible arrows between the outcome and the objective). It is possible to achieve the outcome of an intervention without the intervention contributing to attaining the Strategic Goal. For example, if a Strategic Goal was to prevent dehydration among victims of cholera, and the objective of an intervention was to provide sufficient quantities of oral rehydration solution for 5,000 persons, and a sufficient amount of rehydration solutions was provided to treat 5,000 persons, the outcome of the intervention achieved its objective. But, was the quantity of rehydration solution provided successful in meeting the goal of maintaining the hydration of the victims over the short-, medium-, and long-term? It is not known whether the fluids actually were distributed to the affected population or whether the administration of the fluid prevented the development of dehydration in the population. Other effects of the provision of the solutions may have occurred; the resources required for the transportation of the solutions may have encumbered the ability of the Logistics and Transportation Societal System to provide other needed services (opportunity cost). Thus, attaining the objective may not have contributed to achieving the Strategic Goal.
All evaluations are dependent upon selecting the most appropriate indicators of function. Thus, not only must the goals and objectives be defined, but also the indicators that best reflect the attainment of the respective goals and objectives must be identified and monitored. The same indicators that were utilized to determine the needs and select the intervention must be used to determine if the objectives were accomplished. The indicators related to the goals may be the same as those for the objectives, but generally are of a higher order. Additionally, indicators must be selected and monitored that reflect the other possible effects (positive and negative) of the interventions. The appropriateness of the indicators selected is their construct validity. Indicators may be valid or invalid for determining the effects, outcomes, and impacts of any intervention. Selecting appropriate indicators of function is crucial to the evaluation of the impacts of any intervention.
Unfortunately, to date, the numbers of persons who received the intervention (achievement indices) most often have been the only reported end-points of the intervention.Reference Zint 57 However, achievement indicators cannot describe the impact of any intervention/response. Treating a specified number of patients as a response intervention has no meaning in terms of developing evidence. To be of value, what was done and what difference the intervention made to the population must be known. Did the intervention change the pre-intervention level(s) of function(s)? The successful completion of an intervention, or even the achievement of an objective, is not an adequate end-point of any evaluation of a disaster intervention/response. Attaining the goal may be an important end-point provided the negative impacts (side effects) of the intervention do not render the overall pre-intervention status worse. Achievement indexes are limited to describing how many of something was accomplished by the intervention (eg, how many patients received an immunization or were evaluated/treated in a clinic). Achievement indices contribute little, if anything, to the science of Disaster Health. The effectiveness of the intervention is whether the effect(s) of the intervention met its designated objective.
Impact(s)
Effects generate an impact on the pre-intervention status of the community. Some effects produced by the intervention may have further compromised the pre-intervention status (negative impact). The negative impacts may have greater significance than the positive impacts. In terms of evaluations of interventions, impacts are the social, economic, civic, and/or environmental consequences of a program [intervention]; 58 , 59 a measure of the tangible and intangible effects (consequences) of one thing’s or entity’s action or influence upon another;Reference Pfefferbaum, Noffinger, Sherrib and Norris 11 broad, longer-term changes that occur within a community, organization, society, or environment as a result of the effects [of an intervention]. The value of the new level of function to the community is determined through the synthesis of many variables.
Value is the worth, desirability, or utility of a thing;Reference Thompson 60 a principle, standard, or quality considered worthwhile or desirable.Reference Thompson 61 Worth is the quality that renders something desirable, useful, or valuable.Reference Thompson 61 , Reference Pickett 62 The value (impact) of a specific effect of the intervention on the community either could be positive (benefit) or negative for the community. The impact(s) of the effects of an intervention must be compared with the Strategic Goal(s) for which the intervention was selected.
Even though an intervention met its objective(s), it may not have produced a benefit to the community for which it was provided. The “value” of any effect from an intervention must be judged not only by whether it met all or part of its objectives, but whether it produced a benefit (positive impact) to the affected community or community-at-risk. Positive impacts improve the pre-intervention status, and are called “benefits.” Achieving the Strategic Goal should be associated with a benefit to the community.
If the effect(s) of an intervention did not benefit the community, or a System within the community, it is important to determine the critical point(s) of failure of the intervention, and how the transformation process can be modified to attain the established goal the next time use of that intervention is proposed. Use of the Relief/Recovery or Risk-Reduction Frameworks should assist in evaluating this process.
Variables derived from the effects may facilitate the identification of impacts and comparison with the impacts of other interventions. These derived variables include effectiveness, efficiency, efficacy, needs-effectiveness, and benefit. Definitions of these indicators have been provided earlier in the economic intervention section of this paper. The efficacy of an intervention is the power or capacity to produce a desired effect;Reference Pickett 63 the ability of an intervention to produce the desired beneficial effect in expert hands and under ideal circumstances.Reference Pickett 7 However, the “expert hands” and “ideal circumstances” are difficult to achieve in the settings of an emergency or disaster, so efficacy in Disaster Health must be modified to apply to similar settings and competencies. In emergencies and disasters, an intervention may be considered efficacious if it is likely to produce the same beneficial effects (see External Factors below) using the same equipment and supplies by personnel possessing the same competencies in similar settings.
The impacts of interventions/responses may be short-, medium-, or long-term, and may be the direct or indirect effects of the intervention. In part, this depends on the defined goal—the goal may be short-, medium-, or long-term, but the medium- and long-term impacts should be considered in terms of the overall value of the intervention to the affected population or the population-at-risk. The impact of an intervention aimed at providing relief must be evaluated relative to its goal of limiting the progression of functional damage from an event, while recovery interventions must be evaluated relative to the goal of returning the functional level to its pre-event state. What may seem to be an ideal outcome and positive direct impact from an intervention may be outweighed by those often unanticipated or longer-term indirect effects. Therefore, all impacts of an intervention must be described including: (1) impacts relative to the goal of the project, including how and when they were assessed; (2) indirect impacts of the intervention, including how and when they were assessed; and (3) short-, medium-, and long-term results, including how and when they were assessed.
External Factors
Factors that are external to the transformation process may facilitate or impair the ability to provide the intervention selected, as well as to achieve the outcomes and impacts desired.Reference Taylor-Powell, Jones and Henert 34 (p10) External factors include: the setting in which the intervention is implemented; the culture of the community; weather; terrain; safety; and/or other interventions planned or being implemented in the area. External factors pertinent to the evaluation of risk-reduction interventions include other interventions previously undertaken to reduce the risk of an event from a hazard, and/or increase the absorbing, buffering, and/or response capacities of the community or System being studied.
For relevance and classification purposes, the evaluation of a Relief or Recovery intervention must be placed into the context of the hazard responsible for the event, the characteristics of the event, and the setting in which the intervention was provided. These are accounted in the DLM as “External Factors” that may affect the results and process(es) of the intervention (Figure VI-3).
Costs
The DLM outlines a production function designed to transform the current status into a new (and hopefully, better) state. In order to achieve the new state, an intervention is implemented and resources are consumed.Reference Taylor-Powell, Jones and Henert 34 (p4) Inputs consist of what is put in or taken in or operated on by any process or system.Reference Pickett 64 Within the DLM, the inputs (costs) consist of the resources (human, financial, opportunity, goods and services, environmental, and political) that are consumed by the intervention undertaken. Costs are a loss or sacrifice; an expenditure of resources.Reference Taylor-Powell, Jones and Henert 34 (p4) Therefore, the costs of an intervention consist of all the resources consumed by the intervention.
To date, the primary costs tracked relative to disasters have been financial estimates describing the costs of repair or replacement of damaged structures—how much money it would cost to fix the damage. The inclusiveness of the extent of the financial resources used has not been defined clearly. Does it include only repair and rehabilitation, or does it include the losses in other Societal Systems affected as well? If so, which ones? As noted previously, this is the primary focus of Economic Evaluations. The purpose of Economic Evaluations is to identify the lowest cost for the same effects and benefits; they examine the financial costs associated with the implementation of the intervention and the consequences of implementation of the intervention. The effectiveness of the intervention in attaining the objective is examined per unit of cost (resources consumed by the process; cost-effectiveness).
Several derived variables (computed from assessment data) assist in this process, including: (1) efficiency; (2) cost-effectiveness; (3) cost-utility; and (4) cost-benefit. These variables have been defined under Economic Evaluations.
Using these derived variables facilitates comparisons between interventions in a given setting and supports decisions to provide a specific best intervention in a given situation with the lowest possible costs. The costs accounted may be direct (used by the intervention) or indirect (associated with the intervention). Not only must the direct financial costs be examined, but also, when possible, the opportunity, environmental, utilization of other resources including goods and services consumed, and administrative costs. Use of any resource has an opportunity cost—could the resource have been used for another purpose?
Analysis of the costs of an intervention should be part of any Developmental Evaluation. Knowledge of the costs encumbered should help to focus the consequences of an intervention and facilitate essential comparisons. Some evaluations may focus entirely on the economics associated with the evaluations while others may focus on the impacts of the intervention.
Interventional Evaluation Designs
The design of any interventional evaluation can use one or more of the six types of designs described by Øvertveit in his text, Evaluating Health Interventions: (I) descriptive; (II) audit; (III) before-after; (IV) comparative-experimental; (V) RCTs; and (VI) impact on providers/patients.Reference Pickett 7 All of the designs are useful in examining the changes in the pre-intervention state as a result of a specific intervention. The study design selected depends on the objectives and goals of the evaluation. Each type of study design is described briefly.
Type I: Descriptive
Descriptive studies are designed to describe the intervention by asking “What is it? and “What happens?” The purpose of descriptive studies is to describe, in detail, the intervention being evaluated and the external factors that played a role in determining the effects of the intervention. The descriptive design is used for developmental and managerial evaluations (see prior discussion). The results of descriptive studies help to clarify the objectives of the intervention and are used to identify problems with the study. Often, the results are dependent on the background and skills of the evaluators. Descriptive studies often are used to test/pilot other evaluation designs.
Type II: Audits
Audit designs are used to compare what was done with what the intervention was designed to do. Did the evaluation do what was requested in the Strategic Plan? The audit design is used primarily for managerial monitoring; what was done is compared with an accepted standard. It is the same as the Type I: Descriptive study design, except that it compares the intervention and effects with a standard or best practices for the setting in which the intervention was implemented. The audit study design requires the existence of an established standard or set of best practices.
Type III: Before-After
This study design is quasi-experimental in that it examines the effects of the intervention by comparing the functional status before with the functional status following the conclusion (at steady state) of the intervention. The actual effects of the intervention are compared with those that were expected—the reasons the intervention was implemented. The evidence produced by such an evaluation is not conclusive as to cause and effect as the effects may have occurred due to factors external to the intervention (confounding factors), or the effects could have occurred without implementation of the intervention. Thus far, this design has been the most frequently used for the evaluation of disaster responses/interventions.
Type IV: Comparative-Experimental
Comparative-experimental study designs compare the effects of the intervention with an alternative intervention or the results of another “similar” intervention (eg, similar interventions in the same or different settings). The design is used to suggest the settings in which the intervention is most likely to produce a benefit to the community. The comparative-experimental design produces results that have greater internal validity than does the single before-after design.
Type V: Randomized, Controlled Trials
Randomized, controlled trials have long been the gold standard for scientific research. This study design is used to validate cause-effect relationships and requires that the effects of the intervention be compared with the effects in a control group in which the intervention was not provided. Currently, due to the inability to identify a control group and randomize the communities, such studies are difficult, if not impossible, to perform in the settings of emergencies or disasters.
Type VI: Impact
Impact study designs assess the impact(s) (benefits) of an intervention on the affected population or population-at-risk, or on the providers of the intervention. The importance of determining the impact of an intervention has been stressed in the above discussions. Impact designs go beyond judging the value of the effects of the intervention to include the social and other consequences of the effects of the intervention. The greatest value of impact studies rests with comparisons with other interventions with similar objectives in similar settings or similar interventions in different settings.
With the exception of the RCT designs, using the DLM in the design of a disaster intervention evaluation encompasses an amalgamation of all of the above study designs. Although each of the study designs noted above has value in disaster research, using the DLM will provide the most valuable information, especially when the impacts and costs are compared with other interventions directed at contributing to the same goal in similar settings. While the DLM may seem somewhat complex initially, it should become the standard for evaluating interventions implemented before, during, or after an emergency or disaster.
Finally, all evaluation studies must be compared with other similar studies, and therefore, they must incorporate the same structure in reporting the evaluation. A repository of interventional studies using the same structure will facilitate the development of best practices that are so essential for the design and implementation of capacity building interventions and responses.
Interventional Process Evaluation
The structures required to perform an evaluation of the processes involved in providing a disaster-related intervention are provided by the Relief/Recovery and Risk-Reduction Frameworks and are based on the DLM. These frameworks outline the steps used in evaluating relief/recovery and/or risk-reduction interventions.Reference Taylor-Powell, Jones and Henert 34 (p4), 35 Each step in the Frameworks has consequences on the transformations that follow in the series, and ultimately affects the results as well as the evidence obtained from the intervention, and each step in the series can be evaluated using the DLM. The success or failure of a specific intervention may be related to a component of the process rather than to the intervention itself. Identifying strengths and weaknesses in the processes involved in the intervention is an important component of interventional disaster research.
The Disaster Relief-Recovery and Risk-Reduction Frameworks are used to identify strong and weak points in the process used for the implementation of the disaster-related intervention. These two Frameworks are based on deconstructions of the processes currently used in production functions (transformations). Each step has consequences on the transformations that follow in the series and affects the evidence resulting from the intervention. Each of the steps in the series is amenable to evaluation using the DLM (Figure VI-3).
The Relief-Recovery FrameworkReference Taylor-Powell, Jones and Henert 34 (p4) is used to implement and evaluate the processes of interventions undertaken for the purposes of relief or recovery from the damages that resulted from an event. These processes include: (1) assessments; (2) identification of needs; (3) strategic planning; (4) selection of intervention(s); (5) operational planning; (6) implementation of interventions; (7) assessments of the effects and changes in levels of functions resulting from the intervention(s) being studied; and (8) synthesis of the effects, outcomes, costs, and impacts into evidence.
The Risk-Reduction Framework 35 describes the processes involved in undertaking a risk-reduction intervention. It builds on the Relief/Recovery Framework by adding the following steps: (1) identification of the hazard and risk(s); (2) background research; (3) selection of hazard(s) to address; (4) identification of the Societal System to be studied; (5) assessment of risk(s); and (6) examination of evidence.
The use of these Frameworks will facilitate the identification of critical points of success and/or failure in the overall process of developing, testing, and implementing relief, recovery, and risk-reducing interventions. Further descriptions of the Relief/Recovery and Risk Reduction Frameworks are provided in other papers in this series.Reference Taylor-Powell, Jones and Henert 34 (p4), 35
Example of an Interventional Disaster Study
In this hypothetical example, the disaster being studied is caused by a wildfire and the intervention being evaluated is a Relief response intervention to provide potable water to the affected community.
-
1. Identify the Societal System(s) being studied.
Example: The Water and Sanitation Societal System is selected.
-
2. Identify the question and the contribution towards attaining the overarching goal of the study.
Example: As part of strategic planning, the goal articulated was to provide a sufficient quantity of potable water to meet the basic functional threshold level of water supply. What was the effectiveness of an intervention of providing potable water to the affected community?
-
3. Identify the objective of the intervention provided.
Example: The objective of the intervention was to provide three liters/person/day of potable water to the affected population.
-
4. What was the pre-event status?
Example: Describe the population location and demographics and the average daily delivery and consumption of water/person/day prior to the event per water utility records.
-
5. What was the event?
Example: Describe the wildfire event in terms of the hazard that caused it, its onset, duration, amplitude, scope, and scale. Include any risk modification efforts that may have been undertaken, such as fire prevention and control programs.
-
6. What was the structural damage to the Water and Sanitation System from the event?
Example: Describe overall structural damage(s) as well as specific damage to the structure(s) of the Societal System related to the intervention being studied. In this case, it would be damage to facilities/equipment of the Water and Sanitation System. Define how and when the damage was assessed. Include descriptions of any actions that had been taken to increase the absorbing capacity, such as measures taken to protect the water supply components of the Water and Sanitation System from damage.
-
7. What was the functional damage (loss of function) of the specific Societal System being studied, and how was it assessed?
Example: Compare current water delivery and consumption to the pre-wildfire water consumption from utility company records and cluster neighborhood sampling. Determine any health effects in terms of morbidity and mortality from information obtained from health care providers; compare this information to pre-wildfire information and with the status just before implementation of the intervention. Include descriptions of any actions that had been taken to increase the buffering capacity, such as backup water supply systems (on an individual or community level) and community education efforts.
-
8. What was the result of the assessments of levels of functions, what needs were identified, and what plans were formulated?
Example: Describe the assessment tool(s) used, how the information was obtained, which indicators of function were used, and when (including which phase of the disaster) the assessment occurred. Detail the identified needs (goods, services, and resources) and the planned intervention to meet these needs. In this hypothetical case, the assessed current (pre-intervention) available supply of water was two liters/person/day. The functional threshold for water in the affected community was determined to be five liters/person/day in those conditions. Therefore, an additional three liters/person/day was needed to meet the functional threshold. Thus, a Strategic Plan was developed to determine mechanisms to provide an additional three liters of water/person/day to the affected population. The final intervention selected was the provision of water to affected neighborhoods from an outside source using tanker delivery trucks. A company from outside of the area was contracted to obtain and deliver the required amount of water.
-
9. What was the intervention and how was it implemented?
Example: Describe in detail the intervention provided including who provided the intervention, how, when, where, to whom, and the duration that it was provided, as well as any pre-existing arrangements that may have facilitated the intervention. In this example, the objective was to provide three liters of potable water/person/day to the affected population. The contractor developed an Operational Plan to meet this objective that included: (1) identifying a source for the needed water; (2) obtaining access to the water needed; (3) obtaining the vehicles required for the transport, the required drivers and load masters, and the required fuel; (4) accessing, loading, transporting the required water; (5) defining the reception and distribution process to be used in the affected neighborhoods; (6) acquiring the staff necessary to unload the water and the storage facilities for the transported water; (7) providing an equitable distribution system; (8) providing a timetable for the project; and (9) providing an evaluation of the project’s effectiveness, efficacy, costs (economic, opportunity), efficiency with which it was conducted, as well as other effects generated by the project, the outcomes relative to the objectives, and the impacts (benefits) achieved. An outside water delivery company was contracted to supply potable water to the community in the amounts indicated through a pre-event memorandum of understanding.
-
10. What were the effects of the intervention?
Example: Describe what happened related to the intervention: the effects of the intervention in terms of the levels of functions resulting from implementation of the intervention. The effects are the facts with no comparisons of levels of function—just the facts obtained from the assessments of levels of function. Include the effects not related to the objectives, and the costs (resources consumed including financial, opportunity, environmental, or human) of the intervention. In this example, the outside delivery of potable water resulted in members of each family having access to five liters/day of potable water. Define any encumbrances encountered in the delivery and receipt of the water by each family. The benefits related to changes in the health status of the population were obtained by comparing information provided by health care providers before the event, after the event but before the intervention, and after the intervention. The costs of the intervention were obtained from the delivery company. Other effects, such as the creation of large quantities of empty, plastic water containers, and disputes among residents regarding unfair water usage should be reported.
-
11. Synthesis (putting it all together)
-
a. Did the intervention meet its objectives? The objective to provide three liters potable water/day/person was achieved by the intervention.
-
b. Did the intervention contribute to the goal of supplying enough water to achieve the functional threshold? The functional threshold of five liters water/person/day was achieved.
-
c. What were the impacts on the population? The community was able to operate at or above the functional threshold that existed prior to the event, and at a higher level than had existed prior to the intervention. Other impacts included an increase in social networking as neighbors looked out for other neighbor’s water supply; a decreased incidence of dehydration among the elderly also was noted.
-
d. What was the efficiency of achieving the objectives, the cost-effectiveness, and cost-benefit of the intervention?
-
e. What were the other effects of the intervention on the Water and Sanitation System and on other Societal Systems (and their costs)?
-
f. What role did external factors play?
-
g. Did the Water and Sanitation System recover (return to its pre-event level of function) as a result of the intervention? In this example, the intervention restored the community’s functional status related to the quantity of potable water as determined by repeated assessments. However, the intervention did not contribute to the recovery of the Water and Sanitation System. This required a different intervention.
-
h. Describe the results of repeated assessments using the same indicators of function and compare them to the pre-event status to determine when and if recovery occurred.
-
i. How did the findings compare with those from other studies? Do the findings contribute to the development of evidence?
-
j. Should the intervention be used again in similar or other circumstances? Compare the results with results of other interventions provided during or following the same or other events. In this example, the evidence from this evaluation indicates that the described intervention of providing additional water to an affected population may be useful in similar settings during a disaster.
-
Evidence
The major reason to perform disaster-related evaluations is to generate evidence that contributes to developing the science of health in disasters. Science is the pursuit and application of knowledge and understanding of the natural and social world following a systematic methodology based on evidence.Reference Thompson 65 Evidence is something that should show that something else exists or is true; 66 the available body of facts or information indicating whether a belief or proposition is true or valid. 67 The results of quality studies produce evidence, which then can be synthesized into standards of practice. These standards become the basis for defining best practices that are likely to produce a positive change in the functions of a Societal System in a given setting.
In a review of existing evidence and its role in disaster decision making, Bradt describes a hierarchical classification of evidence. 68 , Reference Bradt 69 As RCTs are difficult and often impossible to conduct during disasters and other emergencies, disaster studies must use lower level designs for data collection, such as quasi-experimental (before and after) studies, surveys, and structured and unstructured interviews. Non-randomized, controlled studies may be possible for the evaluation of interventions aimed at recovery or risk reduction. And while the findings from individual case studies do not have high internal validity (prove cause and effect), each study contributes to the body of evidence; the strength of the evidence comes from repeated documentation of similar findings in the same or similar settings (external factors). Thus, the findings from any evaluation of disaster interventions must be compared with the findings from other similar studies in similar settings. In order to perform such comparisons, identical structures and definitions must be used. This strengthens the evidence and facilitates the development of best practices. As Bradt points out, data and information beget knowledge and knowledge begets wisdom. 68 Evidence to support best practices in Disaster Health comes from such wisdom.
Synthesis of Evidence into Standards and Best Practices
Standards are not based on findings from a single study. This is of particular significance in reports related to Disaster Health. In the absence of RCTs (and even studies that have employed RCTs), experts have attempted to derive standards of practice (and best practices) by combining and comparing the results of many studies. Codification of the findings have been published in literature reviews, narrative reviews, critical reviews, and commentaries.Reference Bradt 70 In the previously cited review of 487 disaster-related publications, five percent were literature reviews and seven percent were commentaries (opinions). 2 Noted problems that limit the value of both the reviews and commentaries include: (1) "not led by a peer-reviewed protocol" (as in systematic reviews) and therefore, are "difficult to replicate;" (2) a lack optimal rigor; (3) possible bias by the views of the authors to build support for personal beliefs by "selectively citing appropriate studies;" and (4) authors may have been selected "because of their accumulated experience and professional opinions."Reference Bradt 70 In an effort to overcome these short comings, Systematic Reviews have been implemented. A Systematic Review attempts to bring the same level of rigor to reviewing research evidence as should be used in producing that research.Reference Bradt 70 The Cochrane Collaboration (London, United Kingdom) defines a Systematic Review for health care as a summary of "available carefully designed health care studies (controlled trials) and provides a high level of evidence on the effectiveness of health care interventions."Reference Hemingway and Brereton 71 Systematic Reviews differ from literature reviews in that they are based on a peer-reviewed protocol and pre-determined questions. They seek to: (1) identify all relevant published and unpublished evidence; (2) select studies/reports for inclusion; (3) assess the quality of each study or report; (4) synthesize the findings without bias; and (5) interpret the findings and present a balanced/impartial summary.Reference Bradt 70
The Cochrane Collaboration has been conducting Systematic Reviews for health care since the mid-1990s. Unfortunately, they have not been focused on disasters. Evidence Aid (Oxford, United Kingdom) has evolved since the 2004 Earthquake and Tsunami in South East Asia, and has the mission of "inspiring and enabling those guiding the humanitarian sector to apply an evidence-based approach in activities and decisions." 72 Evidence Aid Systematic Reviews use the methodology promoted by Cochrane and others conducting Systematic Reviews, and has developed a method for selection of questions to be addressed. 73
Systematic Reviews of Disaster Health intervention studies are necessary in order to transform evidence from specific studies into standards and best practices to be applied in disaster risk reduction. Priorities for such reviews must be established and their use must be coordinated. This process will be facilitated by the using the structure provided by the Frameworks and by a universal terminology to be used in all reports on the health aspects of disasters. Such structure allows the development and implementation of repositories of information from studies of the epidemiology of disasters and for the interventions provided before, during, and following disasters.
Summary
Interventions are designed to change an element of the current status of a community, or a Societal System or its components. The over-riding goal of any intervention is to provide the goods, services, and other resources that are required to meet the needs of a Societal System—to change the current status in a positive direction from where it was prior to the intervention. Evaluations of interventions can take many forms depending on their designated purpose. Regardless of the type of evaluation or the design of the evaluation, the DLM is useful in determining the value of interventions. In addition, it underpins the application of the Relief-Recovery and Risk-Reduction Frameworks in evaluating the processes used in disaster-related interventions. Use of the DLM provides the structure required for building the science associated with the disaster-related interventions and provides the information required for accountability and the evidence needed to support standards and best practices.
