Although health care institutions regularly perform rigorous quality assessments of routine clinical and administrative services, few metrics are available for health care institutions to evaluate the quality of their emergency management initiatives.
In assessing hospital emergency management (HEM) performance, institutions must prepare for events that rarely occur while simultaneously attempting to mitigate the likelihood that they will occur at all. Because of the infrequent nature of major disasters, health care organizations remain ill equipped to systematically evaluate the strengths and weaknesses of their emergency management programs. In addition to the key role of hospitals as stakeholders in regional and global emergencies,Reference Berman and Lazar1 societal expectations of hospitals during times of disaster have shifted considerably. Health care institutions are expected to provide emergency care regardless of the volume and demand, but recently they have also become sites of community refuge, bastions of safety in a threatening and dangerous environment. The public perceives that hospitals will have light, heat, air conditioning, water, food, and communications capabilities, regardless of the fact that the institution may itself be a “victim.” In the terrorist attacks of September 11, 2001 and the northeast blackout of 2003, for instance, the public flocked to hospitals even when they did not require medical care. Finally, the concept of health care institutions as a potential terrorist target has forced institutions to focus limited resources on safety and security rather than assess comprehensive emergency management efforts.
STATEMENT OF THE ISSUE
In 1999 the Institute of Medicine's seminal report To Err is Human: Building a Safer Health System Reference Kohn, Corrigan and Donaldson2 outlined a comprehensive strategy “by which government, health care providers, industry, and consumers could improve overall health care quality by noting that poor quality is caused by faulty systems, processes, and conditions that lead people to make mistakes or fail to prevent them.” Since the publication of the report, significant advances have been documented in a variety of areas, including treatment of hospital-acquired infections and development of rapid response teams and Centers of Excellence,Reference Lenape and Berwick3 but no similar strategy exists for these institutions to assess, and thereby improve, their capabilities in emergency management. Existing metrics such as the Federal Health Resources and Services Administration's critical benchmarks and sentinel indicators for its Bioterrorism Hospital Preparedness Program have not been fully validated and are not evidence based. The recent focus of The Joint Commission on revamping its emergency management standards provides an impetus to strengthen HEM performance measures; however, there is a lack of specific guidance. Given the recent push for pandemic influenza preparedness, the Centers for Disease Control and Prevention has developed 23 performance measures related to its cooperative agreement and supplemental pandemic influenza guidance. Although helpful, more emphasis on standardizing best practices and measures that relate to HEM is required. Congressional funding for hospital emergency management since 9/11 has been in excess of $3.6 billion,4 yet uniformly accepted performance measures to judge the effectiveness of these efforts do not exist.
Lurie et alReference Lurie, Wasserman and Nelson5 note that, “the lack of well-accepted, standardized measures and metrics makes it difficult to satisfy the demands for accountability, or gauge the level of preparedness.” Moreover, years after 9/11, there are still few defined performance standards for state bioterrorism and emergency public health preparedness programs and activities.Reference Lurie, Wasserman and Nelson5 BirnbaumReference Birnbaum6 echoes these ideas, pointing out that, “A major problem affecting the outcome of disaster health care is the lack of internationally accepted standards of performance for disaster health management and response.” Most striking is an article by Nelson et al,Reference Nelson, Lurie and Wasserman7 who argue that, “the situation is not because of a shortage of measures of preparedness,” given that numerous entities have crafted definitions of preparedness, but that the only consistency across them is inconsistency.
The goal of this article, therefore, is to offer recommendations on how health care institutions can apply traditional quality principles to the assessment of HEM efforts, and to use innovative analytic methodologies to develop comprehensive approaches to performance measurement in HEM.
THE HUMAN FACTOR: ADVANTAGES OF EXTERNAL ASSESSMENT OVER SELF-ASSESSMENT
The reluctance of health care institutions and professionals to document shortcomings as the beginning of improvement development is an important factor that influences how practitioners and researchers alike can measure health care emergency management efforts. The National Incident Management System clearly describes the expectation that every emergency drill or exercise, and every actual emergency activation, should be followed by a rigorous critique of performance, identification of needed improvements, and assignment of responsibility for corrective action. There is little evidence, however, that this process is routinely followed beyond the small circle of individuals directly involved in ensuring that minimum drill frequency is met. Anecdotally, experience with crafting after-action reports following drills, exercises, or actual incidents almost always includes substantial editing by involved parties to minimize gaps in the plan or failures of performance, and to create the shortest possible list of needed corrective actions. Agreement to curative steps at the highest level of the institution is often described as part of the emergency preparedness coordinator's wish list, rather than a routine part of HEM.
INTERVENTION VERSUS INTERROGATION: ADVANTAGES OF PROFESSIONAL OVER UNPROFESSIONAL CRITICISM
The constant interrogation regarding “are we ready?” by government, the media, and health care institutions is a potent stimulus to redouble our HEM efforts, despite being hampered by the lack of rigorous assessment capabilities. Furthermore, little has been done in the way of rigorous performance assessment of the response to actual disasters despite the occasional tabloid accusations of failure. This lack of rigorous self-evaluation by the health care community is a problem on several counts. Most important, a given entity is limited in its self-improvement efforts because assessment of both readiness and response remains highly subjective. Leaders and governance bodies of these organizations are forced to rely on reports from emergency managers, who understandably have a significant self-interest, and may underreport issues of concern. Benchmarking across institutions is not possible due to a lack of standardized performance metrics. Finally, options for best practices sharing are limited to the perceived value, rather than the actual value, of particular initiatives. The risk in the current environment is obvious. Whereas hazard vulnerability analyses often accurately identify gaps and areas for improvement, institutions remain susceptible to being quickly overwhelmed, and possibly incapacitated, by large-scale, regional events.
VALUE MEASUREMENTS
The value of measurement in this area is self-evident. It is, after all, difficult to manage, let alone improve, what one does not measure. As The Joint Commission8 notes, “America's communities, its public health infrastructure, and its health care delivery system are literally living far closer to the brink of disaster than they have since the turn of the 20th century.” To best manage our crucial resources, we must analyze our health care institutions' emergency management activities in as rigorous and objective a way as possible. Clearly, the divergent nature of hospital emergency management compared to traditional “routine” health care requires extrapolation of basic health care quality principles, as well as creative approaches when standardized ones are not applicable.
DONABEDIAN'S QUALITY CONSTRUCTS
In his seminal works on health care quality, DonabedianReference Donabedian9, Reference Donabedian10 advanced the idea that traditional health care quality improvement measures revolve around the concepts of structure, process, and outcome. More recently, volume has become an important predictor of clinical outcomes. Although we believe that this categorization can be applied to HEM functions, there are some fundamental differences when compared with traditional health care metrics, as outlined in Table 1.
Comparison of Performance Metrics in Traditional Health Care vs Health Care Emergency Management
Volume as a metric underscores the principle that increased frequency of a task or procedure improves quality. Volume is rooted as a quality metric principally in procedural areas such as surgery. Fortunately, in hospital emergency management, volume (ie, the number of actual events) is extremely low, and this diminishes the value of volume as a significant performance metric in HEM. Some have used frequent training, drills, and exercises as reasonable proxies, but even when conducted often, annual volumes rarely exceed single digits. Thus, the principle of volume metrics may not be fully applicable to HEM or response. There may, however, be surrogate metrics that hospitals can analyze (eg, intensive care patient volumes, emergency department visits for major trauma, patients brought in via ambulance). When evaluating these proxy metrics on an institutional basis, low volume may not be correctable. For instance, how does an institution increase its volume of ambulance visits or care for trauma patients without trauma center designation? Moreover, volume standards in traditional health care are nascent, traditionally being empiric. It is only recently that actual data relating to volume and outcomes have become available. Because the gold standard for emergency management is performance during a disaster, rigorous data relating nondisaster volume performance proxies to outcomes during disaster does not exist. Nevertheless, it seems reasonable to suggest that facilities without significant volume in areas such as patients arriving via ambulance or trauma cases would not be able to respond to a large mass casualty incident as well as a trauma center or busy emergency department. This may be due to numerous factors including lack of specific clinical resources such as trauma clinicians, minimal experience with trauma care, or the need for specific equipment.
There may be partial solutions. For example, a small, rural emergency department could collaborate with a large, urban center to rotate personnel, thereby increasing the breadth of experience for each. Or a facility could simply increase their drill and exercise frequency. Sites also can examine institutional choke points (eg, which resource categories, such as physicians or nurses, limit expansion of surge capacity). It is fair to say that volume is the metric least applicable to HEM, and even when proxies are assessed, low volume situations are difficult to remediate.
Structure measures in health care quality are binary metrics relating to facilities, plans, and procedures. Most of the limited HEM metrics available are of the structural category (eg, whether a site has a dedicated staff position serving as emergency management coordinator, whether it has a decontamination facility). Structural metrics are often the most easily addressed but frequently have little evidence-based rationale for improved HEM performance. To apply these elements to HEM, a combination of binary and scaled metrics must be developed that encompass the all-hazards approach used in emergency management efforts nationwide. For example, if a site has a decontamination facility (ie, it responds yes to the binary question), scaled metrics (eg, the number of patients it can handle simultaneously, for instance 1–5, 6–10, 11–15, etc), can provide additional detail as to the capabilities of the site.
In health care, outcome measures traditionally have focused on morbidity and mortality. Over the last decade, the focus has shifted to include additional indicators such as quality of life and functional outcomes. Evaluation of actual outcomes, during disaster situations, poses significant challenges including lack of frequency, differences in type and scale of the incident, variables such as triage and prehospital care, and a lack of longitudinal and benchmarking comparison data.
Process measures pertain to activities for which there is solid evidence indicating that they will improve outcomes (e.g., the administration of aspirin and beta blockers to patients experiencing an acute myocardial infarction). In HEM, process data may include time to triage, emergency department length of stay, and door to operating time, whereas outcomes data would include morbidity and mortality.
POTENTIAL SOLUTIONS
Conceptually, there are relatively frequent occurrences that, although not true disasters, approach the level of chaos, increased patient volume, and resource demand that are seen with catastrophic events. For example, emergency departments frequently experience periods of extreme volume and demand for services. Traditionally, these are viewed as outlier periods and often ignored or aggregated along with routine-period data to minimize attention to problems. We suggest that a specific focus on outlier periods may most closely replicate disaster situations and thus should be evaluated in terms of outcomes and process. For example, applying traditional quality metrics (eg, emergency department length of stay) to outlier periods (eg, during a blackout) and comparing those findings with local, regional, and even national peers can provide opportunities to replicate disaster incidents. Such metrics may also include waiting times, traditional process measures such as Center for Medicare & Medicaid Services core measures, elopement rates, and missed diagnoses. Admittedly, performance targets and metric definitions may be different during a disaster as compared with normal periods. For example, although wait times may be lengthened during these periods, they can nonetheless prove useful when sequentially compared or benchmarked against other institutions during the same times of system stress. What is important is that performance standards can be established for both normal and disaster times to ensure uniformity across institutions, systems, and regions.
Central to the ability to effectively use quality data is the need to specifically and rigorously define each metric. Data sources, indicators of compliance, and inclusion and exclusion criteria must be explicitly delineated as data definitions. Here, too, normal and disaster definitions must be developed to ensure applicability across the countless incidents that hospitals face.
We suggest that collection protocols ensure that uniform data acquisition can occur and, as is vital to the success of any quality endeavor, comparison groups be established, longitudinal for the same institution over time and horizontal across institutions. Hospitals, industry groups, and public and private players must reach consensus on how to accomplish this formidable yet necessary task.
Finally, adherence to Donabedian's constructs Reference Donabedian9–Reference Donabedian11 of structure, process, and outcomes, enhanced by the incorporation of volume as an additional category of metric, can aid in developing assessment instruments for health care emergency management activities. Even so, key questions to ponder remain. First, what constitutes emergency management? Should quality in health care emergency management be defined? Who should measure it? What should they measure? What level of care is to be expected (maximal versus optimal)? Where should they measure it? From an all-hazards perspective? From a hazard-specific view? When should the measurements occur? These are but a few of the outstanding issues as we seek the answer to the question, “What will it take for health care institutions to meet the most pressing emergency management challenges?”
CONCLUSIONS
Ultimately, the field of HEM must advance achievable recommendations in performance measurement that will guide resource allocation. It is imprudent to think that the continued dedication of resources to these efforts without agreed upon, uniform methods to analyze their effectiveness can continue. The need for universally accepted, evidence-based performance measures continues to grow, and without such measures, hospitals will be unable to demonstrate their progress or needs. The current notion of “percentage of completed grant deliverables” often serves as a key structural performance metric, but using structural metrics as the sole criterion of institutional readiness has significant shortcomings. Nor is it likely that generic emergency management experts will ever arrive at a meaningful approach to measuring hospital readiness for and performance during emergencies.
We suggest that a more rigorous and comprehensive assessment be instituted by applying traditional health care quality metric paradigms to develop performance measures in HEM. The Pandemic and All-hazards Preparedness Act of December 2006 is encouraging. The act requires localities to create preparedness initiatives consistent with “measurable, evidence-based benchmarks and objective standards.”12, Reference Hodge, Gostin and Vernick13 This is an important step that supports the need for evidence-based preparedness policies that must model the proven effectiveness of current health care quality improvement programs, as well as the need to advance this critical aspect of our nation's preparedness.
Authors' Disclosures The authors report no conflicts of interest.
