Review of Existing Dispensing Practices

Before convening the panel, we conducted a review of mass dispensing practices in CRI sites. RAND and CDC DSNS staff collected information on current POD infrastructure, plans, and operations from 19 of the 21 original CRI sites (2 sites declined to provide data) to learn about current POD planning and the considerations and tradeoffs that inform POD planners in addressing POD location, staffing, operational, and security issues. It was originally envisioned as an effort to collect best practices, but in the absence of clear evidence regarding outcomes that would result from these practices, it was difficult to declare any practice to be “best.”

Mathematical Models

We also used mathematical models of POD operations and of POD locations to frame discussion of “what if ” questions about what levels of population coverage could be achieved if cities were to conform to various proposed standards and under various assumptions about communities' geographical characteristics.

Expert Panel

The expert panel included 13 representatives from federal, state, and local health departments, emergency management agencies, and security agencies, and included a blend of subject-matter expertise on countermeasure dispensing and practical experience with health departments (a list of panelists appears at the end of this article). The panel commented on the 4 POD standards areas and was provided with data from the surveys and mathematical modeling to inform the discussion. Ideally, panel deliberations would be backed with evidence linking practices of mass countermeasure delivery to outcomes of reduced morbidity and mortality, the type of research synthesis typically provided to panels on clinical or public health interventions. Although such evidence is unavailable, we were able to present information that helped panelists weigh tradeoffs in stringency and uniformity against flexibility and practicality.

For instance, deliberations about internal POD operations were informed by model-based predictions about the number and composition of staff required by various levels of care. This helped panelists weigh tradeoffs between level of care provided (benefits) and staffing requirements (costs). Use of the models also helped ensure that the resulting standards were aligned with the rather aggressive 48-hour goal of the CRI. Technical detail on the models and the results of the modeling can be found elsewhere.Reference Nelson, Chan and Chandra11

After the expert panel meeting, a first draft of the standards was critiqued by CDC DSNS staff and then by the expert panel. CDC DSNS then distributed a revised version of the draft standards to all 72 CRI sites for review and comment. We received 38 sets of written comments from state and local health departments in 26 states and oral feedback from some 4 dozen individuals during a pair of 2-hour teleconference sessions. The standards were finalized after consulting with key staff from the Health and Human Services Office of the Assistant Secretary for Preparedness and Response and CDC DSNS.

Uniform Requirements

The first category of standards imposes a single, uniform requirement on all awardees, regardless of community characteristics. Standards that fall into those categories are less flexible and build on observable outcomes and results of modeling. Given the lack of data from randomized controlled trials and strong comparison group studies based on real incidents, we used our mathematical models to extrapolate from observed evidence to predict outcomes in situations that have not been observed directly. For instance, epidemiological models conclude that delivering countermeasures to affected individuals within 48 hours would likely prevent ≥95% of anthrax cases in a metropolitan population.Reference Wilkening13 Reference Wein, Craft and Kaplan14 These models were used to drive the overall CRI target of full-community dispensing within 48 hours, which is an example of a uniform requirement.

Consistency Standards

In some instances, the evidence base is not sufficient to support a single uniform requirement, but it is strong enough to mandate internal consistency among planning elements. These standards prescribe a set of mathematically defined relations among infrastructure elements but leave it to grantees to select which combination of elements is best for their jurisdictions. For instance, simple arithmetic indicates that 10 PODs processing 500 people per hour would, other things being equal, produce the same level of operational output as 20 PODs that process 250 people per hour. Therefore, instead of prescribing a set number of PODs or a required minimum throughput at all PODs, standard 1.2 (Appendix) ensures internal consistency between the number of PODs and other critical planning elements by requiring that jurisdictions' plans adhere to the following mathematical relation:

Analytical Standards

In other instances, some knowledge exists but is incomplete. Consequently, expert judgment is required but can be informed by modeling results and other forms of analysis. Given the paucity of real-world experience with mass countermeasure delivery, much of the information considered by the panel fell into this category. These analyses helped panelists to weigh the tradeoffs between the benefits of requiring high levels of care at PODs and low travel distances to PODs (as assessed by their judgment) vs the costs of setting up and staffing the PODs (as assessed by our models), but they could not point directly to specific standards. In most such instances, this led panelists to select standards that require jurisdiction to undertake an auditable analytical process, but that do not prescribe specific plans or actions.

For example, when considering standards for the number and location of PODs, the panel was hampered by a lack of evidence regarding the benefit of reducing patient travel distance to PODs. But it was able to calculate the cost (in terms of additional POD sites) of imposing minimal travel distance requirements. Applying mathematical location models to 3 “case” metropolitan areas showed that a standard on maximum travel distance to PODs could be easily met in a dense urban area. However, the same standard, applied to a larger suburban area, would force a jurisdiction to open large numbers of sparsely attended PODs, potentially wasting resources.

Process Standards

In some cases, the information available could be at best described as requiring a judgment call because there was no basis for modeling or relatively few data. These panel discussions were informed by the aforementioned survey data on current practice at CRI sites. On the basis of this review, panelists concluded that these standards should simply focus on (nonanalytical) planning processes. For instance, rather than enumerating all of the security requirements at PODs, standard 4.1 (Appendix) requires health department planners to have consulted with security officials in drafting their POD security plans.

In some instances, a combination of weak evidence and failure to achieve consensus-produced standards were not well categorized into 1 of the 4 groupings described above. For instance, standard 4.3, alternative 1 (Appendix) requires the presence of at least 1 law enforcement officer at each POD location. This recommended standard, which was strongly endorsed by the security experts on the panel, would be categorized as a uniform requirement, even though the outcome evidence was less than clear. Because of the lack of evidence and consensus, the process also produced an alternative version (standard 4.3, alternative 2), which did not require the physical presence of law enforcement at each POD.

Consistency, Analytical, and Process Standards Can Help Strike a Balance Between Uniformity and Flexibility

An important function of performance standards is to reduce unwarranted variability. With POD infrastructure, however, it appears that there is a considerable amount of warranted variability, which argues for a considerable degree of flexibility. Standards might address the issue of local variation by focusing on outcomes or outputs, holding jurisdictions or other service providers accountable for demonstrating (through exercises or small-scale incidents) a certain level of operational capability but allowing them to use whatever infrastructure configurations can achieve those goals effectively, efficiently, and reliably.Reference Coglianese, Nash and Olmsted16 Fair enforcement of operational capability standards, however, would almost necessarily rely heavily on the ability to measure operational capabilities. Although considerable progress has been made, the science of public health preparedness measurement is still in its infancy. Thus, in the near term, standards likely must focus to a significant extent on infrastructure configurations. We believe that the types of standards presented in this article (eg, consistency standards, analytical standards, process standards) provide a reasonable approach to finding the right degree of flexibility in national standards.

Nature of Evidence Behind Standards

Given the state of the science in public health preparedness, the congressional mandate for evidence-based standards will be difficult to achieve if “evidence-based” implies the standards of proof that are normally required for clinical and other public health interventions. Thus, it is necessary to take immediate action to improve the evidence base for public health preparedness, including the development of a more systematic approach to collecting exercise-based performance data and information on response processes. This action will facilitate more systematic identification of best practices around which to craft standards. Such data could also support more vigorous attempts to further develop and validate the kind of computer models used in the development of the POD standards.Reference Nelson, Beckjord and Dausey17 Research that may inform the development of these data systems could be undertaken by, among others, the PAHPA-mandated preparedness and emergency response research centers, which conduct public health systems research on preparedness and response capabilities at the national, state, local, and tribal levels.

Given the congressional mandate, standards development cannot be put off until the evidence base has matured. Expert panels and consensus-based methods will remain important standards-development methods for the foreseeable future. However, as this article demonstrates, consensus-based approaches can be guided and supplemented by systematic analysis, if not always by direct empirical evidence from responses. Countermeasure delivery is unique in the extent to which key processes can be represented and modeled mathematically, but it is far from the only such capability; other standards may be usefully informed by disease progression models and behavioral responses to public health interventions.Reference Moore, Chan and Lurie18 Reference Fowler, Sanders and Bravata19 Reference Kaplan, Craft and Wein20 Reference Bozzette, Boer and Bhatnagar21 Additional lessons for standards may be gleaned from smaller-scale proxy events, such as routine outbreaks of food- and waterborne disease.Reference Nelson, Beckjord and Dausey17 Reference Rendin, Welch and Kaplowitz22 Reference Stoto, Dausey and Davis23

Need for Additional Policy Guidance

Even with better data, however, it is unlikely that standards development will ever be fully evidence driven. Policymakers must be prepared to make decisions about how much preparedness (and therefore stringency in standards) is worth paying for and how to weigh the benefits of various preparedness investments. They must also be prepared to make tough decisions about how much to hold public health agencies accountable for the actions of other actors. For instance, debate over alternate versions of standard 4.3 raised the vexing issue of whether health departments should be held accountable for the ability and willingness of law enforcement agencies to assign at least 1 officer to each POD.

Importance of Incorporating Stakeholders Into Standards Development

Finally, the POD standards development process demonstrated the importance of involving stakeholders in it. Gaining some level of buy-in is likely to promote more workable standards and higher degrees of compliance. Development of standards can often be contentious, especially when the stakes may be high. Although the POD standards process often led to heated debates, anecdotal evidence suggests a broad degree of acceptance among grantees. However, policymakers and others should bear in mind that extensive stakeholder engagement can add considerably to project timelines.