Hostname: page-component-7b7567c8c-7pwfn Total loading time: 0 Render date: 2026-11-15T22:34:04.130Z Has data issue: false hasContentIssue false
  • English
  • Français

School Closure Decisions Made by Local Health Department Officials During the 2009 H1N1 Influenza Outbreak

Published online by Cambridge University Press:  17 July 2015

Harvey KaymanOpen the ORCID record for Harvey Kayman[Opens in a new window],
Sarah Salter,
Maanvi Mittal,
Winifred Scott,
Nicholas Santos,
Diana Tran and
Ryan Ma
Harvey Kayman*
Affiliation:
Department of Epidemiology, University of California School of Public Health, Berkeley, California
Sarah Salter
Affiliation:
University of California Goldman School of Public Policy, Berkeley, California
Maanvi Mittal
Affiliation:
University of California, Berkeley, California
Winifred Scott
Affiliation:
Department of Epidemiology, University of California School of Public Health, Berkeley, California
Nicholas Santos
Affiliation:
University of California, Berkeley, California
Diana Tran
Affiliation:
University of California, Berkeley, California
Ryan Ma
Affiliation:
University of California, Berkeley, California
*
Correspondence and reprint requests to Harvey Kayman, MD, MPH, 1301 Quarry Court #404, Richmond, CA 94801 (Hkay20@berkeley.edu).
Rights & Permissions [Opens in a new window]

Abstract

Objectives

The goal of this study was to gain insights into the decision-making processes used by California public health officials during real-time crises. The decision-making processes used by California public health officials during the 2009 H1N1 influenza pandemic were examined by a survey research team from the University of California Berkeley.

Methods

The survey was administered to local public health officials in California. Guidelines published by the Centers for Disease Control and Prevention had recommended school closure, and local public health officials had to decide whether to follow these recommendations. Chi-squared tests were used to make comparisons in the descriptive statistics.

Results

The response rate from local public health departments was 79%. A total of 73% of respondents were involved in the decision-making process. Respondents stated whether they used or did not use 15 ethical, logistical, and political preselected criteria. They expressed interest in receiving checklists and additional training in decision-making.

Conclusions

Public health decision-makers do not appear to have a standard process for crisis decision-making and would benefit from having an organized decision-making model. The survey showed that ethical, logistical, and political criteria were considered but were not prioritized in any meaningful way. A new decision-making tool kit for public health decision-makers plus implementation training is warranted. (Disaster Med Public Health Preparedness. 2015;9:464–471)

Keywords

Boyd (OODA) loopdecision-makingorganizationalemergency respondersethicsintuition

Information

Type
Original Research
Information
Disaster Medicine and Public Health Preparedness , Volume 9 , Supplement 4 , August 2015, pp. 464 - 471
Copyright
Copyright © Society for Disaster Medicine and Public Health, Inc. 2015 

Public health officials (PHOs) routinely make decisions affecting the health and safety of the communities they serve, and the decisions they make during crisis situations can dramatically impact the lives of their constituents. Whether providing life-saving services during a natural disaster or implementing communicable disease control measures during an epidemic, PHOs can save lives and prevent the spread of disease. However, the decisions made by PHOs also have the potential to curtail individual freedoms, create unfair distribution of benefits and burdens, and neglect some of the most vulnerable populations.Reference Cauchemez, Ferguson and Wachtel 1 The standard emergency response commitment to the utilitarian doctrineReference Driver 2 of “the greatest good for the greatest number” 3 can have unanticipated consequences. Given the broad and possibly unpredictable impact of the decisions made by public health professionals, it is important to understand how such decisions are made.

The 2009 H1N1 influenza pandemic was, at the time, considered to be a major, global public health threat. On April 15, 2009, the Centers for Disease Control and Prevention (CDC) reported the first case of H1N1 in the United States. 4 Within 2 days, a second infection was confirmed and a preemptive public health response began. Our survey research team (SRT) focused on this potential public health threat to identify the decision-making processes used by PHOs as they determined whether to close schools in accordance with the CDC’s guidelines.

METHODS

In 2011 to 2012, we developed a survey instrument to assess how California PHOs made their decisions regarding school closures in 2009. The survey was designed to document the respondents’ recall of what they did in 2009 on the basis of our assumption that memory was a measure of impact and importance to the respondents.

We reviewed 40 articles on ethics and over 100 articles on various aspects of decision-making and performed an extensive literature review on the topics of surge and crisis preparedness management and business, economic, psychological, cultural, and educational aspects of emergency response. The information in these articles helped to form the basis for some of the questions in the survey.

Study Population

The primary population of interest was health officials at local public health departments (LPHDs) in the State of California. Owing to the variety of administrative structures within LPHDs, both health officers (Chief Executive Officers) and non-health-officers took part in the decision-making process. As a consequence, we included deputy public health officers, communicable disease control officers, and public health emergency preparedness (PHEP) managers as survey participants. The research team chose contacts to survey from lists of PHOs employed in LPHDs in 2012. On the basis of the narratives in the survey results, the SRT estimated that about 10 additional officials completed the survey, although they were not included on the 2012 lists.

Recruitment and Marketing

To recruit survey subjects, the SRT marketed the survey at meetings and conferences attended by local California PHOs. The team used contact information obtained through these meetings, phone numbers and e-mail addresses with officials at county and city health departments, and other contacts to reach out to potential survey participants. The SRT directly contacted all listed potential survey-takers. Potential participants were sent e-mails that included a link to a website where they could take the study survey.

Survey Design

The survey questions were developed by using input from public health researchers, epidemiologists, and local public health practitioners as well as from published sources cited in the bibliography. The survey instrument was not pretested.

Demographic and background questions asked respondents to report their years of public health experience, training in decision-making, and their job description in 2009, as well as their department’s location and jurisdiction size. The participants’ responses to whether they were involved in the 2009 decision-making process determined the questions they were asked next. Those involved in the actual decision-making were asked about the processes used to make their decisions. Those PHOs who were not actually involved in the 2009 decision-making process were asked about processes and criteria they would have used if they had been decision-makers. These 2 levels of questioning allowed for comparisons of actual decision-makers versus “would-be” decision-makers during the analysis. Questions on 14 prelisted criteria were grouped into categories of ethical, logistical, and political concerns. Respondents were also asked to identify the role of peer participation in the decision-making process, ie, from one person alone, to one person with input from others, to arriving at a group consensus. Finally, we asked participants to gauge their familiarity with the Boyd or OODA (Observe, Orient, Decide, and Act) loop in decision-making. The steps in the Boyd or OODA loop encourage decision-makers to investigate what has happened, gather information about what happened, make a decision, and, finally, take action.Reference Angerman 5 - Reference Boyd 7

Analysis

We used Qualtrics, an online survey instrument, to collect the study data. 8 An SRT member tabulated descriptive statistics and performed chi-squared or Fisher’s exact tests to compare categorical variables and t-tests to compare continuous variables. We considered a two-tailed P value <0.05 to be statistically significant. An SRT member analyzed the survey data by using Stata 12 software. 9

RESULTS

Respondents represented 47 of the 58 county LPHDs and 1 of the 3 city public health departments within California, a response rate of 79%. The individual-level response rate was estimated to be 42%; of the 160 individuals who received the survey, 67 completed at least 85% of the questionnaire and were included in the data analysis (Table 1). Survey participants were approximately equally distributed among health officers, PHEP managers, and other public health workers. The research team received more than one response from 14 LPHDs. The SRT included all data collected even for those individuals who worked in the same LPHD.

TABLE 1

Characteristics of the Survey ParticipantsFootnote a

a Abbreviation: PHEPs, public health emergency preparedness managers.

The distribution of the respondents was consistent with LPHD sizes and geographical areas throughout California. A total of 30% of respondents were health officers and 36% were PHEP managers; the remaining 34% included respondents with other job titles, such as communicable disease control officer, deputy health officer, nursing manager, and epidemiologist. Four respondents who reported more than one job title were coded as PHEP managers for analysis.

The positions held by the respondents, as well as other characteristics, such as training in decision-making, years of public health experience, and status as decision-makers in May 2009 are shown in Table 1. The respondents were asked to suggest answers not included in the list of choices to enhance the information given in the listed answers. The respondents wrote narrative answers to the questions about training, methods, processes, prioritization, and satisfaction. Many narrative answers provided insights and study-relevant information.

Training in Decision-Making

A total of 23 respondents reported having received training in decision-making. However, a review of the written narrative descriptions of their training revealed that many of the respondents had not actually had specific training in decision-making. Because military personnel may take a formal military decision-making course, respondents who said they had military experience were considered as actual decision-making trainees.Reference Lieberman, Jarcho and Satpute 10 - 12 Even using these loose criteria, only 13% of the respondents were deemed to have potentially completed a training course in decision-making. Most decision-makers were public health officers (P<0.001). We found that a self-reported job title of public health officer was associated with a greater probability of having received decision-making training and that greater years of public health experience (>5 years vs. ≤5 years) was associated with a greater likelihood of being involved in the local decision-making process (both P<0.05).

Decision-Making Methods and Processes

The percentages of respondents who recalled using various decision-making processes in response to the 2009 influenza threat are reported in Table 2. Respondents were asked about 8 processes determined by the SRT that they might have used to make their decision; they could choose none, some, or all of the items listed. There was also a category for including additional processes not listed among the response options in the check boxes. Fifty-six percent of the respondents from LPHDs reported using a decision-making method in which one person made the decision with input from others, 29% reported using group consensus to make decisions, and 4% did not report what method was used. In the multiple-choice section, no respondent reported that a person in authority made the decision alone, although in their written comments, several respondents indicated that a single authority made the decision alone. Most, but not all, of the respondents reported using the same decision-making methods as their coworkers.

TABLE 2

Processes Used or Considered by Survey ParticipantsFootnote a

a Abbreviation: SOP, standard operating procedure.

The most frequently cited process for decision-making, named by 69% of respondents, was use of one or more of the predetermined list of criteria. Approximately 50% recalled using criteria not listed.

In their narrative responses to this question, respondents commented on processes that they used that we had not listed (noted as “other” in Table 2). The processes described by the respondents were grouped into 4 categories. These included use of governmental guidance, consideration of the biology and epidemiology of influenza, collaboration, and consideration of consequences.

Approximately half of the respondents indicated that they would have liked or would have used at least one of these other criteria to guide their choices: 28% would have liked to have trustworthy government guidance, 44% wanted to have more evidence so they could consider the biology and epidemiology of influenza, 16% would have liked more opportunity for collaboration with experienced colleagues, and 12% wanted to have a clearer idea about the consequences of their decisions. Standard operating procedure was reported significantly more often by would-be decision-makers than by actual 2009 decision-makers as a process used to make decisions.

No one reported using calculation processesReference Gilmour, Hanna and Mc Keever 11 , Reference Manketlow 13 - 16 or decision-making software technologies, and only a small percentage reported using rules of thumb (heuristics) or “gut feelings.” Of the processes included in the survey, lists of criteria and lists of the pros and cons were the most frequently selected by those involved in decision-making. A number of the preselected choices included unconscious use of intuition,Reference Kahneman 17 , Reference Khatri and Alvin 18 experience,Reference Kahneman and Klein 19 , Reference Klein, Snowden and Pin 20 heuristics,Reference Gigerenzer and Gaissmaier 21 , Reference Schwenk 22 rules of thumb, 23 and “gut feelings.”Reference Higgins, Strange and Scarr 24 , Reference Mikels, Maglio and Reed 25 These options were seldom chosen by the respondents.

Decision-Making Criteria

The survey asked about use of 14 preselected ethical, logistical, and political criteria for decision-making.Reference Beauchamp and Childress 26 - Reference Lee 29 Respondents reported whether each criterion was used in their decision-making process. Table 3 shows the criteria grouped by category with the responses from decision-makers and would be decision-makers. Eighty percent or more of the actual decision-makers reported that 9 of the 14 criteria affected their decisions. Approximately 90% stated that consideration of accountability, administrative feasibility, proportionality, and legality contributed to their decisions. All of the decision-makers reported that consideration of the public good (an ethical criterion) and necessity (a logistical criterion) contributed to their decisions. About half of the decision-makers reported that 3 of the 14 criteria were not used in their decision-making: 2 ethical criteria (fair distribution of benefits and burdens and public participation) and the political criterion (political pressure).

TABLE 3

Decision-Making Criteria to Guide PrioritizationFootnote a

a Criteria included 14 criteria in 3 categories and 1 open-ended criterion.

“Would-be” decision-makers differed somewhat from actual decision-makers in their consideration of the criteria. Decision-makers considered their role as a responsible authority figure of greater decision-making importance compared with would-be decision-makers, whereas would-be decision-makers ranked vulnerable populations more highly as a criterion for decision-making. Similar proportions of the 2 groups, however, reported using the criteria of public good, necessity, and accountability (all P<0.05).

Satisfaction With Decision-Making Process

Survey participants who were either decision-makers or otherwise involved in the decision-making process reported being satisfied with the process (Table 4). Overall, 88% reported being satisfied with the decision-making process, including 100% of the actual decision-makers. However, narrative comments indicated that some of the participants were dissatisfied with political pressure having a role in the process. Numerous respondents (88%) expressed satisfaction with the process, judging with hindsight that their decisions were appropriate given the circumstances and available information.

TABLE 4

Satisfaction Among Decision-Makers and Those Involved in the Decision- Making Process

Future Training and Decision-Making Tools

Participants reported that a number of tools would help them and their departments to make better decisions. The preselected training tools listed in the survey and the percentages who stated they would use them, if available, are shown in Table 5. A majority of respondents reported the desire to have a checklist of criteria as well as training and workshops.

TABLE 5

Training and Decision-Making Tools Survey Participants Would Like to Have in the Future

The final survey question posed the hypothetical situation of a potential influenza epidemic in a school. Participants were asked to respond to the event as the LPHD leader. The question was designed to assess whether the respondents would know from training or sense intuitively that a decision-maker should get more information as the step after hearing of the event and before deciding what to do, and the extent to which they would comply with the organized approach to decision-making exemplified by the Boyd or OODA loop.Reference Huder 6 , Reference Boyd 7 Only 54% of respondents chose the option to obtain more information before making a decision or taking action.

DISCUSSION

Most decision-makers as well as others involved in the process reported being satisfied with the techniques they used. However, approximately half of the participants stated that they would like to receive formal decision-making training. Although 73% of the survey respondents were involved in the decision-making process, only 13% had received such formal training. Interestingly, several respondents whose narratives revealed their lack of training in decision-making nevertheless indicated in their check-box responses that they had received such training. The lack of appreciation of the difference between actual training on decision-making processes and perception of decision-making training supports the need for comprehensive training for PHOs on decision-making in crises. Almost half of the respondents chose the “wrong” OODA option, further supporting a likely benefit from specific training in decision-making. Given that health officers and other PHOs need to be able to make decisions during crisis situations, future decision-making tools should be accessible to all relevant personnel in advance of any public health crisis. We suggest that a training tool kit be developed for LPHDs.

LPHDs in the 2009 example used a decision-making method that involved input from multiple individuals. Approximately 29% of LPHDs used group consensus, which requires discussion, the dual process model,Reference Haidt 30 and a shared mental model.Reference Van den Bossche, Gijselaers and Segers 31 - Reference Tuckman 33 The dual process model is described in detail by Haidt in his 2001 article about the “social intuitionist” model. This model describes how decision-makers unconsciously use intuition yet explain post hoc the rationales they believe determined their decisions. Haidt explains his belief that decisions always use a dual process model that incorporates both the use of intuition and the use of deliberation.

Survey respondents noted that in addition to decision-making within public health, they also collaborated with school officials. Although decisions may fall under the purview of the LPHD, other agencies and organizations have a stake in the outcome and also need to be involved in the decision-making processes; for these reasons, the SRT proposes that having a clear, transparent process for making decisions and cross-sector training on how to use the proposed framework may increase understanding and trust between groups.

In reporting on the methods used to make decisions (authoritarian, consultation, consensus, delegation), there were no reports on the preselected check-box answers that one person alone made the decision. However, in the written narrative answers, 3 respondents did report that the health officer made the decision alone. This discrepancy strengthens our concern that some check-box answers may reflect “political correctness” rather than accuracy.

Some participants within a single LPHD reported that they had used methods different from the methods used by others in the same LPHD, which suggests that the process may have been uncoordinated or inadequately conceptualized by those involved. Training in the dual process model would bring consciousness of metacognition to PHOs and help them to recognize that they are often already using a dual approach involving both intuition and deliberation, which has been recognized by experts in decision-making as universal.Reference Lieberman, Jarcho and Satpute 10 , Reference Kahneman 17 , Reference Kahneman and Klein 19 , Reference Gigerenzer and Gaissmaier 21 , Reference Schwenk 22 , Reference Haidt 30 Awareness of the unconscious but powerful effect of intuition can make the dual processes more defined and therefore more effective.

The SRT found that decision-makers used multiple processes to make school closure decisions. Lists of criteria and pros and cons were reported as the most frequently utilized methodologies and suggest a basis on which training and tool kits could be built. Actual calculation and computer-based methods were not used, and this suggests that either these processes are not suitable in the public health crisis context or that PHOs are simply unfamiliar with implementing these types of process. The complex, often unquantifiable, nature of public health problems makes using rigid, overly determined, expensive, and relatively inaccessible computer-based calculation methods, like Analytical Hierarchy Processing,Reference Saaty 34 unlikely to be of much practical use in an evolving crisis. Importantly, many decision-makers relied on government guidance and knowledge of the specifics of influenza to make decisions, and any new training method must be cognizant of the reliance on these factors.

The SRT was specifically interested in how decision-makers used ethical, logistical, and political criteria for decision-making. Respondents remembered using many of the criteria included in the survey. Because only 2 of the criteria were used less than half of the time and the lowest percentage not used was 40%, participants seem to have considered most of the criteria (12/14) to be important. Therefore, actual prioritization by decision-makers using criteria similar to these may not occur. On the basis of our findings, we believe that many public health teams likely did not employ a coherent, planned crisis decision-making process. The survey results do not suggest a standard decision-making process and no comments by participants alluded to any set procedures for decision-making. However, “necessity” and “public good” were noted as criteria frequently used in decision-making.

Of note, most decision-makers (60%) did not report that they utilized public participation, although 67% of would-be decision-makers reported that public participation should be considered. Getting input from the community during an emergency can be challenging, but should be considered because it adds credibility to decisions, can convey the message that the concerns of those impacted by decisions are considered, and can build popular acceptance of and support for potentially unpopular actions. The category utilizing public participation focused on the public’s input, whereas the political feasibility question focused on the interaction between the public and the politicians to capture how the divergent views of the public and the political decision-makers were considered. We found conflicting evidence regarding the role of political pressure, with only some respondents welcoming political input.

Eighty percent of “would-be” decision-makers and all actual decision-makers reported being satisfied with the decision-making process, despite not being fully aware of the processes they used. The SRT thinks that having a clear articulated dual process model and a tracking tool to document decision-making processes will make it easier for LPHD staff to review exactly how they made decisions throughout their response to a public health crisis. Having a well-organized, standard, intuition and evidenced-based decision-making process and process tracking tool would likely be useful for PHOs to report how they manage future public health crises.

Health officers are in positions of authority in part because they have had experience dealing with similar crises in the past. Although the respondents did not acknowledge that they used intuition, their decisions depended on a dual process model integrating both intuition and deliberation.Reference Kahneman 17 Intuition gets its power from experience and heuristics. With experience, PHOs may utilize an unconscious process trying to match a pattern from the past to the current situation, frequently checking to see how close the current situation and the pattern match, avoiding common cognitive traps, making sense of the incoming data, and trajectory tracking or making plans based on probable progression of the event.Reference Huder 6 , Reference Klein, Snowden and Pin 20 PHOs who are aware of and document their decision-making methods, processes, and prioritization methods will likely be better able to defend their decisions post hoc by providing transparent descriptions of the bases for their choices.Reference Poland and Marcuse 35 , Reference Field and Caplan 36 The effect of formal decision-making training for PHOs should be tested and is worthy of further study.

Limitations

Challenges to carrying out the project included finding current contact information for potential study participants and determining how many individuals should be included in the denominator in response rate calculations. Our calculation that only 42% of the individuals who saw the survey completed it is a limitation to our inferences. The survey instrument was not pretested. Our conclusions would be more defendable had the instrument been validated. Because the event was so dramatic for public health officials (author’s personal observations of officials at the state and local levels), the SRT assumed that the survey participants would recall the events of May 2009 relating to the H1N1 influenza pandemic even though 3 years had elapsed; therefore, the survey was focused on what the participants remembered as being important to them during the process, not on what they actually did. No respondent reported what they actually did in regard to school closure. Hence, both random and systematic errors of recall cannot be excluded.

No data were collected to validate whether identified processes or criteria stated as used by PHOs were the actual processes and criteria used, and no direct observations or assessments of decision-makers in action were made. Because the participants were answering on the basis of their recollections, the SRT inferred that the results reflect only what the respondents remember as being important. When the survey instrument was created, the criteria were listed as if they were mutually exclusive but may in fact overlap.

Although non-decision-makers could participate, some chose not to and we cannot assume there was no sampling bias among the participating respondents. We assumed that most 2009 decision-makers were still present and working at the same LPHD, which was true generally for respondents, but was probably not true for nonrespondents. The research team also assumed that any decision-making training obtained after 2009 would not affect responses regarding recalled events during the 2009 H1N1 influenza pandemic. Because the survey was only sent to PHOs in California, the results may not be generalizable. Furthermore, because there were multiple responses from some LPHDs, some data may not be entirely independent and the counts for whether certain processes or criteria were included may be higher in some cases because they represent multiple respondents from the same LPHD.

CONCLUSIONS

Public health decision-makers do not appear to have a standard process for crisis decision-making and would benefit from having an organized decision-making model. The survey showed that ethical, logistical, and political criteria were considered but were not prioritized in any meaningful way. The SRT proposes that an organized public health approach be based on the Boyd or OODA loop (Observe, Orient, Decide, and Act). A new decision-making tool kit for public health decision-makers plus implementation training is warranted.

Acknowledgments

Input on the survey design and data analysis was also provided by Maureen Lahiff of the UC Berkeley School of Public Health. Input on original concepts and survey design was also provided by Chris Nelson and Andrew Parker of the Rand Corporation, as well as Bela Matyas of the Solano County Public Health Department and Karen Smith of the Napa County Public Health Department. Ferenc Dalnoki-Veress and Raymond Zilinskas of the Monterey Institute of International Studies also worked with the principal investigator. We also thank the public health officials of California who provided information about how health departments are run and about the focus of the survey. Joel Simon, MD, MPH, provided the critical editing that makes the paper cogent. Lynn Hammond made suggestions about language usage and structure that were invaluable. The 2012 staff at CIDER supported the SRT as we worked on this project.

Funding

Funding for the pilot study was obtained through Cal Prepare, under an existing Preparedness & Emergency Response Research Center (PERRC) grant, and the Center for Infectious Disease Emergency Readiness. P01TP000295 (PI: Tomas Aragon) 01/12-09/12. Full approval and exemption by the University of California Committee for Protection of Human Subjects Institutional Review Board (IRB) was obtained before data collection began. Protocol ID: 2012-07-4471.

Supplementary Material

To view supplementary material for this article, please visit http://dx.doi.org/10.1017/dmp.2015.72

References

1. Cauchemez, S, Ferguson, NM, Wachtel, C, et al. Closure of schools during an influenza pandemic. Lancet Infect Dis. 2009;9:473-481.Google Scholar
2. Driver, J. The History of Utilitarianism. Stanford Encyclopedia of Philosophy website. http://plato.stanford.edu/archives/sum2009/entries/utilitarianism-history/. First published March 27, 2009. Accessed June 18, 2014.Google Scholar
3. ForestService. The Greatest Good for the Greatest Number. A Forest Service Centennial Film. Forest Service website. http://www.fs.fed.us/greatestgood/press/mediakit/facts/pinchot.shtml. Accessed June 18, 2014.Google Scholar
4. Flu.gov. History of H1N1 and other pandemics. Flu.gov website. http://www.flu.gov/pandemic/history/. Accessed October 27, 2012.Google Scholar
5. Angerman, WS. Coming Full Circle With Boyd’s OODA Loop Ideas: An Analysis of Innovative Diffusion and Evolution. Master’s thesis. http://www.dtic.mil/dtic/tr/fulltext/u2/a425228.pdf. Published May 23, 2004. Accessed April 18, 2013.Google Scholar
6. Huder, RC. in Disaster Operations and Decision Making. pages 19-37. http://onlinelibrary.wiley.com/book/10.1002/9781118178539;jsessionid=647DBFB934932D9A7DC7F1A9C1320C4C.f02t01. Published online February 28, 2012. Accessed November 3, 2013.Google Scholar
7. Boyd, J. The OODA or Boyd Loop in Performance, Learning, Leadership, & Knowledge Web site. http://www.nwlink.com/~donclark/leadership/ooda.html. Accessed November 8, 2012.Google Scholar
8. Qualtrics [computer program]. Version 2012. United Kingdom: Qualtrics, LLC; 2012. www.qualtrics.com. Accessed June 1, 2015.Google Scholar
9. StataCorp. Stata Statistical Software: Release 12 [computer program] College Station, TX: StataCorp LP; 2011.Google Scholar
10. Lieberman, MD, Jarcho, JM, Satpute, AB. Evidence-based and intuition-based self-knowledge: an fMRI Study. J Pers Soc Psychol. 2004;87:421-435.Google Scholar
11. Gilmour, D, Hanna, J, Mc Keever, W Jr, et al. Military Course of Action Model. http://www.au.af.mil/au/awc/awcgate/afrl/real-time_coa_analysis.pdf. Accessed October 25, 2012.Google Scholar
12. US Army. The Military Decision-Making Process. http://www.au.af.mil/au/awc/awcgate/army/fm101-5_mdmp.pdf. Published 2000. Accessed October 27, 2012.Google Scholar
13. Manketlow, J. The Mind Tools Guide to Group Decision Making. https://ejerciciosmentales.files.wordpress.com/2010/08/groupdecisionmakingguide1.pdf. Published 2009. Accessed April 18, 2013.Google Scholar
14. Dalkey, N. The Delphi Method: An experimental Study of Group Opinion RM-5888-PR http://www.rand.org/content/dam/rand/pubs/research_memoranda/2005/RM5888.pdf. Rand Corporation; June 1969. Accessed April 18, 2013.Google Scholar
15. Smith, W. Cost-Effectiveness and cost-benefit analysis for public health programs. Public Health Rep. 1968;83:899-906.Google Scholar
16. The Skilled Group Leader. Multivoting website. http://www.ca.uky.edu/agpsd/multivot.pdf. Accessed November 8, 2012.Google Scholar
17. Kahneman, D. Thinking, Fast and Slow. New York: Farrar, Straus and Giroux; 2011.Google Scholar
18. Khatri, N, Alvin, H. The role of intuition in strategic decision making. Hum Relat. 2000;53:57-86.CrossRefGoogle Scholar
19. Kahneman, D, Klein, G. Conditions for intuitive expertise: a failure to disagree. Am Psychol. 2009;64:515-526.Google Scholar
20. Klein, G, Snowden, D, Pin, CL. Anticipatory thinking. In: Mosier KL, Fischer UM, eds. Informed by Knowledge: Expert Performance in Complex Situations. New York: Psychology Press; 2011:235-245.Google Scholar
21. Gigerenzer, G, Gaissmaier, W. Heuristic Decision Making. Annu Rev Psychol. 2011;62:451-482.Google Scholar
22. Schwenk, CR. Cognitive simplification processes in strategic decision-making. Strateg Manag J. 1984;5:111-128.CrossRefGoogle Scholar
23. Your Dictionary. Rule of thumb. http://www.yourdictionary.com/rule-of-thumb. Accessed June 18, 2014.Google Scholar
24. Higgins, JW, Strange, K, Scarr, J, et al. “It’s a feel. That’s what a lot of our evidence would consist of”: public health practitioners’ perspectives on evidence. Eval Health Prof. 2011;34:278-296.Google Scholar
25. Mikels, JA, Maglio, SJ, Reed, AE, et al. Should I go with my gut? Investigating the benefits of emotion-focused decision making. Emot Emot. 2011;11:743-753.Google Scholar
26. Beauchamp, TL, Childress, JF. Principles of Biomedical Ethics, 6th ed New York: Oxford University Press; 2009.Google Scholar
27. Thompson, AK, Gibson, JL, Upshur, RE, et al. Pandemic influenza preparedness: an ethical framework to guide decision-making. BMC Med Ethics. 2006;7:1-11.Google Scholar
28. McDevitt, R, Giapponi, C, Tromley, C. A model of ethical decision making: the integration of process and content. J Bus Ethics. 2006;73:219-229.Google Scholar
29. Lee, LM. Public health ethics theory: review and path to convergence. J Law Med Ethics. 2012;40:85-98.Google Scholar
30. Haidt, J. The emotional dog and its rational tail: a social intuitionist approach to moral judgment. Psychol Rev. 2001;108:814-834.Google Scholar
31. Van den Bossche, P, Gijselaers, W, Segers, M, et al. Team learning: building shared mental models. Instr Sci. 2011;39:283-301.Google Scholar
32. Druskat, VU, Wollf, SB. Building the emotional intelligence of groups. Harv Bus Rev. 2001;79:80-90.Google ScholarPubMed
33. Tuckman, B. Developmental sequence in small groups. Psychol Bull. 1965;63:384-399.Google Scholar
34. Saaty, TL. How to make a decision: The Analytic Hierarchy Process. Eur J Oper Res. 1990;48:9.CrossRefGoogle Scholar
35. Poland, GA, Marcuse, EK. Developing vaccine policy: attributes of just policy and a proposed template to guide decision and policy making. Vaccine. 2011;29:7577-7578.Google Scholar
36. Field, RI, Caplan, AL. Evidence based decision making for vaccines: the need for an ethical foundation. Vaccine. 2012;30:1009-1013.CrossRefGoogle ScholarPubMed
Figure 0

TABLE 1Characteristics of the Survey Participantsa

Figure 1

TABLE 2Processes Used or Considered by Survey Participantsa

Figure 2

TABLE 3Decision-Making Criteria to Guide Prioritizationa

Figure 3

TABLE 4Satisfaction Among Decision-Makers and Those Involved in the Decision- Making Process

Figure 4

TABLE 5Training and Decision-Making Tools Survey Participants Would Like to Have in the Future

Supplementary material: PDF

Kayman supplementary material

Kayman supplementary material 1

Download Kayman supplementary material(PDF)
PDF 131.5 KB