In critical incidents or events, it is well established that delays, mistakes, and errors result from team failure and communication breakdowns.Reference Hunziker 1 , Reference Manser 2 To examine the predictors and determinants of team performance during public health emergency response, the University of Minnesota School of Public Health (UMN-SPH) partnered with the Minnesota Department of Health (MDH) to implement an applied research study. This research examined the effects of controller-led in situ simulation training, also referred to as exercises, on team function and dynamics among staff responsible for public health emergency operations and coordination at MDH.
Numerous studies have identified a critical link between teamwork, leadership, and patient safety.Reference Hunziker 1 - Reference Nielsen, Goldman and Mann 6 Studies investigating the factors contributing to critical incidents in health care settings demonstrate that teamwork plays an important role in the causation and prevention of adverse events.Reference Manser 2 , Reference Mazzocco, Petitti and Fong 4 , Reference Thomas, Sexton and Lasky 5 Observational studies on teamwork behaviors related to clinical performance in a hospital setting have identified patterns of communication, coordination, and leadership that support effective collaboration.Reference Manser 2 In one study examining patient outcomes and teamwork effectiveness, patients were more likely to experience death or complications when teams exhibited poor teamwork behaviors.Reference Mazzocco, Petitti and Fong 4 Several researchers address teamwork among interprofessional staff in acute care as it is associated with outcomes, quality, and performance in a hospital setting.Reference Baker 3 , Reference Thomas, Sexton and Lasky 5 , Reference Nielsen, Goldman and Mann 6 However, evidence is lacking within the field of public health linking teamwork and public health emergency response.
Research supports the use of exercises as an effective tool for improving team performance.Reference Shapiro, Morey and Small 7 , Reference Wayne, Didwania and Feinglass 8 Conducting exercises to practice and test emergency plans, procedures, and response actions is routine for emergency response. Exercises are a powerful training tool because they allow for systematic control of the schedule of practice, presentation of feedback, and introduction or suppression of environmental distractions within a safe, controlled learning environment.Reference Shapiro, Morey and Small 7 , Reference Wayne, Didwania and Feinglass 8 Public health was intermittently involved in exercising prior to 2001. The majority of exercises were designed by using the Homeland Security Exercise and Evaluation Program (HSEEP), a program of the Federal Emergency Management Agency.
The UMN-SPH conducted research in partnership with MDH that advances the science of public health preparedness by measuring differences in team performance during functional exercises among 3 research teams. This research is significant because the effectiveness of teamwork in a department operations center (DOC) has not been previously studied, nor have measurement tools been validated to understand how teams function, perform, and make decisions during emergency response training.
METHODS
Between May 2010 and September 2011, MDH and the UMN-SPH facilitated 30 one-hour functional exercises using 6 different scenarios at the MDH DOC. All participants in this research engaged in exercises per MDH’s existing exercise and training program, which was based on HSEEP. Participants were divided into 3 research teams that included a layered intervention. Research teams included a control group that participated in standard HSEEP-designed exercises, a didactic group exposed to training based on team dynamics and communication, and a treatment group that received the didactic training plus the post-exercise facilitated debriefing. The teams were consistent in size throughout the research period with 16 team members per exercise. Participants were assigned to a specific Incident Command System (ICS) role within their team on the basis of their skills, experiences, or interest.
Attitudinal Survey
Before and after participation in this research, participants were asked to complete a self-reported attitudinal survey about their experience with the ICS and the DOC. This attitudinal survey was adapted from 2 validated patient culture surveys, including the 2004 Hospital Survey on Patient Safety Culture and the 2008 Medical Office Survey on Patient Safety Culture, developed by the Agency for Healthcare Research and Quality. 9 The attitudinal survey consisted of 35 questions and assessed public health emergency response, specifically, attitudes toward the use of Incident Command and DOC operations.
Behavioral Markers
Behavioral markers to assess group performance in the DOC were adapted from previous research conducted in a hospital obstetrics surgical unit.Reference Baker 3 Behaviors measured using the behavioral markers included situational awareness; closed-loop communication; the situation, background, assessment, and recommendation (S-BAR) technique; and the shared mental model. An observational tool was developed that listed specific observable criteria linked to each behavior. These markers were measured on a scale of 0 to 2, with 0 indicating a nonobserved behavior, 1 indicating that a behavior occasionally or sometimes occurred, and 2 indicating that the behavior always occurred. Cumulative scores based on the set of markers ranged from 0 to 92, with 92 indicating high team performance. To measure the effectiveness of team formation within each group, all 30 one-hour exercises conducted during the research period were video recorded. Observers trained by the research team reviewed all of the recorded exercises and used the observation tool to assign numeric values to the group behavioral markers. Participants were given the option to opt out of being video recorded. All participants who opted into the research signed both UMN-SPH and MDH consent and media release forms agreeing to be part of the research project and to be recorded. Approval to carry out the study was obtained from the University of Minnesota’s Institutional Review Board.
RESULTS
Over half (60%) of the sample was female and nearly one-third of the respondents (30%) identified themselves as being in a management position at MDH. Most (66%) respondents reported that their primary work experience was in public health. The most commonly reported age range was 40 to 59 years of age (73%), and a graduate degree was the most common reported level of education (64%). A range of divisions from across MDH were represented with the Office of Emergency Preparedness as the most commonly represented division at 31%. The most frequently reported length of employment at MDH among the research participants was 6 to 15 years at 44%. The sample sizes of 64 in 2010 and 25 in 2011, referenced in Table 1, represent the total number of participants who completed the pre- and post-participation self-reported attitudinal survey.
Results From an Attitudinal Survey Showing the Percentage of Research Participants Agreeing With Statements Measured Before Participation in Exercises (n=64) in 2010 and Again After Participation in Exercises (n=25) in 2011Footnote a
a Abbreviations: DOC, department operations center; MDH, Minnesota Department of Health.
Figure 1 compares the team performance of the 3 groups across the study period. This graph demonstrates the highest team performance among the didactic group for 3 out of 5 exercises. Although there were differences in team performance between the groups, there was no trend in team performance over time. Many of the teams demonstrated fluctuation in team performance during the study period. This was particularly evident in the didactic group, which experienced changes in team leadership during the study period.
Team Performance Score Among 3 Research Groups During 5 Exercises Conducted From 2010 to 2011.
Table 1 highlights changes in participant agreement from before to after the survey for 6 questions selected from the 36-item attitudinal survey. Additional findings from the survey demonstrated consistently high agreement (participants answering “agree” or “strongly agree”) between before and after participation to questions concerning feeling supported in the DOC from peers and MDH leadership. Additionally, attitudinal surveys demonstrated that an increase in workplace satisfaction occurred during the study period. Participants reported feeling supported by MDH leadership in the DOC by having the appropriate equipment and resources available to them during exercises.
DISCUSSION
Key findings from this research highlight important factors that contribute to improved team performance during exercise.Reference Thomas, Sexton and Lasky 5 , Reference Nielsen, Goldman and Mann 6 Outcomes observed by the MDH leadership involved in the research included that the increase in the frequency of exercising had a positive impact on the attitudes of MDH staff. After participating in this research, MDH staff expressed an interest in training and a desire to exercise more frequently. The increase in exercising allowed research participants an opportunity to provide constructive feedback to the MDH exercise design staff on ways to improve the exercise experiences. These findings also support previous research showing that frequent exercising helps to build confidence and expertise in staff, thus allowing individuals to trust themselves and their team members.Reference Mazzocco, Petitti and Fong 4 - Reference Nielsen, Goldman and Mann 6
Important attitudinal changes across all research groups occurred throughout the study period. Changes in participants either agreeing or strongly agreeing with questions related to working well together in the DOC and around response to feedback by MDH leadership were documented on the pre- and post-participation attitudinal surveys. For instance, there was a 54% increase in participants either agreeing or strongly agreeing that staff in the DOC freely speak up if they see something that may negatively affect their response. Another important finding was that 84% of post-exercise training participants responded that they agreed or strongly agreed that staff across MDH divisions worked well together in the DOC during a response, compared to 45% of pre-training participants responding positively to this statement.
Although there were no consistent trends over time, changes in team performance were observed during the study period. These differences may have been influenced by changes in team leadership. In this context, team leadership refers to the incident commander who was assigned to lead the team through the exercise. For instance, the didactic group had the most fluctuation in team performance. Performance increased in this group for the first 3 exercises and then showed a steady decrease at the end of the research period. This group had the most fluctuation in team leadership among the 3 research groups. For example, the person assigned as team leader, the incident commander, changed several times during the study period compared with the treatment group, who had 1 consistent leader in the role of incident commander throughout the study period. These findings highlight the need for well-trained, effective, and consistent leadership in the DOC to improve team performance during public health emergency response and exercising.
Significant differences in team performance did not occur between the treatment group and the didactic group or the treatment group and the control group after the first exercise. This could be the result of various factors. One potential significant factor is that the treatment group was exposed to an intervention originally designed for implementation in a hospital setting. This intervention may be more effective in a hospital setting because there is a shorter interval of time between exercises in this environment than in the public health setting. This type of intervention may be more successful in a hospital setting because the practice of exercising may be more novel in that setting.Reference Baker 3
The MDH research team reported important lessons learned from this research including that an employee’s skills, interest, and authority influenced their performance in their assigned role in the DOC. This finding suggests that it is critical to assign the right people to the right role during public health emergency preparedness exercises and response. This key finding reinforces previous research around the importance of leadership and team performance.Reference Hunziker 1 , Reference Baker 3 The research highlighted the importance of providing staff with time prior to the start of the exercise (30 minutes) to allow participants to ease into their role. Additional lessons learned include the importance of providing coaches to staff during the exercise and addressing concerns and issues as they occur rather than waiting until the end of the exercise.
Limitations
There were several important limitations to this research. One significant limitation was the challenge of capturing each behavioral marker owing to limited video recording technology in the DOC during exercises. Reviewing and analyzing all conversations in the complex environment of the DOC was difficult. Owing to these constraints, some conversations were missed, making measurements across teams subject to bias. The use of a nonvalidated observation tool to measure team performance was another significant limitation of this study. Additional challenges included inconsistent team participation rates. Team members across the 3 research teams were constantly changing owing to work conflicts and changes in work positions. This variation in team makeup could have led to biased results. Another notable limitation in this study was the use of self-reported data gathered in the attitudinal surveys that was subject to bias.
CONCLUSIONS
Findings from this research support that a critical link exists between team performance, leadership, and key behavioral markers during public health emergency response.Reference Hunziker 1 - Reference Mazzocco, Petitti and Fong 4 The impact of team function on the success or failure of public health emergency response is often overlooked and undervalued. This research supports that training emergency response workers to effectively work together as a team increases performance during response at a state health department. Effective teams are better equipped to participate in emergency response and to initiate and contribute to response activities. This research demonstrates that emergency response at a state health department is more efficient when team members trust each other, feel supported in asking questions, and feel comfortable providing feedback on incident objectives and strategies. Findings from this research can be used to inform the development of emergency preparedness training at state and local health departments. Examples include the development of public health incident leadership training, designing exercises focused on specific response capabilities such as conducting tactics meetings, and creating knowledge and skills workshops related to core response activities such as deploying staff, communicating with partners, and supporting evacuating health care facilities.
Acknowledgments
This project was supported in part through a grant from the Centers for Disease Control and Prevention (CDC) grant number 5P01TP00000301-05. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the CDC. The authors acknowledge the contributions of Samantha Morgan, Debra Burns, and Sarah Henderson Ramsey.
