Since the well-publicized negative and sometimes fatal experiences of individuals with disabilities during the September 11, 2001, terror attacks and Hurricanes Katrina (2005) and Sandy (2012), advocates and authors have highlighted the importance of inclusive, effective emergency preparedness, response, and recovery services for and with individuals with disabilities.Reference Davis and Styron 1 - Reference White, Fox, Rooney and Cahill 4 During Hurricane Sandy, at least 4000 New Yorkers with intellectual disabilities were evacuated; 1300 remained out of their homes for weeks, 3 challenging the service provider system. Hospital transportation of patients with physical disabilities was reported to be poorly planned, putting these individuals at risk.Reference Powell, Hanfling and Gostin 5 As in prior disasters, some public emergency shelters reportedly turned away people with disabilities,Reference Jan and Lurie 6 and those with disabilities in at least one shelter reported being denied freedom of movement and access to their own funds. 7
These negative effects appear to continue despite legal protection such as the Americans with Disabilities Act (ADA) of 1990, 8 , Reference Jones 9 which requires federal, state, and local governments and emergency responders in their employ to provide equal access to emergency services to individuals with disabilities. The US Department of Homeland Security’s mitigation, response, and recovery frameworks; 10 - 12 recent FEMA guidance; 13 and the US Centers for Disease Control and Prevention public health preparedness capabilities 14 support this obligation.
Several factors may be responsible for these inadequacies. Emergency plans often assume that all Americans can walk, run, talk, hear, drive, and quickly follow directions, thereby ignoring the likely adaptations needed for some with disabilities.Reference Kailes and Enders 15 Emergency responders (such as law enforcement, fire, emergency medical services [EMS], and public health personnel; emergency managers and planners; and Medical Reserve Corps [MRC] and Community Emergency Response Team [CERT] volunteer responders) may lack sufficient education about disability-related emergency needs.Reference Davis and Mincin 16 - Reference Gibbs and Holloway 18 For example, without specialized training, law enforcement personnel may interpret an averted eye gaze as a sign of lying or suspicious behavior, 19 rather than potentially typical behavior of someone with autism. At least one EMS protocol suggests that an unsteady gait or slurred speech may imply hypoglycemia due to diabetes. 20 An equally plausible explanation is that the individual exhibits behaviors associated with cerebral palsy. Educating responders regarding disability-related emergency needs would promote equal access to emergency services and remove barriers to response efficiency,Reference Davis and Phillips 21 , Reference Royal, Donahue and Kirby 22 thus enhancing the safety and well-being of those with disabilities.
Education and Training Options For First Responders
When responder education is provided about the emergency needs of individuals with disabilities, it tends to be delivered via online or classroom-based courses, drills, and guides and is often focused on recommendations to enhance 1:1 interactions and communication with those with disabilities. The relatively few online responder courses related to disability appear to include mainly prerecorded, user-paced offerings, often presented in a PowerPoint (Microsoft Corp, Redmond, WA) format. See the Online Data Supplement for a further review of responder training options.
Benefits of Asynchronous, Online Educational and Training Materials
Computer-based online education is an increasingly beneficial educational method for teaching emergency responders, enhancing and reinforcing knowledge and satisfaction, and motivating learning. 23 - Reference Yellowlees, Cook and Marks 26 Online courses offer the opportunity to educate a potentially large and widely dispersed audienceReference Hamad, Serna, Morrison and Fleming 27 and can provide immersive, participatory, realistic scenarios.Reference Hsu, Li and Bayram 24 Appropriate response is greatly influenced by responder familiarity with the response scenario plus skill, knowledge, and psychological state.Reference Hsu, Li and Bayram 24 Thus, like drills, online courses can offer interactive response scenarios that simulate real life, offering the learner practice before encountering a like situation in the field.Reference Hsu, Li and Bayram 24 , Reference Olson, Hoeppner and Scaletta 25 The US Centers for Disease Control and Prevention notes that simulations, including those with game-like features, are so advantageous that they should be a key component of public health education. 23 An additional advantage of online courses is the ability to repeatedly review a course and score it electronically to aid performance assessment and develop corrective action.Reference Hsu, Li and Bayram 24 Finally, online training is thought to be cost-effective; the US Department of Defense found that computer simulations reduced training costs by 10% compared with in-person training.Reference Olson, Hoeppner and Scaletta 25
Purpose of the Investigation
This study investigated the impact and perception of online instruction, presented via simulation-based scenarios and incorporating game-like features, on emergency responder knowledge and skills regarding the emergency needs of people with disabilities. Specifically, University of Massachusetts Medical School (UMMS) Eunice Kennedy Shriver Center (Shriver Center) and Praxis, Inc, researchers designed, developed, and evaluated an Internet-based distance education course (Rescue-D) intended to teach emergency responders to anticipate, plan for, and address the needs of children and adults with disabilities before, during, and after emergencies. The course consisted of 4 modules (a portion of a more extensive course), incorporating an ongoing storyline and scenario-based learner interactions. The research aim was to assess learner factual knowledge, applied knowledge (skill), and satisfaction with the course to determine the feasibility of this educational format for future potential development and implementation of a more extensive curriculum.
The effort is unique in that it is a first attempt known to the authors to combine scenario-based learning and game-like features in a self-paced online course to teach emergency responders about people with disabilities. This is important because work force training increasingly includes asynchronous online instruction yet continues to require evidence-based real-world training and application.
Methods
Rescue-D Design, Development, and Technology
Design and Development
Prior to development, we consulted with a variety of first responders, geographic information system (GIS) mapping specialists, instructional designers, and individuals with disabilities and their families. Rescue-D is a scenario-based online course incorporating a storyline, role-play, and game-like features designed to simulate real emergency situations for the learner. Set in a fictional East Coast US county, Rescue-D depicts events before, during, and after an explosion, hurricane, and flood. The learner assumes the role of “Responder Brown,” a local emergency responder, and begins by learning about the county, its residents, and physical infrastructure. The learner progresses through the course’s storyline, reviewing information, asking and answering questions, and making decisions (eg, the best way to evacuate a family from home, whether an emergency shelter policy comports with the ADA, 8 and whether a dog would be considered a service animal). While interacting with characters with and without disabilities (eg, the boss, a colleague, or someone who needs rescuing from the flood zone), the learner addresses disability demographics, the negative impacts of disasters on people with disabilities, appropriate 1:1 interactions and communication, evacuation, transportation, and inclusive emergency planning for and with people with disabilities. The course includes a mapping tool (a map with different layers of information and a related editable database) used to locate shelters, evacuation vehicles, the flood zone area, and certain vulnerable populations. The pre-test, post-test, 4 tutorials, and 9 knowledge and applied mapping skill-based scenarios were integrated into the storyline; the learner never encountered obvious tests or lecture. The course topic outline is shown in Table 1.
Rescue-D Topic Outline
Figure 1 depicts a scene from “Rescue From the Flood Zone” (RFFZ), 1 of 9 timed tests of simulated interactions, with differential points awarded on the basis of the quality of decision-making. In this scene, the learner is asked to identify the evacuation and transportation needs of elderly bingo players, some with disabilities (eg, mobility disabilities, oxygen needs). The learner chooses whether to ask questions of the scene’s characters with points awarded for questions that elicit more accurate or useful information. Once the learner determines that his or her information is complete, she or he uses the mapping tool to (1) identify and assess potential evacuation vehicles for the group, (2) “send” an appropriate vehicle to pick up the group, and (3) identify the closest, most appropriate mass care (general public) shelter. The learner gains more points for faster, better decisions.
Rescue From the Flood Zone.
Game elements incorporated into Rescue-D include a quest or central theme, winning and losing points, a scoreboard, and time challenges. Learners also earn points for completion of pre-test and post-test segments (unrelated to accuracy) and only receive performance feedback in the tutorials and RFFZ. Correct responses result in points; error responses result in point loss or no point change. Learners receive 2 scores, one for the course overall and one for RFFZ. Progress is shown by displaying cumulative points and badges that appear upon progression through the course.
Development was guided, in part, by the scientific formative evaluation process outlined by Markle and othersReference Markle 28 , Reference Twyman, Layng, Stikeleather and Hobbins 29 and the Distance Learning Design ProtocolReference Fleming 30 developed and used at the Shriver Center to create distance-learning courses. After the learning objectives, the course outline, and 9 potential application scenarios were designed, 4 responders and 3 individuals with disabilities reviewed the objectives, curriculum content, scope, and storyline. They also reviewed the scenarios, including artwork, for complete and realistic portrayals of responders, the disability community, and disaster experiences. Rescue-D was revised on the basis of their feedback.
Prior to course development, Rescue-D was granted an exemption by the UMMS Institutional Review Board. After design and development, Rescue-D underwent a rigorous internal quality assurance process by 5 Praxis and Shriver Center personnel previously unfamiliar with the course. Reviewers included an individual with a disability, an instructional designer, and behavioral research scientists who completed Rescue-D several times, reviewing content, language, vocabulary, layout, and usability. Following the quality assurance review, the research team revised Rescue-D as necessary and then conducted a formative preliminary beta-test, which is described further below. After the preliminary beta-test, Rescue-D was revised again on the basis of learner feedback regarding content, usability, and analysis of learner responses to course interactions, with the navigation interface, knowledge, and mapping assessments adjusted until the course functioned as intended.
Technology
Rescue-D was designed as an asynchronous course that is available “anywhere anytime” over the Internet by use of a modern browser and broadband connection. The course was developed in Unity 3d, a cross-platform system for creating video games and interactive activities. Learners logged in and out of the course as needed, except for RFFZ, which needed to be completed in one sitting for technical reasons. Data from online learner interactions was recorded by Unity 3d and posted in real time to an online UMMS database that in turn analyzed the raw data and produced individual and group scores for research team review.
Participants
Formative preliminary beta-test participants included a pool of 10 individuals (5 completers) and summative beta-test participants included a pool of 50 individuals (18 completers). Recruitment for preliminary beta-testers and beta-testers occurred simultaneously and from the same participant pool. We e-mailed 2 invitation letters seeking beta-testers to 132 local, county, state, and national public health and safety organizations and to individual responders and asked the recipients to share the e-mail with their networks. The invitation described Rescue-D as an “online course/game” for first responders to learn appropriate and efficient interactions with, and emergency response for, children and adults with disabilities and described the eligibility criteria and participant selection. Potential participants were also informed that the course would include knowledge, demographics, and learning experience questions. Interested individuals completed an online survey (SurveyMonkey, Palo Alto, CA) asking about computer broadband Internet access, online course and game experience, and prior education about, or relationship with, individuals with disabilities.
Inclusion criteria were adults aged 18 or older who were first responders (defined to include emergency response personnel “such as police, sheriffs, fire fighters, emergency medical services personnel, federal, state, county or local government emergency managers, and/or federal, state, county or local public health officers”) currently working full-time or part-time in the field in a paid or volunteer capacity. Responders needed to indicate competence in reading and understanding written English and access to a computer with broadband Internet access.
One hundred ninety-nine individuals from 34 states completed the eligibility survey. Twenty-five individuals who did not meet the inclusion criteria were excluded. The research team analyzed the results and added voluntary MRC and CERT responders to the profession list, then divided potential respondents (n=174) into 6 response profession categories (law enforcement, fire, EMS, public health, CERT/MRC, emergency manager/planner). To ensure that participants represented a broad array of response professions, we used an online random number generator to select 10 participants from each responder group (n=60), with at least one participant from each of the 10 FEMA regions. 31
Ten preliminary beta-testers were randomly selected; each received a welcome letter (with information needed to access the course and other instructions) and unique access to Rescue-D. Five participants (50%) completed the formative preliminary beta-test. After revising the course as described above, we e-mailed the welcome letter and unique URL to the remaining 50 participants. Of these, 28 started and 18 (36%) completed Rescue-D and the satisfaction/usability survey within the requested 2-week window.
Tables 1 and 2 in the Online Data Supplement describe participant characteristics. Preliminary beta-testers and beta-testers completing Rescue-D and the survey received a $100 gift card as compensation for their time.
Pre-Test and Post-Test Percentage Correct
aWilcoxon signed-rank test is the nonparametric equivalent of the paired t-test for testing significant pre to post change.
bEffect size (Cohen’s d) on pre to post difference (ES>0.8=>large effect).
cTotal gain in % correct provides equal weight to all 29 questions.
Data Collection and Measurement
Before the study the researchers performed a statistical power analysis for sample size estimation on the basis of data from another Shriver Center distance learning projectReference Fleming, Sulzer-Azaroff and Hamad 32 in which researchers evaluated pre- and post-test scores of student learning participants (N=26). The effect size was 1.76, which is large according to Cohen’s criteria.Reference Cohen 33 With a two-tailed alpha of 0.05 and power=0.80, the projected sample size for this effect size was approximately N=8. With an N of 10, one would approximate yet a higher power setting (at 0.90). Thus, our actual sample size of 18 beta-testers appeared adequate.
During the formative and summative assessments, we collected data on learner demographics, knowledge, and skill acquisition via pre- and post-tests. The pre- and post-tests each included 29 questions, consisting of multiple choice, multiple selection, and mapping skill application items, which were incorporated throughout the ongoing storyline. Pre- and post-tests were not identified as such, thereby controlling post-test contamination. Pre- and post-test items measured identical concepts, with parallel but not identically worded questions, to lessen the likelihood of learning via pre-test exposure. Pre- and post-test questions covered: (1) disability community demographics and disaster experiences; (2) successful 1:1 interaction and communication with people who are blind and have service animals, have autism, or use a wheelchair; (3) “functional and access need”-based inclusive emergency planning per FEMA guidance 34 and the ADA 8 ; and (4) the use of the mapping tool to identify resources and vulnerable populations. We assessed mapping tool use via simulated activities occurring during scenarios (eg, identify public housing locations, create a map layer showing two disability provider agency locations). An online survey available after completion of the last Rescue-D scene collected satisfaction and usability data via Likert-type ratings and open-ended questions. Data were analyzed by using SAS statistical software. 35
Results
The section below reports data from the summative beta-test.
Knowledge and Skill Acquisition Assessment
Data were analyzed on the basis of total responding (corrects and errors). Each question required a response. Data for a single question were not recorded for 2 of the 18 beta-testers (owing to errors in electronic capturing); thus, the data were analyzed by using 2 approaches. In the first, missing data were assumed incorrect, producing a bias toward the null hypothesis. The second approach imputed missing data with the sample mean, which credits the participants with the mean score of the remaining sample when a question is omitted, producing a bias toward intervention success. Whereas the true estimate of intervention success likely lies between these 2 estimates, Table 2 displays pre-/post-test comparisons by using the former, more conservative, statistical analysis and shows an overall mean knowledge and applied skill gain. Beta-testers demonstrated highly statistically significant pre- to post-test gains in knowledge (mean=16%, P<0.0001), mapping skill (mean=39%, P<0.0001), and overall (mean=19%, P<0.0001). Mean gain in percentage correct for the 25 knowledge questions was 16%, whereas mean gain for the 4 map questions, where pre-test scores tended to be very low, was 40%. When all 29 questions were considered, the overall gain in percentage correct was 19% (ie, an average increase of 5 additional questions correct).
The results indicate (regardless of how missing data were handled) that the intervention had a very large effect for knowledge gain, map skill gain, and overall gain. The effect sizes of the pre- to post-test differences were also large: 1.4 for knowledge gains, 1.4 for mapping skill gains, and 2.3 overall. CohenReference Cohen 33 identifies rules of thumb for measuring effect sizes for mean change as 0.2 indicative of a small effect, 0.5 as a medium effect, and 0.8 as a large effect; thus, these effect sizes in the range of 1.4 to 2.3 indicate a very large effect and highly effective intervention.
Satisfaction and Usability
Upon completing Rescue-D, beta-testers completed a survey with 11 satisfaction and usability items. Eighty-nine percent of beta-testers agreed or strongly agreed that Rescue-D was easy to navigate, that it played well, and that content and activities were presented at a comfortable pace. All beta-testers (N=18) indicated that it was helpful to complete Rescue-D at their own pace and without an instructor. Seventy-eight percent indicated that Rescue-D was preferable over an in-person course. We determined a total usability score from all usability questions. Ninety-three percent of beta-testers rated usability positively (satisfied or very satisfied). Ninety-four percent indicated that the content and activities were logical and clear and that they would recommend Rescue-D to a colleague. All beta-testers indicated that the difficulty level and content depth were appropriate and that Rescue-D increased their awareness of the emergency needs of people with disabilities. A total satisfaction score from all satisfaction questions revealed that 97% of beta-testers rated total content and activities positively (satisfied or very satisfied). Half of the beta-testers completed Rescue-D in 2 to 3 hours, 22% required 1 to 2 hours, 17% needed 3 to 4 hours, and 11% completed Rescue-D in 4 to 5 hours.
The beta-testers rated the game-like elements in the course very highly. All beta-testers indicated that the continuing storyline and learning “in a fun way” enhanced their interest in the content and activities. Seventy-eight percent reported that the role-play and interactive map enhanced interest, whereas 72% indicated that point scoring, and 56% indicated that the timed elements, enhanced interest. Representative comments to open-ended questions about game features, technology used, and suggestions for improvement are presented in Table 3 in the Online Data Supplement.
Discussion
This study demonstrated the feasibility of an online role-play simulation-type course to effectively teach public health and safety responders about the emergency needs of individuals with disabilities. Results indicate significant knowledge and skill acquisition gains from pre- to post-test. Learner usability and satisfaction scores were high. These results provide support for further developing Rescue-D into a complete distance-learning curriculum.
The literature suggests potential reasons for these results. We developed Rescue-D as an immersive educational simulation, with a storyline consisting of rescue and other scenarios. Many emergency responder trainings consist of practical exercises in the form of simulated emergenciesReference Davis and Phillips 21 designed to mimic actual potential events. The simulated real-life scenarios and interactivity in Rescue-D may have felt more familiar or comfortable to learners used to participating in such drills. Online simulations and games have been found to be particularly motivating to learners;Reference Olson, Hoeppner and Scaletta 25 satisfaction and motivation impact performance.Reference Olson, Scheller and Larson 36 Adults learn best when applying theory to practice and prefer experiential to passive learning.Reference Olson, Hoeppner and Scaletta 25 During development we learned that our intended audience didn’t want the pressure of “tests”; thus, embedding pre- and post-tests into the storyline may have also encouraged learner motivation and enthusiasm, leading to knowledge and skill gains. Another factor impacting the results may have been ease of technology use. Learners needed only brief instructions, with no special training or familiarity with Unity 3d, to complete Rescue-D. By contrast, online simulations and games developed in other platforms might require more extensive training in platform use. For example, authors describe a required 1-hour tutorial before beginning online trainings developed in the Second Life platform.Reference Olson, Hoeppner and Scaletta 25 , Reference Yellowlees, Cook and Marks 26 Finally, simulated online scenarios are risk-free for the learner; in real life, disaster victims may become injured or die, and an emergency responder may make potentially harmful or fatal mistakes. Rescue-D’s lack of risk may have contributed to the results.
Future Curriculum Development and Dissemination
We consider Rescue-D as a prototype and intend to revise and expand the course with additional topics and depiction of a greater number and variety of disabilities, including people with multiple disabilities or disabilities combined with chronic health conditions. Segments could be included to teach learners to incorporate disability community strengths and abilities into local emergency planning and response. Additional features might be developed to enable learners to ask content-related questions (as one could in a classroom-based training) and a follow-up resource guide might be developed for future consultation during actual response. Additional scenarios could be developed on the basis of traditional response protocols to encourage learners to reconsider those conventions (eg, traditional response protocols may call for evacuating victims from a building as fast as possible;Reference Davis and Phillips 21 however, following this procedure might endanger the health or well-being of someone with a particular disability).
Effective dissemination of such a course, aimed at attitude and system change in addition to education, might prove challenging. Providing continuing education units or college credits might provide incentives for course completion. Some writers suggest that, in addition to stand-alone courses, disability curriculum content should be integrated into existing undergraduate, graduate, and continuing education responder courses.Reference Davis and Phillips 21 , Reference Rowland, White, Fox and Rooney 37 Because people with disabilities represent such a large population segment (approximately 19% of the noninstitutionalized populationReference Brault 38 ), a disability-related course could be required for responders (at least one state, New Jersey, mandates a responder course on developmental disabilities 39 ).
Limitations and Future Research
The current study evaluated a subpart of a larger course, which led to some research limitations. Further research will be needed to assess learner knowledge and skill gains across a wider variety of disabilities and emergency-related challenges. This study did not assess the role of individual instructional components to determine which, if any, components may have had a greater impact on knowledge and skill gains. Future research might examine which components were more or less beneficial to the learning process, with suggestions for re-design for maximum learner knowledge gain. We also did not evaluate whether learner knowledge and skill gains persisted over time or translated into actual practice. Additional assessment via post-surveys might shed greater light on knowledge retention and translation into practice. Although the benefits of cross-training (training multiple responder audiences together and regarding multiple roles)Reference Davis and Phillips 21 is widely recognized, this study did not evaluate whether learners from different response services differed in performance in relation to each other or needed education individualized to their specific service. While it is known how knowledge is disseminated to emergency responders, gaps exist in our understanding of the knowledge acquisition process and best training practices.Reference Davis and Phillips 21 Future study could examine these issues and survey responders about knowledge acquisition needs.
Conclusions
Computer-based distance learning is recognized as an acceptable means of teaching emergency responders and a viable alternative to traditional training modes. 23 , Reference Hsu, Li and Bayram 24 The results of the current study indicate that an online course with simulation-based scenarios and game-like features can successfully transfer knowledge and skills about the emergency needs of people with disabilities to emergency responders. Experts posit that prior training influences emergency responder readiness self-perception,Reference Hsu, Li and Bayram 24 and past experience improves future response.Reference Chandra, Acosta and Stern 40
Emergency responders play an important role in protecting Americans from harm due to manmade or natural disasters. Responders who are better trained and have practiced response techniques under simulated conditions may be better prepared to address the likely needs of individuals with disabilities during an actual emergency and incorporate the needs and skills of people with disabilities into “whole community” inclusive planning, enhancing community resilience and speeding future disaster recovery.
Acknowledgments
We gratefully acknowledge the work of our Rescue-D Development Team colleagues: Christophe Gerard, PhD; Anne Hunt, PhD; Ann Maloney, MD; and Nancy Shea, JD, MPA.
This research was supported by National Institute of Child Health and Human Development (NICHD) grants R41 HD069070 and P30 HD004147. The contents of this article are solely the responsibility of the authors and do not necessarily represent the official view of NICHD.
Supplementary material
To view supplementary material for this article, please visit http://dx.doi.org/10.1017/dmp.2014.129
