Crises, regardless of their context, are by nature fraught with uncertainty. Some crises, such as floods or hurricanes, come with discernable timelines of pre-crisis, acute crisis, and post-crisis. In such crisis events, learning is situated in the post-crisis phase and lessons are distributed during the subsequent return to normality.Reference Sellnow and Seeger1 Crises like the COVID-19 pandemic, however, are protracted and volatile, demand collaborative response efforts world-wide, and require years to fully resolve.Reference Croucher and Diers-Lawson2 During the COVID-19 pandemic, health care workers (HCWs) and organizations did not have the luxury of waiting until post-crisis to reflect on lessons learned. Instead, they had to learn with urgency during the acute phase of the crisis, while pacing themselves throughout an evolving and lengthy ordeal.
Limited research exists examining learning and instructing processes during the acute phase of lengthy crisis events. Similarly, no evidence-based systems-focused guidance for training health care workers (HCWs) is currently available to member states of the World Health Organization (WHO) in this type of situation. Therefore, the Learning and Capacity Development unit in the Executive Director’s Office of the WHO Health Emergency Programme set out to develop Guidance on Learning and Capacity Building in Health Emergencies. With this aim in mind, the WHO issued a call for proposals to undertake mixed methods systematic reviews pertaining to “Learning approaches, interventions, and dissemination strategies used in public health emergencies.” Conclusions drawn from synthesis of existing evidence would inform the guidance development process for just-in-time (JIT) learning.
The focus of the current systematic review was to address the following multi-part question:
What types of learning approaches and interventions and learning dissemination strategies have organizations and individuals developed and used in health emergencies and why? How were they designed and developed and by whom?
Two other reviews were commissioned at the same time to investigate the effects of training interventions during health emergencies and barriers to effectiveness. Therefore, this review does not address those issues.
Methods
The review was designed as a mixed-methods systematic review.Reference Higgins, Thomas and Chandler3 No studies with human or animal participants were conducted, so institutional review board approval was not necessary.
Search Strategy
Search strategies were collaboratively developed with input from an Information Scientist at WHO, scholars at Harvard University and Wayne State University who were working on associated review questions, and the team member, who is the Librarian for the College of Nursing at our institution. Additionally, all members of the review team performed individual searches to locate key articles for use by the team librarian to refine the search strategy.
Three databases were identified for searching within the condensed time frame of the original project, in consultation with the WHO guidance team: PubMed, CINAHL, and Communication and Mass Media Complete (EBSCO) (June 2023). After completion of the initial project report, a search of Web of Science was also conducted (September 2023).
Using a Boolean approach, we searched 3 distinct subject areas: a) disaster/emergency preparedness b) staff development/capacity building/training/education, and c) just-in-time training. Specific search strategies and terms were tailored for each individual database. For each database searched, we consulted the database thesaurus for subject terms, allowing words to be “exploded” to locate additional synonyms and related terms. Exact search strategies for each database are available in Appendix A. To locate literature from low- and middle-income countries, we did not restrict the search to specific languages.
Related systematic reviews and other key articles were flagged in the systematic review software for ancestry and forward citation searching. To ensure that no articles remained uncaptured, additional subject headings and keywords were derived from these articles, as well as previously unexplored controlled vocabulary and keywords located in other core articles.
Study Screening Method
Titles and abstracts of search results were uploaded to the Covidence systematic review management platform, and duplicates were removed. A total of 11 914 records were subjected to title and abstract screening by 2 team members working independently. Disagreements were resolved by discussion or by the PI. Full texts of 556 articles were screened by teams of 2 researchers working independently, and disagreements were resolved by discussion or by the PI. Studies that were judged to be relevant were subjected to quality appraisal and extraction of key findings. The article selection process is graphically portrayed in Figure 1.Reference Page, McKenzie and Bossuyt4
Inclusion and exclusion criteria
Inclusion and exclusion criteria were developed based on information in the WHO request for proposals and in further consultation with WHO guidance team members.
PRISMA Flow Diagram of Selection Process. Note: A list of excluded studies can be obtained from the second author.
Inclusion criteria. 1. Articles which included some element of empirical research about training outcomes (e.g., survey, case study, interviews, observation, experimental design); 2. JIT training which aims to bridge a specific gap in knowledge or skills in a health emergency or impending health emergency; 3. Learning provision to existing health workers, volunteers, and community workers who are providing health care in areas relevant to health emergency management in real time; 4. Continuous learning for existing health workers, volunteers, and community workers who are providing health care and health response (e.g., sanitation, logistics) in health emergencies or impending health emergencies; 5. Any learning approach/intervention delivery type (e.g., virtual, face-to-face, hybrid, etc.); 6. Training that took place within the past 10 years (2013-2023). This time frame was by request of the WHO. A preliminary search by one of their consultants had indicated relatively little data existed more than 10 years prior.
Exclusion criteria. 1. Degree-building, professional, and vocational training programs. Certificate and accredited programs outside the formal degree building program were excluded except learning approaches and interventions that accrued continuous profession development (CPD) points; 2. Institutional capacity building; 3. Health workforce professional training, unless it was adjusted to focus on JIT training for the specific health emergency; 4. Training that focused on mental health outcome of health professionals and/or stigma; 5. General learning readiness in preparedness for health hazards or emergencies, unless there was evidence on use and usability during real health events; 6. Training which evaluated needs in the manner but went on to create resource for training in the specific emergency was included; 7. Interventions that tested a new modality of an ongoing educational effort due to the health emergency without any adjustment of content; 8. Training about changes to specific systems, (e.g., changing triage systems for COVID-19 was included); 9. Systematic reviews, meta-analyses, and protocols.
Quality Appraisal
Appraisal tools
In consultation with a WHO research expert, we initially identified the following appraisal tools for research methods identified in the WHO Minimum Methodological Expectations document, Section 2.2.
-
• Observational quantitative studies – questionnaires (Specialist Unit for Review Evidence (SURE) Questions to Assist with Critical Appraisal of Cross-Sectional Studies checklist, n.d.).5
-
• Qualitative – open-ended questionnaire survey, interview, focus group, ethnography/ participant observation, and textual analysis such as open-ended questions in surveys (Critical Appraisal Skills Program, CASP, 2023).6
-
• Mixed-method – use of both quantitative and qualitative methods, where the different methods usually address different hypotheses and/or research questions (QuADS, Specialist Unit for Review, 2018).Reference Harrison, Jones and Gardner7 Studies in which 1 method was clearly predominant were categorized under that method.
-
• Comparison/control studies – quantitative studies which included a control group, whether or not participants were randomly assigned (Critical Appraisal Skills Program, CASP, 2023).8
-
• Case study – use of several methods, where methods address the same research question and focus on 1 event/person/location (Critical Appraisal of a Case Study checklist, Center for Evidence-Based Management, July 2014).9
-
• Before-after (pre-post) studies with no control group (National Heart, Lung, & Blood Institute, 2013).10
Appraisal process
Studies selected for evidence synthesis were each independently appraised by teams of 2 researchers each. Disagreements were resolved by consensus, and failing that, via discussion with the PI. We employed the following “traffic light” rating as per request of the WHO methodology expert.
-
• Green light: No/minor methodological concerns
-
• Yellow light: Moderate methodological concerns
-
• Red light: Major methodological concerns
All studies, whatever their overall assessment, were included in the analysis. However, at the data synthesis stage, the proportion of studies with each quality assessment was taken into account when considering the overall methodological limitations in contributing primary studies to each finding. See Table 3 for details.
Data extraction
Mixed-methods framework synthesis was used as the question did not aim to establish the effectiveness of interventions.Reference Brunton, Oliver and Thomas11
The coding framework for data extraction was developed and tested over multiple team sessions, and in consultation with WHO personnel. The following headings were created against which data were extracted and subsequently developed and reported as themes: the planning process, specifics of the intervention, training evaluation results, recommendations, rationale, and implementation considerations.
Articles were then independently coded by 2 researchers each. Disagreements in coding were resolved through discussion, and failing that, by the PI.
Descriptive information
The following descriptive information was recorded for each study.
Country of focus. The nation or nations in which the training intervention took place.
Type of health emergency. As per WHO categories, we coded 6 types of health emergencies: infectious diseases, industrial accidents (chemical, radiological, nuclear, food poisoning), armed conflict, terrorism and mass shooting, and environmental disasters (e.g., earthquakes, floods, landslides, etc.). It had become clear by the time we were extracting data that most reports pertained to COVID-19, so we created a separate category for that. If interventions addressed more than 1 health emergency, both were recorded.
Evaluation design. Observational, mixed method, qualitative, case study, comparison/control, before-and-after.
Modality of training. Online, face-to-face, or hybrid. Training that took place face-to-face with back-up material loaded to a website as an optional supplemental resource was coded as face-to-face.
Type of training. Simulation, course, table-top, and other. When training involved apparently equal components of simulation and course, these were coded as 2 types of training in a single intervention.
Entity conducting training. Hospital or clinic; university or institute; governmental organization; professional organization; other; unable to determine. In cases where training was developed through multi-organizational partnerships, all entities were coded.
Topic of training. The following categories were induced from open-ended recording of topics: hygiene/infection management, detection/diagnosis/prevention, assessing and managing illness, upskilling/cross-department training for critical care, general disease knowledge, administration/communication, and other.
Development process. Studies were coded with respect to presence or absence of information about the training development process, reference to outside experts or guidelines, use of pedagogical expert or learning model, pilot testing, and evaluation of skills (as opposed to knowledge or satisfaction only).
Types of personnel trained. Doctors, assistants/nurse practitioners, nurses, technicians/therapists/pharmacists, residents/students, pre-hospital personnel, and other. When training interventions included multiple types of personnel, all were coded.
Results
Key Characteristics of Selected Studies
Basic descriptive characteristics of the studies are presented in Table 1. As noted in the table, the largest number of studies by far were conducted in the US, followed by Canada and India. Conspicuously absent from the body of evidence were evaluations of training in South America, with just 1 study identified from Brazil and 1 from Colombia. Overwhelmingly, studies focused on COVID-19. Just 9 studies focused on JIT training amid other health emergencies: 8 on other infectious diseases (Ebola, cholera, influenza, and Zika), and 2 on environmental disasters (super typhoon and water contamination). One study addressed both an infectious disease and an environmental disaster. A table with descriptions of individual studies is provided in Appendix B.
Overview of Study Context and Method
* Some studies reported on training conducted in more than 1 country, and 2 reported on more than 1 health emergency. Therefore, numbers in these sections do not add up to 187.
Types of learning innovations developed and used during health emergencies
The first element of the WHO question addressed by this review was what type of learning approaches, interventions, and learning dissemination strategies have been developed and used in health emergencies. Table 2 presents characteristics of studies.
Characteristics of Trainings
Training participants were most often doctors and nurses, although many programs addressed all clinical staff in relevant units including technicians, therapists, pharmacists, physician assistants, nurse practitioners, and even non-clinical workers. Residents were also the focus of training, especially for upskilling to work in critical care contexts. Medical school students were trained for diagnostic and prevention work. A few training programs were geared toward first responders.
The training represented in these articles was primarily organized and conducted by hospitals and secondarily by universities and institutes. These efforts not uncommonly took place in partnership with governmental agencies and professional organizations. Several training programs of global scope were mounted by international organizations like the WHO.
For each of the 6 categories of training topics, the following were typical. Training in detection/diagnosis/ prevention included contact tracingReference Riza, Kakalou and Nitsa12 nasopharyngeal specimen collection,Reference Carlson, Klute and Dowdall13 chest radiology,Reference Xiberta, Boada and Thió-Henestrosa14 and vaccine administration.Reference Fahey, Clucas and Elia15
Training in hygiene/infection management was common, given that all but 2 training programs dealing with infectious diseases. These included donning and doffing personal protective equipment,Reference Jen, Chou and Chang16 and studies testing revised protocols for care to minimize the risk of the COVID-19 virus being highly aerosolized.Reference Diaz and Dawson17
Training for assessing and managing of illness concentrated on procedures for aligning treatment with the needs of the patients arising from rapid onset of infections and availability of health care, such as training personnel for triage, training for specific skills,Reference Chiu, Goldberg and Moses18 and scaling up for anticipated COVID-19 surges.Reference Duffy and Vergara19
Anticipating COVID-19 surges also led to design of training programs for upskilling a range of medical professionals including residents, anesthesiology fellows, nurses, and physicians of other specialties to work in critical care units. Most frequently mentioned specific training topics within this category were airway managementReference Favier, Kimmoun and Gatin20 and ventilation.Reference Almomani, Sullivan and Hajjieh21
Many training programs focused on general epidemiological and other disease knowledge, often through online courseworkReference Magaña-Valladares, Rosas-Magallanes and Montoya-Rodríguez22 but also in face-to-faceReference Sharara-Chami, Sabouneh and Zeineddine23 and mixed modalities.Reference Salehi, de Young and Asamoah24
Finally, administrative/communication topics included developing or explicating new guidelines and protocols, specifically in response to COVID-19, including several utilizing a COVID-19 version of the telemedicine education program.Reference Zhao, Rozenberg and Nourouzpour25
Modality of training was split between online, in-person, and hybrid. Other types of training ranged from online topical discussion groups and mentoring to on-the-job training. Multiple programs initially taught knowledge in an online course, and followed up with in-person discussion or, more frequently, simulation exercises.
Two recommendations arising from the types of interventions that have been developed and used in health emergencies center around training modality (Table 3.) First, experiential learning such as simulations and tabletops was viewed as a powerful way to develop and reinforce skills during health emergencies. For certain topics, virtual simulation provided a safe and effective alternative to in-person simulation in a pandemic, although no clear guidelines were presented in the studies analyzed for what topics would be appropriate for online versus in situ simulation.
Synthesis of Findings and Certainty of Evidence
Second, for situations in which knowledge dissemination was the primary goal, many authors argued for online learning as a good option so long as potential participants had access to the internet and compatible devices. Designing materials for JIT training should make employ the most widely used technology possible. Nevertheless, because of the lack of experimental or quasi-experimental research designs, it is not clear for which purposes online courses alone are sufficient and which require an in-person component to be effective.
Process of Development
The second aspect of the question assigned to our research team was to investigate how interventions were designed and developed and taken up in practice. Although many studies dedicated substantial space to describing content of interventions, 115 of 187 studies contained a description of the process by which interventions were created. Most of those were brief, alluding to creating task forces and partnerships but little more. Interventions appeared to have been developed through culling experiences of qualified internal experts in hospitals or university faculties. Pilot studies of interventions were infrequent (although several authors lamented that there was no time to conduct them). Only 45 studies—approximately one-quarter of the total—evaluated skills as an outcome of training.
Four major recommendations regarding implementation emerged among the 115 studies that devoted some level of analysis to the planning process. First, many training vehicles were developed with the assumption that organizations should plan and implement training rapidly during health emergencies, not only because of the pace at which the impending crisis was approaching, but also because quick action reduced employee stress and promoted trust.Reference Xiberta, Boada and Thió-Henestrosa14, Reference Zucco, Chen and Levy72, Reference Hemann, Blizzard and Carver83 That said, this rapid race to training was not without costs. Many training task forces were unable to do formal needs assessment prior to implementing training, and many post-training interventions incorporated no evaluation mechanism other than a brief satisfaction and self-confidence survey. Ideally, randomized controlled trials should be conducted to provide robust evaluation of actual training effectiveness. When this is not possible, quasi-experimental studies with control groups using a pretest-posttest design should be employed. Furthermore, evaluation of programming needs to measure skill acquisition, not just satisfaction and perceived learning.
Second, many trainings were developed collaboratively through intersectoral and/or international partnerships. Academic medical centers worked with state and local health departments and national ministries of health, as well as with hospitals and hospital systems. Partnerships enabled local public health systems to take advantage of an existing workforce to respond to fluctuating capacity needs.Reference Ledesma, Maroofi and Sabin118 Intersectoral, inter-organizational, and international partnerships should be cultivated in times of non-emergency so that they can be more readily leveraged toward rapid development and dissemination of training when the crisis came.
Third, although some interventions appeared to be primarily the work of a single motivated individual, most depended on thoughtfully constructed task forces, representing a range of skills and backgrounds. Leggott et al.Reference Leggott, Willis and Saseen119 observed that such diverse task forces are essential for taking an agile approach that can respond to the needs of diverse worker roles. OthersReference Lin, Bookman and Sieja121, Reference Garvey and Justice162 cautioned that trust-building in task forces must be intentional, given the natural distrust between segregated teams. Communication, ideally daily, could help develop a shared consensus among normally compartmentalized workers. Despite the potential challenges, several studiesReference Magaña-Valladares, Rosas-Magallanes and Montoya-Rodríguez22 strongly urged that senior management should be involved in planning, noting that their visible presence can lead to greater participation and validation of training efforts. If nothing else, senior personnel often bring with them needed clerical resources, which could enable faster pivoting.
A few articles indicated that using established training models such as the ADDIE model (ISFET, n.d.) and its derivatives could assist task forces and developers in ensuring that important considerations are taken into account. Their generic character makes them flexible. Trainers in need of a structural framework around which to plan might adopt, and possible test, learning models for emergency use.
Evidence Gaps
Several substantial gaps in evidence are clear among studies in this review. Most notably, of 187 studies, only 45 evaluated meaningful outcomes beyond participant satisfaction and self-reported knowledge following the intervention. Although many simulation-based programs did evaluate participants’ post-training skills with checklists, very few studies evaluated ongoing behaviors, or results of training on the health of HCWs and their patients. Among those that did, severalReference Zucco, Chen and Levy72, Reference Couto, PDdTR and Silva77, Reference Curran, Murdoch and van Rensburg160 observed changes in practice and infection rates after training which aimed to reduce COVID-19 infection among HCWs. Additionally, Castillo et al146 reported site visits to clinics after a typhoon in the Philippines; Knochel et al85 evaluated patient length of stay and inpatient mortality; Stickney et al134 checked violations after training in a new surgery protocol during COVID-19; Chiu et al18 tracked prone team interventions and Hands et al95 assessed triage decisions of nursing staff, as well as mortality rate over 2 years. Strelau et al135 collected numbers on learner performance in contract tracing in the field, and how trainees felt the experience had impacted their careers 18 months later. Thus, with a few exceptions, we do not know whether the skills participants learned were practiced in the field, or whether they had any impact.
Furthermore, with few comparative studies, little empirical evidence was found about relative effectiveness of different approaches, optimum frequency and length of training programming, or value of using training models.
Finally, evidence was limited geographically. The disproportionately high number of studies that investigated these issues in the US makes it difficult to contextualize findings with confidence outside of Western, and particularly North American, settings.
Discussion
Much of the previous crisis learning literature favored post-crisis debriefing and assessment to acquire lessons learned.Reference Sellnow, Lane and Sellnow179 However, during the COVID-19 pandemic, HCWs and organizations did not have the luxury of waiting to reflect on lessons learned until the post-crisis period.Reference Croucher and Diers-Lawson2 Instead, they had to learn with urgency while pacing themselves throughout an evolving and lengthy ordeal. The severity and duration of the COVID-19 pandemic created a compelling demand for learning during the most acute phase of the crisis, rather than during recovery and return to normalcy.
Until recently, a gap has existed with respect to studies focused on just-in-time learning in this acute phase. Research examined in this review begins to fill that gap, simultaneously focusing on managing the disease from a public health perspective, protecting health care workers, and upscaling staff to address the surge of patients needing intensive care. It describes experiential learning and knowledge dissemination directed to doctors, nurses, other clinical professionals, and even non-clinical workers, via both online and face-to-face interventions. Principles of training implementation that were extracted from included studies emphasized partnerships, rapid training development and dissemination, institutional support and carefully constructed planning task forces, and use of training models.
Although every effort was made to locate all relevant studies, it is possible that we missed some studies. Furthermore, this was a rapid review, and it was conducted under a highly accelerated time frame. Given the nature of the process, and the limitation in the number of databases searched, it is possible that studies were missed. In particular, although we did not limit our search to English, we did not specifically search any non-English databases. Therefore, studies from non-English speaking nations may have been particularly susceptible to being missed. Additionally, the preponderance of studies examined training in high income nations, which may not be fully applicable in resource-poor settings. Although several projects in this review included learning modules that could work with limited technology. For example, modules were designed for those whose only access to technology was through smart phones, designing materials for just-in-time training should make use of the most widely used technology possible.
Clearly, the potential for broadscale learning observed previously by scholars such as Reierson et al180 and Antonacopoulou & SheafferReference Antonacopoulou and Sheaffer181 in organizations and industries is also relevant to health care professions during pandemics. Existing models depicting crisis learning would benefit from incorporating JIT learning and training into the acute phase of health emergencies. Future research should employ experimental designs to determine relative effectiveness of different training techniques and identify optimal training characteristics, as well as evaluate skills and long-term health outcomes.
Conclusion
This review is the first to look at just-in-time learning and provides novel insights in what is a rapidly evolving type of learning. The full guidance developed by the WHO can be found at: https://iris.who.int/handle/10665/379019.
Supplementary material
The supplementary material for this article can be found at http://doi.org/10.1017/dmp.2025.59.
Data availability statement
Upon article acceptance, data will be available at: https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstars.library.ucf.edu%2Fcgi%2Fpreview.cgi%3Farticle%3D2345%26context%3Ducfscholar&data=05%7C02%7Cann.miller%40ucf.edu%7Ca5aa46e8fd2e48a936bd08dca1d82457%7Cbb932f15ef3842ba91fcf3c59d5dd1f1%7C0%7C0%7C638563196945127669%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=fpQAC94mNeXxi7%2Bv74dJ0RszKQweww0k3s5xMcoY3xA%3D&reserved=0
Acknowledgements
This project was funded by the World Health Organization. We also acknowledge the logistical assistance of Luke Miller.
Funding statement
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the World Health Organization [Grant #2023/1359909-0].
Competing interests
None.
Ethical standard
This article does not contain any studies with human or animal participants.
Open access
