CLINICIAN’S CAPSULE
What is known about the topic?
EM is a specialty with a broad knowledge base making it daunting for a junior resident to know where to begin the acquisition of evidence-based knowledge.
What did the study ask?
What list of “top papers” was formulated in the field of EM using a national Canadian Delphi approach to achieve an expert consensus?
What did the study find?
A list was produced of top studies relevant for Canadian EM physicians in training.
Why does this study matter to clinicians?
The list produced can be used as an educational resource for junior residents.
INTRODUCTION
The ability to critically appraise and apply evidence to clinical practice is a skill required by emergency medicine (EM) specialists and is an accreditation requirement for residency programs.Reference Reisdorff, Hayes, Carlson and Walker 1 - Reference McKibbon, Wilczynski and Haynes 4 However, because EM is a generalist specialty, it can be daunting for a junior learner to know where to even begin their acquisition of evidence-informed EM knowledge. There have been many attempts to create lists of sentinel papers for junior learners to use as a springboard to developing an evidence-based clinical practice. Lists of “top cited papers” in EM have been previously compiled, but these lists are often arbitrary and based on the idiosyncrasies and experiences of the curators.Reference Shuaib, Acevedo and Khan 5 , Reference Tsai, Lee, Chen and Yen 6 In the growing age of FOAM (Free Open Access to Medical Education), many online resources have created lists of “top papers.”Reference Grayzel and Wiley 7 - 9 These lists can be an excellent starting point but tend to be compiled off of the opinions of one or two authors or are focused on recent publications, often excluding sentinel papers that have been published in the past.Reference Huis in’t Veld, Nguyen, Martinez and Mattu 10 , Reference Mattu 11 , Reference Junck, Bender, Ilgen and Diller 14
The Colorado Compendium created a list of 100 sentinel articles for EM residents.Reference Druck, Pearson and Claud 12 , Reference Claud, Druck and Pearson 13 However, it reflects the opinions of a single American academic centre. Similarly, the best evidence in emergency medicine (BEEM) scoreReference Worster, Kulasegaram and Carpenter 15 has been created to define lists of more recent evidence but not used to aggregate papers that have historically changed practice. While other groups have reported aggregating papers via crowdsourcing from senior residentsReference Archambault, Blouin and Poitras 16 , Reference Archambault, Thanh and Blouin 17 at a national review course,Reference Blouin, Dagnone and O’Connor 18 no systematic survey of Canadian academic EM faculty responsible for training junior emergency physicians has been conducted to date.
The objective of this study was to systematically generate a list of “top EM papers” (i.e., practice changing results based on strong methods) to assist junior Canadian EM trainees in developing evidence-informed clinical practice.
METHODS
Participants were recruited from all Royal College of Physicians and Surgeons of Canada specialist EM programs. All 14 program directors were contacted via email with a description of the study and a request to nominate two EM experts from their institution with an interest or background in emergency medical education or research.
Delphi survey
Our modified, three-round Delphi survey was conducted electronically with the aim of gathering and aligning opinions from a national set of experts with different perspectives with the hope of defining a more generalizable list of initial papers.Reference Akins, Tolson and Cole 19 - Reference Humphrey-Murto, Varpio and Wood 21 The Delphi rounds were distributed in 3-week intervals using Google Forms (Google, Mountain View, CA), and delivered via email. The nationwide Delphi was conducted between July and November 2016.
Round 1 consisted of an open-ended questionnaire in which participants were asked to generate a list of what they felt were the most important and influential papers related to the field of EM. Results of Survey 1 were compiled for use in Survey 2. In round 2, participants were asked to rank each paper on the compiled list as Include, Not include, Important but not top priority, or Unfamiliar. Participants were also asked to provide a rationale for their scoring. In round 3, participants viewed the percentage of participants who thought each paper should be included in the final list and the free-text comments. Participants then selected whether they agreed or disagreed with the inclusion. Study results were then categorized into six categories for analysis: 100% inclusion, 70% inclusion, 50% inclusion, primarily “good to know,” no consensus (less than 50% agreement), and primarily exclude or unfamiliar.
Ethics
We received approval from the Hamilton Integrated Research Ethics Board (HIREB) to conduct this study.
RESULTS
Participants
A total of 13 participants participated in all three rounds of the study. Not all participants were involved in each round of the study. Eight of the 14 sites with Royal College training programs in Canada were included. Appendix A lists the breakdown of the sites and participants.
Delphi results
From the first round of the Delphi, a total of 120 papers were nominated (Appendix B). By round 3, there were 10 studies with “100% agreement” for final inclusion. Nineteen other studies included in the final list were endorsed by more than 70% of participants as “must include” (Table 1).
Top studies in Delphi survey
DISCUSSION
We identified 29 top papers for junior Canadian EM trainees to use in developing evidence-informed practice. Interestingly, even amongst experts in the field, there is a significant variability in what is considered “must know” studies. This represents a structured attempt with a national consultation of experts to ascertain foundational papers for Canadian trainees.
Although other studies have created similar reading lists for EM,Reference Mattu 11 - Reference Claud, Druck and Pearson 13 , Reference Worster, Kulasegaram and Carpenter 15 - Reference Archambault, Thanh and Blouin 17 this study uses a more robust methodology to systematically create a consensus list with national input. The variability seen in Survey 1 demonstrates how much literature exists that is relevant to the field of EM. In comparing our list with the popularly cited 2016 list from the Academic Life in Emergency Medicine blog,Reference Junck, Bender, Ilgen and Diller 14 there is some degree of overlap; 11 studies were included on both lists. Some reasons for discrepancies between the two lists include a higher prevalence of Canadian-based EM literature, newer studies, and a smaller final list for our study. Future directions may include using a similar methodology to create an annual “must read” lists for continuing professional development and revalidation.Reference Levinson 22 - 24
Limitations
Firstly, the sample size was much smaller than originally anticipated. Although there is no consistent standard in the literature for a Delphi sample,Reference Akins, Tolson and Cole 19 a total of 13 participants is within the typical size range for this type of study. We were also able to achieve only 50% representation from a diversity of geographical regions; the exclusive use of an English language survey may have been a barrier with a francophone faculty.
CONCLUSIONS
Using a modified Delphi technique, we aggregated a nationally endorsed list of top papers for junior EM trainees. This list may anchor an evidence-informed reading list for junior EM trainees.
Acknowledgement
We thank Dr. Yasmin Jajarmi for her contributions on this project while she was a medical student.
Competing interests
None declared.
SUPPLEMENTARY MATERIAL
To view supplementary material for this article, please visit https://doi.org/10.1017/cem.2018.404
