Introduction
Background
Mass-casualty incident (MCI) triage algorithms are intended to allow prehospital providers to quickly and accurately identify patients who require immediate life-saving care and to appropriately allocate limited resources. Emotional obstacles, triage rationale and efficiency, and unfamiliarity with pediatric physiology have been identified as impediments to appropriate triage in pediatric MCIs. Reference Koziel, Meckler and Brown1 While previous studies have compared test characteristics and over- and under-triage rates, the inter-rater reliability (IRR) of most disaster algorithms in the pediatric population has not been established. Reference Wallis and Carley2–Reference Cicero, Overly and Brown5 This deficit has been highlighted by both Jenkins, et al in their 2008 paper addressing the need for an evidence-based approach with mass-casualty triage as well as by Heffernan, et al in their 2018 publication on MCI algorithm accuracy in the pediatric population. Reference Heffernan, Lerner and McKee4,Reference Jenkins, McCarthy and Sauer6 Currently, several MCI algorithms, which guide allocation of responders’ time and resources, are utilized for pediatric patients in MCIs. A comparison of the clinical factors used by five of these MCI triage algorithms to designate patients into categories ranging from deceased to minor injuries are presented in Tables 1 and 2.
Factors Utilized for Triage in each MCI Algorithm - Part 1
Abbreviations: MCI, mass-casualty incident; J-START, Combined Simple Triage and Rapid Treatment and JumpSTART; SALT, Sort Assess Life-Saving Intervention Treatment; PTT, Pediatric Triage Tape; CF, CareFlight; STM, Sacco Triage Method.
a Raters instructed to assume apnea if patient required bag valve mask ventilation.
Factors Utilized for Triage in each MCI Algorithm - Part 2
Abbreviations: MCI, mass-casualty incident; J-START, Combined Simple Triage and Rapid Treatment and JumpSTART; SALT, Sort Assess Life-Saving Intervention Treatment; PTT, Pediatric Triage Tape; CF, CareFlight; STM, Sacco Triage Method; AVPU, alert/verbal/responsive to pain/unresponsive; GCS, Glasgow Coma Score.
a Raters inferred these subjective factors from other information provided.
Simple Triage and Rapid Treatment (START), and its pediatric modification, JumpSTART (initially created for eight years of age and under), combined as J-START, are widely used in the United States. Reference Jenkins, McCarthy and Sauer6 One national survey found that it is used by 71% of Emergency Medical Service (EMS) systems employing a pediatric disaster triage algorithm. Reference Nadeau and Cicero7 As such, START is one of the most well-studied disaster algorithms in the adult and pediatric population and is the only algorithm with literature on the IRR of provider application in the pediatric population. Reference Cicero, Overly and Brown5,Reference DeGeorge, Glick, Ackerman and Wu8
The Sort, Assess, Life-Saving Interventions, Treatment/Transport (SALT) algorithm was proposed in 2008 as a new national standard. Currently, SALT is the only disaster algorithm meeting the key components in the Model Uniform Core Criteria, Reference Cicero, Riera, Northrup, Auerbach, Pearson and Baum9,Reference Lerner, Schwartz and Coule10 a consensus guideline by the Federal Interagency Committee on EMS (Washington, DC USA) that recommends 24 criteria for all MCI systems with four general categories: general considerations, global sorting, life-saving interventions, and individual assessment. The Centers for Disease Control and Prevention (Atlanta, Georgia USA) funded a workgroup that developed the Model Uniform Core Criteria and created the SALT triage algorithm. The SALT algorithm is unique in its sequence, with global sorting followed by life-saving intervention and then categorization of triage. It is the only commonly used algorithm utilizing providers’ subjective assessments--whether the patient is “likely to survive given resources” and whether there are “minor injuries only.” Reference Lerner, Schwartz and Coule10
The Pediatric Triage Tape (PTT) is the pediatric adaptation of the Trauma Sieve, a disaster algorithm commonly used in the United Kingdom and parts of Australia which evaluates ability to walk, airway patency, respiratory rate, and pulse rate. Reference Jenkins, McCarthy and Sauer6 The PTT incorporates patient height and/or weight, relying upon measurements obtained on the backboard to determine appropriate vital sign ranges for pediatric patients.
CareFlight (CF), developed by Nocera and Garner in 2001 and widely used in Australia, is arguably the simplest of the algorithms with no need for measurements or determination of heart rate or respiratory rate. Reference Nocera and Garner11 It is intended for use in all ages and consists of simple yes/no questions with no adjustment for differences in developmental level or physiology of pediatric patients.
The Sacco Triage Method (STM) is also designed for use in all ages. This triage algorithm requires the most calculations, Reference Cross and Cicero3,Reference Lerner, Cone and Weinstein12 a respiratory rate, heart rate, and motor response, followed by an adjustment for age. The STM yields a score of zero to 14 which can be translated into five more traditional categories of Likely Expectant (score of 0-1), Critical (2-4), Compromised/Salvageable (5-8), Delayed/Slow (9-10), and Minor (11+). Reference Jenkins, McCarthy and Sauer6
Importance and Goals
The goal of this pilot study was to measure IRR of five MCI algorithms in the pediatric population. It was hypothesized that algorithms with the most objective criteria would have the highest IRR and that SALT, the only algorithm incorporating subjective findings, would have the lowest IRR. The secondary aims were to evaluate the inter-rater agreement and inter-MCI algorithm agreement.
Methods
This was a comparative analysis of MCI scoring systems using a pre-existing research database. This study was approved by the Institutional Review Board at the University of California, San Diego (San Diego, California USA; protocol #170467).
Dataset Identification
An existing, de-identified dataset of 253 patients less than 14 years of age who were transported by ambulance and had a trauma team activation to a single Level 1 Pediatric Trauma Center in Los Angeles (California USA) from July 2010 through November 2013 was used. The selected patient dataset was purposely weighted to include a high proportion of those severely injured. Prehospital data abstracted included: age; size including weight, height in centimeters, and/or color designation as measured using a Broselow Tape (2017 Edition; Armstrong Medical; Lincolnshire, Illinois USA); all EMS free-text documentation and EMS interventions performed; ability to walk; level of responsiveness using the AVPU scoring system (A = alert, V = verbal, P = responsive to pain, U = unresponsive); field Glasgow Coma Score (GCS); motor component of the GCS; strength of pulse; heart rate in the field; presence of respiratory distress such as apnea, retractions, grunting, tachypnea, or wheezing; field respiratory rate; and perfusion quality. Poor perfusion was defined as documentation of cool or mottled extremities, capillary refill over three seconds, or explicit EMS documentation of poor perfusion. Given the inability to distinguish whether the patient began breathing with repositioning and five breaths, if bag-mask-ventilation was required during EMS care, apnea and a respiratory rate of zero was assumed. Information collected but not provided to the raters included: gender, injury mechanism, Injury Severity Score (ISS), length-of-stay in days, and disposition from emergency department (ED).
Study Design and Measurements
Training
An attending physician who had completed fellowships in both Pediatric Emergency Medicine and EMS trained three independent raters. Raters were volunteer research assistants: one pediatric resident, one medical student who was also an emergency medical technician (EMT), and one pediatric resident with prior EMT experience. Each rater was separately trained in the five triage algorithms with review and discussion of each algorithm. The attending physician provided a sample set of 25 patients from the full data set. The sample dataset consisted of five randomly selected patients from each triage designation (five Green, five Yellow, five Red, and five Black), plus an additional five randomly selected patients. Raters used the sample set for baseline evaluation. A 90% concordance with the attending’s triage of the 25 patients was determined to be adequate IRR to proceed to triage of the full dataset. All three raters achieved greater than 90% concordance on each algorithm on the first attempt.
Triage
A Microsoft Excel (Microsoft Office 2011; Microsoft Corp.; Redmond, Washington USA) sheet containing the 253 patients’ prehospital information followed by six columns (J-START, SALT, PTT, CF, STM color, and STM number) was given to each rater; patients were in the same order for each rater. Raters were instructed to assign each patient a triage designation in each column. For children eight years of age and younger, JumpSTART was utilized, and for patients over eight years of age, the adult START was utilized for the combined triage algorithm (J-START). Both SALT and STM normally sort patients into five triage categories. For consistency, all five algorithms were collapsed into four possible triage designations: Black, Red, Yellow, or Green. For the SALT triage method, the expectant and dead categories were combined into a Black designation. For STM, likely minor (a score of 11-12) was considered a Green designation; delayed/slow (a score of 9-10) Yellow; compromised/salvageable (a score of 5-8) Red; and critical and likely expectant (0-4) were combined into the Black category. Raters could reference a PDF document that included the five algorithms, including the combined J-START algorithm, instructions on converting STM number into a color category, and a list of vital sign parameters based on height/weight for the PTT. The order in which the triage should be performed was not specified and there was no time limit. Time to complete triage of each patient or the complete dataset was not recorded. Raters did not discuss the process with each other.
Characteristics of the Dataset
Patient age ranged from 25 days to 13 years with a mean age of nine years. There was a wide distribution of age, with the majority in the 9-13 (34%) and 2-to-4-year-old (27%) ranges. The male to female ratio of patients was 2:1. The mechanism was vehicle versus pedestrian or bicycle in over one-third of patients and fall was the mechanism in one-quarter of patients. Almost one-half were discharged home after ED evaluation, 18% were admitted to an intensive care unit without going to the operating suite first, 8.5% went directly to the operating suite, and 10% died. The median ISS was 3.5 with a range of one to 75.
Outcomes and Analysis
The primary outcome measure was the overall IRR for each of the five MCI triage algorithms. Total numbers of Black, Yellow, Red, and Green triage designations assigned by each rater for each algorithm was compared for dyads of raters using Spearman’s correlation. The IRR was reported as weighted kappa scores with 95% confidence intervals (CI) for each of the triage algorithms overall. A kappa of 0.81-1.0 was defined as excellent reliability, 0.61-0.8 as substantial, 0.41-0.6 as moderate, 0.21-0.4 as fair, and 0-0.2 as poor.
Secondary outcome measures were the inter-rater agreement (unadjusted proportion of patients for which the raters agreed on the triage level for each MCI algorithm) and inter-MCI algorithm agreement (the proportion of patients who were triaged to the same color by each MCI triage algorithm). Descriptive statistics were used to report inter-rater agreement and the inter-MCI algorithm agreement.
Results
Of the 253 patients in the dataset, six were excluded because they lacked one or more triage designations by one of the raters. Two-hundred forty-seven patients were included in the analysis.
The IRR for the five MCI algorithms is presented in Table 3. Overall, J-START and CF had the highest reliability with a kappa of 0.94 or higher (0.9-1.0, 95% CI for overall weighted kappa). Reliability for STM varied from substantial to excellent (0.79-1.0) and SALT varied from poor to substantial (0.19-0.68).
Spearman Correlation Inter-Rater Reliability between Triage Methods
Abbreviations: J-START, Combined Simple Triage and Rapid Treatment and JumpSTART; SALT, Sort Assess Life-Saving Intervention Treatment; PTT, Pediatric Triage Tape; CF, CareFlight; STM, Sacco Triage Method.
The least variability was found in patients triaged to the Black category; only two patients who were categorized as Black at least once did not have uniform agreement across all algorithms. Yellow and Green categories had the highest variability among raters. Table 4 details the proportion of patients triaged into each color within each MCI algorithm. In total, STM categorized 90.6% of patients as Green; SALT had the highest number of patients in the Red and Yellow categories, the least in the Green category, and the highest variability between raters in the Yellow and Green categories. There was less than 2.5% variability between all three raters for all algorithms except for SALT (Figure 1). Across algorithms, the Yellow designation had the greatest variability (0.9%-39.7%), followed by Green (0.4%-38.9%; Figure 2). Specific designations per rater for each algorithm are listed in Table 4.
Triage Designation by MCI Algorithm and Rater
Abbreviations: MCI, mass-casualty incident; J-START, Combined Simple Triage and Rapid Treatment and JumpSTART; SALT, Sort Assess Life-Saving Intervention Treatment; PTT, Pediatric Triage Tape; CF, CareFlight; STM, Sacco Triage Method.
Variability between Raters within MCI Algorithm.
Abbreviations: MCI, mass-casualty incident; J-START, Combined Simple Triage and Rapid Treatment and JumpSTART; SALT, Sort Assess Life-Saving Intervention Treatment; PTT, Pediatric Triage Tape; CF, CareFlight; STM, Sacco Triage Method.
Variability between Raters within Each MCI Color Designation.
Abbreviation: MCI, mass-casualty incident.
The lowest inter-rater agreement was seen in application of the SALT algorithm with the three raters having 100% agreement in only 53% of patients (Table 5). Inter-rater agreement was high with CF, J-START, STM, and PTT algorithms with 100% agreement between raters in 96%-98% of cases. The three raters had 100% agreement for 93.9% of patients when applying the PTT algorithm. The PTT algorithm was the only algorithm with complete disagreement agreement for any patients; no raters agreed on the triage category for two patients (0.8%).
Inter-Rater Agreement
Abbreviations: START, Simple Triage and Rapid Treatment; SALT, Sort Assess Life-Saving Intervention Treatment; PTT, Pediatric Triage Tape; CF, CareFlight; STM, Sacco Triage Method.
There were several clusters of triage color cohorts across the algorithms. There was a cohort of eight patients who were triaged Black across all five MCI algorithms by all raters. There was a “Red Cohort”- 20 patients primarily categorized as Red among J-START, SALT, and CF; a “Yellow Cohort”- 73 primarily categorized Yellow by J-START, SALT, PTT, and CF; and a “Green Cohort”- 146 primarily categorized Green among J-START, SALT, PTT, CF, and STM. Within the color cohorts, several patterns of disagreement were identified and these are listed in Figure 3. Both STM and PTT had a high level of disagreement in the “Red Cohort;” SALT had the highest level of disagreement among the “Yellow Cohort” and “Green Cohort.”
Inter-MCI Algorithm Agreement by Color Cohort.
Abbreviations: MCI, mass-casualty incident; J-START, Combined Simple Triage and Rapid Treatment and JumpSTART; SALT, Sort Assess Life-Saving Intervention Treatment; PTT, Pediatric Triage Tape; CF, CareFlight; STM, Sacco Triage Method; R1, Rater 1; R2, Rater 2; R3, Rater 3.
Discussion
An ideal MCI algorithm would have a high IRR, being consistent regardless of who is applying the tool. It had been hypothesized that an increase in subjective components of a triage algorithm would lead to a decrease in the IRR of that algorithm. In this pilot study applying MCI algorithms to a retrospective cohort of pediatric trauma patients, the algorithm with the most subjective components, SALT, had the lowest IRR. In contrast, J-START, PTT, CF, and STM all had excellent IRR.
Both CF and J-START had similar inter-rater agreement and inter-MCI algorithm agreement across all triage colors. Overall, PTT had the greatest variance in the Red Cohort but performed similarly across groups in the Black, Yellow, and Green Cohorts; SALT performed similarly within Black and Red Cohorts, but stood out as highly variable within Yellow and Green Cohorts. Lastly, STM performed well among Black and Green Cohorts, but had substantial differences among the Yellow Cohort and had a high variability in the Red Cohort.
In comparison to these findings of high IRR in the J-START algorithm, a research abstract published in Annals of Emergency Medicine found moderate IRR (kappa = 0.44) in emergency physicians versus fair in pediatricians (0.38 kappa) triaging 11 pediatric patients using JumpSTART. Reference DeGeorge, Glick, Ackerman and Wu8 A 2013 study of 37 resident trainees applying JumpSTART to a hands-on mannequin simulation of a 10-patient MCI found IRR to be generally higher than 0.70 with exception of one ambulatory patient. Additionally, there was no difference in triage by post-graduate year. Reference Cicero, Riera, Northrup, Auerbach, Pearson and Baum9 This tabletop study found that J-START has an excellent IRR with minimal variance and strong agreement between the three raters, but results may have differed in a hands-on simulation. One possibility is that variability was introduced by inconsistency in the assessment and interpretation of physical findings. Of mention, CF has not previously been evaluated but performed just as well as J-START in this study.
The SALT algorithm was found to have the highest variability, particularly among the Green and Yellow categorizations. This finding was not surprising as the branch point between the Green and Yellow triage classification was raters’ response to the question: “minor injuries only?” As SALT is the algorithm most adherent with the Model Uniform Core Criteria for MCI triage, and is the algorithm being promoted at a national level in the United States, determining how users apply the algorithm is vital. Reference Lerner, Schwartz and Coule10 A provider performing triage who classifies only a few injuries as minor will lead to more transports while a provider who classifies more injuries as minor may lead to more Green categorizations, potentially resulting in under-triage. Regardless, the inconsistency between raters seen in this pilot study is concerning.
As a proprietary system with a numerical output intended to prioritize resources, STM has been found to predict mortality well in prior studies. Reference Cross and Cicero3,Reference Lerner, Cone and Weinstein12 The age-adjusted STM was used even though that version has not been demonstrated to improve mortality prediction in the pediatric population. Reference Sacco, Navin and Fiedler13 No prior studies have evaluated how STM compares to other disaster algorithms when the STM number is split into triage color categories. Zero-to-four was used as a cut off for Black in the STM and found that cut point to have a high level of agreement with Black categorization in all four of the other algorithms. In comparing STM categorization to the other algorithms, an overwhelming number of patients in STM were categorized as Green, including all of the patients triaged as Yellow in J-START, SALT, and CF and some who were triaged as Red by a majority of the other algorithms. While this study does not look at the inherent accuracy of an algorithm, or the accuracy of the rater applying the algorithm, the fact that STM has a remarkably higher proportion of Green patients compared to other algorithms suggests the possibility of under-triage. The clinical significance of this under-triage when STM is applied to a pediatric population includes non-detection of significant illness and injuries.
An area of particular interest that deserves further study is the 20 patients who were found by J-START, SALT, and CF to be primarily triaged as Red (ie, the Red Cohort). In this cohort, the algorithms with the most objective criteria, STM and PTT, had the least amount of inter-rater and inter-algorithm agreement. This may be in part attributed to the limitations of only using the documented first set of prehospital vital signs as the objective data for MCI algorithm triage. For example, in a five-year-old patient with a gunshot wound to the head with flexural posturing requiring bag-mask-ventilation, the patient had a documented heart rate of 120 and respiratory rate of 33. Given these normal for age vital signs, despite a catastrophic injury, both the STM and PTT triage the patient as a Yellow. Additionally, the PTT algorithm does not take mental status into account, which may account for a large proportion of the patients triaged as Yellow by PTT and Red by other algorithms.
Limitations
This is a pilot study to allow comparison of five triage systems in a known dataset of trauma patients. There are a number of limitations that can be addressed with further study of selected triage methods in a more realistic simulation. First and foremost, the reliability of the algorithms in a high-stress environment and the timeliness of application were not analyzed. This study evaluated disaster algorithms utilizing a retrospective dataset in a tabletop drill. Raters did not assess the patients themselves and were not limited in time. This created an artificial environment without the pressure of physical triage.
The visual presentation of the data may have influenced the IRR within and between some of the systems. In this study, the raters were provided a data sheet with patients’ prehospital data; raters filled in six columns for each of the triage algorithms (including STM color and number). This led to an artificial environment and the potential to replicate the same triage category across algorithms. Similarly, those systems with redundant elements would naturally score closer to each other if raters were scoring them consecutively. As the order of triage completion and time to triage were not controlled in this study, it cannot be determined to what extent these elements may have affected the outcomes.
The raters were provided with actual prehospital vitals along with the documented paramedic narrative of findings and treatments. However, response to repositioning of jaw and administration of rescue breaths was not consistently documented. The raters were taught to assume apnea if respiratory rate was recorded as zero or if patient required assisted ventilation. However, in the only two patients with inter-rater differences between Red and Black, both patients received assisted ventilation but a respiratory rate was documented for both. This is likely a systemic data entry issue as presumably paramedics are entering the respiratory rate at which they are bagging the patient. If true, use of large datasets, such as the National Emergency Medicine Services Information System or the National Trauma Data Bank, to evaluate MCI algorithms may have similar limitations.
Finally, forcing the STM from its numerical designations into the four categories required by the other algorithms may have affected the IRR of that method, if the score fell above or below the cutoff for color. Comparing individual scores demonstrated the least variability with this triage method. Changing the cutoff points for Yellow or Red designations may have altered the IRR between this method and the others. These may require re-examination for the pediatric population as most of the patients in this data set were triaged Green.
Conclusions
In this study, J-START and CF were found to have the highest IRR, highest inter-rater agreement, and highest level of agreement within MCI algorithms across all triage colors. Overall, J-START, PTT, and CF were found to have excellent IRR. Predictably, SALT was found to have the lowest IRR, particularly in the Yellow and Green designations, but it performed well in Red and Black categories. As had been hypothesized, the algorithm with the subjective components, SALT, had the highest rate of variability, and the most concrete objective algorithm, STM, had the least variability. Future studies should explore IRR and agreement in a hands-on simulation of these MCI algorithms.
Conflicts of interest
The authors declare none.
