The 2019 novel coronavirus (SARS-CoV-2) was discovered following an unexplained viral pneumonia case in Wuhan, China, in December 2019, and was named by the World Health Organization (WHO) on January 12, 2020. Although most patients with novel coronavirus disease 2019 (COVID-19) only suffered cold-like symptoms and recovered without special therapy,Reference Rothan and Byrareddy1 a few patients became dangerously ill.Reference Yang, Yu and Xu2 Previous reports described a mortality rate of 11% to 62% among severely or critically ill patients with COVID-19.Reference Chen, Zhou and Dong3-Reference Weiss and Murdoch6 The mortality inpatients aged ≥65 y was higher than that in younger patients.Reference Yang, Yu and Xu2 Hitherto, identifying patients with life-threatening illness and enabling them to access early advanced medical intervention may contribute to reducing mortality. Due to the large number of patients with COVID-19 and relatively insufficient medical resources, emergency screening of such patients has become challenging.Reference Ji, Ma and Peppelenbosch7
Currently, although clinicians use indices such as the white blood cell count,Reference Rothan and Byrareddy1 C-reactive protein concentration,Reference Rothan and Byrareddy1 interleukin,Reference Rothan and Byrareddy1 or d-dimerReference Weiss and Murdoch6 to classify COVID-19 cases for severity, these prognostic factors are nonspecific and may involve a delay. Therefore, these indicators were unsuitable for screening, and there is no particular scoring system created for COVID-19 patients. Thus, the second-best choice is applying existing scoring tools. We focused on existing early warning scores (EWSs) to evaluate the prognosis of COVID-19 patients. The EWSs were widely applied in the emergency department (ED) for assisting emergency physicians to predict the risk of deterioration, monitor the evolution of the patient, and make clinical decisions, specifically to promote the safety of the critical patient. So far, many models of EWSs have been developed, such as National Early Warning Score (NEWS), National Early Warning Score 2 (NEWS2), Hamilton Early Warning Score (HEWS), Modified Early Warning Score (MEWS), and Standardized Early Warning Score (SEWS) (see Supplemental Table 1, which is available online). In 2012, the Royal College of Physicians (RCP) proposed NEWS,Reference Day and Oxton8 whose parameters included heart rate, systolic blood pressure, temperature, respiratory rate, oxygen saturation, supplemental oxygen, and mental status. NEWS has now undergone extensive validation.Reference Silcock, Corfield and Gowens9,Reference Hodgson, Dimitrov and Congleton10 Five years later, the RCP updated NEWS to NEWS2,Reference Zaidi, Bader-El-Den and McNicholas11 which used the same parameters but altered their weights. HEWS, whose parameters are also the same as those of NEWS but with different weights, was developed in 2015, has successfully completed pilot-testing in the ED setting,Reference Skitch, Tam and Xu12-Reference Fernando, Fox-Robichaud and Rochwerg14 and is being used in some medical centers in Canada. MEWS was originally developed in 2000 to facilitate timely recognition of patients with established or impending critical illness.Reference Subbe, Kruger and Rutherford15 Its parameters are heart rate, systolic blood pressure, respiratory rate, body temperature, and mental status. Some studiesReference Xie, Huang and Liu16-Reference Jiang, Jiang and Mao18 have suggested that MEWS may be suitable for use in the ED and may allow improvement in the quality and safety of care. SEWS was proposed by Paterson’s teamReference Paterson, MacLeod and Thetford19 in 2006. Its parameters are respiratory rate, temperature, systolic blood pressure, oxygen saturation, heart rate, and mental status. Paterson et al. illuminated that SEWS correlates with in-hospital mortality.Reference Paterson, MacLeod and Thetford19
Currently, the EWS series is widely used in EDs.Reference Smith, Chiovaro and O’Neil20 Although some studiesReference Gautret, Lagier and Parola21-Reference Fusi-Schmidhauser, Preston and Keller23 on clinical therapies for severely ill COVID-19 patients have used EWSs as tools to stratify risk, to our knowledge, no studies comparing the performance of these scores in COVID-19 patients. Thus, the purpose of our study was to evaluate the use of these EWSs as in-hospital mortality indicators for COVID-19 patients.
METHODS
Study Design
This was a retrospective analysis of data obtained from an electronic register of COVID-19 patients who visited our ED in early 2020. The data accessed were anonymized.
The Diagnosis and Treatment Plan of Novel Coronavirus Pneumonia (Version 6), which was issued by the National Health Commission of China, was adopted for diagnosis, classification, and treatment of COVID patients (see Supplemental Table 2). The 5 EWSs were applied to the data, and their capacity to predict in-hospital mortality of patients was assessed.
Study Setting
The emergency medical team, containing more than 100 members, was deployed to Wuhan city during the COVID-19 epidemic and managed the temporary COVID-19 ward independently in a hospital in Wuhan.
The local Institutional Review Committee approved the study and waived the need for obtaining informed consent from the study subjects, owing to the study design. The study complied with the international ethical guidelines for human research, including the Declaration of Helsinki.
Subjects
We enrolled 367 cases aged ≥18 y diagnosed with COVID-19 between January 13 and April 13, 2020. Excluding 48 cases with missing data, 319 were analyzed (Figure 1). The following data were retrieved from the electronic register: basic information (sex, age, diagnosis, and chronic diseases), initial vital signs (body temperature, heart rate, blood pressure, and respiratory rate), consciousness on patients’ visit, oxygen saturation level on patients’ visit, and other variables used to calculate the 5 scoring systems. The outcome was the patient’s death or survival at discharge.
Flow chart of cases exclusion.
From all 367 cases, we excluded 21 cases missing information on oxygen saturation, 16 cases missing record on oxygen therapy and 11 cases missing information on mental status. Finally, we analyzed 319 cases.
Statistical Analysis
The data analysis was conducted using the IBM Statistical Program for Social Sciences Statistics 20.0 (SPSS; IBM Corp., Armonk, NY) and MedCalc Statistical Software (Version 18.2.1; MedCalc Software Ltd., Ostend, Belgium). Continuous variables are presented as mean ± standard deviation or median (25%quartile, 75%quartile), as appropriate. Categorical variables are described as composition ratios (%). The comparison of continuous variables used Student’s t test or the Mann-Whitney U test, and that of categorical variables used the chi-squared test or Fisher’s exact test, as appropriate. Receiver operating characteristic (ROC) curves were used to evaluate the capacities of the 5 scoring systems to predict mortality. The area under the ROC curve (AUC) and its 95% confidence interval (95% CI) were calculated. An AUC > 0.75 was regarded as indicating a test of acceptable clinical value and an AUC > 0.97 as a test of high clinical value.Reference Fan, Upadhye and Worster24 The AUCs were compared using the DeLong test,Reference DeLong, DeLong and Clarke-Pearson25 and the best demarcation point of each scoring scale was determined as the maximum value of the Youden index. Then, the corresponding accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) corresponding to the best cutoff point of each score were calculated. Finally, we calculated the worst Sequential Organ Failure Assessment (SOFA) score during the first 24 h. And we analyzed the correlation between the worst SOFA score and each EWS through Spearman’s rank correlation test. For all analyses, a P value < 0.05 was considered to indicate statistical significance.
RESULTS
Baseline Analysis
The mean age of survivors and nonsurvivors was 53.93 ± 15.32 and 71.63 ± 13.16 y, respectively, and this difference was significant (P < 0.05).The length of hospital stay of survivors, which refers to the duration from admission to discharge, was 17[9.25, 29] d. And the length of hospital stay of nonsurvivors, which refers to the duration from admission to death, was 4.5[3, 11.25] d. These 2 groups differed significantly with respect to gender, respiratory rate, pulse rate, heart rate, peripheral oxygen saturation, consciousness, and NEWS, NEWS2, HEWS, MEWS, and SEWS scores (P < 0.05). Details are shown in Table 1.
Comparison of the Baseline Characteristics of Survivors and Non-survivors
Abbreviations: PR, pulse rate; HR, heart rate; RR, respiratory rate; SBP, systolic blood pressure; DBP, diastolic blood pressure; MAP, mean arterial pressure; SpO2, peripheral oxygen saturation; GCS, Glasgow Coma Scale score; AVPU: AVPU assessment (A, alert; V, respond to verbal commands; P, respond to painful stimuli; U, unconscious); NEWS, National Early Warning Score; NEWS2, National Early Warning Score2; HEWS, Hamilton Early Warning Score; MEWS, Modified Early Warning Score. SEWS, Standardized Early Warning Score
* P < 0.05.
Overall Analysis
The results of ROC curve analysis were showed in Table 2 and Figure 2. The AUCs of SEWS, HEWS, NEWS, NEWS2, and MEWS were 0.841, 0.821, 0.809, 0.809, and 0.670 (P < 0.05). The optimal cutoff values of SEWS, NEWS, NEWS2, HEWS, and MEWS were 7, 10, 10, 8, and 5, respectively (Table 2).
The ROC curves of five EWSs for overall cases
ROC: Receiver operating characteristic; NEWS: National Early Warning Score; NEWS2: National Early Warning Score2; HEWS: Hamilton Early Warning Score; MEWS: Modified Early Warning Score. SEWS: Standardized Early Warning Score.
Performance of 5 Scoring Systems in Predicting in-Hospital Mortality of COVID-19 Patients
Abbreviations: AUC, area under the curve of the receiver operating characteristic; 95% CI, 95% confidence interval; Sen., sensitivity; Spe., specificity; PPV, positive predictive value; NPV, negative predictive value; LR+, likelihood ratio positive; LR-, likelihood ratio negative; NEWS, National Early Warning Score; NEWS2, National Early Warning Score2; HEWS, HamiltonEarly Warning Score; MEWS, Modified Early Warning Score; SEWS, Standardized Early Warning Score; N/A, Not available because the denominator is zero.
* P < 0.05.
Based on the best Youden index, an optimum cutoff value was used to predict in-hospital mortality using each EWS. The cutoff values for each score, together with the sensitivity, specificity, PPV, and NPV are shown in Table 2 and Supplemental Table 3.
Pairwise comparisons of the AUCs associated with the 5 EWSs, showed significant differences among 4 pairs, including SEWS versus MEWS, NEWS versus MEWS, NEWS2 versus MEWS and HEWS versus MEWS. There was no statistical difference between other paired values (Table 3).
Pairwise Comparison of AUC of 5 Scoring Systems for Predicting in-Hospital Mortality of COVID-19 Patients
Abbreviations: △AUC, difference between areas under curves; 95% CI, 95% confidence interval; NEWS, National Early Warning Score; NEWS2, National Early Warning Score2; HEWS, Hamilton Early Warning Score; MEWS, Modified Early Warning Score; SEWS, Standardized Early Warning Score.
* P < 0.05.
Subgroup Analysis
In the subgroup of age ≥ 65 y, the AUCs of SEWS, NEWS, NEWS2, HEWS, and MEWS were 0.808, 0.829, 0.829, 0.809, and 0.650, respectively. The optimal cutoff values of SEWS, NEWS, NEWS2, HEWS, and MEWS were 7, 8, 8, 6, and 4, respectively (Table 2; Figure 3A). Paired comparisons showed a significant difference between the MEWS score and all other scores. There was no statistical difference between other paired values (Table 3).
Classification performances of five early warning scores (EWS) for subgroups.
(A) The ROC curves of five EWSs in the subgroup of persons aged ≥65 years; (B) The ROC curves of five EWSs in the subgroup of persons aged <65 years. NEWS: National Early Warning Score; NEWS2: National Early Warning Score2; HEWS: Hamilton Early Warning Score; MEWS: Modified Early Warning Score. SEWS: Standardized Early Warning Score.
In the subgroup of age <65 y, the AUCs of SEWS, NEWS, NEWS2, HEWS, and MEWS were 0.893, 0.803, 0.803, 0.820, and 0.704. The optimal cutoff values of SEWS, NEWS, NEWS2, HEWS, and MEWS were 7, 10, 10, 8, and 5, respectively (Table 2; Figure 3B).
Pairwise comparisons showed significant differences among 2 pairs, including SEWS versus MEWS, NEWS versus MEWS, and NEWS2 versus MEWS. There was no statistical difference between other paired values (Table 3).
Correlation Between Each EWSs and SOFA
As 45 cases lacked the information needed to calculate the worst SOFA in the first 24 h, we exclude these cases and analyzed 274 cases for the correlation between each EWS and SOFA. The median and quartiles of SOFA score were 4[4, 6]. The Correlation coefficients of SOFA versus SEWS, NEWS, NEWS2, HEWS and MEWS were 0.294, 0.330, 0.330, 0.278, and 0.221, respectively (P < 0.05, see Supplemental Table 4).
DISCUSSION
COVID-19, a novel infectious disease caused by SARS-CoV-2, led to more than 9 million confirmed cases worldwide from outbreak to June 2020. Although most cases suffered mild symptoms, some progressed to viral pneumonia and multi-organ dysfunction. Rapid and accurate identification of severely ill patients would promote the allocation of appropriate medical resources.
The application of scoring systems can facilitate effective evaluation by emergency or critical care physicians to screen severe patients. At present, however, there are no specific scoring systems for the evaluation of COVID-19 patients. Another option is to adopt existing scoring systems that are used for predicting the mortality of severe patients.Reference Hu, Yao and Qiu26 EWSs, which are generally used in the patients with sepsis, trauma, and other critical ills, are such scoring systems and are widely used in hospitals to identify patients who are clinically deteriorating.Reference Mitsunaga, Hasegawa and Uzura17,Reference Smith, Chiovaro and O’Neil20,Reference Gerry, Bonnici and Birks27,Reference Corfield, Lees and Zealley28
On the whole, the performance of each EWS was acceptable for screening the COVID-19 patients. However, there are differences in the performance among 5 EWSs.
Overall cases and in both age subgroups the AUC of SEWS was acceptable, demonstrating moderate discriminatory power and potential utility as a predictor of mortality in severe COVID-19 patients. An NPV of 98.57% for SEWS would enable emergency physicians to decisively exclude COVID-19 patients with a SEWS score below 7 from the high-risk group. For COVID-19 patients, the advantage of SEWS is the inclusion of more appropriate parameters, including those that differ between survivors and nonsurvivors in univariate analysis, such as heart rate, respiratory rate, oxygen saturation, and mental status.
The AUCs of NEWS and NEWS2 were 0.809 in overall analysis and a higher NPV of 99.28% suggest that these scores may be better screening tools for emergency physicians. In both age subgroups, the predictive performance of NEWS and NEWS2 was acceptable. Some therapeutic studies of COVID-19 patientsReference Gautret, Lagier and Parola21,Reference Cao, Wang and Wen22 have used NEWS or NEWS2 as baseline comparison tools. Although these studies did not evaluate the predictive value of NEWS and NEWS2, the researchers believed that these EWSs could be used as tools for stratification of critical illness.
HEWS performed moderately among the 5 EWSs overall and the subgroup analysis. Its AUC suggest that HEWS is similar to SEWS, NEWS, and NEWS2 for screening COVID-19 patients.
As it does not take into account oxygen saturation (which is very important for COVID-19 patients), it was not surprising that MEWS performed worst in predicting mortality. Therefore, we do not recommend it as a risk stratification tool for severe COVID-19 patients.
We also found that each EWS had only weak positive correlation with SOFA (P < 0.05; correlation coefficient 0.221-0.330). In our study, the SOFA includes laboratory testing parameters, which cannot be calculated at the same time as EWS being calculated while the patients just arrived at the hospital. Therefore, the difference of assessing time may be the reason why these 2 variables are only weakly correlated. This thus makes us realize that the advantage of EWSs is likely to lie in the rapid triage of COVID-19. Further study could explore the relationship between EWSs and SOFA at the same scoring time.
We believe this study was the first to explore the practical value of 5 existing EWSs for rapid screening of severe COVID-19 patients. However, its limitations must be acknowledged. First, we used in-hospital mortality as the main outcome, which means all the living patients at time of discharge (discharge criteria see Supplemental Table 2) would be considered a survivor. Although the sudden death after discharge would not be predicted, as no report on sudden death of COVID-19 patients after discharge was found, we inferred that sudden death of COVID-19 patients after discharge is rare. Thus, we considered in-hospital mortality can be used for evaluating the performance of EWSs of identifying severely ill patients to promote the allocation of appropriate medical resources in hospital. In addition, we also used the worst SOFA as the secondary outcome. Nevertheless, further prospective studies of survival analysis could be done for COVID-19 patients.
Second, whether an oxygen supply was given during SpO2 measurement may be a confounder. In this study, all SpO2 data are from the first record of the patient when they just arrive at the hospital. Although the SpO2 measurement at this time is usually on room air or after a short time of oxygen supplying, we cannot confirm that the SpO2 values of all patients are measured without oxygen supply due to a retrospective study design. Thus, the prospective study with well-designed should be done to control the condition during SpO2 measurement. Third, due to the lack of COVID-19 patients with hypercapnia, our study was unable to distinguish the performance of NEWS and NEWS2. Because the difference between them is that the oxygen saturation scale of NEWS is uniform for all patients, while that of NEWS2 distinguishes patients with hypercapnic respiratory insufficiency. Fourth, we neglected prehospital care because this information was not available. Last, as the study was confined to a single center, it may have been affected by selection bias.
CONCLUSIONS
The SEWS, NEWS, NEWS2, and HEWS have the potential to be used as tools for screening severely and critically ill COVID-19 patients. Inclusion of these tools in decision strategies could provide a more effective evaluation of mortality rate, thus avoiding delayed medical attention. And MEWS is unsuitable for COVID-19 patients because its performance is inferior to that of other 4 EWSs.
Author Contributions
Both Hai Hu and Ni Yao contributed equally to this work and should be considered co-first authors
Acknowledgments
We thank all the physicians and nurses who worked in the COVID-19 ward for helping us collect the data. We also thank Editage (www.editage.cn) for English language editing.
Funding Statement
This work was financially supported by the Novel Coronavirus Research Fund of West China Hospital, Sichuan University (HX2019nCoV063).
Conflict of Interest
The authors declare there are no conflicts of interest.
Supplementary Material
To view supplementary material for this article, please visit https://doi.org/10.1017/dmp.2020.324
Open access
