The pandemic of coronavirus disease 2019 (COVID-19), caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), has affected people worldwide. Reference Machhi, Herskovitz and Senan1 COVID-19 poses a public health threat that burdens the healthcare system, particularly hospitals. To best respond, diagnostic testing has been prioritized. 2 The ability of this virus to spread from individuals who are asymptomatic is especially problematic; thus, accurately diagnosing asymptomatic patients is of utmost importance. Reference Nikolai, Meyer, Kremsner and Velavan3 For this reason, admission testing of all hospitalized patients has become the standard at many US hospitals.
The current standard assay for SARS-CoV-2 is a reverse transcriptase polymerase chain reaction (RT-PCR) assay performed on a nasopharyngeal swab. Reference Ravi, Cortade, Ng and Wang4 These assays have several limitations including need for resources, trained laboratory personnel, and turnaround times that potentially exceed 24 hours. Reference Reusken, Broberg and Haagmans5–Reference Esbin, Whitney, Chong, Maurer, Darzacq and Tjian7 Recent studies have demonstrated that fast turnaround times are critical for effective control of COVID-19. Reference Larremore, Wilder and Lester8 For this reason, the World Health Organization (WHO) emphasized rapid, point-of-care (POC) diagnostics as a top priority to contain COVID-19. 9 Long turnaround times of RT-PCR tests have negatively impacted the emergency department (ED) to inpatient admission process at hospitals because the RT-PCR result is needed prior to inpatient placement. To overcome this problem, the Sofia rapid antigen fluorescent immunoassay (FIA) was introduced at our hospitals with the goal of rapid testing to guide initial bed placement of patients being admitted through the ED.
The Sofia SARS rapid antigen FIA has a manufacturer-published assay turnaround time of 15 minutes, and an emergency use authorization for SARS-CoV-2 diagnostic testing has been granted by the US Food and Drug Administration. 10 Previous studies evaluating rapid antigen assays have suggested that these tests have low sensitivity at lower viral loads and among asymptomatic persons, Reference Buchan, Hoff and Gmehlin11 and published data on their utility among hospitalized patients are limited.
We conducted a real-world evaluation of rapid antigen testing among patients admitted through the ED through assessment of (1) the sensitivity, specificity, positive and negative predictive values of the antigen FIA compared to RT-PCR, (2) comparisons of turnaround times of antigen and RT-PCR testing, and (3) clinical correlates of antigen-negative/RT-PCR positive patients.
Methods
Setting and patients
This study was designed as a clinical quality improvement project at University of Maryland Upper Chesapeake Health, which includes 2 hospitals, the Upper Chesapeake Medical Center (220 beds with 29 double occupancy and an average of 32 daily admissions), and Harford Memorial hospital (104 beds with 49 double occupancy and an average of 12 daily admissions). From September 21, 2020, to December 3, 2020, all patients evaluated in the ED and considered candidates for hospital admission were told the purpose of the study and were asked to provide consent to undergo both Sofia SARS rapid antigen FIA and SARS-CoV2 RT-PCR assays. The study included both symptomatic (persons under investigation or PUIs) and asymptomatic patients.
Clinical practice and infection prevention measures during the study period
All hospitalized patients were required to undergo admission SARS-CoV-2 RT-PCR testing at the 2 hospitals. The test order required clinicians to indicate whether the patient was asymptomatic or a COVID-19 PUI. RT-PCR tests were performed for all patients. When caring for asymptomatic patients, staff were required to wear a medical-grade face mask and eye protection. For PUIs and laboratory-confirmed COVID-19 patients, staff were required to don a respirator, eye protection, gloves, and gowns.
Sample collection and testing
For each patient, trained staff collected 2 nasopharyngeal specimens. One specimen was placed in viral transport media for RT-PCR testing. The other specimen was placed dry in a tube for antigen testing. Results for both assays were reported in patient charts in the electronic medical record system and reported to the Maryland Department of Health.
SARS-CoV-2 RT-PCR assay
Three real-time RT-PCR-based methods were utilized for SARS-CoV-2 detection. The Abbott RealTime SARS-CoV-2 assay (Abbott, Des Plaines, IL) and the cobas SARS-CoV-2 (Roche Diagnostics, Indianapolis, IN) assay were performed at a centralized laboratory. The Xpress Xpert (Cepheid, Sunnyvale, CA) assay was utilized for patients presenting for emergency surgery, women presenting for delivery, prior to initial admission to behavioral health units, and for situations requiring rapid medical decision making based on the institutional testing protocol that guides the use of RT-PCR tests. The Roche cobas assay detects the ORF1 and a region of the E gene are specific to SARS-CoV-2. The Abbott RealTime assay detects RdRp and N genes. For Xpress Xpert, RT-PCR detects the pan-sarbecovirus E gene and the N2 region of N gene. Reference Burbelo, Riedo and Morishima13 The cycle threshold (Ct) values eere obtained for positive RT-PCRs tests as a surrogate measure for viral load for samples tested using the cobas and Xpress Xpert assays. Presumptive results were repeated by another method, and only the final results of “detected” or “not detected” were used in this study.
Sofia SARS rapid antigen FIA
This assay, which uses sandwich immunofluorescence-base lateral flow to qualitatively detect the SARS-CoV-2 nucleocapsid protein antigen, was performed according to the manufacturer’s guidelines. Reference Burbelo, Riedo and Morishima13 Sofia analyzers were used for detection. 10
Patient placement
Patient room-placement guidelines were based on presenting symptoms and test results using a 2-step algorithm with the antigen assay available first and the RT-PCR over the following 24–48 hours. Briefly, antigen-negative, asymptomatic patients could be placed in any room with standard precautions and potentially in double occupancy rooms with other patients. Antigen-positive patients and symptomatic patients (regardless of antigen test results) were placed in a private room. Upon receipt of RT-PCR results, patients who were RT-PCR and antigen positive remained in private rooms, and antigen-negative patients who were found to be RT-PCR positive were moved to private rooms (Supplementary Table 1 online).
Turnaround times
Turnaround times were calculated as time (hours) between NP swab collection and result reporting in the electronic medical record.
Clinical evaluation
All antigen-negative, RT-PCR–positive patients were evaluated for the presence of COVID-19 symptoms, potential COVID-19 exposure in preceding 2 weeks, and potential SARS-CoV-2 transmission to patients and staff based on placement in a double occupancy room and compliance with institutional infection prevention protocols.
Outcome measures and statistical analyses
Daily positivity rates, 7-day moving averages, and trends in positivity based on RT-PCR results were plotted. Overall agreement, sensitivity, and along with 95% confidence intervals (CIs) were calculated for antigen testing with RT-PCR as the reference standard. Patients were stratified into symptomatic and asymptomatic and the aforementioned performance measures were calculated separately for both groups. Positive predictive value (PPV) and negative predictive value (NPV) of antigen testing were calculated for overall prevalence, symptomatic prevalence, and asymptomatic prevalence of COVID-19 based on RT-PCR, as well as for theoretical scenarios of 10% and 20% disease prevalence to evaluate WHO standards. The χ2 test was conducted to statistically compare the performance measures among all patients, symptomatic patients, and asymptomatic patients.
The sensitivity of the antigen assay relative to RT-PCR was also calculated in different categories based on the following ranges of Ct values: <14, 14–17.9, 18–21.9, 22–25.9, 26–29.9, 30–33.9, and ≥34. Results from Abbott RealTime assay were excluded since the assay reports copy number instead of Ct values. Mean RT-PCR Ct values between true positives and false negatives and between symptomatic and asymptomatic patients were compared using the Student t test. Mean turnaround times for antigen and RT-PCR were compared using the Student t test. Statistical analyses and graphs were done using GraphPad Prism version 8 software (GraphPad, San Diego, CA).
This study was deemed non–human-subjects research by the University of Maryland Institutional Review Board.
Results
In total, 2,887 patients were enrolled in this study and received simultaneous SARS-CoV-2 RT-PCR and Sofia antigen testing; 235 patients were positive by RT-PCR, for an overall positive prevalence of 8.1%. For RT-PCR, 1,838 patients received the Roche cobas assay, 675 patients received the Xpress Xpert assay, and 374 received the Abbott RealTime assay. Of 1,675 patients presenting with COVID-19 symptoms, 193 (11.5%) were positive. Among 1,206 asymptomatic patients, 42 were positive (3.5%).
Positivity rates and trends
The 7-day moving average ranged from 2.5% to 20.9% during the study period, peaking between November 8 and November 24, and remained above 10% until the end of the study (Supplementary Fig. 1 online).
Sofia SARS rapid antigen FIA performance
Overall agreement, sensitivity, and specificity between antigen and RT-PCR for all participants were 97.9%, 76.6% (95% CI, 71%–82%), and 99.7% (95% CI, 99%–100%), respectively (Table 1). Among 1,675 symptomatic patients, the overall agreement was 97.1%, sensitivity was 76.2% (95% CI, 70%–82%), and specificity was 99.9% (95% CI, 99%–100%). Among 1,206 asymptomatic patients, the overall agreement was 98.9%, sensitivity was 78.6% (95% CI, 67%–91%), and specificity was 99.7% (95% CI, 99%–100%) (Table 1). Symptomatic and asymptomatic groups were not statistically different.
Comparison of Paired SARS-CoV-2 Rapid Antigen Assay and RT-PCR Assay Results Among Emergency Department Patients Presenting for Hospital Admission (N=2,887)
Note. RT-PCR, reverse-transcriptase polymerase chain reaction; PCR, polymerase chain reaction; +, positive; –, negative; IQR, interquartile range.
With an overall prevalence of 8.1% for this study, the PPV was 96.3% (95% CI, 92%–98%) and the NPV was 98.0% (95% CI, 97%–98%). Among symptomatic individuals, the prevalence was 11.5%, the PPV was 98.7% (95% CI, 97%–100, and the NPV was 97.0% (95% CI, 96%–98%). The prevalence among asymptomatic patients was 3.5%, the PPV was 89.2% (95% CI, 79%–99%), and the NPV was 99.2% (95% CI, 99%–100%). For WHO scenarios of 10% prevalence, the PPV was 96.5% and the NPV was 97.4 %. For 20% prevalence, the PPV was 98.4% and the NPV was 94.5% (Table 2).
Positive Predictive Value (PPV) and Negative Predictive Values (NPV) of SARS-CoV-2 Sofia SARS Rapid Antigen Assay for Prevalences of COVID-19
a Scenarios include overall, symptomatic, and asymptomatic, prevalence in our sample, and 2 theoretical values that are used to determine quality of SARS-CoV-2 test by the World Health Organization.
Cycle thresholds were available for 166 RT-PCR–positive patients in this study (Fig. 1A): 146 from the Roche cobas assay and 20 from Xpress Xpert. The mean Ct value was 22.7 (95% CI, 21.8–23.6) for true positives and 32.1 (95% CI, 31.0–33.2) for false negatives (P < .001) on rapid antigen testing. At Ct values <17.9, the sensitivity was 100%. Between Ct values of 18 and 21.9, the sensitivity was 97.2%. For Ct values of 22–25.9, the sensitivity was 96.8%. For CT values of 26–29.9, the sensitivity was 71.0%. For Ct values of 30–33.9, the sensitivity was 29.2%. For Ct values ≥34, the sensitivity was 22.2% (Fig. 1B). We detected no difference in the mean Ct values between symptomatic patients (25.0; 95% CI, 24.1–25.9) and asymptomatic patients (25.4; 95% CI, 23.2–27.5; P = .78) (Fig. 2).
(A) Box plot comparing cycle thresholds of RT-PCR assays between true-positive and false-negative results for the Sofia Rapid antigen assay. True-positive antigen results had a significantly lower cycle threshold on corresponding RT-PCR assays than false-negative results. (B) Sensitivity of Sofia rapid antigen assay compared to RT-PCR assay based on cycle threshold. Sensitivity decreases as cycle threshold increases. Thus, at lower viral loads, the likelihood of false-negative antigen test results is higher.
Box plot comparing cycle thresholds of RT-PCR assays between symptomatic and asymptomatic individuals. No statistical differences were detected between the 2 groups (P = .78).
Turnaround time
The turnaround time for the Sofia SARS rapid antigen FIA was significantly shorter at 1.2 hours (95% CI, 1.0–1.3) compared to 20.1 hours (95% CI, 18.9–40.3) for RT-PCR (P < .001).
Clinical evaluation
In total, 55 patients with positive SARS-CoV-2 PCR assays were falsely negative on Sofia SARS rapid antigen FIA; of these patients, 38 (69%) were symptomatic. Symptomatic patients were placed in private room and treated as PUIs according to hospital policy, whereas 9 asymptomatic patients were placed in rooms with a roommate. Roommates of 3 patients still in house were tested for SARS-CoV-2 at 5–7 days from exposure; all were negative. No high-risk staff exposures were identified due to use of universal masking and eye protection. None of the staff or patient contacts developed symptoms. The median duration from symptom onset to testing for the 38 symptomatic false-negative results was 7 days (IQR, 2–13) and 17 (44%) received testing ≥8 days from symptom onset. Of 55 false-negative results, 9 (16%) had had prior a positive RT-PCR in the preceding 4-week period. Excluding those known prior positive test results, 20 (43%) patients had possible SARS-CoV-2 exposure risk.
Discussion
In this real-word evaluation among ED patients being considered for hospital admission, the Sofia rapid antigen FIA assay had a sensitivity of 76.6% (95% CI, 71%–82%) and specificity of 99.7% (95% CI, 99%–100%) compared to RT-PCR, with no significant differences between symptomatic and asymptomatic patients. Importantly, the predictive value estimates varied by prevalence, and the percent agreement increased with decreasing RT-PCR cycle thresholds, with 100% sensitivity with Ct values < 22. Average turnaround times were significantly lower for antigen versus RT-PCR testing. With limited testing of contacts, transmission to other patients or staff was not observed from antigen-negative/RT-PCR–positive patients.
Several studies have evaluated and compared rapid POC diagnostics for SARS-CoV-2. These assays can be administered via saliva, nasopharyngeal swab, or nasal swab, and they are antigen or molecular based. Reference SoRelle, Mahimainathan and McCormick-Baw20 Some current examples of rapid POC assays include BD Veritor, COVID-19 Ag Respi-Strip, Lumipulse, and the Abbott ID NOW. Reference Sheridan21–Reference Young, Taylor and Cammarata26 These rapid assays have reported low sensitivities, especially at high Ct. Reference Scohy, Anantharajah, Bodeus, Kabamba-Mukadi, Verroken and Rodriguez-Villalobos27–Reference Kucirka, Lauer, Laeyendecker, Boon and Lessler29 The Sofia SARS rapid antigen FIA has previously been compared to other POC diagnostics in symptomatic individuals, and it is either competitive with or outperforms other POC assays. Reference Sheridan21–Reference Scohy, Anantharajah, Bodeus, Kabamba-Mukadi, Verroken and Rodriguez-Villalobos27
For a rapid SARS-CoV-2 diagnostic test to be considered acceptable by the World Health Organization (WHO), sensitivity and specificity must be 80% and 97%, respectively. 14 The desirable thresholds are even higher at 90% and 99%, respectively. Although the overall sensitivity for the Sofia rapid antigen FIA assay does not meet WHO guidelines, it is only slightly lower than the acceptable level, with the 95% confidence interval crossing the threshold. For predictive values, the WHO requires a second or confirmatory assay for PPV <50%. By these standards, a second assay would not be required for the Sofia SARS rapid antigen FIA. Furthermore, for prevalence between 10% and 20%, the WHO-recommended acceptable ranges are >78%–89% for PPV and 95%–98% for NPV. At 10% prevalence, the PPV based on our study is 96.5% and the NPV is 97.4%; both meet acceptable criteria. At 20%, the NPV drops to 94.5%, slightly below guidelines. Based on recommended PPV and NPV ranges by the WHO, the ideal prevalence range for use of the Sofia SARS rapid antigen FIA assay is between 10% and 18.5%, and caution should be exercised when using this assay’s PPV at lower and NPV at higher prevalence of disease.
A significant proportion of transmission of SARS-CoV-2 occurs from asymptomatic and presymptomatic infected individuals. Reference Nikolai, Meyer, Kremsner and Velavan3 It is critical to properly identify these carriers in a timely manner. Previous studies lack thorough analysis of antigen-based assay performance in asymptomatic individuals. In this study, the Sofia SARS rapid antigen FIA was evaluated in both symptomatic and asymptomatic individuals, providing information not previously explored. When stratified by symptomatic and asymptomatic, overall agreement, sensitivity, and specificity were similar, demonstrating the Sofia SARS rapid antigen FIA is just as accurate in asymptomatic individuals as those who are symptomatic. Importantly, mean Ct values were not significantly different between asymptomatic and symptomatic individuals suggesting viral load and not symptoms as the primary determinant of test performance. These results differ from previous publications that have suggested its use only in symptomatic patients, and they reflect the wide spectrum of clinical manifestations of SARS-CoV-2 infection such that patients can be asymptomatic and highly infectious or severely ill and past their infectiousness. Reference Xu, Chen and Yuan16,Reference He, Lau and Wu17,Reference Young, Taylor and Cammarata26
We also observed a decrease in the sensitivity of the Sofia SARS rapid antigen FIA at lower viral loads as estimated by RT-PCR Ct values, and we observed significantly higher Ct in the antigen false-negative group. This finding is consistent with previous studies that have shown lower sensitivity at high Ct values. Reference Buchan, Hoff and Gmehlin11,Reference Lanser, Bellmann-Weiler and Ottl15 Lack of sensitivity at high Ct values makes antigen testing less suitable than RT-PCR for detection of both very early and late SARS-CoV-2 infection when viral load is low. This aspect was noted in the clinical evaluation of discordant (antigen negative, RT-PCR positive) results in our study patients, with most representing either recent exposure or delayed shedding in prior RT-PCR positive patients. Although it is likely that antigen testing detects most patients with transmissible SARS-CoV-2 infection, infected individuals with low viral loads detected on NP swabs may still transmit the virus to others, particularly in the setting of early incubating disease. Reference Xu, Chen and Yuan16–Reference Liu, Yan and Wan19,Reference Mina, Peto, Garcia-Finana, Semple and Buchan30
No evidence of transmission from false-negative patients was found. Low or noninfectious viral shedding, lack of transmission was likely mitigated by infection control policies in place including universal masking, universal use of eye protection among staff providing patient care, and maintenance of symptomatic patients in appropriate precautions while RT-PCR test was still pending.
A significant reduction in turnaround time from sample collection to result was observed using the Sofia SARS rapid antigen FIA compared to RT-PCR with a shorter turnaround time by an average of 18.9 hours for antigen testing. Previous studies have suggested that time from sample to results is even more critical than sensitivity in reducing the spread of SARS-CoV-2. Reference Larremore, Wilder and Lester8 Many healthcare systems depend on assay results to decide patient placement. Delays in results could lead to patient placement in nonprivate rooms, spreading the virus, or unnecessarily isolating the patient, wasting limited hospital space and resource. During the study period, inpatient room placement was successful in 1,160 of 1,169 asymptomatic antigen-negative patients in this study, and only 9 false negative results required a re-evaluation of placement, reflecting the benefit of the high NPV of antigen testing in this setting. Collectively, this finding suggests that when used together with clinical symptoms, exposure history, infection control practices, and confirmatory RT-PCR testing, rapid antigen tests can be useful in guiding initial patient placement. Although not formally assessed, anecdotally, ED staff and inpatient providers expressed significant satisfaction in being able to make quicker decisions based on the significantly improved turnaround time of antigen testing relative to RT-PCR in a hospital with a relatively high proportion of semiprivate rooms.
Our study has several limitations. RT-PCR can provide inaccurate results and might not be the perfect comparison. However, RT-PCR remains the current recognized standard for SARS-CoV-2 diagnosis. Designations of symptomatic and asymptomatic were at provider discretion and subject to bias; however, this reflects real-world conditions of use of SARS-CoV-2 diagnostic testing. Furthermore, a higher prevalence or percent positivity in symptomatic versus asymptomatic patients in our sample indicates that designations were accurate within the inherent limitations of recognizing early or nonspecific symptoms of COVID-19 PCR Ct values are limited surrogate measures for viral load. However, these values do not provide an absolute count and are dependent on the assay, sample collection, and collection site. We tested only a small number of contacts of antigen-negative/RT-PCR–positive patients, which limits conclusions about transmission in this study. However, no symptomatic staff or patients were identified through contact tracing. Lastly, this study may not be generalizable to hospitals with different prevalences of SARS-CoV-2. Confirmatory testing of the Sofia SARS rapid antigen FIA may be required for negative tests in symptomatic individuals in high-prevalence populations and for positive asymptomatic patients in low-prevalence populations.
Despite not meeting the requirements to replace the RT-PCR assay for detection of SARS-CoV-2 infection, our findings suggest the usefulness of Sofia SARS rapid antigen FIA to guide initial patient placement in a burdened, limited rapid PCR-capacity hospital setting for both symptomatic and asymptomatic patients.
Acknowledgments
Financial support
No financial support was provided relevant to this article.
Conflicts of interest
All authors report no conflicts of interest relevant to this article.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/ice.2021.281
