Introduction
Bloodstream infections (BSI) are a significant clinical and public health challenge, and timely and accurate identification of causative pathogens and their antimicrobial resistance (AMR) profiles are crucial for effective treatment.Reference Coulter, Roberts, Hajkowicz and Halton1,Reference Kontula, Skogberg, Ollgren, Järvinen and Lyytikäinen2 Traditional methods for diagnosing BSI, such as blood cultures and antibiograms, though reliable, are time-intensive and often delay the initiation of effective antimicrobial therapy. Delays in pathogen identification and the start of appropriate antimicrobial treatment result in worse outcomes.Reference Zasowski, Bassetti and Blasi3,Reference Bonine, Berger and Altincatal4
Standard blood cultures generally need to be incubated for at least 12 to 24 hours to observe growth. Subsequently, an additional 1 to 2 days are required to identify the pathogen and its antimicrobial susceptibility. Reduction in the total testing time may result in earlier targeting of antibiotic therapy, potentially resulting in decreased mortality. The introduction of molecular diagnostics, including polymerase chain reaction (PCR) panel tests, has advanced the diagnosis of BSI, facilitating quicker identification of pathogens and resistance markers.Reference Banerjee, Teng and Cunningham5 PCR detection and differentiation of Staphylococcus aureus has been employed for almost a decade, being a well-established practice in many clinical labs. The introduction of PCR-based BioFire Blood Culture Identification 2 Panel (BCID2) represents a recent development in standard clinical laboratory practices. It has contributed to reducing the turnaround time after a positive blood culture, enabling the rapid identification of over 30 pathogens, including resistant strains such as vancomycin-resistant Enterococcus (VRE), by identifying genes associated with AMR. Enterococcal infections, especially those caused by AMR strains like VRE, are known for their high morbidity and mortality.Reference Billington, Phang and Gregson6 The prompt identification of pathogens through molecular tests such as BCID2 can reduce the time required to adjust empiric antibiotic treatments. Thus, expediting identification and decreasing time to adjust antibiotic treatment could potentially decrease mortality among these patients.Reference Banerjee, Teng and Cunningham5,Reference Cortazzo, D’Inzeo and Giordano7 However, the impact of rapid testing on mortality in clinical settings remains unclear.
Previous studies have shown mixed results regarding the impact of rapid molecular testing on reducing mortality resulting from BSI in general and enterococcal BSI in particular. A set of randomized trials did not find conclusive evidence regarding the effect of the implementation of PCR testing on mortality at 30 daysReference Banerjee, Teng and Cunningham5,Reference Beuving, Wolffs and Hansen8 . A systematic reviewReference Timbrook, Morton, McConeghy, Caffrey, Mylonakis and LaPlante9 included 31 different studies evaluating the effect of the introduction of molecular test for grouped BSI on mortality outcomes. Most of these studies were observational pre-post-intervention analysis without control groups, and only 2 randomized studies were included. The review concluded that, overall, the new diagnostic methods reduced mortality when aggregating all the results from these diverse studies. For the specific case of enterococcal infections, observational studies encountered contradictory effects: one early study found a reduction in mortality after molecular test implementationReference Forrest, Roghmann and Toombs10; no effect on mortality was reported in two other studies.Reference Roshdy, Tran and LeCroy11,Reference MacVane, Hurst, Boger and Gnann12 The conflicting results reported might be because the observational studies included in the review implemented a pre-post analysis without a comparison groupReference Timbrook, Morton, McConeghy, Caffrey, Mylonakis and LaPlante9.
This study aimed to evaluate the effectiveness of the BCID2 PCR test for enterococcal BSI in a large hospital system in New York City. We employed a difference-in-differences (DiD) analysis to estimate causal effects of the BCID2 PCR implementation on 30-day mortality risk. The analysis compared changes in mortality risk among patients with enterococcal BSI before and after introduction of the BCID2 PCR testing protocol in 2022, against the change in mortality risk among patients with S. aureus BSI over the same period, a patient group who has not been subject to changes in diagnostic procedures for nearly a decade, and that can provide an stable indicator as a reference of no broader changes in the treatment of BSI. Our hypothesis was that the introduction of the BCID2 PCR test for enterococcal BSI led to improved patient survival. The study was designed to evaluate recent changes to clinical practice, with the goal of identifying effective diagnostic approaches for BSI.
Methods
Study design and setting
We employed a quasi-experimentalReference Shadish, Cook and Campbell13 difference-in-differences (DiD) design to estimate the causal effect of introducing the BCID2 PCR test on 30-day all-cause mortality in patients with vancomycin-resistant Enterococcal (VRE) bloodstream infections. As a comparison group, we used patients with BSI caused by methicillin-sensitive S. aureus (MSSA), since the detection algorithm for this type of bacteria has not changed—a different PCR test has been continuously performed for molecular identification of MSSA since 2014. This study was conducted within the NYU Langone Health system, a large hospital network in New York City, encompassing both tertiary and community hospital settings.
Data collection and study population
This study sourced data from electronic health records (EHR) using SQL queries to extract records of patients with BSI caused by Enterococcus faecalis, Enterococcus faecium, and S. aureus. Patients with BSI were defined as those with a positive blood culture for the pathogens of interest. Records extracted ranged from October 2019 to October 2024. We focused on patients 18 years or older with positive blood cultures for vancomycin-resistant enterococci (VRE), defined as our treatment group, and methicillin-sensitive Staphylococcus aureus (MSSA) as our control group. We selected VRE since they would benefit the most from a rapid adjustment of antibiotic treatment, as many empiric treatments do not cover VRE, thus the effect would be more noticeable. A similar criterion was used to select MSSA: these patients usually benefit from adjusting treatment to a narrow-spectrum antimicrobial (i.e., oxacillin), whereas methicillin-resistant S. aureus (MRSA) is usually treated empirically because empiric treatment of sepsis includes vancomycin given the high frequency of MRSA in both communities and hospitals. The timeframe selected allowed us to compare outcomes before and after the implementation of the BCID2 test on April 7, 2022.
Intervention
The BCID2 PCR panel (BioFire Blood Culture Identification 2) is an FDA-cleared multiplex PCR assay, capable of simultaneous detection and identification of 33 common bacteria and yeasts causing bloodstream infection and 10 antimicrobial resistance genes from positive blood cultures, including enterococci.Reference Aguero-Rosenfeld14 At NYU Langone Health, this panel was integrated into the clinical lab workflow for BSI samples starting on April 7, 2022. Since then, the test was run on the first positive blood culture sample per episode for a defined set of microorganisms, as detailed in Figure 1.
Process to identify bloodstream infection pathogens and antimicrobial susceptibility from blood samples.
Laboratory workflow
Starting April 7, 2022 at NYU Langone Health, once a culture turns positive and a Gram stain is performed, the BCID2 kit is applied to all gram-negative bacteria, all gram-positive cocci in pairs and/or chains (including those suspicious for Listeria sp.), and all yeast and yeast-like organisms. No rapid molecular tests for these pathogens were routinely performed before April 7, 2022. However, for Gram-positive cocci in clusters, including S. aureus, the workflow used since June 2014 includes the Cepheid MRSA SA BC rapid test, with no changes in the detection algorithm for S. aureus cases. (See Figure 1). Positive blood culture results, including Gram stain findings, are called to the clinical floor. The PCR test results are not called directly but are reported in the patient’s electronic medical record. All clinical staff were notified in advance about the introduction of the PCR test. Antimicrobial adjustments are made as needed by the care team upon receiving results. Time to effective antimicrobial was extracted from the microbiological test reports.
Outcome measures
The primary outcome measure was all-cause mortality within 30 days of the positive blood culture, as this is the main clinical outcome measured in RCTs. Mortality occurrences outside of hospital facilities are documented by individual states and reported to the Social Security Administration. The records at NYU Langone are regularly cross-checked against these records, providing a high degree of accuracy.
Statistical analysis
We employed an event study framework using inverse probability weights to adjust for covariates. To estimate dynamic effects of the intervention over time, we first estimated the propensity score via logistic regression with independent variables being age (linear and squared terms), comorbidities (measured as Charlson comorbidity index, linear), and history of antibiotic use (binary). The response variable was the binary treatment group indicator (being enterococcal or S. aureus infection). Then a weighted linear regression model was fit, weighted by the propensity scoreReference Abadie15, regressing the binary mortality variable on time and treatment, with time specified as a restricted cubic spline with 5 degrees of freedom fully interacted with the binary treatment group indicator. Interaction terms were used to calculate estimates of treatment effects at each time post-intervention by computing the difference in differences on any day as follows: we subtracted change in the control group from change in the treatment group (i.e., difference in predicted values for day 0 and for day n for the treated minus the difference in predicted values for day 0 and for day n for the control). 95% confidence intervals were calculated using a bootstrap analysis of 1000 resamples and using the 0.025th and 0.975th quantiles from the distribution of the estimate. We then calculated the cumulative effect of the intervention effect by taking an average of effects at all timepoints post-treatment and calculated 95% CIs using a bootstrap analysis involving percentiles of those cumulative effect estimates.Reference Callaway and Sant’Anna16
Our approach relied on the conditional parallel trends assumption. Specifically, we must assume that had the PCR intervention not been implemented, additive changes in mortality risk over time would have been the same between enterococcal BSI and MSSA, within levels of age, comorbidities and history of antibiotics. Though the assumption is untestable, we provided contextual information regarding its plausibility by examining the extent to which trends in the pre-treatment period deviate from parallelism. Specifically, we derived estimates of these deviations from the weighted outcome model, based on the time × treatment interaction terms from times before the intervention occurred.Reference Cunningham17 In addition to these “event study” estimates, we also visually compared the weighted risk estimates for each arm, derived from the same weighted outcome model described above. Two sensitivity analyses were included to address possible misspecification of the model. One model included time as a categorical variable (instead of as restricted cubic spline). A different model without weights for covariates was also performed. Lastly, two sensitivity analysis were performed using different subset of pathogens: we tested all enterococcal BSI (VRE and vancomycin susceptible strains, VSE) as treatment group and all S. aureus BSI (MRSA+MSSA) as comparison group; another sensitivity analysis was done with only E. faecium in the treated group to exclude the potential problem of different distribution of enterococci in each period. Data analyses were conducted using R version 4.0.2.
Results
A total of 548 inpatients admitted between October 1, 2019 to October 30, 2024 were included in this study; 164 were diagnosed with VRE BSI (treatment group) and another 384 had diagnosed MSSA BSI (Table 1). The use of BCID2 panels began on April 7, 2022, which was used to define the pre and post-intervention periods of the study. Pre-intervention, the average 30-day mortality risk for patients with MSSA BSI was 10.8%, whereas VRE BSI mortality was 40.2%. History of antimicrobial treatment was more common among those with VRE than MSSA, and VRE patients presented with a higher average number of comorbidities. In the post-intervention period, the average 30-day mortality risk was 7.9% for patients with MSSA and 23.9% for VRE patients. Characteristics of the patients pre- and post-intervention for the treatment and control groups are shown in Table 1.
Patient’s characteristics by type of BSI and for pre and post-intervention periods
Notes. BSI: Bloodstream infections; SD: standard deviation; MSSA: methicillin-sensitive S. aureus; VRE: vancomycin-resistant enterococci.
Figure 2 displays event study plots, which present effect estimates in the post-intervention period and deviations from parallel pre-trends in the pre-intervention period. Pre-intervention values were centered near 0 relative to the available precision, suggesting that parallel trends may be plausible. Figure 3 shows weighted mortality risk and 95% confidence intervals for VRE BSI (treated) and MSSA (control) groups separately, derived from linear regression with time modeled as cubic restricted splines.
Event study plot. Event study coefficients along with their 95% confidence intervals. After the intervention (weeks > 0), estimates correspond (under assumptions) to the impact of PCR introduction for diagnosis of vancomycin-resistant enterococci on mortality risk; and before the intervention (weeks < 0), deviations from parallel pre-trends. Error bars depict 95% confidence intervals calculated using the 2.5th and 97.5th percentiles from 1000 bootstrap resamples.
Weighted mortality risk along with their 95% confidence intervals for vancomycin-resistant enterococci bloodstream infection (treated) and methicillin-sensitive S. aureus (control) groups derived from linear regression with time modeled as cubic restricted splines (DF=5). Study 95% confidence intervals represented as error bars calculated using a bootstrap analysis of 1000 resamples.
The estimates around the intervention were as follows: at 180 days before the intervention (-180 days) we estimated a deviation from parallel trends corresponding to a risk difference of 0.053 (95% CI: -0.099 to 0.11), and at -60 days, the estimated deviation was 0.036 (95% CI: -0.038 to 0.64). At 60 days post-intervention, the estimated risk difference was -0.041 (95% CI: -0.071 to 0.039), at 180 days it was -0.098 (95% CI: -0.18 to 0.99) (Table 2). The estimated mortality risk for individuals in the enterococcus group, relative to those in the control group, showed a shift post-intervention. After the intervention, all point estimates were negative, corresponding to a lower mortality risk in the patients with enterococcus relatively to the control group, though wide confidence intervals associated with these point estimates include large negative and positive values. The cumulative effect, calculated as the average of all the post-treatment timepoints, was -0.0603 (95% CI: -0.1035, -0.0171).
Effect estimates for the impact of PCR introduction for diagnosis of VRE on 30-day mortality risk. Estimates are derived from the time × treatment interaction in the weighted event study model, and 95% confidence intervals were calculated using the 2.5th and 97.5th percentiles from 1000 bootstrap resamples
Notes. VRE: vancomycin-resistant enterococci; CI: Confidence interval.
The time to concordant antimicrobial treatment was reduced for patients with a VRE infection. The median time in hours for linezolid decreased from 43.8 pre-intervention to 38.8 post-intervention, and for daptomycin from 57.4 to 32.3. For MSSA, the most common treatment when pathogens are identified is oxacillin, and the time to antimicrobial switch remained very similar (33.9 vs. 33.4 hours).
We conducted a sensitivity analysis using alternative model specifications. First, a model treating time as a categorical variable instead of using restricted cubic splines produced results similar to the main analysis, with negative estimates after the intervention and wide confidence intervals encompassing both negative and positive values. Second, an unweighted regression approach yielded consistent results, also showing negative post-intervention estimates with wide confidence intervals including both negative and positive values.
In the sensitivity analyses including MRSA and MSSA and all enterococci, the treated group (VRE and VSE) showed lower average mortality risk before and after the intervention relative to the VRE-only group, which is consistent with known patterns of higher mortality among patients with VRE BSI compared to VSEReference Prematunge, MacDougall and Johnstone18 (Supp. Table 1). The DiD model resulted in similar trends: a change in the direction of the estimates post-intervention indicated a potential decrease in mortality risk in the patients with enterococcus relatively to the control group, but with wide confidence intervals making the estimates imprecise (Supp. Figure 1 and 2). The sensitivity analysis including only resistant E. faecium showed similar results: a change in trends from positive risk difference values in the pre-intervention period to negative risk differences (reduced mortality for resistant E. faecium) in the post-period (Supp. Table 2 and Supp. Figure 3).
Discussion
The recent hospital-wide implementation of the BCID2 PCR panel testing, a new technology in the standard laboratory process for identifying BSI pathogens, presented a unique opportunity to assess its impact on patient mortality. The benefits of rapid pathogen identification have previously been shown to effectively reduce delays in targeted antibiotic therapy, though they have not produced unequivocal results on clinical outcomes.Reference Banerjee, Teng and Cunningham5,Reference Timbrook, Morton, McConeghy, Caffrey, Mylonakis and LaPlante9 We observed a shift in mortality risk estimates for VRE BSI for the period after the intervention, suggesting a potential reduction in mortality for these patients. The cumulative effect (average effect among all timepoints post-intervention) showed a reduction in mortality among the patients with VRE BSI. We found that deviations from parallel pre-trends were small relative to the available precision, but of a similar magnitude to the effect estimates. Thus, while we do not have strong reason to doubt the parallel trends assumption, actual deviations from parallel counterfactual trends in the post-period to the degree observed in the pre-period would be large enough to cause bias. Overall, our data suggest a decrease in mortality risk, but not conclusive, indicating the need for cautious interpretation of these results.
Previous RCTsReference Banerjee, Teng and Cunningham5,Reference Beuving, Wolffs and Hansen8 , signaled no causal effect of the PCR test on mortality risk of patients with BSI compared to standard of care. Two other observational studies using a design without controls also did not find a definitive answer regarding the effect of PCR testing on mortality among enterococci BSI patients.Reference Roshdy, Tran and LeCroy11,Reference MacVane, Hurst, Boger and Gnann12 MacVane reported a lower point estimate for mortality risk for the period after implementing PCR testing, although stated as non-significant. Roshdy found no difference in mortality, reporting 10% in both groups. Since the previous studies produced imprecise estimates and/or used a weaker design, further research is needed with larger samples and sound methodology to establish a conclusion on the effect of PCR testing on mortality.
The impact of PCR testing on mortality in cases of VRE BSI may be influenced by various factors involved in the diagnostic and treatment processes in our settings. The observed effect is likely due to a reduction in the time to effective antimicrobial treatments, as our data suggest. However, other factors might also contribute, such as increased awareness of stewardship policies. For example, clinicians may have become more attentive to the need to expedite switching to concordant antimicrobials, coinciding with the introduction of the new test into the workflow. PCR testing might have a reduced impact in locations where infectious disease specialists are available and can prescribe appropriate empiric treatment. However, in communities with limited access to specialists, rapid identification of pathogens through PCR testing could play a larger role in guiding treatment selection.Reference Yadav, Shah, Sayed, Horton and Schroeder19 Other potential benefits of the intervention, such as its long-term effect in reducing antimicrobial resistance, are more challenging to quantify. Faster diagnoses might lead to shorter exposure periods to broad-spectrum antibiotics for patients, benefiting not only individual outcomes but also aligning with antibiotic stewardship policies.Reference Barlam, Cosgrove and Abbo20
Our study has limitations. Because VRE BSI events are rare, our sample size was limited, leading to relatively imprecise estimates. Expanding our data sources to include different providers could help in achieving results that are more precise. Another limitation is that any changes in the bacterial burden of S. aureus or in the efficacy of controlling the source of infection in our patient population could have affected the control group. However, we did not observe major shifts in the control group during this time period.
Despite these challenges, the DiD method effectively accounts for unmeasured factors that do not change over time, enhancing the credibility of our intervention’s evaluation. Additionally, our study, utilizing EHR data in a real-world healthcare setting, provides a practical example of how to evaluate the introduction of changes in clinical care. By demonstrating the potential benefits of rapid PCR testing in reducing mortality for VRE BSI patients, our findings contribute to the ongoing efforts to improve patient outcomes through timely and accurate antimicrobial treatment strategies.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/ice.2025.15
Financial support
National Institutes of Health grants AI137336 and AI140754 (B.S. and V.J.T.); grants CDC U01CK000590 (B.S.); U48 DP006396-01 (L.E.T.); AI149350 (V.J.T.); and funds from the NYU Langone Health AMR Program (B.S., V.J.T.).
Competing interests
B.S. consulted for Regeneron and MicroGenDx. V.J.T. is an inventor on patents and patent applications filed by New York University, under commercial license to Janssen Biotech Inc. Janssen Biotech Inc. provides research funding and other payments associated with a licensing agreement. All other authors certify no potential conflicts.
