Hospital performance-based payment programs have become a critical tool that incentivize hospitals to implement better practices and reduce adverse events including healthcare-associated infections (HAIs). The premier program is the Center for Medicare and Medicaid Services (CMS) Hospital Value-Based Purchasing Program (VBP). In 2017, the CMS added a safety domain metric including hospital-onset (HO) Clostridioides difficile infection (CDI) as reported to the Center for Disease Control and Prevention (CDC) National Healthcare Safety Network (NHSN). 1 CDI has emerged as the most common cause of HAI in US hospitals Reference Magill, Edwards and Bamberg2 ; therefore, targeting this preventable infection is a priority. The VBP uses HO-CDI to rank hospitals linking millions of dollars in healthcare payments to hospital performance based on the NHSN CDI standardized infection ratio (SIR) calculation. 3
The CDI SIR is a summary measure that compares the number of observed HO-CDI laboratory-identified events (HO-CDI events) for each facility with the number of predicted HO-CDI events, based on a predictive model. 4,5 Any overestimation in the number of predicted HO-CDI would increase the denominator and lower the SIR. If these overestimations are biased, the SIR would misrepresent performance. Within a year after CMS penalties were first linked to this NHSN metric, investigators demonstrated that the NHSN adjustment for laboratory test type was insufficient; the NHSN HO-CDI events observed drastically differed depending on the laboratory test used in the facility. Reference Marra, Edmond, Ford, Herwaldt, Algwizani and Diekema6 Additionally, the model did not adjust for independent risk factors for CDI that hospitals cannot readily change (ie, patient age or fixed hospital characteristics), even with exemplary performance. Reference Thompson, Edwards, Dudeck, Fridkin and Magill7 In response, in 2015, the NHSN modified the predicted HO-CDI events model to improve the CDI test type adjustment and account for additional fixed factors. 5
Increasing evidence indicates that current NHSN model adjustments are still inadequate. Reference Figueroa Castro, Palen and Johnson8 Currently, only the final CDI test result placed in the medical record, regardless of testing sequence, is reported to the NHSN. 5 Changing the sequence of testing used for CDI diagnosis would alter the SIR without improving prevention performance. We estimated the reduction in NHSN CDI events and its impact on the SIR performance metric in 13 Atlanta area hospitals implementing testing by toxin enzyme immunoassay (EIA) only after a positive nucleic acid amplification test (NAAT) (ie, reflex EIA testing).
Methods
Data sources
The Georgia Emerging Infections Program (GA EIP, funded by the CDC) conducts active population-based CDI surveillance in the 8-county metropolitan Atlanta area (population 4.16 million, 2019). Reference Magill, Dumyati, Ray and Fridkin9 As part of ongoing surveillance since 2009, all CDI test results are reported regardless of which tests are performed. We reviewed test results from April 2018 to July 2019 at facilities that adopted reflex EIA testing. We also accessed facility data reported to NHSN through data-sharing rights with the Georgia Department of Public Health. Values reported for 2017 were utilized for CO-CDI rate, medical school affiliation, ICU bed size, facility type, bed size, ED/observation status, and CDI patient days. 4
Derivation of estimated SIR
Test positivity values were calculated using EIP data reported during months of reflex testing; frequency of testing results were aggregated by facility and test type to calculate facility-specific test positivity rates (total no. EIA positive divided by total no. NAAT positive) and percent reductions in test positivity (1 – % positive). Then, percent positivity was applied to the NHSN reported HO-CDI and CO-CDI event data for each facility; for this analysis, we assumed facility-specific percent positivity would be the same for both HO-CDI and CO-CDI events. We used these estimated values and the NHSN CDI SIR model 5 to calculate a revised SIR with reflex testing. We compared this revised SIR to the SIR calculated with the same NHSN CDI SIR model 5 (without estimated values from reflex testing) to quantify the percent change in SIR with reflex EIA testing implementation.
Statistical analysis
Hospital-specific percent positivity values were compared by facility characteristic using the Kruskal-Wallis test. To simulate a range of percent reductions in test positivity and corresponding changes in SIR, we applied 1% reductions to each hospital’s CO-CDI rate and HO-CDI events, which allowed us to identify a threshold reduction where the effects of reduced CO-CDI rates counteracted the effects of reduced HO-CDI events in the calculation of the SIR. We plotted the facilities with the highest (a teaching hospital) and lowest (a nonteaching hospital) number of predicted CDI events as the upper and lower limits of this simulation. All statistical tests were performed with SAS version 9.4 software (SAS Institute, Cary, NC).
Results
Overall, 13 acute-care hospitals reported a switch to reflex EIA testing during the study period. These facilities varied greatly in size, reporting a range of 52–633 beds (8–105 critical care beds). The facilities were from 3 healthcare systems; approximately half were teaching affiliated and 3 sent specimens off site for testing (Table 1). Facilities reported using reflex EIA testing for a mean of 7 months (range, 6–9 months), resulting in 550 positive NAAT tests reflexing to 180 positive EIA tests (pooled mean, 58% reduction in test positivity). The overall facility-specific percent reduction varied (mean, 67%; range, 42%–81%). The percent reduction did not differ when comparing hospital size, 61% among larger hospitals (>217 beds) compared to 50% among smaller hospitals (≤217 beds; P > .05), or testing site (65% at off-site testing compared to 54% at on-site testing; P > .05). The pooled mean SIR without reflex testing was 0.35, and the pooled mean SIR with reflex testing was 0.19, a 46% reduction in the estimated SIR. The percent estimated change in SIR for each facility ranged from −12% to −71% (Fig. 1, solid circles). The simulations indicated that the percent reduction with reflex EIA testing would need to be very small for the SIR to remain unchanged, between 26% and 32% (Fig. 1).
Variation in C. difficile Testing Results and Facility Characteristics Relevant to SIR Estimates Among 13 Acute-Care Hospitals Performing Reflex EIA testing and reporting to Georgia Emerging Infections Program (2018–2019) and the CDC NHSN (2017)
Notes: SIR, standardized infection ratio as predicted by NHSN methodology 4,5 ; NHSN, National Health Safety Network; CDC, Centers for Disease Control and Prevention; EIA, enzyme immunoassay; EIP, Emerging Infections Program; CDI, Clostridioides difficile infection; NAAT, nucleic acid amplification test; ICU, intensive care unit.
a To maintain the anonymity of the facility the following variables were rounded or modified: admissions, patient days and beds to a standard range, SIR to the nearest one-tenth, and teaching status indicated any of undergraduate, graduate, or major teaching.
Change in standardized infection ratio for different percent reductions in C. difficile test results reportable to the National Healthcare Safety Network with adoption of reflex enzyme immunoassay testing (EIA) after positive nucleic acid–based testing, simulation models (shaded area) and experience at 13 hospitals (solid circles) in the Atlanta metropolitan area, 2018–2019.
Discussion
Data from 13 acute-care hospitals allowed for a robust estimation of the impact of reflex EIA testing on the NHSN derived SIR. Universally, there were SIR reductions at all facilities, and there were substantial overall estimated changes in SIR of ˜50%. These data reaffirm what has been illustrated on a smaller scale, Reference Marra, Edmond, Ford, Herwaldt, Algwizani and Diekema6,Reference Figueroa Castro, Palen and Johnson8,Reference Polage, Gyorke and Kennedy10 and they demonstrate the consistency and magnitude of this impact. In addition, the simulations highlight that reductions in SIRs would be expected with hospitals adopting reflex EIA testing with few exceptions. These exceptions would be at facilities where most (ie, 68%–76%) NAAT-positive patients also test positive by EIA; this occurrence would be rare or nonexistent, and it is well below the level experienced in any of our study hospitals or of those reported elsewhere in the literature. Reference Marra, Edmond, Ford, Herwaldt, Algwizani and Diekema6,Reference Figueroa Castro, Palen and Johnson8,Reference Polage, Gyorke and Kennedy10
Percent reduction in test positivity with EIA reflex testing likely depends on the prior probability that the tested patients have clinical illness with CDI. A better understanding of the drivers of variability in percent reductions may shed light on possible ways to mitigate the effect of different testing strategies on this performance metric. Switching to reflex EIA testing may reflect best clinical practice, minimizing unnecessary treatment for patients with false-positive NAAT tests, but reflex EIA testing is more expensive than NAAT alone. Currently, variation in testing practice is acceptable as a practice standard; a performance metric should not change when changed testing practices are acceptable standards. The key limitations of our findings include a modest geographic representation, so findings may differ in other areas. Also, in applying the facility-specific percent positivity to their NHSN data, we assumed that the same percent positivity would apply to both the facilities CO-CDI rate and HO-CDI events, which may not be true. Notably, we did not compare changes over time (before and after study) which would lack external validity due to other temporal changes which may affect SIR. By applying the percent reduction estimated to the same SIR model parameters, we minimized threats to external validity.
In summary, changing C. difficile testing methodology to EIA reflex testing is expected to substantially improve the current CMS VBP safety domain metric regarding hospital-onset C. difficile prevention as operationalized by the NHSN, without changing any CDI prevention practices. Because this performance metric is so influenced by testing practice, the methodology needs correction to ensure that the SIR is a fair measure of infection prevention performance.
Acknowledgments
We thank Dr Monica Farley, Stephanie Thomas, and Stacy Carswell of the Georgia Emerging Infections Program. We also thank our hospital laboratory, infection prevention, and information technology partners.
Financial support
EIP surveillance was funded through the Centers for Disease Control and Prevention Emerging Infection Program (grant no. U50CK000485).
Conflicts of interest
S.K.F. reports that Emory University has received a services agreement from Pfizer Inc. for public health research on C. difficile starting in 2019 for 1 year. All other authors report no conflicts of interest relevant to this article.
