Hostname: page-component-7b7567c8c-kwt9d Total loading time: 0 Render date: 2026-11-14T09:45:22.507Z Has data issue: false hasContentIssue false
  • English
  • Français

Increasing the Reliability of Fully Automated Surveillance for Central Line–Associated Bloodstream Infections

Published online by Cambridge University Press:  02 September 2015

Rachael E. Snyders,
Ashleigh J. Goris,
Kathleen A. Gase,
Carole L. Leone,
Joshua A. Doherty and
Keith F. Woeltje
Rachael E. Snyders*
Affiliation:
Center for Clinical Excellence, BJC HealthCare, St. Louis, Missouri
Ashleigh J. Goris
Affiliation:
Infection Prevention and Control, Missouri Baptist Medical Center, St. Louis, Missouri
Kathleen A. Gase
Affiliation:
Center for Clinical Excellence, BJC HealthCare, St. Louis, Missouri
Carole L. Leone
Affiliation:
Center for Clinical Excellence, BJC HealthCare, St. Louis, Missouri
Joshua A. Doherty
Affiliation:
Center for Clinical Excellence, BJC HealthCare, St. Louis, Missouri
Keith F. Woeltje
Affiliation:
Center for Clinical Excellence, BJC HealthCare, St. Louis, MissouriDivision of Infectious Diseases, Department of Internal Medicine, Washington University School of Medicine, St. Louis, Missouri
*
Address correspondence to Rachael E. Snyders, MPH, BSN, RN, CIC, 8300 Eager Rd, Ste 400-A, St. Louis, MO 63144 (res8897@bjc.org).
Rights & Permissions [Opens in a new window]

Abstract

OBJECTIVE

To increase reliability of the algorithm used in our fully automated electronic surveillance system by adding rules to better identify bloodstream infections secondary to other hospital-acquired infections.

METHODS

Intensive care unit (ICU) patients with positive blood cultures were reviewed. Central line–associated bloodstream infection (CLABSI) determinations were based on 2 sources: routine surveillance by infection preventionists, and fully automated surveillance. Discrepancies between the 2 sources were evaluated to determine root causes. Secondary infection sites were identified in most discrepant cases. New rules to identify secondary sites were added to the algorithm and applied to this ICU population and a non-ICU population. Sensitivity, specificity, predictive values, and kappa were calculated for the new models.

RESULTS

Of 643 positive ICU blood cultures reviewed, 68 (10.6%) were identified as central line–associated bloodstream infections by fully automated electronic surveillance, whereas 38 (5.9%) were confirmed by routine surveillance. New rules were tested to identify organisms as central line–associated bloodstream infections if they did not meet one, or a combination of, the following: (I) matching organisms (by genus and species) cultured from any other site; (II) any organisms cultured from sterile site; (III) any organisms cultured from skin/wound; (IV) any organisms cultured from respiratory tract. The best-fit model included new rules I and II when applied to positive blood cultures in an ICU population. However, they didn’t improve performance of the algorithm when applied to positive blood cultures in a non-ICU population.

CONCLUSION

Electronic surveillance system algorithms may need adjustment for specific populations.

Infect. Control Hosp. Epidemiol. 2015;36(12):1396–1400

Information

Type
Original Articles
Information
Infection Control & Hospital Epidemiology , Volume 36 , Issue 12 , December 2015, pp. 1396 - 1400
Copyright
© 2015 by The Society for Healthcare Epidemiology of America. All rights reserved 

Surveillance for hospital-acquired infections has long been established as an essential component in successful infection prevention programs.Reference Haley, Culver and White 1 Standardized surveillance, using definitions provided by the Centers for Disease Control and Prevention National Healthcare Safety Network (NHSN), has aided the steady reduction of central line–associated bloodstream infection (CLABSI) rates in intensive care units (ICUs).Reference Jarvis 2 However, manual application of the NHSN CLABSI definitions to positive blood culture results is often labor-intensive and time-consuming for infection preventionists (IPs). For this reason, manual CLABSI surveillance is often limited to ICUs.

CLABSI rates are similar in ICU and non-ICU patient populationsReference Marschall, Leone, Jones, Nihill, Fraser and Warren 3 – Reference Son, Daniels and Eagan 7 ; moreover, the number of patients in non-ICU locations greatly surpasses those in ICU locations. Thus, in many hospitals, there may actually be more patients with CLABSIs outside the ICUs than in the ICUs.Reference Marschall, Leone, Jones, Nihill, Fraser and Warren 3 , Reference Climo, Diekema and Warren 8 The widespread availability of electronic patient and laboratory data offers the potential for electronic surveillance of CLABSIs outside of the ICUs.Reference Woeltje, McMullen, Butler, Goris and Doherty 9 A semiautomated electronic surveillance system has been used in BJC HealthCare to facilitate identification of positive blood culture results and potential ICU CLABSIs for 7 yearsReference Woeltje, Butler and Goris 10 and is now part of our routine ICU surveillance. A fully automated electronic surveillance system has been used to track non-ICU CLABSIs for 5 years.Reference Woeltje, McMullen, Butler, Goris and Doherty 9 The algorithm used in this surveillance system was designed to be as consistent with the NHSN CLABSI definition as possible given that there is no formal guidance from NHSN on how best to conduct fully electronic surveillance for CLABSIs. Feedback from IPs is that if the fully automated electronic surveillance system were better able to identify and eliminate secondary bloodstream infections (BSIs), this would increase specificity. The objective of this study was to increase the reliability of the algorithm used in our fully automated electronic surveillance system by incorporating new rules to better identify BSIs that are secondary to other healthcare-associated infections, as defined by NHSN.

METHODS

BJC HealthCare is a large, nonprofit healthcare organization comprising 11 acute care facilities and multiple community health locations, serving urban, suburban, and rural communities in Missouri and Illinois. Patients in 17 ICUs with positive culture results of blood drawn from January 1 through June 30, 2011, were identified from the BJC HealthCare informatics database. Medical, surgical, cardiothoracic surgical, and neurosurgical ICUs were represented from 8 of the 11 hospitals; pediatric and neonatal ICUs were excluded. These patients were chosen for the study because IPs conducted prospective CLABSI surveillance in this population using the NHSN definitions. This afforded us the opportunity to compare CLABSIs identified from routine surveillance with those identified from the fully automated electronic surveillance system.

Positive blood culture results are considered CLABSIs by the existing fully automated electronic surveillance algorithm if they meet the following rules:

  1. 1. Blood cultures must be collected more than 48 hours after hospital admission.

  2. 2. Central line must be in place at the time of culture or discontinued within 48 hours prior to the culture.

  3. 3. Blood cultures with organism(s) that are common skin contaminants, such as coagulase-negative staphylococcus, must have a second culture with the same organism(s) within 3 calendar days, and have clinical documentation of a fever (temperature, >38.0°C) within 48 hours of the first blood culture collection date.

  4. 4. Matching organisms from different blood cultures are considered the same CLABSI episode if they were isolated within 14 days of one another.

  5. 5. Two or more different organisms from different blood cultures are considered the same CLABSI episode if they were isolated within 7 days of one another.

  6. 6. Matching organisms seen in both a nonblood specimen and blood culture, where the nonblood specimen was collected within a window beginning 21 days prior and ending 7 days after the blood culture, are used to exclude the blood culture from being a CLABSI episode; these are considered secondary BSIs.

Positive blood culture results were classified as discrepant if they were confirmed as a CLABSI by either routine surveillance or fully automated electronic surveillance, but not both. All discrepant cases were evaluated by 3 IPs credentialed with Certification in Infection Prevention and Control to determine root causes of the discrepancies. The root causes were then used to brainstorm proposed new rules to include in the algorithm. Owing to the fact that the algorithm rules are applied to each organism in a blood culture separately, all analysis was performed at the level of organism, rather than at the patient level, accounting for patients with polymicrobial bacteremias. Sensitivity, specificity, positive predictive value, negative predictive value, and kappa scores were used to evaluate the proposed new rules.

Evaluation of the proposed new rules was first performed with our aforementioned original ICU study population. Next, evaluation of the proposed new rules was performed with a non-ICU study population. This population included patients in non-ICUs with positive culture results of blood drawn from January 1 through December 31, 2012, identified from the BJC HealthCare informatics database. Various types of non-ICUs were represented from 9 of the 11 acute care facilities, including medical, surgical, oncology, pediatric, and numerous other specialty units. These blood cultures were previously evaluated for the presence or absence of CLABSI events by routine surveillance by an IP credentialed with a Certification in Infection Prevention and Control.

RESULTS

During the study period, 643 blood cultures from 518 ICU patients were reviewed. Routine surveillance by a hospital-based IP identified 38 organisms (5.9%) as CLABSIs; fully automated electronic surveillance identified 68 organisms (10.6%) as CLABSIs (Table 1). A total of 46 organisms were discordantly called by routine and fully automated electronic surveillance methods, most of which were called CLABSIs by the fully automated electronic surveillance system but not by the IP. Secondary infection sites that did not meet rule 5 of the existing algorithm were identified in most discrepant cases. On the basis of the findings, several new rules to identify secondary sites were investigated for inclusion into the existing algorithm of our fully automated electronic surveillance system.

TABLE 1

Comparison of Routine Surveillance and Fully Automated Electronic Surveillance by Number of Organisms Confirmed as CLABSIs–ICU Only

NOTE. CLABSI, central line–associated bloodstream infection; ICU, intensive care unit.

Each of the new rules below, if met within 14 days before or after the positive blood culture, would classify an organism as a secondary BSI rather than a CLABSI.

  1. I. Matching organisms, by genus and species, cultured from any other site

    1. Ia. Excluding yeast if cultured from the respiratory tract, skin, and/or wound(s)

    2. Ib. Allows matching for yeast at the species level if cultured from any other site (ie, Candida albicans cultured from the blood would “match” any Candida spp. cultured from the urine), while still excluding yeast cultured from the respiratory tract, skin, and/or wound(s)

  2. II. Any organisms cultured from a sterile site

  3. III. Any organisms cultured from skin and/or wound(s)

  4. IV. Any organisms cultured from the respiratory tract

Rules I, Ia, and Ib were each included individually with the rules of the existing algorithm and applied to the ICU study population. They were also tested in various combinations with rules II, III, and IV with the rules of the existing algorithm. The addition of rules I, Ia, and Ib individually improved the performance of the algorithm in the ICU population, as well as many of the aggregate rule sets. The best-fit model included rules Ia and II, decreased the discrepant organisms from 46 to 30 (Table 1), and increased the kappa score from 0.530 to 0.650 (Table 2). Importantly, adding rules Ia and II to the algorithm increased the positive predictive value from 44.1% to 57.4% without adversely affecting the negative predictive value, which was already at 98%.

TABLE 2

Performance of New Rules and Aggregate Rule Sets for CLABSI Prediction–ICU Only

NOTE. CLABSI, central line–associated bloodstream infection; ICU, intensive care unit; NPV, negative predictive value; PPV, positive predictive value.

The best-fit model was then applied to the non-ICU study population of 796 blood cultures from 563 patients. The existing algorithm identified 515 organisms (64.7%) as CLABSIs, while routine surveillance identified 428 organisms (53.8%) as CLABSIs (specificity, 71.4%; sensitivity, 96.0%) (Table 3). Applying the best-fit model resulted in the identification of 489 organisms (61.4%) as CLABSIs; this improved the specificity slightly to 75.7%, but at the cost of lower sensitivity (92.5%), resulting in an overall lower kappa score (0.680 vs 0.689) (Table 4). Application of other combinations of rules did not measurably improve performance of the algorithm (data not shown).

TABLE 3

Comparison of Routine Surveillance and Fully Automated Electronic Surveillance by Number of Organisms Confirmed as CLABSIs–Non-ICU Only

NOTE. CLABSI, central line–associated bloodstream infection; ICU, intensive care unit.

TABLE 4

Performance of New Rules Ia + II for CLABSI Prediction–Non-ICU Only

NOTE. CLABSI, central line–associated bloodstream infection; ICU, intensive care unit; NPV, negative predictive value; PPV, positive predictive value.

DISCUSSION

Adding new rules designed to identify secondary BSIs to the algorithm increased the reliability of our fully automated electronic surveillance system when applied to ICU patients, but adversely impacted the negative predictive value and kappa score when applied to non-ICU patients. We initially chose to develop the proposed new rules based on an ICU patient population, rather than a non-ICU patient population, because we had access to detailed surveillance data. This allowed us to classify and examine the discrepancies between the 2 surveillance methods. Given that CLABSI rates are similar in ICU and non-ICU patients,Reference Marschall, Leone, Jones, Nihill, Fraser and Warren 3 – Reference Son, Daniels and Eagan 7 we believed it was a reasonable population as well.

The premise of rule I was to capture other site cultures in a different timeframe than the existing algorithm. Rule I allowed for other site cultures to be considered within 14 days before or after the positive blood culture result as opposed to the existing algorithm; the existing algorithm allowed 7 days after the blood culture, but allowed up to 21 days before the blood culture, which was thought to be too expansive. Rule Ia was designed to ensure that yeast species cultured from the respiratory tract, skin, or wounds did not negate a primary CLABSI with candidemia, given that yeast cultured from these sites often represents colonization rather than infection. The addition of rule II to rule Ia increased the kappa score from 0.590 to 0.650 in the ICU study population, highlighting the importance of considering polymicrobial sterile site infections as the primary site of a bacteremia, as opposed to the central line in place. Accounting for potential skin, wound, soft-tissue, and respiratory infections with nonmatching organisms, in an effort to identify secondary BSIs, did not add reliability or improve the algorithm.

Addition of these rules did not improve performance of the fully automated electronic surveillance system for patients outside of the ICU. The design of the current study did not allow for a detailed investigation of the reasons. It is plausible that non-ICU patients with secondary BSIs have different characteristics in general. It is also plausible that clinicians’ patterns of obtaining samples for culture differ between non-ICU settings and ICU settings. As semiautomated and fully automated electronic surveillance systems are refined, it may be necessary to implement different algorithms depending on the location of patient care in order to optimize the balance of sensitivity and specificity.

There is widespread and increasing interest in fully automated electronic surveillance for healthcare-associated infections; however, there remain several large issues that need to be addressed before this can be accomplished.Reference Dumyati, Concannon and van Wijngaarden 5 , Reference Freeman, Moore, Alvarez, Charlett and Holmes 11 , Reference Woeltje, Lin, Klompas, Wright, Zuccotti and Trick 12 Similar to previous studies, our fully automated electronic surveillance system likely overestimates the number of CLABSIs. A major limitation to using fully automated electronic surveillance for CLABSIs is the inaccessibility to radiographic imaging reports or other diagnostic criteria that can be used to identify secondary site infections during routine surveillance, such as narrative physician or nursing documentation. Improvements to electronic medical records that ensure consistent documentation of these types of criteria may further enhance automated surveillance.

Fully automated electronic surveillance offers great potential for performing housewide CLABSI surveillance without a dramatic increase in resources.Reference Woeltje, McMullen, Butler, Goris and Doherty 9 Since our initial work, NHSN has updated the CLABSI definition, including a transition from hours to calendar days (eg, 48 hours after admission vs 2 calendar days after admission to quantify a potential healthcare-associated infection, among others). We will be updating both our semiautomated and fully automated electronic surveillance systems to incorporate these changes. Although we were not able to improve the performance of the algorithm used in our fully automated electronic surveillance system to identify non-ICU secondary BSIs in this study, the new Centers for Medicare and Medicaid Services mandate for CLABSI surveillance in non-ICU medical and surgical units 13 will provide us with a population of IP-validated CLABSIs that we can use to further refine our algorithms. In addition, more clinical data are becoming available electronically for use in decision support algorithms. This will allow us to expand our efforts to improve our algorithms both for semiautomated and fully automated electronic surveillance systems.

ACKNOWLEDGMENTS

We thank Kathleen M. McMullen, MPH, CIC, for her assistance with the work of the manuscript.

Financial support. None reported.

Potential conflicts of interest. All authors report no conflicts of interest relevant to this article.

References

REFERENCES

1. Haley, RW, Culver, DH, White, JW, et al. The efficacy of infection surveillance and control programs in preventing nosocomial infections in US hospitals. Am J Epidemiol 1985;121:182–205.Google Scholar
2. Jarvis, WR. Benchmarking for prevention: the Centers for Disease Control and Prevention’s National Nosocomial Infections Surveillance (NNIS) system experience. Infection 2003;31:44–48.Google Scholar
3. Marschall, J, Leone, C, Jones, M, Nihill, D, Fraser, VJ, Warren, DK. Catheter-associated bloodstream infections in general medical patients outside the intensive care unit: a surveillance study. Infect Control Hosp Epidemiol 2007;28:905–909.CrossRefGoogle ScholarPubMed
4. Edwards, JR, Peterson, KD, Mu, Y, et al. National Healthcare Safety Network (NHSN) report: data summary for 2006 through 2008, issued December 2009. Am J Infect Control 2009;37:783–805.Google Scholar
5. Dumyati, G, Concannon, C, van Wijngaarden, E, et al. Sustained reduction of central line-associated bloodstream infections outside the intensive care unit with a multimodal intervention focusing on central line maintenance. Am J Infect Control 2014;42:723–730.CrossRefGoogle ScholarPubMed
6. Vonberg, RF, Behnke, M, Geffers, C, et al. Device-associated infection rates for non–intensive care unit patients. Infect Control Hosp Epidemiol 2006;27:357–361.Google Scholar
7. Son, CH, Daniels, TL, Eagan, JA, et al. Central line–associated bloodstream infection surveillance outside the intensive care unit: a multicenter survey. Infect Control Hosp Epidemiol 2012;33:869–874.Google Scholar
8. Climo, M, Diekema, D, Warren, DK, et al. Prevalence of the use of central venous access devices within and outside of the intensive care unit: results of a survey among hospitals in the prevention epicenter program of the Centers for Disease Control and Prevention. Infect Control Hosp Epidemiol 2003;24:942–945.Google Scholar
9. Woeltje, KF, McMullen, KM, Butler, AM, Goris, AJ, Doherty, JA. Electronic surveillance for healthcare-associated central line–associated bloodstream infections outside the intensive care unit. Infect Control Hosp Epidemiol 2011;32:1086–1090.Google Scholar
10. Woeltje, KF, Butler, AM, Goris, AJ, et al. Automated surveillance for central line–associated bloodstream infection in intensive care units. Infect Control Hosp Epidemiol 2008;29:842–846.Google Scholar
11. Freeman, R, Moore, LSP, Alvarez, LG, Charlett, A, Holmes, A. Advances in electronic surveillance for healthcare associated infections in the 21st century: a systematic review. J Hosp Infect 2013;84:106–119.Google Scholar
12. Woeltje, KF, Lin, MY, Klompas, M, Wright, MO, Zuccotti, G, Trick, WE. Data requirements for electronic surveillance of healthcare-associated infections. Infect Control Hosp Epidemiol 2014;35:1083–1091.Google Scholar
13. Healthcare facility HAI reporting requirements to CMS via NHSN: current requirements. Centers for Disease Control and Prevention website. http://www.cdc.gov/nhsn/PDFs/CMS/CMS-Reporting-Requirements.pdf. Published 2013. Accessed September 5, 2014.Google Scholar
Figure 0

TABLE 1Comparison of Routine Surveillance and Fully Automated Electronic Surveillance by Number of Organisms Confirmed as CLABSIs–ICU Only

Figure 1

TABLE 2Performance of New Rules and Aggregate Rule Sets for CLABSI Prediction–ICU Only

Figure 2

TABLE 3Comparison of Routine Surveillance and Fully Automated Electronic Surveillance by Number of Organisms Confirmed as CLABSIs–Non-ICU Only

Figure 3

TABLE 4Performance of New Rules Ia + II for CLABSI Prediction–Non-ICU Only