Hostname: page-component-7b7567c8c-kwt9d Total loading time: 0 Render date: 2026-11-12T06:54:22.598Z Has data issue: false hasContentIssue false
  • English
  • Français

Natural Language Processing for Real-Time Catheter-Associated Urinary Tract Infection Surveillance: Results of a Pilot Implementation Trial

Published online by Cambridge University Press:  26 May 2015

Westyn Branch-Elliman,
Judith Strymish,
Valmeek Kudesia,
Amy K. Rosen and
Kalpana Gupta
Westyn Branch-Elliman*
Affiliation:
Department of Medicine, Eastern Colorado Veterans Affairs (VA) Healthcare System, Denver, ColoradoDepartment of Medicine, University of Colorado School of Medicine, Denver, Colorado
Judith Strymish
Affiliation:
Department of Medicine, VA Boston Healthcare System, Boston, MassachusettsHarvard University Medical School, Boston, Massachusetts
Valmeek Kudesia
Affiliation:
Department of Medicine, VA Boston Healthcare System, Boston, MassachusettsMassachusetts Veterans Epidemiology and Information Center, Boston VA Medical Center, Boston, Massachusetts
Amy K. Rosen
Affiliation:
Boston University School of Medicine, Boston, MassachusettsCenter for Healthcare Organization and Implementation Research, VA Boston Healthcare System, Boston, MassachusettsDepartment of Surgery, Boston University School of Medicine, Boston, Massachusetts
Kalpana Gupta
Affiliation:
Department of Medicine, VA Boston Healthcare System, Boston, MassachusettsBoston University School of Medicine, Boston, MassachusettsCenter for Healthcare Organization and Implementation Research, VA Boston Healthcare System, Boston, Massachusetts
*
Address correspondence to Westyn Branch-Elliman, MD, MMSc, Medical Director of Infection Prevention, ECHCS, 1055 Clermont Avenue, Mailstop 111L, Denver, CO 80220 (westyn.branch-elliman@ucdenver.edu).
Rights & Permissions [Opens in a new window]

Abstract

BACKGROUND

Incidence of catheter-associated urinary tract infection (CAUTI) is a quality benchmark. To streamline conventional detection methods, an electronic surveillance system augmented with natural language processing (NLP), which gathers data recorded in clinical notes without manual review, was implemented for real-time surveillance.

OBJECTIVE

To assess the utility of this algorithm for identifying indwelling urinary catheter days and CAUTI.

SETTING

Large, urban tertiary care Veterans Affairs hospital.

METHODS

All patients admitted to the acute care units and the intensive care unit from March 1, 2013, through November 30, 2013, were included. Standard surveillance, which includes electronic and manual data extraction, was compared with the NLP-augmented algorithm.

RESULTS

The NLP-augmented algorithm identified 27% more indwelling urinary catheter days in the acute care units and 28% fewer indwelling urinary catheter days in the intensive care unit. The algorithm flagged 24 CAUTI versus 20 CAUTI by standard surveillance methods; the CAUTI identified were overlapping but not the same. The overall positive predictive value was 54.2%, and overall sensitivity was 65% (90.9% in the acute care units but 33% in the intensive care unit). Dissimilarities in the operating characteristics of the algorithm between types of unit were due to differences in documentation practice. Development and implementation of the algorithm required substantial upfront effort of clinicians and programmers to determine current language patterns.

CONCLUSIONS

The NLP algorithm was most useful for identifying simple clinical variables. Algorithm operating characteristics were specific to local documentation practices. The algorithm did not perform as well as standard surveillance methods.

Infect. Control Hosp. Epidemiol. 2015;36(9):1004–1010

Information

Type
Original Articles
Information
Infection Control & Hospital Epidemiology , Volume 36 , Supplement 9 , September 2015, pp. 1004 - 1010
Copyright
© 2015 by The Society for Healthcare Epidemiology of America. All rights reserved 

Catheter-associated urinary tract infections (CAUTIs) are the most common healthcare-associated infections and the third-leading cause of death related to healthcare-associated infection.Reference Klevens, Edwards and Richards 1 Thus, CAUTI prevention is a top priority for hospitals, administrators, and infection prevention programs.

In most facilities, measurement of CAUTI rates is accomplished by manual chart review. Downsides of manual review include that the strategy is time intensive and often requires interdisciplinary coordination, and reproducibility and interrater reliability are low.Reference Mayer, Greene and Howell 2 – Reference Worth, Brett, Bull, McBryde, Russo and Richards 4 Automated surveillance of healthcare-associated infections is a theoretically attractive alternative but few data are available regarding the resources required for development, implementation, and maintenance of fully automated systems.Reference Klompas and Yokoe 5 , Reference Meddings, Rogers, Krein, Fakih, Olmsted and Saint 6 Most automated algorithms focus on administrative and microbiologic variables,Reference Shepard, Hadhazy and Frederick 7 – Reference Landers, Apte, Hyman, Furuya, Glied and Larson 9 but the sensitivity and specificity of these methods are questionable. The use of International Statistical Classification of Disease, Ninth Revision, coding alone is objective but has poor sensitivity.Reference Landers, Apte, Hyman, Furuya, Glied and Larson 9 , Reference Cass, Kelly, Probst, Addy and McKeown 10

Natural language processing (NLP)Reference Friedman, Rindflesch and Corn 11 offers an alternative means of automated surveillance not reliant upon traditional data extraction methods. NLP-augmented systems are able to mine data from free-text in unstructured documents, such as clinical progress notes. Data extracted by NLP algorithms can be combined with traditional structured data to create a fully automated surveillance program without the need for manual review. 12

We previously reported the derivation of an NLP-based algorithm for identifying indwelling urinary catheter (IUC) days (IUCDs) in a static snapshot of 8 hospital days.Reference Kudesia, Strymish, D'Avolio and Gupta 13 – Reference Wright, Fisher, John, Reynolds, Peterson and Robicsek 15 However, few data are available regarding tangible implementation of NLP-augmented algorithms. Thus, we sought to evaluate whether manual chart review could be supplemented or replaced with an NLP-augmented algorithm in a real-world clinical setting, and to characterize the resources required to develop, implement, and sustain an NLP-augmented system.

METHODS

Setting

The Boston Veterans Affairs (VA) Healthcare System is a 545-bed facility, with approximately 110,000 admissions annually. The system includes acute care units (ACU) and a 30-bed intensive care unit (ICU).

Participants

All patients admitted to the ACU and ICU from March 1, 2013, through November 30, 2013, were included.

Resource Utilization Required for Algorithm Development and Refinement

Time and resources required for developing, implementing, and sustaining the NLP-augmented algorithm in real time were recorded and reviewed. A qualitative review of necessary steps was compiled.

Documentation

Documentation of indwelling urinary catheter (IUC) varies by unit. In the inpatient wards, IUCs are documented in progress notes. In the ICU, IUCs are documented in an electronic database not directly linked to the electronic medical record. For full details of documentation practice, see Online Appendix 1.

Gold-Standard Infection Prevention Surveillance

National Healthcare Safety Network (NHSN) definitions are used for standard surveillance. To identify cases, a combination of electronic data extraction (IUCDs in the ICU, microbiologic data in all units) and manual medical chart review and data extraction (IUCDs in the ACU, vital signs and symptoms in all units) is used. Coordination of multiple services (nursing, healthcare quality, infection control) is required for data extraction and management. For full details of standard practice, see Online Appendix 2.

NLP for Real-Time Surveillance in Hospital Wards

IUCDs were classified using an NLP-augmented algorithm developed within the VA, used prospectively (Figure 1).Reference Kudesia, Strymish, D'Avolio and Gupta 13 For CAUTI detection, the NLP programming was expanded to search for clinical documentation of fever using temperatures at least 100.4°F and/or keywords (for full details of programming, see Online Appendix 3). The phrases included in the NLP were chosen on the basis of input from key stakeholders, followed by a detailed medical chart review to determine local clinical documentation patterns. Of note, for patients in the ACU, nurses are required to document a minimum of 1 temperature per shift; however, there are no specific guidelines outlining that this be the highest or lowest temperature. Thus, relying solely on vital signs data may miss clinical “fevers.”

FIGURE 1

Data extraction and processing by the natural language processing (NLP) algorithm, followed by logic of electronic catheter-associated urinary tract infection (CAUTI) determination. Key data points for the electronic algorithm were extracted from several sources. Data were then combined to determine whether the patient had an indwelling urinary catheter present. If there was no indwelling urinary catheter, or if the indwelling urinary catheter was in place for less than 3 days, then no further evaluation for CAUTI was undertaken by the electronic algorithm. If an indwelling urinary catheter was present for 3 or more days, then urine culture results and vital signs were evaluated to determine whether CAUTI was present. Fever and urine cultures had to be documented either on the same day or on consecutive days to count as a CAUTI. Note that the criteria for CAUTI identification used in the NLP-augmented algorithm differs from standard National Healthcare Safety Network (NHSN) definitions. VA, Veterans Affairs.

Data collected from free-text were combined with structured data (vital signs, microbiology, urinalysis). Fever, urine cultures, and urinalysis had to be documented either on the same day or on consecutive days to qualify as a potential CAUTI. Searches were performed independently for the 2 types of unit.

Of note, manual review recorded a “snapshot” view of IUCDs at the same time every day (ie, 6 AM), in accordance with NHSN recommendations. Conversely, the NLP-augmented algorithm used a “video” approach to recording IUCDs and therefore counted any IUC placed at any time during a given day (Figure 2).

FIGURE 2

Schematic of indwelling urinary catheter (IUC) day counting by natural language processing (NLP)–based algorithm versus infection control manual review. Manual review recorded a “snapshot” view at the same time every day (ie, 6 AM). If an indwelling urinary catheter was placed after the standard evaluation time or removed before the standard evaluation time, the day did not count as an indwelling urinary catheter day, per National Healthcare Safety Network guidelines. The NLP program used a “video” approach to recording indwelling urinary catheter days and so counted indwelling urinary catheters placed at any time during the day as an indwelling urinary catheter day. In scenario 1, in which the indwelling urinary catheter was placed at 9 PM and removed at 2 AM, there was a 50% higher number of IUCDs counted by the NLP program compared with manual review. In scenario 2, in which the indwelling urinary catheter was placed at 2 AM and removed at 9 AM, the 2 methods counted the same number of days. Two is the maximum difference between the 2 approaches.

Validation and Accuracy of NLP Report

IUCDs and CAUTIs identified by the NLP-augmented algorithm were compared with the number generated from routine practice. Test characteristics of the NLP-augmented algorithm were calculated and stratified by inpatient unit. In cases where the NLP- algorithm failed to correctly classify a patient, extensive medical chart review was undertaken to determine the root cause of the misclassification.

Implementation Barriers

Barriers to full automation of the NLP-augmented algorithm were reviewed qualitatively. Estimates of resources required for algorithm development were determined on the basis of experiences of programmers and team managers. Time required for standard CAUTI surveillance was based on estimates from experienced infection prevention practitioners.

Ethical Considerations

The VA Boston Healthcare System Institutional Review Board approved the study (study no. 2730).

RESULTS

Setting and Participants

Veterans admitted during the period from March 1, 2013, through November 30, 2013, to our large, urban VA medical center were included. A total of 43,609 patient-days from the ACU and ICU were identified.

Resource Utilization Required for Algorithm Development and Refinement

This NLP algorithm was based on a simple regression approach, which uses keywords and key phrases to identify clinical data. Development of this algorithm for identification of IUCDs was resource intense. First, a team of clinicians developed a list of phrases that might be used to identify presence of an IUC. Then, 500 hospital days were manually reviewed to identify any other words and phrases used to identify IUCs. Finally, an electronic algorithm was developed by programmers on the basis of all identified words and phrases. Once collected, extracted data were reviewed by infection prevention staff to determine validity.

NLP algorithm development required excellent “gold standard” data for comparison to ensure that the electronic program correctly identified and extracted data. Identifying a clear standard for comparison to fine-tune the NLP programming presented problems independent of the NLP itself. For example, electronic systems identified “two north” and “two north telemetry” as different patient care areas. In the manually generated list, many errors were identified (incorrect patient medical record numbers, incorrect ward assignment). Thus, the NLP algorithm required significant data cleaning before the algorithm could be implemented in real time.

Real-Time Algorithm Performance

Indwelling urinary catheter days

In the ACU, the NLP-augmented algorithm consistently identified more IUCDs than standard evaluation (mean, 27% more IUCDs) (Table 1). Qualitative review found the primary reason for the higher number of IUCDs was due to differences in timing of assessment for catheter presence (Figure 2). Conversely, in the ICU the NLP-augmented algorithm consistently identified fewer IUCDs than standard surveillance (mean, 28% fewer) (Table 1). The primary reason NLP underestimated the number of IUCDs in the ICU was lack of systematic documentation in the progress notes.

TABLE 1

Performance of Natural Language Processing (NLP) Algorithm Versus Standard, Manual Review Surveillance for Indwelling Urinary-Catheter Days (IUCDs) in 2 Different Care Settings

CAUTI

Overall, NLP and manual review identified a similar total number of CAUTI (24 by NLP and 20 by manual review). Although the gross numbers appeared similar, the patients identified by the 2 review strategies were overlapping but not the same. The positive predictive value for identifying CAUTI in inpatient settings was 54.2%.

In the ACU, the NLP-augmented algorithm flagged 19 CAUTIs (10 true-positive and 9 false-positive), compared with 11 identified by standard surveillance, and did not identify any cases not flagged by standard surveillance methods. False-positives flagged by the electronic system were due to differences between how clinicians document symptoms and results and the NHSN definitions, as well as lack of specific filters for evaluating urine cultures and urinalyses (for full details, see Online Appendix 4). The cause of the 1 false-negative case was unclear (sensitivity, 90.9%; Table 2).

TABLE 2

Test Characteristics of Automated Natural Language Processing (NLP) Algorithm for Catheter-Associated Urinary Tract Infection (CAUTI) Surveillance

NOTE. Results were based on incidence of CAUTI per number of indwelling urinary catheter days (IUCDs). IUCDs were calculated on the basis of manual review for the intensive care unit, and via the NLP algorithm for the acute care units.

In the ICU, the NLP-augmented algorithm demonstrated even lower performance, identifying 5 CAUTI, and only 3 of the 9 true CAUTI. All 6 false-negatives were secondary to incorrect flagging of temperatures. The reason for misclassification by the NLP-augmented algorithm was vital signs documentation in the ICU. In our ICU, vital signs are entered into an unlinked electronic system (Careview). The NLP is directed toward the electronic medical record and thus was unable to mine these data elements. In both false-positive identifications, the patient did not meet NHSN criteria for CAUTI owing to temperatures less than 100.4°F; the NLP flagged these cases on the basis of keywords present in clinical progress notes.

All discrepancies between the NLP-augmented algorithm flag and the standard surveillance determination were compared to determine whether adding additional clinical variables (eg, dysuria, suprapubic pain, among others) would improve algorithm performance. None of the 7 cases missed by the NLP algorithm would have been reclassified.

Implementation Barriers

In this center, CAUTI surveillance requires an average of 8 hours per month of infection prevention practitioner time, not including the substantial time required for data input and extraction by other services (nursing and healthcare quality). The time and resources required for further development of this NLP program are unknown; however, initial development and implementation required substantial upfront commitment by a team of clinicians and programmers, followed by extensive time for validation and data cleaning. Utilization requires ongoing updates and data extraction, which is also time intensive. Refinement may be a particularly large barrier for using NLP for identification of healthcare-associated infections because NHSN definitions are updated frequently, and thus the algorithm would require frequent re-programming and re-validation (Table 3).

TABLE 3

Barriers to Implementation of Natural Language Processing (NLP) and Suggested Solutions

Moving to an NLP-augmented algorithm requires not only technological development of the algorithm but also a targeted system for clarifying documentation. Changing documentation practices requires coordination of multiple services and providers in order to enact significant culture change, and this may be infeasible. Other barriers to implementation include lack of interinstitutional standardization for IUCD documentation and a broad range of data collection methods both within a single institution and across institutions. Further refining the NLP algorithm to function outside of a single hospital network would be time consuming and require additional data management support, and it is unclear whether the long-term benefits would outweigh the significant costs.

DISCUSSION

Electronic surveillance has the potential to improve the accuracy of measurement by reducing the clinical subjectivity present in manual reviewReference Stevens, Kachniarz and Wright 3 and by increasing the efficiency and standardization of healthcare-associated infection detection.Reference Shepard, Hadhazy and Frederick 7 NLP can theoretically be used to integrate many data collection systems and circumvent manual chart review.Reference Jones, DuVall, Spuhl, Samore, Nielson and Rubin 16 – Reference Dublin, Baldwin and Walker 18 The purpose of this study was to evaluate how well an NLP-augmented algorithm can function in a real-world clinical setting. This information is critical because automated systems are increasingly being adopted.

Overall, the NLP had reasonable accuracy for identifying IUCDs but performed suboptimally for CAUTI detection. Although only a narrow piece of the NHSN definitions was included in the NLP algorithm development, no additional CAUTI would have been identified if additional clinical variables had been programmed into the NLP algorithm. Thus, it remains unclear whether refining the algorithm to include additional keywords would significantly improve its operating characteristics. Furthermore, integrating these variables would be extremely resource intense, requiring significant time and effort before updates could be integrated.

During the implementation, we found that CAUTI capture was dramatically better when a systematic approach to documentation was partnered with the use of the NLP program; use of a templated note mandated that all documentation be completed appropriately and that language specifically recognized by the NLP algorithm be used in favor of true free-text. If a templated note was not implemented simultaneously, then the NLP performed poorly. Templated notes, however, may present other barriers for patient care. Progress notes may include a litany of imported items and check boxes, which may improve data collection and compliance while also worsening readability and communication between healthcare providers.

A significant barrier to expansion of the NLP-augmented system is that the algorithm itself is heavily nuanced for use in a particular institution. We used a simple regression approach based upon local language patterns. This approach is time consuming upfront but has the upside of being transparent to the end-user. Machine learning is an alternate programming approach that may bypass the need for upfront review; however, this strategy is less transparent and more difficult to adapt, and it requires robust review to ensure proper functioning. Furthermore, the machine learning approach “learns” only local language patterns and thus may not be reliable when expanded beyond a single healthcare system. As practice patterns change and as documentation practices and definitions evolve, NLP algorithms require continuous updating. Thus, NLP-augmented surveillance programs require a significant commitment from programmers, clinicians, and technologic experts. Infection prevention review requires approximately 8 hours per month in our institution. Whether transitioning from current practice to an NLP-augmented system is an efficient use of resources is unclear. Other barriers to expansion include staff turnover, particularly in facilities with trainees that rotate frequently, because achieving compliance with templated notes may be difficult.

Our prospective study represents real-world implementation data in several different patient-care areas; however, it is limited because it represents the experience of a single, closed, healthcare system, and the operating characteristics of the algorithm may vary considerably in other clinical settings. Indeed, we found marked variability between different units within our facility. Another limitation is our use of standard surveillance as the gold standard of comparison, because CAUTI identification via usual methods is known to have low sensitivity. However, there is no other clear gold standard, and a goal of this study was to determine whether an NLP-augmented algorithm would perform better than traditional methods.

Conclusions

The NLP algorithm performed well for identifying IUCDs but had poor operating characteristics for identifying CAUTI in real time. Given the resource-intensive nature of developing NLP-based systems compared with more traditional data extraction methods, it remains unclear whether developing NLP for this purpose is an appropriate use of limited resources. However, further evaluation of NLP as a tool to complement other electronic surveillance to identify CAUTI is warranted.

Acknowledgments

We thank Suzanne Mosesso, RN (Infection Prevention), Anne Brogan, RN (Nursing), and the entire Infection Prevention Staff at the Boston VA Healthcare System for their help with implementation and data collection.

Financial support. Patient Safety Center of Inquiry “Measurement to Advance Patient Safety,” grant number XVA 68-023, Boston, Massachusetts.

Potential conflicts of Interest. All authors report no conflicts of interest relevant to this article.

Supplementary material

To view supplementary material for this article, please visit http://dx.doi.org/10.1017/ice.2015.122

References

1. Klevens, RM, Edwards, JR, Richards, CL Jr, et al. Estimating health care-associated infections and deaths in U.S. hospitals, 2002. Public Health Rep 2007;122:160–166.Google Scholar
2. Mayer, J, Greene, T, Howell, J, et al. Agreement in classifying bloodstream infections among multiple reviewers conducting surveillance. Clin Infect Dis 2012;55:364–370.Google Scholar
3. Stevens, JP, Kachniarz, B, Wright, SB, et al. When policy gets it right: variability in U.S. hospitals' diagnosis of ventilator-associated pneumonia. Crit Care Med 2014;42:497–503.CrossRefGoogle ScholarPubMed
4. Worth, LJ, Brett, J, Bull, AL, McBryde, ES, Russo, PL, Richards, MJ. Impact of revising the National Nosocomial Infection Surveillance System definition for catheter-related bloodstream infection in ICU: reproducibility of the National Healthcare Safety Network case definition in an Australian cohort of infection control professionals. Am J Infect Control 2009;37:643–648.Google Scholar
5. Klompas, M, Yokoe, DS. Automated surveillance of healthcare-associated infections. Clin Infect Dis 2009;48:1268–1275.Google Scholar
6. Meddings, J, Rogers, MA, Krein, SL, Fakih, MG, Olmsted, RN, Saint, S. Reducing unnecessary urinary catheter use and other strategies to prevent catheter-associated urinary tract infection: an integrative review. BMJ Qual Saf 2014;23:277–289.CrossRefGoogle ScholarPubMed
7. Shepard, J, Hadhazy, E, Frederick, J, et al. Using electronic medical records to increase the efficiency of catheter-associated urinary tract infection surveillance for National Health and Safety Network reporting. Am J Infect Control 2014;42:e33–e36.CrossRefGoogle ScholarPubMed
8. Lo, YS, Lee, WS, Liu, CT. Utilization of electronic medical records to build a detection model for surveillance of healthcare-associated urinary tract infections. J Med Sys 2013;37:9923.CrossRefGoogle ScholarPubMed
9. Landers, T, Apte, M, Hyman, S, Furuya, Y, Glied, S, Larson, E. A comparison of methods to detect urinary tract infections using electronic data. Jt Comm J Qual Patient Saf 2010;36:411–417.Google Scholar
10. Cass, AL, Kelly, JW, Probst, JC, Addy, CL, McKeown, RE. Identification of device-associated infections utilizing administrative data. Am J Infect Control 2013;41:1195–1199.Google Scholar
11. Friedman, C, Rindflesch, TC, Corn, M. Natural language processing: state of the art and prospects for significant progress, a workshop sponsored by the National Library of Medicine. J Biomed Inform 2013;46:765–773.CrossRefGoogle ScholarPubMed
12. Centers for Disease Control and Prevention. Urinary tract infection (catheter-associated urinary tract infection [CAUTI] and non-catheter-associated urinary tract infection [UTI] and other urinary system infection [USI]) events. Device-associated module CAUTI. http://www.cdc.gov/nhsn/PDFs/pscManual/7pscCAUTIcurrent.pdf. Published 2014. Accessed March 24, 2014.Google Scholar
13. Kudesia, V, Strymish, J, D'Avolio, L, Gupta, K. Natural language processing to identify Foley catheter–days. Infect Control Hosp Epidemiol 2012;33:1270–1272.Google Scholar
14. Choudhuri, JA, Pergamit, RF, Chan, JD, et al. An electronic catheter-associated urinary tract infection surveillance tool. Infect Control Hosp Epidemol 2011;32:757–762.CrossRefGoogle ScholarPubMed
15. Wright, MO, Fisher, A, John, M, Reynolds, K, Peterson, LR, Robicsek, A. The electronic medical record as a tool for infection surveillance: successful automation of device-days. Am J Infect Control 2009;37:364–370.CrossRefGoogle ScholarPubMed
16. Jones, M, DuVall, SL, Spuhl, J, Samore, MH, Nielson, C, Rubin, M. Identification of methicillin-resistant Staphylococcus aureus within the nation's Veterans Affairs medical centers using natural language processing. BMC Med Inform Decis Mak 2012;12:34.CrossRefGoogle ScholarPubMed
17. Forbush, TB, Gundlapalli, AV, Palmer, MN, et al. "Sitting on pins and needles": characterization of symptom descriptions in clinical notes. AMIA Jt Summits Transl Sci Proc 2013;2013:67–71.Google Scholar
18. Dublin, S, Baldwin, E, Walker, RL, et al. Natural language processing to identify pneumonia from radiology reports. Pharmacoepidemiol Drug Saf 2013;22:834–841.Google Scholar
Figure 0

FIGURE 1Data extraction and processing by the natural language processing (NLP) algorithm, followed by logic of electronic catheter-associated urinary tract infection (CAUTI) determination. Key data points for the electronic algorithm were extracted from several sources. Data were then combined to determine whether the patient had an indwelling urinary catheter present. If there was no indwelling urinary catheter, or if the indwelling urinary catheter was in place for less than 3 days, then no further evaluation for CAUTI was undertaken by the electronic algorithm. If an indwelling urinary catheter was present for 3 or more days, then urine culture results and vital signs were evaluated to determine whether CAUTI was present. Fever and urine cultures had to be documented either on the same day or on consecutive days to count as a CAUTI. Note that the criteria for CAUTI identification used in the NLP-augmented algorithm differs from standard National Healthcare Safety Network (NHSN) definitions. VA, Veterans Affairs.

Figure 1

FIGURE 2Schematic of indwelling urinary catheter (IUC) day counting by natural language processing (NLP)–based algorithm versus infection control manual review. Manual review recorded a “snapshot” view at the same time every day (ie, 6 AM). If an indwelling urinary catheter was placed after the standard evaluation time or removed before the standard evaluation time, the day did not count as an indwelling urinary catheter day, per National Healthcare Safety Network guidelines. The NLP program used a “video” approach to recording indwelling urinary catheter days and so counted indwelling urinary catheters placed at any time during the day as an indwelling urinary catheter day. In scenario 1, in which the indwelling urinary catheter was placed at 9 PM and removed at 2 AM, there was a 50% higher number of IUCDs counted by the NLP program compared with manual review. In scenario 2, in which the indwelling urinary catheter was placed at 2 AM and removed at 9 AM, the 2 methods counted the same number of days. Two is the maximum difference between the 2 approaches.

Figure 2

TABLE 1Performance of Natural Language Processing (NLP) Algorithm Versus Standard, Manual Review Surveillance for Indwelling Urinary-Catheter Days (IUCDs) in 2 Different Care Settings

Figure 3

TABLE 2Test Characteristics of Automated Natural Language Processing (NLP) Algorithm for Catheter-Associated Urinary Tract Infection (CAUTI) Surveillance

Figure 4

TABLE 3Barriers to Implementation of Natural Language Processing (NLP) and Suggested Solutions

Supplementary material: File

Branch-Elliman supplementary material

Appendix 1-4

Download Branch-Elliman supplementary material(File)
File 19.3 KB