Hostname: page-component-7b7567c8c-tnzcf Total loading time: 0 Render date: 2026-11-12T16:19:01.087Z Has data issue: false hasContentIssue false
  • English
  • Français

Internal and External Validation of a Computer-Assisted Surveillance System for Hospital-Acquired Infections in a 754-Bed General Hospital in the Netherlands

Published online by Cambridge University Press:  04 August 2016

H. Roel A. Streefkerk,
Ivar O. Lede,
John L. V. Eriksson,
Marije G. Meijling,
Conrad P. van der Hoeven,
Jan C. Wille,
Titia E. M. Hopmans,
Alex W. Friedrich,
Henri A. Verbrugh and
Nashwan al Naiemi
H. Roel A. Streefkerk*
Affiliation:
Erasmus University Medical Center (Erasmus MC), Rotterdam, NetherlandsAlbert Schweitzer Hospital/Regionaal Laboratorium medische Microbiologie, Dordrecht, Netherlands
Ivar O. Lede
Affiliation:
Ziekenhuisgroep Twente, Almelo/Hengelo, NetherlandsLabMicTA, Hengelo, NetherlandsDepartment of Medical Microbiology, University of Groningen, University Medical Center Groningen, Groningen, Netherlands
John L. V. Eriksson
Affiliation:
Erasmus University Medical Center (Erasmus MC), Rotterdam, NetherlandsKatholieke Universiteit Leuven, Belgium
Marije G. Meijling
Affiliation:
Ziekenhuisgroep Twente, Almelo/Hengelo, Netherlands
Conrad P. van der Hoeven
Affiliation:
Erasmus University Medical Center (Erasmus MC), Rotterdam, Netherlands
Jan C. Wille
Affiliation:
National Institute for Public Health and the Environment, Center for Infectious Disease Control Bilthoven, Netherlands
Titia E. M. Hopmans
Affiliation:
National Institute for Public Health and the Environment, Center for Infectious Disease Control Bilthoven, Netherlands
Alex W. Friedrich
Affiliation:
Department of Medical Microbiology, University of Groningen, University Medical Center Groningen, Groningen, Netherlands
Henri A. Verbrugh
Affiliation:
Erasmus University Medical Center (Erasmus MC), Rotterdam, Netherlands
Nashwan al Naiemi
Affiliation:
Ziekenhuisgroep Twente, Almelo/Hengelo, NetherlandsLabMicTA, Hengelo, NetherlandsDepartment of Medical Microbiology, University of Groningen, University Medical Center Groningen, Groningen, Netherlands
*
Address correspondence to H.R.A. Streefkerk, Albert Schweitzerplaats 25, 3318 AT Dordrecht, the Netherlands, P.O. Box 899, 3300 AW Dordrecht, Netherlands (r.streefkerk@rlm-microbiologie.nl).
Rights & Permissions [Opens in a new window]

Abstract

OBJECTIVE

To evaluate a computer-assisted point-prevalence survey (CAPPS) for hospital-acquired infections (HAIs).

DESIGN

Validation cohort.

SETTING

A 754-bed teaching hospital in the Netherlands.

METHODS

For the internal validation of a CAPPS for HAIs, 2,526 patients were included. All patient records were retrospectively reviewed in depth by 2 infection control practitioners (ICPs) to determine which patients had suffered an HAI. Preventie van Ziekenhuisinfecties door Surveillance (PREZIES) criteria were used. Following this internal validation, 13 consecutive CAPPS were performed in a prospective study from January to March 2013 to determine weekly, monthly, and quarterly HAI point prevalence. Finally, a CAPPS was externally validated by PREZIES (Rijksinstituut voor Volksgezondheid en Milieu [RIVM], Bilthoven, Netherlands). In all evaluations, discrepancies were resolved by consensus.

RESULTS

In our series of CAPPS, 83% of the patients were automatically excluded from detailed review by the ICP. The sensitivity of the method was 91%. The time spent per hospital-wide CAPPS was ~3 hours. External validation showed a negative predictive value of 99.1% for CAPPS.

CONCLUSIONS

CAPPS proved to be a sensitive, accurate, and efficient method to determine serial weekly point-prevalence HAI rates in our hospital.

Infect Control Hosp Epidemiol 2016;1–6

Information

Type
Original Articles
Information
Infection Control & Hospital Epidemiology , Volume 37 , Issue 11 , November 2016, pp. 1355 - 1360
Copyright
© 2016 by The Society for Healthcare Epidemiology of America. All rights reserved 

Surveillance of HAI is the systematic collection of data, consolidation and analysis of these data into useful information, and dissemination of results to persons who need to know and can take action.Reference Langmuir 1 Traditional hospital-wide, ward-based point-prevalence surveys are labor intensive and therefore are performed infrequently, eg, only twice a year by the current infection control staff in Ziekenhuis Groep Twente (ZGT), a 754-bed general hospital. This low frequency of surveys does not facilitate the detection of trends or outbreaks and provides little insight into the hospital-wide burden of HAI.Reference Brusaferro, Regattin and Faruzzo 2 Using automated algorithmsReference Leth and Moller 3 to discriminate between patients with and without HAI, point-prevalence surveys can be performed much more efficiently, making it possible to increase the number of surveys performed per year and to introduce analysis at ward and service levels. In a 1,200-bed university hospital, we previously showed that, with an automated selection algorithm (ie, IDM-Surveillance software), which reliably and consistently excluded 70% of patients from detailed review by the ICP, it is indeed possible to perform a point-prevalence survey in significantly less time.Reference Freeman, Moore, Garcia Alvarez, Charlett and Holmes 4 – Reference Streefkerk, Moorman and Parlevliet 6 Because the case mix of patients, complexity of care, length of stay, and prevalence of HAI differ among hospitals, we evaluated the algorithm and the use of IDM-Surveillance in a large general hospital.

METHODS

Ziekenhuis Groep Twente is a 754-bed teaching hospital with 2 locations. Annually, ZGT has 37,188 admissions with a total of 189,153 hospital days (Annual Report, 2012). The Department of Infection Control of ZGT has 5 ICPs (3.4 full-time equivalents).

A hospital-wide, ward-based, point-prevalence survey is routinely performed twice each year to determine the HAI burden in ZGT. For each ward, an appointment with an attending nurse is made and a registration form (including instructions) 7 is sent in preparation for an interview in which the answers are discussed with the ICP. All data and results are submitted to the national Preventie van Ziekenhuisinfecties door Surveillance (PREZIES) database (see Supplementary Table).

IDM-Surveillance is a clinical decision support system and workflow management tool that facilitates surveillance of HAI based on frequently performed CAPPS and automates the dissemination of information to managerial and medical staff. The algorithm, method, and software were developed and validated at the Department of Medical Microbiology and Infectious Diseases, Erasmus University Medical Center (Erasmus MC), Rotterdam, Netherlands.

The software consists of a standardized and normalized database with interfaces to the hospital information systems and laboratory and clinical databases. Through daily automated import services, this database is uploaded with all laboratory, clinical, and census data needed for the software to execute its algorithms. Furthermore, electronic radiology and surgical operation reports as well as documented care plans by attending physicians and nurses are imported into the database.

A computer-assisted point-prevalence survey (CAPPS) consists of an automated algorithm-based selection of patients followed by manual assessment by the ICP to determine the presence of an HAI. The algorithm used in the CAPPS was integrally adopted from the Erasmus MC (where it was developed and validated).Reference Streefkerk, Borsboom, van der Hoeven, Vos, Verkooijen and Verbrugh 5 , Reference Streefkerk, Moorman and Parlevliet 6 For each patient, the automated algorithm calculates a daily nosocomial infection index (Nii), a score that is routinely saved. On the day of prevalence measurement, all patients with an Nii score above a specified threshold (ie, 8) are selected. According to the PREZIES national CDC-based criteria, 8 selected patients were subsequently listed for manual assessment by 2 independent ICPs for 5 groups of HAI: surgical site infection (SSI), bloodstream infection (BSI), lower respiratory tract infection (LRTI), urinary tract infection (UTI), and other. This manual assessment procedure is facilitated by an on-screen timeline representation of all relevant laboratory, pharmacy, and clinical data. Keywords to predict infection are automatically highlighted in the written reports. A dedicated workflow facilitates the assessment results, which are entered into the program by the ICPs. After each ICP has finished assessing the selected patients, the software detects discrepancies between their results. These discrepancies are resolved by consensus in a meeting of the 2 ICPs. After being medically authorized, the final results are automatically disseminated to the physicians (Figure 1).

FIGURE 1

Workflow computer-assisted point prevalence survey for hospital-acquired infection.

For internal validation of CAPPS, 2,526 patients from 6 consecutive traditional hospital-wide, ward-based, prevalence surveys performed between 2010 and 2013 were selected and re-reviewed in depth by 2 senior infection control professionals, 1 ICP and 1 medical doctor microbiologist, who had not participated in the original surveys in 2010–2013. Patients whose symptoms had already subsided but who were still being treated on the prevalence survey date were considered positive according to the PREZIES criteria. This in-depth review was used to represent true HAI prevalence. The same cohort was also assessed by a CAPPS using IDM-Surveillance software. The sensitivity, specificity, and accuracy of both the original, traditional, ward-based surveys and CAPPS were subsequently calculated using the same gold standard, ie, the results of the in-depth retrospective reassessment by the 2 ICPs.

Following this retrospective validation study, we conducted a prospective study in which 13 consecutive weekly CAPPS were performed in a period from January to March. The time spent per CAPPS was recorded.

To externally validate the results of a CAPPS, 3 infection control experts from the Dutch national organization for HAI surveillance (PREZIES, Bilthoven, Netherlands) evaluated 1 of the 13 CAPPS. During a 2-day visit, they assessed all 328 patients that had been scored negative by the software algorithm. For each patient, all laboratory and clinical data in the electronic patient record were assessed. Positive findings were discussed with the ICPs of ZGT and were resolved by consensus where possible. The chair of the PREZIES team had the final say on patient positivity for HAI.

RESULTS

Retrospective review by 2 ICPs of all 2,526 patients, previously included in 6 consecutive, hospital-wide, ward-based surveys from 2010 to 2013 yielded 55 separate episodes of HAI, resulting in an average overall HAI prevalence rate of 2.2%. Compared to this true HAI rate, the original ward-based surveys detected only 40 episodes of HAI, representing a sensitivity of 73%. The software algorithm marked 2,098 of 2,526 (83%) of the patients automatically negative for an HAI and detected 50 HAI episodes. Comparison of the results of the CAPPS and the traditional ward-based survey to the results of the retrospective review showed sensitivities of 91% (95% CI, 80%–97%) versus 73% (95% CI, 59%–83%) and accuracies of 99.8% versus 98.5%, respectively. With CAPPS, 100% of the SSIs (21 of 21), 83% of the BSIs (5 of 6), 94% of the LRTIs (16 of 17), and 67% of the UTIs (6 of 9) were selected by the software for ICP review (Table 1).

TABLE 1

A Summary of the Results of 6 Consecutive Point-Prevalence Surveys According to the Traditional Ward-Based Survey and CAPPS.Footnote a

a All patients who were scored positive by CAPPS were included in the pre-selection of patients made by the algorithm.

Following the internal validation study, 13 consecutive CAPPS were prospectively performed each Thursday from January to March 2013. The prevalence rate of patients with HAI was 1.8% (range, 0.9%–3.4%) (Figure 2). In total, 100 of 5,447 included patients suffered from 122 HAIs, including 26 SSIs, 37 UTIs, 32 LRTIs, 25 BSIs, and 2 other HAIs.

FIGURE 2

Prevalence of hospital acquired infection detected by 13 consecutive, weekly, hospital-wide, point-prevalence surveys using IDM-Surveillance software. Data are presented for all ward as a weekly, monthly, and quarterly scores, including 95% confidence intervals (upper panels), and for patients cared for by 2 medical specialties (lower panels).

In the 13 consecutive CAPPS, on average, 349 patients were automatically marked negative and 70 patients were scored positive. After detailed review by the ICP, on average, 8 of 70 patients (11%) were deemed truly positive for HAI. The time needed by ICPs to perform a single CAPPS was ~3 hours. During these 3 hours, ICPs reviewed the patients preselected by the IDM-Surveillance algorithm and solved their discrepancies (see Figure 1, point 4a, 4b, and 5).

An external audit team of PREZIES assessed each of the 328 patients that were scored negative by the software algorithm in 1 of the 13 CAPPS. Among these presumably HAI-negative patients, the external auditors found that 3 patients did have HAIs, 1 patient with an LRTI (a case of clinical pneumonia without microbiological confirmation), and 2 patients with deep SSIs.

DISCUSSION

In this study, we showed that CAPPS, conducted using IDM-Surveillance software, is a highly accurate and efficient method of performing hospital-wide screening for all types of HAI. Its efficiency is such that it can be performed on a weekly basis, even in a large teaching hospital because it requires only 3 hours of ICP time. Internal validation showed a sensitivity of 91% and accuracy of 99.9% with a CAPPS. In contrast, for a traditional ward-based survey, the sensitivity was 73% and the accuracy was 98.5%. The enhanced performance of the CAPPS can be explained by several factors. First, CAPPS allows ICPs to focus on high-risk patients; only 17% of the patients needed detailed assessment to find 91% of the HAIs.Reference van Mourik, Troelstra, van Solinge, Moons and Bonten 9 In addition, the intuitive, chronologically orderly (ie, time line), on-screen presentation of all relevant clinical (including the patient’s decursus and radiological findings) and laboratory data greatly facilitates the assessment of individual patients by the ICP.Reference Trick 10 Finally, the independent assessment of each patient by 2 ICPs and the subsequent discrepancy analysis by consensus meeting of these 2 ICP clearly enhanced the accuracy of the survey system.Reference Klompas 11 In the retrospective analysis, 5 HAIs (1 LRTI, 3 UTIs, and 1 BSI) were not picked up by the algorithm used in the CAPPS. In the case of the LRTI and UTI, only clinical criteria to score an HAI were met at the time of the point-prevalence survey. The infections were treated empirically, but CRP, leukocyte count, or microbiological examinations were not performed. The patient with BSI was still being treated with antibiotics at the time of the survey but had become free of signs and symptoms of invasive infection. On the other hand, 15 cases were missed in the traditional, ward-based, point-prevalence surveys. Several factors might have contributed to the higher detection of HAI during the retrospective review. In the ward-based surveys, each patient was assessed at a different point in time during working hours. During the retrospective review, each patient was assessed at 24:00 hours and, therefore, all clinical and diagnostic data of the whole day were taken into account. In this respect, the CAPPS method resembled the retrospective review because it also used 24:00 hours as the census time. Another factor may be that the level of expertise of the 2 senior ICPs who performed the in-depth retrospective review in a research setting exceeded that of the ICP performing the routine traditional surveys on the wards.Reference Keller, Linkin, Fishman and Lautenbach 12 Other studies have also reported improved case findingsReference Bolon, Hooper and Stevenson 13 – Reference Yokoe, Khan and Olsen 15 with the use of electronically assisted surveillance (EAS).Reference Woeltje 16

In the second part of the study, 13 consecutive CAPPS were performed in the first quarter of 2013, resulting in an average point prevalence of 1.8% (range, 0.9%–3.4%). Reporting the weekly point prevalence rates aggregated by month and quarter showed the potential of this method to detect trends in HAIs earlier and with statistical significance. Even when the numbers of hospitalized patients were small for 1 medical specialty, quarterly presentation of the results still provided useful insights regarding the burden of HAI.

The external validation showed a 99.1% (325 of 328) concordance between cases automatically marked negative for HAI during the CAPPS of March 21 and the findings of the PREZIES experts. The 3 discrepant cases consisted of 2 deep SSIs and 1 pneumonia. Further analysis showed that the 2 deep SSIs were still being treated (ie, by topical wound treatment and/or antibiotics) but that laboratory markers of infection were not present at the time of the survey. This finding suggests that the algorithm may need to be adjusted to further optimize case finding (eg, using text mining tools to find the clinical criteria on which a ‘pneumonia without microbiological confirmation’ is scored) when a single point-prevalence survey is performed only once every 6 months, as is the case for the current PREZIES point-prevalence surveys. It is also possible to lower the threshold of the automated algorithm to increase its sensitivity further, but this would result in a decrease in specificity and thus an increase in workload. During the development of the algorithm, we used receiver operator curve (ROC) analysis to determine the optimal trade-off between sensitivity and specificity, requiring a sensitivity of ≥90%. On the other hand, when point-prevalence surveys are repeated every week, HAI episodes are less likely to be missed. Indeed, the 2 patients with deep SSIs that were scored HAI negative by CAPPS were flagged positive by the algorithm in the weeks before; thus, they were selected for detailed review during their hospital stay (March 7 and 14, 2013). Because overall HAI prevalence is low and length of stay has been steadily decreasing over the last decade, the current sensitivity of a weekly CAPPS may become compromised in the future. By focusing surveillance activities on wards with an HAI risk and by simultaneously reducing the time interval between successive CAPPS combined with correction for repeatedly identified HAI in the same patient, this CAPPS system can be readily transformed to yield HAI incidence data of very good quality.

Performing repeated CAPPS in itself will not decrease HAI rates without interventions, but we are confident that based on the sensitivity of the CAPPS method, it can be used to measure the effects of interventions. However, in the 13 weeks we performed CAPPS, no major interventions were conducted in this hospital.

Both strategies, point-prevalence and incidence measurements, are useful to inform infection control teams about the presence of HAIs and each method has strengths. Incidence rates can be used to assess the risk for HAI, while point-prevalence is a more informative metric regarding the burden (morbidity) of HAI for the organization. In the Netherlands, hospitals voluntarily participate in hospital-wide point-prevalence surveys and/or incidence surveillance of SSI or CLABSI.

In the Erasmus MC in 2008, we started the development of the computer-assisted point-prevalence survey (CAPPS). A hospital-wide point-prevalence survey of HAI was conducted twice yearly thereafter within the framework of the national HAI surveillance system PREZIES. We then noted that, although we fully complied with national standards, the impact of the hospital-wide point-prevalence survey was limited. These point-prevalence surveys were labor intensive and took at least 2 weeks to complete. Importantly, the value of the information reported back to the clinical departments was limited by the small numbers of patients included per medical discipline. Notably, surveillance of HAIs by continuously measuring hospital-wide incidence rates is seldom reported, probably because it was too labor-intensive when performed without the aid of a computer-based search system.

A complete PREZIES point-prevalence survey basically consists of 3 parts: (1) gathering all data and risk factors for each patient included, (2) assessing whether an HAI is present, and (3) uploading the data and findings into the national database. On average in this hospital, 419 patients were included in a point-prevalence survey, 411 patients were negative, and 8 patients were positive for HAI. Based on the estimates presented in the supplementary data, the time periods needed to complete the 3 parts of a PREZIES survey are 105 hours, 43 hours, and 43 hours. All types (or subtypes) of HAIs are documented during a PREZIES survey. In a CAPPS, this documentation is limited to 5 categories: SSI, LRTI, UTI, BSI, and other. Furthermore, a PREZIES survey requires reading all patient records, and not only is the presence or absence of an HAI specified but also the use of antibiotics, the presence of catheters, and several other parameters are also included.

These 2 methods differ and serve different goals. A CAPPS in its current form can support the ICPs performing a PREZIES survey, but a CAPPS cannot replace this survey. Depending on the current hospital practice, full automation of data retrieval, decision making supported by automated algorithms, and visualization of information can lead to increased efficacy.

We report a time requirement of 3 hours per CAPPS in this study. This time period was achieved by a team consisting of a senior ICP and an MD microbiologist who performed all 13 surveys in consecutive sessions. In practice, a CAPPS performed each week by the regular infection control staff is, we believe, feasible in 1 working day.

In conclusion, an automated electronic surveillance method based on an expert rule-driven algorithm proved sensitive and efficient in a large general hospital. Only a minority of patients (17%) remains to be assessed in detail by the infection control team. Because this detailed assessment is supported by a time-line representation of all laboratory and clinical parameters, a net labor investment of only 3 hours was required per hospital-wide, 754-bed point-prevalence survey. Weekly CAPPS can provide hospital managerial and medical staff timely and valuable information regarding the presence of HAIs in their institutions.

ACKNOWLEDGMENTS

Financial support: No financial support was provided relevant to this article.

Potential conflicts of interest: AWF and NA are collaborating within a common publicly funded EU/Interreg-V grant. All other authors report no conflicts of interest relevant to this article.

SUPPLEMENTARY MATERIAL

To view supplementary material for this article, please visit http://dx.doi.org/10.1017/ice.2016.159

Footnotes

a

These authors contributed equally to this manuscript.

References

REFERENCES

1. Langmuir, AD. The surveillance of communicable diseases of national importance. N Engl J Med 1963;268:182–192.Google Scholar
2. Brusaferro, S, Regattin, L, Faruzzo, A, et al. Surveillance of hospital-acquired infections: a model for settings with resource constraints. Am J Infect Control 2006;34:362–366.Google Scholar
3. Leth, RA, Moller, JK. Surveillance of hospital-acquired infections based on electronic hospital registries. J Hosp Infect 2006;62:71–79.Google Scholar
4. Freeman, R, Moore, LS, Garcia Alvarez, L, Charlett, A, Holmes, A. Advances in electronic surveillance for healthcare-associated infections in the 21st century: a systematic review. J Hosp Infect 2013;84:106–119.Google Scholar
5. Streefkerk, RH, Borsboom, GJ, van der Hoeven, CP, Vos, MC, Verkooijen, RP, Verbrugh, HA. Evaluation of an algorithm for electronic surveillance of hospital-acquired infections yielding serial weekly point prevalence scores. Infect Control Hosp Epidemiol 2014;35:888–890.Google Scholar
6. Streefkerk, RH, Moorman, PW, Parlevliet, GA, et al. An automated algorithm to preselect patients to be assessed individually in point prevalence surveys for hospital-acquired infections in surgery. Infect Control Hosp Epidemiol 2014;35:886–887.Google Scholar
7. PREZIES PPS forms 2015. Rijksinstituut voor Volksgezondheid en Milieu website. http://www.rivm.nl/Onderwerpen/P/PREZIES/Formulieren. Published 2015. Accessed June 15, 2016.Google Scholar
8. PREZIES HAI definitions 2015. Rijksinstituut voor Volksgezondheid en Milieu website. http://www.rivm.nl/Onderwerpen/P/PREZIES/Prevalentieonderzoek_Ziekenhuizen/Definities. Updated 2016. Accessed June 15, 2016.Google Scholar
9. van Mourik, MS, Troelstra, A, van Solinge, WW, Moons, KG, Bonten, MJ. Automated surveillance for healthcare-associated infections: opportunities for improvement. Clin Infect Dis 2013;57:85–93.Google Scholar
10. Trick, WE. Decision making during healthcare-associated infection surveillance: a rationale for automation. Clin Infect Dis 2013;57:434–440.Google Scholar
11. Klompas, M. Interobserver variability in ventilator-associated pneumonia surveillance. Am J Infect Control 2010;38:237–239.Google Scholar
12. Keller, SC, Linkin, DR, Fishman, NO, Lautenbach, E. Variations in identification of healthcare-associated infections. Infect Control Hosp Epidemiol 2013;34:678–686.Google Scholar
13. Bolon, MK, Hooper, D, Stevenson, KB, et al. Improved surveillance for surgical site infections after orthopedic implantation procedures: extending applications for automated data. Clin Infect Dis 2009;48:1223–1229.Google Scholar
14. Choudhuri, JA, Pergamit, RF, Chan, JD, et al. An electronic catheter-associated urinary tract infection surveillance tool. Infect Control Hosp Epidemiol 2011;32:757–762.Google Scholar
15. Yokoe, DS, Khan, Y, Olsen, MA, et al. Enhanced surgical site infection surveillance following hysterectomy, vascular, and colorectal surgery. Infect Control Hosp Epidemiol 2012;33:768–773.Google Scholar
16. Woeltje, KF. Moving into the future: electronic surveillance for healthcare-associated infections. J Hosp Infect 2013;84:103–105.Google Scholar
Figure 0

FIGURE 1Workflow computer-assisted point prevalence survey for hospital-acquired infection.

Figure 1

TABLE 1A Summary of the Results of 6 Consecutive Point-Prevalence Surveys According to the Traditional Ward-Based Survey and CAPPS.a

Figure 2

FIGURE 2Prevalence of hospital acquired infection detected by 13 consecutive, weekly, hospital-wide, point-prevalence surveys using IDM-Surveillance software. Data are presented for all ward as a weekly, monthly, and quarterly scores, including 95% confidence intervals (upper panels), and for patients cared for by 2 medical specialties (lower panels).

Supplementary material: File

Streefkerk supplementary material

Supplementary Table

Download Streefkerk supplementary material(File)
File 18.1 KB