Central-line–associated bloodstream infections (CLABSIs) are bloodstream infections associated with a present (or recently present) central venous access device (CVAD) in the absence of infection at another site. 1 This type of hospital-acquired infection (HAI) carries significant financial burden for health services (ie, staff time and treatment costs) and often disincentive penalties.Reference Pakyz and Edmond 2 In the United States of America alone, CLABSI is estimated to cost $1.9 billion annually.Reference Zimlichman, Henderson and Tamir 3 Mortality among patients diagnosed with CLABSI is also high, with extra attributable mortality estimated between 14% and 16%.Reference Kaye, Marchaim and Chen 4 , Reference Hu, Tao and Rosenthal 5 A large proportion of these infections are considered preventable (65%–70%)Reference Umscheid, Mitchell and Doshi 6 and as a result, quality improvement activities and implementation of national guidelines aimed at prevention have become core business for healthcare institutions.Reference O’grady, Alexander and Burns 7 Continuous and reliable surveillance of CLABSI is essential; any underestimation (false negatives) or overestimation (false positives) of CLABSI are problematic because they impair the ability of the healthcare facility to set targets and benchmark healthcare quality between similar facilities; to monitor the efficacy of quality improvement strategies; to measure trends over time; and/or to quickly identify outbreaks.Reference DiGiorgio, Fatica and Oden 8 , Reference Venable and Dissanaike 9
The National Healthcare Surveillance Network (NHSN), in partnership with the Centers for Disease Control (CDC), have established definitions that are used to classify CLABSI. 1 These definitions are considered gold standard and have been adopted in whole (eg, AustraliaReference Worth, Daley and Spelman 10 ) or in part (eg, European CDC 11 ) by central reporting agencies worldwide. CLABSI classification with this definition is the responsibility of a variety of healthcare professionals including infection preventionists, infectious disease physicians, or specialist nurses.Reference Lin, Hota and Khan 12 The number of staff, facility resources, and time allocated for this duty often depend upon the size, budget, and structure of the healthcare facilities.Reference Venable and Dissanaike 9 Validity and reliability of these data is essential. However, despite standardized methods and definitions for reporting, application in practice is not consistent.Reference Lin, Hota and Khan 12 This inconsistency may be a result of subjective clinician selective reporting (ie, ‘overruling’), which is widely discouraged;Reference Talbot, Bratzler and Carrico 13 uncertainty; inadequate education or unfamiliarity with the NHSN/CDC criteria;Reference Lin and Marc 14 insufficient resources; and/or lack of external validation.Reference Lin, Hota and Khan 12 , Reference Wright, Allen-Bridson and Hebden 15
One of the first retrospective cohort studies to explore the reliability of HAI reporting was conducted by Emori et alReference Emori, Edwards and Culver 16 in 1998. Their findings highlighted inconsistencies of reported bloodstream, surgical, urinary catheter, and respiratory infections.Reference Emori, Edwards and Culver 16 This seminal work prompted other researchers and clinicians to report on inconsistencies, specifically related to CLABSI, within their own institutions.Reference Backman, Melchreit and Rodriguez 17 , Reference Thompson, Makvandi and Baumbach 18 In this systematic review, we evaluated how the application of the NHSN/CDC CLABSI definitions varies between hospital infection control teams and/or preventionists, compared with expert adjudicators with advanced knowledge of the definitions and their application. The findings of this review synthesize known levels of specificity and sensitivity for the NHSN/CDC CLABSI criterion and provide an estimated margin of error in CLABSI reporting for institutions not actively validating local data.
Methods
We conducted a diagnostic-test accuracy systematic review to explore previously published literature exploring reliability of CLABSI reporting. Methods and outcomes of interest were prospectively registered (June 7, 2015), with the International Prospective Register of Systematic Reviews (PROSPERO; ID CRD42015021989).
Search strategy
A systematic search of literature was conducted April 17, 2018, using the Cochrane Library; MEDLINE (Ovid), CINAHL (EbscoHost), PubMed (NCBI), and SCOPUS (Elsevier) with MESH terms. We used the following key search terms: ‘Catheterisation, Central Venous’ (MESH); ‘central line’; ‘central venous access device’; ‘vascular access’; ‘Bacteremia’ (MESH); ‘Catheter-Related Infections’ (MESH); ‘Sepsis’ (MESH); ‘central line-associated bloodstream infections’; ‘Diagnostic Errors’ (MESH); ‘Reproducibility of Results’ (MESH); ‘Validation Studies’ (MESH); ‘interrater’; ‘variation’; ‘reliability’; ‘test–retest’; ‘Centers for Disease Control and Prevention’ (MESH); ‘National Health and Safety Network’; ‘surveillance system’; and ‘infection surveillance.’
Inclusion and exclusion criteria
Cohort studies from January 2008 that compared publicly reported CLABSI rates using NHSN/CDC criteria (reference test), with the CLABSI rates determined by blinded, independent, and formally trained expert adjudicators (index test) were eligible for inclusion. Tertiary facilities using surveillance definitions reproducing NHSN/CDC criteria (ie, the current version at the time of the study) were also eligible for inclusion. There were no age limits or facility- or discipline-related exclusions. Studies were eligible if they had selected a whole cohort or randomly selected cohort of patients over a pre-established period with CVADs. Grey literature and studies reported in a language other than English were excluded. Furthermore, studies that used either a program or algorithm (electronic) or example scenarios (vignettes) for CLABSI diagnosis were excluded.
Data extraction
Data from eligible studies were extracted separately by 2 independent reviewers using a purpose-built data extraction instrument established by the investigators (E.L. and N.G.). Disagreements and/or discrepancies identified between the 2 independent investigators were resolved by a third adjudicator (J.W.), who informed the final decision. True positives, true negatives, false positives and false negatives, both (1) publicly reported (reference test) and (2) classified by the independent reviewer team (index test, ie, ‘gold standard’) were extracted and compared for congruence. Sensitivity, specificity, positive predictive values (PPVs) and/or negative predictive values (NPVs) with 95% confidence intervals (CIs) were calculated using SPSS software (IBM, Armonk, NY) if the authors did not report these values. Accuracy and incidence of positive and negative cases were extracted; however, because these measures should only be used in cases where all cases (both positive and negative for CLABSI) were represented in the data,Reference Baratloo, Hosseini, Negida and Ashal 19 they were excluded for studies including a random sample only.
Two investigators (E.L. and N.G.) conducted a risk of bias (quality) assessment of each included study, using the Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS 2) tool.Reference Whiting, Rutjes and Westwood 20 The investigators conducted the assessment independently and were blinded to each other’s judgments. In disagreements, a third adjudicator (J.W.) was consulted for a final decision.
Statistical analysis
Cohort studies eligible for inclusion were presented using descriptive statistics. Forest plots were constructed to display sensitivity and specificity with 95% CI. A summary receiver operating characteristic (SROC) curve was plotted to demonstrate overall accuracy of CLABSI classification.
Results
The systematic search yielded 1,259 titles and abstracts. References lists were hand-searched for titles missing from the original search. Duplicates were removed, and studies were initially excluded by title, followed by an assessment of the remaining abstracts and (if indicated) full texts (Fig. 1).
PRISMA flow diagram.
In total, 9 studies, including 7,093 participants (with 7,160 central lines), met the inclusion criteria and were included in the review. All studies included data from adult intensive care units (ICUs), and 3 reports included additional data from pediatric ICUs.Reference Backman, Melchreit and Rodriguez 17 , Reference Hazamy, Van Antwerpen and Tserenpuntsag 21 , Reference Rich, Reese, Bol, Gilmartin and Janosz 22 A single study also included data from a long-stay acute-care unit.Reference Rich, Reese, Bol, Gilmartin and Janosz 22 Most studies (n = 5) were conducted in the United States using NHSN ReportingReference Backman, Melchreit and Rodriguez 17 , Reference Thompson, Makvandi and Baumbach 18 , Reference Hazamy, Van Antwerpen and Tserenpuntsag 21 – Reference Oh, Cunningham and Beldavs 23 ; the remaining studies were conducted in Australia,Reference McBryde, Brett and Russo 24 Canada,Reference Fontela, Rocher and Platt 25 Korea,Reference Kwak, Choi and Yoo 26 and Spain,Reference López-Pueyo, Olaechea-Astigarraga and Palomar-Martinez 27 each with their own central reporting systems replicating NHSN/CDC criterion. Four studies included a consecutive series of patientsReference Backman, Melchreit and Rodriguez 17 , Reference Thompson, Makvandi and Baumbach 18 , Reference Hazamy, Van Antwerpen and Tserenpuntsag 21 , Reference Kwak, Choi and Yoo 26 ; the remaining studies included a random sample, either stratifiedReference Rich, Reese, Bol, Gilmartin and Janosz 22 , Reference Fontela, Rocher and Platt 25 , Reference López-Pueyo, Olaechea-Astigarraga and Palomar-Martinez 27 or unstratified and/or unclear.Reference Oh, Cunningham and Beldavs 23 , Reference McBryde, Brett and Russo 24 Table 1 outlines characteristics of each included study, as well as sensitivity, specificity, PPV and NPV, incidence and accuracy results.
Characteristics of Included Studies and Findings
Note. CI, confidence interval; NPV, negative predictive value; PPV, positive predictive value; ICU, intensive care unit; SPIN-BACC, Surveillance Provinciale des Infections Nosocomiales; KONIS, Korean Nosocomial Infections Surveillance System; ENVIN-HELICS, Estudio Nacional de Vigilancia de Infección Nosocomial en Uci; VICNISS: Victorian Nosocomial Infection Surveillance System; NHSN, National Health Safety Network.
a Reported sample and results in devices (not patients).
b 14 records not found.
Specificity (true negative rate) results were high, ranging from 0.70 (95% CI, 0.58–0.81) to 0.99 (95% CI, 0.99–1.00). In most cases, publicly reported CLABSI rates did not include patients who experienced a true case of CLABSI. Sensitivity (true positive rate) results trended lower, ranging from 0.42 (95% CI, 0.15–0.72) to 0.88 (95% CI, 0.77–0.95); patients were more likely to be incorrectly classified as having a non-CLABSI event. In all included studies, sensitivity was lower than specificity (Fig. 2). The SROC curve (Fig. 3) demonstrates overall accuracy of CLABSI classification.
Forest plot, sensitivity and specificity.
SROC.
Accuracy measures from the 4 studies that included data from an entire cohort (n = 4,109 devices) demonstrated an overall probability of accurate classification between 91.9% (95% CI, 85.6–96.0) and 94.6% (95% CI, 91.6–96.6). Furthermore, within these 4 studies, the proportion of patients experiencing true CLABSI was underreported by 3.3%–4.4%.Reference Backman, Melchreit and Rodriguez 17 , Reference Thompson, Makvandi and Baumbach 18 , Reference Hazamy, Van Antwerpen and Tserenpuntsag 21 , Reference Kwak, Choi and Yoo 26 A single study reported this difference per 1,000 catheter days (1.97 vs 3.51 per 1,000 catheter days).Reference Backman, Melchreit and Rodriguez 17
The overall quality of the included studies was high and applicability concerns were consistently low (Table 2). The risk of bias was unclear to high in relation to patient selection for 2 of the 9 (22%) included studiesReference Hazamy, Van Antwerpen and Tserenpuntsag 21 , Reference Rich, Reese, Bol, Gilmartin and Janosz 22 ; the index test had an unclear to high risk of bias in 3 studies,Reference Hazamy, Van Antwerpen and Tserenpuntsag 21 , Reference Kwak, Choi and Yoo 26 , Reference López-Pueyo, Olaechea-Astigarraga and Palomar-Martinez 27 and a further 2 studies demonstrated a high risk of bias related to flow and/or timing.Reference Rich, Reese, Bol, Gilmartin and Janosz 22 , Reference López-Pueyo, Olaechea-Astigarraga and Palomar-Martinez 27 Table 2 displays the risk of bias and applicability concerns for each of the 9 studies using the QUADAS 2 tool.
Quality Assessment (QUADAS 2)
Discussion
The findings of this systematic review demonstrate consistent underestimation of the true CLABSI incidence in publicly reported rates. This underestimation may have widespread effects. Hospital staff, epidemiologists, and researchers may be using inaccurate data to inform benchmarking, to identify outbreaks, and to prioritize resource allocation (eg, access to electronic medical records and staff for data collection, and classification).Reference Beekmann, Diekema and Huskins 28 The ‘gold standard’ NHSN/CDC definitions are intended to enable standardized classification and diagnosis; however, despite supplemental CDC educational documents and available online web-based training,Reference Wright, Allen-Bridson and Hebden 15 these definitions are sometimes misinterpreted and/or inappropriately applied. Concern exists that underreporting may be intentional (1) to avoid financial penaltiesReference Sexton, Chen, Moehring, Thacker and Anderson 29 or (2) to demonstrate ‘zero CLABSI’ to improve the perceived quality and safety of care of the reporting facility.Reference Weeks, Goeschel, Cosgrove, Romig and Berenholtz 30 Observational studies comparing interobserver variability between 2 independent infection preventionists (not against publicly reported rates, according to our approach) found similar classification disparities.Reference DiGiorgio, Vinski and Bertin 31 , Reference Mayer, Greene and Howell 32 This finding supports the argument that variability in classification errors may be related to clinician understanding/application variations rather than deliberate omissions.Reference Lin and Marc 14 , Reference Sexton, Chen, Moehring, Thacker and Anderson 29 Moreover, ongoing changes to NHSN/CDC HAI definitions, most notably the 2013 addition of the mucosal-barrier injury related BSI (MBI-LCBI) classification,Reference Steinberg, Robichaux, Tejedor, Reyes and Jacob 33 while necessary, may create further confusion. This concern is not only limited to CLABSI however, as comparisons with interobserver agreement of other varieties of HAI (eg, ventilator-associated pneumonia) have similarly found poor concordance of classification.Reference Klompas 34
Clearly, validation of CLABSI rates is essential moving forward for accuracy of data. Sexton et alReference Sexton, Chen, Moehring, Thacker and Anderson 29 suggest that cyclic or random audits of facilities reporting either high or low CLABSI (based on local benchmarking) may improve quality of these measures. However, validation of large cohort datasets requires significant time and resources. Consequently, limitations such as these may necessitate the use of prospective surveys or vignettes among clinicians applying CLABSI classifications to explore the variability of definition application. This method has been executed in various facilities applying NHSN/CDC CLABSI definitions, and we were able to highlight concordance/disagreement in classification application, as well as the common source of these variations.Reference Beekmann, Diekema and Huskins 28 , Reference Keller, Linkin, Fishman and Lautenbach 35 Although this method is a more feasible a systemwide assessment of CLABSI accuracy, a whole or randomly selected cohort of patients should be used to assess accuracy of data reported which is likely to influence changes in clinical practice (eg, trial interventions and quality improvement activities). This more rigorous method of validation is required because randomized controlled trials and/or other studies often aim to reduce the incidence of primary outcomes, such as CLABSI, by margins of <5%.Reference Zacharioudakis, Zervou and Arvanitis 36 Small margins such as this may be impacted significantly by misclassification, particularly when, as we have identified, CLABSI may be underreported by margins as high as 4.4%.
Other methods for systemwide CLABSI reporting include the use of automated electronic surveillance systems, which has been proposed as a means to reduce this risk of unintentional misclassification and improve efficiency.Reference Venable and Dissanaike 9 These systems are reportedly able to determine the presence of HAI (including CLABSI) using data sourced from microbiological reports, with or without considering patient symptoms and/or other criteria required to fit the NHSN/CDC definitions.Reference Venable and Dissanaike 9 The advantages of this process may include reduced staff expenditure; surveillance of all admitted patients; early identification of outbreaks; and reduced susceptibility of classification subjectivity. However, their use has yet to be perfected.Reference Venable and Dissanaike 9 A systematic review of HAI electronic surveillance identified 2 studies that assessed the accuracy of CLABSI (NHSN/CDC); which ranged from 94.3% to 95.2% for sensitivity and from 68.0% to 97.5% for specificity.Reference Freeman, Moore, Alvarez, Charlett and Holmes 37 A further 8 studies assessing the accuracy of reporting for BSI (other classification, non-NHSN/CDC) reported sensitivity ranging from 72.0% to 100% and specificity ranging from 37.3% to 100%.Reference Freeman, Moore, Alvarez, Charlett and Holmes 37 Although this finding demonstrates a higher sensitivity rate, there was a lower specificity compared with the findings of this systematic review. Therefore, in contrast to manually classified rates, these electronic surveillance methods may overestimate the incidence of CLABSI. The effects of overestimation may be equal to the those of underestimation because it would similarly influence benchmarking, outbreak identification, and resource allocation.Reference Beekmann, Diekema and Huskins 28
Accuracy and reliability of reported CLABSI rates are not only reliant upon objective classification but also on the reported denominator.Reference Wright, Fisher and John 38 CLABSI incidence is commonly reported against 3 key denominators: device/central line days; patient days; and (less commonly) neutropenic days.Reference Backman, Nobert, Melchreit, Fekieta and Dembry 39 Standardization and validation of these denominators is essential because variation (eg, inclusion/exclusion of admission/discharge days; or differences in accounting for coexisting catheters) may influence a measurable difference in reported rates.Reference Backman, Nobert, Melchreit, Fekieta and Dembry 39
Accurate CLABSI reporting is essential to facilitate strategic benchmarking between institutions and evaluate the success of interventions aimed at reducing the incidence of HAIs. This systematic review has highlighted the likely underestimation of CLABSI in publicly reported rates, which contrasts to the likely overestimation of electronically classified CLABSI rates previously reported in literature, and we have further identified a need to validate local data, particularly that which is likely to lead practice change.
Our study has several limitations. The findings of this diagnostic-test accuracy systematic review are limited by the homogeneity of the data presented; most studies were undertaken within the United States, and 8 of 9 studies presented data from ICU only. Moreover, most studies (5 of 9) reported randomly selected patient sets rather than entire cohorts; therefore, incidence and accuracy measures were only calculated for the remaining 4 studies. Additionally, the sensitivity and positive predictive values for most studies reported wide CI margins, compared with the CI margins for specificity and negative predictive values, therefore introducing some uncertainty to the results. Results may also have been impacted by NHSN/CDC CLABSI definitions, which continue to change over time (eg, the 2013 inclusion of MBI-LCBI).Reference Steinberg, Robichaux, Tejedor, Reyes and Jacob 33
Although other systematic reviews have compared electronicReference Freeman, Moore, Alvarez, Charlett and Holmes 37 and administrative code dataReference van Mourik, van Duijn, Moons, Bonten and Lee 40 regarding CLABSI classification accuracy, this systematic review is the first to explore the concordance of CLABSI classification between experts and clinicians entering publicly reportable data, which more accurately reflects current reporting practices.Reference Hazamy, Van Antwerpen and Tserenpuntsag 21
Acknowledgments
The authors would like to acknowledge the support of the Nursing and Midwifery Research Centre, Royal Brisbane and Women’s Hospital.
Financial support
No financial support was provided relevant to this article.
Conflicts of interest
All authors report no conflicts of interest relevant to the findings of this systematic review.
