Discussion

In a detailed chart review analysis of an electronic surveillance definition of NV-HAP using clinical data in a large healthcare system, we found moderate correlation between an electronic NV-HAP definition and existing manual surveillance criteria. The PPV was as high as 82% using the most permissive definition (NV-HAP according to either CDC-NSHN criteria, reviewer assessment, treating clinician diagnosis, or discharge summary diagnosis according to at least 1 reviewer) but as low as 42% using the strictest definition (both reviewers agreed that CDC-NHSN criteria were met). In contrast, a claims-based strategy to identify NV-HAP using diagnosis codes detected <10% of patients flagged by the electronic surveillance definition and correlated poorly with other definitions. The variable PPV of the electronic surveillance criteria mirrors the high rates of reviewer variability that we found in all strategies for identifying clinical diagnoses of NV-HAP. Agreement levels between reviewers were moderate regardless of whether reviewers were applying formal CDC-NHSN criteria (κ = 0.50), assessing whether bedside clinicians diagnosed NV-HAP (κ = 0.64), or whether the discharge summary documented pneumonia (κ = 0.72). These findings underscore the complexity and subjectivity of NV-HAP diagnosis and surveillance using manual chart review, even when trained reviewers apply formal criteria.

The moderate accuracy and reviewer variability that we detected for the electronic criteria is similar to that of other definitions used to identify hospital-acquired pneumonia, including facility reporting, diagnosis codes, and medical record review. In a retrospective chart review by See et al,^{Reference See, Chang and Gualandi19} CDC medical epidemiologists independently reviewed 250 cases reported to the CDC-NHSN with pneumonia or lower respiratory tract infection and found that 8% of reported adult pneumonia cases did not meet CDC-NHSN criteria for NV-HAP and that 15% lacked clinician diagnoses. Similarly, Wolfensberger et al^{Reference Wolfensberger, Meier, Kuster, Mehra, Meier and Sax20} found that the PPV of ICD codes for NV-HAP was 35% and sensitivity was 59% compared to validated surveillance definitions. A systematic review summarizing the accuracy of diagnosis codes for NV-HAP reported similar performance, with sensitivity and specificity of 40% compared to clinical review.^{Reference van Mourik, van Duijn, Moons, Bonten and Lee5}

To address problems with diagnosis codes and manual evaluation of medical records, others have begun to develop approaches that either augment or replace these approaches. Wolfensberger et al^{Reference Wolfensberger, Jakob and Faes Hesse21} validated a semi-automated surveillance system for NV-HAP. By using an EHR-based surveillance definition to identify patients at-risk for NV-HAP, they were able to rule out NV-HAP in 94% of patients and significantly reduce the workload of manual review with high sensitivity and negative predictive value (NPV).^{Reference Wolfensberger, Jakob and Faes Hesse21} Similar to our study, Ramirez Battle et al^{Reference Ramirez Batlle, Klompas and Program7} found similar accuracy of electronic surveillance criteria for NV-HAP and CDC-NHSN criteria relative to expert chart review at a single center among 120 cases with oxygen deterioration. The electronic surveillance definition demonstrated sensitivity of 71%, PPV of 48%, and NPV of 90%. The CDC-NHSN definition demonstrated sensitivity of 61%, PPV of 59%, and NPV of 88%.^{Reference Ramirez Batlle, Klompas and Program7} These findings raise the possibility that EHR-based surveillance strategies could improve reproducibility and efficiency without dramatically reducing accuracy.

We detected substantial reviewer variability, despite adhering to a rigorous training process and formal consensus guide. This finding mirrors previous observations of low reliability of human assessment of pneumonia in hospital-acquired pneumonia.^{Reference Klompas8,Reference Kerlin, Trick and Anderson10,Reference Naidech, Liebling, Duran, Moore, Wunderink and Zembower22} Klompas et al^{Reference Klompas8} reported 62% agreement with a κ = 0.40 among 3 infection control personnel using CDC criteria for the identification of VAP. Kerlin et al^{Reference Kerlin, Trick and Anderson10} reported interreviewer agreement of 66%–83%, with a κ of 0.12 among infection preventionists and a κ of 0.34 among intensivists assessing VAP. In the same vein, humans demonstrate substantial variability in identifying pneumonia by chest imaging, both among reviewers interpreting reports and radiologists evaluating images.^{Reference See, Chang and Gualandi19,Reference Melbye and Dale23} Human review has historically been considered the gold standard for case detection, but our study adds to the growing evidence suggesting that it may not be an ideal form of measurement.^{Reference Vassar and Holzmann24} High levels of disagreement between reviewers despite using a common framework to apply agreed-upon definitions demonstrate the subjectivity of pneumonia diagnosis and the difficulty that human reviewers have applying complex definitions in a consistent fashion. This finding supports the development of surveillance approaches that are independent of human review to increase consistency, reduce burden, and ensure scalability.^{Reference Schreiber, Krauss, Blake, Boone and Almonte25}

Our study had several limitations. The small sample size of charts reviewed may not be truly representative of the variability of the population. Additionally, by restricting our analysis to cases meeting the electronic NV-HAP surveillance definition, we did not assess its sensitivity, which could have been affected by missing data. To provide a reliable measure that is amenable to large-scale examination of system-wide quality improvement interventions, electronic surveillance requires high-quality and stable clinical data that are routinely collected and entered without variation across settings or time. The criteria require physical signs of pneumonia (oxygenation, WBC count, temperature) as well as clinical recognition and responses to those physical signs (antibiotic use and chest imaging). Thus, variation in diagnosis and treatment patterns across settings or time could also influence the surveillance measure. Although we have previously validated several of the data elements used for the surveillance criteria,^{Reference Jones, Haroldsen and Madaras-Kelly26} continuous validation and analysis of variation in the quality of detailed clinical data across settings, systems, and time is essential before facility comparisons or intervention tracking can be pursued. Finally, our estimates of the accuracy of surveillance strategies are limited by the challenges of identifying a reference standard of “true” pneumonia. Although it does not entirely circumvent these challenges, electronic surveillance that does not rely upon diagnostic labels increases reproducibility and efficiency with accuracy that appears consistent with other approaches.

Our findings have important implications for clinical care and public health. NV-HAP is one of the most common and morbid hospital-acquired infections.^{Reference Giuliano, Baker and Quinn1,Reference Baker and Quinn2,Reference Magill, O’Leary and Janelle27} Robust surveillance and prevention programs are needed, but robust prevention programs need robust surveillance programs to measure and inform progress.^{Reference Kazaure, Martin, Yoon and Wren28–Reference Lacerna, Patey and Block32} We cannot improve what we cannot measure, and measurement must be timely and consistent as well as accurate. Implementation of systemwide surveillance and prevention efforts are limited by the poor reliability and validity of current approaches, a reflection of the highly variable and often subjective clinical diagnosis of pneumonia. Electronic surveillance has the potential advantage of being more reproducible and more amenable to large-scale examination of systemwide quality-improvement interventions. Our analysis highlights the ongoing challenge with accurately identifying pneumonia but also suggests a potential strategy to increase the scale, efficiency, and reliability of surveillance. No surveillance approach for NV-HAP is perfect. However, applying clinical criteria using data that are routinely entered into the EHR may provide a practical means to characterize the frequency and morbidity of NV-HAP, catalyze prevention programs, and reliably measure impacts on NV-HAP rates and outcomes.