Introduction
Promoting adherence to evidence-based antibiotic treatment guidelines is recommended by the Centers for Disease Control and Prevention (CDC) Core Elements of Hospital Antibiotic Stewardship Programs and is a high priority for pediatric antimicrobial stewardship programs (ASPs). 1,Reference O’Leary, Neuhauser, McLees, Paek, Tappe and Srinivasan2 Community-acquired pneumonia (CAP) is among the most common indications for antibiotic treatment in children and is therefore a key target condition for ASPs to ensure optimal antibiotic use. Reference Gill, Anwar and Thavam3–5 Several reports have demonstrated the impact of prospective audit with feedback-based interventions on improving antibiotic use for CAP. Reference Avdic, Cushinotto and Hughes6–Reference Vaughn, Gandhi and Hofer10 However, these interventions have relied on manual review by ASP personnel, which is a resource-intensive approach that may not be feasible in all settings.
The use of electronic health record (EHR) data has the potential to efficiently identify children hospitalized with CAP and evaluate the appropriateness of antibiotic choice and duration; however, validated metrics of antibiotic appropriateness in pediatrics are lacking. National guidelines recommend ampicillin as first-line treatment for most children hospitalized with non-severe CAP, and an antibiotic treatment duration of five days is supported by several studies and recommended by the American Thoracic Society (ATS)/Infectious Diseases Society of America (IDSA) adult CAP guidelines. Reference Bradley, Byington and Shah11–Reference Metlay, Waterer and Long18 We sought to evaluate whether we could leverage data available in the EHR to validate metrics classifying the appropriateness of antibiotic choice and duration in children with CAP.
Methods
Study design
We performed a retrospective cross-sectional study to validate metrics of inappropriate antibiotic choice and duration in children with clinician-diagnosed CAP admitted between January 2019 and October 2022. This study was reviewed and approved as exempt research by the Institutional Review Board at the Children’s Hospital of Philadelphia (CHOP).
Study sites
This study was conducted within a pediatric health system that includes a 567-bed quaternary care hospital and a 108-bed community-based hospital. The CHOP care network also has 31 primary care practices, four urgent care centers, and two emergency departments. All CHOP sites use an integrated EHR (Epic Systems, Inc., Verona, WI, USA).
CAP study subjects
To identify children 6 months to 17 years being treated for clinician-diagnosed CAP, we first identified all encounters with an International Classification of Diseases version ten (ICD-10) code for viral or bacterial CAP who were discharged from an inpatient or observation encounter between January 2019 and October 2022 (Supplemental Table 1). Encounters with viral diagnosis codes were included to enhance generalizability, as these children may also be treated with antibiotics for clinician-diagnosed bacterial pneumonia and should therefore adhere to recommended antibiotic choice and duration recommendations. We additionally required that at least one dose of antibiotics be administered within the first 48 hours of admission and a second dose of antibiotics be administered at least 22 hours after the first, with the goal of excluding children treated with a single day of antibiotics whose antibiotics were ultimately discontinued. Patients discharged prior to meeting these criteria were included if they were discharged with an antibiotic prescription (Supplemental Table 2). We further limited the cohort to patients who had a chest X-ray or chest CT scan within the 48 hours before or after admission. Children transferred from hospitals outside of our health system to an inpatient unit were excluded to ensure capture of only community-onset infections.
To identify patients with non-severe pneumonia in whom standardized definitions of inappropriate antibiotic choice and duration could be applied using only data from the EHR, we also excluded those who died within 48 hours of admission and those with pediatric intensive care unit (PICU) admissions greater than 48 hours, complex chronic conditions (CCCs) (including severely immunocompromised children and those with tracheostomies), Reference Feudtner, Feinstein, Zhong, Hall and Dai19 complicating pulmonary conditions (including chronic bronchitis, bronchiectasis, emphysema) (Supplemental Table 3), and a length of stay greater than seven days. At our health system, PICU admission is required for patients requiring new noninvasive mechanical ventilation, including high flow nasal canula exceeding age-specific thresholds. Finally, we excluded those with ICD-10 codes for a concurrent bacterial infection, other than acute otitis media (AOM) and streptococcal pharyngitis (Supplemental Table 4).
Antibiotic appropriateness indicators
Appropriate antibiotic choice was defined based on the 2011 Pediatric Infectious Diseases Society/IDSA CAP Guidelines and assessed within the first 48 hours of hospitalization. Reference Bradley, Byington and Shah11 Amoxicillin or ampicillin were considered appropriate in all cases. For patients admitted to the PICU within the first 48 hours of their hospitalization and those with an ICD-10 code for complicated pneumonia (Supplemental Table 5), cefotaxime or ceftriaxone were additionally considered appropriate. For patients with documented β-lactam allergies or β-lactam antibiotic use within the 30 days, any antibiotic was classified as appropriate. Any drug with activity against methicillin-resistant Staphylococcus aureus (MRSA) was considered appropriate for patients who had a positive clinical or screening culture for MRSA in the preceding 1 year (including vancomycin [IV only], linezolid, clindamycin, trimethoprim-sulfamethoxazole, or ceftaroline). Azithromycin monotherapy was considered appropriate only in encounters in which a patient was at least 5 years old or in which there was positive Mycoplasma pneumoniae polymerase chain reaction (PCR) testing. Antibiotic choice was classified as appropriate if the patient received one or more of the antibiotics classified as appropriate, even if the patient may have received additional antibiotics beyond those listed (Table 1).
Measures of appropriate antibiotic choice in children hospitalized with community-acquired pneumonia
Abbreviations: MRSA, methicillin- resistant Staphylococcus aureus; PCR, polymerase chain reaction.
1 Antibiotic choice was classified as appropriate if the patient received one or more of the antibiotics classified as appropriate within the first 48 hours, even if the patient may have received additional antibiotics beyond those listed.
Antibiotic duration was defined as the total duration of both inpatient and discharge antibiotic treatment. Antibiotic durations of 6 days or less were considered appropriate in all cases. While five days is the recommended total antibiotic duration for CAP, the metric allowed for one additional day, given the imprecision of counting partial days of antibiotics and to avoid over-penalizing clinicians for these partial days. For patients with positive blood cultures, a positive respiratory culture for Pseudomonas aeruginosa or Staphylococcus aureus, any positive Legionella pneumophilia testing (PCR, sputum, or culture), concurrent otitis media, streptococcal pharyngitis, or a complicated pneumonia diagnosis code, any antibiotic duration was considered appropriate (Table 2).
Measures of appropriate antibiotic duration in children hospitalized with community-acquired pneumonia
Abbreviations: MRSA, methicillin- resistant Staphylococcus aureus; PCR, polymerase chain reaction; ICD-10, international classification of diseases version 10.
1 Prescriptions written for durations >30 days were assumed not to be for uncomplicated CAP. These prescriptions were presumed to be either treating a complicated pneumonia or prescribed for prophylaxis.
Data collection
We randomly selected 100 encounters for validation. Two physicians (K.C. and D.M.) performed manual chart review of clinical documentation (reference standard) and compared the findings from this manual chart review to EHR extracted data from the clinical data warehouse. In cases where the two physician reviewers did not agree, a third physician (J.S.G.) resolved discrepancies. To evaluate the accuracy of our electronic algorithm in identifying children treated for clinician-diagnosed CAP, we reviewed physician documentation at both admission and discharge to determine whether antibiotics were ordered for treatment of CAP. To evaluate the performance characteristics of the electronic indicators of appropriate antibiotic choice and duration, the classification based on the EHR extracted electronic data was compared to the classification obtained from the manual chart review.
Analysis
Binary variables were reported using numbers and proportions, and continuous variables were reported using medians and interquartile ranges. Statistical differences were assessed using chi-square tests for categorical variables and the Wilcoxon rank sum tests for continuous variables, with a P-value of <0.05 considered statistically significant. Using the assessments derived from the manual chart review as the reference standard, we assessed the sensitivity, specificity, positive predictive value, and negative predictive value of the electronic algorithm in identifying inappropriate antibiotic choice and duration.
Results
Cohort identification
There were 3641 unique encounters with an ICD-10 code for CAP among children ages 6 months to 17 years. Of these, 922 (25%) were excluded because they did not receive an antibiotic within 48 hours for at least two consecutive days or were discharged within the first 48 hours with no prescription for antibiotics. An additional 103 (3%) were excluded because they did not have a chest X-ray or CT scan within 48 hours of admission. Four hundred and thirteen (11%) were excluded because they were transferred from another institution. The final cohort with CAP, therefore, included 2203 children (Figure 1A). To identify the subset with non-severe, uncomplicated CAP in whom our electronic indicators of antibiotic appropriateness would apply, we then excluded patients who did not survive for at least 48 hours (n = 1, <1%) or had a PICU stay longer than 48 hours (n = 481, 22%), a CCC (n = 628, 29%), a complicating pulmonary condition (n = 5, <1%), a length of stay >7 days (n = 23, 1%), or a concurrent bacterial infection other than AOM or streptococcal pharyngitis (n = 7, <1%). The final cohort meeting the electronic definition for non-severe CAP was 1,058 children (Figure 1B).
A. Identification of children with community-acquired pneumonia, January 2019-October 2022. B. Identification of children with non-severe community-acquired pneumonia, January 2019–October 2022. Abbreviations: ICD-10, International Classification of Diseases version 10; CAP, community-acquired pneumonia; CT, computed tomography; ICU, intensive care unit; CCC, complex chronic condition; LOS, length of stay.
Clinical and demographic characteristics
The median age of children meeting the electronic definition of CAP was two years (interquartile range, 1–5 years). Fifteen percent were admitted to the PICU during the first 48 hours of admission, while 79% and 91% were admitted to and discharged from the general pediatrics service, respectively. Fourteen percent had a diagnosis code for viral pneumonia, and 15% had a concurrent diagnosis of AOM or streptococcal pharyngitis. Complicated pneumonia ICD-10 codes were present in <1% of children. The demographic and clinical characteristics were similar in the full cohort versus the validation cohort, with statistically significant differences noted only in the distribution of race and prevalence of β-lactam allergies (Table 3).
Baseline clinical and demographic characteristics of children with community-acquired pneumonia, January 2019–October 2022
Abbreviations: AOM, acute otitis media; GAS, Group A Streptococcus; ICU, intensive care unit.
Validation of CAP diagnosis
A total of 100 charts were reviewed by two physicians to evaluate whether the electronic definition of CAP identified children being treated for clinician-diagnosed CAP. Of these, 98 encounters (98%) were treated for CAP based on manual chart review. The two patients who were not being treated for CAP were both diagnosed with viral pneumonia and were receiving antibiotics for AOM only.
Appropriate antibiotic choice validation
On manual chart review, 99 encounters received the appropriate initial antibiotic choice, while 1 encounter received an inappropriate initial antibiotic choice. The electronic algorithm classified 93 encounters as receiving the appropriate initial antibiotic choice, all of which were classified as appropriate by manual chart review. The electronic algorithm classified 7 encounters as receiving an inappropriate initial antibiotic choice, including the 1 encounter classified as inappropriate based on manual chart review. Of the 6 patients classified as receiving the inappropriate initial choice based on the electronic definition but the appropriate initial antibiotic choice based on manual chart review, three patients had failed treatment with amoxicillin or amoxicillin-clavulanate prior to admission, but these antibiotics were prescribed outside of our health system and therefore were not detected by the electronic algorithm; two patients had concern for complicated pneumonia based on clinician documentation, without a corresponding ICD-10 code; and one patient did not have CAP and received a dose of ceftriaxone for AOM. The electronic algorithm, therefore, had a sensitivity of 100%, specificity of 94%, positive predictive value of 14%, and negative predictive value of 100% for detecting inappropriate antibiotic choice.
Appropriate antibiotic duration validation
Of the 100 encounters reviewed manually, 25 received the appropriate duration of antibiotics (25%), while 75 received an inappropriate duration of antibiotics (75%). The electronic algorithm classified 26 encounters as receiving the appropriate antibiotic duration, including three encounters that were classified as receiving the inappropriate duration of antibiotics based on a manual chart review. The three encounters classified as having an inappropriate antibiotic duration based on the manual chart review but an appropriate antibiotic duration by the electronic algorithm included two patients who had been started on antibiotics as outpatients for durations exceeding six days with no new prescription provided at discharge and one patient who had a billing code for AOM but no clinical diagnosis of AOM. A total of 74 encounters were classified as receiving an inappropriate antibiotic duration by the electronic algorithm, including two encounters classified as receiving an appropriate duration based on manual chart review. The two encounters classified as having an appropriate antibiotic duration on manual chart review but not by the electronic algorithm included a patient who was treated for AOM rather than CAP, and a patient treated for complicated pneumonia, neither of which had ICD-10 billing codes assigned for these diagnoses. The electronic algorithm, therefore, had a sensitivity of 97%, specificity of 88%, positive predictive value of 96%, and negative predictive value of 92% for inappropriate antibiotic duration (Table 4).
Performance characteristics of electronic algorithm, January 2019-October 2022
Discussion
We performed a cross-sectional study to validate indicators of inappropriate antibiotic choice and duration based on EHR data in a cohort of children with clinician-diagnosed CAP. The electronic indicators for inappropriate antibiotic choice and duration had acceptable performance characteristics when compared to a reference standard of manual chart review. This work demonstrates the feasibility of deriving measures of appropriate antibiotic choice and duration for CAP based solely on EHR data, as well as the accuracy of our electronic definition in identifying a low-risk cohort of children with clinician-diagnosed CAP in whom these indicators would be applicable. This approach may promote lower-maintenance ASP activities focused on tracking clinician adherence to evidence-based guidelines for treating CAP, thereby facilitating improvement efforts as well as adherence to Joint Commission accreditation standards. Reference Chiotos, Dutcher and Grundmeier20,21
ASPs have historically relied on aggregate measures of antibiotic use, such as days of therapy per 1000 patient days or days present, which lack specificity in assessing antibiotic appropriateness, or medication use evaluations, which require time-intensive chart review. We instead focused on identifying a cohort of children with CAP in whom antibiotic appropriateness could be accurately classified using electronic data alone. Consistent with other reports, we found that antibiotic duration exceeded guideline recommendations in the majority of children treated for CAP. Reference Vaughn, Gandhi and Hofer10,Reference McCarthy, Baggs and Wolford22,Reference Madaras-Kelly, Burk and Caplinger23 Our electronic measure was highly accurate in identifying inappropriate antibiotic duration with a PPV of 96% and NPV of 92%. Reference Vaughn, Gandhi and Hofer10,Reference McCarthy, Baggs and Wolford22,Reference Madaras-Kelly, Burk and Caplinger23 While the PPV of the antibiotic choice measure was very low, this was in part driven by the low prevalence of inappropriate antibiotic choice. ASPs may therefore choose to manually review the small number of encounters labeled inappropriate by the electronic indicator, as the two scenarios in which the electronic measure misclassified antibiotic choice are scenarios not readily identifiable in structured EHR data, including (1) prior use of antibiotics outside of our health system and (2) clinician concern for complicated pneumonia.
Several prior studies have attempted to develop approaches to evaluate appropriate antibiotic use using natural language processing (NLP), billing codes, clinical data, or a combination of these approaches. Reference Jones, Amoah and Klein24–Reference Williams, Shah and Myers28 For example, Yarahuan and colleagues derived an algorithm to evaluate adherence to a local clinical practice guideline among children with CAP using structured EHR data, followed by regular expressions. Reference Yarahuan, Kisvarday and Kim27 Consistent with other studies, NLP enhanced the performance of the electronic algorithm by specifically improving identification of children with complicated pneumonia and those with missing outpatient antibiotic use data, both of which negatively impacted the positive predictive value of our antibiotic choice metric. Reference Jones, Amoah and Klein24,Reference Jones, South and Shao25,Reference Yarahuan, Kisvarday and Kim27 However, NLP was only able to classify an additional 17% of patients as guideline-concordant or non-concordant beyond those classified using structured data, with the remainder requiring manual review. Reference Yarahuan, Kisvarday and Kim27 While this certainly represents an incremental improvement, NLP also introduces significant additional complexity and requires specific expertise that may not be available everywhere. Given these considerations, our electronic algorithm would be most useful for stewardship programs that seek a pragmatic approach to performing syndrome-based stewardship in common clinical scenarios.
For example, after completing this validation, we leveraged this electronic algorithm to create clinician feedback reports, including the monthly rate of appropriate antibiotic choice and duration among CAP patients cared for by our general pediatrics service. Implementation of these reports was associated with a 32% increase in the proportion of CAP encounters receiving the recommended antibiotic choice and duration. Reference Chiotos, Dutcher and Grundmeier20 This demonstrates the feasibility of EHR-based approaches to facilitate stewardship efforts, which may be particularly beneficial for programs with limited personnel.
There are several limitations to our study. First, there is no “gold standard” for diagnosing bacterial CAP. Because the goal of deriving our electronic definition for CAP was to assess the appropriateness of antibiotic choice and duration, we focused on identifying children whom clinicians were treating for CAP, acknowledging that some children may not require any antibiotics at all. Second, we did not perform a formal validation of our electronic definition of CAP; instead, we ensured that it identified a cohort of children with clinician-diagnosed CAP for whom our electronic indicators of antibiotic choice and duration would apply. Third, we were intentionally permissive in assessing antibiotic appropriateness in this proof-of-concept approach of EHR-based syndromic stewardship efforts to promote scalability and acceptability of the metrics. In future iterations, more stringent indicators of appropriateness incorporating measures of overly broad or use of multiple antimicrobials would be beneficial in facilitating robust stewardship efforts. Fourth, while our indicators of appropriate antibiotic use account for antibiotics administered prior to admission within our health system, we were unable to comprehensively evaluate prior antibiotic use due to the limitations of electronic data, which contributed to the lower PPV for the antibiotic choice metric. However, at our center and in most published studies, the greatest opportunity for improvement in antibiotic use for CAP was in optimizing antibiotic duration, so ASPs may wish to either prioritize implementation of the antibiotic duration metric or use the antibiotic choice metric as a means to identify encounters warranting manual review, particularly if adherence to the antibiotic choice metric was lower than was observed in our system. Fifth, we excluded approximately 50% of patients with CAP because they had chronic medical conditions or required a PICU stay exceeding 48 hours. This was necessary to ensure that we could reliably evaluate antibiotic appropriateness using only EHR data. Nevertheless, because these populations make up an increasing proportion of hospitalized children, tools to improve stewardship in these populations should be prioritized in future efforts. Reference Killien, Keller, Watson and Hartman29,Reference Burns, Casey, Lyle, Bird, Fussell and Robbins30 Finally, our validation was performed at a single health system, and external validation of both the cohort definition and the measures of appropriate antibiotic use are necessary, given that there are likely differences in coding, case mix, and antibiotic use patterns across settings and clinician types.
Overall, we demonstrated that an electronic definition of CAP based solely on EHR data was highly accurate in identifying children treated for CAP at our center, and that electronic indicators of antibiotic choice and duration performed well when compared to a reference standard of manual chart review. Future studies should focus on refining and validating these measures in a multicenter fashion, as well as evaluating their implementation for stewardship operations and interventions.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/ice.2025.10314
Financial support
This work was supported by the U.S. Centers for Disease Control and Prevention (CDC) Safety and Healthcare Epidemiology Prevention Research Development (SHEPheRD) 2017 Domain 1: A-002, “Assessing Appropriate Antibiotic Use in Hospitals and Outpatient Settings Through Electronic Health Records” and 2021 Domain 1: A-009, “Using Electronically Derived Automated Reports of Appropriate Antibiotic use to Inform Stewardship Interventions.”
Competing interests
None.
Disclaimer
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the CDC.
