Methods

Participating practices

Three pediatric primary care clinics (A, B, and C) and three emergency departments (D, E, and F) were included in the study (Figure 1). These sites consisted of all the non-main campus primary care practices and emergency departments affiliated with Vanderbilt University Medical Center at the time of the study start. The primary care clinics were each staffed by 4–6 prescribers, all but 2 of which were physicians and averaged between 600 and 1,236 encounters per month in the baseline period. The emergency departments were staffed by a rotating team of adult emergency medicine physicians and advanced practice providers (APPs) and averaged between 339 and 401 pediatric encounters per month in the baseline period. The project sites are connected to a central electronic health record (EHR, Epic, Verona, WI) (Figure 1). A Tableau dashboard used for routine antimicrobial stewardship monitoring tracks EHR data at both the clinic and prescriber level and was used to track antibiotic prescriptions, indications, and use of diagnostic tests. No prior stewardship educational efforts had taken place at these sites.

Figure 1.

Project sites. Figure illustrates the counties in which the primary care clinics and emergency departments involved in the study were located throughout the state of Tennessee. Home academic site from which the tele-stewardship interventions were carried out is marked as well.

Study period and outcomes

We evaluated all encounters with patients less than 18 years of age at participating sites.

The baseline period was January 2022–December 2022, and the intervention period was February 2023–December 2024, with a washout period in January 2023 that was not included in the analysis. Antibiotic indications were extracted from a required question field at the time of antibiotic order entry which went live in May 2022, and we previously demonstrated highly concordance with documentation in the patient’s chart. ^{Reference Oertli, Staub, Zhang and Katz7} We used ICD-10 codes associated with the order encounter in which an antibiotic was prescribed to extract indication for baseline data prior to May 2022. Concordance with diagnosis-specific national (e.g. acute otitis media (AOM), ^{Reference Lieberthal, Carroll and Chonmaitree8} streptococcal pharyngitis, ^{Reference Shulman, Bisno and Clegg9} community-acquired pneumonia (CAP), ^{Reference Bradley, Byington and Shah10} sinusitis, ^{Reference Wald, Applegate and Bordley11} ) and institutional (e.g. urinary tract infection (UTI)) guidelines was assessed for each antibiotic prescription. Concordant antibiotic choice and duration were evaluated separately, as defined in Figure 2. The study’s primary outcome was the percent of all encounters that resulted in an antibiotic prescription. Secondary outcomes included the percent of encounters resulting in a guideline-concordant antibiotic choice for AOM, streptococcal pharyngitis, CAP, sinusitis, and UTI; percent of encounters resulting in a guideline-concordant antibiotic duration for AOM, sinusitis, and CAP; percent of prescriptions with a 5-, 7- or 10-day duration (excluding azithromycin prescriptions); and percent of encounters resulting in a rapid streptococcal pharyngitis antigen or polymerase chain reaction test. Percent of encounters with a return visit anywhere within the healthcare system in the two weeks following the initial encounter was tracked over the course of the study as a balancing measure.

Figure 2.

Percent change in outcome. Panel A shows percent change in overall prescription rate by site, with sites that showed a significant improvement marked with an asterisk. Panel B shows percent change in guideline concordant antibiotic choice by site. Panel C shows percent change in guideline concordant antibiotic choice by antibiotic indication. Guideline concordant antibiotics were defined as follows: amoxicillin, penicillin, or amoxicillin/clavulanate for acute otitis media (AOM) or sinusitis, amoxicillin or penicillin for streptococcal pharyngitis and community acquired pneumonia (CAP), and nitrofurantoin or cephalexin for urinary tract infection (UTI). Panel D shows percent change in guideline concordant antibiotic prescription duration by antibiotic indication. Panel E shows percent change in streptococcal pharyngitis testing rates by site. *Denotes statistically significant improvement.

Interventions

The interventions in the tele-stewardship bundle consisted of disseminating patient education materials to participating sites including posters and handouts available in English, Spanish, and Arabic, providing access to educational materials for clinicians including communication skills training modules, ¹² vaccine educational material from the American Academy of Pediatrics (AAP), ¹³ and online microlearning modules addressing common causes of inappropriate outpatient antimicrobial prescribing, ¹⁴ creating and disseminating local guidelines for common pediatric outpatient infections based on national guidelines when available, providing access to local clinical guidelines via the Firstline phone app, ¹⁵ and quarterly audit and feedback with peer comparison. The quarterly feedback included an online meeting with each site where overall prescribing trends were reviewed at the clinic site level and de-identified individual prescriber level. Each meeting concluded with a brief teaching point about a common pediatric infection. Each individual prescriber also received a quarterly personalized email that included their rate of antibiotic prescription and rapid streptococcal pharyngitis testing compared to all other prescribers in the study (eFigure). Dates of implementation for each intervention can be found in Figure 3. Continuing medical education credit (CME) credit and Maintenance of Certification (MOC) Part IV Credit were offered for participating in the meetings, review of educational materials and review of feedback reports. The study was approved by the Vanderbilt University Institutional Review Board.

Figure 3.

Implementation timeline for various interventions. Feedback meetings and emails occurred quarterly, with the exact time depending on the availability of providers at each site.

Results

Among 147,357 total encounters (29.3% baseline, 68% intervention), 106,651 (72.4%) occurred in a pediatric primary care clinic while 40,739 (27.6%) occurred in an emergency department. The median patient age was 5 years (interquartile range 1–10 years). Most patients identified as White (63.0%), non-Hispanic/Latino (70.0%), and had public insurance (65.0%). Additional demographics are shown in Table 1.

Table 1.

Patient and participating practice characteristics

Overall antibiotic prescription rate decreased slightly from 12.4% (5338/43157) of encounters in the baseline period to 11.9% (11914/100200) of encounters in the intervention period (P = .01). Adjusted comparison via regression discontinuity analysis showed an odds ratio of 0.81 (P < .001) implying lower odds of antibiotic prescribing in the intervention period. Change in prescription rates varied by individual site (Figure 2). Two primary care clinics showed significant decreases in antimicrobial prescribing, while one showed a significant increase. There was greater variability among the emergency departments, with one having a significant decrease, one having a non-significant decrease, and one having a significant increase in overall prescription rate (Figure 2). There were statistically significant but non-clinically significant changes in prescription rate by race and primary spoken language (eTable).

Overall guideline-concordant prescribing for all indications increased from 44.3% in the baseline period to 47.8% in the intervention period (P < .001). Adjusted comparison via regression discontinuity analysis after adjusting for viral season and preexisting time trend showed an odds ratio of 0.78 (P = .02) suggesting lower guideline-concordant prescribing in the intervention period. Changes again varied by site: two primary care clinics had a non-significant decrease in guideline concordance, and one had a significant increase; two emergency departments had significant increases in guideline concordance while one had no significant change. When stratified by diagnosis, a significant overall increase in guideline-concordant prescriptions was found for AOM, streptococcal pharyngitis, and UTI, though individual sites had variable results. The percent of guideline-concordant prescriptions for sinusitis and CAP did not differ between the study periods (Figure 2).

For all diagnoses combined, there was a significant increase in 5-day prescriptions (6.3% baseline vs 19.7% intervention, P < .001) and a significant decrease in 10-day prescriptions (67.7% baseline vs 54.3% intervention, P < .001) (Figure 2).

There was a significant decrease in rapid streptococcal pharyngitis testing (10.9% baseline vs 7.6% intervention, P < .001), a trend which was present in all sites aside from one primary care clinic.

The rate of return visits within 14 days was low and did not differ significantly between the periods (5849/43157 [13.6%] baseline vs 13591/100200 [13.6%] postimplementation; P = .96).

Discussion

In this large prepost intervention study, we demonstrated that implementation of a bundled tele-stewardship intervention across rural and suburban pediatric primary care clinics and emergency departments was associated with lower antibiotic prescription rate and less diagnostic testing for streptococcal pharyngitis with mixed overall results for guideline-concordant prescribing. Our findings support the feasibility of tele-stewardship interventions in pediatric ambulatory rural settings, as has been previously reported for adults. ^{Reference Stenehjem, Wallin and Willis17}

The impact of the intervention was not similar across all practices. Outcomes including overall antibiotic prescription rate and guideline-concordant prescribing varied by site. The sites where the intervention did not reduce antibiotic prescribing did not significantly differ from others in provider or practice characteristics like number of clinicians, number of patients seen, or number of advanced practice providers. Data on how long each prescriber was in practice was not available but has been shown to impact antibiotic prescribing. ^{Reference Baillie, Merlo, Van Driel, Magin and Hall18} We did not see associations between intervention effectiveness and ED or primary care location. Site C had more patients seen by advanced practice providers compared to the other primary care clinics and had a larger decrease in prescribing. Provider engagement in the intervention, as assessed by measuring utilization of educational modules also did not differ between sites and was low overall. We did not track site engagement with quarterly educational sessions but did coordinate the sessions in previously scheduled site-specific administration meetings and offer CME to increase attendance. However, sites were different in terms of patient demographics such as race, ethnicity, and primary spoken language. For example, the patient population at sites C and E (which both had decreases in overall antibiotic prescribing) was over 80% White, while the population of site B (which had an increase in overall antibiotic prescribing) was less than 40% White. Whether unconscious clinician bias, systemic racism, language barriers, patient expectations, and clinicians’ expectations of patients contributed to the variable outcomes at different sites requires further study.

While our study had many strengths including a large sample size, inclusion of multiple sites and outpatient settings, rural and suburban settings, analysis of multiple diagnoses, and a two-year intervention period to reduce the effects of seasonal variation in prescription rates, it also had several limitations. Approximately 10% of the prescriptions had missing information in the duration field and were excluded from the analysis. In the latter half of 2024, national and local rates of Mycoplasma pneumoniae increased far above their usual rates. ^{Reference Raghuram, Furmanek, Chandler, Rashid, Mattingly and Ramirez19} Since this infection is usually treated with a 5-day course of azithromycin which is not considered to be a first-line antibiotic for CAP, high community prevalence of Mycoplasma may have lowered the rate of guideline concordance we found for CAP. However, we performed a sensitivity analysis of guideline concordance excluding the diagnosis of CAP and found similar results to the primary analysis. Another limitation was the delay in the roll-out of some of the interventions. Implementation of the Firstline application and dissemination of patient educational materials occurred in the latter half of the postimplementation period due to delays receiving institutional approvals, possibly limiting the effectiveness of the intervention. The intervention was bundled, so it is impossible to discern if any one of the components of the intervention was more effective than others. Finally, each patient’s diagnosis was dependent on the clinical judgement of their individual clinician, which could not practically be verified or standardized given the number of encounters involved.

Despite these limitations, the results of this study support the use of implementing bundled interventions using tele-stewardship to improve appropriate antimicrobial prescribing in pediatric rural and suburban primary care clinics and EDs. Further research into what factors drive the effectiveness of stewardship interventions in these settings would help sites tailor which parts of the bundle to implement for their needs. Tele-stewardship has great potential to improve appropriate antibiotic prescribing and quality of care in rural areas that lack access to in-person stewardship expertise.