Nearly 80% of antibiotic use in the United States occurs in the outpatient setting, accounting for 60% of antibiotic expenditures. Reference Sanchez, Fleming-Dutra, Roberts and Hicks1,Reference Hicks, Bartoces and Roberts2 Within the United States, antibiotic prescribing rates are highest in the southeast, particularly in Tennessee. Reference Fleming-Dutra, Hersh and Shapiro3,Reference Hersh, Shapiro, Pavia, Fleming-Dutra and Hicks4 In 2016, the pediatric antibiotic prescribing rate in Tennessee was 1.5-times higher than the national pediatric prescribing rate. Reference Katz, Staub and Ouedraogo5
Multiple different outpatient stewardship interventions such as prospective audit and feedback, antibiotic use commitment posters, and watchful-waiting methods have been proven to improve appropriate antibiotic prescribing in clinical trials, but optimal strategies to implement outpatient antimicrobial stewardship initiatives in real-world settings are not known. Reference Kronman, Gerber and Grundmeier6–Reference Meeker, Knight and Friedberg9 The objective of this study was to improve appropriate antibiotic prescribing for children in Tennessee. We developed an online tool kit with resources for implementation of outpatient pediatric antimicrobial stewardship, and we paired the tool kit with quarterly peer comparison and monthly practice-specific data review. Furthermore, we customized other antibiotic stewardship implementation strategies selected by each practice. We hypothesized that bundled implementation strategies would lead to fewer outpatient antibiotic prescriptions and would increase guideline-concordant antibiotic use in primary care practices in Tennessee.
Methods
Study design, participants, and setting
We performed a before-and-after intervention study to assess the impact of bundled implementation strategies on antibiotic prescribing in 5 primary-care pediatric practices not affiliated with an academic institution. One additional practice started the study but subsequently dropped out because of an upgrade of the electronic health record (EHR) that occurred during the study. The practice was unable to complete data validation after the EHR upgrade due to competing priorities.
Data were collected from May 2018 through November 2020 for 4 practices and through January 2021 for 1 practice. Although we initially planned a before-and-after intervention study comparing a baseline and intervention period, the coronavirus disease 2019 (COVID-19) pandemic began 3.5 months into our intervention period so we decided after the intervention to compare 3 study periods: period 1 (P1, baseline, May 2018–September 2019), period 2 (P2, intervention, before the COVID-19 pandemic, November 11, 2019–March 20, 2020), and period 3 (P3, intervention during the COVID-19 pandemic, March 21, 2020–November 10, 2020) (Fig. 1). One practice began their intervention period on January 1, 2020, so they had a longer washout period, a shorter prepandemic intervention period (P2), and a shorter COVID-19 intervention period (P3), through December 31, 2020. We included a 1-month washout period prior to the intervention prepandemic period (P2) for the remaining 4 practices.
Overall antibiotic prescribing for all practices. Black bars represent the count of monthly encounters with an antibiotic prescription. Grey bars represent the total count of monthly encounters. The black line represents the percent of encounters with an antibiotic prescription.

Visualize Health (ShareCare, Franklin, TN), a third-party software company, which was able to extract and display data from any EHR system, was used to collect data. The study was approved by the Vanderbilt University Institutional Review Board with a waiver of patient consent.
Study interventions
Prior to the start of the study, each practice identified a physician champion and operations champion for the project who led the selection and implementation of interventions. The study employed a bundled implementation strategy for outpatient antimicrobial stewardship, incorporating implementation strategies previously shown to be effective at reducing outpatient antibiotic use. Components of the bundled implementation strategy are outlined in Table 1. Practices chose 2–4 implementation strategies (listed in Table 1) to implement in their practice throughout the study intervention period.
Components of Bundled Implementation Strategy

Study personnel also held in-person or videoconference quarterly meetings with all providers in each practice to review their outcomes and provide peer comparison feedback both across practices enrolled in the study and at the provider level within each practice. Practices and providers were deidentified, but each practice and provider were given their unique study identifier prior to the meeting. Meetings were held in person prior to the COVID-19 pandemic and online during the COVID-19 pandemic.
Outcomes
The primary outcome was the percentage of encounters with an antibiotic prescription (stratified by all visits and sick visits only). A secondary outcome was antibiotic prescription appropriateness, measured as the percentage of encounters among children with specific diagnoses and no penicillin allergy that resulted in a guideline-concordant antibiotic prescription for community-acquired pneumonia, streptococcal pharyngitis, otitis media, sinusitis, and skin and soft-tissue infection. Reference Bradley, Byington and Shah10–Reference Stevens, Bisno and Chambers14 Another secondary outcome was the percentage of encounters with a diagnosis of viral upper respiratory tract infection and an antibiotic prescription. A third secondary outcome was the percentage of encounters with an antibiotic prescribed that occurred within 1 week of an initial encounter in which no antibiotic had been prescribed, to serve as a balancing measure for the study. Further details are included in the Appendix (online).
After the study began, we classified practices as having high or low study fidelity based on the number of times that someone from the practice accessed the Visualize Health online website, the number of online tool-kit logins per practice, the number of meetings the practice had with study personnel during the intervention period, and the number of plan–do–study–act (PDSA) cycles the practice submitted to study personnel throughout the intervention period.
Provider surveys
Before and after the study intervention period, we surveyed volunteer providers from each practice to assess provider knowledge, attitudes, practices, and perceived barriers to appropriate antibiotic prescribing. The survey was created using a modified version of the MITIGATE Antimicrobial Stewardship Toolkit survey, with permission from Dr May. 15
Statistical analyses
We compared medians for each outcome across the 3 study periods using the Kruskal-Wallis test. In addition to analyzing data for all practices combined, we analyzed data for each individual practice. We compared the pre- and postintervention survey responses using Wilcoxon signed-rank tests. All statistical analyses were performed using R version 3.6.3 software (R Foundation for Statistical Computing, Vienna, Austria) or Stata/IC version 15.1 software for Mac (StataCorp, College Station, TX).
Results
In total, 81 providers from 5 practices (range, 7–31 providers per practice) participated in the study (Table 2). Average yearly patient encounters per practice ranged from 15,075 to 106,500. (Table 2). The practices were all located in middle Tennessee and included pediatricians, family medicine physicians, and nurse practitioners.
Baseline Characteristics of the Five Participating Practices

Note. EHR, electronic health record.
a Providers watched the Dialogue Around Respiratory Illness Treatment (DART) online modules and met to discuss what they learned from the modules. 24
b Providers were given handouts for history elements to stratify patients into high-risk and low-risk penicillin allergy categories. They could then either delabel the patient, perform an in-office oral challenge, or refer to an allergist for further testing based on risk.
c EHR templates were changed to default to guideline-concordant recommendations for antibiotic prescribing, forcing providers to manually change orders if they chose to override the template.
Across all practices, 689,165 encounters were included: 411,279 (59.7%) in the baseline period (P1); 114,088 (16.6%) in the prepandemic intervention period (P2); and 163,798 (23.8%) in the period during COVID-19 pandemic (P3).
Implementation strategies and adoption
All practices chose to implement antibiotic-use commitment posters and parent education as implementation strategies (Table 2). Additional implementation strategies selected by practices included development of local clinical practice guidelines and EHR template changes, telephone consultation with the antibiotic stewardship team, in-office penicillin allergy delabeling initiatives, and online communication skills training. None of the practices that chose telephone consultation with the antibiotic stewardship team as an intervention utilized this resource.
Study fidelity varied by practice (Supplementary Table 1 online). Practices accessed the online Visualize Health dashboard between 2 and 229 times throughout the study period. However, we were unable to assess whether they were checking their antibiotic use data or other data relevant to their practice (vaccine completion and insurance status were also available to some practices on the dashboard). The online tool kit was minimally accessed, and total logins ranged from 3 to 15 per practice. The number of PDSA cycles submitted to study personnel related to practice-specific implementation strategies ranged from 3 to 11 per practice. Also, 3 practices participated in American Board of Pediatrics Maintenance of Certification part 4 credit for the entire study period; 1 practice participated for the first 6 months of the study only; and 1 practice chose not to participate.
Overall antibiotic use
Compared to the baseline (P1), we detected an increase in the overall percentage of encounters with an antibiotic prescribed in the prepandemic intervention period (P2) (as expected because antibiotic prescribing is generally higher during winter months), followed by a significant decrease in the intervention period during the pandemic (P3). For P1, the median percentage of encounters with an antibiotic prescribed was 15.65% (IQR, 12.13%–17.52%; n = 411,279); for P2 the median was 18.33% (IQR, 17.65%–18.72%; n = 114,088); and for P3, the median was 7.75% (IQR, 6.09%–8.71; n = 163,798) (P < .001) (Fig. 1). A similar trend was observed when restricting the data to sick visits only. For P1, the median percentage of sick visits with an antibiotic prescribed was 22.94% (IQR, 19.63%–24.55%; n = 265,950); for P2 it was 24.82% (IQR, 24.19–25.27; n = 82,484); and for P3 it was 14.09% (IQR, 13.21–15.45; n = 80,870) (P < .001). Compared to baseline, we similarly detected a significant decrease in the percentage of return encounters for an antibiotic prescription in P3, likely because of the COVID-19 pandemic. For P1, the percentage of return encounters for an antibiotic prescription was 0.64% (IQR, 0.42%–0.82%; n = 349,651); for P2 it was 0.73% (IQR, 0.66%–0.82%; n = 96,676); and for P3 is increased to 0.22% (IQR, 0.15%–0.27%; n = 143,350) (P < .001).
Guideline-concordant antibiotic use
The percentage of viral upper respiratory tract infection encounters with an antibiotic prescribed was close to 0 for all study periods. For P1, viral upper respiratory tract infection encounters with an antibiotic prescribed was 0.13% (IQR, 0–0.68%); for P2 it was 0.04% (0–0.11%); and for P3 it was 0 (0–0.26%). We detected an improvement in guideline-concordant antibiotic use for otitis media in P2, and it was sustained in P3 for all practices combined and for each individual practice, except practice E, which had a slight decrease in P2 (Table 3 and Supplementary Table 2 online). Similarly, all practices combined had a significant improvement in guideline-concordant prescribing for group A streptococcal pharyngitis, and 4 practices made significant improvements at the individual practice level (Table 3 and Supplementary Table 2 online). We detected significant improvements in guideline-concordant prescribing for all practices combined for pneumonia and sinusitis, but sample sizes for these diagnoses were small during the intervention periods (Table 3). Although there were slight improvements in guideline-concordant antibiotic use for skin and soft-tissue infections during both intervention periods, the baseline percentage of appropriate prescriptions was already high at 97.2% for all practices combined (Table 3). Practices B and D made improvements for both otitis and group A streptococcal pharyngitis prescribing and were ranked as the “more engaged” practices, although practice B had less of an improvement after P3.
Percent of Encounters with Guideline-Concordant Antibiotic Prescribing Across 5 Practices Participating in the Study

Surveys
Of the 81 providers in the study, 18 participated in the the pre- and postintervention surveys (22.2%). Most survey participants were female (n = 15, 83.3%), were physicians (n = 16, 88.9%), and had been in practice 10–29 years (n = 11, 61%). After the study, significantly more providers reported referring to guidelines to make decisions for prescribing antibiotics, including local practice guidelines: preintervention survey (n = 1, 5.6%) versus postintervention survey (n = 7, 38.9%) (P = .01). Most providers agreed that antibiotic stewardship programs are important to optimize antibiotic use in the primary care setting: preintervention survey agree or strongly agree (n = 16, 88.9%) versus postintervention survey (n = 18, 100%) (P = .21). Similarly, most did not feel that an antibiotic stewardship program would interfere with their usual approach to clinical decision making: preintervention survey (n = 12, 66.7%) versus postintervention survey (n = 18, 88.9%) (P = .90).
Discussion
In this study of real-world implementation of outpatient antibiotic stewardship bundled implementation strategies in outpatient community pediatric practices, the bundled implementation strategies led to significant increases in guideline-concordant antibiotic prescribing for all diagnoses. Despite the COVID-19 pandemic, we acquired large sample sizes for acute otitis media and streptococcal pharyngitis, and we detected significant increases in appropriate antibiotic prescribing for these diagnoses. Although the study was underpowered to assess antibiotic use for pneumonia and sinusitis due to few encounters for these diagnoses after the pandemic, we nevertheless observed significantly higher guideline-concordant prescribing for these diagnoses as well. Similar to national data during the COVID-19 pandemic, we observed decreases in overall outpatient antibiotic use across all practices and decreases in the numbers of outpatient encounters, especially for sick visits. Reference King, Lovegrove and Shehab16–Reference Chua, Volerman and Conti18 The survey results demonstrated that the bundled implementation strategies were well accepted by providers.
Practices with higher fidelity to the implementation strategies demonstrated higher guideline-concordant antibiotic prescribing than those with lower fidelity. This finding is not surprising and supports data from prior studies that indicate that the implementation strategy is equally as important as the intervention itself. Reference Kronman, Gerber and Grundmeier6 Practices that participated in the maintenance of certification (MOC) credit and submitted more PDSA cycles demonstrated higher fidelity than practices that either submitted the minimum number of PDSA cycles or did not participate in the MOC credit. PDSA cycle submission was a surrogate of providers setting aside time to discuss interventions outside of meetings scheduled by study personnel for these discussions. Although practice B was ranked as the practice with highest fidelity, it had minimal change in guideline-concordant treatment for streptococcal pharyngitis between P3 and P1. This practice had the highest rate of guideline-concordant treatment for streptococcal pharyngitis at baseline, which likely explains why they saw minimal improvements. Future outpatient antibiotic stewardship studies should focus on provider engagement and should plan to measure engagement as an implementation outcome.
A key part of our study design was allowing practices to choose their own implementation strategies, which increased study participation but also inhibited our ability to compare effectiveness of specific implementation strategies. Although improvements observed in our study may have been due to audit and feedback alone, audit and feedback as a single strategy is more successful for increasing adoption of evidence-based practices than for deimplementation. Reference Linder, Schnipper and Tsurikova19,Reference Carney, Bowles and Sickles20 Prior systematic reviews have reported that studies using audit and feedback alone to decrease use of an ineffective practice have not resulted in significant reductions. Reference Tuti, Nzinga and Njoroge21 In addition, the use of bundled or multifaceted implementation strategies is associated with improved implementation, with a correlation between the number of implementation strategies used and the probability of implementation success. Reference Rogal, Yakovchenko and Waltz22 Thus, our findings are consistent with earlier studies in implementation science from other areas of clinical medicine.
Despite these encouraging results, our study had several limitations. We were unable to distinguish whether the decrease in overall antibiotic prescribing we observed during the intervention periods was due to the intervention or the COVID-19 pandemic. Pandemic-associated declines in well- and sick-child visits and in antibiotic prescriptions have been well documented. Reference King, Lovegrove and Shehab16,Reference Heintzman, O’Malley and Marino23 However, we attribute the increase in guideline-concordant antibiotic prescriptions in the postintervention periods to the stewardship interventions because the pandemic was unlikely to have affected antibiotic choices of providers. Additionally, competing priorities of the enrolled practices (ie, financial concerns due to lower patient volumes, need for rapid uptake of telemedicine, and other stressors due to the pandemic) may have reduced their study fidelity.
This study had several limitations. We excluded patients with a penicillin allergy from the calculation of guideline-concordant antibiotic use, which may have limited our ability to see benefit of the penicillin allergy delabeling strategy implemented by 2 of the practices. Practices implemented strategies for only 6 months, and it is possible that this implementation period was too short to yield meaningful results. Additionally, we did not collect intervention-specific results such as number of patients with penicillin allergy delabeled. Due to the COVID-19 pandemic, we were also unable to compare different implementation strategies within the same practice because patient encounter volumes differed greatly between the intervention periods. We attempted to separate out in-person and telemedicine encounters after the start of the pandemic but were unable to reliably extract these data. Anecdotally, providers shared that few encounters resulting in an antibiotic prescription occurred over telemedicine as they preferred to perform a complete physical exam on sick children. We did not evaluate sustainability of the implementation strategies. Finally, the before-and-after surveys were completed by only 22% of providers, and it is possible that they do not represent the beliefs of study participants who did not respond.
In conclusion, despite the COVID-19 pandemic, this study demonstrates that outpatient antibiotic stewardship implementation strategies are feasible in community general pediatrics practices that are not affiliated with an academic institution, and it provides a framework for real-world implementation of outpatient antibiotic stewardship. We saw significant improvements in guideline-concordant antibiotic prescribing, and allowing practices to customize their implementation strategies improved acceptance by participants. The passive online tool kit was minimally accessed by providers, suggesting that more resources should focus on active implementation strategies like peer comparison and regular meetings where providers prioritize time for stewardship and relationship building with antimicrobial stewards. Future studies should expand to other outpatient settings including urgent cares, retail health clinics, and rural settings, and they should evaluate the sustainability of the implementation strategies.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/ice.2021.534
Acknowledgments
The authors thank the participating general pediatrics practices for their participation and enthusiasm for the study. We are grateful to the Tennessee Department of Health for support for this project. The Health Department was not involved in study design, conduct, data analysis, manuscript preparation or publication.
Financial support
This study was supported by the Tennessee Department of Health and the Centers for Disease Control and Prevention Leadership in Epidemiology, Antimicrobial Stewardship and Public Health (LEAP) fellowship (to S.K.), sponsored by the Society for Healthcare Epidemiology of America (SHEA), the Infectious Diseases Society of America (IDSA), and the Pediatric Infectious Diseases Society (PIDS).
Conflicts of interest
The authors have no conflicts of interest to disclose.



