Antibiotic resistance is estimated to cause >2.8 million infections and >35,000 deaths in the United States each year.1 Antibiotic overuse is a major driver of antibiotic resistance. Most antibiotic prescribing takes place in the ambulatory setting, and at least 30% of all outpatient antibiotic prescriptions are unnecessary.Reference Suda, Hicks and Roberts2,Reference Fleming-Dutra, Hersh and Shapiro3 Most of the avoidable outpatient prescribing is for respiratory tract infections.Reference Gonzales, Malone, Maselli and Sande4
The Centers for Disease Control and Prevention (CDC) Core Elements of Outpatient Antibiotic Stewardship call for tracking and reporting ambulatory antibiotic prescribing by individual clinician and facility.Reference Sanchez, Fleming-Dutra, Roberts and Hicks5 However, it is not practical to perform retrospective chart review to audit prescribing, so metrics that can be easily extracted are needed. In an effort to organize administrative data more meaningfully, Fleming-Dutra et alReference Fleming-Dutra, Hersh and Shapiro3 grouped International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes for infections into 3 tiers based on whether or not antibiotics are indicated. Tier 1 diagnoses are conditions for which antibiotics are almost always indicated (eg, bacterial pneumonia); tier 2 diagnoses are conditions for which antibiotics may be indicated depending on the clinical scenario (eg, sinusitis); and tier 3 diagnoses are conditions for which antibiotics are almost never indicated (eg, bronchitis).Reference Fleming-Dutra, Hersh and Shapiro3
Metrics derived from administrative data have the potential to make tracking and reporting antibiotic prescribing data more efficient. However, determining which metrics are most strongly associated with inappropriate prescribing is essential. The primary objective of this study was to determine metrics using prespecified respiratory tract diagnoses (RTDs) that are most strongly associated with inappropriate antibiotic prescribing. The secondary objective was to determine provider characteristics associated with inappropriate prescribing.
Methods
Data collection
We conducted a retrospective cohort study from January 1, 2016, through December 31, 2016, at the 32 primary care practices in the University of Pennsylvania Health System, which included 227 primary care providers. There were no ongoing antibiotic stewardship interventions at any of the practices throughout the study period. Likewise, no clinical decision support tools for treatment of respiratory tract infections were in use.
Office visits were eligible for inclusion if (1) there was an ICD-10-CM code for at least 1 RTD (Supplemental Table 1, online), (2) the patient was at least 18 years of age, and (3) the prescriber was either an attending physician or advanced practice provider (APP). Office visits were excluded if an infection other than a respiratory infection was coded in the same visit. Only office visits in which a patient was seen primarily by an attending physician or APP were included in the analysis, given that resident trainees were not present in all practices. Only the first office visit during the study period per patient was included. Demographics were collected for all patients in the study cohort. Comorbidities were collected using ICD-9-CM and ICD-10-CM codes within 1 year prior to the index office visit.
Demographics and Comorbidities of Patients in Study Cohort and the Random Cohort

Note. SD, standard deviation; COPD, chronic obstructive pulmonary disease; HIV, human immunodeficiency virus.
To determine a gold standard of antibiotic appropriateness, encounters were randomly selected from office visits in which an oral antibacterial agent (excluding rifamycins, ethambutol, isoniazid, pyrazinamide, vancomycin, rifaximin, fidaxomicin, fosfomycin, metronidazole, and nitrofurantoin) or intramuscular penicillin was prescribed. To identify the random subset, 60 prescribers were randomly selected. Then, 20 office visits for each of these prescribers were randomly selected, for a total of 1,200 randomly selected office visits (Fig. 1).
Methods of determining the association of appropriate antibiotic prescribing with respiratory tract diagnosis (RTD) metrics and provider demographic characteristics.

Electronic health records (EHRs) from each office visit in the random subset were reviewed by at least 1 of 4 infectious diseases physicians (V.C., K.D., M.D., and K.H.). The decision to prescribe an antibiotic was coded as appropriate or inappropriate based on diagnostic criteria for bacterial respiratory tract infections. Diagnostic criteria for bacterial community-acquired pneumonia, group A Streptococcus pharyngitis, and bacterial rhinosinusitis were based on Infectious Diseases Society of America (IDSA) guidelines.Reference Mandell, Wunderink and Anzueto6–Reference Shulman, Bisno and Clegg8 Diagnostic criteria for bacterial otitis media were modified from American Academy of Pediatrics guidelines.Reference Lieberthal, Carroll and Chonmaitree9 Diagnosis of pertussis was based on CDC criteria.10 Prescribing for bronchitis was considered appropriate only if the patient had an established diagnosis of chronic obstructive pulmonary disease or bronchiectasis and the patient fit clinical criteria for antibiotic treatment based on Global Initiative for Obstructive Lung Disease criteria.11
Antibiotics were classified as appropriate if the patient fit diagnostic criteria for a bacterial respiratory tract infection. Otherwise, they were classified as inappropriate. Only the decision to prescribe an antibiotic was considered when determining appropriateness. Antibiotic choice, dose, and duration were not considered. Chart abstraction for appropriateness was performed using REDCap electronic data capture tools hosted at the University of Pennsylvania.Reference Harris, Taylor and Thielke12 Also, 15% of office visits were double-coded by 2 infectious diseases physicians. If there was a discrepancy in the determination of appropriateness for an office visit, 3 infectious diseases physicians discussed the case and came to a consensus on appropriateness. This adjudicated decision was used as the final determination of appropriateness for the double-coded encounters.
RTD metrics that were evaluated to determine their association with appropriateness included the proportion of antibiotic prescribing for visits in which (1) any RTD was coded; (2) a tier 3 RTD was coded; (3) an upper RTD was coded; (4) a lower RTD was coded; (5) bronchitis was coded; and (6) nonspecific symptoms (eg, cough) were coded without a specific RTD. Determination of tier 3 RTDs were based on modified criteria previously described using ICD-10-CM codes for diagnoses in which antibiotics are almost never indicated; office visits were excluded if the visit included a tier 1 or tier 2 RTD.Reference Fleming-Dutra, Hersh and Shapiro3 The denominator for all of these RTD metrics was calculated by identifying all of the visits coded with a relevant ICD-10-CM code. The proportion for which an antibiotic was prescribed was determined for each of the visit types. Only the first visit for each patient was utilized in these calculations.
To examine the impact of severity of illness, the mean Charlson comorbidity index (CCI) was calculated for each primary care provider by first identifying all patients seen by that prescriber with a code for an RTD during the study period. Next, a CCI for each of these patients was calculated using ICD-9-CM and ICD-10-CM codes for relevant comorbidities during the year prior to the office visit using a method previously described.Reference Quan, Sundararajan and Halfon13 A mean CCI was then calculated for each provider.
Prescriber demographics were collected, including medical degree type, specialty, year of board certification, and gender. The ambulatory care practices were characterized as either teaching or nonteaching depending on whether resident trainees saw patients at the clinic, and they were characterized as either urban or nonurban.
Statistical analysis
Categorical variables were summarized using frequencies, and continuous variables were summarized using mean and median when appropriate. The association between demographic factors and appropriate prescribing was determined using a t test or a Wilcoxon rank-sum test where appropriate. Linear regression was used to determine the RTD metrics that were associated with appropriate antibiotic prescribing, as determined from a manual EHR review of randomly selected patients. Multivariate regression was performed to adjust for CCI. R2 was used to evaluate association of these RTD metrics with appropriateness. Data were analyzed using STATA version 15.0 software (StataCorp, College Station, TX). This study was approved by the University of Pennsylvania Institutional Review Board.
Results
Of 109,049 total office visits during the study period, 42,196 visits (39%) were identified in which an antibiotic was prescribed with a respiratory diagnosis. When duplicate patients were removed, 26,671 unique patient visits (37%) of 71,144 office visits were identified in which an antibiotic was prescribed. There were 89,240 ICD-10-CM codes for RTDs in these 71,144 unique office visits. For the visits in which an antibiotic was prescribed, there were 31,677 ICD-10 codes for RTDs. Demographic characteristics and comorbidities of patients in the study cohort and random subset are shown in Table 1. RTDs are shown in Table 2, and antibiotics prescribed in these visits are shown in Supplementary Table 2 (online). The total number of antibiotic prescriptions from office visits in which an RTD was coded and proportion of antibiotic prescribing for each RTD metric by month are shown in Figure 2.
Respiratory Tract Diagnoses (RTDs) in Visits With an Antibiotic Prescribed for All Patients in the Study Cohort as Well as Subset of 1,200 Randomly Selected Patients, Identified by ICD-10-CM Code

Note. ICD-10-CM, International Classification of Disease, Tenth Revision, Clinical Modification; URI, upper respiratory tract infection; COPD, chronic obstructive pulmonary disease.
Proportion of antibiotic prescribing for respiratory tract diagnosis (RTD) metrics and total number of antibiotic prescriptions for RTDs by month.

In the entire study cohort, there was considerable variability in the proportion of prescribing for the different RTD metrics (Supplementary Figure 1 online). Of all the RTD metrics, bronchitis had the highest overall median of provider prescribing (ie, 59%) as well as the widest range (0–100%). Nonspecific symptoms had the lowest median provider prescribing (ie, 13%) and the narrowest range excluding outliers (0%–60%).
Of the 180 office visits that were double-coded in the random subset, there was 92% concordance between coders for determination of antibiotic prescribing appropriateness. Overall, 69% of all antibiotic prescriptions for RTDs were determined to be inappropriate, with a range of 15%–100% for individual providers. Certain provider characteristics were associated with higher proportion of inappropriate prescribing including APP, family medicine specialty, initial board certification 1997 or later, nonteaching practice, and nonurban practice (Table 3). Female clinicians had higher rates of inappropriate prescribing, but this difference was not statistically significant. Given the seasonal variation in antibiotic prescribing for RTDs (Fig. 1), a post hoc analysis of inappropriateness by time of year was performed, demonstrating that fall and winter months (October–March) were associated with higher proportion of inappropriate prescribing than spring and summer months (April–September) (74% vs 63%; P < .01).
Primary Care Provider Demographic Characteristics and Association With Proportion of Inappropriate Antibiotic Prescribing for RTDs for the 60 Providers Randomly Selected to Determine Appropriateness of Antibiotic Prescriptions

Note. RTD, respiratory tract diagnosis; APP, advance practice provider.
The individual RTD metrics that were most strongly associated with inappropriate prescribing in univariate analysis were proportion of prescribing for tier 3 RTDs (R2 = 0.24; P < .01) and proportion of prescribing for all RTDs (R2 = 0.23; P < .01) (Table 4 and Fig. 3). These RTD metrics, as well as all other metrics reporting proportion prescribing for various groupings of RTDs, had a positive association with inappropriateness as determined by manual chart review. In other words, primary care providers with a higher proportion of visits with antibiotic prescriptions also tended to prescribe inappropriate antibiotics more frequently.
Association of RTD Metrics With Appropriateness of Antibiotic Prescribing as Determined by Manual Chart Review for 60 Randomly Selected Primary Care Providers

Note. RTD, respiratory tract diagnosis; CI, confidence interval; CCI, Charlson comorbidity index.
Proportion of inappropriate prescribing and association with (A) percentage prescribing for tier 3 diagnosis visits and (B) percentage prescribing for all respiratory tract diagnosis (RTD) visits.

Notably, univariate analysis showed a nonsignificant negative association between a provider’s mean CCI and that provider’s proportion of inappropriate prescribing. Incorporating CCI in multivariate analyses did demonstrate a modest improvement in the models (Supplementary Table 3 online). For instance, incorporating CCI in a multivariate model with proportion of prescribing for tier 3 RTDs demonstrated an increase in R 2 over the univariate analysis of proportion of prescribing for tier 3 RTDs (0.32 vs 0.24). However, we detected a significant association between CCI and inappropriateness in the multivariate models only when CCI was combined with proportion of prescribing for tier 3 RTDs and not when it was combined with other RTD metrics.
In a post hoc analysis, providers with lower mean CCI tended to have training in family medicine rather than internal medicine (mean CCI, 4.1 vs 5.2; P < .01) and to be APPs rather than physicians (mean CCI, 4.7 vs 5.1; P = .02). Other provider characteristics were not associated with significant differences in CCI.
Discussion
Based on national prescribing trends, outpatient antibiotic stewardship programs should prioritize interventions targeting inappropriate prescribing for RTDs. Developing meaningful and validated prescribing metrics has the potential to decrease inappropriate antibiotic use through benchmarking of providers and to facilitate tracking of antibiotic use by healthcare systems, health plans, and public health agencies. This practice of auditing can be used on a clinician or institution level to highlight patterns in prescribing, to guide stewardship interventions, and to assess the impact of interventions. However, it is not practical to perform retrospective medical record reviews to audit prescribers. In this study, we sought to determine which RTD metrics that are accessible from administrative data were associated with inappropriate antibiotic prescribing using manual chart review.
The RTD metrics with the strongest association with inappropriate antibiotic prescribing were proportion of prescribing for office visits in which a tier 3 RTD was coded and proportion of prescribing for office visits in which any RTD was coded. Adding CCI to a multivariate model with each of these metrics to adjust for chronic comorbidities yielded a modest improvement compared to each metric alone. CCI had a negative association with inappropriate prescribing. In other words, primary care providers who cared for patients with lower mean CCI tended to prescribe antibiotics more inappropriately. The original intention of including CCI was to adjust for the potential of higher antibiotic utilization by providers with patients with higher CCIs. However, we detected a trend toward lower utilization by these providers.
This finding between CCI and inappropriate prescribing may have been confounded by provider characteristics. In a post hoc analysis discussed previously, providers with lower mean CCI tended to have training in family medicine as opposed to internal medicine and to be APPs as opposed to physicians, both of which factors were associated with higher rates of inappropriate prescribing. Further supporting this hypothesis that the relationship between CCI and antibiotic prescribing may be a result of confounding, prior studies have demonstrated that variation in antibiotic prescribing is not explained by differences in comorbidities of individual patients.Reference Pouwels, Dolk and Smith14 Therefore, antibiotic stewardship programs may wish to report RTD metrics that are not adjusted by CCI for the following reasons: (1) adjusting for CCI is associated with only a modest improvement in models; (2) higher CCI is actually associated with more appropriate prescribing; (3) the association between CCI and appropriateness may be a result of confounding; and (4) calculating mean CCI by provider on an ongoing basis is time intensive.
Given that providers may attempt to circumvent antibiotic prescribing audits by inappropriately shifting ICD-10-CM codes to tier 1 or tier 2 RTDs, stewardship personnel should consider reporting both proportion of prescribing for office visits with tier 3 diagnoses and proportion of prescribing for office visits with any RTD to capture shifts in diagnostic coding.Reference Calfee, Brooks and Zirk15 Using both metrics could detect and possibly minimize shifting of diagnostic codes that could occur during an audit and feedback intervention. For example, providers could shift diagnostic coding from bronchitis (tier 3 diagnosis) to pneumonia (tier 1 diagnosis) to justify prescribing an antibiotic. If this practice did occur, monitoring both metrics would allow the stewardship team to detect coding shifts because the proportion of prescribing for tier 3 diagnoses would change, but not prescribing for all RTDs because bronchitis and pneumonia would both be captured in the latter diagnostic grouping. However, more study is required to evaluate these metrics in the setting of audit and feedback and to understand provider perceptions of these metrics.
Notably, we detected considerable seasonable variation not only in total antibiotic prescribing but also in proportion of antibiotic prescribing for some RTD metrics, including prescribing for office visits in which any RTD was coded. Therefore, in fall and winter months not only was the number of antibiotics higher, but the proportion of antibiotic prescribing per office visit was also higher. Seasonal variation in prescribing for RTDs has been noted in prior studies as well.Reference Ferech, Coenen and Malhotra-Kumar16–Reference Curtis, Walker, Mahtani and Goldacre19 In a post hoc analysis of inappropriateness by time of year discussed previously, fall and winter months were associated with higher proportion of inappropriate prescribing compared to spring and summer months. More study needs to be performed to understand the drivers of this seasonal variation; however, this finding suggests that, at least in part, the increased antibiotic prescribing for RTDs in fall and winter months may be related to an increase inappropriate antibiotic prescribing during these months.
Macrolides made up the greatest proportion of antibiotic prescriptions for RTDs (41%), followed by β-lactams (38%) and fluoroquinolones (12%). Although we did not evaluate the appropriateness of antibiotic choice in this study, the high frequency of antibiotic prescribing with macrolides and fluoroquinolones could indicate high rates of inappropriate antibiotic choice because the high rates of inappropriate use of these agents in RTDs is well documented.Reference Suda, Hicks and Roberts20 However, additional study is warranted to evaluate appropriateness of antibiotic choice.
Our findings also suggest that APPs, family medicine providers, clinicians in nonteaching practices, clinicians in nonurban practices, and clinicians who are newer to practice have higher proportion of inappropriate antibiotic prescribing compared to physicians, internal medicine providers, clinicians in teaching practices, clinicians in urban practices, and clinicians who have been in practice longer, respectively. Prior studies have identified provider demographics to be associated with higher proportion of antibiotic prescribing or with surrogate metrics of inappropriateness, including female gender (compared to male), APP (compared to physician), family medicine specialty (compared to pediatrician and internal medicine), older age (compared to younger), nonurban practice setting (compared to urban), and southern United States geographic region (compared to western).Reference Saliba-Gustafsson, Hampton and Zarb21–Reference Ference, Min and Chandra33
All of the characteristics that could be assessed in this study had similar associations to prior studies except for time in practice. In this study, less time in practice was associated with a higher proportion of inappropriate antibiotic prescribing, whereas in other studies older providers were associated with higher proportion of inappropriate antibiotic prescribing. This difference could have resulted from assessing different infectious conditions or different practice settings. The association with time in practice for physicians may not be reproducible and requires further study. Likewise, more study is needed to understand how provider demographic characteristics affect antibiotic decision making and whether targeted interventions that address these characteristics are beneficial.
This study has several limitations. First, it was a retrospective single health-system study, which could limit the generalizability of results. Second, RTD metrics were based on diagnostic codes. Although, the determination of appropriateness was based on manual chart review by infectious diseases physicians, the validity of these RTD metrics depends on providers choosing ICD-10-CM diagnostic codes that accurately capture the clinical scenario, which may not always be the case. Third, in determination of association of provider demographic characteristics with appropriate prescribing, the small number of providers in the random subset precluded multivariate analysis and exploration of association of different provider characteristics with each other.
In conclusion, RTD metrics most strongly associated with inappropriate antibiotic prescribing for RTDs were proportion of prescribing for office visits in which a tier 3 RTD was coded and proportion of prescribing for office visits in which any RTD was coded. Antibiotic stewardship programs may be able to utilize these metrics to provide periodic reporting, benchmarking, or academic detailing to improve antibiotic prescribing for RTDs. Future study should focus on understanding provider perceptions of these RTD metrics and outcomes following implementation of these metrics for the purposes of audit and feedback.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/ice.2021.44
Acknowledgments
The authors would like to acknowledge the Antimicrobial Resistance and Healthcare Epidemiology Study Collaborative (ARES) at the University of Pennsylvania for expertise in study design and data analytics.
Financial support
This research was supported by a CDC Cooperative Agreement (FOA no. CK16-004) from the Epicenters for the Prevention of Healthcare Associated Infections.
Conflicts of interest
No authors have conflicts of interest to report.



