Data source

Data covering the period from January 2020 to June 2021 were extracted on inpatients from each of four hospitals in our healthcare network where Hospital Medicine Services are provided. Hospitals A and B were 582-bed and 537-bed academic medical centers, respectively, while hospitals C and D were 373-bed and 125-bed community hospitals, respectively (Supplemental Table S1). Staffing models were mostly similar at each with 2–4 Nocturnists at each facility, but Advanced Practice Providers worked only at the two larger facilities. For each hospitalist in each hospital, we utilized billing data to identify patient encounters for which the hospitalist provided care and calculated the number of days each provider cared for each patient (billed patient days, bPD). All hospitals have a robust antibiotic stewardship team in place with harmonization of clinical practice guidelines and effort across all facilities.

Days of therapy

We extracted antibiotic prescribing data from the electronic medication administration records (eMAR), which accounts for returned medications and captures only administered medications. For each bPD, matching transaction dates in the eMAR with an order for antibacterial agents belonging to any of the three NHSN antibiotic groups—broad spectrum hospital-onset (BSHO), broad-spectrum community-acquired (BSCA), and anti-methicillin-resistant Staphylococcus aureus (Anti-MRSA)—were captured as billed days of antibiotic therapy (bDOT). bDOT differs from DOT in that a bDOT only captures administration of the antibiotic if the provider also billed on that same date, thus excluding DOT for the patient if cared for by a different service team (care escalated to ICU, transfer to surgery service). To streamline the focus and eliminate the need to track specific antibiotics, all bDOTs were summed by NHSN group for each provider. If three separate antibiotic agents belonging to the same NHSN group had the same transaction date for the same provider, this equated to three bDOTs for that NHSN group on that date for the provider.

Patient encounter characteristics

Second, we extracted patient characteristics linked to each encounter, including age, antibiotic indications (pneumonia, urinary tract infection [UTI]), and comorbidities/disease severity using ICD-10 codes. These included sepsis, obesity, COVID-19, end-stage renal disease (ESRD), neurological conditions, sepsis, malignancies, and the Charlson Comorbidity Index (CCI). These characteristics were identified a priori as patient factors that were potential predictors of antibiotic prescribing in hospital settings. ^{Reference Sundararajan, Henderson, Perry, Muggivan, Quan and Ghali19}

Exclusions, attribution, and aggregation parameters

Based on feedback from Division of Hospital Medicine Leadership regarding this initiative, as well as a series of focus groups with volunteer Hospital Medicine Providers, we defined some parameters for eligible data for inclusion into a summary metric. These parameters included exclusion of data related to patients with cystic fibrosis (roughly 5% of encounters) from the analysis since antibiotic prescriptions for this subset were determined by pulmonary clinicians, not hospitalists. We also excluded data on hospitalists who worked nights exclusively (i.e., 8 nocturnists) since the data captured were largely reflective of prescribing practices of other service lines such as emergency medicine and critical care. Over the study period 12 Advanced Practice Providers worked with a variety of Faculty, and prescribing of Advanced Practice Providers (APPs) were attributed to the Faculty billing for the patient care that day. All measures (bPD, patient characteristics, and bDOT) were aggregated into bimonthly (every two months) intervals, resulting in provider-level datasets stratified by hospital. Next, provider-specific data for any single bimonthly period was excluded from analysis if the provider accrued fewer than 80 bPD, suggested they worked < 2 weeks during the preceding month. The bimonthly metric was the consensus aggregation level to allow providers more regular feedback in a period they still could recollect their patient care experience. This approach was adopted to reduce data sparsity for providers with variable work schedules reflective in a monthly metric. Finally, both Hospital Medicine Service (HMS) leadership and the HMS focus groups universally agreed they would prefer hospital-specific risk adjustments (i.e., facility-specific models as described below) due to the drastically different nature of patient mix between the four facilities (as reflected in Table 1).

Table 1.

Summary of billed patient-days and patient-mix characteristics for hospital medicine providers at four hospitals within an academic healthcare network in Georgia (2020–2021)

BSHO, Broad-spectrum hospital-onset antibiotics; BSCA, Broad-spectrum community-acquired antibiotics; Anti-MRSA, Anti-Methicillin-resistant Staph, Aureus antibiotics; ESRD, End-Stage Renal Disease; UTI, Urinary tract Infection; CCI, Charlson Comorbidity Index.

Observed-to-expected ratios (OERs)

The main outcome was provider-specific observed-to-expected ratios (OERs), a metric quantifying the deviation of observed prescribing from the expected, given the characteristics of patients seen by the provider within the specified period. OERs were calculated by dividing observed prescribing by the expected prescribing derived using methods detailed below.

Statistical analysis

The prescribing characteristics of the providers, total bPD, and bDOT for the three NHSN groups were presented as medians and interquartile ranges (IQR). The proportions of the bimonthly bPD attributed to care for patients with each patient-mix characteristic were calculated for each provider and described for each hospital using medians and IQR. For each provider, the observed prescribing of each NHSN antibiotic group was calculated for each bimonthly interval as a ratio of the observed bDOT for that antibiotic group and the total bPD within the interval. These data were described across hospitals using medians and interquartile ranges (IQR).

Expected prescribing was estimated using patient-mix adjusted linear mixed regression models with random intercepts to account for repeated measurements of provider prescribing over time. The expected prescribing predictive model selection process involved two stages. First, we employed backward elimination in linear mixed models across 1000 bootstrapped samples to identify variable inclusion frequencies and establish candidate models. ^{Reference Altman20,Reference Heinze, Wallisch and Dunkler21} Bootstrap samples were created by sampling at the provider-level with replacement and included all bimonthly observations associated with selected providers in the bootstrap sample. ^{Reference Myers, Brincks, Ames, Prado, Penedo and Benedict22,Reference Austin and Leckie23} Providers selected more than once in a bootstrap sample were assigned different subject-level identifiers to ensure treatment within the model as distinct subject clusters. A linear mixed-effects model, adjusted for all ten patient characteristics measured, was fitted to each bootstrap sample, and statistically significant covariates (alpha = 0.05) were identified using backward elimination (fixed effects only). The frequency of each covariate being selected as a significant predictor of antibiotic prescribing across all bootstrap samples used was calculated and ranked. Ten candidate models were then created by sequentially adding these potential predictors to a mean model, starting with the highest-ranked predictor.

Model comparisons

The predictive accuracy of the candidate models was quantified and ranked using the bootstrap-based optimism-correction method. ^{Reference Harrell, Lee and Mark24,Reference Iba, Shinozaki, Maruo and Noma25} This method was selected due to the small number of hospitalists and/or bimonthly observations (number of unique hospital medicine providers per hospital <50; number of bimonthly observations per hospital <300). First described by Harrell et al, the bootstrap-based optimism-correction method is an internal predictive model validation method that involves repeatedly sampling from a study sample to measure and correct for anticipated optimism from model overfitting. ^{Reference Harrell, Lee and Mark24–Reference Smith, Seaman, Wood, Royston and White26} This approach provides an alternative way of internally validating predictive models for small samples where more robust model validation methods that require data splitting may not be prudent and has been shown to perform similarly to other model validation methods. ^{Reference Harrell, Lee and Mark24,Reference Smith, Seaman, Wood, Royston and White26} We estimated optimism-corrected mean square errors (OC-MSE) for each candidate model (CM_k) by fitting the model to the hospital dataset (D) and calculating the mean square error (MSE), C (step 1). Then, in a bootstrap sample of the hospital dataset (D*), we refitted CM_k and calculated the MSE (C_boot, step 2). Using the coefficients of model fit in D*, we predicted outcomes in the original dataset, D, and re-evaluated the MSE (C_orig, step 3). The optimism was then calculated as C_boot-C_orig (step 4). Steps 2 through 4 were repeated 200 times and the average optimism (O) was calculated (step 5). For each CM_k, OC-MSE was calculated as C – O (step 6). The OC-MSEs for all 10 candidate models were then ranked and the model with the lowest OC-MSE was selected as the final prediction model. These final risk-adjusted models were fitted to original hospital datasets and used to estimate expected prescribing adjusted for patient-mix characteristics for each provider in each bimonthly interval. Serial provider-specific OERs were then calculated as the ratio of observed prescribing to expected prescribing in each bimonthly interval.

Sensitivity analysis

We evaluated expected prescribing predictive model selection, as described above, against information criterion-based model selection. This was done by assessing agreement in model selections across OC-MSE, marginal Akaike information criterion (mAIC), and conditional Akaike information criterion (cAIC) methods. ^{Reference Müller, Scealy and Welsh27–Reference Vaida and Blanchard29} For each NHSN group and hospital, we derived mAIC and cAIC for all candidate models considered and identified the models that most minimized each information criterion. ^{Reference Vaida and Blanchard29,Reference Säfken, Rügamer, Kneib and Greven30} Then, we assessed how frequently the OC-MSE selected models matched the final models selected using the information criterion (absolute agreement). We also assessed how frequently the information criteria estimates of the OC-MSE selected models were within +/–2 of the lowest information criteria estimated for the group of candidate models (relative agreement). All analyses were performed using R Statistical Software (v4.2.0; R Core Team 2021).