Discussion

We describe a machine learning model that accurately stratifies inpatient antibiotic orders according to predicted likelihood of being not indicated or suboptimal and thus needing review by an ASP team. The model employs ubiquitously available data that can be automatically extracted from the EHR without the need for manual extraction, and can be interrogated to make interpretations explainable. Its predictions demonstrate accuracy over several spectra of antibiotics, although its performance appears slightly higher in community-acquired compared to hospital-acquired infections, potentially reflecting differences in complexity between infections arising in the community versus those in patients with significant healthcare exposure.

The most influential clinical features in our model include markers of hemodynamic instability, inflammation, and end-organ dysfunction. Relatively little model performance comes from antibiotic orders themselves. This emphasis on markers of infection mirrors clinicians’ approach to empiric antibiotic use; empiric treatment is life-saving in newly septic patients, and is often guided by clinical status rather than microbiological data. Interestingly, markers of instability or inflammation did not yield positive predictions more than 48 hours in advance of review. In these cases abnormal values (leukocytosis, low systolic blood pressure) actually correlate with predictions that antibiotic use might not be indicated/optimal. One potential explanation is that abnormal values in this setting can arise from preexisting non-infectious causes, such as reactive leukocytosis in malignancy or stable hypotension in a patient with cirrhosis. Alternatively, abnormal values 2–3 days prior to review may function as a proxy for prior antibiotic use, and the need to revisit prior empiric treatment.

We have previously described a machine learning approach to screen for antibiotic orders in need of ASP review, describing an approach that similarly found markers of clinical instability or systemic inflammation as key predictors. ^{Reference Bystritsky, Beltran, Young, Wong, Hu and Doernberg20} In contrast to this prior work, which used data from a single time point, our model makes use of time series data, which lends context to data that would otherwise be interpreted in isolation. This likely accounts for the increased discriminative ability of the model described in this paper.

Microbiological and MAR data contributed little to model predictions in our study. Models trained on such data alone did retain predictive ability but with poorer performance compared to vital sign or laboratory data when each was used in isolation. In contrast, work recently published by Tran-The et al, who describe a model for recommending antibiotic discontinuation, oral switch, or de-escalation, places greater emphasis on specific classes of antibiotics as well as susceptibility data. ^{Reference Tran-The, Heo and Lim21} Beyond slightly different use cases for the models, this may be in part due to differences in the time between antibiotic initiation and ASP review. 60% of orders in our data set were less than 48 hours old with a median of 1.2 days, whereas data described in the authors’ paper had a mean antibiotic duration of 13.2 days in the discontinuation arm, 8.2 in their early de-escalation, and 17.3 in their late de-escalation (ibid). Microbiological data are unlikely to yield actionable information in fewer than 48 hours, a period in which decisions about antibiotics are often made empirically and influenced by markers of clinical instability or active infection. Sample size likely also influenced which features Goodman et al (2022), in a similar paper modeling antibiotic appropriateness, were able to make use of. ^{Reference Goodman, Heil, Claeys, Banoub and Bork22}

Ultimately, the goal of a model such as this one would be to use an automated pipeline to triage antimicrobial orders in order for ASP staff to make better use of finite resources by prioritizing orders for review. Surveys of antibiotic use in United States hospitals suggest more than half of patients admitted will receive antibiotics, with a median of 776 days of therapy per 1,000 patient-days. ^{Reference Polk, Fox, Mahoney, Letcavage and MacDougall23} Given the high proportion of patients receiving antibiotics, manually reviewing all orders is impractical, and our institution’s current approach is to select limited subsets of the hospital at a time in tandem with periodic point prevalence studies. While local implementation might vary in its specifics, the core data used by a model such as this one would be able to be updated on a daily basis, providing a regular, automated means for directing ASP staff’s attention to high-impact areas. This might extend ASP reach to a much broader swath of hospitalized patients.

Several limitations pertain to this model. While NAPS is a standardized and validated tool, inter-rater variability is higher at our institution for infectious syndromes with clear diagnostic and treatment guidelines compared to those where more weight is placed on clinical gestalt. ^{Reference Bystritsky, Gruenberg, Abdoler, Hilts-Horeczko and Doernberg24,Reference Khanina, Douglas and Yeoh25} In cases such as documented bacteremia where diagnostic criteria are straightforward, data labeling—and therefore model predictions—are likely more reliable than in less clear-cut instances like undifferentiated sepsis. Sample size, use of hospital medicine patients for training, and the single-center nature of the model, both of which reflect the time-intensive nature of ASP review, limit the model’s generalizability and complexity. Many potentially useful features, such as considering antibiotics or microbes individually rather than by spectrum of coverage, would also be more feasible with a larger sample size. While we chose to focus on clinical variables relevant to the need for antibiotics and the spectrum of coverage required, several other factors contribute to “optimal” use, including considerations around patient comorbidities, toxicities, and social factors that are not accounted for in the model. External validation, both in academic and community settings, would strengthen the case for widespread use of a model such as this one.

From an implementation standpoint, while the model was optimized to identify high-yield orders for ASP review, unstructured data such as clinical notes detailing the prehospitalization use of suppressive/prophylactic antibiotics might lead to divergent assessments. Structured data such as vital signs are key factors in identifying and diagnosing infection. However, clinical notes, radiology reports, and pathology reports lend additional context—nuances gleaned from these sources cannot be accounted for by our model. Large language models, a type of neural network that can be appropriated for text classification tasks, could be used to abstract some of this information if given a large enough corpus of training data and represent a promising development in future efforts to more effectively implement stewardship interventions, although outstanding questions remain regarding privacy and regulatory structures. ^{Reference Bhayana26,Reference Yang, Liu and Deng27}

In summary, we describe an approach toward automatically reviewing new inpatient antibiotic orders using machine learning. In contrast to labor-intensive manual review, this model uses data that can be abstracted from the medical record without the need for clinician involvement. The model’s higher throughput might be used to optimize ASP teams’ limited staffing resources; antibiotics predicted to be non-indicated or suboptimal could be prioritized, freeing up ASP resources for other tasks. Bearing our model’s strengths and limitations in mind, it is best framed as a tool for helping ASP teams triage the highest-yield orders for PAF, and defer review of those least likely to need intervention. Future extensions of this work include incorporating additional data sources, validation in broader practice settings, and investigating similar models’ impact when used to provide direct feedback to providers.