The 2019 American Thoracic Society/Infectious Diseases Society of America (ATS/IDSA) guideline for management of patients with community-acquired pneumonia (CAP) recommends treating hospitalized adults empirically with a respiratory quinolone or a β-lactam plus macrolide. Reference Metlay, Waterer and Long1 However, in 2%–4% of patients, the infecting pathogen demonstrates resistance to these antibiotics. Reference Aliberti, Cilloniz and Chalmers2,Reference Gross, Van Schooneveld and Olsen3 The challenge for clinicians is to predict (without benefit of culture results) which patients are likely to harbor resistant organisms and require empiric broad-spectrum agents, while simultaneously avoiding overutilizing these antibiotics. Several models have been developed to aid in this prediction, but they were validated on small samples or not at all. Reference Madaras-Kelly, Remington, Fan and Sloan4–Reference Ceccato, Mendez and Ewig10 The best studied is the Drug Resistance in Pneumonia (DRIP) score, Reference Webb, Dascomb and Stenehjem8 whose creators noted that their findings required confirmation in a larger cohort. Indeed, in a small validation study, the DRIP score was inferior to clinical judgment. Reference Babbel, Sutton, Rose, Yarbrough and Spivak11
The ATS/IDSA guidelines recommend that hospitals derive their own local risk factors for resistant organisms but acknowledge that most hospitals will be unable to do this. Reference Metlay, Waterer and Long1 Alternatively, they recommend using broad-spectrum antibiotics for patients with methicillin-resistant S. aureus (MRSA) or Pseudomonas infection in the past year, or hospitalization with intravenous antibiotics in the past 90 days (the “2-factor rule”).
We sought to validate the DRIP score in a large sample, to create and validate an improved risk assessment model, and to compare our model and the DRIP score to both the prescribing behaviors of physicians and the simplified 2-factor rule recommended by the ATS/IDSA guidelines. Reference Metlay, Waterer and Long1
Methods
In this retrospective cohort diagnostic study, we derived a CAP antibiotic resistance model (CARM) to predict resistance to the antibiotics recommended for empiric treatment of CAP. We assessed its performance internally by repeated 10-fold cross validation and in a random 20% holdout sample and an independent data set. We then compared its performance to that of 2 previously published approaches: (1) the DRIP score Reference Webb, Dascomb and Stenehjem8 and (2) the 2-factor rule from the 2019 CAP guidelines. The study was approved by the Cleveland Clinic Institutional Review Board.
Derivation set
We derived the CARM using the Premier Healthcare Database (Premier, Charlotte, NC), which is frequently used for research. Reference Lindenauer, Pekow, Gao, Crawford, Gutierrez and Benjamin12,Reference Rothberg, Pekow, Lahti, Brody, Skiest and Lindenauer13 Premier hospitals are located throughout the United States. They vary in size, and they include nonprofit and nongovernmental, urban and rural, and community and academic medical centers representing ∼25% of US admissions. 14 Data elements include sociodemographic information, International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnosis and procedure codes, hospital and physician information, treatments received, source of admission, and discharge status. A subset of hospitals also provides microbiology data, including cultures, organisms, and antibiotic sensitivity testing. We examined admissions between July 2010 and June 2015 to any of the 177 hospitals providing such data of patients ≥18 years old with a principal diagnosis of pneumonia; or with a secondary diagnosis of pneumonia paired with a principal diagnosis of respiratory failure, ARDS, respiratory arrest, or sepsis; and who had blood or respiratory cultures collected on admission. We studied all patients with CAP, but we excluded patients with hospital-acquired (HAP) or ventilator-associated pneumonia (VAP). 15 Interhospital transfers and admissions with a diagnosis code for an alternate site of infection (eg, cholecystitis) were also excluded. For patients with multiple hospitalizations, we included 1 randomly selected admission.
Predictor variables
We considered 43 candidate predictors of antibiotic resistance, many of which have been shown to be associated with resistant organisms: sociodemographics [ie, age, sex, race (White, Black, or other)], HCAP factors (eg, admission from skilled nursing facility, dialysis, immunosuppression); lifestyle factors (eg, alcohol abuse, drug abuse, smoking); previous hospital and inpatient antibiotic exposures in the past year; 16 comorbidities; and severity of disease indicators (ie, intensive care, invasive mechanical ventilation, vasopressor, wound care, tube feeding) on the first hospital day. Comorbidities were identified from ICD-9-CM secondary diagnosis codes and diagnosis-related groups using Healthcare Cost and Utilization Project Comorbidity version 3.1 software, based on the work of Elixhauser. Reference Elixhauser, Steiner, Harris and Coffey16 Lifestyle factors were based on ICD codes or billing for nicotine replacement therapy.
Outcome variable
The primary outcome was growth of a resistant organism from any blood or respiratory source other than the nares. Cultures for which an organism and antibiotic sensitivities were reported were considered positive. We excluded probable contaminants (eg, coagulase negative Staphylococcus) and organisms not known to cause pneumonia (eg, Enterococcus spp). Resistance was determined by participating laboratories in accordance with clinical microbiology laboratory standards. 17 Any organism resistant to either a quinolone or the combination of a third-generation cephalosporin and a macrolide was considered resistant to CAP therapy. Patients whose cultures did not grow antimicrobial-resistant organisms were considered negative for antimicrobial resistance, including patients who grew no organisms and those who had no sample via a particular route. For example, patients without respiratory cultures were assumed not to have a resistant organism in the respiratory culture.
Validation sets
We validated the CARM in both a 20% random Premier holdout sample and data from 10 northeastern Ohio hospitals and 1 Florida Cleveland Clinic Health System (CCHS) hospital. The latter included all adult patients discharged between 2017 and 2019, with the same inclusion criteria as our Premier data set. Data elements were derived using electronic chart review to match, as closely as possible, the corresponding elements from the Premier database. CCHS does not contribute to Premier, and some variable definitions required adaptation. Hence, this secondary database provided the opportunity for a fully independent external validation.
Statistical analysis
We randomly split the Premier data into a “derivation set” consisting of 80% of hospitalizations for model development, and a “validation set” of the remaining 20%, used only to assess model performance. In the derivation set, we used multiple logistic regression with successive backward elimination of variables, maximally reducing the Akaike information criterion (AIC) Reference Akaike18 until no additional removals reduced this statistic. The resulting model was further simplified by removing variables that contributed little to discrimination (as measured by the C-statistic) based on prevalence, odds ratio, and clinical and statistical judgment. Reference Kattan and Gerds19
We used 20 repeats of 10-fold cross validation to estimate the model C-statistic in the derivation set. We applied the CARM’s linear predictor and the DRIP score to both validation sets. In each set, we plotted their receiver operating characteristic (ROC) and cumulative gain curves (fraction of patients with resistant organisms vs fraction of all patients flagged for resistance by the model), then we plotted calibration by deciles. We superimposed recommendations of the 2-factor rule and the actual care provided (ie, proportion of patients who received broad-spectrum antibiotics). In each set, we also examined slopes and intercepts of logistic regressions of resistance on the CARM’s linear predictor, and we compared the C-statistics of the CARM and DRIP score using the test developed by Kang et al. Reference Kang, Chen, Petrick and Gallas20 SAS version 9.4 software (SAS Institute, Cary, NC) was used for data management, and we used R version 3.5 and version 4.0.0 software and SAS/STAT version 15.1 software for statistical analyses.
Results
Of 151,715 adult patients with pneumonia who met enrollment criteria in the Premier database, 138,762 (91.4%) had blood or respiratory cultures on admission and were included in the analysis (Fig. 1). Of these 12,181 (8.8%) had a positive culture, and 5,200 (3.8%) had organisms resistant to CAP therapy. Table 1 compares the demographics and comorbidities of patients with and without resistant pathogens. Table 2 shows the CARM predictors, odds ratios, and 95% confidence intervals, and Supplementary Table 1 (online) shows their prevalence among CCHS patients, where resistant organisms were found more than twice as often (8.4% vs 3.8%), as in Premier patients. Infection with an organism resistant to CAP therapy in the previous year was the strongest predictor of resistance, and markers of acute illness (eg, need for mechanical ventilation, vasopressors) and chronic illness (eg, pressure ulcer, paralysis) also predicted resistant infections. Smoking was inversely associated with resistance.
Flow chart-inclusion and exclusion from cohort.
Comparison of Patients with and without Resistant Infections in the Derivation Set
Note. MRSA, methicillin-resistant Staphylococcus aureus; SNF/ICF, skilled nursing facility or intermediate care facility; ICU, intensive care unit; IMV, invasive mechanical ventilation.
a Units unless otherwise noted.
Odds Ratios for Factors in the Cleveland Clinic Antibiotic Resistance Model (CARM) for Resistance to Community-Acquired Pneumonia Therapy in the Derivation Set
Note. CI, confidence interval.
a All factors are present at the time of admission. C-statistic = 0.70.
b Resistant either to a third-generation cephalosporin, ampicillin, or ertapenem, and a macrolide or to a fluoroquinolone.
Both the CARM and the DRIP score performed better in the Premier validation set than in the CCHS validation sample. Their respective C-statistics in Premier were 0.71 for the CARM versus 0.63 for the DRIP score (P < .001) (Fig. 2A), and in the CCHS data, the C-statistics were 0.65 and 0.58, respectively (P < .001) (Fig. 2B). Calibration plots for risk deciles of the CARM are shown in Figure 3 and for the DRIP score in Supplementary Figure 1 (online). Observed risk for the CARM in the Premier data ranged from 1.38% in the lowest decile of predicted risk to 12.71% in the highest (Fig. 3A). However, in the CCHS setting, the intercepts and slopes of both models were miscalibrated, as evident in the highest deciles in Figure 3B, where both CARM and the DRIP score overpredicted. Supplementary Figure 2 (online) shows improvement, particularly for the CARM, after recalibrating by adjusting the intercepts and slopes of the linear predictors.
Receiver operating characteristic curves for the CARM in the derivation (2A) and validation sets (2B).
Calibration plots for the CARM in the derivation (3A) and validation sets (3B).
Figure 4 shows the cumulative gain curves for each validation set, demonstrating the tradeoff between the use of broad-spectrum antibiotics (on the x-axis) and fraction of patients with antimicrobial-resistant organisms adequately covered (on the y-axis). The curves include different cutoffs for each model, as well as the 2-factor rule and the actual use of broad-spectrum antibiotics by clinicians. Smaller Premier hospitals used fewer broad-spectrum antimicrobials than larger hospitals. Clinicians at Premier facilities used broad-spectrum antibiotics for 20%–30% of patients, corresponding to a DRIP score cutoff between 2 and 3 (Fig. 4A). At any cutoff, CARM probability was superior to both physician performance and the DRIP score, with physicians slightly outperforming the DRIP instrument. For example, using a DRIP cutoff of ≥2 risk factors to initiate broad-spectrum antibiotics would have covered 53.5% of resistant organisms but would have required treating 34.1% of all patients with broad-spectrum antibiotics. Using a 4% threshold on the CARM, the same 53.5% coverage could have been obtained while treating only 25% of all patients. Using a DRIP cutoff of ≥4 would reduce the use of broad-spectrum antibiotics substantially but would also leave >80% of resistant infections uncovered. Using a predicted probability of >11.5% on the CARM could achieve the same coverage while treating fewer patients.
Cumulative gain curves for the CARM, DRIP score and 2-factor rule in the derivation (4A) and validation sets (4B). Utilization of broad-spectrum antibiotics for patients with antimicrobial-resistant organisms versus all patients: the % of patients with resistant organisms is on the Y axis, the % of patients receiving broad-spectrum antibiotics is on the X axis, showing the tradeoffs between these 2 factors. The blue line shows the tradeoffs using different thresholds of the CARM, the red line shows the DRIP score factors, and the purple Xs show the average among small, medium and large hospitals in the Premier data set. Panel A: Premier data. Panel B: Cleveland Clinic data.
In CCHS data, clinician performance again fell between that of the DRIP score and the CARM, although considerably more patients (57.3%) were treated with broad-spectrum antibiotics because the population had more risk factors for resistance, reflecting the more complex CCHS patient mix (Fig. 4B). The physicians’ prescribing of broad-spectrum antibiotics suggested a DRIP score threshold of 2.
Discussion
From this large national sample of 177 hospitals admitting 138,948 patients with pneumonia, we developed a risk prediction model, the CARM, that outperformed the DRIP score and the 2-factor IDSA rule in both our holdout validation set and an independent CCHS sample. Importantly, the CARM outperformed the physicians’ actual antibiotic prescribing, which the DRIP score did not, in both data sets. Infection with an antibiotic-resistant organism in the prior year was strongly predictive of current resistance. Several markers of illness severity, including need for invasive mechanical ventilation and vasopressors, and markers of chronic illness such as presence of a pressure ulcer, Reference McGee, Nathanson, Lederman and Higgins21 were also important predictors.
Our work builds on that of Webb et al, Reference Webb, Dascomb and Stenehjem8 who derived the DRIP score from a cohort of 200 patients in a single geographic region and validated it in 200 patients from 4 geographically distinct tertiary-care hospitals. The DRIP score’s reported high C-statistic, sensitivity, specificity, and positive and negative predictive values were notable, but given the small sample sizes, required validation within a larger cohort. Our findings confirm some associations in the original study, but the DRIP score’s C-statistic dropped from 0.88 in the authors’ validation sample to 0.63 and 0.58 in our 2 validation samples. This degree of degradation in model performance is often seen when models are applied in other settings, which is why external validation is so important. We developed a model with better discrimination based on somewhat different variables, which are also readily available to clinicians. Our study’s strengths include the large sample size, nationally representative data, and multiple data elements examined. We improved on previous attempts to derive such models, the limitations of which are nicely summarized by Webb et al. Reference Webb, Dascomb, Stenehjem and Dean22
Our analysis differs from the original DRIP score derivation and validation in 2 important ways. First, Webb et al enrolled only patients with positive cultures to derive predictions for the risk of antibiotic resistance. Clinically, however, the score is applied to patients before culture results are known, even though a sizable majority will have negative cultures. In contrast, the new model predicts the combined outcome of a positive culture and an antimicrobial-resistant organism among all comers, congruent with the clinical situation faced by clinicians. As a result, most patients have low predicted risks of antimicrobial-resistant infection; only ∼1-in-10 has a predicted risk exceeding 10%. In our CCHS external validation sample, both the CARM and DRIP score overestimated absolute risk among high-risk patients, but the DRIP score overestimated it by somewhat more.
Second, we tested our model against physicians’ actual prescribing. The purpose of clinical prediction models is to improve on clinical judgment. In the case of antimicrobial resistance, models should help to increase the percentage of patients with resistant organisms who receive adequate coverage, while reducing the unnecessary use of broad-spectrum antimicrobials. Reference Falcone, Tiseo and Menichetti23 The DRIP score was originally compared to risk factors for healthcare-associated pneumonia (HCAP) Reference Webb, Dascomb and Stenehjem8 but not physician practice. Our findings comport with another VA analysis showing that the DRIP model could improve stewardship relative to the HCAP factors but not compared to physician judgment. Reference Babbel, Sutton, Rose, Yarbrough and Spivak11 In that study, following the DRIP score would have unnecessarily increased use of broad-spectrum antibiotics by 9% without increasing the proportion of patients with antimicrobial-resistant organisms who received them. In contrast, the CARM outperformed the DRIP and clinical judgment in 2 validation cohorts and can therefore aid in tailoring appropriate empiric therapy.
An inescapable tradeoff exists between the imperative to prescribe empiric broad-spectrum antibiotics to patients with antimicrobial-resistant infections and the importance of avoiding them for patients without antimicrobial-resistant infections. Although models can help to shift the balance between these 2 groups, no model can perfectly distinguish between them. Physicians must decide how many patients with antimicrobial-resistant organisms they are willing to let go initially untreated in order to help other patients avoid unnecessary broad-spectrum antibiotic exposures. In comparing the 2 models, we plotted several potential thresholds of risk for antimicrobial-resistant organisms that would spur treatment with broad-spectrum antibiotics. Choosing a lower threshold (ie, fewer DRIP score points or a lower model-predicted probability) would identify more patients with resistant infections but would require treating more patients overall. Increasing the threshold does the reverse. The optimal threshold would balance the risks of resistant infections being missed, and therefore undertreated, against the risks of overtreating susceptible ones, weighting each for the severity of the misclassification. Such optimization could be done with a decision analytic model, but such a model does not yet exist.
The DRIP-score developers recommended treating patients with a score ≥4, which in both of our samples would sharply reduce physician use of broad-spectrum antibiotics but would cover smaller proportions of patients with antimicrobial-resistant infections than current practice, yielding performance almost identical to the much simpler 2-factor rule. Based on their prescribing behavior, physicians in both of our samples chose a lower implied threshold, between 2 and 3 DRIP points. Capturing a similar fraction of patients with antimicrobial-resistant organisms requires a predicted risk of only 4% to 5% from the CARM, suggesting that physicians are generally more concerned about missing an antimicrobial-resistant infection than about overtreating. Individual hospitals will have to choose their own thresholds, but our cumulative gain plots should help clarify what that decision will mean in terms of balancing these 2 concerns.
Our study had several limitations, primarily related to the data sources used. We developed the model from an administrative database that did not include specific laboratory values or vital signs, inclusion of which could potentially strengthen future versions of the model. Cultures in this study were obtained in real-world settings during routine patient care, and we were unable to determine why some patients did not have cultures performed. Hence, selective use of cultures—respiratory cultures in particular—could have introduced bias. Similarly, antimicrobial-resistant organisms could have been missed. Our diagnoses and comorbidities were based on ICD codes, which are subject to differential coding at different hospitals and coding drifts over time. Some patients may have received outpatient antibiotics that could have decreased culture yields, although we somewhat mitigated this problem by excluding patients who were transferred from other hospitals. The CARM was developed from 2010–2015 Premier data and performed less well on 2017–2019 CCHS data; this could reflect temporal or geographical changes in resistance or diagnostic coding differences. Finally, it is impossible to determine antibiotic susceptibilities for any organism for which no CLSI break points exist, Reference Jorgensen and Hindler24 and thus, these patients were excluded.
In conclusion, the CARM outperformed both the DRIP score and the 2-factor rule proposed by the ATS/IDSA guidelines in a national database and an independent validation sample from a multihospital healthcare system. The CARM may therefore be the model of choice for hospitals that cannot develop their own models. Further study is needed to determine whether locally derived models, as suggested by the updated CAP guidelines, outperform the CARM, and whether implementation of any such models improves patient outcomes. The 12-factor CARM is available at https://riskcalc.org/PneumoniaAntibioticResistance/ and could be incorporated into the electronic health record as clinical decision support.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/ice.2022.229
Acknowledgments
Financial support
This study was supported by the Agency for Healthcare Research and Quality (grant no. R01HS024277), which had no role in the study design, conduct, or reporting. All authors except Dr. Higgins received salary support or consulting fees from this grant.
Conflicts of interest
Dr. Marya Zilberberg reports grants from Merck, grants from Lungpacer, Tetraphase, The Medicines Company, J&J, and Spero Therapeutics, and personal fees from Nabriva, Melinta, and Paratek outside the submitted work. Dr. Haessler reports her role as an SHEA board member for the 2020–2022 term. Dr. Deshpande reports personal funding from Merck and Ferring Pharmaceuticals, both outside the submitted work. No other authors have potential conflict(s) of interest to disclose.
