Algorithm background: development

As a use case to develop this protocol, we consider our extreme gradient-boosting decision tree (XGBoost) algorithm that predicts the results of blood cultures in the emergency department (ED). We first developed the model at the Amsterdam University Medical Center (AUMC) in the Netherlands. ^{Reference Boerman, Schinkel and Meijerink5,Reference Schinkel, Boerman and Bennis6} Its development, including the rationale for feature selection, extraction of data, patient exclusion criteria, handling of missing data, target development as well as its final performance characteristics in earlier external validation sets has been described extensively elsewhere. ^{Reference Boerman, Schinkel and Meijerink5,Reference Schinkel, Boerman and Bennis6} To complement the initial external validation and aid the implementation process in other hospitals, a large-scale external validation will be performed on data from electronic health records (EHR) of all adult patients visiting the ED of the St. Antonius Hospital between January 1^t, 2018, and July 1, 2023. The St. Antonius Hospital is a 500-bed teaching hospital in the Netherlands. On this data set, an in-depth clinical evaluation of potential harms of (retrospective) application of the algorithm will be performed according to the protocol described below.

Inclusion of data

A graphic representation of the inclusion and categorization of cases is depicted in Figure 1. The first step in our proposed protocol is to identify the cases that are at potential risk of harm by retrospectively applying the algorithm to the described data set (Figure 1). Data will be extracted from the Electronic Health Record system (EHR, EPIC, Epic Systems Corporation). We then select cases in which a blood culture yielded a positive result, while the model predicted a probability of positivity below 5% and would therefore recommend not obtaining a culture. This constitutes a false-negative prediction. A probability of less than .05 (5%) was chosen in consensus during the development phase of this algorithm, as higher probabilities of missing a positive blood culture were regarded as clinically unacceptable. After the initial selection, an additional screening is performed to filter out blood cultures which are marked as contaminants in the EHR, which constitute false-positives and should not be considered. Those cultures will be recategorized as true negative. In a final step, a manual check is performed to the remaining cases at potential harm to filter out those in whom the algorithm would not be used in practice (eg, patients with neutropenia, suspected endocarditis, or follow-up cultures in patients with previously positive cultures). Those cultures will be recategorized not eligible for inclusion.” An overview of the counts in the various groups of inclusion and exclusion will be provided, as well as aggregate data on characteristics such as culture positivity rates in those groups. These data specifically may be interesting to estimate case-mix differences between the development setting and our hospital.

Figure 1.

Flowchart to evaluate eligibility of the case for in-depth retrospective clinical evaluation.

Study design: clinical evaluation

The main part of the protocol aims to help identify the possible clinical consequences of false-negative predicted blood cultures in detail through a case-by-case evaluation (Table 1). Through panel discussions, experts in the fields of microbiology, infectious diseases, internal medicine, intensive care, and epidemiology reached consensus on nine questions in three domains to cover the potential (detrimental) consequences of a false-negative blood culture prediction for the individual patient. The domains in which harm might occur because of a false-negative prediction were identified to lie within the areas of antibiotics prescription, diagnostics, and patient outcomes. The panel discussions, though unstructured, were deliberately focused on outcomes that are also being evaluated in a prospective randomized controlled trial using the same algorithm, to ensure alignment of domains for future comparison. ^{Reference Van Der Zaag, Bhagirath and Boerman7} With respect to antibiotics, inappropriate choice of empirical antibiotics or of narrow-spectrum antibiotics (Q1 and Q2, respectively), and insufficient length of antibiotics therapy (Q3) were deemed to be of greatest concern. Within the area of diagnostics, the greatest concern was that unnecessary (invasive) diagnostics would be ordered, causing discomfort for the patient (Q4). Concerns about missing the need for admission, unnecessary additional days in the hospital or worse-than-expected outcomes, were captured in the final three questions (Q5–7). As the probability exists that the patient may still experience harm, despite zero or only minor alterations in therapeutics, diagnostics, or outcomes, two final questions (Q8–9) on general harm and costs were added. For the financial category, the reviewers will be instructed to consider potential biases such as survivorship in their judgment.

Table 1.

Domains and subdomains of interest for clinical evaluation on case-by-case basis

FN, false-negative.

Questions will be scored on a 3-point forced-choice Likert scale without a neutral midpoint. The absence of a neutral midpoint was deliberate, to force the reviewing expert to make a definitive judgment on the case, rather than defaulting to an undecided position. A 3-point scale is thought to simplify the decision process and therefore appeal to time-constrained experts having to judge a high number of cases. Per question, exemplary cases were constructed as a reference to help the medical expert judge the severity of the case and correctly interpret the question. Before consolidation of the questionnaire, two experts each assessed ten cases to judge the correct interpretability of the questions.

Statistical analysis

Two observers (MS and IM) will individually score all false-negative predicted cases. First, each case will be screened to confirm that the case should not be recategorized as “true negative or is not eligible for inclusion. Scores per question will be expressed on a 3-point forced-choice Likert scale without a neutral midpoint and expressed as percentages. Overall scores per question will be calculated per reviewer as well as for both reviewers combined and expressed as percentages.

To identify any possible differences between observers’ ratings, a linearly weighted Cohen’s kappa as well as percent agreement will be calculated for individual scores between the two observers. Krippendorff’s alpha will be calculated for the full questionnaire (Q1–7). As the final questions on general harm and costs are related to all other questions, these will not be included in the Krippendorff alpha calculation. The analyses will be performed in Python (version 3.7 +).

To come to a final consensus on clinical impact of a false-negative prediction per case, small discrepancies between the observers’ ratings will be resolved among the two initial observers. Large, or unresolvable discrepancies in a case, that is more than 2 questions scored on opposite ends of the Likert scale, will be resolved by an expert panel of two independent medical specialists to come to a consensus on a final adjudication. Through these steps, a final consensus on all cases is necessary to present the results.

Ethical considerations

Both the AUMC and St. Antonius Hospital institutional review boards waived the review of this study as the Medical Research Involving Human Subjects Act did not apply (IRB number: IRB00002991; case: 2020.486). No data will be extracted for patients who have registered that their data should not be used in scientific research. Publication of results of this study will adhere to the Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD + AI) reporting guidelines.