Although central-line–associated bloodstream infection (CLABSI) remains one of the core healthcare-associated infection (HAI) measures reported to the US Centers for Disease Control and Prevention (CDC) National Healthcare Safety Network (NHSN), CLABSIs represent only a proportion of all hospital-onset bacteremia and fungemia (HOB). Reference Goto and Al-Hasan1–Reference Page, Klompas and Chan3 Due to decreasing rates and small numbers with low variability, CLABSI may no longer discriminate between facilities as a quality metric. Reference Masnick, Morgan and Sorkin4 In addition, subjectivity limitations of the CLABSI surveillance definition have led to a divergence between reported cases and relevant bedside clinical diagnoses. Reference Lin, Hota and Khan5,Reference Szymczak and Coffin6 As a result, prior work has examined HOB as a broader and more objective HAI outcome measure. Reference Rock, Thom and Harris7–Reference Dantes, Abbo and Anderson9
The assumption that HAIs are preventable forms the basis of HAI measurement and improvement efforts. Reference Warye and Murphy10 Although healthcare facilities use processes such as root-cause analyses to identify and improve upon aspects of care delivery that may have contributed to HAI incidence, there is no standardized method of determining the potential preventability of any type of HAI, and particularly for HOB, which can have an extensive range of clinical etiologies. In the absence of an existing standardized method, we previously developed a risk matrix and used a corresponding 6-point Likert scale to rate HOB preventability based on clinical chart reviews in a pilot study. Reference Dantes, Rock and Milstone8 However, it was unclear whether this method (1) accurately captured preventability and (2) whether results could be replicated between reviewers. Therefore, prior to embarking on a large multicenter study of HOB preventability through the CDC Prevention Epicenters program, we expanded on that previous work to develop a structured and more comprehensive HOB preventability rating guide, with a focus on validity and reliability. 11 The aims of this study were (1) to develop “consensus” on preventability rating of hypothetical clinical scenarios of HOB using expert panel methodology and (2) to evaluate the performance of a structured preventability rating guide against expert consensus.
Methods
To evaluate the validity and reliability of our structured rating guide, it was necessary to first identify a reference standard for the measure of HOB preventability. We used the approach of expert panel consensus on preventability of hypothetical HOB case scenarios as the reference standard against which our structured guide could be compared.
Expert panel composition
We assembled a 10-member panel of clinical experts in hospital medicine, pediatrics, critical care, healthcare epidemiology, and infectious diseases with additional expertise in infections in immunocompromised and community hospital-based practice. Among them, 6 were affiliated with CDC Prevention Epicenters sites participating in the multicenter study of HOB and 4 others identified based on complementary clinical expertise.
Construction of HOB case scenarios
Hypothetical HOB case scenarios were constructed by 4 infectious diseases physicians and healthcare epidemiologists (G.M.S., A.S.S., S.B., and S.L.) based on cases at their institutions, aiming for observed distributions of HOB sources and causative organisms in prior research and clinical experience. Reference Magill, O’Leary and Janelle12,Reference Wisplinghoff, Bischoff and Tallent13 Scenarios were outlined using a standard template (Supplementary Fig. 1 online). In total, 82 HOB case scenarios were created, representing a spectrum of sources of HOB attribution in adult and pediatric patients (Table 1).
Study protocol of hospital-onset bacteremia and fungemia (HOB) scenario review, discussion, and rerating by the expert panel (left) and review, independent feedback, and rerating by the physician reviewers (right).
Categorization and Causative Organisms of Hospital-Onset Bacteremia and Fungemia (HOB) Hypothetical Clinical Case Scenarios Constructed by Infectious Disease Physicians and Healthcare Epidemiologists for Review by an Expert Panel and Independent Review by Physician Reviewers Using a Structured Rating Guide
Expert panel rating of HOB case scenarios and consensus development
First, each expert panelist was asked to independently review and rate each HOB case scenario using a 6-point Likert scale of preventability ranging from “definitively or almost certainly preventable” (1) to “definitively or almost certainly not preventable” (6). Reference Dalkey and Helmer14 This scale is modeled on preventability rating of adverse events in health care, Reference Schwendimann, Blatter and Dhaini15 and it has previously been used in healthcare epidemiology in preventability evaluations of sepsis-related mortality, healthcare-associated bloodstream infections at a French university hospital, and in our pilot study of HOB preventability in the United States. Reference Dantes, Rock and Milstone8,Reference Rhee, Jones and Hamad16,Reference Bonnal, Mourvillier and Bronchard17 Experts were instructed to assess preventability as (1) unintended bacteremia or fungemia related to medical management, (2) as a failure to meet expected standard of care while accounting for a patient’s risk of infection, or (3) as a deficiency of known preventive measures and standards of care including diagnosis and management.
We used guidance outlined in the RAND/UCLA Appropriateness Method User’s Manual Reference Bernstein18 to define agreement, disagreement, and development of expert consensus, using 3 clinically meaningful categories of preventability ratings based on the 6-point Likert scale: preventable (1–2), uncertain (3–4), or not preventable (5–6). We defined the level of agreement among the 10-member expert panel as follows: disagreement occurred when ≥20% of experts rated a scenario 1–2 (preventable) and ≥20% of experts rated the same scenario 5–6 (not preventable). In the absence of disagreement, consensus was concurrence on the same preventability category among ≥70%, and majority agreement was concurrence on the same preventability category among >50%. Case scenario review and rating were conducted independently by each expert panelist using an online survey and data collection tool (Qualtrics, Provo, UT). We created summaries of ratings for each scenario including median, 75th percentile, and whether criteria for consensus, majority agreement, or disagreement were met. A distribution of ratings by the panel and the individual’s rating in relation to others was provided to each expert (Supplementary Fig. 2 online).
Preventability matrix for rating of hospital-onset bacteremia and fungemia (HOB) case scenarios. Examples of intrinsic risk conditions include desquamating skin condition (high), neutropenia (high), solid-organ transplant >30 days prior (medium), and acute myocardial infarction (low). Examples of preventability relative to extrinsic healthcare-related risk include temporary central venous catheter infection (high), pressure ulcers that develop or worsen during the hospital stay (high), mechanical ventilation complicated by pneumonia (medium), and infection following contaminated or dirty surgical procedures (low).
Expert panel discussions
Using Delphi methodology, case scenarios that did not meet consensus criteria were deliberated by the expert panel via 2 teleconferences and a web-based discussion board (Slack Technologies, San Francisco, CA) moderated by the principal investigator (S.L.). and and Reference Rock, Thom and Harris7,Reference Dantes, Rock and Milstone8,Reference Dalkey and Helmer14 During discussions, panel members were asked to voluntarily offer rationale for their choices. Individual ratings were not disclosed to the group by the moderator unless explicitly offered by the individual themselves. Teleconferences were recorded with permission from all participants. Following deliberation, a second round of independent review and rating was performed of the scenarios that did not meet consensus criteria during the first round (Fig. 1).
We reviewed the content of teleconference and web-based discussions to inform the development and refinement of the structured rating guide and to provide insight into the aspects of the case that were deemed to be preventable.
Structured preventability assessment guide development
In a previous pilot study of HOB preventability assessment, we conceptualized the risk of developing HOB as a function of “intrinsic” or patient-related factors and “extrinsic” or healthcare-related factors. To explicitly account for both components of HOB risk, we developed a matrix to separately assign a patient’s intrinsic risk of HOB and preventability related to extrinsic healthcare factors. The overall preventability of HOB was then derived as the combined effect of the intrinsic and extrinsic risk rating from this matrix and translated to the 6-point Likert scale (Fig. 2). During the pilot study, limited guidance around classification of intrinsic risk and extrinsic preventability was provided and variability in reviewer assessment of risks was recognized as a limitation. Reference Dantes, Rock and Milstone8 To overcome that limitation and minimize subjectivity of ratings, we created a more comprehensive list of clinical conditions and developed the present HOB preventability rating guide (Supplementary Appendix online).
In this guide, intrinsic risk guidance was based upon the acute illness and underlying comorbidities present upon index admission and prior to the HOB event, informed by both literature review and known pathogenesis of bloodstream infections related to these conditions. Extrinsic risk accounted for preventability relative to the suspected source of infection. The guide was not made available to the reference standard expert panel.
Independent review with structured preventability rating guide
Two additional healthcare epidemiology experts (A.D.H. and J.J.) were asked to independently review and rate the same 82 hypothetical case scenarios utilizing the structured guide. For each scenario, reviewers assigned an intrinsic and extrinsic health-care related risk of HOB using the guide, which informed the overall HOB preventability rating. The 2 reviewers were asked to provide justification for their decision making for each rating. Summaries of each scenario, including the difference between reviewers’ overall preventability rating, intrinsic risk, and extrinsic healthcare-related risk assignments, were created.
Following the initial round of scenario rating with the structured guide, 2 investigators (G.M.S. and S.L.) performed review of all scenario ratings where agreement between the 2 reviewers was not achieved and provided feedback to reviewers on any notable deviations from intended use of the guide. Reviewers’ ratings were maintained as independent and not shared with the other. Reviewer feedback on the structured guide was also incorporated and the guide was refined to improve usability. This procedure was followed by repeated independent review and rating of scenarios by the same 2 reviewers (Fig. 1).
Statistical analysis
For the purposes of analysis, HOB scenarios were categorized by predefined sources of infection, and Likert ratings were assigned to the appropriate category of preventability as previously described: 1–2 (preventable), 3–4 (uncertain), or 5–6 (not preventable). For expert consensus development, descriptive analyses of expert panel and reviewer ratings were performed and summarized as consensus, majority agreement, and disagreement after each round of review. For guide-based rating by the 2 independent reviewers, results were compared to those of the expert panel as a reference standard. The proportion of cases (and 95% confidence intervals) where guide-based preventability category (ie, 1–2, 3–4, or 5–6) matched expert consensus was used to calculate overall agreement, sensitivity, and specificity by preventability category. These calculations were only performed for cases in which expert consensus was achieved, given the lack of an available reference standard in the absence of that consensus. Interrater reliability of ratings by the 2 reviewers using the structured rating guide was evaluated using the κ statistic using the same preventability categories. Reference Landis and Koch19,Reference Viera and Garrett20 Analyses were performed using SAS version 9.4 software (SAS Institute, Cary, NC). This study was approved by the Institutional Review Board of the University of Maryland, Baltimore.
Results
Expert panel scenario rating
After the first round of expert panel preventability rating, 29 scenarios (35%) met criteria for consensus; an additional 17 (21%) had majority agreement. Moreover, 19 scenarios (23%) met criteria for disagreement. During panel discussion of scenarios that did not meet consensus criteria, a number of themes emerged regarding the assessment of HOB preventability. These are summarized with illustrative quotations in Table 2.
Themes That Emerged During Hospital-Onset Bacteremia and Fungemia (HOB) Scenario Rating Discussion by Expert Panel Along With Selected Quotations From Individual Experts
Following the second round of expert panel rating, 52 scenarios (63%) met criteria for consensus: 14 (27%) preventable, 23 (44%) uncertain, 15 (29%) not preventable. Overall, 85% of scenarios had at least majority agreement, and 2 (3%) met criteria for disagreement. Expert consensus was achieved more frequently for HOB attributed to contaminants, respiratory, and surgical site infection sources than for urinary tract and central lines (Fig. 3).
Predefined sources of hospital-onset bacteremia and fungemia (HOB) and associated ability of expert panel to achieve consensus on preventability after 2 rounds of rating.
Note. PIV, peripheral intravenous.
Independent review using structured rating guide
Following an initial round of adjudication using the rating guide, there was agreement between the 2 reviewers in 45 (55%) scenarios (κ, 0.31; 95% CI, 0.12–0.49). Reviewers acknowledged challenges in separating the concept of intrinsic from extrinsic risk in several cases. Key issues included prolonged hospitalization, patients with numerous comorbidities with dependence on multiple indwelling devices and for whom the intrinsic risk was so great that HOB seemed “inevitable,” and scenarios in which there were deviations from the expected standard of safe, quality care delivery such that considering the patient’s intrinsic risk seemed inappropriate when determining HAI preventability. Following the second round of reviewer adjudication, agreement between the 2 reached 84% overall (κ, 0.76; 95% CI, 0.64–0.88). Comparison of preventability ratings between the first and second rounds of review is demonstrated in Supplementary Figure 3 (online).
After 2 rounds of rating, comparing reviewer adjudication of preventability ratings with those of the expert panel, among cases with expert consensus (n = 52), guide-based rating agreed with the expert panel consensus in 40 (77%; 95% CI, 65%–88%) scenarios for reviewer 1 and 39 (75%; 95% CI, 63%–87%) for reviewer 2. For both reviewers, the sensitivity and specificity were higher for cases rated as preventable or nonpreventable, and lower for expert panel ratings in the uncertain range (Table 3). Interreviewer agreement was 87% (κ, 0.79; 95% CI, 0.65–0.94).
Performance Characteristics of the Structured Preventability Assessment Guide for the Rating of Hospital-Onset Bacteremia and Fungemia (HOB) Hypothetical Clinical Case Scenarios Compared to Reference Expert Panel With Consensus
Discussion
We developed a structured preventability rating guide and evaluated its performance for standardized assessment of HOB scenarios in comparison with consensus preventability ratings by a panel of healthcare epidemiology and clinical experts. Using hypothetical HOB clinical scenarios, we identified varying perspectives on HOB preventability and through Delphi methodology were able to achieve consensus in 63% of scenarios after 2 rounds of rating. Preventability ratings of these HOB case scenarios by 2 independent reviewers using a structured preventability rating guide matched expert consensus in most cases and demonstrated moderately high interreviewer reliability.
Iterative rating and discussion of HOB scenarios by the expert panel demonstrated differing perspectives and opinions regarding the preventability of HOB; only 35% of scenarios achieved consensus after the first round of rating. Similarly, even with the structured rating guide, initial agreement between the 2 reviewers was only 55% and despite individualized feedback to ensure use of the guide as intended, differences in assessments of preventability persisted. Complex clinical cases in which there is pre-existing uncertainty—both in pathogenesis of infection and relative contribution of intrinsic and extrinsic risks—inevitably lead to differing opinions with regards to preventability and challenges in establishing consensus. This potential disagreement was also noted in a prior French study of preventability of healthcare-associated bloodstream infections in which 49 (25%) of 196 cases with an identifiable source and related to medical management were of uncertain preventability. Reference Bonnal, Mourvillier and Bronchard17 Our findings are also consistent with prior literature showing that when evaluating other non-HAI adverse events in healthcare, the level of agreement, even among expert reviewers, is not high. Reference Nabhan, Elraiyah and Brown21 Open discussion amongst experts provided some insight into reasons for this observation. These included lack of agreement on the underlying source of infection, differences in perceived preventability of specific sources of infection, and the interaction of patient risk with healthcare risk. An expert’s clinical familiarity with a particular field or specialty may also contribute. For example, a pediatrician may feel less confident in determination of preventability in an adult HOB scenario and vice versa. Lastly, collapsing the Likert scale ratings from 6 ordinal levels to 3 to determine expert consensus may also contribute to lack of agreement via measurement distortion due to some neighboring ratings being combined and other neighboring ratings being placed in separate ordinal categories.
Our goal in creating and applying a risk matrix to HOB scenarios was to reduce subjectivity and variability in assessing both the risk related to underlying conditions and perceived preventability. We believe that we were able to accomplish these objectives, as demonstrated both by moderate to high agreement of guide-based rating to expert consensus reference standard as well as interreviewer agreement for guide-based rating. However, the rating guide does not eliminate the subjectivity and uncertainty inherent in this assessment. In this regard, there were 2 key findings in our study. First, the reference standard of “expert consensus” against which the guide-based rating was compared was itself an elusive entity for more than one-third of the cases. Second, the guide performed well for the 2 extremes of preventable and nonpreventable cases but not for those considered uncertain by the expert panel. This reflects current paradigms of HAI preventability that are largely focused on those associated with specific devices or procedures, whereas an understanding of preventability across the entire spectrum of HOB sources which have not been a focus of prior surveillance efforts (eg, non–catheter-associated UTI or non–ventilator-associated pneumonia) is relatively lacking in comparison. The risk matrix additionally provided insight into whether differences in ratings by reviewers were driven by differences in patient risk versus extrinsic provision of medical care. For example, one reviewer consistently rated the intrinsic risk of patients at a higher level, thereby leading to more scenarios being categorized as “not preventable” compared to their counterpart reviewer.
Despite these limitations, there are advantages to using a standardized risk matrix both for surveillance and as a tool when communicating with frontline clinicians. From a surveillance perspective, measuring and understanding the “preventable” portion of HOB cases can be the basis for reducing clinically meaningful HAIs. Identifying HOB cases in which there is uncertainty about preventability can provide the impetus for the healthcare epidemiology field to focus on novel prevention measures based on the underlying sources of those cases.
Our findings of expert reviewer perception of preventability of these scenarios, with only 27% of consensus scenarios identified as preventable, contrasts with the general perception that HAIs are mostly preventable and underlies their inclusion as quality metrics. Our results are more closely aligned with actual data from systematic reviews of studies of interventions to reduce HAIs, which estimate preventability of HAIs in the range of 35%–70% across different infections, settings, and periods of study. Reference Umscheid, Mitchell and Doshi22,Reference Schreiber, Sax and Wolfensberger23 Acknowledgement of the inherent risk contributed by the patient’s presenting conditions prior to HOB events in a standardized way could help bridge the gap between those responsible for oversight of surveillance and prevention activities and bedside clinicians in expectations of the preventability of HAI events. In the future, this structured guide could be adapted for use by clinicians to evaluate HOB cases at the bedside.
This study had several limitations. A major limitation was that the case scenarios represented an artificial collection developed by the investigators and was likely skewed toward more complex presentations that may not be reflective of the entire spectrum of HOB etiologies or patient case mix across all facility types and geographic locations. Additionally, the scenarios were intended to facilitate the development of the structured rating guide rather than assess the preventability of various HOBs by source. A larger study of real-world HOB cases is more appropriate for such analysis. However, we attempted to capture a spectrum of HOB events with respect to underlying conditions, sources, and etiologic microorganisms, including contaminants. Another concern is the inability to develop expert consensus in more than one-third of the cases, which limited the reference standard against which the rating guide could be evaluated. Lastly, in many scenarios, the use of a 70% threshold to define consensus led to a determination of consensus without unanimity, though other infection control and hospital epidemiology researchers have utilized a similar threshold. Reference Meddings, Saint and Fowler24,Reference Bramesfeld, Wrede and Richter25
In summary, using an iterative process, we developed a structured HOB preventability rating guide that allowed a moderately high degree of agreement with a reference standard of expert panel consensus, and between reviewers, in rating preventability. Although the diversity of expert opinions and uncertainty of HOB preventability merit further exploration, we demonstrated that this rating guide can be applied to a broad spectrum of HOB case scenarios, and account for underlying patient risk and the extrinsic aspects of care that contribute to HOB. A larger study of HOB evaluation using this rating guide for actual HOB events across multiple institutions is underway and will provide further insight into preventability based on HOB source and etiologic organism.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/ice.2021.528
Acknowledgments
The findings and conclusions are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
Financial support
This work was supported by the Centers for Disease Control and Prevention Epicenters Program (grant no. U01CK000556-01). S.K.S. is supported in part by the National Institutes of Health and the National Cancer Institute (NIH/NCI grant no. P30 CA008748).
Conflicts of interest
All authors report no conflicts of interest related to this article.
