Peripherally inserted central venous catheters (PICCs) are a common type of central venous catheters (CVCs) used for hospitalized patients. Reference Chopra, Anand, Krein, Chenoweth and Saint1,Reference Rubin, Apfelbaum and Tung2 However, PICC-associated bloodstream infections (PBSI) have an incidence rate of up to 7.71 per 1,000 catheter days, which prolongs hospital stay and increases costs and mortality. Reference Chopra, Anand, Krein, Chenoweth and Saint1
The Michigan PICC catheter-associated bloodstream infections score (MPC score) is a risk prediction model to estimate an individual’s risk of developing PBSI. Reference Herc, Patel, Washer, Conlon, Flanders and Chopra3 Although the US multicenter derivation cohort involving 23,088 adult medical patients reported a promisingly good discriminative performance (C statistic, 0.67–0.77), Reference Herc, Patel, Washer, Conlon, Flanders and Chopra3 its external validation had yet to be performed. Therefore, in this research, we aimed to validate its discrimination and calibration performance in a Japanese university hospital setting involving all hospitalized patients regardless of the departments to which they were admitted.
Methods
Source of data and participants
This study was a retrospective analysis using routinely collected data from Fujita Health University Hospital, a 1,435-bed, tertiary-care, academic center in Japan. The institutional review board of Fujita Health University approved this study (approval no. HM19-479) and waived the requirement for informed consent.
Between October 2015 and January 2020, all adult (≥18 years old) hospitalized patients, regardless of the departments in which they were admitted, who had PICCs inserted by the dedicated PICC nursing team were prospectively and consecutively registered with the hospital PICC database to survey CVC-related complications. Patients were operationally subcategorized into 4 groups based on the departments to which they were admitted (ie, medicine, surgery, ICU, and others).
Outcome
Following the National Healthcare Safety Network Laboratory Confirmed Bloodstream Infection (LCBI) Checklist, PBSIs were defined as bloodstream infections that satisfied the LCBI-2 criterion. 4
Predictors
Predictors of PBSI (points assigned by the score Reference Herc, Patel, Washer, Conlon, Flanders and Chopra3 ) included (1) past and present history of hematological malignancy (3 points); (2) history of central-line–associated bloodstream infection (CLABSI) within 3 months of PICC insertion (2 points); (3) active cancer with receipt of chemotherapy (2 points); (4) multilumen PICC (2 points); (5) presence of another CVC upon PICC placement (1 point); and (6) receipt of total parenteral nutrition through the PICC (1 point).
Statistical analysis
In this study, we followed the recommended framework for assessing discrimination and calibration in external validation of a Cox prognostic model. Reference Royston and Altman5 Full details of the methods are available as Supplementary Material (online). To compare a clinical “net benefit” among strategies based on the original or updated MPC score and among default strategies performing some interventions for all or no patients, a decision curve analysis was performed. Reference Vickers and Elkin6
In our main analysis, we used the original MPC score and grouped patients into 7 risk categories according to the total points (0, 1, 2, 3, 4, 5, and ≥6 points). Reference Herc, Patel, Washer, Conlon, Flanders and Chopra3 In sensitivity analyses, we used the linear predictor with the originally reported coefficients to calculate the prognostic index.
Results
Patient characteristics
Our case-mix variation explicitly differed from the derivation cohort in the inclusion of patients hospitalized in nonmedical departments, higher patient age, and higher prevalence of malignancy and history of CLBSI (Table 1).
Baseline Characteristics of Derivation and Validation Cohorts
Note. CLABSI, central-line–associated bloodstream infection; CVC, central venous catheterization; PICC, peripherally inserted central catheter; PBSI, peripherally inserted central catheter–associated bloodstream infection.
a Results were for patients who did not develop CLABSI (n = 22,839). The median dwell time for patients who developed CLABSI (n = 249) was 15 days (IQR, 9–26).
Outcomes
Over a median (per catheter) follow-up duration of 15 days (IQR, 12–31 days; total, 29,812 days), 89 PBSIs occurred (incidence, 6.1%; incidence rate, 2.99 per 1,000 catheter days). In our cohort, the median PICC dwell time in patients who did develop a PBSI was 17 days and in those who did not develop a PBSI it was 15 days. These dwell times were respectively longer than those in the derivation cohort, which were 15 and 11 days, respectively. Of the 89 causative organisms in our PBSI cases, 60 (67.4%) included gram-positive cocci, and coagulase-negative Staphylococcus were the most prevalent (n = 44, 49.4%) (Supplementary Table 1 online).
External validation of the MPC score
The median values of the MPC score and prognostic index calculated based on the linear predictor were 1 (IQR, 1–2) and 0.60 (IQR, 0.60–1.33), respectively; the distributions of both the values were skewed to the right (Supplementary Fig. 1 online).
Discrimination
Visual assessment of the Kaplan–Meier curves did not demonstrate discrimination between the risk groups (log-rank χ Reference Rubin, Apfelbaum and Tung2 = 11.66; P = .11) (Fig. 1A). The discrimination measures were generally poor (eg, C statistic = 0.608 and K statistic = 0.544) (Supplementary Table 2 online).
Observed and predicted PBSI even curves. Kaplan–Meier plots for the observed PBSI events (A, left panel) and predicted mean event curves for the original (B, center panel) and updated (C, right panel) MPC scores. The curves are color-coded based on the assigned scores in dark green (0 points), orange-red (1 point), navy (2 points), maroon (3 points), teal (4 points), sienna (5 points), and orange (≥ 6 points). Note. MPC score, Michigan peripherally inserted central catheter–associated bloodstream infections score; PBSI, peripherally inserted central catheter–associated bloodstream infection.
Calibration
The predicted risks according to the MPC score were systematically too low and had insufficient variety (calibration slope, 1.16; 95% CI, 1.02–1.32; P = .024; χ Reference Rubin, Apfelbaum and Tung2 of model misspecification, 23.74; P < .001). The observed event rates for the lower-risk groups (≤3 scores) were generally higher than the corresponding predicted mean curves (Fig. 1A and B). The calibration plots at any observed timing consistently showed underestimation (Supplementary Fig. 2 online).
Update of the MPC score
Updating the baseline survival function increased the predicted mean event rates that were higher than the original score (Fig. 1B and 1C), which improved the poor calibration of the original score (updated calibration slope, 1.02; 95% CI, 0.04–1.99; P = .051). See Supplementary Fig. 3 (online) for the calibration plots.
Decision curve analysis
The updated MPC score outperformed the other strategies when the threshold probability ranged from 4% to 13%, and the cumulative PBSIs up to 28 days were concerning (Supplementary Fig. 4 online). However, the maximal net benefit was only 0.016 (ie, 16 more cases appropriately managed per 1,000 cases) compared to the “intervention for all” strategy without model when the threshold probability was 8.7% and the cumulative PBSI at 28 days was the outcome of interest. The “intervention for all” strategy outperformed the original MPC score up to 28 days regardless of the threshold probability.
Sensitivity analysis
Although the prognostic index based on the original linear predictor slightly improved the discrimination of the MPC score (log-rank χ Reference Rubin, Apfelbaum and Tung2 = 14.5; P = .006), the discriminative and calibration performance measures were still unsatisfactory (Supplementary Table 1 and Supplementary Figs. 5 and 6 online). The maximal net benefit of the updated MPC score was similar to that of the main analysis (Supplementary Fig. 7 online).
Discussion
In this study, we failed to replicate the good performance of the original MPC score. Our cohort had a clinically more heterogeneous case mix with a higher PBSI incidence compared to the more uniform derivation cohort consisting of only hospitalized medical patients. Specifically, the discriminative performance was poor, and the model largely underestimated the risk of PBSI. However, with a recalibrated baseline hazard function, the updated model had improved calibration. A decision curve analysis showed that the updated MPC score marginally outperformed reference baseline strategies without the model, although the increase in the net benefit was not substantial.
Our study had several strengths. It was the first external validation of the MPC score of its kind. We included consecutive, prospectively registered patients who underwent PICC catheterizations uniformly performed by a dedicated PICC team with no missing information. We used the standard approach for updating and adapting an existing model to a local setting. Reference Steyerberg7,Reference Riley, van der Windt, Croft and Moons8 This study also had limitations. It was a retrospective, single-center design with only 89 PBSI events, which did not satisfy the minimum target events to ensure precise estimates of model performance Reference Collins, Ogundimu and Altman9 and did not allow subgroup analysis on hospitalized medical patients or a full model update.
The case mix and baseline risk of our cohort differed from that of the derivation cohort. Reference Herc, Patel, Washer, Conlon, Flanders and Chopra3 Therefore, the observed poor calibration of the original score is not surprising. Recalibrating the baseline hazard function is only the first step in locally updating the model, Reference Riley, van der Windt, Croft and Moons8,Reference Debray, Vergouwe, Koffijberg, Nieboer, Steyerberg and Moons10 and the sample size was small. Thus, implementing a multicenter collaborative study, re-estimating the coefficients for the individual predictors, and incorporating new, additional parameters, Reference Debray, Vergouwe, Koffijberg, Nieboer, Steyerberg and Moons10 would be the next steps to making better use of the MPC score.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/ice.2021.497
Acknowledgments
Financial support
No financial support was provided relevant to this article.
Conflicts of interest
All authors report no conflicts of interest relevant to this article.
M.I. has received honoraria from Otsuka Pharmaceutical, Astellas Pharma, Eisai, and Daiichi Sankyo outside the submitted work.
