Hostname: page-component-6898cfcb55-zkb6p Total loading time: 0 Render date: 2026-11-11T11:06:14.432Z Has data issue: false hasContentIssue false
  • English
  • Français

Severity of Disease Estimation and Risk-Adjustment for Comparison of Outcomes in Mechanically Ventilated Patients Using Electronic Routine Care Data

Published online by Cambridge University Press:  17 April 2015

Maaike S. M. van Mourik,
Karel G. M. Moons,
Michael V. Murphy,
Marc J. M. Bonten,
Michael Klompas and
MICU Registry
Maaike S. M. van Mourik*
Affiliation:
Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute, Boston, MassachusettsDepartment of Medical Microbiology, University Medical Center Utrecht, Utrecht, The Netherlands
Karel G. M. Moons
Affiliation:
Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands
Michael V. Murphy
Affiliation:
Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute, Boston, Massachusetts
Marc J. M. Bonten
Affiliation:
Department of Medical Microbiology, University Medical Center Utrecht, Utrecht, The NetherlandsJulius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands
Michael Klompas
Affiliation:
Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute, Boston, MassachusettsDepartment of Medicine, Brigham and Women’s Hospital, Boston, Massachusetts
MICU Registry
Affiliation:
Department of Medicine, Brigham and Women’s Hospital, Boston, Massachusetts
*
Address correspondence to Maaike S. M. van Mourik, Department of Medical Microbiology, HP G04.614, Heidelberglaan 100, 3584 CX Utrecht, The Netherlands (M.S.M.vanMourik-2@umcutrecht.nl).
Rights & Permissions [Opens in a new window]

Abstract

BACKGROUND

Valid comparison between hospitals for benchmarking or pay-for-performance incentives requires accurate correction for underlying disease severity (case-mix). However, existing models are either very simplistic or require extensive manual data collection.

OBJECTIVE

To develop a disease severity prediction model based solely on data routinely available in electronic health records for risk-adjustment in mechanically ventilated patients.

DESIGN

Retrospective cohort study.

PARTICIPANTS

Mechanically ventilated patients from a single tertiary medical center (2006–2012).

METHODS

Predictors were extracted from electronic data repositories (demographic characteristics, laboratory tests, medications, microbiology results, procedure codes, and comorbidities) and assessed for feasibility and generalizability of data collection. Models for in-hospital mortality of increasing complexity were built using logistic regression. Estimated disease severity from these models was linked to rates of ventilator-associated events.

RESULTS

A total of 20,028 patients were initiated on mechanical ventilation, of whom 3,027 deceased in hospital. For models of incremental complexity, area under the receiver operating characteristic curve ranged from 0.83 to 0.88. A simple model including demographic characteristics, type of intensive care unit, time to intubation, blood culture sampling, 8 common laboratory tests, and surgical status achieved an area under the receiver operating characteristic curve of 0.87 (95% CI, 0.86–0.88) with adequate calibration. The estimated disease severity was associated with occurrence of ventilator-associated events.

CONCLUSIONS

Accurate estimation of disease severity in ventilated patients using electronic, routine care data was feasible using simple models. These estimates may be useful for risk-adjustment in ventilated patients. Additional research is necessary to validate and refine these models.

Infect. Control Hosp. Epidemiol. 2015;36(7):807–815

Information

Type
Original Articles
Information
Infection Control & Hospital Epidemiology , Volume 36 , Supplement 7 , July 2015, pp. 807 - 815
Copyright
© 2015 by The Society for Healthcare Epidemiology of America. All rights reserved 

Evaluating quality of care has become increasingly important over the past years and mandatory public reporting and nonpayment programs for medical complications have strengthened the need for reliable and comparable measures of performance.Reference Panzer, Gitomer, Greene, Webster, Landry and Riccobono 1 , Reference Rosenthal 2 Reporting mandates have been introduced for healthcare-associated infections in a majority of the United States as well as other countries, and nonpayment regulations are expanding. 3 – Reference Haustein, Gastmeier and Holmes 5 The emphasis on measuring quality of care by patient outcomes requires not only reliable ascertainment of clinical outcomes but also accurate and practical methods of adjustment for underlying disease severity so as to allow for valid comparisons between healthcare facilities.Reference Panzer, Gitomer, Greene, Webster, Landry and Riccobono 1 , Reference Mu, Edwards, Horan, Berrios-Torres and Fridkin 6 – Reference Kritsotakis, Dimitriadis and Roumbelaki 10

The ventilator-associated event (VAE) paradigm for surveillance of complications of mechanical ventilation (MV) was introduced as an alternative to traditional assessment of ventilator-associated pneumonia.Reference Magill, Klompas and Balk 11 Two tiers of the new framework—ventilator-associated condition (VAC) and infection-related VAC (IVAC)—are being considered for future use in benchmarking and pay-for-performance programs and are reportable to the Centers for Disease Control and Prevention’s National Healthcare Safety Network (NHSN).Reference Magill, Klompas and Balk 11 However, case-mix adjustment in NHSN is currently limited to stratification by type of intensive care unit (ICU) and better methods of incorporating disease severity are needed to facilitate more meaningful comparison of rates. 12 , Reference Dudeck, Weiner and len-Bridson 13

Several ICU mortality risk scoring systems, such as the Acute Physiology and Chronic Health Evaluation (APACHE) and Simplified Acute Physiology Score, have been in use for numerous years to measure disease severity in research and outcome evaluations.Reference Breslow and Badawi 8 , Reference Moreno, Metnitz and Almeida 14 – Reference Higgins, Teres, Copes, Nathanson, Stark and Kramer 16 Although these models have good predictive accuracy for in-hospital mortality, their universal adoption is hampered by their inclusion of many clinical parameters that are labor-intensive to gather and infrequently available in a structured electronic format.Reference Jha, DesRoches and Campbell 17 , Reference Kuzniewicz, Vasilevskis and Lane 18 Increasing use of electronic health records (EHRs) has fueled interest in the possibility of using routine care electronic data as a reliable alternative to time-consuming manual data collection. Although some risk scoring systems have been adapted to electronic data, their generalizability is questionable since they use diagnosis codes from administrative data, include clinical variables that are available in only a fraction of EHRs, and incorporate some variables that are not routinely measured in all patients.Reference Johnson, Kramer and Clifford 19 – Reference Liu, Turk, Ragins, Kipnis and Escobar 21 Furthermore, existing disease severity models may not be optimal for VAE risk-adjustment since they were derived from both ventilated and nonventilated patients and in some cases excluded cardiovascular surgery patients.Reference Moreno, Metnitz and Almeida 14 – Reference Higgins, Teres, Copes, Nathanson, Stark and Kramer 16 , Reference Render, Deddens and Freyberg 20

In this study, we aimed to develop a disease severity prediction model targeted specifically at mechanically ventilated patients using routinely collected, electronically stored data. We strove to optimize the balance between model precision, complexity, and generalizability. In addition, we assessed the relation between estimated disease severity and the risk of VAEs.

METHODS

Study Population

All patients initiated on MV, through an endotracheal tube or tracheostomy, and requiring ICU care in the Brigham and Women’s Hospital (Boston, MA) from January 1, 2006, through December 31, 2012, were eligible for study inclusion. An interruption of MV of more than 1 calendar day defined the start of a new ventilation episode. If multiple hospital admissions or MV episodes occurred for a patient within the study period, a random hospitalization was chosen and the first episode within the hospital admission was selected. Patients younger than 18 years, those who could not be successfully linked to electronic data repositories, and those who were not admitted to an ICU were excluded. This study was reviewed by the institutional review board of Brigham and Women’s Hospital and the requirement for individual patient consent was waived (protocol number: 2012P002052).

Outcomes

The primary outcome in developing a severity of disease prediction model was in-hospital mortality. As a sensitivity analysis, models were also developed for mortality within 30 days of initiation of MV as determined from hospital vital statistics records.Reference Graham and Cook 22 – Reference Pouw, Peelen, Moons, Kalkman and Lingsma 24

VACs and IVACs were selected as targets for risk-adjustment. A VAC reflects a sustained period of worsening oxygenation requiring increased ventilator settings whereas an IVAC is defined as a VAC with concurrent initiation of new antimicrobial therapy in combination with signs of infection (abnormal temperature, white blood cell count). Both were operationalized electronically in accordance with January 2013 NHSN specifications using ventilator settings derived from the respiratory therapy department. 12

Predictors

Potential predictors were collected from the hospital’s electronic Research Patient Data Registry. 25 This database compiles a broad variety of structured electronic data elements recorded during routine clinical care. We extracted data on patient demographic characteristics, laboratory tests, microbiology results, procedure codes, diagnosis codes, billing information, and medication use. ICU admission and discharge dates, however, were not available. Variables were selected for extraction on the basis of clinical associations with severity of disease and previously published models.Reference Moreno, Metnitz and Almeida 14 , Reference Zimmerman, Kramer, McNair and Malila 15 , Reference Render, Deddens and Freyberg 20 , Reference Higgins, Kramer, Nathanson, Copes, Stark and Teres 26

Subsequently, all candidate variables were assessed for potential inclusion. A qualitative estimate of feasibility was based on the extent of missing data and the expected likelihood that similar data will be available in electronic form in other hospitals. We also evaluated candidate variables for generalizability based on the probability that predictors would have similar interpretation when applied in other settings or across time (Table 1). When applicable, predictors were classified in broad groups that are expected to translate to other settings. ICU types were grouped into medical, surgical, neuroscience, and cardiovascular surgery categories, and procedure types were classified by anatomic or functional location (eg, musculoskeletal, cranial, digestive tract).

TABLE 1

Overview of Electronic Data Sources

NOTE. Data were collected during the process of routine care on the first calendar day of mechanical ventilation (MV) or the day prior when available. All groups of predictors are classified by anticipated feasibility and generalizability of electronic data collection. For details on variable definitions please refer to supplementary material (section 1). ALT, alanine transaminase; CABG, coronary artery bypass graft; DRG, diagnosis-related group; ICD9, International Classification of Disease, Ninth Edition; INR, international normalized ratio.

a Feasibility was estimated on the basis of the amount of missing data in our sample and the expected availability from electronic health records in other settings.

b Generalizability was defined as the anticipated consistency of predictor interpretation in other settings. +: good anticipated feasibility/generalizability, +/−: some concerns, −: limited anticipated feasibility/generalizability.

c Surgical intensive care unit (ICU): all surgical ICU types (eg, general, trauma, thoracic); Neuroscience ICU: neurology and neurosurgery.

We collected values for predictors from the calendar day of MV onset and the day prior when available. Data were cleaned and processed as specified in sections 1A–1C of the online-only material. For each predictor, the observation reflecting the worst prognosis was taken (eg, lowest hematocrit, highest creatinine). Comorbid conditions were assessed based on administrative data using the Elixhauser method. 27 , Reference Elixhauser, Steiner, Harris and Coffey 28

Model Development and Statistical Methods

Patterns of missing data were explored and multiple imputation was performed (mice package, version 10, R).Reference van Buuren and Groothuis-Oudshoorn 29 Exploratory univariable analysis of the association between predictors and in-hospital mortality was performed and restricted cubic splines were used for continuous variables that did not have a linear association with the log-odds of mortality (online-only material, section 1B).Reference Harrell 30

Regression models of incremental complexity were built, first including groups of candidate predictors with the best expected feasibility and generalizability (defined in Table 1). All variables within each group of predictors were added to the model irrespective of their individual significance. Interactions detected by backward elimination were judged relevant if P<.0001; very rare combinations of predictors were excluded. If a group of predictors no longer contributed to model performance, measured by the Akaike information criterion, it was eliminated in its entirety. To limit overfitting and to obtain bias-corrected estimates of model performance, internal validation and uniform shrinkage were performed using bootstrap samples (100 samples per imputation). This method generates more accurate estimates of performance than split-sample or cross-validation approaches.Reference Steyerberg, Harrell, Borsboom, Eijkemans, Vergouwe and Habbema 31 Parameter estimates from the imputation sets were pooled by Rubin’s rule.Reference Rubin 32

Model discrimination and calibration were assessed using the area under the receiver operating characteristic curve (AUROC), the Brier score, the Hosmer-Lemeshow test, and examination of calibration plots. Given the large heterogeneity in mortality across ICU types, the model with the most favorable balance of complexity to performance was developed stratified by ICU type.

Finally, the possibility of VAE risk-adjustment using the severity of underlying disease estimates was assessed by examining the relation between predicted risk of mortality and observed incidence of VAEs. For the overall population and for each type of ICU separately, patients were classified by decile of predicted mortality risk and the incidence of VACs and IVACs was determined within each decile. In addition, attention was paid to the relation between disease severity and the competing outcomes for VAEs. Because patients must by definition have received at least 4 days of MV—2 days of stability or improvement followed by 2 days of deterioration—in order to be at risk of VAE, early death or early discharge within 4 days can be considered competing outcomes. 33

All analyses were performed using SAS, version 9.2 (SAS Institute), and R (www.r-project.org).

RESULTS

During the study period, 24,366 episodes of MV were recorded in 20,581 unique patients. Of these patients, 59 were younger than 18 years; 350 did not have a successful medical record match; and 144 were not admitted to an ICU, thus leaving 20,028 patients for analysis. Median age was 64 years, 60.3% were male, and 40.7% were admitted to the cardiovascular surgery unit (Table 2). Average in-hospital mortality was 15.1%, ranging from 3.3% for cardiovascular surgery to 38% for the medical ICUs.

TABLE 2

Selected Patient Characteristics in Comparison Study of Outcomes in Mechanically Ventilated Patients Using Electronic Routine Care Data, 2006–2012

NOTE. Data are no. (%) unless otherwise indicated. IQR, interquartile range; MV, mechanical ventilation.

a On day of MV onset or day prior.

As expected, models using larger numbers of predictors had improved performance characteristics, although the gains with incrementally more complicated models became smaller (Table 3). Model 2, using only demographic information, unit type, microbiology data, procedure codes, and commonly performed laboratory tests, achieved an optimism-corrected AUROC of 0.870 (95% CI, 0.864–0.876) (parameter estimates in the online-only material Tables S1, S2). On internal validation, overoptimism was very limited (0.003–0.006 decrease in AUROC). Calibration was adequate with slight overprediction for high-risk patients (online-only material, Figure S1); this is also reflected by the Hosmer-Lemeshow test statistics (χ2 test, 49.9 to 59.5 on 10 degrees of freedom). A sensitivity analysis for 30-day mortality demonstrated similar performance of all models (online-only material, Table S3).

TABLE 3

Results of Model Development and Internal Validation for In-Hospital Mortality

NOTE. Models are ordered by increasing complexity as classified by feasibility and generalizability of data collection. AUROC, area under the receiver operating characteristic curve; CV, cardiovascular; H-L, Hosmer-Lemeshow; ICU, intensive care unit.

a Microbiology (blood culture performed) did not contribute to model performance in models 3 and up and was therefore removed.

b Physiology I-III, please refer to Table 1 and supplementary data for definitions.

c Bias-corrected.

Model 2 also had reasonable performance when applied within ICU types (AUROC, 0.758 to 0.852); as can be expected, the observed performance was somewhat lower because differences between patients within each of the unit types were smaller than across units. Given the large heterogeneity in mortality rates between types of ICUs and the intended application of achieving adequate estimation of disease severity within ICU types, the model was refit stratified by ICU type (Table 4, Table S4), resulting in similar performance yet less complex models because interaction terms were no longer necessary. In addition, the performance of this model was compared with APACHE II scores of a subset of 144 patients included in the hospital’s Registry of Critical Illness who were mechanically ventilated on admission to the ICU.Reference Dolinay, Kim and Howrylak 34 This revealed an AUROC of 0.66 for APACHE II versus 0.80 for the final model.

TABLE 4

Model Development for In-Hospital Mortality Stratified by Unit Type for Model 2

NOTE. AUROC, area under the receiver operating characteristic curve; H-L, Hosmer-Lemeshow; IVAC, infection-related ventilator-associated condition; MV, mechanical ventilation; VAC, ventilator-associated condition.

a For mortality, after correction for overoptimism. Models were developed by unit type; procedure types that did not occur within the unit types were excluded. There were no significant interactions in all models.

Depending on the type of ICU, the overall risk of VAC ranged from 0.9% to 9.1% and of IVAC from 0.4% to 3.2% (Table 4). Overall, there was a positive association between the estimated severity of disease (predicted mortality from model 2 stratified by ICU type) and the occurrence of VAC and IVAC (Figure 1). Specifically, for surgical and cardiovascular surgery units there was a positive association between estimated disease severity and VAE incidence, whereas for medical and neurological ICUs the picture was less clear. As can be seen in the bar charts, the association between observed risk of VAE and estimated disease severity is largely explained by the proportion of patients who remain on MV 4 days or longer and are thus at risk of developing a VAE. In particular for medical units, the decline in VAC incidence in higher mortality risk groups is caused by an increasing proportion of patients experiencing the competing event of dying within 4 days of MV onset.

FIGURE 1

Association between estimated disease severity (predicted risk of in-hospital mortality) and the occurrence of ventilator-associated events (VAEs). Predicted risk of in-hospital mortality was derived from the final model stratified by unit type and incidence of ventilator-associated condition (VAC, left column) and infection-related VAC (IVAC, middle column). Horizontal lines represent the overall event incidence. The bar charts (right column) depict the proportion of patients at risk of developing a VAC or IVAC and those either discharged alive off the ventilator within 4 days of ventilator onset (early discharge) or deceased within 4 days of ventilator onset (early death). Top row: deciles of predicted mortality risk calculated for the overall population; bottom 4 rows: patients are stratified by decile of mortality risk within each intensive care unit type. MV, mechanical ventilation.

DISCUSSION

Adjustment for severity of underlying disease is a necessary requirement to achieve valid comparisons of patient outcomes across healthcare providers. This study is meant as a proof-of-concept that estimating underlying disease severity in mechanically ventilated patients is feasible using fairly simple models applied to electronic routine care data. The performance characteristics obtained were in the same range as those observed in the derivation studies of more labor-intensive methods such as APACHE and Simplified Acute Physiology Score.Reference Breslow and Badawi 8 Models of increasing complexity showed marginally better performance but at the cost of more extensive data collection and rising concerns with generalizability to different settings.

In addition, this study found an intricate relation between estimated disease severity and the incidence of VAEs, thus providing a starting point towards improved correction for differences in patient populations when these outcomes are used for assessment of quality of care. Currently, event rates reported to NHSN are stratified by unit type for comparison.Reference Dudeck, Weiner and len-Bridson 13 Although unit type does explain an important fraction of the observed variation in VAE incidence, this can be further refined by stratifying on underlying disease severity within ICU types. This association between estimated disease severity and VAE risk seems to be mediated by identifying the proportion of patients who remain on MV long enough to be eligible for developing a VAE (MV≥4 days). We purposely included all patients initiated on MV regardless of whether they achieved 4 days of MV because all MV days are counted in the denominator of VAE rates and the duration of MV is also unknown at the time of ventilation onset (the moment of prediction). Future research is needed to determine how to assess the exact relationship between severity of underlying disease and the likelihood of developing VAE and how to account for the competing outcomes of early death and early discharge. Because the models were developed in a single tertiary care US hospital, further validation studies are needed to determine their performance and utility in different hospitals and types of ICU. These studies will also provide an opportunity to refine estimates for a broader range of (specialized) ICU types, including mixed medical-surgical units and units affiliated with smaller community hospitals, ideally in alignment with the ICU classifications currently employed by NHSN.

Although numerous models have previously been developed to predict in-hospital mortality, these were not developed for the purpose of comparing patient outcomes in the specific domain of mechanically ventilated patients, and several concerns limit their application in risk-adjustment for VAEs, including the highly divergent degrees of sophistication in EHR contents across hospitals.Reference Moreno, Metnitz and Almeida 14 , Reference Zimmerman, Kramer, McNair and Malila 15 , Reference Higgins, Kramer, Nathanson, Copes, Stark and Teres 26 Existing models, such as the APACHE, require extensive amounts of manual data collection, including physiologic variables that are not routinely documented in EHRs (eg, Glasgow coma scale, blood pressure, urine output) and reason for admission, which is subject to considerable interrater variability.Reference Kuzniewicz, Vasilevskis and Lane 18 , Reference Polderman, Thijs and Girbes 35 Strikingly, models in some cases included administrative data collected after the first 2 days of admission, which may better reflect the evolution of disease within hospitals rather than patients’ presenting conditions.Reference Render, Deddens and Freyberg 20 , Reference Liu, Turk, Ragins, Kipnis and Escobar 21 Moreover, translating a model directly to an electronic environment may introduce unanticipated loss of comparability.Reference Liu, Turk, Ragins, Kipnis and Escobar 21 , Reference Bosman, Oudemane van Straaten and Zandstra 36 The models developed in this study aim to address these concerns and do have similarities with existing risk-adjustment models, in particular the model developed within the Veteran’s Affairs system.Reference Render, Deddens and Freyberg 20 However, that model makes extensive use of admission diagnoses and procedure codes and includes more laboratory variables than the simpler models developed in our study.

The process of model development in this study was targeted specifically for use with electronic data and an important component was the a priori assessment of the feasibility and generalizability of data collection through EHRs. Preference was given to measuring information “close to the patient” as opposed to proxies such as administrative coding data. We found that simple models without admission diagnoses, disease codes, or medication use—variables for which the use and classification are more likely to differ across settings—were still able to perform robustly. We further incorporated methods for handling of outliers and approaches to handling of missing observations other than imputing a normal value since the latter weakens the reliability of risk-adjustment.Reference Afessa, Keegan, Gajic, Hubmayr and Peters 37

Several important additional limitations need to be addressed. Firstly, this study was developed in a single tertiary medical center using data from a 6-year period and did not include a separate validation sample; use of a random split-sample for validation was judged not to provide sufficient new insights in comparison with the internal validation strategy used.Reference Steyerberg, Harrell, Borsboom, Eijkemans, Vergouwe and Habbema 31 As described above, validation studies will be needed before large-scale adoption. Nevertheless, the results of this study will inform the design and data collection of such studies. The Hosmer-Lemeshow statistics for our models suggested significant miscalibration but this test is known to be unreliable with large sample sizes and assessment of calibration plots showed adequate calibration.Reference Kramer and Zimmerman 38 Finally, examination of yearly observed-to-expected ratios by unit type did not reveal any secular decreases in model performance (online supplementary data).

We could perform only a very limited head-to-head comparison of our models with other, existing, methods (eg, APACHE or Simplified Acute Physiology Score) since these severity-of-disease estimators are not routinely collected in our setting; their absence was the driving force behind the current study. It has been argued that using 30-day mortality is more reliable than in-hospital mortality for the purpose of outcome comparison; however, data on in-hospital mortality is often collected more consistently than 30-day mortality.Reference Drye, Normand and Wang 23 , Reference Pouw, Peelen, Moons, Kalkman and Lingsma 24 In this study, model performance characteristics were similar for both. Prior studies have identified several possible predictors that could not be collected from our EHRs but that may be of predictive value in future studies, including (hospital) location prior to ventilator onset, time to ICU admission, transfer status, urgency of admission and/or surgery, and the presence of do-not-resuscitate orders.Reference Moreno, Metnitz and Almeida 14 , Reference Zimmerman, Kramer, McNair and Malila 15 , Reference Render, Deddens and Freyberg 20 , Reference Keegan, Gajic and Afessa 39 In addition, inclusion of ICU type as a predictor requires assumptions to be made regarding similarity of patient characteristics across ICU types in different hospitals. Future validation studies will need to assess robustness to differences in admission policies and define methods to handle specialty ICUs. The use of more sophisticated modeling strategies such as supervised machine learning have been suggested in the recent literature as efficient methods of model development, and focusing on using parameters that can be collected reliably and efficiently would be of added value.Reference Johnson, Kramer and Clifford 19

Valid comparison of patient outcomes and occurrence of VAEs requires adequate methods of correction for severity of underlying disease. This study demonstrates the feasibility of estimating disease severity in ventilated patients using electronic routine care data and provides insight into the balance between complexity of data collection and model performance. The estimates of underlying disease severity obtained from these models show utility in future adjustment of VAE rates, although further research is necessary to validate and refine the proposed models and develop practical and robust methods of benchmarking.

Acknowledgments

We thank the Brigham and Women’s Hospital Registry of Critical Illness investigators for contributing APACHE II scores for our comparative analysis: Carolina Quintana, MD, Rebecca M. Baron, MD, Laura E. Fredenburgh, MD, Joshua A. Englert, MD, Lee Gazourian, MD, Anthony F. Massaro, MD, Anna Trtchounian, Tamas Dolinay, MD, PhD, and Augustine M. K. Choi, MD. In addition, we thank N.P.A. Zuithoff, PhD, for his statistical advice.

Financial support. Netherlands Organization for Scientific Research (project 9120.8004 and 918.10.615 to K.G.M.M.); Centers for Disease Control and Prevention (grant 1U54CK000172 to M.K.).

Potential conflicts of interest. All authors report no conflicts of interest relevant to this article.

Supplementary material

To view supplementary material for this article, please visit http://dx.doi.org/10.1017/ice.2015.75

Footnotes

*

Shared last author.

Presented in part: IDWeek 2013; San Francisco, California; October 4, 2013 (Abstract p1076).

References

1. Panzer, RJ, Gitomer, RS, Greene, WH, Webster, PR, Landry, KR, Riccobono, CA. Increasing demands for quality measurement. JAMA 2013;310:1971–1980.Google Scholar
2. Rosenthal, MB. Nonpayment for performance? Medicare’s new reimbursement rule. N Engl J Med 2007;357:1573–1575.Google Scholar
3. Centers for Medicare and Medicaid Services(CMS). Hospital-acquired conditions. CMS website. Available at: http://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/HospitalAcqCond/Hospital-Acquired_Conditions.html. Updated September 20, 2012. Accessed November 25, 2013.Google Scholar
4. Association of Professionals in Infection Control and Epidemiology (APIC). HAI reporting laws and regulations. APIC website. Available at: http://www.apic.org/Resource_/TinyMceFileManager/Advocacy-PDFs/Advocacy_Updates/Static_map_-_HAI_revised_7-6-11.gif. Updated July 6, 2011. Accessed February 13, 2012.Google Scholar
5. Haustein, T, Gastmeier, P, Holmes, A, et al. Use of benchmarking and public reporting for infection control in four high-income countries. Lancet Infect Dis 2011;11:471–481.Google Scholar
6. Mu, Y, Edwards, JR, Horan, TC, Berrios-Torres, SI, Fridkin, SK. Improving risk-adjusted measures of surgical site infection for the National Healthcare Safety Network. Infect Control Hosp Epidemiol 2011;32:970–986.CrossRefGoogle ScholarPubMed
7. Harris, AD, McGregor, JC. The importance of case-mix adjustment for infection rates and the need for more research. Infect Control Hosp Epidemiol 2008;29:693–694.Google Scholar
8. Breslow, MJ, Badawi, O. Severity scoring in the critically ill, part 1: interpretation and accuracy of outcome prediction scoring systems. Chest 2012;141:245–252.Google Scholar
9. van Dishoeck, AM, Koek, MB, Steyerberg, EW, van Benthem, BH, Vos, MC, Lingsma, HF. Use of surgical-site infection rates to rank hospital performance across several types of surgery. Br J Surg 2013;100:628–636.Google Scholar
10. Kritsotakis, EI, Dimitriadis, I, Roumbelaki, M, et al. Case-mix adjustment approach to benchmarking prevalence rates of nosocomial infection in hospitals in Cyprus and Greece. Infect Control Hosp Epidemiol 2008;29:685–692.Google Scholar
11. Magill, SS, Klompas, M, Balk, R, et al. Developing a new, national approach to surveillance for ventilator-associated events. Crit Care Med 2013;41:2467–2475.Google Scholar
12. Centers for Disease Control and Prevention (CDC). Surveillance for ventilator-associated events. CDC website. Available at: http://www.cdc.gov/nhsn/acute-care-hospital/vae/. Updated November 25, 2013. Accessed November 26, 2013.Google Scholar
13. Dudeck, MA, Weiner, LM, len-Bridson, K, et al. National Healthcare Safety Network (NHSN) report, data summary for 2012, device-associated module. Am J Infect Control 2013;41:1148–1166.Google Scholar
14. Moreno, RP, Metnitz, PG, Almeida, E, et al. SAPS 3: from evaluation of the patient to evaluation of the intensive care unit, part 2: development of a prognostic model for hospital mortality at ICU admission. Intensive Care Med 2005;31:1345–1355.CrossRefGoogle ScholarPubMed
15. Zimmerman, JE, Kramer, AA, McNair, DS, Malila, FM. Acute Physiology and Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for today’s critically ill patients. Crit Care Med 2006;34:1297–1310.Google Scholar
16. Higgins, TL, Teres, D, Copes, WS, Nathanson, BH, Stark, M, Kramer, AA. Assessing contemporary intensive care unit outcome: an updated Mortality Probability Admission Model (MPM0-III). Crit Care Med 2007;35:827–835.Google Scholar
17. Jha, AK, DesRoches, CM, Campbell, EG, et al. Use of electronic health records in US hospitals. N Engl J Med 2009;360:1628–1638.Google Scholar
18. Kuzniewicz, MW, Vasilevskis, EE, Lane, R, et al. Variation in ICU risk-adjusted mortality: impact of methods of assessment and potential confounders. Chest 2008;133:1319–1327.Google Scholar
19. Johnson, AE, Kramer, AA, Clifford, GD. A new severity of illness scale using a subset of Acute Physiology, Age, and Chronic Health Evaluation data elements shows comparable predictive accuracy. Crit Care Med 2013;41:1711–1718.Google Scholar
20. Render, ML, Deddens, J, Freyberg, R, et al. Veterans Affairs intensive care unit risk adjustment model: validation, updating, recalibration. Crit Care Med 2008;36:1031–1042.CrossRefGoogle ScholarPubMed
21. Liu, V, Turk, BJ, Ragins, AI, Kipnis, P, Escobar, GJ. An electronic Simplified Acute Physiology Score-based risk adjustment score for critical illness in an integrated healthcare system. Crit Care Med 2013;41:41–48.Google Scholar
22. Graham, PL, Cook, DA. Prediction of risk of death using 30-day outcome: a practical end point for quality auditing in intensive care. Chest 2004;125:1458–1466.Google Scholar
23. Drye, EE, Normand, SL, Wang, Y, et al. Comparison of hospital risk-standardized mortality rates calculated by using in-hospital and 30-day models: an observational study with implications for hospital profiling. Ann Intern Med 2012;156:19–26.CrossRefGoogle ScholarPubMed
24. Pouw, ME, Peelen, LM, Moons, KG, Kalkman, CJ, Lingsma, HF. Including post-discharge mortality in calculation of hospital standardised mortality ratios: retrospective analysis of hospital episode statistics. BMJ 2013;347:f5913.Google Scholar
25. Partners Healthcare. Research Patient Data Registry (RPDR). Partners Healthcare website. Available at: http://rc.partners.org/rpdr. Updated November 1, 2013. Accessed November 11, 2013.Google Scholar
26. Higgins, TL, Kramer, AA, Nathanson, BH, Copes, W, Stark, M, Teres, D. Prospective validation of the intensive care unit admission Mortality Probability Admission Model (MPM0-III). Crit Care Med 2009;37:1619–1623.Google Scholar
27. Healthcare Cost and Utilization Project. Comorbidity Software, version 3.7. Available at: http://www.hcup-us.ahrq.gov/toolssoftware/comorbidity/comorbidity.jsp. Updated 2012. Accessed November 1, 2013.Google Scholar
28. Elixhauser, A, Steiner, C, Harris, DR, Coffey, RM. Comorbidity measures for use with administrative data. Med Care 1998;36:8–27.Google Scholar
29. van Buuren, S, Groothuis-Oudshoorn, K. MICE: Multivariable Imputation by Chained Equations in R. J Stat Softw 2011;45:1–67.Google Scholar
30. Harrell, FE. Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis. New York: Springer-Verlag, 2001.Google Scholar
31. Steyerberg, EW, Harrell, FE, Borsboom, GJ, Eijkemans, MJ, Vergouwe, Y, Habbema, JD. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol 2001;54:774–781.Google Scholar
32. Rubin, DB. Multiple Imputation for Nonresponse in Surveys. Hoboken: J. Wiley & Sons, 1987.Google Scholar
33. Centers for Disease Control and Prevention (CDC) Division of Healthcare Quality Promotion. The National Healthcare Safety Network Device-Associated Module: Ventilator-Associated Event Protocol. Atlanta: CDC, 2013.Google Scholar
34. Dolinay, T, Kim, YS, Howrylak, J, et al. Inflammasome-regulated cytokines are critical mediators of acute lung injury. Am J Respir Crit Care Med 2012;185:1225–1234.Google Scholar
35. Polderman, KH, Thijs, LG, Girbes, AR. Interobserver variability in the use of APACHE II scores. Lancet 1999;353:380.Google Scholar
36. Bosman, RJ, Oudemane van Straaten, HM, Zandstra, DF. The use of intensive care information systems alters outcome prediction. Intensive Care Med 1998;24:953–958.Google Scholar
37. Afessa, B, Keegan, MT, Gajic, O, Hubmayr, RD, Peters, SG. The influence of missing components of the Acute Physiology Score of APACHE III on the measurement of ICU performance. Intensive Care Med 2005;31:1537–1543.Google Scholar
38. Kramer, AA, Zimmerman, JE. Assessing the calibration of mortality benchmarks in critical care: the Hosmer-Lemeshow test revisited. Crit Care Med 2007;35:2052–2056.Google Scholar
39. Keegan, MT, Gajic, O, Afessa, B. Comparison of APACHE III, APACHE IV, SAPS 3, and MPM0-III and influence of resuscitation status on model performance. Chest 2012;142:851–858.Google Scholar
Figure 0

TABLE 1Overview of Electronic Data Sources

Figure 1

TABLE 2Selected Patient Characteristics in Comparison Study of Outcomes in Mechanically Ventilated Patients Using Electronic Routine Care Data, 2006–2012

Figure 2

TABLE 3Results of Model Development and Internal Validation for In-Hospital Mortality

Figure 3

TABLE 4Model Development for In-Hospital Mortality Stratified by Unit Type for Model 2

Figure 4

FIGURE 1Association between estimated disease severity (predicted risk of in-hospital mortality) and the occurrence of ventilator-associated events (VAEs). Predicted risk of in-hospital mortality was derived from the final model stratified by unit type and incidence of ventilator-associated condition (VAC, left column) and infection-related VAC (IVAC, middle column). Horizontal lines represent the overall event incidence. The bar charts (right column) depict the proportion of patients at risk of developing a VAC or IVAC and those either discharged alive off the ventilator within 4 days of ventilator onset (early discharge) or deceased within 4 days of ventilator onset (early death). Top row: deciles of predicted mortality risk calculated for the overall population; bottom 4 rows: patients are stratified by decile of mortality risk within each intensive care unit type. MV, mechanical ventilation.

Supplementary material: PDF

van Mourik supplementary material

van Mourik supplementary material 1

Download van Mourik supplementary material(PDF)
PDF 123.4 KB