DISCUSSION

Too much of the information reported to the public increases confusion. We found substantial variation across the 3 SIR measures of SSI performance for 20 California hospitals. One-third of the hospitals had significantly discordant quality ratings across these 3 metrics, which could result in different actions by patients seeking high-quality care or by hospital leaders aiming to improve and maintain patient safety and high-quality care. This confusion is heightened because all 3 metrics are identified using the same nomenclature, “SIR,” thus implying that they are comparable. While serious discrepancies have been reported among other aggregate quality metrics, those metrics are at least known by different names.Reference Austin, Jha and Roman ^{4 –} Reference Birnbaum ⁶

At least 3 possible reasons underly the observed discrepancies. First, the inclusion criteria of SSI differ; only 1 model includes superficial SSI events, whereas the other 2 models do not. Also, the CDPH model includes only SSIs for patients readmitted to the operative hospital. Yokoe et alReference Yokoe, Avery, Platt and Huang ⁷ report that limiting surveillance to the operative hospital for SSI readmissions underestimates SSIs for total knee arthroplasty and total hip arthroplasty by missing 17% of cases. Failing to count SSIs detected at nonoperative hospitals resulted in a better relative rank for 61% of hospitals performing total hip and total knee arthroplasties.Reference Yokoe, Avery, Platt and Huang ⁷ Whether a large proportion of colon SSIs are missed due to restriction of surveillance to the operative hospital remains to be determined.

Second, the 3 models use different variables in their risk adjustment calculations. For example, the CMS adjustment model includes only age and American Society of Anesthesiologists (ASA) Physical Status score but excludes wound class, a factor known to impact SSI risk in colon surgery.Reference Hübner, Diana and Zanetti ^{8 ,} Reference Serra-Aracil, Garcia-Domingo and Pares ⁹ This exclusion could explain the smaller number of expected infections compared to observed infections. Furthermore, the CMS model does not account for facility factors that may be associated with SSI risk. Consequently, institutions could be penalized because of their patient population, the calculation method selected, and/or the degree of appropriate risk stratification. Therefore, it is not surprising that hospitals with higher hospital quality summary scores (eg, those that had quality accreditations, offered more services, were major institutions, and/or demonstrated better than average performance on other process and outcome measures) were frequently poor performers and were more often penalized by the Hospital-Acquired Condition Reduction Program.Reference Rajaram, Chung and Kinnier ^{10 –} Reference Minami, Dahke and Barnard ¹²

Third, it is unknown whether the shifting-base distortion demonstrated with central line-associated bloodstream (CLA-BSI) SIRs might also occur with SSI SIRs.Reference Birnbaum, Zarate and Marfin ¹³ SSI SIRs are calculated using logistic regression unlike CLA-BSI SIRs that are calculated as a simple weighted average.

Extremely favorable SIRs from a single institution could represent either excellence in SSI prevention processes or underreporting of SSIs. Therefore, validation of HAI reporting processes is critical to ensuring that each hospital’s surveillance program meets predetermined minimum sensitivity and specificity standards for case detection and case classification.Reference Letourneau, Calderwood and Huang ¹⁴ In 2013, the CDPH conducted a validation study on the accuracy of SSI reporting following colon surgery using a sample of 47 hospitals that represented ~10% of all SSI-reporting acute-care hospitals in California. The sensitivity of self-reporting was 50%. When diagnostic ICD-9 flag codes were used, it was estimated that 1 SSI would be found for every 2–3 records reviewed, which would improve the sensitivity by ~34%.Reference Calderwood, Huang and Keller ¹⁵

This pilot study has important limitations. First, with only 20 hospitals, it has limited statistical power. A broader sample is needed to more fully explore statistical differences among the various models. Second, teaching status was accepted based upon self-report; the accuracy with which California hospitals categorize their extent of teaching is unknown, but discrepancies have been reported elsewhere.Reference Zarate and Birnbaum ¹⁶

An ideal model might include the following: better stratification of colon surgery procedures; NHSN’s current risk adjustment factors; nonmodifiable risk factors that addresses the case-mix index differences between institutions; and all SSIs (including superficial incisional that require hospitalization for management of SSI) and deep incisional and organ/space SSIs, regardless of readmission to the operative or nonoperative hospital. The inclusion of superficial SSIs remains controversial because most superficial incisional SSIs are not serious and are often managed in the outpatient setting. However, significant financial and physical morbidity can be associated with superficial SSIs that require hospitalization. From a public health perspective, all SSIs present opportunities for infection prevention interventions.

In summary, this study is the first to examine the extent of agreement and discord among SSI models. A hospital’s colon surgery SSI SIR values and associated ranks differ depending on the SIR calculation used. Readers knowledgeable in the various models might recognize only those differences that are statistically significant; however, the broader public audience might consider any difference in the numbers or rank as important, and CMS incentives and penalties are triggered according to rank. While no single metric is ideal, healthcare quality would be better served if there were a single well-adjusted model that would better reflect quality outcomes.