Most cardiac surgery centers collect surveillance data on surgical site infections (SSIs) through their infection prevention and control (IPAC) departments using the standardized Center for Disease Control and Prevention/National Healthcare Safety Network (CDC/NHSN) criteria. 1 The majority of recent trials on antibiotic prophylaxisReference Niederhauser, Vogt, Vogt, Genoni, Kunzli and Turina 2 – Reference Saginur, Croteau and Bergeron 5 and other cardiac surgery studies have used the same or similar definitions.Reference Eklund, Valtonen and Werkkala 6 , Reference Walsh, Greene and Kirshner 7 Given that identical definitions are accepted for both research and surveillance, surveillance data could be used for the outcome assessment in clinical trials, reducing both costs and redundancy.
Surveillance programs may however, systematically underestimate the true infection rate.Reference Niedner 8 In particular, post-discharge capture of superficial SSIs has proven difficult.Reference Petherick, Dalton, Moore and Cullum 9 Issues stem from a lack of a standardized postoperative follow-up process, including both the timing and the specific methods used.Reference Petherick, Dalton, Moore and Cullum 9 One approach is telephone contact at the end of the surveillance period.Reference Bianco, Roccia, Nobile, Pileggi and Pavia 10 Although patients cannot reliably self-diagnose surgical site infections,Reference Petherick, Dalton, Moore and Cullum 9 they can accurately report antibiotic usage or revision surgery, which would flag the patient for further review.
Blinded outcome adjudication committees are used to reduce the risk of observer bias with potentially subjective outcomes. To the best of our knowledge, no published studies have assessed the accuracy of routine deep or organ-space sternal SSI (s-SSI) surveillance compared to independent research surveillance with review by an outcome adjudication committee.
METHODS
All patients undergoing cardiac surgery by sternotomy from 2 high-volume Canadian sites (Hamilton, Ontario, and Edmonton, Alberta) during a 4-month period were included. Patients were independently reviewed by IPAC and a research team using CDC/NHSN criteria. 1 The last criterion, “diagnosis of s-SSI by physician,” defined s-SSI for surveillance but was not considered diagnostic for the outcome adjudication committee due to its subjectivity. As a quality-assurance project, the study was exempt from review by the institutional research ethics board.
The IPAC programs, as part of their routine surveillance, reviewed patient records available at the hospital site on all patients at ≥90 days post-surgery. Patients were categorized as having no infection, a superficial infection, or a deep or organ-space s-SSI. Patients with SSIs were routinely reviewed by a physician at the Edmonton site. No routine physician involvement was in place at the Hamilton site.
The research assistants reviewed patient records at hospital discharge and at post-operative day 90. Additionally, patients were contacted by telephone at postoperative day 90 using a standardized, structured script. Patients with any indicators of a potential s-SSI (antibiotic treatment, fever, culture from sternal wound, local symptoms, revision surgery, diagnosis of SSI by physician or surgeon) were asked to provide further information. For patients with 1 or more positive indicators in the absence of an obvious alternative diagnosis at either discharge or the post-operative day 90 review, a case report was drafted and presented to the outcome adjudication committee for assessment. This committee consisted of 2 infectious diseases physician/hospital epidemiologists and 1 cardiac surgeon. If the committee felt that a case report at postoperative day 90 did not have sufficient information, missing information was requested through the IPAC team.
The primary objective was to assess the sensitivity and specificity of IPAC surveillance in detecting deep or organ-space s-SSI compared to research surveillance. The secondary outcome was the sensitivity and specificity for all types of s-SSIs, including superficial s-SSIs. We required a total of 20 deep or organ-space s-SSIs for a sensitivity of >95% (95% confidence interval [CI], 75.1%–99.9%) if 1 or fewer cases were missed by surveillance, and a specificity of >99.5% (95% CI 99.0%–99.9%) if 4 or fewer cases were falsely labelled as events. Assuming an incidence of 2% for deep or organ-space s-SSIs, our planned sample size was 1,000 patients. We report the sensitivity and specificity and the 95% confidence intervals (CI) of surveillance compared to research surveillance (SPSS 18, SPSS, Chicago, IL), and the reasons for disagreements.
RESULTS
A total of 966 patients were included. Among them, 12 deep or organ-space infections were identified (1.2%) by surveillance and 11 (1.1%) were identified by research surveillance (Table 1a). There was disagreement between research surveillance and IPAC surveillance in 7 of these cases (κ=0.69; 95% CI, 0.48–0.91). In addition, 3 cases with a deep or organ-space s-SSI were missed by IPAC surveillance because the patients were not readmitted to the study hospital and were identified only by a follow-up phone call by the research team. On the other hand, 4 cases were identified by IPAC surveillance but were not deemed to be an infection by research surveillance: 3 of these 4 patients (75%) were treated for presumed deep or organ-space s-SSIs by an infectious disease physician but did not meet the other CDC/NHSN criteria, and 1 case did not meet any of the criteria. Using research surveillance as the reference standard, the sensitivity and specificity of surveillance to accurately detect deep or organ-space s-SSIs were 72.7% (8 of 11; 95% CI, 39.0%–94.0%) and 99.6% (951 of 955; 95% CI, 98.9%–99.9%), respectively. When considering the diagnosis of an SSI by a physician or surgeon as a criterion that would overrule the outcome adjudication’s assessment, sensitivity, specificity, and κ were 78.6% (95% CI, 49.2%–95.3%), 99.9% (95% CI, 99.4%–100.0%), and 0.84 (95% CI 0.69–0.99), respectively.
Agreement Between Routine Surveillance and the Outcome Adjudication Committee (Research Surveillance)
In addition to the 12 deep or organ-space s-SSIs, surveillance identified 14 superficial s-SSIs (26 total SSIs), for an overall infection rate of 2.7% (Table 1b). In comparison, research surveillance identified 34 s-SSIs overall for an s-SSI rate of 3.5%. IPAC surveillance missed 19 cases and reported 11 false-positive cases, for a sensitivity of 44.1% (95% CI, 27.2%–62.1%) and a specificity of 98.8% (95% CI, 97.9%–99.4%). The κ for any type of s-SSI was 0.48 (95% CI, 0.33–0.64).
DISCUSSION
While the s-SSI rates identified by IPAC surveillance and research surveillance were similar, IPAC surveillance was not sufficiently accurate to be used for research purposes due to its low sensitivity for identifying true cases. The reason for this discrepancy is that patients whose s-SSIs were missed by surveillance were not readmitted to the same hospital and therefore were not identified. Specificity was excellent; however, several cases were misclassified based on a surgeon’s or physician’s clinical diagnosis of an s-SSI.
Because patients with deep or organ-space s-SSI need surgical revision and long-term antibiotic treatment, they would not be missed by IPAC surveillance if they were readmitted to the same hospital. However, 3 patients that were readmitted to another hospital were identified only through a follow-up phone call. Therefore, either a follow-up phone call or notes from a clinic appointment at the end of the surveillance period would improve the accuracy of surveillance programs, as previously suggested.Reference Petherick, Dalton, Moore and Cullum 9 , Reference Bianco, Roccia, Nobile, Pileggi and Pavia 10
The other group of misclassified patients were cases identified by IPAC surveillance as having an s-SSI but the committee found that they did not meet the definition. Moreover, 3 of these 4 patients were labelled as deep or organ-space s-SSIs by IPAC surveillance based on a physician’s or surgeon’s clinical diagnosis while not meeting the other CDC/NHSN criteria. Given the subjectivity of the criterion “diagnosis of s-SSI by physician,” these cases should not be defined as s-SSIs when using surveillance data for research purposes due to the potential risk of observer bias. Thus, patients identified as having an s-SSI through surveillance, regardless of the extent of the infection, may need to be assessed independently by a blinded outcome adjudication committee to ascertain whether they had objective evidence of deep or organ-space s-SSIs.
In conclusion, IPAC surveillance alone was not sufficiently accurate to identify deep or organ-space s-SSIs. Researchers could improve the accuracy of IPAC surveillance data such that it could be used in clinical trials by calling patients 90 days after their operations and by having a committee independently review all s-SSI cases identified by IPAC surveillance.
ACKNOWLEDGMENTS
Financial support: D.M. is a recipient of a Research Early Career Award from Hamilton Health Sciences Foundation (Jack Hirsh Fellowship). The study was funded through a Hamilton Health Sciences New Investigator Fund.
Potential conflicts of interest: All authors report no conflicts of interest relevant to this article.
