Hand hygiene (HH) is the foundation of infection prevention as the simplest and most effective intervention to prevent transmission in the hospital.Reference Allegranzi and Pittet 1 Nevertheless, HH compliance is suboptimal among healthcare workers.Reference Boyce and Pittet 2 Monitoring and feedback regarding HH performance are used to drive compliance improvement. Direct observation by a trained observer is considered the gold standard for monitoring, despite limitations, which include low volume of observations and the Hawthorne effect.Reference Masroor, Doll, Stevens and Bearman 3
Recently, interest in automated approaches to HH monitoring using technology has increased substantially. Data to support these technologies are limited to predominantly before-and-after, system-defined, provider compliance.Reference Ward, Schweizer and Polgreen 4 Accuracy of systems is rarely evaluated.Reference Ward, Schweizer and Polgreen 4 We have described the implementation of 2 HH monitoring systems, and we compared their accuracy using concurrent data from trained HH observers.
Methods
We tested 2 types of electronic hand hygiene monitoring systems (EHHMS type A and type B) at an 865-bed, academic medical center in 90-day trials across 4 distinct units. The trials were separated by 18 months. The timing of each trial and unit characteristics are displayed in Table 1.
Inte-rater Reliability of Electronic Hand Hygiene Monitoring Systems (EHHMSs) Compared to Direct Observation
Note. N, no. of observations; RFID, radio frequency infrared detection.
a Functioning on a proprietary network, badge dimensions shown in inches, and weight in grams. The κ statistic range is 0–1, with 0 being no agreement and 1 being perfect agreement.
Each EHHMS was a completely automated system in which providers wore individually identifying badges, and HH events were captured by sensors embedded in or attached to all soap and alcohol-based product dispensers. Patient zones were defined by each system’s network to include areas around a patient’s bed, and semiprivate rooms were included with zones distinct for each patient. System settings expected the providers to perform HH on entry and exit to each patient zone. Data for providers were only captured if the badge was worn. The type B EHHMS badge displayed colored lights and beeped to remind providers to perform HH; the type A badge provided no immediate feedback. All hospital employees who had contact with patients and/or the patient environment on selected units were eligible to participate; participation was strongly encouraged by hospital leadership but was not mandated. In each trial, participants received standard training and were then asked to wear EHHMS badges while continuing normal workflow. During the trials, existing infection prevention program HH observers continued to evaluate HH through direct observation of providers on the units and recorded compliance. One observer remained consistent throughout the study; the other position turned over between periods. Observers dressed in hospital scrubs and documented using an iPod to remain inconspicuous. Observer rounds schedules were not standardized, but they targeted all units at least monthly for infection control committee reporting. During the trial, the observers made monitoring rounds approximately twice weekly on each study unit.
Calculation of a Cohen’s κ was performed by comparing a subset of observer and technology data. For these matched observations, the HH observer and coordinator (N.M.) watched providers in the course of clinical care; there was no special effort to remain inconspicuous for these observations. The time stamps of observer data were compared against the time and location stamps in the technology database. Calculations were performed using Microsoft Excel software (Microsoft, Redmond, WA) (Appendix A).
Hand hygiene compliance goals were set at 90% as a reasonable increase over baseline performance. Providers continued to receive feedback on HH from monthly reported observer data. The EHHMS data were shared once in aggregate to unit leadership during each pilot test; individual data were not distributed.
Results
In the first pilot test, 279 employees volunteered to use the type A EHHMS for 14 weeks, with an overall HH compliance of 30% (87,688 opportunities). In the second pilot test, 169 employees volunteered to use the type B EHHMS for 12 weeks, with overall HH compliance of 93% (363,272 opportunities). Comparisons of the weekly compliance data for each unit are shown in Figure 1. Implementation of the type A EHHMS was extended by 2 weeks because initial badge distribution required 2 weeks longer than anticipated.
Weekly system-defined hand hygiene technology compliance by unit and EHHMS type. Unit 1 is a pediatric intensive care unit. Unit 2 is a general medical unit. Unit 3 is an acute-care oncology unit. Unit 4 is a general medical unit. Team observations for κ statistic calculation took place on weeks designated by an asterisk in the bar of the graph.
Matched observations by the HH observer team were performed during the weeks indicated by an asterisk in Figure 1. Overall observer-defined compliance rates were 90% (778 opportunities) in the type A EHHMS units and 91% (108 opportunities) for the type B EHHMS units (Table 1). Routine observer rounding on these units occurring at least weekly throughout the study revealed overall compliance rates of 87% (2,282 of 2,634 opportunities) and 92% (880 of 953 opportunities) for trial A and B, respectively. Average baseline HH observer data in the 6 months preceding each trial were 86% adherence in unit 1 (216 of 251 opportunities), 78% in unit 2 (395 of 503 opportunities), 92% in unit 3 (989 of 1,080 opportunities), and 91% in unit 4 (1,154 of 1,273 opportunities).
Discussion
We describe the implementation of 2 distinct electronic hand hygiene monitoring systems with markedly different accuracy in capturing provider hand hygiene events. Provider feedback alerted us early in the trial of EHHMS A that the devices were not capturing provider badges. Lights on the dispenser were supposed to flash during badge capture; this was inconsistent according to provider reports. We intensified observer presence on those units. Due to the discrepancy between observer and EHHMS data, we performed the matched observations confirming that some HH events were not captured by the system. The total number of observations recorded during the EHHMS type A trial was less than the number of observations recorded during the type B trial, although the type A trial had more participating providers, which also suggests that events were not reliably captured.
Provider belief in the performance of EHHMS is critically important for participation in automated monitoring of HH.Reference Ellingson, Polgreen and Schnieder 5 , Reference Boyce, Cooper, Lunde, Yin and Arbogast 6 The goal of EHHMS is ultimately to facilitate behavioral change in providers; it is difficult to envision this occurring outside of provider confidence in the data. Our providers did not receive individual data, but they were aware that aggregated data from the type A EHHMS trial were dramatically different from the observation data.
The Hawthorne effect is often used to explain differences in EHHMS system-defined compliance and directly observed compliance. There may be differences in the Hawthorne effect between electronic versus in-person observation monitoring, yet the data demonstrate a greater compliance discrepancy with the type A EHHMS compared to the type B system, suggesting that system performance is an important variable. Only type B provided an HH reminder, which could have influenced provider compliance, but it does not explain differing accuracy.
Rigorous path-based validation of EHHMS has been suggested as a standard evaluation,Reference Limper, Garcia-Houchins, Slawsky, Hershow and Landon 7 but this intentional, methodological testing fails to capture the complexity of clinical workflows. A strength of this study is that the validation occurred during actual patient care encounters. A corresponding weakness is that in comparing the time stamps for observer and EHHMS data, the events may not have been perfectly matched; it is possible that some observations assumed to be the same event may in fact have been different episodes of HH.
Other reports of EHHMS have indicated suboptimal performance.Reference Boyce, Cooper, Lunde, Yin and Arbogast 6 , Reference Pineles, Morgan and Limper 8 , Reference Swoboda, Earsing, Strauss, Lane and Lipsett 9 However, even poorly performing systems are often framed as “improving compliance” and decreasing infection, despite limitations.Reference Swoboda, Earsing, Strauss, Lane and Lipsett 9 The growing body of literature on EHHMS is composed of diverse product types and evaluation methods, which reflects the differing objectives of centers testing these products.Reference Ward, Schweizer and Polgreen 4 , Reference Boyce 10 Furthermore, EHHMS products are continuously evolving. The 2 systems evaluated here are in no way comprehensive and cannot assist others in selecting specific products. Rather, we emphasize that healthcare systems considering implementation of EHHMS should rigorously evaluate the technology before committing. Furthermore, although some products perform well in a test environment, the addition of diverse clinical workflows can result in variable performance and failure to accurately capture HH events. Deficits in EHHMS accuracy may compromise the potential for enhancing patient safety.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/ice.2019.209
Acknowledgments
None.
Financial support
No financial support was provided relevant to this article.
Conflicts of interest
All authors report no conflicts of interest relevant to this article.
