Despite the critical importance of hand hygiene in the prevention of healthcare-associated infections (HAIs), the extent to which healthcare workers engage in appropriate levels of hand hygiene is unclear. The standard audit method of overtly measuring hand hygiene compliance (HHC) is subject to observer biases, and many hospitals report HHC rates that are misleadingly high. For example, in 2009 The Joint Commission led a quality improvement study targeting HHC, in which facilities reported compliance as high as 85%, rather than the 48% measured using more accurate methods.Reference Chassin and Loeb 1 One reason that data collected using standard audit methods might overestimate compliance is that audits are frequently conducted during a scheduled time by an auditor that is known by the unit staff. Thus, HHC rates observed using standard audit methods may be susceptible to bias from the Hawthorne effect.Reference Chen, Vander Weg, Hofmann and Reisinger 2 – Reference Maury, Moussa, Lakermi, Barbut and Offenstadt 6 The Hawthorne effect is a phenomenon in which people engage in higher levels of socially desirable behavior when they feel like they are being observed. The consequence of the Hawthorne effect for hand hygiene is that any HHC measurement that occurs when individuals are actively aware of being observed will likely lead to higher HHC estimates.Reference Dhar, Tansek and Toftey 7 – Reference Yin, Reisinger and vander Weg 13 Consistent with this prediction, a recent review of studies highlighted that HHC rates are significantly lower when observations are unannounced, short (<15 minutes), and conducted by “secret shoppers”: individuals hired to covertly observe HHC by surreptitiously appearing as a member of the public or as researchers studying a topic unrelated to HHC.Reference Chen, Vander Weg, Hofmann and Reisinger 2
This rapid (<15-minute) benchmark using covert observations was first established in a study at 5 ICUs and 6 medical/surgical wards.Reference Yin, Reisinger and vander Weg 13 A “secret shopper” sat and “read” in a hallway, recording entry and exit HHC covertly for 1 hour. The Hawthorne effect occurred most rapidly for exit compliance, shifting HHC from 56.2% to 60.5% after 14 minutes, and to 66.0% after 50 minutes of observation.Reference Yin, Reisinger and vander Weg 13 These shifts, which occurred during covert observation, are likely exacerbated when healthcare workers are being observed overtly. Electronic surveillance of HHC, which employees are aware of but habituate to, may be another method of reducing the Hawthorne effect. However, a recent systematic review concluded that there is not enough available evidence about the cost-effectiveness of these technologies, so the “secret shopper” methodology is currently the preferred method of covertly observing HHC.Reference Ward, Schweizer, Polgreen, Gupta, Reisinger and Perencevich 14
Despite evidence that standard audit methods produce inflated HHC estimates, no studies have examined the impact of providing feedback about HHC rates based on rapid, covert human observations. Feedback is a frequent fixture in bundled interventions to improve HHCReference Schweizer, Reisinger and Ohl 15 and should be incorporated into studies examining the impact of the Hawthorne effect on HHC estimates. While infectious disease specialists may be aware of the Hawthorne effect, it is unlikely that simply being aware of the effect would result in changes to audit methods. Most people exhibit a bias blind spot in which a person is aware of a bias that affects people but believe themselves to be immune or less susceptible to the bias.Reference Pronin 16 , Reference Scopelliti, Morewedge, McCormick, Min, Lebrecht and Kassam 17 Another potential issue is that designating additional resources to change audit methods may not be feasible if HHC is not a priority for the institution. As a result, further evidence is needed to highlight the influence of the Hawthorne effect on reported HHC estimates and whether feedback using the more accurate estimates obtained using rapid, covert observations impact HHC. Thus, the objectives of this study were (1) to compare HHC rates estimated using a rapid “secret shopper” methodology, which we refer to as the new audit method, to those using the standard audit method of overt observation and (2) to pilot test the impact of a novel feedback tool that communicated information obtained using the new audit method on HHC.
Methods
Setting
Data were collected from October 2016 to September 2017 at 5 acute-care Veterans Health Affairs (VHA) hospitals in the Midwest, constituting a total of 17 wards and 5 intensive care units. Reported monthly hospital HHC rates using the standard audit method ranged from 52% to 99% prior to participation in the study.
Design
The new audit method and feedback tool (both described below) were implemented using a nonrandomized (quasi-experimental) stepped wedge design as part of a quality improvement project focused on monitoring and improving HHC. The purpose of this initial period was to establish a baseline HHC rate using covert observation prior to the introduction of feedback. The baseline period began in October 2016 (1 site), November 2016 (2 sites), and February 2017 (2 sites; see Table 1 for a visual timeline). After 3 months of baseline data collection at a site, the person in charge of HHC at the site was provided with a feedback tool at the beginning of each month in the form of a poster that communicated information obtained from the new audit method. We purposefully provided no specific instructions on how the sites should use the feedback tool to get a sense of how hospitals might naturally utilize the tool. The Institutional Review Board of the University of Iowa granted an exemption for this quality improvement project.
Study Timeline
Materials and procedures
Audit methods
The new audit method, also described elsewhere,Reference Yin, Reisinger and vander Weg 13 , Reference Morgan, Pineles and Shardell 18 , Reference Chang, Reisinger and Jesson 19 involved having an employee previously unknown to hospital staff covertly record entry and exit HHC for a maximum duration of 15 minutes outside of a randomly determined room before moving to a new room on a different unit. Observers attempted to remain inconspicuous and were provided with a prepared story about observing human factors related to the healthcare worker movement, if asked (which happened rarely after the first few weeks of study). An average of 630 observations per month were recorded by the observer at each site. Observers received 2-day standardized virtual observation training conducted by the project coordinator. Day 1 included a 1.5-hour review of the study and observation protocol and 1 hour of practice observations on their units. Day 2 consisted of a review of their observations and time for questions and concerns. A recurring weekly call with the observers provided an opportunity to review the previous week and have any questions answered. The standard audit method consisted of the existing HHC monitoring protocol at each site, all of which included observations being recorded by a hospital employee known to the other hospital staff.
Novel feedback tool
The research team members responsible for the design of the feedback poster included social scientists with experience designing feedback tools for HHC and laboratory test results (HSR, AMS), infectious disease specialists (MG, ARM, EP), and qualitative researchers (EEC, CCG). The feedback posters were designed to communicate 3 key pieces of information about the HHC rate obtained using the new audit method.
1. Relative change in monthly HHC rates. Our motivation to present relative change in HHC rate, rather than the exact numeric HHC rate, was inspired by The Joint Commission recommendation that facilities should focus on improving their HHC rates each month, rather than on hitting a specific target percentage. 20 An upward arrow was used to indicate improvement in HHC relative to the previous month, a downward arrow was used for decreased HHC, and a dash was used to indicate no change (Fig. 1). Relative change information was provided for each month going back a maximum of 6 months. To increase the salience of the most recent HHC rate, each arrow became smaller and increased in transparency as the temporal distance increased.
2. “Gist” of the absolute HHC rate. After viewing a generic version of the feedback tool during the baseline phase, some sites expressed concern about the absence of the exact numeric HHC rate. As a result, we indicated the gist (ie, the essential take-away) of the site’s numeric HHC rate in 2 ways. The first method involved the arrows and dashes indicating the relative change in HHC rates being colored to indicate whether the site’s numeric HHC was in the “good” (green), “okay” (yellow), or “bad” (red) range. The values in each range (0–50% = bad; 50%–75% = okay; 75%–100%=good) were determined by the infectious disease specialists on the research team (M.G., A.R.M., and E.P.), and the stoplight color theme was selected based on previous research testing feedback tools.Reference Thirkell, Chambers, Gilbart, Thornhill, Arbogast and Lacey 21 – Reference Zikmund-Fisher, Scherer, Witteman, Solomon, Exe and Fagerlin 24 The legend communicated the color-evaluative label correspondence without specifying what values constituted those ranges (Fig. 1). The second method, which was never mentioned to the participating sites, was a picture containing a Petri dish with bacteria growth from a handprint. The picture of the Petri dish contained a small, moderate, or large amount of bacteria to indicate that HHC was in the “good,” “okay,” or “bad” range, respectively (see Fig. 1 for an example of a “bad” range Petri dish). Gist information represented via color was provided for each month going back a maximum of 6 months, whereas the Petri dish picture only represented the gist information for the current month.
3. Social-comparison information. Comparisons to others can serve as a motivator to positive behavior change, including HHC.Reference Huis, van Achterberg, de Bruin, Grol, Schoonhoven and Hulscher 25 We included how the site’s HHC rate for the current month compared to a 3-month weighted average (based on facility size) for all 5 participating VHA hospitals. This social-comparative information was communicated in a stand-alone box indicating whether the current HHC rate at the site was “above average” in green font, “average” in yellow font, or “below average” in red font (see Fig. 1 for an example).
Example of the feedback tool. VISN, Veterans Integrated Service Network. Notes: Arrow direction indicates whether rate improved, stayed the same or decreased; Color indicates whether HH rate is good (green), ok (yellow), or bad (red); VISN average based on 3-month weighted average based on facility sizes.
Measures
HHC rate. HHC was calculated as the percentage of times hospital staff washed their hands with soap and water or used alcohol-based hand sanitizer prior to entering or after exiting a patient’s room. HHC rates from both audit methods were calculated monthly and for each study phase (ie, baseline and feedback phases).
Statistical analyses
Using the power formula of Hussey and HughesReference Hussey and Hughes 26 and assuming a type I error rate of 5% for a 2-sided test, we calculated that our planned sample size (200 observations per month) had 87.9% power to detect a 0.05 (eg, 60% to 65%) difference in HHC. We used descriptive statistics to report HHC rates using data from both audit methods. We used a generalized linear mixed model to examine changes in HHC rates based on study phase and audit method, controlling for calendar month and bed size as fixed effects and facility as a random effect. Data weights were calculated by dividing the number of beds at a site by the total number of beds at the 5 sites.
Results
In total, 9,301 observations were recorded using the standard audit method for the baseline phase and 28,457 were recorded for the feedback phase. In addition, 3,281 observations were recorded using the new audit method for the baseline phase, and 7,576 observations were recorded for the feedback phase. The overall unweighted and weighted HHC rates for the 2 study phases obtained via both audit methods are presented in Fig. 2, and Fig. 3 presents the HHC rates from both audit methods for each study month (study month 1 = first month of site participation). Although the weighted HHC rates for the baseline and feedback phases using the standard audit method were ~80%, the rates from the new audit method were both slightly less than 50%, an ~30% absolute difference between the 2 audit methods for both study phases.
Unweighted and weighted hand hygiene compliance rates by audit method and study phase.
Hand hygiene compliance rates by study implementation month. Study month 1 = first month a site began baseline observations.
Results from the generalized linear mixed model with a random effect confirmed the observed differences in HHC rates between the 2 audit methods, with an almost 10-fold increase (odds ratio [OR], 9.83; 95% confidence interval [CI], 8.82–10.95) in the likelihood of reported compliance using the standard audit method compared to the new audit method (Table 2). This means that a healthcare worker would be 10 times more likely to be recorded as engaging in appropriate hand hygiene with the standard audit method compared to the new audit method. We found no observed association of study phase on HHC, but we detected a significant interaction between study phase and audit method. Specifically, the likelihood of reporting appropriate HHC was higher in the feedback phase compared to the baseline phase for the standard audit method, but we found no evidence of change in HHC across the 2 phases for the novel audit method (see Fig. 2). We also noted a significant decrease in the likelihood of reporting compliance as the bed size of the hospital increased (controlling for method).
Generalized Linear Mixed Model for Hand Hygiene Compliance Rates for Baseline and Feedback Phases With a Random Effect for Facility
Note. OR, odds ratio; CI, confidence interval.
Discussion
The current study contributes to the growing body of research demonstrating that the Hawthorne effect introduces a significant amount of bias when HHC is measured via overt observation, the method almost universally utilized by hospitals.Reference Chen, Vander Weg, Hofmann and Reisinger 2 – Reference Maury, Moussa, Lakermi, Barbut and Offenstadt 6 In our study, there was an ~30% absolute difference in estimated HHC rates between the standard and new audit methods. As the evidence mounts that hospitals and healthcare systems are significantly overestimating HHC, there should be increased demand that they change how they measure HHC to provide more accurate representations of their compliance rates. The standard audit method has value insofar as estimates of HHC using overt observation are negatively associated with rates of HAIs,Reference Gould, Moralejo, Drey and Chudleigh 27 , Reference Stiller, Salm, Bischoff and Gastmeier 28 revealing relative differences in HHC across facilities. However, because the standard method also seems to inflate estimates of HHC, using overt observations may undermine additional efforts to improve HHC by promoting the belief that one’s unit, hospital, or healthcare system is already near the top of what is possible for HHC and, as a result, put patients at risk.
Our study also represents the first attempt to provide feedback based on rapid, covert observations of HHC by pilot testing a feedback tool that we developed. While a more formal analysis of the potential barriers and facilitators to the implementation of the feedback tool is needed (and is currently underway in our research group), preliminary analyses indicate that the primary reason the feedback tool may not have produced any meaningful change in HHC is that the tool was not widely disseminated beyond the leadership and nurse managers at each facility. We purposefully provided no guidance about how to use the feedback tool, but simply sent it to the person responsible for HHC at a given site to see how it would be naturally used and disseminated. Although it is possible that the feedback was used to make top-down changes in policy to promote HHC, it is also possible that the tool needed to be shared more broadly with hospital staff to produce bottom-up changes in HHC behaviors. There are a number of potential explanations for why the feedback tool was not widely disseminated including, but not limited to, ambiguity aversion Reference Camerer and Weber 29 , Reference Politi, Han and Col 30 resulting from the absence of precise numeric HHC estimates; risk aversion Reference Tversky and Kahneman 31 , Reference Kahneman 32 over concerns about presenting information indicating substantially lower HHC rates than believed; and/or increased scrutiny or disbelief about the feedback information since it conflicted with expectations or desires (a process called biased assimilation).Reference Lord, Ross and Lepper 33 , Reference Bastardi, Uhlmann and Ross 34 Also, the feedback tool may have presented information in a way that was difficult to interpret. Identifying the specific barriers will be necessary to determine what changes are needed in future attempts to provide feedback about HHC based on rapid, covert observations.
Additional research is also necessary to determine whether the size of the Hawthorne effect varies across different locations (ie, unit, facility, healthcare system) and, if variability exists, whether the size of the effect is independently associated with HAIs. For example, would HAI risk be higher for a hospital that reports HHC of 90% but covert observations suggest HHC is 65% (difference of 25%) compared to a hospital that reports HHC of 80% when covert observations suggest HHC is 60% (difference of 20%). In other words, we do not know whether either method of measuring HHC, or the size of the discrepancy between the 2 methods, is more predictive of HAIs. Although more accurate measures of HHC would have a stronger association with HAI rates, it is also possible that facilities may be more responsive to the less accurate, but more familiar HHC rates calculated via overt observation. Another possibility is that, similar to the “unskilled and unaware” phenomenon in which individuals with the lowest levels of skill significantly overestimate their abilities,Reference Kruger and Dunning 35 hospitals with the largest discrepancies between HHC estimates from overt and covert observations may have the highest risk of patients developing HAIs. Which of these possibilities is true is currently an open question and should be addressed in future research exploring the Hawthorne effect in HHC to demonstrate the superiority of assessing HHC using overt observations, covert observations, or both. Relatedly, this research could also try to determine the influence of the Hawthorne effect relative to other potential sources of bias, such as observers ‘selectively’ observing compliance to avoid confrontations with staff.
Our study was not without limitations. First, the study was limited to a specific region of the United States and VA hospitals. However, our results were consistent with the growing body of evidence for the Hawthorne effect in HHC. Additionally, we purposefully allowed each site to determine how they would use the feedback tool which a priori excludes generalizability. Another potential limitation was our decision to not feedback a site’s numeric HHC rate based on the new audit method. It is possible that HHC information provided in numeric form (1) may have been more widely disseminated and/or (2) may have had a stronger effect on HHC rates. Although we wanted the sites to focus on improving HHC, rather than simply striving for a target value, future research should compare the efficacy of feedback tools with and without numeric HHC rates.
The current study contributes to the growing body of research highlighting that HHC rates from overt observation, which many consider the current “gold standard,” should be highly suspect. Our initial attempt at providing feedback about a site’s HHC from rapid, covert observations, which produced substantially lower estimates than those obtained from overt observation, appeared to have no impact on HHC. Despite the apparent failure of our feedback tool to improve HHC, the rates observed through the new audit method highlight that hospitals and healthcare systems should take the impact of the Hawthorne effect seriously, while researchers should continue to develop and test better ways of accurately measuring and meaningfully communicating HHC rates.
Financial support
This work was supported by Quality Enhancement Research Initiative (QUERI) Award QUE 15-269 (grant no. 1IP1HX001993) from the US Department of Veterans Affairs Health Services Research and Development Service. The contents do not represent the views of the US Department of Veterans Affairs or the US government.
Conflicts of interest
All authors report no conflicts of interest relevant to this article.
