BACKGROUND

The fields of healthcare epidemiology and antimicrobial stewardship (HE&AS) frequently apply interventions at a unit level (eg, the intensive care unit [ICU]). These interventions are often rapid responses to outbreaks or other patient safety problems requiring prompt, nonrandomized interventions. Quasi-experimental studies evaluate the association between an intervention and an outcome using experiments in which the intervention is not randomly assigned.Reference Harris, Bradham, Baumgarten, Zuckerman, Fink and Perencevich ^{1 ,} Reference Shadish, Cook and Campbell ² Quasi-experimental studies can be used to measure the impact of large-scale interventions or policy changes in which data are reported in aggregate and multiple measures of an outcome over time (eg, monthly rates) are collected.

Quasi-experimental studies vary widely in methodological rigor and can be categorized into 3 types: interrupted time-series designs, designs with control groups, and designs without control groups. The HE&AS literature contains many uncontrolled before-and-after studies (also called pre-post studies), but advanced quasi-experimental study designs should be considered to overcome the biases inherent in uncontrolled before-and-after studies.Reference Grimshaw, Campbell, Eccles and Steen ³ In this article, we highlight methods to improve quasi-experimental study design, including the use of a control group that does not receive the interventionReference Shadish, Cook and Campbell ² and the use of the interrupted time series study design, in which multiple equally spaced observations are collected before and after the intervention.Reference Shardell, Harris, El-Kamary, Furuno, Miller and Perencevich ⁴

ADVANTAGES AND DISADVANTAGES

The greatest advantages of quasi-experimental studies are that they are less expensive and require fewer resources than individual randomized controlled trials (RCTs) or cluster randomized trials (Table 1). Quasi-experimental studies are appropriate when randomization is deemed unethical (eg, in studies of the effectiveness of hand hygiene protocols).Reference Harris, Bradham, Baumgarten, Zuckerman, Fink and Perencevich ¹ With IRB approval as appropriate, quasi-experimental studies are often performed at a population level rather than an individual level; thus, they can include patients who are often excluded from RCTs, such as those too ill to give informed consent or urgent surgery patients.Reference Thorpe, Zwarenstein and Oxman ⁵ Quasi-experimental studies are also pragmatic because they evaluate the real-world effectiveness of an intervention implemented by hospital staff rather than the efficacy of an intervention implemented by research staff under research conditions.Reference Thorpe, Zwarenstein and Oxman ⁵ Therefore, quasi-experimental studies may also be more generalizable and have better external validity than RCTs.

TABLE 1

Advantages, Disadvantages, and Important Pitfalls in Using Quasi-Experimental Designs in Healthcare Epidemiology Research

NOTE. RCT, randomized controlled trial.

The greatest disadvantage of quasi-experimental studies is that randomization is not used, which limits the study’s ability to reveal a causal association between an intervention and an outcome. A practical challenge to quasi-experimental studies may arise when some hospital units are encouraged to introduce an intervention, while other units retain the standard of care and may feel excluded.Reference Shadish, Cook and Campbell ² Importantly, researchers need to be aware of the biases that may occur in quasi-experimental studies that may lead to a loss of internal validity, especially selection bias in which the intervention group may differ from the baseline group.Reference Shadish, Cook and Campbell ² Types of selection bias that can occur in quasi-experimental studies include maturation bias, regression to the mean, historical bias, instrumentation bias, and the Hawthorne effect.Reference Shadish, Cook and Campbell ² Lastly, reporting bias is prevalent in retrospective quasi-experimental studies in which researchers publish only quasi-experimental studies with positive findings and do not publish null or negative findings.

PITFALLS AND TIPS

Key study design and analytic approaches can help avoid common pitfalls of quasi-experimental studies. Quasi-experimental studies can be as small as an intervention in a single ICU or as large as implementation of an intervention in multiple countries.Reference Lee, Cooper and Malhotra-Kumar ⁶ Multisite studies generally have stronger external validity. Subtypes of quasi-experimental study designs are shown in Table 2 and the Supplemental Figure.Reference Harris, Bradham, Baumgarten, Zuckerman, Fink and Perencevich ^{1 ,} Reference Shadish, Cook and Campbell ^{2 ,} Reference Harris, Lautenbach and Perencevich ⁷ In general, the higher numbers assigned to the designs in the table are associated with more rigorous study designs. Quasi-experimental studies meet some requirements for causality, including temporality, strength of association, and dose response.Reference Harris, Bradham, Baumgarten, Zuckerman, Fink and Perencevich ^{1 ,} Reference Hill ⁸ The addition of concurrent control groups, time-series measurements, sensitivity analyses, and other advanced design elements can further support the hypothesis that the intervention is causally associated with the outcome. These design elements aid in limiting the number of alternative explanations that could account for the association between the intervention and the outcome.Reference Shadish, Cook and Campbell ²

TABLE 2

Major Quasi-Experimental Design Types and Subtypes

NOTE. Classification types adapted prior publicationsReference Harris, Bradham, Baumgarten, Zuckerman, Fink and Perencevich ^{1 ,} Reference Shadish, Cook and Campbell ² ; A=primary group of interest; B=control group; 1,2,3, etc.= observations for a Group; X=intervention; remove X=remove intervention; ^v=variable of interest; ⁿ=non-equivalent dependent variable; t=treatment group; c=no treatment. Time moves from left to right. Citations are published examples from the literature.

Quasi-experimental studies can use observations that were collected retrospectively, prospectively, or a combination thereof. Prospective quasi-experimental studies use baseline measurements that are calculated prospectively for the purposes of the study, then an intervention is implemented and more measurements are collected. It is often necessary to use retrospective data when the intervention is outside the researcher’s control (eg, natural disaster response) or when hospital epidemiologists are encouraged to intervene quickly in response to external pressure (eg, high central-line–associated bloodstream infection [CLABSI] rates).Reference Shadish, Cook and Campbell ² However, retrospective quasi-experimental studies are at a higher risk of bias than prospective quasi-experimental studies.Reference Shadish, Cook and Campbell ²

The first major consideration in quasi-experimental studies is the addition of a control group that does not receive the intervention (Table 2, subtypes 6–9, 11, and 15). Control groups can assist in accounting for seasonal and historical biases. If an effect is seen among the intervention group but not the control group, then causal inference is strengthened. Careful selection of the control group can also strengthen causal inference. Detection bias can be avoided by blinding those who collect and analyze the data to which group received the intervention.Reference Shadish, Cook and Campbell ²

The second major consideration is designing the study to reduce bias, either by including a non-equivalent dependent variable or by using a removed-treatment design, a repeated treatment design, or a switching replications design. Non-equivalent dependent variables should be similar to the outcome variable except that the non-equivalent dependent variable is not expected to be influenced by the outcome (Table 2, subtypes 3 and 12). In a removed-treatment design, the intervention is implemented then taken away, and observations are made before, during, and after implementation (Table 2, subtypes 4, 5, and 13). This design can only be used for interventions that do not have a lasting effect on the outcome that could contaminate the study. For example, once staff members have been educated, that knowledge cannot be removed.Reference Shadish, Cook and Campbell ² Researchers must clearly explain before implementation that the intervention will be removed; otherwise, this can lead to frustration or demoralization by the hospital staff implementing the intervention.Reference Shadish, Cook and Campbell ² In the repeated-treatment design (Table 2, subtypes 5 and 14) interventions are implemented, removed, then implemented again. Similar to the removed-treatment design, the repeated-treatment design should only be used if the intervention does not have a lasting effect on the outcome. In a switching replications design, also known as a crossover design, one group implements the intervention while the other group serves as the control. The intervention is then stopped in the first group and implemented in the second group (Table 2, subtypes 9 and 15). The crossovers can occur multiple times. If the outcomes are only impacted during intervention observations but not in the control observations, then there is support for causality.Reference Shadish, Cook and Campbell ²

A third key consideration for quasi-experimental studies with an interrupted time-series design is to collect many evenly spaced observations during both the baseline and intervention periods. Multiple observations are used to estimate and control for underlying trends in data, such as seasonality and maturation.Reference Shadish, Cook and Campbell ² The frequency of the observations (eg, weekly, monthly, or quarterly) should have clinical or seasonal meaning so that a true underlying trend can be established. There are conflicting recommendations regarding the minimum number of observations needed for a time-series design, but they range from 20 observations before and 20 after intervention implementation to 100 observations overall.Reference Shadish, Cook and Campbell ^{2 –} Reference Shardell, Harris, El-Kamary, Furuno, Miller and Perencevich ^{4 ,} Reference Crabtree, Ray, Schmidt, O’Connor and Schmidt ⁹ The interrupted time-series design is the most effective and powerful quasi-experimental design, particularly when supplemented by other design elements.Reference Shadish, Cook and Campbell ² However, time-series designs are still subject to biases and threats to validity.

The final major consideration is ensuring an appropriate analysis plan. Time-series study designs collect multiple observations of the same population over time, resulting in auto-correlated observations.Reference Shadish, Cook and Campbell ² For instance, carbapenem-resistant Enterobacteriaceae (CRE) counts collected 1 month apart are more similar to one another than CRE counts collected 2 months apart.Reference Shardell, Harris, El-Kamary, Furuno, Miller and Perencevich ⁴ Basic statistics (eg, χ² test) should not be used to analyze time-series data because they cannot take into account trends over time and they rely on an independence assumption. Time-series data should be analyzed using either regression analysis or interrupted time-series analysis (ITSA).Reference Shardell, Harris, El-Kamary, Furuno, Miller and Perencevich ⁴ Linear regression models or generalized linear models can be used to evaluate the slopes of the observed outcomes before and during implementation of an intervention. However, unlike regression models, ITSA relaxes the independence assumption by combining a correlation model and a regression model to effectively remove seasonality effects before addressing the impact of the intervention.Reference Shadish, Cook and Campbell ^{2 ,} Reference Shardell, Harris, El-Kamary, Furuno, Miller and Perencevich ⁴ ITSA assesses the impact of the intervention by evaluating the changes in the intercept and slope before and after the intervention. ITSA can also include a lag effect if the intervention is not expected to have an immediate result, and additional sensitivity analyses can be performed to test the robustness of the findings. We recommend statistician consultation while designing the study to choose the most appropriate model and to help perform power calculations that account for correlation.

Key considerations for designing, analyzing, and writing a quasi-experimental study can be found in the Transparent Reporting of Evaluations with Nonrandomized Designs (TREND) statement and are summarized in Table 3.Reference Des Jarlais, Lyles and Crepaz ¹⁰

TABLE 3

Checklist of Key Considerations When Developing a Quasi-Experimental Study

EXAMPLES OF PUBLISHED QUASI-EXPERIMENTAL STUDIES IN HE&AS

Recent quasi-experimental studies illustrated strengths and weaknesses that require attention when employing this study design.

A recent prospective quasi-experimental study (Table 2, subtype 10) implemented a multicenter bundled intervention to prevent complex Staphylococcus aureus surgical-site infections.Reference Schweizer, Chiang and Septimus ¹¹ The study exemplified the strengths of quasi-experimental design using a pragmatic approach in a real-world setting that even enabled identification of a dose response to bundle compliance. To optimize validity, the authors included numerous observation points before and after the intervention and used time-series analysis. This study did not include a concurrent control group, and outcomes were collected retrospectively for the baseline group and prospectively for the intervention group, which may have led to ascertainment bias.

Quach et alReference Quach, Milstone, Perpete, Bonenfant, Moore and Perreault ¹² performed a quasi-experimental study (Table 2, subtype 11) to evaluate the impact of an infection prevention and quality improvement intervention of daily chlorhexidine gluconate (CHG) bathing to reduce CLABSI rates in the neonatal ICU. The primary strength of this study was that the authors used a non-bathed concurrent control group. Given that the baseline rates of CLABSI exceeded the National Healthcare Surveillance Network (NHSN) pooled mean and that the observation that the concurrent control group did not see a reduction in rates post-intervention, the treatment effect was more likely due to the treatment than to regression to the mean, seasonal effects, or secular trends.

Yin et alReference Yin, Schweizer, Herwaldt, Pottinger and Perencevich ¹³ performed a quasi-experimental study (Table 2, subtype 14) to determine whether universal gloving reduced HAIs in hospitalized children. This retrospective study compared the winter respiratory syncytial virus (RSV) season during which healthcare workers (HCWs) were required to wear gloves for all patient contact and the non-winter, non-RSV season when HCWs were not required to wear gloves. Because the study period extended many calendar years, the design facilitated multiple crossovers removing the intervention as well as the use of time-series analysis. This study did not have a control group (another hospital or unit that did not require universal gloving during RSV season) nor did it have a nonequivalent dependent variable.

MAJOR POINTS

Quasi-experimental studies are less resource intensive than RCTs; they test real-world effectiveness; and they can support a hypothesis that an intervention is causally associated with an outcome. These studies are subject to biases that can be limited by carefully planning the design and analysis. Several key strategies to limiting bias should be considered: including a control group, including a non-equivalent variable or removed-treatment design, collecting adequate observations before and during the intervention, and using appropriate analytic methods (ie, ITSA).