Studies using quasi-experimental study designs or pre- and postintervention studies are nonrandomized studies used to assess the effectiveness of specific interventions.Reference Shadish, Cook and Campbel 1 The nonrandom assignment of study interventions in quasi-experiments poses threats to causal inference and frequently requires careful selection of comparison periods and/or groups to control for potential confounders.Reference Shadish, Cook and Campbel 1 – Reference Schweizer, Braun and Milstone 5 Additionally, nonrandomization and other design-related factors create statistical challenges in analyzing quasi-experimental studies.Reference Shardell, Harris, El-Kamary, Furuno, Miller and Perencevich 2 In quasi-experimental studies, internal validity and strength of evidence are dependent on the study design and the ability to control for data-related factors such as confounding, correlation, and possible seasonal or time trends.Reference Shadish, Cook and Campbel 1 – Reference Harris, Bradham, Baumgarten, Zuckerman, Fink and Perencevich 3 Despite these limitations, a quasi-experimental design may allow for causal interpretation of observed association when randomization is not feasible or ethical, as is often the case in infection control and antibiotic resistance research. Thus, quasi-experiments are a valuable alternative to randomized controlled trials, requiring less time and resources to evaluate the effectiveness of specific interventions.Reference Schweizer, Braun and Milstone 5
In the field of infectious diseases, quasi-experimental study designs are frequently used to assess the effectiveness of prevention and control measures for healthcare-associated infections and antibiotic-resistant pathogens,Reference Harris, Bradham, Baumgarten, Zuckerman, Fink and Perencevich 3 , Reference Harris, Lautenbach and Perencevich 6 as well as to study the effectiveness of vaccines. A hierarchy of quasi-experimental designs in studies of infectious diseases has been published to elucidate their ability to establish causal associations.Reference Harris, Bradham, Baumgarten, Zuckerman, Fink and Perencevich 3 Furthermore, statistical techniques and their application to quasi-experiments in studies of infectious diseases have also been published previously.Reference Shardell, Harris, El-Kamary, Furuno, Miller and Perencevich 2
The scientific literature has required increasingly rigorous reporting standards for different types of epidemiological study designs. The Consolidated Standards of Reporting Trials (CONSORT) guidelines for randomized trials,Reference Schulz, Altman, Moher and Group 7 the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement for observational studies,Reference von Elm, Altman and Egger 8 and the Preferred Reporting Items for Systematic Review and Meta-analysis (PRISMA) guidelines for systematic reviewsReference Moher, Liberati, Tetzlaff, Altman and Group 9 have improved the reporting of these study designs. While the use of quasi-experiments is frequent in the fields of infection control and antibiotic resistance, literature regarding the implementation and optimization of the quasi-experimental design in studies of infectious diseases remains scarce. In 2005, a systematic review was conducted to assess the use and knowledge of quasi-experimental studies related to infection control and antibiotic resistance among those published in 4 major journals in this field between January 1, 2003, and December 31, 2004.Reference Harris, Lautenbach and Perencevich 6 The aim of the previous review was to identify possible limitations in the published literature and areas for potential improvement. Studies were evaluated based on the following criteria: “type of quasi-experimental study design used,” “justification of the use of the design,” “use of correct nomenclature to describe the design,” and “recognition of potential limitations of the design.” Overall, 73 articles that used the quasi-experimental study design were identified. Among them, 16% of articles utilized a design involving a control group; 4% justified the use of a quasi-experimental design; 22% used the correct nomenclature; and 23% reported at least 1 potential limitation of the design.Reference Harris, Lautenbach and Perencevich 6 In 2007, the Outbreak Reports and Intervention Studies of Nosocomial infection (ORION) statement was published.Reference Schweizer, Braun and Milstone 5 In contrast to the criteria outlined in the 2005 systematic review, which provided specific recommendations on the design and nomenclature of quasi-experimental studies, the ORION statement provides a checklist for reporting outbreak and other intervention studies of hospital-acquired infections.
In this review, we identified and systematically reviewed quasi-experimental studies related to infection control and antibiotic-resistance published in 2013–2014 in the same 4 journals included in the 2005 review. In addition to the aforementioned criteria, we also sought to report the statistical methods used to analyze the quasi-experimental data. The aim of this study was to determine the number of quasi-experimental studies published, to determine whether there were improvements in the design and reporting of quasi-experimental studies 10 years after the initial review and to re-evaluate areas of potential improvement.
METHODS
To identify quasi-experimental studies, we systematically reviewed articles published between January 1, 2013, and December 31, 2014, in the journals included in the original review: American Journal of Infection Control, Clinical Infectious Diseases, Emerging Infectious Diseases, and Infection Control and Hospital Epidemiology. Among the 5 authors of the current review, 2 authors (R.A. and A.H.) reviewed the title and abstract of all original-research articles published in these journals during the aforementioned study period to identify articles utilizing a quasi-experimental study design. Outbreak reports were not considered quasi-experimental studies because they are often unplanned, involve simultaneous interventions, or are not useful in assessing the effectiveness of specific interventions.Reference Stone, Cooper and Kibbler 10 Quasi-experimental studies focusing on infection control and antibiotic resistance were subsequently classified on the basis of the following 4 criteria: (1) type of quasi-experimental study design used, (2) justification of the use of the design, (3) use of correct nomenclature to describe the design, and (4) statistical methods used.
Criterion 1: Type of Quasi-experimental Study Design Used
The hierarchy for quasi-experimental designs in the field of infectious diseases was described previously (Figure 1).Reference Harris, Lautenbach and Perencevich 6 Briefly, quasi-experimental designs were separated into 2 categories: quasi-experimental designs that do not use a control group (category A) and quasi-experimental designs that use control groups (category B). Quasi-experiments that feature the use of a control group are generally stronger from a methodological perspective than studies that do not. Additionally, between and within each category, the strength of the designs increases moving downward in the hierarchy (ie, the strength of design A5 is higher than that of A1) (Figure 1). We reviewed each article to determine the type of quasi-experimental study design used. Articles that displayed multiple preintervention data points within tables or figures, even when a statistical comparison between preintervention time points was not conducted, were rated A2 if the study did not utilize a control group or were rated B2 when a control group was present and not as A1 or B1, respectively.
Hierarchy of quasi-experimental study designs.Reference Harris, Lautenbach and Perencevich 6 Note: O, observational measurement; X, intervention under study; time moves from left to right.
Criterion 2: Justification of the Use of the Quasi-experimental Study Design
Randomized controlled trials are considered the “gold standard” to establish a causal link between an intervention and the study outcome. The use of quasi-experimental studies is a valuable alternative when randomization is ethically or logistically infeasible. We reviewed the articles to determine whether the authors mentioned why they chose the quasi-experimental design. This criterion was rated “yes” or “no.” If the authors explained or justified the use the quasi-experimental study design (eg, because a randomized controlled trial was not possible), the design was given a “yes” for this criterion.
Criterion 3: Use of Correct Nomenclature to Describe the Quasi-experimental Study Design
We reviewed the articles to determine whether the authors correctly identified their study as a quasi-experimental study. This criterion was rated as “yes” or “no,” and acceptable nomenclature included “quasi-experimental,” “before–after,” “pre–post,” and/or “interrupted time-series.”
Criterion 4: Statistical Methods Used to Analyze Data From a Quasi-experimental Study
This criterion was the only criterion not assessed in the original paper. In 2007, a study outlining statistical techniques and their application to quasi-experimental studies in infection control and antibiotic resistance was published.Reference Shardell, Harris, El-Kamary, Furuno, Miller and Perencevich 2 These statistical techniques include 2-group tests, regression analysis, and time-series analysis and have been described in detail previously.Reference Shardell, Harris, El-Kamary, Furuno, Miller and Perencevich 2 Briefly, 2-group tests (eg, Student’s t test or χ2 test) make crude or unadjusted comparisons of the study outcome between pre- and postintervention periods. They are simple and require minimal data, but they are limited by their incorrect assumption of independence between individuals and between time periods and their inability to control for confounders or detect changes in temporal trends. Simple regression models allow for multivariable analyses when evaluating associations between an intervention and an outcome, permitting statistical adjustments for measured confounders; however, they also assume independence between observations. Time-series analyses account for autocorrelation between measurements collected at different time points by specifying a correlation model in addition to the regression model, thereby relaxing the independence assumption.Reference Shardell, Harris, El-Kamary, Furuno, Miller and Perencevich 2 Time-series analyses are used to predict future values based on data collected at successive time points, but they are limited by the number of observations required (≥50 overall and ≥10 per parameter).Reference Shardell, Harris, El-Kamary, Furuno, Miller and Perencevich 2 Models in regression and time-series analyses can be either standard or segmented. Segmented models estimate changes in slopes (trends) and intercepts (mean outcome levels) from pre- to postintervention periods. In general, a segmented model is preferable to a standard model, and time-series analysis is preferable to regression analysis and 2-group tests, when applicable. We reviewed the articles to determine the type of statistical methods used, which were categorized as follows: 2-group analysis, standard regression analysis, segmented regression analysis, standard time-series analysis, or segmented time-series analysis. If no statistical comparisons were made, the study was classified as “not applicable” for this criterion.
The 2 reviewers independently reviewed all quasi-experimental studies in their entirety and classified them per criteria 1–4; R.A. reviewed all articles, and the second review was equally divided among the remaining 4 authors. In cases where the 2 reviewers disagreed on any classification, a third investigator reviewed the article, and/or a group discussion resolved the disagreement.
RESULTS
Among 2,600 articles published in the 4 infectious disease journals within the 2-year study period, we identified 173 (7%) studies that featured a quasi-experimental study design related to infection control and antibiotic resistance (Figure 2, see online supplementary material), more than twice the number identified from the previous review (73 of 2,320; 3%; P<.01). The 173 articles fell into the following category topics: healthcare-associated infections (n=52, 30%), hand hygiene (n=27, 16%), environment (n=24, 14%), antibiotic resistance (n=23, 13%), vaccination (n=18, 10%), other (n=16, 9%), and antibiotic stewardship (n=13, 8%). Moreover, 63 articles (36%) explicitly used the term “hypothesis” or “aim” in reference to their study. However, 164 articles (95%) articulated the hypothesis or aim using “hypothesis,” “aim,” or other terms.
Table 1 outlines the results for criteria 1–4 from the current and original reviews. Among the 173 quasi-experimental studies, 21 articles (12%) featured a design that used control groups (category B). Furthermore, 143 reviewed articles (83%) used an A1 (n=76, 44%) or A2 (n=67, 39%) design.
Quasi-experimental Design Type, Justification, Nomenclature, and Statistical Methods Comparing the 2003–2004 Systematic Review to the 2013–2014 Systematic Review From 4 Infectious Disease Journals
Our second criterion assessed whether the authors justified the use of the quasi-experimental study design; 6 articles (3.5%) were rated “yes” for this criterion.
Our third criterion assessed whether the authors correctly identified their study as a quasi-experimental study. Only 68 articles (39%) clearly identified their study design using the correct nomenclature described above. Among those, 34 articles (20%) used the terms “quasi-experimental” to describe their study design. Notably, 50 additional articles (29%) used the terms “pre–post” or “before–after” throughout their papers, but did not explicitly identify their study design as a “pre–post” or “before–after” intervention study. Some examples of inaccurate nomenclature used included “case-control design,” “cohort study,” “repeated cross-sectional study,” and “self-controlled case series design.”
Our fourth criterion described the type of statistical method used to analyze quasi-experimental data. Of the 173 studies, 75 (43%) used 2-group tests, 58 (34%) used standard regression analysis, 18 (10%) used segmented regression analysis, 7 (4%) used standard time-series analysis, 5 (3%) used segmented time-series analysis, and 10 (6%) did not utilize statistical methods for comparisons.
DISCUSSION
In this review, 173 studies in these 4 journals (7%) were quasi-experimental studies on the topics of infection control and antibiotic resistance. This proportion is most likely an underrepresentation of the overall use of quasi-experimental studies in the field as the review was limited to the content areas of infection control and antibiotic resistance. Notably, the proportion of published quasi-experimental studies more than doubled (P<.01) when comparing the findings from 2003–2004 (3%) to those from 2013–2014 (7%) indicating that this design is becoming increasingly popular in this field.
Overall, very few studies used higher-quality study designs; 76 articles (44%) featured an A1 design, the weakest within the hierarchy. However, this percentage is lower than the proportion of studies utilizing an A1 design observed in the previous review (n=39, 53%), suggesting some improvement over the decade (P=.17). Previously, no studies were classified as A3, A4, or B2, whereas 16 studies in this review (9%) were characterized as one of these designs, also suggesting some improvement (P<.01). While there is more variability in the type of quasi-experimental study designs used compared to the previous review, the proportion of studies featuring a study design that used a control group (category B) was lower in the current review (n=21, 12% in 2013–2014 vs n=16, 22% in 2003–2004; P=.05). Importantly, as previously stated, articles that reported multiple preintervention data points within tables or figures, even when statistical comparisons between preintervention time points were not conducted, were considered as designs that used double pretests (A2 or B2). Thus, the classification relative to criterion 1 could have been even lower with more stringent criteria.
Only 6 studies (3.5%) justified the use of the quasi-experimental design, a slightly lower proportion than observed in the previous review (n=3, 4%; P=.81). We evaluated this criterion to assess any improvements from the original review, but we do not believe this is a critical criterion because in the field of infection control and antibiotic resistance, randomization is often not ethically or logistically feasible. On the other hand, we found that a higher proportion of authors correctly identified their studies using accurate nomenclature in this review compared to the previous review (n=68, 39% vs n=16, 22%, respectively; P<.01). In an effort to standardize nomenclature, it was previously recommended that all pre- and postintervention studies be uniformly referred to as quasi-experimental studies.Reference Harris, Lautenbach and Perencevich 6 Although we observed considerable improvement in the 10 years between the previous and current reviews, of the 68 studies that correctly identified their study design, only half explicitly referred to it as “quasi-experimental.” The use of inaccurate or nonstandard nomenclature further increases the difficulty for readers to understand these studies.
Finally, this study, in contrast to the prior study, evaluated the statistical methods used by the included studies. We found that a large proportion of studies (n=75, 43%) used 2-group tests to compare pre- and postintervention outcomes. These 2-group statistical tests are limited because they do not adjust for confounding variables, because they cannot detect changes in temporal trends, and because they do not account for the correlated nature of observations in studies of infectious diseases. Additionally, time-series analysis is the preferred analytic method for designs with ≥50 observations given its ability to control for confounders and time trends, to estimate changes in time trends (segmented model), and to account for autocorrelation, but it was rarely used.Reference Shardell, Harris, El-Kamary, Furuno, Miller and Perencevich 2
While causal inferences from quasi-experimental studies are challenged by the nonrandomized nature of the design, for many questions and areas of clinical importance, a randomized trial is neither feasible nor ethical. For these questions, well-designed and well-reported quasi-experimental studies can play an important role in informing policy and practice. The presence of confounding factors, maturation effects, and the possibility of regression to the mean are among the major threats for establishing causal associations from quasi-experimental studies of infectious diseases.Reference Harris, Bradham, Baumgarten, Zuckerman, Fink and Perencevich 3 Additionally, the resulting data structure and correlated nature of observations in quasi-experiments of infectious diseases pose methodological challenges for statistical analysis.Reference Shardell, Harris, El-Kamary, Furuno, Miller and Perencevich 2 Thus, the design of a quasi-experimental study is key to the strength of evidence derived from its data and the ability of certain conclusions to be drawn from the data.Reference Shadish, Cook and Campbel 1 , Reference Harris, Bradham, Baumgarten, Zuckerman, Fink and Perencevich 3 , Reference Harris, Lautenbach and Perencevich 6 Generally, studies that utilize control groups and multiple preintervention observations are preferred to those that do not; importantly, however, the described hierarchy may not always be applicable because it may be logistically infeasible to use a higher-quality design in certain instances.Reference Harris, Lautenbach and Perencevich 6
Based on our review, we urge researchers to design their quasi-experimental studies with the hierarchy in mind and to choose study designs that are of the highest possible methodological quality. We offer 3 main recommendations for designing a quasi-experimental study: (1) use control groups when possible; (2) collect multiple preintervention data points; and (3) use appropriate statistical methods. Using control groups and collecting multiple preintervention data points reduces the inherent threats to establishing causal associations in quasi-experiments. Using strong design methods enables investigators to use statistical techniques that are more appropriate for quasi-experimental studies in infectious diseases. Specifically, investigators should use statistical methods that are capable of controlling for confounding factors and accounting for temporal trends and that are not restricted by assumptions of independence (eg, segmented time-series analysis). We further recommend that authors justify why randomization was not used when not implied by the nature of the intervention and that investigators continue to advocate for more standardized nomenclature to describe quasi-experimental studies. While the aforementioned acceptable nomenclature (eg, pre- and posttest intervention study, before–after intervention study, and interrupted time series) is accurate, referring to these studies as “quasi-experimental” could make it easier for readers to understand study design and limitations. We also urge the editors and editorial staff of these journals to consider applying some of these points in their review and standardization of nomenclature for accepted quasi-experimental studies, in addition to the guidelines outlined in the ORION and other published statements for reporting of nonrandomized studies.Reference Schweizer, Braun and Milstone 5 , Reference Des Jarlais, Lyles, Crepaz and Group 11 With these recommendations, we hope to continue improving the quality of future quasi-experimental studies so that more effective interventions can be assessed and implemented in the study of infection control and antibiotic resistance.
PRISMA flow diagram.Reference Moher, Liberati, Tetzlaff and Altman 12
ACKNOWLEDGMENTS
Financial support: This research was funded by the National Institutes of Health (grant no. 5K24AI079040-05 to Dr Harris) and by the CDC Prevention Epicenters Program (grant no. 1U54CK000450-01).
Potential conflicts of interest: All authors report no conflicts of interest relevant to this article.
SUPPLEMENTARY MATERIAL
To view supplementary material for this article, please visit https://doi.org/10.1017/ice.2017.296
