Increasing replication quality
Zwaan et al. discussed criticisms of the limited theoretical value of replication and the role of contextual variable in replications. This criticism stems from a well-known discussion in psychology whether quality of research should be results- or theory-centered (e.g., Greenberg et al. Reference Greenberg, Solomon, Pyszczynski and Steinberg1988; Greenwald et al. Reference Greenwald, Pratkanis, Leippe and Baumgardner1986). One strategy to resolve the conflict between theoretical value and obtained results is to follow the guidelines outlined in the Replication Recipe (RR; Brandt, IJzerman et al. Reference Brandt, IJzerman, Dijksterhuis, Farach, Geller, Giner-Sorolla, Grange, Perugini, Spies and van ‘t Veer2014). The RR suggests that replications include 36 “ingredients” for high-quality replications (including, but not limited to, choosing a finding with high replication value, sufficient power, exclusion criteria that are defined a priori, identified differences between original and replication studies, and pre-registration). Following the RR helps replication researchers identify the central parameters of a study and thus the key components of the replication, so that the replication is as convincing as possible. This not only facilitates communication between original and replication researchers, but also between readers of both the replication and the original research. The RR, for example, suggests that replication researchers list contextual features that likely differ between the original and replication research (e.g., Different cultural setting? Different population?). This helps communicate to the original authors and readers what the differences in the studies are and the degree the study is a direct or more of a conceptual replication. There may not always be agreement on these designations, but at least the information is clearly available for the reader to make up their own mind. The results from the RR can also be used by future scholars to identify (and then test with pre-registered studies) potential moderators of the effect across both original studies and replication studies, increasing the theoretical value of replications.
Interestingly, Zwaan et al. misinterpreted the RR as something that should be included in original articles. Our original paper was focused on replications and so we did not discuss original articles, but this misinterpretation highlights the important point that many, if not all, of the qualities of a convincing and high-quality replication are exactly the same as the qualities of a convincing and high-quality original study. Therefore, authors can specify the conditions they consider necessary and relevant for their finding and any limits on generalizability (Simons et al. Reference Simons, Shoda and Lindsay2017), resulting in increasingly specified psychological theories.
Making replications habitual
Another key facet to making replication mainstream is making replications habitual. One way of doing so is by developing an appreciation for replication early in the academic career. We created the Collaborative Education and Research Project (CREP; Grahe et al. Reference Grahe, Brandt, IJzerman, Cohoon, Peng, Detweiler-Bedell and Weisberg2015) with the goal of training undergraduate researchers to conduct high quality replication research through standardized procedures as part of research methods courses. The CREP board selects - through a rigid selection process - impactful studies that are feasible to conduct by bachelor students. Prior to data collection, the CREP board communicates with original authors that we selected their study and invite them to provide any original materials and to comment about any conditions that would facilitate successful replication. Students – often in groups and always under the supervision of a faculty supervisor– create a project page on the Open Science Framework, submit their proposed protocol (including video, methods, and evidence of international review board approval) for review by a CREP review team (three advanced researchers and a student administrative advisor). This review process is at least as stringent (and perhaps sometimes more so) than the journal review process. After receiving approval, they complete a general registration of their study, and then collect data. Upon project completion, they go through a second review where the CREP review team reviews their presentation of their data and findings.
CREP projects directly contribute to the research literature by reporting high-quality replications (with one manuscript published [Leighton et al. Reference Leighton, Legate, LePine, Anderson and Grahe2018] and two more in progress [Ghelfi et al. Reference Ghelfi, Christopherson, Fischer, Legate, Lenne, Urry, Wagemans, Wiggins, Barrett, Glass, Guberman, Hunt, Issa, Paulk, Peck, Perkinson, Sheelar, Theado and Turpinin preparation; Wagge et al. Reference Wagge, Johnson, Meltzer, Baciu, Banas, Nadler, IJzerman and Grahein preparation]). Additionally, and more importantly, the CREP educates students about modern psychological research methods, training them to be the researchers with the relevant expertise we need. These skills transfer to original research. Students must understand the hypothesis and theory from the original study as they identify which materials are necessary in an original study. They learn to properly document a study (including, but not limited to, obtaining informed consent, collecting and analyzing data, and reporting findings requires the same resources as original research). Further, by interacting with the CREP team, these students experience a review process with faculty at different institutions than their own. As a bonus, instructors are not challenged with reading and supervising poorly conceptualized or poorly planned research that is developed quickly, without adequate preparation that can understandably be typical of students' first research project.
Zwaan et al. integrated perspectives on replication to argue why replications should be made mainstream. The initiatives we describe have and we hope will continue to help make replications mainstream. Over the course of 5 years, 233 RRs have been registered on the Open Science Framework and 356 students at 49 institutions started 106 CREP replications. The RR and the CREP have already substantively contributed to increased replication quality and to making replications habitual. We hope this exchange further stimulates making replications mainstream.
You have
Access
Zwaan et al. integrated previous articles to promote making replications mainstream. We wholeheartedly agree. We extend their discussion by highlighting several existing initiatives that aim to make replications mainstream and that have already helped resolve several of the concerns discussed by Zwaan et al. Specifically, we discuss how to Increase Replication Quality and how to Make Replications Habitual. These facets should facilitate addressing the concerns of not having a standard method and that expertise of the original and replication authors may not be sufficiently relevant.
Increasing replication quality
Zwaan et al. discussed criticisms of the limited theoretical value of replication and the role of contextual variable in replications. This criticism stems from a well-known discussion in psychology whether quality of research should be results- or theory-centered (e.g., Greenberg et al. Reference Greenberg, Solomon, Pyszczynski and Steinberg1988; Greenwald et al. Reference Greenwald, Pratkanis, Leippe and Baumgardner1986). One strategy to resolve the conflict between theoretical value and obtained results is to follow the guidelines outlined in the Replication Recipe (RR; Brandt, IJzerman et al. Reference Brandt, IJzerman, Dijksterhuis, Farach, Geller, Giner-Sorolla, Grange, Perugini, Spies and van ‘t Veer2014). The RR suggests that replications include 36 “ingredients” for high-quality replications (including, but not limited to, choosing a finding with high replication value, sufficient power, exclusion criteria that are defined a priori, identified differences between original and replication studies, and pre-registration). Following the RR helps replication researchers identify the central parameters of a study and thus the key components of the replication, so that the replication is as convincing as possible. This not only facilitates communication between original and replication researchers, but also between readers of both the replication and the original research. The RR, for example, suggests that replication researchers list contextual features that likely differ between the original and replication research (e.g., Different cultural setting? Different population?). This helps communicate to the original authors and readers what the differences in the studies are and the degree the study is a direct or more of a conceptual replication. There may not always be agreement on these designations, but at least the information is clearly available for the reader to make up their own mind. The results from the RR can also be used by future scholars to identify (and then test with pre-registered studies) potential moderators of the effect across both original studies and replication studies, increasing the theoretical value of replications.
Interestingly, Zwaan et al. misinterpreted the RR as something that should be included in original articles. Our original paper was focused on replications and so we did not discuss original articles, but this misinterpretation highlights the important point that many, if not all, of the qualities of a convincing and high-quality replication are exactly the same as the qualities of a convincing and high-quality original study. Therefore, authors can specify the conditions they consider necessary and relevant for their finding and any limits on generalizability (Simons et al. Reference Simons, Shoda and Lindsay2017), resulting in increasingly specified psychological theories.
Making replications habitual
Another key facet to making replication mainstream is making replications habitual. One way of doing so is by developing an appreciation for replication early in the academic career. We created the Collaborative Education and Research Project (CREP; Grahe et al. Reference Grahe, Brandt, IJzerman, Cohoon, Peng, Detweiler-Bedell and Weisberg2015) with the goal of training undergraduate researchers to conduct high quality replication research through standardized procedures as part of research methods courses. The CREP board selects - through a rigid selection process - impactful studies that are feasible to conduct by bachelor students. Prior to data collection, the CREP board communicates with original authors that we selected their study and invite them to provide any original materials and to comment about any conditions that would facilitate successful replication. Students – often in groups and always under the supervision of a faculty supervisor– create a project page on the Open Science Framework, submit their proposed protocol (including video, methods, and evidence of international review board approval) for review by a CREP review team (three advanced researchers and a student administrative advisor). This review process is at least as stringent (and perhaps sometimes more so) than the journal review process. After receiving approval, they complete a general registration of their study, and then collect data. Upon project completion, they go through a second review where the CREP review team reviews their presentation of their data and findings.
CREP projects directly contribute to the research literature by reporting high-quality replications (with one manuscript published [Leighton et al. Reference Leighton, Legate, LePine, Anderson and Grahe2018] and two more in progress [Ghelfi et al. Reference Ghelfi, Christopherson, Fischer, Legate, Lenne, Urry, Wagemans, Wiggins, Barrett, Glass, Guberman, Hunt, Issa, Paulk, Peck, Perkinson, Sheelar, Theado and Turpinin preparation; Wagge et al. Reference Wagge, Johnson, Meltzer, Baciu, Banas, Nadler, IJzerman and Grahein preparation]). Additionally, and more importantly, the CREP educates students about modern psychological research methods, training them to be the researchers with the relevant expertise we need. These skills transfer to original research. Students must understand the hypothesis and theory from the original study as they identify which materials are necessary in an original study. They learn to properly document a study (including, but not limited to, obtaining informed consent, collecting and analyzing data, and reporting findings requires the same resources as original research). Further, by interacting with the CREP team, these students experience a review process with faculty at different institutions than their own. As a bonus, instructors are not challenged with reading and supervising poorly conceptualized or poorly planned research that is developed quickly, without adequate preparation that can understandably be typical of students' first research project.
Zwaan et al. integrated perspectives on replication to argue why replications should be made mainstream. The initiatives we describe have and we hope will continue to help make replications mainstream. Over the course of 5 years, 233 RRs have been registered on the Open Science Framework and 356 students at 49 institutions started 106 CREP replications. The RR and the CREP have already substantively contributed to increased replication quality and to making replications habitual. We hope this exchange further stimulates making replications mainstream.