The reproducibility of scientific findings, whereby a study is replicated by independent investigators in order to assess the robustness of the research and of its findings, is fundamental to the scientific process (Dunlap Reference Dunlap and Murchison1926; Popper Reference Popper1959). Overall, we strongly agree with the authors of the target article that replication should be made mainstream. Although replication is typically discussed in terms of reproducing previously published work, we further advocate for making mainstream the independent replication of findings prior to publication (see also Schooler Reference Schooler2014). Pre-publication independent replication (PPIR) is a collaborative, crowdsourced approach to science in which original study authors nominate their own findings to be replicated in independent laboratories around the world. This approach complements existing replication initiatives that focus on published findings and has different strengths and weaknesses. Importantly, PPIR further addresses three of the key concerns from replication skeptics counterargued so effectively in the target article.

In our first pre-publication independent replication initiative (Schweinsberg et al. Reference Schweinsberg, Madan, Vianello, Sommer, Jordan, Tierney, Awtrey, Zhu, Diermeier, Heinze, Srinivasan, Tannenbaum, Bivolaru, Dana, Davis-Stober, du Plessis, Gronau, Hafenbrack, Liao, Ly, Marsman, Murase, Qureshi, Schaerer, Thornley, Tworek, Wagenmakers, Wong, Anderson, Bauman, Bedwell, Brescoll, Canavan, Chandler, Cheries, Cheryan, Cheung, Cimpian, Clark, Cordon, Cushman, Ditto, Donahue, Frick, Gamez-Djokic, Hofstein Grady, Graham, Gu, Hahn, Hanson, Hartwich, Hein, Inbar, Jiang, Kellogg, Kennedy, Legate, Luoma, Maibuecher, Meindl, Miles, Mislin, Molden, Motyl, Newman, Ngo, Packham, Ramsay, Ray, Sackett, Sellier, Sokolova, Sowden, Storage, Sun, Van Bavel, Washburn, Wei, Wetter, Wilson, Darrous and Uhlmann2016; Tierney et al. Reference Tierney, Schweinsberg, Jordan, Kennedy, Qureshi, Sommer, Thornley, Madan, Vianello, Awtrey, Zhu, Diermeier, Heinze, Srinivasan, Tannenbaum, Bivolaru, Dana, Davis-Stober, du Plessis, Gronau, Hafenbrack, Liao, Ly, Murase, Schaerer, Tworek, Wagenmakers, Wong, Anderson, Bauman, Bedwell, Brescoll, Canavan, Chandler, Cheries, Cheryan, Cheung, Cimpian, Clark, Cordon, Cushman, Ditto, Amell, Frick, Gamez-Djokic, Hofstein Grady, Graham, Gu, Hahn, Hanson, Hartwich, Hein, Inbar, Jiang, Kellogg, Legate, Luoma, Maibeucher, Meindl, Miles, Mislin, Molden, Motyl, Newman, Ngo, Packhan, Ramsay, Ray, Sackett, Sellier, Sokolova, Sowden, Storage, Sun, Van Bavel, Washburn, Wei, Wetter, Wilson, Darroux and Uhlmann2016), 10 unpublished moral judgment effects from the last author's research pipeline were replicated by 25 independent research groups who collected data from more than 11,000 participants. The findings were mixed – while some studies replicated successfully, others did not replicate according to the a priori established criteria. Overall, 6 findings successfully replicated; one study replicated but with a much smaller effect size than the original (a decline effect [Schooler Reference Schooler2011]), two findings were not supported, and one study was culturally moderated (replicating consistently in the original country but not in five other countries). The culturally moderated effect provides evidence that contextual factors can play an important and unexpected role in replications. In total, 40% of the original findings failed at least one major criterion for reproducibility.

We have expanded the scope of our crowdsourcing approach in a second PPIR initiative, the Pipeline Project 2. This initiative opens pre-publication independent replication to the world, providing original authors the opportunity to nominate their unpublished work for replication in partner laboratories as well as graduate methods classes. We currently have 13 original findings being replicated at more than 50 sites around the world (Schweinsberg et al., in preparation). Original authors opt into the PPIR process and help select replicators they regard as suitable and as having access to relevant subject populations, leading to collaborative rather than adversarial interactions. Notably, original authors are asked to specify beforehand in what cultures and research sites they do and do not expect their effect to emerge. We are further conducting a prediction market (Dreber et al. Reference Dreber, Pfeiffer, Almenberg, Isaksson, Wilson, Chen, Nosek and Johannesson2015) to see if members of the scientific community at large can anticipate contextual variability in effects. These aspects of the PPIR process further address a key challenge raised by replication skeptics, by systematically taking context into account.

Concerns have also been raised about reputational damage to those involved in replications, both to original authors whose published findings are not reproduced by other research groups, and replicators whose results question established findings (Bohannon Reference Bohannon2014; Kahneman Reference Kahneman2014; Schnall Reference Schnall2014a; Reference Schnall2014b; Reference Schnall2014c). By replicating findings in independent laboratories before (rather than after) the findings are published, PPIRs minimizes reputational costs to both original authors and replicators because (1) no one's reputation depends on the outcome, and (2) original authors voluntarily opt into the PPIR process and help select their replicators.

Another common argument is that failed replications are uninterpretable and low in theoretical value (Schnall Reference Schnall2014a; Reference Schnall2014b; Reference Schnall2014c). Although in our view replications are always informative and valuable (Dreber et al. Reference Dreber, Pfeiffer, Almenberg, Isaksson, Wilson, Chen, Nosek and Johannesson2015), it is at the same time true that there are other plausible explanations for null findings other than the original effect being false (Open Science Collaboration 2015). We suggest that the theoretical value of PPIR in terms of identifying false positives is even higher than for traditional replications, because most alternative explanations for null effects are ruled out. In particular, defenders of the original finding have little basis to attribute an unsuccessful replication to a lack of replicator expertise or use of irrelevant subject populations, because the original authors helped select what they regarded as qualified replicators and specified a priori which participant populations they expected to exhibit the effect. However, informational value is correspondingly lower for successful PPIRs, relative to traditional replications, because the original authors participate in selecting their own replicators, who may be biased in favor of the hypothesis. Indeed, research demonstrates that the theories investigators endorse strongly predict the effect sizes they obtain (Berman & Reich Reference Berman and Reich2010).

The biggest challenge to making pre-publication independent replication mainstream is the lack of professional incentives, especially for replicators. One potential solution is to build PPIRs into the education of graduate students (Everett & Earp Reference Everett and Earp2015) as part of crowdsourced projects on which they and the instructors of their methods courses are co-authors. These student PPIRs can examine findings that the original authors identify as straightforward for a junior researcher to conduct. To facilitate the integration of pre-publication independent replication into graduate methods courses, as part of the Pipeline Project 2 we have developed an open source curriculum on Crowdsourcing Science including instructions for student PPIR projects (https://osf.io/hj9zr/). Researchers of any level of experience who wish to initiate projects can use the Study Swap website (https://osf.io/view/StudySwap/), a new forum where interested parties can engage with the PPIR process, both as original authors looking for labs to replicate their findings or as independent investigators looking to replicate findings. Networks of partner laboratories such as the Psychological Science Accelerator (Chartier Reference Chartier2017) might also be leveraged to conduct replications of unpublished, rather than published findings.

In sum, conducting independent replications earlier in the research process – before findings are even submitted for publication – can further address what the target article identifies as three of the key concerns raised by skeptics of replication. The pre-publication independent replication approach minimizes reputational costs to original authors and replicators, systematically takes into account context, and maximizes the informational value of failed replications.