Altough running a replication study is difficult to impossible in some domains, it is quite easy in others. Zwaan et al. (2017) state, in their concern III (sect 5.3), that direct replications are not feasible in some domains, for example, large-scale observational studies, or even not possible, for example, for studies capitalizing on rare events. This argument pertains to domains, but more directly to journal articles, that is, experimental studies. We argue that, beyond the larger obstacles described above, studies that are easier to replicate are indeed more frequently replicated, which introduces selection bias into the replication enterprise.
The selection of a to-be-replicated experiment often depends on how easy it is to do a direct replication. An instructive example is the multilab preregistered replication of the ego-depletion effect (Hagger et al. Reference Hagger, Chatzisarantis, Alberts, Anggono, Batailler, Birt, Brand, Brandt, Brewer, Bruyneel, Calvillo, Campbell, Cannon, Carlucci, Carruth, Cheung, Crowell, De Ridder, Dewitte, Elson, Evans, Fay, Fennis, Finley, Francis, Heise, Hoemann, Inzlicht, Koole, Koppel, Kroese, Lange, Lau, Lynch, Martijn, Merckelbach, Mills, Michirev, Miyake, Mosser, Muise, Muller, Muzi, Nalis, Nurwanti, Otgaar, Philipp, Primoceri, Rentzsch, Ringos, Schlinkert, Schmeichel, Schoch, Schrama, Schütz, Stamos, Tinghög, Ullrich, vanDellen, Wimbarti, Wolff, Yusainy, Zerhouni and Zwienenberg2016). The authors selected, as the target of their replication, a procedure introduced by Sripada et al. (Reference Sripada, Kessler and Jonides2014) and not by the original authors of ego depletion (Baumeister et al. Reference Baumeister, Bratslavsky, Muraven and Tice1998; note that Baumeister recommended the alternative procedure!). The reason for selecting this procedure was described as follows: “tasks used in the original experiments were deemed too elaborate or complex to be appropriate for a multilab replication” (Sripada et al. Reference Sripada, Kessler and Jonides2014, p. 548). Why is selection bias at work here? Ease of application is frequently related to the quality of manipulating the independent variable, such that the strength of the manipulation is often limited in easy-to-administer operationalization. A vicious circle is generated: Ease of application breeds a multitude of primary studies (e.g., using simple procedures like questionnaires or vignettes). Many of these studies lead to significant results and therefore publication, but often they are false positives and effect sizes are overestimated (e.g., because of publication bias). If such studies become predominantly targets for direct replication, these replications have little power and are doomed to fail. We end up with many failed replications that are also published, even using the new gold standards of preregistration and multilab collaborations. This circle artificially increases the number of replication failures. The choice of the to-be-replicated target study thus is crucial.
Another selection criterion could be even more harmful: doubt. Many papers become replication targets, not because they are theoretically interesting or important, but because other researchers doubt their results. If there is something to researchers' intuitions of whether a result is likely to be true or not, less likely results have a lower base rate to be true. Even after a significant result, the posterior probability of the hypotheses tested in studies with a small prior is low. Selecting “doubted studies” as targets for direct replication also is doomed to result in failure under most definitions of successful replication. Again, if people select replication targets because they doubt the original findings, and if their doubt is reasonable, the literature will be filled with many failed replications.
The process for choosing the to-be-replicated target study thus is crucial. Ease of application and doubt may contribute to the selection of target papers, leading to an overestimation of replication failures. The best way to avoid this is random selection of replication targets. We pick the Replication Project: Psychology (Open Science Collaboration 2015) as an example of such a procedure. However, inspection of this selection process reveals a variety of judgments, deviating from a purely random choice. The decision tree in Figure 1 illustrates the selection that cuts down an overall 488 articles in the 2008 issues of three journals (Psychological Science; Journal of Experimental Psychology: Learning, Memory and Cognition; Journal of Personality and Social Psychology) to an ultimate 100 completed replications.
Figure 1.Selection process for to-be-replicated papers in the Replication Project: Psychology (RPP). Texts in small print are citations from the Open Science Collaboration (2015).
We identify the following nonrandom selections in the Replication Project: Psychology: (a) publication (only published papers are included); (b) year (papers published in 2008); (c) journal (Psychological Science, Journal of Experimental Psychology: Learning, Memory and Cognition, Journal of Personality and Social Psychology); (d) type (488 original research papers); (e) eligibility (158/488, i.e., 32.4%); (f) claim (113 of 158 claimed by replicators, i.e., 71.5%); (g) completion (100 of 113 papers completed and data uploaded to the Open Science Framework within given time frame, i.e., 88.5%). Eventually, a fifth (100/488, i.e., 20.5%) of all possible replications were run and ultimately published. Bias caused by the difficulty of doing a replication surely exists for eligibility (step e, see description in Open Science Collaboration 2015, Methods appendix) and is likely for claiming (step f) and completion (step g). Bias caused by doubt influences claiming (step f). In sum, there is plenty of room for bias in the selection of replication targets. In our opinion, this problem has not yet been addressed adequately.
Altough running a replication study is difficult to impossible in some domains, it is quite easy in others. Zwaan et al. (2017) state, in their concern III (sect 5.3), that direct replications are not feasible in some domains, for example, large-scale observational studies, or even not possible, for example, for studies capitalizing on rare events. This argument pertains to domains, but more directly to journal articles, that is, experimental studies. We argue that, beyond the larger obstacles described above, studies that are easier to replicate are indeed more frequently replicated, which introduces selection bias into the replication enterprise.
The selection of a to-be-replicated experiment often depends on how easy it is to do a direct replication. An instructive example is the multilab preregistered replication of the ego-depletion effect (Hagger et al. Reference Hagger, Chatzisarantis, Alberts, Anggono, Batailler, Birt, Brand, Brandt, Brewer, Bruyneel, Calvillo, Campbell, Cannon, Carlucci, Carruth, Cheung, Crowell, De Ridder, Dewitte, Elson, Evans, Fay, Fennis, Finley, Francis, Heise, Hoemann, Inzlicht, Koole, Koppel, Kroese, Lange, Lau, Lynch, Martijn, Merckelbach, Mills, Michirev, Miyake, Mosser, Muise, Muller, Muzi, Nalis, Nurwanti, Otgaar, Philipp, Primoceri, Rentzsch, Ringos, Schlinkert, Schmeichel, Schoch, Schrama, Schütz, Stamos, Tinghög, Ullrich, vanDellen, Wimbarti, Wolff, Yusainy, Zerhouni and Zwienenberg2016). The authors selected, as the target of their replication, a procedure introduced by Sripada et al. (Reference Sripada, Kessler and Jonides2014) and not by the original authors of ego depletion (Baumeister et al. Reference Baumeister, Bratslavsky, Muraven and Tice1998; note that Baumeister recommended the alternative procedure!). The reason for selecting this procedure was described as follows: “tasks used in the original experiments were deemed too elaborate or complex to be appropriate for a multilab replication” (Sripada et al. Reference Sripada, Kessler and Jonides2014, p. 548). Why is selection bias at work here? Ease of application is frequently related to the quality of manipulating the independent variable, such that the strength of the manipulation is often limited in easy-to-administer operationalization. A vicious circle is generated: Ease of application breeds a multitude of primary studies (e.g., using simple procedures like questionnaires or vignettes). Many of these studies lead to significant results and therefore publication, but often they are false positives and effect sizes are overestimated (e.g., because of publication bias). If such studies become predominantly targets for direct replication, these replications have little power and are doomed to fail. We end up with many failed replications that are also published, even using the new gold standards of preregistration and multilab collaborations. This circle artificially increases the number of replication failures. The choice of the to-be-replicated target study thus is crucial.
Another selection criterion could be even more harmful: doubt. Many papers become replication targets, not because they are theoretically interesting or important, but because other researchers doubt their results. If there is something to researchers' intuitions of whether a result is likely to be true or not, less likely results have a lower base rate to be true. Even after a significant result, the posterior probability of the hypotheses tested in studies with a small prior is low. Selecting “doubted studies” as targets for direct replication also is doomed to result in failure under most definitions of successful replication. Again, if people select replication targets because they doubt the original findings, and if their doubt is reasonable, the literature will be filled with many failed replications.
The process for choosing the to-be-replicated target study thus is crucial. Ease of application and doubt may contribute to the selection of target papers, leading to an overestimation of replication failures. The best way to avoid this is random selection of replication targets. We pick the Replication Project: Psychology (Open Science Collaboration 2015) as an example of such a procedure. However, inspection of this selection process reveals a variety of judgments, deviating from a purely random choice. The decision tree in Figure 1 illustrates the selection that cuts down an overall 488 articles in the 2008 issues of three journals (Psychological Science; Journal of Experimental Psychology: Learning, Memory and Cognition; Journal of Personality and Social Psychology) to an ultimate 100 completed replications.
Selection process for to-be-replicated papers in the Replication Project: Psychology (RPP). Texts in small print are citations from the Open Science Collaboration (2015).
We identify the following nonrandom selections in the Replication Project: Psychology: (a) publication (only published papers are included); (b) year (papers published in 2008); (c) journal (Psychological Science, Journal of Experimental Psychology: Learning, Memory and Cognition, Journal of Personality and Social Psychology); (d) type (488 original research papers); (e) eligibility (158/488, i.e., 32.4%); (f) claim (113 of 158 claimed by replicators, i.e., 71.5%); (g) completion (100 of 113 papers completed and data uploaded to the Open Science Framework within given time frame, i.e., 88.5%). Eventually, a fifth (100/488, i.e., 20.5%) of all possible replications were run and ultimately published. Bias caused by the difficulty of doing a replication surely exists for eligibility (step e, see description in Open Science Collaboration 2015, Methods appendix) and is likely for claiming (step f) and completion (step g). Bias caused by doubt influences claiming (step f). In sum, there is plenty of room for bias in the selection of replication targets. In our opinion, this problem has not yet been addressed adequately.