Zwaan et al. (2018) convincingly argue that replication attempts should become mainstream, but they say little as to how this can best be achieved. The problem is that there are currently few mechanisms in place to encourage replication attempts. For example, a survey conducted in 2015 found that only 3% of psychology journals explicitly state that they will consider publishing replications (Martin & Clarke Reference Martin and Clarke2017). Although there have been some notable attempts to encourage more replications (Klein et al. Reference Klein, Ratliff, Vianello, Adams, Bahník, Bernstein, Bocian, Brandt, Brooks, Brumbaugh, Cemalcilar, Chandler, Cheong, Davis, Devos, Eisner, Frankowska, Furrow, Galliani, Hasselman, Hicks, Hovermale, Hunt, Hunstinger, IJerzman, John, Joy-Gaba, Kappes, Krueger, Kurtz, Levitan, Mallett, Morris, Nelson, Nier, Packard, Pilati, Rutchick, Schmidt, Skorinko, Smith, Steiner, Storbeck, Van Swol, Thompson, van't Veer, Vaughn, Vranka, Wichman, Woodzicka and Nosek2014a; Open Science Collaboration 2015), they have been of limited scope, and replications remain scarce: A survey of the top 100 psychology journals found that only 1% of reported studies involved replication (Makel et al. Reference Makel, Plucker and Hegarty2012). Given the enormous publication pressures on academics, if replications are rarely publishable, then a mainstream culture of replication will not emerge.
Here, we propose a novel solution to this problem: Make it standard practice for journals to pre-commit to publishing adequately powered, technically competent direct replications (at least in online form) for any article they publish and link to it from the original article. This would be comparatively simple to implement and would have a relatively low cost, but would greatly change the incentive structure for researchers. It would also lead to a virtuous cycle in which the more replications are published, the more other people would be encouraged to perform replications of their own. Indeed, performing replications might become an important part of academic training: Running replications would enable early postgraduate students to gain valuable skills in research implementation and analysis while also contributing to the scientific literature.
If our proposal were to be adopted, one expectation might be that authors of the original article would discuss the extent to which they predict that their findings would replicate. For instance, authors might become more explicit in identifying when they believe that their findings are likely to apply only to a particular demographic or to occur only in particular circumstances. These discussions would enhance the interpretability of the original article and encourage authors to think more clearly about these issues during the design and analysis of their studies.
Why should journals adopt our proposal? We suggest that a simple modification to the calculation of impact would encourage journals to publish replications of original articles, regardless of how those replications turn out. Currently, the Thomas Reuters journal's impact factor is determined by the number of citations of that journal within a designated period, divided by the number of citable documents published overall during that period. Importantly, the denominator does not include documents considered to be “Editorial Material” — a term covering a wide range of document types from true editorials to commentaries such as this one (even when the commentaries report original data). It should be comparatively simple to agree that non–peer-reviewed, online-only, direct replication attempts should also not count toward the denominator. If so, then hosting direct replication attempts on the journal's website would never hurt. Indeed, if these replication attempts could still be cited (just like editorials can be cited), they would only increase the journal's impact factor. This creates an incentive for journals to publish replications, which is a necessity for replications to become mainstream.
What about funding agencies? Like journals, grant agencies greatly value novelty, but they even more greatly value reliable science; a novel finding can have a long-term impact only if it is true. It should, therefore, be in a funding body's interest either to offer grants that are focused solely on replication or to mandate that a certain percentage of each grant be devoted to replicating previous research.
In one sense, our suggestion is a minor alteration in how science is traditionally done but, in another sense, it is a paradigm shift in how to evaluate scientific work. Although novelty and originality are clearly vital, replicability is no less important. Our failure to systematically replicate our findings results in biased estimates of effect sizes, hampers future work, and makes it hard to obtain a realistic evaluation of what we know (Anderson et al. Reference Anderson, Kelly and Maxwell2017). Because the best way to obtain accurate estimates of a finding's effect size and robustness is to combine multiple independent replication attempts, we need to actively encourage replications. Within our paradigm, the initial publication of an article is just the starting point in an extended conversation that will conclude with a multitude of replication attempts, an increasingly accurate estimate of the effect size, and a much greater understanding of the circumstances for which the findings hold.
How might we appropriately acknowledge replication attempts for the purposes of career advancement? One obvious possibility would be to adopt a convention on curricula vitae in which replication attempts are classified as distinct from other types of publications — much as books, journal articles, and conference proceedings are classified separately now. It would then be up to the individual's university, grant review panel or promotion committee to decide how much to value replication attempts relative to other forms of publication.
Our proposal represents a “win” for academics, journals, and the progress of science as a whole. The ability to easily publish replications would mean that academics would be incentivised to perform replications. Indeed, doing so may become a routine and accepted part of academic training. Within a culture of replicability, the impact of any single replication failure would diminish, making replications less personally threatening and simply part of the process (much as reviews are part of science now). Journals would increase in prestige and citation rates by publishing replications. Fundamentally, incentivising replication attempts is the only way to achieve a mainstream culture of replicability. It is vital for our future that science is built on truth rather than sand.
Zwaan et al. (2018) convincingly argue that replication attempts should become mainstream, but they say little as to how this can best be achieved. The problem is that there are currently few mechanisms in place to encourage replication attempts. For example, a survey conducted in 2015 found that only 3% of psychology journals explicitly state that they will consider publishing replications (Martin & Clarke Reference Martin and Clarke2017). Although there have been some notable attempts to encourage more replications (Klein et al. Reference Klein, Ratliff, Vianello, Adams, Bahník, Bernstein, Bocian, Brandt, Brooks, Brumbaugh, Cemalcilar, Chandler, Cheong, Davis, Devos, Eisner, Frankowska, Furrow, Galliani, Hasselman, Hicks, Hovermale, Hunt, Hunstinger, IJerzman, John, Joy-Gaba, Kappes, Krueger, Kurtz, Levitan, Mallett, Morris, Nelson, Nier, Packard, Pilati, Rutchick, Schmidt, Skorinko, Smith, Steiner, Storbeck, Van Swol, Thompson, van't Veer, Vaughn, Vranka, Wichman, Woodzicka and Nosek2014a; Open Science Collaboration 2015), they have been of limited scope, and replications remain scarce: A survey of the top 100 psychology journals found that only 1% of reported studies involved replication (Makel et al. Reference Makel, Plucker and Hegarty2012). Given the enormous publication pressures on academics, if replications are rarely publishable, then a mainstream culture of replication will not emerge.
Here, we propose a novel solution to this problem: Make it standard practice for journals to pre-commit to publishing adequately powered, technically competent direct replications (at least in online form) for any article they publish and link to it from the original article. This would be comparatively simple to implement and would have a relatively low cost, but would greatly change the incentive structure for researchers. It would also lead to a virtuous cycle in which the more replications are published, the more other people would be encouraged to perform replications of their own. Indeed, performing replications might become an important part of academic training: Running replications would enable early postgraduate students to gain valuable skills in research implementation and analysis while also contributing to the scientific literature.
If our proposal were to be adopted, one expectation might be that authors of the original article would discuss the extent to which they predict that their findings would replicate. For instance, authors might become more explicit in identifying when they believe that their findings are likely to apply only to a particular demographic or to occur only in particular circumstances. These discussions would enhance the interpretability of the original article and encourage authors to think more clearly about these issues during the design and analysis of their studies.
Why should journals adopt our proposal? We suggest that a simple modification to the calculation of impact would encourage journals to publish replications of original articles, regardless of how those replications turn out. Currently, the Thomas Reuters journal's impact factor is determined by the number of citations of that journal within a designated period, divided by the number of citable documents published overall during that period. Importantly, the denominator does not include documents considered to be “Editorial Material” — a term covering a wide range of document types from true editorials to commentaries such as this one (even when the commentaries report original data). It should be comparatively simple to agree that non–peer-reviewed, online-only, direct replication attempts should also not count toward the denominator. If so, then hosting direct replication attempts on the journal's website would never hurt. Indeed, if these replication attempts could still be cited (just like editorials can be cited), they would only increase the journal's impact factor. This creates an incentive for journals to publish replications, which is a necessity for replications to become mainstream.
What about funding agencies? Like journals, grant agencies greatly value novelty, but they even more greatly value reliable science; a novel finding can have a long-term impact only if it is true. It should, therefore, be in a funding body's interest either to offer grants that are focused solely on replication or to mandate that a certain percentage of each grant be devoted to replicating previous research.
In one sense, our suggestion is a minor alteration in how science is traditionally done but, in another sense, it is a paradigm shift in how to evaluate scientific work. Although novelty and originality are clearly vital, replicability is no less important. Our failure to systematically replicate our findings results in biased estimates of effect sizes, hampers future work, and makes it hard to obtain a realistic evaluation of what we know (Anderson et al. Reference Anderson, Kelly and Maxwell2017). Because the best way to obtain accurate estimates of a finding's effect size and robustness is to combine multiple independent replication attempts, we need to actively encourage replications. Within our paradigm, the initial publication of an article is just the starting point in an extended conversation that will conclude with a multitude of replication attempts, an increasingly accurate estimate of the effect size, and a much greater understanding of the circumstances for which the findings hold.
How might we appropriately acknowledge replication attempts for the purposes of career advancement? One obvious possibility would be to adopt a convention on curricula vitae in which replication attempts are classified as distinct from other types of publications — much as books, journal articles, and conference proceedings are classified separately now. It would then be up to the individual's university, grant review panel or promotion committee to decide how much to value replication attempts relative to other forms of publication.
Our proposal represents a “win” for academics, journals, and the progress of science as a whole. The ability to easily publish replications would mean that academics would be incentivised to perform replications. Indeed, doing so may become a routine and accepted part of academic training. Within a culture of replicability, the impact of any single replication failure would diminish, making replications less personally threatening and simply part of the process (much as reviews are part of science now). Journals would increase in prestige and citation rates by publishing replications. Fundamentally, incentivising replication attempts is the only way to achieve a mainstream culture of replicability. It is vital for our future that science is built on truth rather than sand.