As Zwaan et al. have repeatedly emphasized, replication is a cornerstone in the edifice called “science.” It is therefore somewhat surprising that its structure has not yet collapsed under the burden of the virulent “replication crisis.” Instead, lively discussions have sprung up focusing on various characteristics of replication endeavors, for example, about the distinction between “direct” procedural and theoretically driven “conceptual” replications (Stroebe & Strack Reference Stroebe and Strack2014). In this comment, we discuss the benefits of the frequent replication exercises and suggest a more fruitful way to pursue in the future.
There is no doubt that thus far, the number of “replication failures” has attracted great public attention. “Over half of psychology studies fail reproducibility test” was the title of a cover story in Nature (Baker Reference Baker2015), and frequently, the news that “yet another classic study was not replicated” makes the headlines in popular newspapers and magazines. Typically, these replication failures were based on an attempted copy of the original study that did not reach conventional significance levels and/or yielded effect sizes below the original standard. Rarely were there significant reversals of the original outcomes.
What can be learned from such procedural nonreplications? There are two possibilities. Other than suspecting fraud, the original effect may be diagnosed as fragile and not sufficiently “robust” to emerge under changing conditions. Second, the effect is declared as being not “real” or “true,” but merely a “false positive.”
To be sure, it is certainly important to ascertain the robustness of an effect if a procedure is meant to be employed as an intervention in a natural setting. There, it is more important to determine the effectiveness of an intervention than to understand its underlying mechanisms. For this purpose, procedural replications are diagnostic and highly appropriate. For example, to assess the therapeutic value of a cancer drug, it is necessary to ascertain its effectiveness under varying contexts.
Understanding underlying mechanisms, however, requires us to resort to the theoretical level and to assess the validity of the theory. Unlike experimental effects, theoretical statements have a truth value that can be supported or undermined by empirical findings. Theories, however, are formulated on a level that transcends the concrete evidence, and their validity does not rest on the outcome of one specific experimental paradigm. Thus, even if a procedural nonreplication implies a lack of reliability for a concrete effect, consequences about truth and falseness of an otherwise well-supported theory must be weighed in the context of supporting evidence from these other investigations.
In particular, inferences about truth or falseness that are based solely on selected parameters from Null Hypothesis Significance Testing are highly problematic. Recently, this statistically myopic approach of determining truth or falsity has been sharply criticized. For example, Blakeley et al. (Reference Blakeley, McShane, Gal, Gelman, Robert and Tackett2018) have advocated moving “beyond the paradigm of routine ‘discovery,’ and binary statements about there being ‘an effect’ or ‘no effect,’ to one of continuous and inevitably flawed learning that is accepting of uncertainty and variation” (p. 9). In a similar vein, the Deutsche Forschungsgemeinschaft (German National Science Foundation) has issued a statement on the “Replicability of Research Results” (2017), arguing that “ascertaining the replicability or non-replicability of a scientific result is itself a scientific result. As such, it is not final; rather, like all scientific knowledge, it is subject to methodological skepticism and further investigation” (p. 2).
These insights imply the lack of a direct route from statistical parameters to scientific truth (Strack Reference Strack2017). Instead, empirical data must be recognized as arguments that must enter the scientific discourse to persuade its participants. For this purpose, the collection of data must be guided by theoretically grounded hypotheses that generate specific predictions. Such hypothesis-guided approaches are conspicuously missing when it comes to procedural replications. As a consequence, little insight is gained about the underlying causal dynamics.
If science has the goal of finding the causes of things (“rerum causas cognoscere”), it is not enough to merely devalue a finding as “random.” Instead, replicators must make the effort to identify the actual determinants of the original finding if they believe that it was not caused by the factors claimed in the original study. In other words, the evidence of a replication should consist of an interaction where the original finding is replicated under certain conditions, but not under others. To be sure, making a theoretical statement contingent on the fulfillment of other conditions may decrease its falsifiability. Zwaan et al. are right in pointing out that repeatedly adding conditions may cause a research program to become “degenerative.” On the other hand, the price of being closer to the truth may be a decreased theoretical power. At the same time, proposing new theories that go beyond what has been known so far and include previous causalities as special cases is always more desirable.
When it comes to theory testing, it is not justified to yield replications a special status that sets them apart from the theoretical discourse. It is not justified to assign the original researcher the onus of identifying the contextual confound that may have prevented the effect from occurring in the replication. Like any other basic research, replications should be driven by theoretical assumptions generating results that go beyond demonstrating a mere failure to replicate and identifying yet another “false positive.” To advance scientific knowledge, replicators should be able to identify the reasons why an effect did not replicate. Without such evidence, simple nonreplication of a finding that is part of a well-supported theory is uninformative. Thus, to answer the question stated in our title, the theoretical knowledge gained from replication failures has been negligible. Therefore, mechanistically continuing multilab replications of yet another set of empirical studies is a waste of valuable scientific resources that would better be employed in developing and testing better and more powerful theories.
As Zwaan et al. have repeatedly emphasized, replication is a cornerstone in the edifice called “science.” It is therefore somewhat surprising that its structure has not yet collapsed under the burden of the virulent “replication crisis.” Instead, lively discussions have sprung up focusing on various characteristics of replication endeavors, for example, about the distinction between “direct” procedural and theoretically driven “conceptual” replications (Stroebe & Strack Reference Stroebe and Strack2014). In this comment, we discuss the benefits of the frequent replication exercises and suggest a more fruitful way to pursue in the future.
There is no doubt that thus far, the number of “replication failures” has attracted great public attention. “Over half of psychology studies fail reproducibility test” was the title of a cover story in Nature (Baker Reference Baker2015), and frequently, the news that “yet another classic study was not replicated” makes the headlines in popular newspapers and magazines. Typically, these replication failures were based on an attempted copy of the original study that did not reach conventional significance levels and/or yielded effect sizes below the original standard. Rarely were there significant reversals of the original outcomes.
What can be learned from such procedural nonreplications? There are two possibilities. Other than suspecting fraud, the original effect may be diagnosed as fragile and not sufficiently “robust” to emerge under changing conditions. Second, the effect is declared as being not “real” or “true,” but merely a “false positive.”
To be sure, it is certainly important to ascertain the robustness of an effect if a procedure is meant to be employed as an intervention in a natural setting. There, it is more important to determine the effectiveness of an intervention than to understand its underlying mechanisms. For this purpose, procedural replications are diagnostic and highly appropriate. For example, to assess the therapeutic value of a cancer drug, it is necessary to ascertain its effectiveness under varying contexts.
Understanding underlying mechanisms, however, requires us to resort to the theoretical level and to assess the validity of the theory. Unlike experimental effects, theoretical statements have a truth value that can be supported or undermined by empirical findings. Theories, however, are formulated on a level that transcends the concrete evidence, and their validity does not rest on the outcome of one specific experimental paradigm. Thus, even if a procedural nonreplication implies a lack of reliability for a concrete effect, consequences about truth and falseness of an otherwise well-supported theory must be weighed in the context of supporting evidence from these other investigations.
In particular, inferences about truth or falseness that are based solely on selected parameters from Null Hypothesis Significance Testing are highly problematic. Recently, this statistically myopic approach of determining truth or falsity has been sharply criticized. For example, Blakeley et al. (Reference Blakeley, McShane, Gal, Gelman, Robert and Tackett2018) have advocated moving “beyond the paradigm of routine ‘discovery,’ and binary statements about there being ‘an effect’ or ‘no effect,’ to one of continuous and inevitably flawed learning that is accepting of uncertainty and variation” (p. 9). In a similar vein, the Deutsche Forschungsgemeinschaft (German National Science Foundation) has issued a statement on the “Replicability of Research Results” (2017), arguing that “ascertaining the replicability or non-replicability of a scientific result is itself a scientific result. As such, it is not final; rather, like all scientific knowledge, it is subject to methodological skepticism and further investigation” (p. 2).
These insights imply the lack of a direct route from statistical parameters to scientific truth (Strack Reference Strack2017). Instead, empirical data must be recognized as arguments that must enter the scientific discourse to persuade its participants. For this purpose, the collection of data must be guided by theoretically grounded hypotheses that generate specific predictions. Such hypothesis-guided approaches are conspicuously missing when it comes to procedural replications. As a consequence, little insight is gained about the underlying causal dynamics.
If science has the goal of finding the causes of things (“rerum causas cognoscere”), it is not enough to merely devalue a finding as “random.” Instead, replicators must make the effort to identify the actual determinants of the original finding if they believe that it was not caused by the factors claimed in the original study. In other words, the evidence of a replication should consist of an interaction where the original finding is replicated under certain conditions, but not under others. To be sure, making a theoretical statement contingent on the fulfillment of other conditions may decrease its falsifiability. Zwaan et al. are right in pointing out that repeatedly adding conditions may cause a research program to become “degenerative.” On the other hand, the price of being closer to the truth may be a decreased theoretical power. At the same time, proposing new theories that go beyond what has been known so far and include previous causalities as special cases is always more desirable.
When it comes to theory testing, it is not justified to yield replications a special status that sets them apart from the theoretical discourse. It is not justified to assign the original researcher the onus of identifying the contextual confound that may have prevented the effect from occurring in the replication. Like any other basic research, replications should be driven by theoretical assumptions generating results that go beyond demonstrating a mere failure to replicate and identifying yet another “false positive.” To advance scientific knowledge, replicators should be able to identify the reasons why an effect did not replicate. Without such evidence, simple nonreplication of a finding that is part of a well-supported theory is uninformative. Thus, to answer the question stated in our title, the theoretical knowledge gained from replication failures has been negligible. Therefore, mechanistically continuing multilab replications of yet another set of empirical studies is a waste of valuable scientific resources that would better be employed in developing and testing better and more powerful theories.