The plea of making direct replication mainstream by Zwaan et al. seeks to return the scientific method to its roots. Emphasis on direct replication heralded the origins of science. In the Royal Society (established in 1660), early experimenters replicated in front of their colleagues whatever claims they made.
In the modern era, replication fell out of favor. Scientists publish articles that one must trust almost with blind faith. Typically, it is impossible even to perform a re-analysis of the published data (because data, protocols, and scripts are unavailable), let alone replicate in new studies. The implicit assumption is that the large majority of these claimed discoveries are right. If science faces a flood of major discoveries, using resources to replicate or to increase methodological scrutiny is low priority. Similarly, “negative” results seem low priority, if true, important, significant results abound.
The narrative of an oversupply of major, true discoveries and few false findings is untenable (Ioannidis Reference Ioannidis2005). Sometimes this narrative coexists with additional urges to rush (e.g., “patients are dying, license new drugs immediately”), avoid replication, and even be methodologically sloppy (e.g., use routinely collected data instead of randomized trials).
Clearly, genuine discoveries do occur, but they are inhomogeneous across different scientific fields. Science is a difficult endeavor. Numerous biases affect different designs and disciplines (Fanelli et al. Reference Fanelli, Costas and Ioannidis2017). Even with best intentions, getting wrong or exaggerated (Ioannidis Reference Ioannidis2008) findings is easy. Many fields make little progress, but they still keep pouring thousands of peer-reviewed papers. Peer review without checking data, protocols, and methods is superficial at best. What we do get, predictably, is an oversupply of statistically significant results; for example, 96% of the papers using P values in the biomedical literature have statistically significant P values (Chavalarias et al. Reference Chavalarias, Wallach, Li and Ioannidis2016). However, very few of these millions of papers with seemingly significant and novel results translate into something useful in medicine (Bowen & Casadevall Reference Bowen and Casadevall2015), and similar inefficiencies probably occur in many other fields. The overblown rhetoric of “success” (e.g., the repeated unfulfilled claims of eliminating cancer or Alzheimer's disease) makes science seem untrustworthy in public.
If genuine discoveries are not that torrential, then replication has more value that an original discovery claim. If the proportion of false original findings is relatively high and/or if false findings have substantial consequences (e.g., if they lead researchers and translators of research down the wrong path wasting effort and resources, or if they cause harm for patients or other “users”), then original discovery work may even have negative value.
Let R be the prestudy odds for a research finding, BF the Bayes factor conferred by the discovery study, and h the ratio of the weight of negative consequences from a false-positive (FP) discovery claim versus the positive consequences from a true-positive (TP) discovery. The value of the original research is proportional to TP – (h * FP) or equivalently to (TP/FP) – h=(R * BF) – h. Given that R and h are rather field specific and cannot be modified (unless a researcher moves to a different research field or entirely changes investigative strategy), investigators can increase the value of their discoveries mostly by increasing BF, for example, by running larger studies and ensuring greater protection from biases. To avoid negative values, one needs BF>h/R. Often this is difficult. With limited resources, most original discoveries come from small studies, where biases are common. A P value slightly less than 0.05 corresponds typically to a BF less than 3 (Benjamin et al. Reference Benjamin, Berger, Johannesson, Nosek, Wagenmakers, Berk, Bollen, Brembs, Brown, Camerer, Cesarini, Chambers, Clyde, Cook, De Boeck, Dienes, Dreber, Easwaran, Efferson, Fehr, Fidler, Field, Forster, George, Gonzalez, Goodman, Green, Green, Greenwald, Hadfield, Hedges, Held, Ho, Hoijtink, Hruschka, Imai, Imbens, Ioannidis, Jeon, Jones, Kirchler, Laibson, List, Little, Lupia, Machery, Maxwell, McCarthy, Moore, Morgan, Munafò, Nakagawa, Nyhan, Parker, Pericchi, Perugini, Rouder, Rousseau, Savalei, Shönbrodt, Sellke, Sinclair, Tingley, Van Zandt, Vazire, Watts, Winship, Wolpert, Xie, Young, Zinman and Johnson2017), and biases erode this further. Furthermore, most current research areas lack obvious low-hanging fruit (i.e., domains with high values of R).
In these circumstances, replication can easily have more value than original research, provided that it has reasonable ability to help differentiate eventually what is true and what is false, when properly done, or to identify the proper boundaries where some claimed phenomena hold true.
In some fields, even replications cannot have positive value. Then their conduct in a field where original research already had negative value would not make things better. For example, observational nutritional epidemiology of single nutrients affecting health outcomes may belong to this category (Ioannidis Reference Ioannidis2013a). Biases in this field are far stronger than whatever true signals may exist. Replication efforts using the same biased observational methods would just add negative value. Such fields should simply be abandoned, acknowledging that the methods that we possess cannot yield positive value. Efforts should be diverted to other methodological strategies (e.g., randomized trials) and other fields of investigation. However, for most fields, it is reasonable to expect that well-done, carefully executed, pre-registered, transparent direct replications in an environment of high methodological standards (Munafò et al. Reference Munafò, Nosek, Bishop, Button, Chambers, Nosek, Percie du Sert, Simonsohn, Wagenmakers, Ware and Ioannidis2017) would have positive value and would help correct the mess of original discovery research. Some fields of modern science, such as population genetics, have learned that replication matters more than discovery (Ioannidis Reference Ioannidis2013b). We hope this will become more widely recognized.
Replication may have more value than original research even in situations where indeed many discoveries are made and a large proportion reflect true signals. It could still be the case that these true discoveries are then difficult to prioritize for further steps (e.g., translation, application, implementation in the real world to reap benefits) unless the magnitudes of their effects are known with substantial accuracy and their context-sensitivity is well understood. For example, it could be that 200 technologies or tentative interventions are discovered, but implementing all of them is impossible. Understanding which ones are best would require narrowing the uncertainty around the relative benefits and harms of each proposed discovery. This would require large-scale replication evidence. It would also require testing in diverse settings to understand better the architecture of the heterogeneity of the observed effects. It is unlikely that the original study alone will offer such insights, regardless of how well it is designed and executed (IntHout et al. Reference IntHout, Ioannidis and Borm2016). Again, replication would matter more than the original discovery study. Eventually, the cumulative evidence, comprising both the original and the replication studies, may be more informative than either component.
The plea of making direct replication mainstream by Zwaan et al. seeks to return the scientific method to its roots. Emphasis on direct replication heralded the origins of science. In the Royal Society (established in 1660), early experimenters replicated in front of their colleagues whatever claims they made.
In the modern era, replication fell out of favor. Scientists publish articles that one must trust almost with blind faith. Typically, it is impossible even to perform a re-analysis of the published data (because data, protocols, and scripts are unavailable), let alone replicate in new studies. The implicit assumption is that the large majority of these claimed discoveries are right. If science faces a flood of major discoveries, using resources to replicate or to increase methodological scrutiny is low priority. Similarly, “negative” results seem low priority, if true, important, significant results abound.
The narrative of an oversupply of major, true discoveries and few false findings is untenable (Ioannidis Reference Ioannidis2005). Sometimes this narrative coexists with additional urges to rush (e.g., “patients are dying, license new drugs immediately”), avoid replication, and even be methodologically sloppy (e.g., use routinely collected data instead of randomized trials).
Clearly, genuine discoveries do occur, but they are inhomogeneous across different scientific fields. Science is a difficult endeavor. Numerous biases affect different designs and disciplines (Fanelli et al. Reference Fanelli, Costas and Ioannidis2017). Even with best intentions, getting wrong or exaggerated (Ioannidis Reference Ioannidis2008) findings is easy. Many fields make little progress, but they still keep pouring thousands of peer-reviewed papers. Peer review without checking data, protocols, and methods is superficial at best. What we do get, predictably, is an oversupply of statistically significant results; for example, 96% of the papers using P values in the biomedical literature have statistically significant P values (Chavalarias et al. Reference Chavalarias, Wallach, Li and Ioannidis2016). However, very few of these millions of papers with seemingly significant and novel results translate into something useful in medicine (Bowen & Casadevall Reference Bowen and Casadevall2015), and similar inefficiencies probably occur in many other fields. The overblown rhetoric of “success” (e.g., the repeated unfulfilled claims of eliminating cancer or Alzheimer's disease) makes science seem untrustworthy in public.
If genuine discoveries are not that torrential, then replication has more value that an original discovery claim. If the proportion of false original findings is relatively high and/or if false findings have substantial consequences (e.g., if they lead researchers and translators of research down the wrong path wasting effort and resources, or if they cause harm for patients or other “users”), then original discovery work may even have negative value.
Let R be the prestudy odds for a research finding, BF the Bayes factor conferred by the discovery study, and h the ratio of the weight of negative consequences from a false-positive (FP) discovery claim versus the positive consequences from a true-positive (TP) discovery. The value of the original research is proportional to TP – (h * FP) or equivalently to (TP/FP) – h=(R * BF) – h. Given that R and h are rather field specific and cannot be modified (unless a researcher moves to a different research field or entirely changes investigative strategy), investigators can increase the value of their discoveries mostly by increasing BF, for example, by running larger studies and ensuring greater protection from biases. To avoid negative values, one needs BF>h/R. Often this is difficult. With limited resources, most original discoveries come from small studies, where biases are common. A P value slightly less than 0.05 corresponds typically to a BF less than 3 (Benjamin et al. Reference Benjamin, Berger, Johannesson, Nosek, Wagenmakers, Berk, Bollen, Brembs, Brown, Camerer, Cesarini, Chambers, Clyde, Cook, De Boeck, Dienes, Dreber, Easwaran, Efferson, Fehr, Fidler, Field, Forster, George, Gonzalez, Goodman, Green, Green, Greenwald, Hadfield, Hedges, Held, Ho, Hoijtink, Hruschka, Imai, Imbens, Ioannidis, Jeon, Jones, Kirchler, Laibson, List, Little, Lupia, Machery, Maxwell, McCarthy, Moore, Morgan, Munafò, Nakagawa, Nyhan, Parker, Pericchi, Perugini, Rouder, Rousseau, Savalei, Shönbrodt, Sellke, Sinclair, Tingley, Van Zandt, Vazire, Watts, Winship, Wolpert, Xie, Young, Zinman and Johnson2017), and biases erode this further. Furthermore, most current research areas lack obvious low-hanging fruit (i.e., domains with high values of R).
In these circumstances, replication can easily have more value than original research, provided that it has reasonable ability to help differentiate eventually what is true and what is false, when properly done, or to identify the proper boundaries where some claimed phenomena hold true.
In some fields, even replications cannot have positive value. Then their conduct in a field where original research already had negative value would not make things better. For example, observational nutritional epidemiology of single nutrients affecting health outcomes may belong to this category (Ioannidis Reference Ioannidis2013a). Biases in this field are far stronger than whatever true signals may exist. Replication efforts using the same biased observational methods would just add negative value. Such fields should simply be abandoned, acknowledging that the methods that we possess cannot yield positive value. Efforts should be diverted to other methodological strategies (e.g., randomized trials) and other fields of investigation. However, for most fields, it is reasonable to expect that well-done, carefully executed, pre-registered, transparent direct replications in an environment of high methodological standards (Munafò et al. Reference Munafò, Nosek, Bishop, Button, Chambers, Nosek, Percie du Sert, Simonsohn, Wagenmakers, Ware and Ioannidis2017) would have positive value and would help correct the mess of original discovery research. Some fields of modern science, such as population genetics, have learned that replication matters more than discovery (Ioannidis Reference Ioannidis2013b). We hope this will become more widely recognized.
Replication may have more value than original research even in situations where indeed many discoveries are made and a large proportion reflect true signals. It could still be the case that these true discoveries are then difficult to prioritize for further steps (e.g., translation, application, implementation in the real world to reap benefits) unless the magnitudes of their effects are known with substantial accuracy and their context-sensitivity is well understood. For example, it could be that 200 technologies or tentative interventions are discovered, but implementing all of them is impossible. Understanding which ones are best would require narrowing the uncertainty around the relative benefits and harms of each proposed discovery. This would require large-scale replication evidence. It would also require testing in diverse settings to understand better the architecture of the heterogeneity of the observed effects. It is unlikely that the original study alone will offer such insights, regardless of how well it is designed and executed (IntHout et al. Reference IntHout, Ioannidis and Borm2016). Again, replication would matter more than the original discovery study. Eventually, the cumulative evidence, comprising both the original and the replication studies, may be more informative than either component.