A distinguishing feature of science is that its claims are expected to be verifiable. This has been long appreciated by scientists and philosophers. In 1660 the Royal Society, the oldest continuously operating scientific society in the world, adopted as its motto the Latin phrase nullius in verba, meaning “take nobody's word for it.” Merton (Reference Merton and Merton1942) wrote that scientific norms are upheld through scrutiny and verification of our work by other scientists. For Popper (Reference Popper1959), scientific statements must be intersubjectively testable, capable of being evaluated similarly by any person thinking rationally about them. Lupia and Elman (Reference Lupia and Elman2014) argued that scientific claims get their credibility from being publicly available for inspection and critique.
Verifiability is a broad idea that can be applied to every step in the derivation of a scientific claim, and it is the basis for many current norms, practices, and expectations in scientific discourse. If an author of a paper asserts that a prediction follows from some theory, scientific norms hold that the arguments should be presented in enough detail that readers can evaluate whether the theory is coherent and the prediction does indeed follow. If an author asserts that a measurement procedure produces valid scores, scientists expect the author to support that claim with data or through citations to previous validity studies. If an author employs a novel statistical method, scientists expect to be able to verify how the method works by inspecting proofs or the results of simulations.
Verifiability is important for empirical results, as well. Consider a famous example from philosophy, the black swan. Once upon a time, Europeans believed that all swans were white. But in the seventeenth century, Dutch explorers reported that they had observed black swans in Australia, forcing Europeans to update their beliefs about the possible colors of swans. In the stylized telling of this story, this was a purely logical operation: The empirical statement “black swans exist” falsifies the theoretical statement “all swans are white” via modus tollens. But the logic applies only after we accept the premises. Philosophers have long recognized that scientists are not required to accept a premise like “black swans exist” just because someone says they do, or else theories would have to accommodate all kinds of outlandish claims (Mill Reference Mill1882/2014). Instead, scientists establish methodological rules to decide whether to refuse or admit new findings into the corpus of scientific knowledge, with verifiability being one key principle guiding those rules (Meehl Reference Meehl1990b; Popper Reference Popper1959). There are many reports from people who say they have seen black swans, but there are also many reports from people who say they have seen Bigfoot. Part of why scientific taxonomies include black swans but not Bigfoots is because of differences in the verifiability of the reports.
Replication research makes scientific knowledge more verifiable in several ways. The most straightforward way, and the one that receives the greatest emphasis by Zwaan et al. in the target article, is that direct replication studies are a way to evaluate empirical claims and decide which ones to accept as dispositive for theories. If a result is a statistical fluke, or if the necessary conditions for getting a result include critical elements that were not specified in the original report, replication research will produce discrepant results and scientists may be justifiably cautious in updating their theories. Conversely, because new empirical results are almost never definitive (Srivastava Reference Srivastava2011), a successful direct replication can increase scientists' confidence in the definitiveness of a claim.
A second way replication research can advance verifiability is by promoting open dissemination of data, analysis code, materials, and details of experimental procedures. All of these things help researchers carry out high-quality direct replications, so norms and policies that are designed to enable replications should promote their dissemination. But all of them are useful for other kinds of verification beside replication. Open data and code allow other scientists to evaluate how well statistical analyses support conclusions. Open materials and procedures allow other scientists to evaluate whether research protocols work as intended. Thus, even if no replication study is ever conducted, when original authors provide the information that makes replication possible, they make other aspects of verification possible too.
A third way that replication research promotes verifiability is through the accumulation of meta-scientific knowledge. The claim “this experimental result is verifiable” can be treated like a hypothesis and tested empirically in a direct replication. But in practice, scientists do not have the time or resources to replicate every single experiment. Instead they must rely on meta-scientific ideas about which results are more or less credible. Should scientists put more trust in findings with low p values? Studies with more thorough disclosures? Papers by high-status authors? When scientists read one another's work, they make informal judgments about how much they believe the results. Replication research, especially large-scale systematic replication research like the Many Labs projects (e.g., Ebersole et al. Reference Ebersole, Atherton, Belanger, Skulborstad, Allen, Banks, Baranski, Bernstein, Bofiglio, Boucher, Brown, Budima, Cairo, Capaldi, Chartier, Chung, Cicero, Coleman, Conway, Davis, Devos, Fletcher, German, Grahe, Hermann, Hicks, Honeycutt, Humphrey, Janus, Johnson, Joy-Gaba, Juzeler, Keres, Kinney, Kirschenbaum, Klein, Lucas, Lustgraff, Martin, Menon, Metzger, Moloney, Morse, Prislin, Razza, Re, Rule, Sacco, Sauerberger, Shrider, Shultz, Siesman, Sobocko, Sternglanz, Summerville, Tskhay, van Allen, Vaughn, Walker, Weinberg, Wilson, Wirth, Wortman and Nosek2016a; Klein et al. Reference Klein, Ratliff, Vianello, Adams, Bahník, Bernstein, Bocian, Brandt, Brooks, Brumbaugh, Cemalcilar, Chandler, Cheong, Davis, Devos, Eisner, Frankowska, Furrow, Galliani, Hasselman, Hicks, Hovermale, Hunt, Hunstinger, IJerzman, John, Joy-Gaba, Kappes, Krueger, Kurtz, Levitan, Mallett, Morris, Nelson, Nier, Packard, Pilati, Rutchick, Schmidt, Skorinko, Smith, Steiner, Storbeck, Van Swol, Thompson, van't Veer, Vaughn, Vranka, Wichman, Woodzicka and Nosek2014a) can tell us what makes research more replicable and help scientists make those judgments in a more principled way.
A fourth way that replication research promotes verifiability is by reinforcing it as a norm of scientific work. As Merton (Reference Merton and Merton1942) noted, it is unrealistic to expect individual scientists to have only pure scientific motives. Many scientists want to discover true things about nature; but they also want to make money, gain status, look and feel smart, confirm their preconceptions, and all sorts of other entirely human things. Psychologists should especially appreciate that individual scientists cannot be expected to perfectly manage their own conflicting motivations and biases. So scientific disinterestedness must be upheld through the social process of scientists scrutinizing and verifying each other's claims. The more replication research is supported formally through professional incentives, and informally through scientists speaking favorably about replication as valued work and treating it as an expected part of their jobs, the more it will promote a broader culture that values verification of scientific knowledge above the advancement of scientists' individual interests.
A distinguishing feature of science is that its claims are expected to be verifiable. This has been long appreciated by scientists and philosophers. In 1660 the Royal Society, the oldest continuously operating scientific society in the world, adopted as its motto the Latin phrase nullius in verba, meaning “take nobody's word for it.” Merton (Reference Merton and Merton1942) wrote that scientific norms are upheld through scrutiny and verification of our work by other scientists. For Popper (Reference Popper1959), scientific statements must be intersubjectively testable, capable of being evaluated similarly by any person thinking rationally about them. Lupia and Elman (Reference Lupia and Elman2014) argued that scientific claims get their credibility from being publicly available for inspection and critique.
Verifiability is a broad idea that can be applied to every step in the derivation of a scientific claim, and it is the basis for many current norms, practices, and expectations in scientific discourse. If an author of a paper asserts that a prediction follows from some theory, scientific norms hold that the arguments should be presented in enough detail that readers can evaluate whether the theory is coherent and the prediction does indeed follow. If an author asserts that a measurement procedure produces valid scores, scientists expect the author to support that claim with data or through citations to previous validity studies. If an author employs a novel statistical method, scientists expect to be able to verify how the method works by inspecting proofs or the results of simulations.
Verifiability is important for empirical results, as well. Consider a famous example from philosophy, the black swan. Once upon a time, Europeans believed that all swans were white. But in the seventeenth century, Dutch explorers reported that they had observed black swans in Australia, forcing Europeans to update their beliefs about the possible colors of swans. In the stylized telling of this story, this was a purely logical operation: The empirical statement “black swans exist” falsifies the theoretical statement “all swans are white” via modus tollens. But the logic applies only after we accept the premises. Philosophers have long recognized that scientists are not required to accept a premise like “black swans exist” just because someone says they do, or else theories would have to accommodate all kinds of outlandish claims (Mill Reference Mill1882/2014). Instead, scientists establish methodological rules to decide whether to refuse or admit new findings into the corpus of scientific knowledge, with verifiability being one key principle guiding those rules (Meehl Reference Meehl1990b; Popper Reference Popper1959). There are many reports from people who say they have seen black swans, but there are also many reports from people who say they have seen Bigfoot. Part of why scientific taxonomies include black swans but not Bigfoots is because of differences in the verifiability of the reports.
Replication research makes scientific knowledge more verifiable in several ways. The most straightforward way, and the one that receives the greatest emphasis by Zwaan et al. in the target article, is that direct replication studies are a way to evaluate empirical claims and decide which ones to accept as dispositive for theories. If a result is a statistical fluke, or if the necessary conditions for getting a result include critical elements that were not specified in the original report, replication research will produce discrepant results and scientists may be justifiably cautious in updating their theories. Conversely, because new empirical results are almost never definitive (Srivastava Reference Srivastava2011), a successful direct replication can increase scientists' confidence in the definitiveness of a claim.
A second way replication research can advance verifiability is by promoting open dissemination of data, analysis code, materials, and details of experimental procedures. All of these things help researchers carry out high-quality direct replications, so norms and policies that are designed to enable replications should promote their dissemination. But all of them are useful for other kinds of verification beside replication. Open data and code allow other scientists to evaluate how well statistical analyses support conclusions. Open materials and procedures allow other scientists to evaluate whether research protocols work as intended. Thus, even if no replication study is ever conducted, when original authors provide the information that makes replication possible, they make other aspects of verification possible too.
A third way that replication research promotes verifiability is through the accumulation of meta-scientific knowledge. The claim “this experimental result is verifiable” can be treated like a hypothesis and tested empirically in a direct replication. But in practice, scientists do not have the time or resources to replicate every single experiment. Instead they must rely on meta-scientific ideas about which results are more or less credible. Should scientists put more trust in findings with low p values? Studies with more thorough disclosures? Papers by high-status authors? When scientists read one another's work, they make informal judgments about how much they believe the results. Replication research, especially large-scale systematic replication research like the Many Labs projects (e.g., Ebersole et al. Reference Ebersole, Atherton, Belanger, Skulborstad, Allen, Banks, Baranski, Bernstein, Bofiglio, Boucher, Brown, Budima, Cairo, Capaldi, Chartier, Chung, Cicero, Coleman, Conway, Davis, Devos, Fletcher, German, Grahe, Hermann, Hicks, Honeycutt, Humphrey, Janus, Johnson, Joy-Gaba, Juzeler, Keres, Kinney, Kirschenbaum, Klein, Lucas, Lustgraff, Martin, Menon, Metzger, Moloney, Morse, Prislin, Razza, Re, Rule, Sacco, Sauerberger, Shrider, Shultz, Siesman, Sobocko, Sternglanz, Summerville, Tskhay, van Allen, Vaughn, Walker, Weinberg, Wilson, Wirth, Wortman and Nosek2016a; Klein et al. Reference Klein, Ratliff, Vianello, Adams, Bahník, Bernstein, Bocian, Brandt, Brooks, Brumbaugh, Cemalcilar, Chandler, Cheong, Davis, Devos, Eisner, Frankowska, Furrow, Galliani, Hasselman, Hicks, Hovermale, Hunt, Hunstinger, IJerzman, John, Joy-Gaba, Kappes, Krueger, Kurtz, Levitan, Mallett, Morris, Nelson, Nier, Packard, Pilati, Rutchick, Schmidt, Skorinko, Smith, Steiner, Storbeck, Van Swol, Thompson, van't Veer, Vaughn, Vranka, Wichman, Woodzicka and Nosek2014a) can tell us what makes research more replicable and help scientists make those judgments in a more principled way.
A fourth way that replication research promotes verifiability is by reinforcing it as a norm of scientific work. As Merton (Reference Merton and Merton1942) noted, it is unrealistic to expect individual scientists to have only pure scientific motives. Many scientists want to discover true things about nature; but they also want to make money, gain status, look and feel smart, confirm their preconceptions, and all sorts of other entirely human things. Psychologists should especially appreciate that individual scientists cannot be expected to perfectly manage their own conflicting motivations and biases. So scientific disinterestedness must be upheld through the social process of scientists scrutinizing and verifying each other's claims. The more replication research is supported formally through professional incentives, and informally through scientists speaking favorably about replication as valued work and treating it as an expected part of their jobs, the more it will promote a broader culture that values verification of scientific knowledge above the advancement of scientists' individual interests.