1. Introduction
The reward system of science is the priority rule scheme (Merton Reference Merton1957). In general, the first scientist making a discovery is rewarded with prestige and recognition while second runners get very little or nothing.Footnote 1 Arguably, the priority rule is functionally beneficial for science in various respects. According to economists of science, the priority rule incentivizes the production of scientific findings (Stephan Reference Stephan2012). And foundational work in the social epistemology of science justifies the rationality of the priority rule in epistemic terms. In particular, Strevens (Reference Strevens2003, Reference Strevens2011), following Kitcher (Reference Kitcher1990), argues that the priority rule incentivizes an efficient division of cognitive labor, allocating resources to (roughly) maximize science’s payoff to society. On these grounds, Kitcher-Strevens’s story suggests that a priority-based reward system is normatively adequate. In this article, I dispute this contention. I argue that Kitcher-Strevens’s story depends on strong implicit assumptions about the replicability of findings. Philosophers and scientists alike regard replicability as the gold standard of scientific findings in experimental science (Fisher Reference Fisher1926; Popper Reference Popper1959/2002; Heisenberg Reference Heisenberg1975). I show that the interplay among the priority rule, statistical inference procedures, and publication practices favors novelty and discourages replication, hindering scientific confirmation.Footnote 2 This analysis leads us to qualify the alleged virtues of the reward system of science: while the reward system might be efficient in spreading out resources across novel projects, it is inefficient in allocating resources to produce well-tested theories. My analysis also offers a plausible explanation for the worrisome lack of replication in practice (Smith Reference Smith1970; Campbell and Jackson Reference Campbell and Jackson1979; Evanschitzky et al. Reference Evanschitzky, Baumgarth, Hubbard and Armstrong2007; Begley and Ellis Reference Begley and Ellis2012; Makel, Plucker, and Hegarty Reference Makel, Plucker and Hegarty2012; Makel and Plucker Reference Makel and Plucker2014). As Longino (Reference Longino and Zalta2015) remarks, the gap between the theory and reality of scientific self-correction has yet to be addressed by philosophers. This article also takes some steps to study this gap.
In section 2, I synthesize Kitcher and Strevens’s assessments about the reward system of science. Then, in section 3, I formulate the replicability principle using an example of a recent controversy about purported evidence of extrasensory perception, and then, in section 4, I show how the priority rule creates a tension between novelty and replicability, which threatens the principle.
2. Virtues in the Reward System of Science: Novelty and Progress
Sociologist Robert Merton (Reference Merton1957) was probably the first to document the central role of the priority rule in the reward system of science.Footnote 3 Drawing from his work and also discussions about the priority rule in philosophy (Strevens Reference Strevens2003, Reference Strevens2011) and economics (Stephan Reference Stephan2012), I characterize the priority rule as follows:
(PR1) Only the first scientist (or research team) making a discovery is rewarded. Second runners get very little or nothing.
(PR2) Scientists’ reward is not primarily money but prestige, that is, peer recognition.Footnote 4
(PR3) Prestige comes in different forms: eponymy,Footnote 5 prizes (e.g., Nobel Prize, Fields Medal), and promotion.
Twentieth-century science adds another important characteristic:
(PR4) Scientists’ establish priority for a finding via peer-reviewed publication. Prestige increases depending on the journal’s prestige and citations to the publication.
At first glance, this reward system might seem a capricious manifestation of human ego, not akin to the disinterested and rational scientific spirit. Nonetheless, some authors have exalted its functional significance. I summarize the functional role of the priority rule in two alleged virtues.
2.1. Virtue 1: The Priority Rule Incentivizes the Production of Novel Scientific Findings
Scientific findings are traditionally regarded as public goods. Once findings are published, researchers cannot sell them, and everybody else can benefit from them. This is a problem insofar as a market economy does not provide incentives to produce them (Stephan Reference Stephan2012, 6). Arguably, a priority-based reward system has solved this problem (Stephan Reference Stephan2012). The scientist who makes a discovery appropriates it when her peers, especially her competitors, give her credit for being the first making the discovery. Prestige is the currency, which incentivizes scientists to produce findings that otherwise they would be less likely to produce and share. There could be monetary rewards, but they are derived from priority. For instance, in the twentieth century the patent system motivates research with commercial potential (Scotchmer Reference Scotchmer2004). Still, patents depend on priority.
2.2. Virtue 2: The Priority Rule Incentivizes the Division of Cognitive Labor
The problem of the division of cognitive labor is “how best to allocate resources among different possible scientific projects” (Strevens Reference Strevens2011, 188). Kitcher (Reference Kitcher1990, Reference Kitcher1993) states the problem as follows: imagine research programs that pursue the same goal (e.g., a vaccine). Some programs have higher potential (i.e., higher success probabilities) than others. For truth-driven scientists, the epistemically rational decision from their individual viewpoints is to join only higher-potential programs. But to maximize science’s payoff, the community has to divide labor to explore lower-potential programs as well. After all, the history of science teaches that programs/theories that seem less likely to succeed sometimes turn out to be correct/true. The problem is that achieving this division of labor requires some scientists to join programs that from their individual perspectives would be epistemically irrational to join. More generally: there is a distribution of epistemic goals for individual scientists, different from their rational distribution of goals, that yields a higher probability that the community as a whole will reach its epistemic goals. Then, the problem is, how can cognitive labor be allocated in a way that is epistemically irrational for some scientists?
Kitcher uses a formal decision-theoretic model to show that the reward system of science solves this problem. However, the particular argument that concerns me here can be stated qualitatively: communities in which scientists are both credit-driven and truth-driven will come close to the desired division of cognitive labor. Assume that when a research program succeeds, credit is divided among its contributors. If a scientist is truth-driven only, then she will join primarily higher-potential programs. But if she is also credit-driven, then she will take into account how many other scientists are working on a program. If the higher-potential program is too populated, then she is expected to join a lower-potential program: in case of success, credit would be divided among fewer researchers.
Strevens’s rejoinder is that the priority rule actually serves this end and furthermore promotes an efficient allocation of labor among research programs, that is, an allocation that (roughly) maximizes science’s payoff to society (Strevens Reference Strevens2003). The argument is as follows. In Strevens’s analysis, scientific races have a winner-contributes-all character: “the first discovery brings great benefits to science or society, while the second discovery brings nothing at all” (Reference Strevens2011, 192). And recall PR1, which Strevens calls the winner-takes-all character of the priority rule: scientists in the winner program get rewarded in proportion to their actual contribution to utility, while second runners get nothing. Now, contrast two scenarios: a hypothetical scenario that rewards all programs in terms of their expected utility and one in which PR1 is the case. In both cases, scientists will divide themselves across different programs. However, more scientists will join higher-potential programs in the latter scenario because succeeding first matters. This makes the latter scenario more efficient. In Strevens’s words, “scientists choose among research programs based on their prospects for earning scientific credit—status or reputation—and the reward system in science calibrates the allocation of credit so that the choices made lead to a distribution of labor that (roughly) maximizes the production of social good” (Reference Strevens2011, 194).
I will turn to discuss one core problem in this story.Footnote 6 Virtue 2 depends on the winner-contributes-all assumption. However, if this assumption is true, rewarded scientific findings ought to be for the most part replicable. To explain why this is a problem, I will first introduce some notions about replication using an example.
3. The Replicability Principle
A standard experimental paradigm for studying memory has the following form: (1) participants are presented a set of words, then (2) participants practice a random subset of those words, and then (3) participants take a free recall test. Unsurprisingly, participants are better at remembering practiced than nonpracticed words (otherwise it would not make sense to study for exams). Now, in 2011, Cornell University’s emeritus psychology professor Daryl Bem published an experiment inverting the order of 2 and 3. Participants had to take the test before practicing the words. His results show that participants were better at remembering words that they would practice later (
, 
). Strange as it may sound, Bem’s study seems to provide evidence for extrasensory perception (ESP). After two editors and four reviewers evaluated the paper, it was published in the leading social psychology journal, The Journal of Personality and Social Psychology (JPSP; Bem Reference Bem2011).
Can we regard this as a scientific finding? Leaving aside a complete analysis of the term, by “scientific finding” I mean a (report of a) finding that is truth-tracking (but not necessarily true) and trustworthy to build on. Does this finding have those qualities?
An immediate answer is to consider the replicability of the finding. Philosophers (Popper Reference Popper1959/2002) and scientists (Fisher Reference Fisher1926; Heisenberg Reference Heisenberg1975) have strongly defended replicability as the gold standard of scientific findings. As Popper bluntly puts it, “non-reproducible single occurrences are of no significance to science” (Reference Popper1959/2002, 64).Footnote 7 The relevant concept of replication here is often called “direct replication” in the literature: an experiment that mirrors an original experimental design in all factors that are purportedly causally responsible for the effect (Campbell and Jackson Reference Campbell and Jackson1979; Schmidt Reference Schmidt2009).Footnote 8 Using this concept, we can state a replicability principle. Suppose experiment E produces result F (e.g., some measurable quantity, such as an effect size 
for ESP); then,
(RP) F is a scientific finding only if in principle a direct replication of E produces F.
I formulate RP in a minimal way. RP does not state that actual replication of E has taken place, only that in principle a direct replication should be successful.Footnote 9 Also, RP does not imply that F is scientifically important, or robust, or that it generalizes beyond its experimental conditions (although any F that purportedly has those properties should be directly replicable too).
But how can we ensure replicability? One possible answer is to assess E’s methodological merits. This is what reviewers and editors do during peer review. In psychology, this means checking whether the authors did the appropriate controls, used adequate sample sizes, analyzed their data correctly, and so on. If these checks are positive, then we can expect the experiment to replicate. I call this hypothetical replicability. Explicitly
Hypothetical replicability: If E were directly replicated, then we would obtain F.
Establishing good methodology establishes hypothetical replicability. Consider Bem’s ESP study again. Most researchers received Bem’s findings with suspicion. The initial responses attempted to deny the paper’s hypothetical replicability. It is difficult, however, to show what went wrong. The paper does not have methodological flaws according to the current standards for psychological research. Indeed, Bem is more careful than many researchers, and typical worries about psychology experiments do not apply to his study. For instance, the study rules out the common problem of small sample sizes, given that he tested more than 1,000 participants. Also, the paper is arguably not a lucky shot since he reported nine experiments with similar findings. Now, from a classical statistical perspective, one can still try to speculate about possible methodological problems (Yarkoni Reference Yarkoni2011): Bem’s experiments have uneven sample sizes, which suggests data lumping or splitting; all p-values are close to the .05 threshold, which suggests selection bias; and, some features of the design suggest post hoc hypothesizing. All these together arguably inflate the probability of false positives. Importantly, however, it is hard to confirm these suspicions. Not to mention that one could have the same suspicions about most published papers in experimental psychology, which raises the question whether the suspicions undermine Bem’s paper more than any other paper.
Another reaction is to try to argue that Bem’s research evidences that methodological standards in psychology are inappropriate. In particular, one could blame the statistics. Advocates of Bayesian statistics claim that classical statistics overestimates Bem’s results as evidence for ESP (Rouder and Morey Reference Rouder and Morey2011; Wagenmakers et al. Reference Wagenmakers, Wetzels, Borsboom and van der Maas2011). Bem and colleagues’ response raises a standard worry about Bayesian analysis. In Bayesian statistics, hypotheses should be tested against noncommittal priors, and under what Bem and colleagues take to be the right noncommittal prior, Bayesian analysis favors ESP (Bem, Utts, and Johnson Reference Bem, Utts and Johnson2011).Footnote 10 This discussion shows, if anything, that there is space for disagreement about what constitutes a noncommittal prior and how to apply Bayesian analysis to controversial findings.
The focus on methodological worries does not take us far in assessing Bem’s findings. Furthermore, even if his findings are hypothetically replicable according to standard methodology, there are other possible sources of error beyond methodological flaws. This is true more generally and not only about his findings. Specifically, metascientific evidence in various fields suggests that many successful published findings result from questionable practices: selective reporting (Rosenthal Reference Rosenthal1979), exclusion of data points (Bones Reference Bones2012; John, Loewenstein, and Prelec Reference John, Loewenstein and Prelec2012), p-hacking (Simmons, Nelson, and Simonsohn Reference Simmons, Nelson and Simonsohn2011), HARKing (Kerr Reference Kerr1998), fraud (Fanelli Reference Fanelli2009), and experimenter’s bias.
These sources of error are not the sort that we can rule out as rare exceptions. And since the methodological checks do not reveal them, we cannot be satisfied with hypothetical replicability. Now, we cannot merely say that a finding such as Bem’s is the result of such practices. Instead, if we want to rule out such possibilities of error, we have to attempt to replicate the experiment in question. Consider then another notion of replicability:
Actual replicability: E has been directly replicated, and we have obtained F.
Actual replicability is not sufficient to ensure that F is a scientific finding. Originators and replicators may share miscalibrated instruments, in which case F would be false and still replicable. Also, actual replicability is not necessary either.Footnote 11 However, actual replications are particularly effective to increase trust in science under the right conditions (Simonsohn Reference Simonsohn2015). The intuition is simple. If you wonder whether F is truth-tracking and trustworthy, a particularly convincing answer is that E has been successfully replicated. And while there are diminishing returns in replicating the same finding, when multiple independent investigators succeed in finding F, the possibility of F being one lucky shot or the result of fraud decreases substantially. Now, how does actual replicability square with the alleged virtues of the reward system of science?
4. Vices: Novelty versus Replicability
These considerations about the efficiency of the reward system of science (virtue 2) depend on the winner-contributes-all assumption (Strevens Reference Strevens2011). But is this assumption true? Given a finding F, a winner contributes all only if the replicability principle (RP) is true of F. However, the interplay among the reward system, publication practices, and statistical inference procedures creates particularly inappropriate conditions for original proposers of F and third parties to ensure RP. I will now discuss these two vices.
4.1. Vice 1: The Priority Rule Systematically Rewards Findings Regardless of Their Replicability
Recall PR4: in the twentieth century, the peer review process determines priority (and prestige) over a finding. However, as I discussed in the previous section, the peer review process is useful to uncover methodological flaws in an experiment, which establishes hypothetical replicability. But what about actual replicability? Nothing done in the peer review process can establish actual replicability. In particular, reviewers of experimental papers do not attempt to replicate the work they are refereeing (at least not as part of their refereeing duties). Hence, establishing priority (and obtaining prestige) for a finding does not imply that the finding is actually replicable. Using Strevens’s terms, winners take all but may not contribute all.
Metascientific evidence confirms this theoretical argument. In psychology, a recent attempt to assess the replicability of published research suggests that only around 39% of published findings directly replicate (Open Science Collaboration Reference Collaboration2015). Replicability of published cancer research is also low. Biotechnology firms often try to capitalize on preclinical studies, turning promising findings into drugs. With this end in mind, in the late 2000s Amgen Inc. identified 53 papers of preclinical cancer research deemed as “landmark” in the literature. Scientists at Amgen attempted to directly replicate the experiments and even contacted the original researchers to perform direct replications under their supervision. They report, however, that they could only replicate 11% of the experiments (Begley and Ellis Reference Begley and Ellis2012). Now, Amgen’s report also evidences that authors often get their rewards regardless of the replicability of (and even controversies about) their findings: the report traces citations of the papers in the study and shows that citation counts to the nonreplicable and replicable papers are roughly the same, and the nonreplicable papers have produced secondary publications that do not intend to confirm or falsify them (Begley and Ellis Reference Begley and Ellis2012, 532).
Someone might object that ensuring the replicability principle is not a function of the reward system. This line of reasoning does not make my argument unsound but might make the conclusion seem less like a vice. However, this line of reasoning raises a question for the objector: how does (or should) science ensure the replicability principle?
There are two possible responses, not mutually exclusive but equally unsatisfactory. First, the objector might respond that the original researcher should ensure that her findings are replicable before attempting to publish them. Nonetheless, if the reward system rewards findings regardless of their replicability, then scientists cannot be rationally expected to ensure replicability for most of their findings. After all, ensuring replicability is more costly than not ensuring it. Second, and more important, successful replications by the same proposer of a finding often are not convincing for the community, and should not be. (Recall Bem’s nine experiments.) Such replications could be subject to systematic biases (e.g., experimenter expectancies and miscalibrated instruments).
The second possible response is that others in the community (should) ensure the replicability principle. That is, the objector could argue, actual replication takes place anyway because potential users of a finding attempt to replicate it after it has been published. Thus, in the long run, findings that survive will be replicable. This response, however, depends on users of findings having incentives to replicate. Consider the next vice.
4.2. Vice 2: The Priority Rule Systematically Disincentivizes Direct Replication of Others’ Findings
PR1 and PR4 create a problem for replication of others’ work. There are two possible cases when someone tries to replicate an experiment. If the replication succeeds, then it is not expected to receive high rewards because it is not a novel result, even if it contributes epistemically (e.g., the replication might provide a more accurate estimation of an effect size). If the replication fails, then, in theory, it can be expected to be rewarded: the replicator would be the first to prove the original experimenter wrong. Indeed, following Kitcher (Reference Kitcher1993, 339–40), one may think that when the original finding comes from a famous researcher, there are high incentives to replicate and high rewards on a failed replication.
In practice, however, the credit-driven scientist who wants publications has reasons to do novel research instead of replication work because publishing replication research is very difficult. In classical statistics, a negative result (understood as a failure to reject a null hypothesis) is typically inconclusive (Simmons et al. Reference Simmons, Nelson and Simonsohn2011). Hence, a failed replication qua negative result is difficult to publish in high-profile journals. To get a publication, the credit-driven scientist needs to not only fail to replicate the finding but also make the argument that the null hypothesis is true. And the rules to make this sort of inference are controversial (Machery Reference Machery2012).
It has also been known for a while that the editorial system is biased against replication research. Most editors regard showing novel findings to be more important (Neuliep and Crandall Reference Neuliep and Crandall1990, Reference Neuliep and Crandall1993). Negative results are disappearing in most disciplines (Fanelli Reference Fanelli2010, Reference Fanelli2012). Published direct replication attempts in the social sciences are less than 1% of published findings (Evanschitzky et al. Reference Evanschitzky, Baumgarth, Hubbard and Armstrong2007; Makel et al. Reference Makel, Plucker and Hegarty2012; Makel and Plucker Reference Makel and Plucker2014). And, at least in psychology, such attempts are mostly made by the same team that produced the original finding (Makel et al. Reference Makel, Plucker and Hegarty2012). Although not conclusively, this questions whether the incentives to replicate others’ findings are practically effective.
Bem’s ESP controversy illustrates these vices. After Bem’s publication, Ritchie, Wiseman, and French (Reference Ritchie, Wiseman and French2012a) performed a three-lab replication study, which turned out to be unsuccessful. They struggled to publish their article (Ritchie, Wiseman, and French Reference Wiseman and French2012b). The same journal that published the original article—JPSP—rejected the replication study on the basis that the journal does not accept direct replications (either successful or unsuccessful). Psychological Science rejected the paper on the same grounds (Winerman Reference Winerman2013). Another replication study suggesting no evidence for ESP had more luck, and it was published in JPSP (Galak et al. Reference Galak, LeBoeuf, Nelson and Simmons2012). Unlike Ritchie et al.’s study, Galak et al.’s study is not merely a direct replication since it provides a meta-analysis of replication attempts of Bem’s experiments from different sources, some of them unpublished, which arguably made the article more likely to be published under the journal’s policies.
The lessons from this story go beyond ESP. The perhaps surprising fact is that this situation is the same for many findings that are less controversial. Furthermore, the situation is worse when scarce resources and career pressures exacerbate the vices. Such findings could turn out to be false, but without direct replication attempts of the right sort, it will be hard to know.
5. Conclusion
The thesis that the reward system of science incentivizes an efficient division of cognitive labor is too optimistic. This thesis assumes that the scientific community in most opportunities produces and rewards replicable findings. This assumption is not unreasonable in the contexts that Merton had in mind when he wrote about the priority rule (e.g., historical cases of discovery in physics and mathematics). Nonetheless, in the context of contemporary experimental science, the assumption is highly questionable. Theoretically, a priority-based system can at most ensure hypothetical replicability but not actual replicability. And in practice, many rewarded findings are not replicable at all.
We need further work developing recommendations to address the vices in the reward system of science. We need to design scientific institutions that divide cognitive labor in a way that produces both new discoveries and well-tested theories. Such recommendations should consider that credit-driven scientists may not be suited for the latter task. We should also consider that different scientific fields have different notions of success and therefore might need different reward systems.
More generally, there is also a metaphilosophical upshot: Kitcher and Strevens’s work has inspired highly abstract formal studies on the social epistemology of science. If the aim is to say something relevant about science (and not only to develop formal models), these studies should take statistical inference practices and metascientific evidence more seriously.
