1. Introduction

According to the Requirement of Total Evidence (RTE), when assessing the credibility of hypotheses, we should endeavor to take into account all of the relevant evidence at our disposal instead of just some proper part of that evidence. I interpret the phrase “assessing the credibility” of hypotheses in RTE to mean assessing their epistemic probabilities. Given this interpretation, one implication of RTE is that, in assessing the effect of new evidence on the ratio of the epistemic probability of one hypothesis to the epistemic probability of another,Footnote ¹ one should use a more specific evidence statement that one knows to be true instead of a less specific one whenever different results would be obtained by doing so. For example, when comparing the hypothesis that Mona intends to harm Lisa to the hypothesis that she intends to benefit her, one might seek to determine how probable it is (antecedently) on each of the two hypotheses that Mona is bringing a butter knife to her meeting with Lisa. To ignore this more specific knowledge and focus instead just on the more general knowledge that Mona is bringing a knife (of some sort) to that meeting would commit the “fallacy of understated evidence.”

In a recent article in this journal, Epstein (Reference Epstein2017) offers two alleged counterexamples to this implication of RTE (and thus to RTE itself). Given how plausible this implication of RTE is, Epstein’s project is very ambitious and would be very important if it were to succeed. Unfortunately (for philosophical thrill seekers), it does not. Epstein’s apparent counterexamples are not genuine. After I explain why that is so, I close by saying something more about the importance both of RTE and of the project of defending it. Throughout the article, I assume that Epstein interprets RTE the same way I do, as a principle governing our assessment of the epistemic probabilities of hypotheses.Footnote ²

2. The First Alleged Counterexample

The context of Epstein’s discussion is a debate about the significance of the fact that life depends for its existence on the “fine-tuning” of several physical parameters. Given that this fact obtains, the issue is whether the further fact that our universe is life-permitting supports the hypothesis that there are multiple universes. White (Reference White2000) says it does not. This is because, while the fact that some universe is life permitting is antecedently more likely given the hypothesis that there are multiple universes than given the hypothesis that our universe is unique, the fact that some universe or other is life permitting understates our evidence. We know something more specific than that, namely, this universe (the only one we observe) is life permitting. White names this universe “α” and argues that, given that α exists, the antecedent probability that it is life permitting is the same, regardless of how many universes one assumes to exist.

In defending his position, White considers the following case:

In this case, the most specific evidence Jane has at her disposal, according to White, is just that someone rolled a double six, which is more likely given multiple players than given a single player. White concludes that Jane reasons correctly, but only because the case is not analogous to the fine-tuning case.

Epstein uses this case as his first alleged counterexample. He challenges White’s analysis as follows:

Epstein then asks whether E_S raises the ratio of the probability of H_MP (the hypothesis that there are multiple players) to the probability of H_OP (the hypothesis that there is a single player): “That is, was it more likely that Asha’s roll would come up double-six if there were many players?” (643). He answers, “Certainly not: no matter how many other players there were (zero or a million), Asha’s roll had a 1/36 chance of being double-six. So $P\left(E_S\right|H_{\mathrm{MP}})=P(E_S\left|H_{\mathrm{OP}}\right)$ ” (643). Since, however, Jane’s original reasoning that led her to the conclusion that there were likely multiple players is clearly correct, Epstein concludes that rationality demands in this case that the less specific evidence be used instead of the more specific. In short, RTE is false.

One initial worry about this alleged counterexample to RTE is that in White’s original description of the case an unspecified number of people simultaneously roll two dice, so there is no guarantee that only one player is responsible for Jane’s being awoken and thus no guarantee that a particular player is designated by the description “the player whose roll actually led to Jane being awoken.” This is no doubt why White says that Jane’s evidence is not about the outcome of a particular roll. It is easy, however, to solve this problem by adding a couple details to the Jane case. Suppose that, although all of the players roll their dice simultaneously, the results are checked one player at a time and, if a double six is discovered, then Jane is immediately awoken and no further results are checked. Suppose also that Jane knows all this. On this scenario, the single player whose roll actually led to Jane being awoken will be the first player who is discovered to have rolled a double six.

Epstein believes that Jane can use the name “Asha” to rigidly designate this person in spite of the fact that the referent of this name is fixed by description. Assuming that he is right about that, the fact remains that the identity of the particular player “Asha” rigidly designates is unknown to Jane because the referent of that name is fixed by a description that might, for all Jane knows, pick out a variety of different potential players. By contrast, in the fine-tuning case the referent of the rigid designator α is fixed by ostension, not by description: α is this universe, which could not be confused with any other universe since it is the only one we ever observe (and, arguably, is essentially ours).

No doubt Epstein believes that this difference between the two cases is irrelevant because the designation in both cases is rigid. Suppose, for example, that “Asha” actually refers to a player named Mona who, let us say, is a friend of Jane’s. Since the designation is rigid, it follows that “Asha rolled a double six” and “Mona rolled a double six” have the same truth-value in every metaphysically possible world, including of course those worlds in which Jane’s background knowledge in the actual world is true. Does this show that these two statements have the same epistemic probability? If it did, then Epstein would have a counterexample to RTE because Mona’s rolling a double six is no more probable on the assumption that Mona is the only player than on the assumption that she is just one of multiple players.Footnote ³ Crucially, though, these two statements do not have the same truth-value in every epistemically possible world. In other words, the designation is not rigid in the epistemic sense, and when epistemic and metaphysical possibilities diverge (without irrationality), epistemic probabilities depend on the former not on the latter. This is why, for example, a necessary a posteriori proposition like “water is H₂O” does not have an epistemic probability of 1 relative to the evidential situations of those who have not yet discovered its truth.

In the present case, Asha is Mona but Jane does not know this. Thus, although the statement that Asha is Mona is a necessary truth, it is also an a posteriori truth, and its epistemic probability relative to Jane’s evidential situation is clearly not 1. For Jane, it is epistemically possible that Asha is her friend Mona and also epistemically possible that Asha is her other friend Lisa or any of a variety of other people that she knows to be potential players. This means that, while there is only one metaphysically possible way for the revised evidence statement “Asha rolled a double six” to be true and that is for Mona to roll a double six, there are multiple epistemically possible ways for it to be true that do not directly involve Mona. Therefore, even though “Mona rolled a double six” and “Asha rolled a double six” have the same truth-value in every metaphysically possible world, and even though the statement that Mona rolled a double six is no more likely on the assumption that there are multiple players than on the assumption that there is one player, the statement that Asha rolled a double six is more likely (relative to Jane’s evidential situation) on the former assumption than on the latter. For to assume that there are two or more players instead of just one is (for Jane) to assume that there is at least one person (Lisa perhaps) whose roll could, for all Jane knows, make the statement that Asha rolled a double six true if some other person’s roll (e.g., Mona’s) failed to do so.

This conclusion is hardly surprising. In the fine-tuning case, a person who knows that some universe is life permitting by virtue of knowing that this universe is life permitting really does know more of relevance than someone who (for some bizarre reason) just knows that some universe or another is life permitting. In this case, however, Jane gains no new relevant knowledge simply by making up a name for the unknown someone in a statement like “someone rolled a double six” while leaving the identity of the referent of that name a mystery.

3. The Second Alleged Counterexample

Epstein worries (for good reason, as we have seen) that his first counterexample will not convince everyone because “the clever rigid designation involved in the case might seem fishy” (Reference Epstein2017, 643), at least to some. He is confident, however, that his second counterexample will convince any doubters and, because it is less artificial than the first, reveal “just how far-reaching a threat RTE, taken as a fully general principle, would pose to our ability to confirm hypotheses in the light of our evidence” (643). He describes the new case as follows:

Taking as your evidence the statement (call it E) that you caught some large fish, you correctly conclude that you have evidence favoring H_L over H_S since it is more likely that you would catch a large fish if there are mostly large fish in the lake than if there are mostly small fish in the lake. Of course, you also have more specific evidence than E. After all, you did not just catch some large fish or other, you caught the particular large fish in your boat—the one you strangely decided to name “Asha.” Notice that here the referent of the name “Asha” is fixed by ostension, making this case more analogous than the Jane case to the fine-tuning case, and hence less “fishy,” at least in one respect. Epstein claims that if, following RTE, you use the more specific evidence (call it E*) that you caught Asha, then you will be led astray. For whatever one assumes about the relative numbers of large versus small fish in the lake, the probability that you caught Asha as opposed to any other fish in the lake remains the same. Epstein concludes that by following RTE you will be led to the mistaken conclusion that you have no evidence favoring H_L over H_S.

One initial worry about Epstein’s argument here is that, since Epstein never claims that Asha is essentially large (which would be odd since presumably Asha used to be a hatchling, and fish hatchlings are relatively small), the statement (E*) that you caught Asha does not entail that Asha is large (or even that Asha is a fish) and thus does not entail E, the statement that you caught a large fish. It is crucial, of course, that E* entail E. Otherwise it is not more specific than E in the relevant sense, and using it as one’s evidence statement would actually violate RTE. Fortunately, this worry can be eliminated while simultaneously making the case more analogous to the fine-tuning case. Just as the more specific evidence statement in the latter case can be “this universe (the one we observe) is life permitting,” we can make the more specific evidence statement in the fish case, “this fish (the one you caught) is large,” which is equivalent to “you caught this fish and this fish is large,” or “C&L” for short. Understood this way, the counterexample is not genuine. Epstein is correct, of course, that the probability of C (your catching this fish) given H_L (most of the fish in the lake are large) is the same as the probability of C given H_S (most of the fish are small). But given that you caught this fish, it is clearly more likely that L (this fish is large) given H_L than it is given H_S. Notice that it is of no help to Epstein to assume that Asha is essentially large.Footnote ⁴ For even then your knowledge that this fish is large (like your knowledge that this universe is life permitting) is a posteriori. Thus, given that you caught Asha but prior (in the relevant sense, which may not be temporal) to your observing Asha’s size, it is epistemically possible that Asha—the particular fish you caught—is small. Thus, given that you caught Asha (and whether or not fish have their sizes essentially), Asha’s turning out to be one of the large fish in the lake is antecedently more likely if the lake contains mostly large fish than if it contains mostly small fish. Therefore, C&L is more likely given H_L than given H_S, which implies that C&L raises the ratio of the probability that H_L is true to the probability that H_S is true. So once again, RTE leads to the correct conclusion.

Given some remarks that Epstein makes in the last section of his article, I suspect he would criticize my argument here by saying that there is no probabilistic link between Asha’s large size and the proportion of large fish in the lake. After all, genes, age, and diet determine this fish’s size, not the ratio of large fish to small fish in the lake. From this Epstein would conclude that L and H_L are probabilistically independent, as are L and H_S. If he is right, then given that you caught this fish, the probability of this fish being large is the same whether one assumes H_L or H_S.

This argument, however, either conflates causal independence with probabilistic independence or makes a bad inference from one to the other.Footnote ⁵ It is true that there is no causal link between Asha’s size and the proportion of large fish in the lake, but it does not follow that there is no probabilistic link, at least not if the relevant probabilities are epistemic ones instead of objective ones. To say that two statements are probabilistically independent is to say, roughly, that the assumption that one of the statements is true has no effect on the probability of the other. The fact that there is no causal connection between the events reported by two statements does not show that they are independent in this probabilistic sense. If this is not obvious, consider the following example. Suppose that there is a six-sided die that might, for all one knows, be fair or might, for all one knows, be heavily biased toward the side with a six on it. Then the statements “the first roll of this die came up six” and “the second roll of this die came up six” refer to causally independent events (getting a six on the first roll does not cause one to get a six on the second roll), but they are clearly not probabilistically independent—the assumption that you rolled a six the first time raises the probability that you will roll a six the second time because it raises the probability that the die is biased in favor of rolling sixes.

In the particular case in question, keep in mind that you had no knowledge of Asha’s size before you caught her. Now ask yourself this. If you had to bet your life on correctly predicting the size of the next fish you caught, would you consider the information that most of the fish in the lake are large irrelevant? And would you do this on the grounds that, no matter which fish one catches, there is no causal link between the size of that fish and the proportion of large fish in the lake? Of course not. Even if the fish is already in your boat, and even if you have already taken the time to name that fish “Asha” (in which case you already know that you caught Asha), still, as long as you are in the dark about Asha’s size (e.g., because you are fishing blindfolded), the information that most of the fish in the lake are large will raise the probability that this fish, the one you just caught and named Asha, is large.Footnote ⁶ Thus, the statements that Asha is large and that most of the fish in the lake are large (or small) are not independent in the probabilistic sense even if the states of affairs to which they refer are causally independent. Thus, my conclusion stands. C&L is more probable given H_L than given H_S because, while C is equally probable on the two hypotheses, L is more probable given C&H_L than it is given C&H_S. Therefore, Epstein’s second alleged counterexample is not genuine.

One might argue, however, that a genuine counterexample to RTE can be generated by building L into the background information so that the only new evidence is Epstein’s E*, which states that you caught Asha. For example, imagine that you somehow knew before you caught any fish both that a particular fish named Asha exists and that Asha is large. Suppose further that you are capable (perhaps because of photographs you have seen of Asha) of visually identifying this fish. You then put your net in the lake, and to your surprise you see that you have caught Asha. (You are surprised because you had no idea that Asha was an inhabitant of this particular lake.) Now let C* stand for “You caught Asha” and L* stand for “Asha is large.” In this scenario, like the original one, your new evidence should raise the ratio of the probability of H_L to the probability of H_S. But while in the original case L was more likely given C&H_L than given C&H_S, in this case L* is not more likely given C*&H_L than given C*&H_S. Instead, L* is certain on both assumptions because Asha’s being large is part of your background knowledge. In this case, however, RTE still produces the right verdict because C* is more likely given H_L than given H_S. Since you did not already know that Asha was one of the fish in the lake in which you cast your net, it is antecedently more likely that Asha lives in that lake and hence that you would catch Asha if there are mostly large fish in the lake than if there are mostly small fish in the lake.

Furthermore, changing the case once again so that you knew beforehand not only that Asha is a large fish but also that Asha is an inhabitant of the lake in which you cast your net is also of no help to Epstein. In this case, C*&L* is no more probable on H_L than it is on H_S, but that is as it should be because in this case your new evidence should not raise the ratio of the probability of H_L to the probability of H_S. Why not? Because in this case the ratio of those probabilities would already have been raised by your knowledge that the one fish in the lake whose size is known to you is large. Indeed, violating RTE by employing the more general fact that you caught some large fish would lead one to double count the same evidence and thus mistakenly believe that one had new evidence favoring H_L over H_S. I conclude that there is no way to save Epstein’s second alleged counterexample by revising it.

It should not be surprising that the results in the original fish case are not analogous to White’s results in the fine-tuning case. (Recall that in the fine-tuning case neither hypothesis is favored, while in the unrevised fish case the mostly large fish hypothesis is favored over the mostly small fish hypothesis.) The two cases are not closely analogous. A fine-tuning case more closely analogous to the fish case would be one in which, just as we know that there are many fish in the lake, we know that there are many universes, and the question is whether the fact that our universe is life permitting favors the hypothesis that most universes are life permitting over the hypothesis that most universes are life prohibiting. Or, to reverse directions, a fish case more closely analogous to White’s fine-tuning case would involve comparing the probability that Asha is surprisingly large on the assumption that there are many fish in the lake to the probability that Asha is surprisingly large on the assumption that there is only one fish in the lake. If it is part of one’s background knowledge that Asha is a fish in (or from) the lake and if one has no reason to think that Asha’s size depends on how many other fish are around for Asha to eat, then these two probabilities are the same, just as, if the existence of our universe is a part of our background knowledge, the surprising fact (surprising because of its dependence on fine-tuning) that our universe is life permitting is no more likely if there are many universes than if ours is the only one.

In the rest of his article, Epstein uses his alleged counterexamples to justify his position that our choice of evidence statement should be guided not by RTE but by a version of the so-called Predesignation Requirement, which says, roughly, that scientists should designate which hypotheses are to be tested and decide how to analyze their data before looking at those data. As Staley (Reference Staley2002, Reference Staley2004) has shown, however, the Predesignation Requirement is not a fundamental norm of rationality like RTE. Nor, as we have seen, is it needed to get the right result in the Jane case or in the fish case.

Why, then, do we need the Predesignation Requirement (other than to help scientists avoid the temptation to construct ad hoc hypotheses and draw bad inferences from statistical significance when such significance is significant only in the technical sense)? Epstein claims that “notoriously, … violations of the Predesignation Requirement—sometimes referred to as ‘data mining’ or ‘overfitting’—can lead to the appearance of statistically significant experimental outcomes when no real confirmation has occurred” (Reference Epstein2017, 648; my italics). It is well known, however, that statistical significance does not guarantee confirmation, whether or not the Predesignation Requirement is followed. What is less well known and somewhat ironic in this context is that it is precisely because statistical significance testing violates RTE that statistical significance can be present, not just when confirmation is absent but even when disconfirmation (or, more precisely, confirmation of the null hypothesis) occurs. I explain this last point in the next section, where I turn to the issue of why it is important to show that there are no exceptions to RTE, or at least no exceptions to the specific implication of RTE that I have defended in this article.

4. Significance

One reason it is important to defend RTE is that scientists who use statistical significance testing routinely violate it, sometimes with very bad real-life consequences. Indeed, the assumption that it is sometimes permissible to understate relevant evidence is not the result of misusing the method of statistical significance testing. It is actually built into that method. This defect, it should be noted, is distinct from the more widely discussed problem that statistical significance testing ignores prior probabilities.

Allow me to explain with a personal anecdote. My daughter has autism, so I try to keep informed about the latest treatments. Many years ago, a number of researchers and medical doctors began promoting a new biomedical treatment for autism involving a hormone called secretin. Not only was this treatment expensive, it was also invasive (administered by infusion) and it had some bad potential side effects (e.g., seizures). The FDA had previously approved a natural form of secretin (taken from pigs) for the treatment of a condition unrelated to autism, making it possible for doctors to use secretin “off-label” to treat autism. At the same time, a synthetic form of secretin that was less likely to cause allergic reactions was progressing through FDA trials that were designed to determine its efficacy as a treatment for autism. When it was tested in a relatively large, double-blind, placebo-controlled phase 2 study, the results were statistically significant, which means, roughly, that it was very unlikely that differences between the experimental group and the control group at least as impressive as those obtained would occur given chance (i.e., given the “null hypothesis”). Further, a story about how the treatment would work was thought by some researchers to be biologically plausible, so perhaps the relevant priors were not negligibly low.

The problem, as I explained (or tried to explain) to other parents in the autism support group to which I belonged was that, given the way that the treatment was supposed to work and the way the subjects in the experiment were chosen (they were all prime candidates to benefit from the treatment if the treatment worked the way it was supposed to), the treatment should—if it really were effective—have resulted in even more impressive differences between the experimental group and the control group than the (barely) statistically significant differences that were observed. I concluded that the results actually obtained—as opposed to results at least as impressive as those obtained—were even more unlikely given the denial of chance than given chance (although of course extremely unlikely on both) and thus confirmed the null hypothesis in the sense of raising the probability of its truth. This was diametrically opposed to the conclusion being touted by the treatment’s defenders. Indeed, one highly distinguished researcher referred to the day that the results of the study became available as “the most important day in the history of autism.” As I expected, the parents I knew who took their child to get this treatment regretted it, if only because they spent a couple thousand dollars for a treatment that did not help their child in the least. The treatment was later shown by more than a dozen other studies (including a larger phase 3 FDA study), not to mention by clinical experience, to be ineffective.

Of course, it is possible that my analysis of this particular study is not completely accurate. I am not an expert on the medical issues involved. (I am just a parent with a background in confirmation theory.) Still, whether or not my story is true in all of its details, it illustrates a real problem with statistical significance testing. Again, the problem is that, in order to make the data manageable, the evidence is taken to be the fact that the actual results or more impressive results were obtained, when in fact we know something more specific than that, namely, that the actual results were obtained. Such a violation of the RTE may be harmless in most cases, but it has predictably bad consequences in others.

To see exactly why, notice first that, given how specific the actual results are in the sorts of studies in which statistical significance testing is used, those results are inevitably going to be very unlikely (to say the least) on both chance and the denial of chance. What is crucial, however, is that sometimes they will be even more unlikely on the denial of chance than on chance. This can happen even when the results are statistically significant because results that are surprisingly impressive if the null hypothesis is true can also be even more surprisingly unimpressive if the null hypothesis is false. Typically, these will be cases in which a theory about how a treatment will work if it works at all makes us confident that especially impressive results will be obtained if the null hypothesis is false and then much less impressive (although still statistically significant) results are obtained. By taking the evidence to be that we obtained the results we did or more impressive results, statistical significance testing is doomed to miss what is important in such special cases, and thus results will be mistakenly interpreted as evidence against chance when in fact they are evidence for it. I do not know how often this occurs, but that it occurs in some cases seems inevitable.

I conclude that the project of defending RTE by showing that one should not understate relevant evidence is an important one. Violations of this implication of RTE are common and potentially harmful, and it is much easier to criticize such violations when the waters are not muddied by claims that RTE has exceptions.