2. The narrow argument

Consider the following scenario:

What is the probability that we have randomly selected a person with the disease? The correct answer to this question can be calculated easily. Let $H$ be the hypothesis that we’ve randomly selected a person with the disease and $ + $ be the positive result. Then, by a straightforward application of Bayes’ rule,

$$P(H| + ) = {{{P\left( H \right)P( + |H)}}\over{{P\left( H \right)P( + |H) + P\left( {\neg H} \right)P( + |\neg H)}}}$$

$$ = {{{.001 \times .95}}\over{{.001 \times .95 + .999 \times .05}}} \approx .019.$$

People often give a different answer when asked similar questions (see, e.g., Tversky and Kahneman 1982).Footnote ³ In particular, they often respond to these kinds of questions by saying that the probability is .95—that is, they report the likelihood of $ + $ given $H$ or (in other words) the probability of a positive result given the assumption that the randomly selected person has the disease. This answer illustrates the base-rate fallacy: it confuses the likelihood $P( + |H)$ with the posterior probability $P(H| + )$ .

Howson argues that classical statistics is committed to precisely this fallacy (see Howson Reference Howson1997, S188–89, Reference Howson2000, 52–54). To make this case, he presents examples like Disease Testing as cases of hypothesis/significance testing.Footnote ⁴ The classical recipe for hypothesis testing involves administering a test that is designed so that some outcomes are very improbable if the hypothesis is true. Should we then observe those outcomes, the classical statistician takes this to be powerful evidence against the hypothesis; a good reason to “reject” it. How exactly we should interpret “rejection” here depends on whether we’re working within the broadly falsificationist approach of Fisher, the more behavioristic approach of Neyman and Pearson, or the evidential approach outlined by Mayo. It doesn’t matter for the present discussion which of these interpretations we prefer, however, so we can be sloppy in this respect and say that “rejecting” a hypothesis just involves believing that it’s false—this (ahistorical) interpretation of “rejection” is the most amenable to Howson’s criticisms.

If we treat Disease Testing as a case of hypothesis testing, a positive result is very improbable if the hypothesis that the randomly selected person doesn’t have the disease is true (that is, $P( + |\neg H)$ is very low). Since we observed this outcome, we have good reason to reject $\neg H$ —that is, to believe that $H$ is true. This is the result that we derive even though, as Howson stresses, it’s very likely that $H$ is in fact false. In effect, then, we’ve committed the base-rate fallacy: we’ve believed something that is very unlikely to be true because the classical recipe for hypothesis testing tells us to reject the hypothesis when the likelihood of the evidence is sufficiently low. So, since we’ve just followed the rules of classical statistics here, classical statistics is in some important sense intrinsically committed to the base-rate fallacy. Howson concludes that the rules of classical statistics are “demonstrably unsound” (Howson Reference Howson1997, S188), and remarks that “It can only be in a Pickwickian sense of ‘good’, therefore, that a 2 per cent chance constitutes ‘good grounds for H’ ” (Howson Reference Howson2000, 54).

Howson’s claim that classical statistics commits the base-rate fallacy is essentially a diagnosis of why it’s unsound: classical statistics takes the low likelihood of the observed results on the hypothesis that the randomly selected person lacks the disease as grounds for belief that they have the disease, whereas the (only?) proper grounds for belief is a high posterior probability. As Howson puts it:

Presumably, a probability of approximately .02 is “sufficiently low”; that granted, the argument is valid. The preceding paragraphs demonstrate that (N2) and (N3) are both true. As we’ll see in section 5, the idea behind (N4) is contentious, but we’ll assume that it’s true for the sake of argument for now. And the diagnosis is tendentious and dismissive—Howson (Reference Howson2000, 54) explicitly urges us to think that the “unsoundness” of Mayo’s position is owed to an obvious fallacy rather than a philosophically motivated disagreement about the proper approach to statistics—but (if the argument is correct) it does seem to get at the heart of the problem.

That leaves (N1).

3. The narrow argument rebuffed

The first of Mayo’s three responses that we’ll consider is that (N1) is false. In Mayo (Reference Mayo1997a), for instance, she argues that classical hypothesis testing is only appropriate when there aren’t (classically acceptable) priors. Where such priors exist, the classical statistician should use them:

In another article appearing in the same year, she points out that if we take the perspective of classical statistics, the hypothesis in question should not be considered a proper statistical hypothesis at all:

According to Mayo, in other words, classical statistics views hypothesis testing and calculating the probability that a random variable takes on some value as distinct problems. (N1) is false, therefore, because Disease Testing involves calculating the probability that a random variable takes on some value and is thus not a problem where classical statistics recommends hypothesis testing.Footnote ⁵

Is Mayo’s characterization accurate? To answer this question, we need look no further than Fisher’s Statistical Methods and Scientific Inference, which is unequivocal on the matter:

One can find similar comments elsewhere in Fisher’s work (e.g. Fisher Reference Fisher1958, 9–10), as well in that of the other “fathers” of classical statistics, Neyman (Reference Neyman, Godambe and Sprott1971, 3) and Pearson (Reference Pearson1962, 395–96). Indeed, these quotes are unsurprising. Throughout the first half of the 20th century, Fisher’s “logical distinction” was typically understood to be the distinction between “direct” inferences from populations to (appropriately selected) samples—which admit of straightforward mathematical reasoning—and “inverse” inferences that go the other direction.Footnote ⁶ Only the latter were understood to be the proper domain of hypothesis testing. This framework is largely presupposed in the canonical work of both Fisher (1922, 313–14, Reference Fisher1958, 7) and Neyman and Pearson (1928, 175, Reference Neyman and Pearson1933, 291).

So neither the canonical statements of the fathers of classical statistics nor Mayo’s philosophically sophisticated reconstruction recommend applying hypothesis testing in Disease Testing. Modern textbooks are no more supportive of (N1). Lehmann and Romano (Reference Lehmann and Romano2022), widely regarded as the textbook on hypothesis testing, opens with a specification of the problem of statistical inference that excludes Disease Testing:

Other textbooks are similar. Wasserman (Reference Wasserman2005, ix), for example, contrasts “probability,” which involves calculating the probability of outcomes of a given data-generating process, with “statistical inference” where the problem is the “inverse” (again) of that for which the calculations of probability are appropriate.Footnote ⁷ Neither formulation licenses treating the question of whether a random variable takes on a particular value as a hypothesis in the sense of statistical inference generally speaking, let alone hypothesis testing.

Howson is aware that what we’ve termed (N1) is open to objection. Here is his discussion, quoted in full:

There are two rejoinders here. The second—that the objection is “beside the point”—finds no real purchase against Mayo’s response: the problem is that the account that he is criticizing says that it would be inappropriate to apply hypothesis testing to the alleged example; whether the conditions are met for the inappropriate test to deliver a particular verdict is immaterial. In determining whether a rule or procedure is “sound” or “valid,” what that rule would say outside its explicitly specified domain of proper application is irrelevant.Footnote ⁸ The narrow argument is akin to claiming that first-order logic is unsound because of what it says (or doesn’t say) when applied to strings of symbols that aren’t well-formed formulas.

The first rejoinder—which is essentially that the objection is confused—is more interesting. In the present context, Howson can be read as pointing out that the distinction between inferences from populations to samples and inferences from samples to populations is not recognized by probability theory. That’s correct, of course, and at face value it puts pressure on the distinction that the classical statistician wants to draw. Certainly, it undermines Fisher’s characterization of the distinction as a “logical” one. It does not undermine the distinction itself, however, at least not absent further argument. The classical statistician has at least two routes for response.

The first of these is to appeal to different interpretations of probability. According to a “frequentist” interpretation, as Mayo (Reference Mayo1997a, S205) highlights in her response, we cannot coherently assign probabilities to certain kinds of hypotheses because those hypotheses are either true or false; there’s no sampling procedure or long-run frequency to make the probability claim “meaningful” (see also Spanos Reference Spanos2010, 577). As the hypothesis in Disease Testing is the result of a random sampling process, it can be properly characterized as a random variable according to frequentist strictures. So we can and should calculate its probability distribution in a straightforward mathematical way rather than employing hypothesis testing. In other words, a frequentist interpretation of probability blocks Howson’s argument. Whatever the defects of frequentism, it does license a distinction between Disease Testing and those cases where hypothesis testing is applied.

While talk of different interpretations of probability is familiar to philosophers, it is often misleading when mapped onto the concerns of statisticians. The alternative is to ground the distinction in a view about what’s required for the justified use of probability distributions in science. Roughly: there are some cases where we have sufficient information to specify an intersubjectively acceptable prior probability distribution. Disease Testing is a paradigm example: we know the prevalence of the disease in the population and can treat it as common knowledge. At the same time, there are other cases—thought by the classical statistician to be more common in interesting scientific contexts—where the prior probability distribution is so underdetermined that it’s better to proceed without one. Pearson seems to endorse a version of this view in recollecting about his early work with Neyman:

Pearson’s suggestion, essentially, is not that probabilities are relative frequencies, but that relative frequencies are a way of grounding or justifying the choice of probability distribution.Footnote ⁹ Like the first response, this one blocks Howson’s rejoinder: the classical statistician may be wrong about what’s required for justification, but they aren’t committed to (N1).

The narrow argument rests on the assumption that classical statistics recommends applying the methods of hypothesis testing in cases like Disease Testing, an assumption I’ve labeled (N1). Mayo’s response is that (N1) is false. On this point, she’s clearly correct: given that both the founders of classical statistics and Mayo herself explicitly recommend against doing so, both the founders and modern textbooks adopt frameworks that preclude doing so, and all three do so on grounds that are broadly consistent with their other views on the interpretation and justification of probability functions, there is simply no question of maintaining this assumption. The idea that classical statistics is committed to the base-rate fallacy is simply a misunderstanding of classical theory.

4. The refined argument

With the exception of Sprenger (Reference Sprenger, Hájek and Hitchcock2017), more recent discussions of the relationship between classical statistics and the base-rate fallacy have shied away from insisting that the theory is actually committed to a fallacious inference. Titelbaum, for instance, directs his discussion of what is essentially Howson’s example against the idea that it’s appropriate to (automatically or mechanically) reject the null hypothesis when the alternative passes a test (Titelbaum Reference Titelbaum2022, 464)—a conclusion that everyone involved accepts.

For our purposes, the more interesting argument is raised by Achinstein:

The idea behind Achinstein’s arguments is apparently simple: classical statistics may not be committed to the base-rate fallacy, but it doesn’t give us a high posterior probability. And a high posterior probability is what we really care about.

Note that Howson can also be read in this way, though doing so requires treating his claim that classical theory is “demonstrably unsound” as mere rhetoric. For instance, he complains that “Error statisticians … have for decades given us something quite different from what we want, which is a way of computing the degree of confidence we should invest in hypotheses given empirical evidence” (Howson Reference Howson1997, S189). And his response to the objection that his case involves a random variable could, with great charity, be understood as granting the narrow point but holding that the fact that the kind of properties that the classical statistician cares about are so misleading in that scenario indicates that something is deeply wrong with the approach.

(R1) and (R2) are simply facts about the relevant quantities and methods. The only point of attack, therefore, would seem to be (R3). I suspect that most philosophers sufficiently enmeshed in Bayesian confirmation theory will find (R3) plausible, perhaps even obviously true—once it’s demonstrated that error probabilities and posterior probabilities come apart, the literature on the subject has tended to assume that no further argument is needed to show that classical statistics doesn’t give us what we care about. And while appeals to the base-rate fallacy are both recent and limited to philosophy, the idea that classical statistics doesn’t give us what we care about is an older and broader one: Kyburg (Reference Kyburg1961, 27) calls essentially the same complaint “the most common objection to frequency theory.”

Still, the fact that (R3) is intuitive is no guarantee that it is true. As we’ll see, Mayo argues that it isn’t.

5. The refined argument rebuffed

The second of Mayo’s three responses is what initially appears to be an entirely unpromising decision to bite the bullet. She repeatedly asserts that what we really want (or at least what scientists really want) is not information about how likely it is that the hypothesis is true, but information about how well-“probed” it is; we want to know if it has been subjected to tests that are discriminating and that can be expected to reliably distinguish true hypotheses from false ones (Mayo Reference Mayo1997b, 328, Reference Mayo and Achinstein2005, 109, 121, Reference Mayo, Mayo and Spanos2010, 197, Reference Mayo2018, 226–27). Why is that? Mayo clearly sees her response as grounded in the entirety of the picture that she presents: the point of the various arguments that she offers throughout her work is to convince us that probity is something that we should care about. In some sense, then, Mayo and the Bayesian are simply at loggerheads. Evaluating the refined argument is then impossible without evaluating the position as a whole.

I do not think things should be left here, however. Though Mayo does not make this connection in her published work, the idea that we care about probity has substantial support within contemporary epistemology. In other words, widely held views in epistemology provide us with strong grounds for rejecting (R3) and thus the refined argument. Here, briefly, is the case. First, it’s commonly thought that “what we care about” is knowledge, which comes apart from posterior probability. Second, it’s also commonly thought that knowledge requires truth-tracking and, as we’ll see, error probabilities are best understood as measures of the degree to which our method tracks the truth. Together, these two common views imply that error probabilities are something we care about in at least the same way that posterior probabilities are: they measure something that is necessary but not sufficient for our central epistemic aim.Footnote ¹⁰

Lottery paradoxes of the sort originally put forward by Kyburg (Reference Kyburg1961) can be used to illustrate both of my main points in this section. So consider a lottery in which one winner will be chosen at random from a set of 10,000 tickets. The probability of any individual ticket winning is then 1 in 10,000. Kyburg points out that it is easy to generate a contradiction from these facts using only extremely reasonable inferences. The probability that any individual ticket wins is so low that you can reasonably conclude of any individual ticket that it will not win. If you can conclude of any individual ticket that it will not win, however, then it seems reasonable to conclude that no ticket will win. But this contradicts our assumption that one of the tickets would win.

One common response to the lottery paradox is to hold that no matter how high the probability that an individual ticket loses, there’s something defective about believing that the ticket will lose. The problem is that even if the belief that the ticket loses is true, it isn’t tracking the truth in a way that’s important to knowledge. If the ticket was the winner, you would still think it wasn’t (Dretske Reference Dretske1971). If you were to believe that it was, it still wouldn’t be (Pritchard Reference Pritchard2005). And there’s some important sense in which no explanation would be needed for you to have an incorrect belief in this case—it would be very normal for you to be wrong (Smith Reference Smith2016). Regardless of how we choose to cash out truth-tracking—the three options above are sensitivity, safety, and normalcy, respectively—it’s widely felt that beliefs that track the truth are more valuable than those that don’t, even in situations where both beliefs are true or even both true and justified. Which is essentially to say that “what we care about” is not just truth/accuracy but (modally stable) knowledge. From this perspective, the lottery paradox is a particularly nice illustration of the fact that high posterior probabilities can come apart from “what we care about.”

So the lottery paradox illustrates the first of the two main points of this section: many contemporary epistemologists hold that what we care about is knowledge, and knowledge comes apart from high posterior probability, which is (at best) a necessary but not a sufficient condition for it. To illustrate the second point—that the error probabilities of classical statistics should be thought of as measuring the degree of truth-tracking—it will be helpful to have a more finely specified case. So consider what we’ll call “Lockean Lottery” after Foley (Reference Foley1992):

(Those who feel the potential costs of John’s decision affect our judgments in this case can imagine that he has voluntary control over his belief and he is simply deciding what to believe; nothing will turn on the actual costs and benefits of throwing away the ticket.)

Notice: John’s ticket being the winning ticket doesn’t affect how many tickets have been sold. As a consequence, the error probabilities for John’s decision procedure are just a function of the cutoff he chooses. More precisely, the probability of judging his ticket to be a loser when it isn’t is just the probability that more than 10,000 tickets have been sold; the probability of failing to judge his ticket a loser when it is, is the probability that 10,000 or fewer tickets have been sold. Or, more formally, the probability of the first kind of error is $P(n \gt 10,000)$ ; the probability of the second kind is $1 - P(n\gt 10,000)$ .

That’s bad. Normally, we aim for statistical tests in which both probabilities are very near to 0. Infamously, a .05 probability of a false negative is a standard minimum requirement for publication in most disciplines; there’s no similar standard for false positives, but .2 is a commonly cited cutoff. John’s method can’t achieve anything close to that. In fact, if $P(n\gt10,000)$ is anything other than .5—in which case his method is no different than flipping a coin—he’s guaranteed that the method will be anti-truth-tracking in one respect or the other. No honest statistician would ever recommend a testing method with this error profile.

As Bayesians have rightly stressed for decades, the error probabilities are not telling us that John’s method is unreliable; after all, if the number of tickets sold is greater than 10,000, then John’s method is extraordinarily reliable. It has a success rate of $\left( {n - 1} \right)/n$ ! Instead, what they’re telling us is that John’s method is not tracking the truth of the hypothesis. More precisely, the two error probabilities, $P(n\gt 10,000)$ and $1 - P(n\gt10,000)$ , are quantitative measures of how sensitive John’s method is to the truth and falsity of the hypothesis respectively.Footnote ¹¹ If John’s method were perfectly sensitive, the probability that John would reject the hypothesis if it were true would be 0, as would the probability that he would accept that hypothesis if it were false.Footnote ¹² The poor quality of the actual error probabilities tells us that John’s method is not tracking the truth at all, which means that it cannot support knowledge; what the statistical analysis reveals is that John’s beliefs don’t track the truth in the way knowledge is commonly thought to require.

Contrary to what the critics suggest, therefore, classical statistics has a good claim to giving us something that we care about: error probabilities measure one of the necessary components of knowledge. At best, then, (R3) in the refined argument is an unargued-for assumption that is inconsistent with common views in epistemology. As such, I think it’s fair to conclude that the refined argument is unsuccessful absent substantial further justification of (R3).

In an important sense, however, this conclusion understates the problems with the refined argument. Recall that the argument relies on a kind of sleight-of-hand: the audience is shown a misleading-at-best example in which the classical statistician allegedly commits to the base-rate fallacy, and it is then argued that while they aren’t actually committed to the fallacy, the example still illustrates that the methods of classical statistics are deficient in some deeper sense. The entire intuitive pull of the argument, however, turns on the base-rate case: what makes it compelling is that Disease Testing is the rare situation where we know that posterior probabilities track the truth and thus that results that disagree are unreliable—Howson (Reference Howson1997, S190) himself says as much! In other words, the features that make the refined argument compelling are exactly the features that make it misleading. To me, it thus seems fair to conclude that even if the assumptions identified above can be defended, the argument itself only has the advantages of theft over honest toil to recommend it.

6. Should the Bayesian fear base rates?

The previous two sections have demonstrated that neither the narrow argument offered by Howson and Sprenger nor the refined argument offered by Achinstein is successful. The former is just straightforwardly unsound; the best thing that can be said about the latter is that it relies on assumptions that are both controversial and for which no argument is provided. Nevertheless, there is one more argument of Mayo’s that we have yet to examine—an argument to the effect that it is the Bayesian who should be afraid of base rates.

To makes this case, Mayo appeals to scenarios that look superficially like Disease Testing (Mayo Reference Mayo1997a, S205-6, Reference Mayo1997b, 327–29, Reference Mayo and Achinstein2005, 115–18, Reference Mayo, Mayo and Spanos2010, 195–99, Reference Mayo2018, 368–69). Here is a version of her most common example:

Mayo takes Readiness Testing to be amenable to the methods of hypothesis testing in a way that Disease Testing isn’t (cf. Spanos Reference Spanos2010, 577–79). The crucial difference? Here our concern is with a particular individual, not a randomly selected sample, and the information about the population—the prior, essentially—is no longer acceptable by frequentist strictures.Footnote ¹³

Before turning to the importance of this point, notice that a strict Bayesian can happily agree with Mayo that the previous research doesn’t dictate the choice of prior. Indeed, Levi (Reference Levi1981, Reference Levi1983) makes essentially the same point in his discussion of empirical research on the base-rate fallacy: for a committed subjectivist, there’s no necessary connection between the frequency of college readiness in the population and the prior probability that Isaac is college ready; so long as the relevant probability assignments obey the Kolmogorov axioms, the subjectivist is free to assign any prior probability. About this much, at least, the strict frequentist and the strict subjectivist agree.

So the Bayesian should be willing to grant Mayo that there is no necessary connection here; it’s consistent for her to employ the population-level information in the first case but not in this one. Still, most Bayesians are liable to say that the population-level information is relevant; after all, the whole point of including prior distributions in the analysis—or at least of including priors other than the objectivist’s minimally informative ones—is to account for information like the general rate of college readiness among Isaac’s peers. Here, therefore, we have an example that parallels Disease Testing but where the two accounts actually disagree. Why, then, does Mayo think this example supports the classical viewpoint when she was so careful to reject the earlier application?

To draw out her reasoning, notice an oddity about the Bayesian position in this scenario: for the Bayesian, there is no reason for Isaac to take the TAS. Or, more accurately, there is only reason to carry out the test given a very specific utility distribution: Isaac’s text scores matter only if our decision problem is set up so that the very small change in the probability of Isaac’s readiness alters which of our options has the highest expected utility. There’s something worrisome about this aspect of the Bayesian position: it seems bad that the prior swamps any evidence that we could collect using available methods; it seems worse that we have to jury-rig the decision problem to avoid the situation where the Bayesian recommends rejecting Isaac from college purely on the basis of their priors.

It’s easy to give this intuition some teeth. In the U.S., a long history of discriminatory practices that encourage (and enforce) segregation has rendered the location of a person’s residence a good proxy for their race (Rothstein Reference Rothstein2017; Taylor Reference Taylor2019). Widespread reliance on these proxies continues to contribute to racial disparities in a variety of sectors, including incarceration, housing, health, and education. Against this backdrop, it should be disturbing that even apparently innocuous demographic information about Isaac’s peers can entirely swamp the information that we have about Isaac himself—indeed, can swamp it to such a great extent that there’s no point in Isaac even trying.

It’s worth being precise about the worry here. It is not that priors are “subjective” in some vague sense and therefore guaranteed to be biased.Footnote ¹⁴ The concern is a narrower one: Mayo contends that when we take the critics’ own examples and strip them of the idealizations that ensure the accuracy and/or relevance of the prior, we’re left with a scenario in which the Bayesian’s recommendation—or at least the critics’—is extremely troubling. On the one hand, we have a prior that is subject to few constraints, influenced by information we have good reason to think is biased, and whose relevance to the present testing situation we have no way of evaluating.Footnote ¹⁵ On the other hand, we have data from an extremely probative test that we know is relevant to the present testing situation. But if we adopt the method that the critics apparently believe we should, we allow the former to dominate both our epistemic conclusions (the posterior probabilities) and our decision making.

Bayesians have two traditional responses to worries about the subjective elements of their methodology. Neither is particularly compelling in this case. The first response involves pointing to results (e.g., Hawthorne Reference Hawthorne1993) that show that agents who properly conditionalize will eventually converge on the same posteriors under relatively weak assumptions about their priors. In this scenario, however, it’s hard to see how eventual convergence helps Isaac—here, at least, the Bayesian is susceptible to their own complaint that what we care about is whether the hypothesis is correct, not whether our method has nice features in the long run. The second response is to (correctly) point out that classical methods also rely on the judgment of the individual statisticians. But Mayo can—and indeed does (see Mayo Reference Mayo2018, Excursion 4)—grant the general point while arguing that it isn’t relevant: at least in this specific scenario, we have good reason to think that the priors are susceptible to bias and substantial empirical evidence that constrains the classical approach. Even if the different elements are all generally susceptible to bias, the prior is clearly more susceptible in this specific case.Footnote ¹⁶

What does this argument show? By itself, I think, not a lot. Consider two distinct audiences for this kind of argument. The first views statistics as akin to engineering: the goal is to develop the right test for a messy and complex world where things often don’t behave as they’re supposed to.Footnote ¹⁷ If they are not already users of classical statistics, they are at least sympathetic to the idea that there may be cases where Bayesian tools are ineffective or unreliable and, crucially, take that as a reason not to use Bayesian tools in those specific cases. To this first audience, Mayo has offered a compelling scenario in which potentially biased prior information swamps a probative test and thus where it’s arguable that the classical approach is simply preferable. But the interesting question for this audience is not whether these cases exist but their commonality: is Readiness Testing more representative of the use of statistics in science than the cases that Bayesians like Spielman (Reference Spielman1974, 219) point to when motivating their position? (How are we even to evaluate this question, given that the concern is at least partly about the appropriate questions to ask in different scientific contexts?)

The second distinct audience is more committed to a Bayesian philosophy. This audience, I suspect, is unlikely to be swayed towards a pluralistic—let alone a classical—approach by pointing to specific cases where a reliance on Bayesian tools leads to dubious judgments. After all, one of the common commitments of the Bayesian philosophy is to seeing the problems of statistics as problems of logic; the goal is not a toolbox of different rules, methods, and heuristics specifically suited to their individual situations, but rather something that looks like quantified first-order logic. Howson and Urbach express this view when they remark that the “truth, rationality, objectivity, cogency or whatever of the premises … are exogenous considerations, just as they are in deductive logic” (Howson and Urbach Reference Howson and Urbach2006, 301). They would likely say the same thing about Readiness Testing; on their view, it’s simply not the job of a theory of statistics to say anything about where you got the information on which your prior distribution is based. Bad inputs lead to bad outputs—just as they do in logic.

Like the refined argument, then, Mayo’s response is indecisive without further supplementation (though one might fairly note that her work more broadly can be viewed as an extended argument for the first of these positions). Nor should this be surprising—if the disagreements among statistical approaches could be definitively settled by appealing to simple examples like those found in the philosophical literature, we would have expected it to be settled long ago. Cast in this light, I think Mayo’s third response is best read not as an argument against Bayesianism writ broadly—among other things, not all Bayesians would endorse the kind of flat-footed picture of testing that leads to trouble here—but instead as a demonstration that the defender of classical statistics is just as capable of constructing problematic scenarios involving base rates as the Bayesian.