1. Introduction

The large n problem was initially raised by Lindley (Reference Lindley1957) in the context of the simple Normal model,

where NIID stands for Normal, Independent, and Identically Distributed with mean and variance (assumed known), by pointing out

Lindley went on to claim that this result is paradoxical because, when viewed from the Bayesian perspective, one can show

This result was later called the Jeffreys-Lindley paradox because the broader issue of conflicting evidence between the frequentist and Bayesian approaches was first raised by Jeffreys (Reference Jeffreys1939/1961, 359–60). Claims a and b contrast the behavior of a frequentist test (p-value) and the posterior probability of as , which brings up a potential for conflict between the frequentist and Bayesian accounts of evidence:

This potential conflict is given a more distinct Bayesian slant by

The problem of conflicting evidence pertains to the broader philosophical issue of grounding statistical practice on sound principles of inference and evidence. What has not been adequately explained in this literature is why, given the rejection of by a frequentist test, its posterior probability going to one as (irrespective of the truth or falsity of ) is conducive to a more sound account of evidence.

The primary objective of this article is to consider this issue by comparing the frequentist, Bayesian, and likelihoodist accounts of inference and evidence. The discussion can be seen as part of a wider endeavor to use the error statistical perspective (Mayo Reference Mayo1996) in an attempt to have a closer look at several Bayesian allegations that have undermined the credibility of frequentist statistics in philosophical circles over the past half century.

The brief comments in section 2 provide a prelude to the discussion that follows by clarifying certain key issues at the outset. Section 3 introduces the large n problem in frequentist testing using a numerical example discussed by Stone (Reference Stone1997). This example is based on a very large sample (n = 527,135) that is used to bring out the fallacious claims associated with the Jeffreys-Lindley paradox without the technical difficulties of invoking limiting arguments as . In sections 4 and 5, the Bayesian and likelihoodist approaches are applied to this example with a view to demonstrate that both approaches are far from immune to fallacious results, contrary to the current view among proponents of the Bayesian and likelihoodist perspectives (see Berger Reference Berger1985; Royall Reference Royall1997; Robert Reference Robert2007; inter alia). Section 6 illustrates how the postdata severity evaluation of the p-value and accept/reject results addresses not only the large n problem but also the broader fallacies of acceptance/rejection and calls into question the charges stemming from the Jeffreys-Lindley paradox. The severity perspective is then used in section 7 to shed light on why the Bayesian and likelihoodist accounts of evidence give rise to highly fallacious results.

2. Clarifying What Is Fallacious or Paradoxical

Before discussing the various claims that relate to the Jeffreys-Lindley paradox, it is important to bring out certain key issues that have not been adequately illuminated by the literature.

First, in frequentist testing, which includes both Fisher’s significance and the Neyman-Pearson testing, the large n problem arises naturally because the power of any “good” (consistent) test increases with n. An α-significance level Neyman-Pearson test is said to be consistent when its power to detect any discrepancy from approaches one as . In this sense, there is nothing fallacious or paradoxical about a small p-value or a rejection of the null, for a given significance level , when n is large enough, since a highly sensitive test is likely to pick up on tiny (in a substantive sense) discrepancies from .

Second, Bayesian charge 2 (d) overlooks the fact that the cornerstone of Neyman-Pearson testing is the trade-off between the type I (reject when true) and type II (accept when false) error probabilities. In this sense, these charges ignore the decrease in the type II error probability as n increases, since, for a given discrepancy γ, the power is one minus the type II error probability. Indeed, various attempts have been made to alleviate the large n problem, including decreasing α as n increases in an attempt to counterbalance the increase in power associated with n (see Lehmann Reference Lehmann1986). The difficulty, however, is that only crude rules of thumb for adjusting α can be devised because the power of a test depends on other factors besides n.

Third, the large n problem (a) constitutes an example of a broader problem known as the fallacy of rejection: (mis)interpreting reject (evidence against ) as evidence for a particular ; this can easily arise when a test has very high power. Due to the trade-off between type I and II error probabilities, any attempt to ameliorate the problem by selecting a smaller significance level when n is large might render the result susceptible to the reverse fallacy known as the fallacy of acceptance: (mis)interpreting accept (no evidence against ) as evidence for ; this can easily arise when a test has very low power (e.g., n is very small). As argued below, the large n problem (a), when a rejection of is interpreted as evidence for , and the Jeffreys-Lindley paradox (b) constitute examples of the fallacies of rejection and acceptance, respectively.

The main argument of this article can be stated succinctly as follows: (i) Although there is nothing fallacious about a small p-value, or a rejection of , when n is large (it is a feature of a good frequentist test), (ii) there is a problem when such results are detached from the test itself and are treated as providing the same evidence for a particular alternative regardless of the generic capacity (the power) of the test in question. The large n problem is directly related to points (i) and (ii) because the power depends crucially on . That in turn renders a rejection of with a large n (high power) very different in evidential terms from a rejection of with a small n (low power). That is, the real problem does not lie with the p-value or the accept/reject rules as such but with how such results are fashioned into evidence for or against a particular hypothesis or an inferential claim relating to (). (iii) It is argued that the large n problem can be circumvented by using the postdata severity assessment to provide a sound evidential account for frequentist inference. Whether data provide evidence for or against a particular hypothesis ( or ) depends crucially on the capacity of the test in question to detect discrepancies from the null. This stems from the intuition that a small p-value or a rejection of based on a test with low power (e.g., a small n) for detecting a particular discrepancy γ provides stronger evidence for the presence of γ than using a test with much higher power (e.g., a large n). As first pointed out by Mayo (Reference Mayo1996), this intuition is completely at odds with the Bayesian and likelihoodist intuitions articulated by Berger and Wolpert (Reference Berger and Wolpert1988), Howson and Urbach (Reference Howson and Urbach2006), and Sober (Reference Sober2008). Indeed, Mayo went on to propose a frequentist evidential account based on harnessing this perceptive intuition in the form of a postdata severity evaluation of the accept/reject results. This is based on custom tailoring the generic capacity of the test to establish the discrepancy γ warranted by data . This evidential account can be used to circumvent the above fallacies, as well as other (misplaced) charges against frequentist testing (see Spanos Reference Spanos2011). (iv) In contrast to frequentist testing, no such reasoned remedy exists for the Bayesian and likelihoodist approaches, whose evidential accounts are shown to be equally vulnerable to these fallacies. A strong case can be made, or so it is argued, that the fallacious nature of the Jeffreys-Lindley paradox stems primarily from the fact that the Bayesian and likelihoodist approaches dismiss the relevance of the generic capacity of the particular test in their evidential accounts. Indeed, these accounts cast aside the relevance of the sample space beyond and “controlling” the relevant error probabilities for being the key weaknesses of the frequentist approach.

3. The Large n Problem in Frequentist Testing

The approach to frequentist statistics followed in this article is known as error statistics (Mayo Reference Mayo1996). It can be viewed as a refinement/extension of the Fisher-Neyman-Pearson approach that offers a unifying inductive reasoning for frequentist inference. It refines the Fisher-Neyman-Pearson approach by bringing out the importance of specifying explicitly as well as validating vis-à-vis data the inductive premises of inference. It extends the Fisher-Neyman-Pearson approach by supplementing it with a postdata severity assessment with a view to address a number of foundational problems relating to a sound evidential account (see Mayo and Spanos Reference Mayo and Spanos2004, Reference Mayo and Spanos2006, Reference Mayo, Spanos, Gabbay, Thagard and Woods2011).

Consider the following numerical example discussed by Stone (Reference Stone1997):

How can one test this substantive hypothesis of interest?

The first step is to embed the above material experiment into a statistical model and frame the substantive hypothesis in terms of statistical parameters. With that in mind, let us assume that each of these n trials can be viewed as a realization of a sample of IID random variables defined by

which transforms the observed sequence of particles into data (1, 0, 0, 1, 1, … , 0). These conditions render the simple Bernoulli (Ber) model,

appropriate as a statistical model in the context of which the material experiment can be embedded. Note that in practice one should validate the IID assumptions to secure the reliability of inference. The substantive hypothesis of interest, h = .2, can be framed in terms of the Neyman-Pearson statistical hypotheses:

specified solely in terms of the unknown parameter of the statistical model in (2). A proper framing of the Neyman-Pearson hypotheses requires a partitioning of the parameter space, irrespective of whether one is substantively interested in one or more specific values of θ. From the statistical perspective, all values of θ in are relevant for defining the optimality of the test (see Spanos Reference Spanos2010, 569).

The test , defined by

where and , denote the rejection region and value, respectively, is a uniformly most powerful unbiased Neyman-Pearson test (see Lehmann Reference Lehmann1986). Using (4) one can define the type I error probability (significance level):

The sampling distribution in (4) is based on the fact that for a Bernoulli IID sample (X ₁, X ₂, … , X _n), the random variable , where Y denotes the number of 1’s in n trials, is binomially (Bin) distributed:

In light of the large sample size (n = 527,135), it is often judicious to choose a small type I error (Lehmann Reference Lehmann1986), say α = .003, which yields a rejection value of ; note that the Normal approximation to the binomial distribution is quite accurate in this case.

The power of test at defined by

as well as the type II error probability , for , are evaluated using the sampling distribution of under , which takes the form:

This indicates that test is consistent because its power increases with —a monotonically increasing function of —approaching one as (see Lehmann Reference Lehmann1986). Hence, other things being equal, increasing n increases the power of this test, confirming that there is nothing paradoxical about a larger n rendering a (consistent) test more sensitive.

Applying the Neyman-Pearson test to the above data,

leads to a rejection of . The p-value, defined as the probability of observing an outcome that accords less well with than does when is true, confirms the rejection of :

This definition is preferred to the traditional one, “the probability of observing a result more extreme than under ,” because it blends in better with the postdata severity evaluation discussed in section 6. The result suggests that data indicate “some” discrepancy between and the “true” θ (that gave rise to ), but it provides no information about its magnitude.

As mentioned above, what is problematic is the move from the accept/reject results, and the p-value, to claiming that data provide evidence for a particular hypothesis, because such a move is highly vulnerable to the fallacies of acceptance and rejection. However, in the context of frequentist testing, this vulnerability can be circumvented using a postdata severity evaluation.

How does the Jeffreys-Lindley paradox arise in this context? Using a spiked prior distribution (Lindley Reference Lindley1957) of the form

the formal claim associated with this paradox is

that is, the posterior probability of , conditional on , goes to one as n approaches infinity. What is not so obvious is why this result is considered an indicator of the soundness of the Bayesian account of evidence.

In relation to the nature of the null hypothesis (simple or composite), it is important to elucidate certain issues raised in this literature. First, in the case of two composite hypotheses

the test , where = {: , is both consistent and uniformly most powerful (see Lehmann Reference Lehmann1986). In this sense, the large n problem has nothing to do with the restriction to a point null hypothesis; it is simply a feature of a consistent Neyman-Pearson test. The nature of the null (simple or composite) only matters for the behavior of the likelihoodist ratio, the Bayes factor, and the posterior probability of . Second, the fact that the discrepancy between the p-value and the Bayes factor is smaller for a composite null is not particularly interesting because both measures are misleading in different ways.

Third, the comparison of the posterior , as θ varies over (which represent one’s revised beliefs about θ in light of ), with error probabilities (which measure how often a frequentist procedure errs as varies over ), is dubious. Indeed, as argued by Casella and Berger (Reference Casella and Berger1987, 110), the comparison is made possible by using a blatant misinterpretation of the p-value: “The phrase ‘the probability that is true’ has no meaning within frequency theory, but it has been argued that practitioners sometimes attach such a meaning to the p value. Since the p value, in the cases considered, is an upper bound on the infimum of it lies within or at the boundary of a range of Bayesian measures of evidence demonstrating the extent to which the Bayesian terminology can be attached.”

4. The Bayesian Approach

Consider applying the Bayes factor procedure to the hypotheses (3) using a uniform prior:

This gives rise to the Bayes factor:

It is interesting to note that the same Bayes factor (12) arises in the case of the spiked prior (9) with , where is given prior probability of and the other half is distributed equally among the remaining values of θ. This is because for , the ratio and will cancel out from .

The next step in Bayesian inference is to use (12) as the basis for fashioning an evidential account. A Bayes factor result , for , indicates that data favor the null with the “strength of evidence” increasing with k. In particular, for , the evidence is substantial, for the evidence is strong, and for is decisive (see Robert Reference Robert2007).

Comparing the result in (12) with the p-value in (8), Stone (Reference Stone1997, 263) pointed out: “The theoretician is pleased when the [likelihood–Bayes-minded] statistician reports a Bayes factor of 8 to 1 in favour of his brainchild, but the pleasure is alloyed when he uses his own P-value cookbook to reveal the 3.00 standard deviation excess of type A outcomes.”

A closer scrutiny of the evidential interpretation of the result in (12) suggests that it is not as clear-cut as it appears. This is because, on the basis of the same data , the Bayes factor “favors” not only but each individual value inside a certain interval around :

where the square bracket indicates inclusion of the end point, in the sense that, for each , , that is,

Worse, certain values in are favored by more strongly than :

where the left parenthesis indicates exclusion of the end point. It is important to emphasize that the subsets exist for every data , and one can locate them by trial and error. However, there is a much more efficient way to do that. As shown in section 5, can be defined as a subset of Θ around the maximum likelihood estimate (MLE) This is not coincidental because, as Mayo (Reference Mayo1996, 200) pointed out, is always the maximally likely alternative, irrespective of the null or other substantive values of interest. In this example, the Bayes factor for versus yields

indicating not only decisive evidence for but also that

This result is an instance of the fallacy of acceptance in the sense that the Bayes factor is misinterpreted as providing evidence for : against any value of θ in , when in fact provides stronger evidence for certain values of in .

What is the source of these conflicting evidence? Stone (Reference Stone1997, 263) conjectured the following explanation: “A subhypothesis strongly rejected by a significance test may be strongly supported in posterior probability if the prior puts insufficient weight on the hypotheses of non-negligible likelihood.”

Let us flesh out this conjecture. First, choose as a range of values of θ that favors in the sense given in (14). To this range of values the prior attributes insufficient weight since

Second, one can relate to both an equal-tail Bayesian credible interval as well as the frequentist confidence interval:

where denotes the standard deviation of . In light of this, one can consider to represent a range of values of θ with nonnegligible likelihood. Third, the weight attributed to these values by the Bayes factor,

is rather tiny, providing some support for Stone’s conjecture.

This is connected to the large n problem because for and y≔ that keeps close to its original value, the Bayes factor attribution in (17) would have been much larger () since

This raises the broader problem of how the large n problem might affect the Bayesian results. In light of the fact that for ,

the large n problem does not affect the ratio in (12), but it does affect the Bayes attribution by rendering the numerator and denominator much smaller.

Focusing on the latter problem, the question is whether one can address the “insufficient weight” problem, due to the large , by varying the value of in . That will take an extreme tilting of the prior for a whole range of values of θ to compensate for the particular n. For instance, the tilted spiked-prior

can counteract the maximally likely alternative problem. (For further discussion on data-based priors, see Shafer [Reference Shafer1982].) Reflecting on this issue, Stone (Reference Stone1997, 264) decried such a Bayesian move, arguing that “if the statistician were to withdraw his uniform prior and claim that he ought to have organized some more probability in the neighbourhood of , this would be a confession that his Bayesianity does not have a bedrock quality, that his coherence has only the (doubtfully useful) temporal value.”

Returning to the invariance of the Bayes factor to the sample size , it can be shown that it stems from the Fisher-Neyman factorization theorem, where, for a sufficient statistic s for θ, the likelihood function simplifies into

(see Cox and Hinkley Reference Cox and Hinkley1974, 22). In the case of the simple Normal (1) and Bernoulli (2) models, is a minimal sufficient statistic for θ, and thus the Bayes factor in (12) depends only on the observed value . That is, the factor cancels out because it is common to both the numerator and denominator.

Although it seems sensible that the likelihood ratio depends only on , the claim that it is irrelevant whether results from or when going from to claiming that data provide strong evidence for seems counterintuitive. As argued in section 7, the large n problem also plagues the Bayes factor primarily because its invariance to n renders its evidential interpretation vulnerable to the fallacy of acceptance, the reverse problem plaguing the p-value. Despite this vulnerability, the likelihood ratio has been proposed as an effective way to deal with the large n problem (see Freeman Reference Freeman1993).

5. The Likelihoodist Approach

The likelihoodist approach (Royall Reference Royall1997, 24) evaluates how data compares two simple hypotheses:

using the likelihood ratio (LR):

It is interesting to note that the Law of Likelihood (LL) was first proposed by Hacking (Reference Hacking1965), but a few years later, when reviewing a book on “likelihood,” Hacking changed his mind: “The only great thinker who tried it out was Fisher, and he was ambivalent. Allan Birnbaum and myself are very favourably reported in this book for things we have said about likelihood, but Birnbaum has given it up and I have become pretty dubious” (Hacking Reference Hacking1972, 137).

The idea behind the use of the likelihood ratio in (19) is that it ameliorates the large n problem by affecting both hypotheses equally (see Howson and Urbach Reference Howson and Urbach2006, 155). Strictly speaking, the LR can only be applied to the case where both hypotheses are simple (see Royall Reference Royall1997). For hypotheses such as the Neyman-Pearson hypotheses (3), however, the alternative takes an infinite number of values, and thus one needs to select particular point alternatives of interest to apply the LL.

In the case of the above example, a particularly interesting point alternative to is . For the hypotheses

which reverses the Bayes factor result and suggests that the degree to which data favor over is much stronger (), confirming the maximally likely alternative problem in (16). In fact, when it comes to fallacious results, the LL is in sync with the Bayes factor procedure because the former can be used directly to establish the subset in (13).

To see this, consider pairwise comparisons of different values of with :

The LL reveals that data favor each value in over since

As mentioned above, there is nothing coincidental about the subset since

where

That is, (21) is related to Stone’s remark associated with the p-value indicating “3 standard deviation excess of type A outcomes,” when it is viewed as an observed confidence interval (CI). Having said that, it is important to reiterate that although we used a CI around to define , such a subset exists in Θ for all data , irrespective of the way one locates .

In summary, applying the Bayes factor to the statistical hypotheses,

indicates that data provides substantial evidence for over . However, this inference is undermined by the fact that each value turns out to be favored more strongly than when tested using the generic hypotheses:

falling prey to the fallacy of acceptance. These conflicting results seriously undermine the initial favoring of over , and they bring out the fallacious implications of Bayesian and likelihoodist inferences, calling into question the Jeffreys-Lindley paradox. As aptly put by Stone (Reference Stone1997, 263): “When not misused, they [P-values] still provide some sort of control over the pursuit of weak clues—not a measure of faith in some alternative hypothesis. A P-value is a P-value is a P-value! That some users like to misinterpret it as a posterior probability or odds ratio or other inferential measure … should not detract from the P-value’s intrinsic, if uninterpretable, value.”

As shown in the next section, in the case of the p-value, there is a principled way to circumvent its weaknesses using Mayo’s (Reference Mayo1996) postdata severity assessment.

6. Severity: Addressing the Large n Problem

Severity constitutes a post-data evaluation of the Neyman-Pearson accept/reject results with a view to establish the (smallest/largest) discrepancy γ from warranted by data . As such, the severity evaluation is by definition directional since postdata one has an outcome whose sign indicates the relevant direction of departure from .

A hypothesis H passes a severe test with data if (i) accords with H (using a suitable measure of accordance), and (ii) with very high probability, test would have produced a result that accords less well with H than does, if H were false.

The case of testing the hypotheses (3) using (4) yielded , which led to rejecting . Postdata, the sign of the observed test statistic, , indicates that the rejection is clearly in the direction of values greater than . That is, in light of data , the two-sidedness of the original Neyman-Pearson test is irrelevant. Indeed, the same severity evaluation applies to the case of the one-sided test for the composite hypotheses (10). Condition (i) of severity implies that the generic form of the inferential claim that “passed” is (Mayo and Spanos Reference Mayo and Spanos2006):

It is important to emphasize that (24) is not a reformulation of the original hypotheses but a framing of the relevant inferential claim associated with the rejection of stemming from (see Spanos Reference Spanos2011). The rejection of calls for the appraisal of the largest discrepancy γ from warranted by data . Condition (ii) calls for the evaluation of the probability of “outcomes that accord less well with than does,” which translates into all those outcomes such that []. This gives rise to:

Given the numerical values in (7) and the relevant distribution in (6), one can proceed to evaluate (25) for different using the standardized statistic:

Table 1 reports the severity evaluation for different discrepancies of interest.

Table 1.Reject H_o: θ = (d(x_o) = 2.999)

	Inferential Claim
γ	,
0	θ > .2000	.999
.0009	θ > .2009	.914
.00095	θ > .20095	.900
.001	θ > .2010	.882
.0015	θ > .2015	.609
.00165	θ > .20165	.500
.002	θ > .2020	.264
.0023	θ > .2023	.120
.0024	θ > .2024	.087
.0025	θ > .2025	.062
.003	θ > .203	.007

To explain how one derives the results in table 1, consider the case , that is, the relevant claim is . Evaluating the relevant components

one can proceed to evaluate using the N tables, which yields:

The results in table 1 indicate that, for a severity threshold of say , the claim for which data provide evidence is .

How does this answer relate to the original question of interest of testing the theoretical prediction that the proportion of type A outcomes is .2? One needs to answer the question whether the particular discrepancy from the null, , is substantively significant, which cannot be answered exclusively on statistical grounds because it pertains to the substantive subject matter information. That is, one needs to consider in the context of the theory of particle physics that motivated the above experiment to decide whether it is substantively significant.

It is important to emphasize that the postdata severity evaluation goes beyond avoiding the misuse of p-values, as suggested by Stone (Reference Stone1997) in the above quotation. It addresses the key problem with Fisherian p-values in the sense that the severity evaluation provides the “magnitude” of the warranted discrepancy from the null by taking into account the generic capacity of the test (that includes n) in question as it relates to the observed data .

As shown in Mayo and Spanos (Reference Mayo and Spanos2006), the postdata severity assessment can be used to supplement frequentist testing with a view to bridge the gap between the p-value and the accept/reject rules, on the one hand, and providing evidence for or against a hypothesis in the form of the discrepancy γ from the null warranted by data , on the other hand. Its key difference from the Bayesian and likelihoodist approaches is that it takes into account the generic capacity of the test in establishing γ.

The severity-based evidential interpretation addresses not just the large-n problem but the fallacies of acceptance and rejection more broadly, as well as other charges leveled against frequentist testing, including the “arbitrariness” of choosing the significance level, the one-sided versus two-sided framing of hypotheses, the reversing of the null and alternative hypotheses, and so forth (see Mayo and Spanos Reference Mayo, Spanos, Gabbay, Thagard and Woods2011; Spanos Reference Spanos2011).

7. Bayesian and Likelihoodist Accounts Revisited

Where does the above severity assessment leave the Bayesian and likelihoodist inferences? Both approaches are plagued by the maximally likely alternative problem (Mayo Reference Mayo1996) in the sense that the value is always favored against every other value of θ, irrespective of the substantive values of interest. Any attempt to sidestep that problem will require an extreme data-based tilting of the prior against all values . In contrast, the severity of the inferential claim is always low, being equal to (table 1), calling into question the appropriateness of such a choice. In addition, the severity assessment in table 1 calls seriously into question the results associated with the two intervals and because these intervals include values of θ, for which the severity of the relevant inferential claim is very low (e.g., ).

The question that naturally arises is why the Bayesian and likelihoodist approaches give rise to the above conflicting and confusing results. The severity account gives a straightforward answer: both approaches ignore the generic capacity of a test when going from

indicating that is k times more likely than , to step 2: fashioning the result in (27) into the strength of evidence for or against .

Bayesians and likelihoodists are likely to challenge this criticism as misplaced by invoking the distinction between the logic versus the epistemology of inference to claim that the generic capacity of the test belongs to the latter, but their approaches are primarily focused on the former. Demarcating the logic of inference as pertaining to what follows from given premises (in a deductive sense) and the epistemology of inference as concerned with “how we learn from data ,” the generic capacity of a test belongs squarely within the logic of inference because it follows deductively from the premises (the statistical model) without any reference to data . The inductive dimension of severity emanates from the fact that its evidential account uses the particular data to infer something pertaining to the underlying generating mechanism represented by the statistical model.

The Bayesian objection would have had some merit if the approach were to end after is evaluated, but it does not. Similarly, likelihoodists do not end after is evaluated but proceed to claim an evidential based on benchmarks for the “strength of statistical evidence” for . For instance, , , is considered moderate evidence, while is considered strong evidence (see Royall Reference Royall1997). In contrast, the severity account ensures learning from data by employing reliable procedures to establish trustworthy evidence for hypotheses or claims pertaining to the underlying generating mechanism, the reliability of inference being calibrated in terms of the relevant error probabilities, both predata and postdata.

What is particularly interesting is that the Bayes factor and the likelihood ratio are directly related to the test (4) in the sense that the test statistic is a monotone function of the likelihood ratio statistic , and its rejection region is related to (see Lehmann Reference Lehmann1986). In this sense, the key difference between the frequentist and the other two approaches is that they ignore the sampling distribution and the associated error probabilities of the test in (4) by invoking the likelihood principle (Berger and Wolpert Reference Berger and Wolpert1988), which asserts that no other value of the sample , apart from data , is relevant for inference purposes. Indeed, Bayesian statisticians take delight in poking fun at frequentist testing by quoting Jeffreys’s (Reference Jeffreys1939/1961, 385) remark about the “absurdity” of invoking the quantifier “for all ”: “What the use of P implies, therefore, is that a hypothesis that may be true may be rejected because it has not predicted observable results that have not occurred. This seems a remarkable procedure.”

What these critics overlook is that their attempts to provide an evidential account for statistical hypotheses go astray exactly because they ignore the generic capacity of the test, which calls upon the quantifier “for all ” for its evaluation. Viewed from the severity perspective, the trouble with using a small p-value as a basis for inferring evidence for a particular alternative stems from the fact that it only indicates the presence of “some” discrepancy from , but it provides no information about its magnitude; the latter requires summoning the generic capacity of the test. In light of this fatal flaw of the p-value as a basis for an evidential account, the literature concerned with “reconciling” the p-value (or some modification of it) with various Bayesian measures (see Berger and Delampady Reference Berger and Delampady1987; Berger, Boukai, and Wang Reference Berger, Boukai and Wang1997; Sellke, Bayarri, and Berger Reference Sellke, Bayarri and Berger2001; Berger Reference Berger2003; inter alia) is overlooking the real issue. Any evidential account aiming to provide a sound answer to the question of when data provide evidence for or against a hypothesis (or a claim) can ignore the generic capacity of a test at its peril!

Intuitively, what goes wrong is that the Bayesian factor and the likelihoodist procedures use Euclidean geometry based on to evaluate evidence for different hypotheses ( or ), when in fact the statistical testing space is curved, with the curvature determined primarily by the capacity of the test to detect discrepancies from . This is especially relevant for the large n problem because the power of the test in (4) increases with , The use of Euclidean geometry is clearly apparent in the following quotation from Berger and Wolpert (Reference Berger and Wolpert1988, 7):

For a testing example, suppose it is desired to test

based on X ∼ N(μ, .25). The rejection region gives a test with error probabilities (type I and type II) of .0228. If is observed, it is then permissible to state that is rejected and that the error probability is . Common sense, however, indicates that fails to discriminate at all between and . Any sensible person would be equally uncertain as to the truth of or (based just on data ).

Any “sensible” Bayesian/likelihoodist would be wrong to conclude that provides the same evidence for both and just because it is halfway between their hypothesized values of μ (see Spanos Reference Spanos2011).

Although there is a connection between the “curved statistical space” and Stone’s (Reference Stone1997) conjecture concerning “insufficient weight on the hypotheses of non-negligible likelihood,” there is no way to use the generic capacity of the test to provide a more appropriate weighting scheme within the Bayesian and likelihoodist frameworks.

8. Summary and Conclusions

The Jeffreys-Lindley paradox has played an important role in undermining the credibility of frequentist inference by focusing attention on how the large-n problem renders frequentist testing vulnerable to the fallacy of rejection. It was argued that although there is nothing fallacious about rejecting , when n is large, there is a problem when this result is detached from the test itself and viewed as providing the same evidence for a particular alternative regardless of the generic capacity (that depends on n) of the test in question. This renders the p-value and the accept/reject rules vulnerable to the fallacies of acceptance and rejection.

The discussion has also called into question the basic premise of the Jeffreys-Lindley paradox concerning the sagacity of the Bayes factor favoring as n increases as symptomatic of the fallacy of acceptance, the reverse problem plaguing the p-value. More generally, it was shown that the move from to inferring that provides weak or strong evidence for , depending on the value of , renders the Bayes factor and the likelihood ratio equally susceptible to the same fallacies.

It was argued that in the context of frequentist testing these fallacies can be circumvented using a postdata severity evaluation. The key is that this evaluation takes into account the test’s generic capacity in establishing the discrepancy γ from the null warranted by data . The underlying intuition is that detecting a particular discrepancy γ using a very sensitive (insensitive) test provides less (more) strong evidence that γ is present. In contrast, the Bayesian and likelihoodist approaches have no principled way to circumvent these fallacies.