1. Introduction: Lindley’s Paradox

Lindley’s Paradox exposes a salient divergence between subjective Bayesian and frequentist reasoning when a parametric point null hypothesis is tested against an unspecified alternative . Since the paradox has repercussions for the interpretation of statistical tests in general, it is of high philosophical interest.

To illustrate the paradox, we give an example from parapsychological research (Jahn, Dunne, and Nelson Reference Jahn, Dunne and Nelson1987). The case at hand involved the test of a subject’s claim to affect a series of randomly generated zeros and ones () by means of extrasensory capacities (ESP). The subject claimed that his ESP would make the sample mean differ significantly from 0.5.

A very large data set () was collected to test this hypothesis. The sequence of zeros and ones, , was described by a binomial model . The null hypothesis asserted that the results were generated by a machine operating with a chance of , whereas the alternative was the unspecified hypothesis .

Jahn et al. (Reference Jahn, Dunne and Nelson1987) report that in 104,490,000 trials, 52,263,471 ones and 52,226,529 zeros were observed. Frequentists would now calculate the z-statistic, which is

and reject the null hypothesis on the grounds of the very low p-value it induces:

Thus, the data would be interpreted as strong evidence for the presence of ESP.

Compare this to the result of a Bayesian analysis. Jefferys (Reference Jefferys1990) assigns a conventional positive probability to the null hypothesis, a uniform prior over the alternative, and calculates a Bayesian measure of evidence in favor of the null, namely, the Bayes factor. The evidence x provides for vis-à-vis is written as and defined as the ratio of prior and posterior odds:

Hence, the data clearly favor the null over the alternative and do not provide evidence for the presence of ESP.

This divergence between Bayesians and frequentists has, since the seminal paper of Lindley (Reference Lindley1957), been known as Lindley’s Paradox. In Lindley’s original formulation, the paradox is stated as follows: assume that we compare observation sets of different sample size N, all of which attain, in frequentist terms, the same p-value (e.g., the highly significant value of .01). In that case, as N increases, the Bayesian evaluation of the data will become ever more inclined toward the null hypothesis. Thus, a result that seems to refute the null from a frequentist point of view can strongly support it from a Bayesian perspective. Put formally (for the case of Gaussian models):

As the ESP example makes clear, Lindley’s Paradox actually extends beyond Gaussian models with known variance. It exposes a general divergence between Bayesians and frequentists in hypothesis tests with large sample size.

In this article, I consider the following questions: First, which statistical analysis of the ESP example, and Lindley’s Paradox in general, is most adequate? Second, which implications does Lindley’s Paradox have for the methodological debates between Bayesians and frequentists? Third, does our analysis have ramifications for the interpretation of point null hypothesis tests? I will argue that both the subjective Bayesian and the standard frequentist ways to conceive of Lindley’s Paradox are unsatisfactory and that alternatives have to be explored. In particular, I believe that José Bernardo’s approach (the Bayesian Reference Criterion, or BRC) holds considerable promise as a decision model of hypothesis testing, both in terms of the implied utility structure and as a reply to Lindley’s Paradox.

2. Testing a Precise Null Hypothesis: Frequentist versus Bayesian Accounts

Lindley’s Paradox deals with tests of a precise null hypothesis against an unspecified alternative for large sample sizes. But why are we actually testing a precise null hypothesis if we know in advance that this hypothesis is, in practice, never exactly true? For instance, in tests for the efficacy of a medical drug, it can be safely assumed that even the most unassuming placebo will have some minimal effect, positive or negative.

The answer is that precise null hypotheses often give us a useful idealization of reality. This is rooted in Popperian philosophy of science: “only a highly testable or improbable theory is worth testing and is actually (and not only potentially) satisfactory if it withstands severe tests” (Popper Reference Popper1963, 219–20). Accepting such a theory is not understood as endorsing the theory’s truth but as choosing it as a guide for future predictions and theoretical developments.

Frequentists have taken the baton from Popper and explicated the idea of severe testing by means of statistical hypothesis tests. Their mathematical rationale is that if the discrepancy between data and null hypothesis is large enough, we can infer the presence of a significant effect and reject the null hypothesis. For measuring the discrepancy in the data with respect to the postulated mean value of a normal model, one canonically uses the statistic

that we have already encountered above. Higher values of z denote a higher divergence from the null, and vice versa. Since the distribution of z usually varies with the sample size, some kind of standardization is required. Many practitioners use the p-value or significance level, that is, the “tail area” of the null hypothesis under the observed data, namely, .

On that reading, a low p-value indicates evidence against the null: the chance that z takes a value at least as high as would be very small if the null were indeed true. Conventionally, means significant evidence against the null and very significant evidence. In the context of hypothesis testing, it is then common to say that the null hypothesis is rejected at the .05 level, and so on.

Subjective Bayesians choose a completely different approach to hypothesis testing. For them, scientific inference obeys the rules of probabilistic calculus. Probabilities represent honest, subjective degrees of belief, which are updated by means of Bayesian Conditionalization. A Bayesian inference about a null hypothesis is based on the posterior probability , the synthesis of data x and prior . Bayes’s theorem can be used to calculate the posterior on the basis of the prior and the likelihood of the data.

If we investigate the source of Lindley’s Paradox, one might conjecture that an “impartial” but unrealistically high prior for (e.g., ) is the culprit for the high posterior probability of the null. However, Lindley’s findings persist if the analysis is conducted in terms of Bayes factors, like in the ESP example. These measures of evidence are independent of the particular prior of . For instance, if the prior over the alternatives to the null follows an distribution, then the Bayes factor in favor of the null can be computed as

which converges, for increasing N, to infinity as the second factor is bounded (Bernardo Reference Bernardo and Bernardo1999, 102). This demonstrates that the precise value of is immaterial for the outcome of the subjective Bayesian analysis.

Why is it that this result diverges so remarkably from the frequentist finding of significant evidence against the null? If the p-value, and consequently the value of , remains constant for increasing N, we can make use of the central limit theorem: converges, for all underlying distributions with bounded second moments, in distribution against . Thus, as , we obtain that , and . In other words, the sample mean gets ever closer to , favoring the null over the alternatives. For, the deviance between the variance-corrected sample mean z and will be relatively small compared to the deviance between z and all those hypotheses in that are remote from . By contrast, significance tests do not consider the probability of the data under these alternatives.

In other words, as soon as we take our priors over seriously, as an expression of our uncertainty about which alternatives to are more likely than others, we will, in the long run, end up with results that favor over an unspecified alternative. Bayesians read this as the fatal blow for frequentist inference since an ever smaller deviance of the sample mean from the parameter value will suffice for a highly significant result. Obviously, this makes no scientific sense. Small, uncontrollable biases will be present in any record of data, and frequentist hypothesis tests are unable to distinguish between statistical significance () and scientific significance (a real effect is present). A Bayesian analysis, however, accounts for this insight: as , an ever greater chunk of the alternative will be far away from , favoring the null hypothesis.

These phenomena exemplify more general and foundational criticisms of frequentist inference, in particular the objection that p-values grossly overstate evidence against the null (Cohen Reference Cohen1994; Royall Reference Royall1997; Goodman Reference Goodman1999). For instance, even the minimum of under a large class of priors is typically much higher than the observed p-value (Berger and Sellke Reference Berger and Sellke1987).

Still, also the subjective Bayesian stance on hypothesis tests is not entirely satisfactory. Assigning a strictly positive degree of belief to a precise hypothesis is a misleading and inaccurate representation of our subjective uncertainty. In terms of degrees of belief, is not that different from any value in its neighborhood. Standardly, we would assign a continuous prior over the real line, and there is no reason why a set of (Lebesgue) measure zero, namely, , should have a strictly positive probability. But if we set , then for most priors (e.g., an improper uniform prior) the posterior probability distribution will not peak at the null value but somewhere else. Thus, the apparently innocuous assumption has a marked impact on the result of the Bayesian analysis.

A natural reply to this objection contends that is actually an idealization of the hypothesis , for some small , rather than a precise hypothesis . Then, it would make sense to use strictly positive priors. Indeed, it has been shown that point null hypothesis tests approximate, in terms of Bayes factors, a test of whether a small interval around the null contains the true parameter value (theorem 1 in Berger and Delampady Reference Berger and Delampady1987). Seen that way, it does make sense to assign a strictly positive prior to .

Unfortunately, this will not help us in the situation of Lindley’s Paradox: when , the convergence results break down, and testing a point null is no longer analogous to testing whether a narrow interval contains (Bernardo Reference Bernardo and Bernardo1999, 102). In the asymptotic limit, the Bayesian cannot justify the strictly positive probability of as an approximation to testing the hypothesis that the parameter value is close to —which is the hypothesis of real scientific interest.

This may be the toughest challenge posed by Lindley’s Paradox. In the debate with frequentists, Bayesians like to appeal to “foundations,” but assigning a strictly positive probability to a precise hypothesis is hard to justify as a foundationally sound representation of subjective uncertainty.

Moreover, the Bayesian analysis fails to explain why hypothesis tests have such an appeal to scientific practitioners, even to those that are statistically well educated. Why should we bother testing a hypothesis if only posterior probabilities are relevant? Why even consider a precise hypothesis if it is known to be wrong? The next section will highlight these questions and briefly discuss the function of hypothesis tests in scientific inquiry.

3. Intermezzo: A Note on Precise Hypotheses

Since both Bayesians and frequentists struggle to deliver satisfactory responses to Lindley’s Paradox, one may conjecture that the real problem is with testing a precise hypothesis as such. For instance, if we constructed a 95% confidence interval for in the ESP case, it would not include . But it would be close enough to as to avoid the impression that the null was grossly mistaken.Footnote ¹ Hence, Lindley’s Paradox seems to vanish into thin air if we only adopt a different frequentist perspective.

However, this proposal is not satisfactory either. Confidence intervals do not state which hypotheses are credible—they only list the hypotheses that are consistent with the data, in the sense that these hypotheses would not be rejected in a significance test. Therefore, confidence intervals are intimately connected to significance tests and share a lot of their foundational problems (cf. Seidenfeld Reference Seidenfeld1981; Sprenger Reference Sprenger, Hájek and Hitchcock2013). Second, confidence intervals do not involve a decision-theoretic component; they are interval estimators. In particular, they do not explain why tests of a precise null have any role in scientific methodology. Since any proper resolution of Lindley’s Paradox should address this question, a confidence interval approach evades rather than solves the paradox.

On this note, one ought to realize that tests of a precise null usually serve two purposes: to find out whether an intervention has a significant effect and, since any intervention will have some minute effect, to decide whether the null hypothesis can be used as a proxy for the more general model. The point null is usually much easier to test and to handle than any composite hypothesis, so we have positive reasons to “accept” it, as long as the divergence to the data is not too large.

This view of scientific inference is hardly compatible with an orthodox Bayesian approach. For instance, the assumption neglects that hypothesis tests ask, in the first place, whether is a reasonable simplification of a more general model—and not whether we entertain a high degree of belief in a precise value of . Also, point null hypothesis tests are by definition asymmetric, but a subjective Bayesian analysis in terms of Bayes factors or posterior probabilities is essentially symmetric.

In total, subjective Bayesians have a hard time explaining why informative and precise but improbable hypotheses should sometimes be preferred over more general alternatives. The challenge for the Bayesian consists in modeling that we may be less interested in the truth of than in its usefulness. The next section presents an answer to this effect developed by José Miguel Bernardo (Reference Bernardo and Bernardo1999, Reference Bernardo and Bernardo2012).

4. The BRC Approach to Hypothesis Testing

This section presents a full Bayesian decision model for point null hypothesis testing that addresses Lindley’s Paradox: José Bernardo’s BRC (Reference Bernardo and Bernardo1999, Reference Bernardo and Bernardo2012). The point consists in shifting the focus from the truth of to its predictive value and in stipulating a specific utility structure. While classical Bayesian accounts of hypothesis testing involve simple exogenous utilities (e.g., a loss of zero for correct decisions and one for wrong decisions) and use the posterior probability as the only criterion for accepting or rejecting the null, Bernardo’s approach is based on endogenous, prediction-based utilities. In the remainder, I sketch a simplified version of Bernardo’s BRC in order to elaborate the main ideas of philosophical interest.

Since the work of R. A. Fisher, the replication of previously observed effects has been recognized as a main goal of experimental research in science and as a main motivation for significance tests (cf. Schmidt and Hunter Reference Schmidt, Hunter and Harlow1997). Therefore, a central component of Bernardo’s decision model focuses on the expected predictive accuracy of the null for future data. Hence, we need a function that evaluates the predictive score of a hypothesis, given some data y. The canonical approach consists in the logarithmic score (Good Reference Good1952): if an event considered to be likely occurs, then the score is high; if an unlikely event occurs, the score is low. This is a natural way of rewarding good and punishing bad predictions.

A generalization of this scoring rule describes the score of data y under parameter value as , where is a scaling term, and is a function that depends on the data only. Informally speaking, is decomposed into a prediction term and a term that depends on the desirability of an outcome, where the latter will eventually turn out to be irrelevant. This is a useful generalization of the logarithmic score. Consequently, if is the true parameter value, the utility of taking as a proxy for the more general model is

The overall utility U of a decision, however, should depend not only on the predictive score, as captured in q, but also on the cost of selecting a specific hypothesis . As explained above, should be preferred to ceteris paribus because it is more informative, simpler, and less prone to the risk of overfitting (in case there are nuisance parameters). Therefore, it is fair to set . Writing , we obtain

Note that the utility of accepting is evaluated against the true parameter value and that the alternative is not represented by a probabilistic average (e.g., the posterior mean) but by its best unknown element, . Much better than subjective Bayesianism, this approach represents the essential asymmetry in testing a point null hypothesis. Consequently, the difference in expected utility, conditional on the posterior density of , can be written as

This means that the expected utility difference between inferring to the null hypothesis and keeping the general model is essentially a function of the expected log-likelihood ratio between the null hypothesis and the true model, calibrated against a “utility constant” . For the latter, Bernardo suggests a conventional choice that recovers the well-probed scientific practice of regarding 5 standard deviations as compelling evidence against the null.Footnote ² The exact value of depends, of course, on the context—on how much divergence is required to balance the advantages of working with a simpler, more informative, and more accessible model (Bernardo Reference Bernardo and Bernardo1999, 108).

Wrapping up all this, we will reject the null if and only if , which amounts to the

Bayesian Reference Criterion: Data x are incompatible with the null hypothesis , assuming that they have been generated from the probability model , if and only if

This approach has a variety of remarkable features. First, it puts hypothesis testing on firm decision-theoretic grounds, with predictive value being the primary criterion. This foundational soundness distinguishes BRC vis-à-vis frequentist procedures.

Second, accepting the null, that is, using as a proxy for , amounts to claiming that the difference in expected predictive success of and the true parameter value will be offset by the fact that is more elegant, more informative, and easier to test. Hence, BRC does not only establish a trade-off between different epistemic virtues: it is also in notable agreement with Popper’s view that “science does not aim, primarily, at high probabilities. It aims at high informative content, well backed by experience” (Reference Popper1934/1959, 399). In marked difference to the orthodox Bayesian approach, accepting no longer involves commitment to the truth or likelihood of .

Third, the approach is better equipped than subjective Bayesianism to account for frequentist intuitions since under some conditions, the results of BRC agree with the results of a frequentist analysis, as we shall see below. Fourth, it is invariant of the particular parametrization; that is, the final inference does not depend on whether we work with or a 1:1 transformation . Fifth, it is neutral with respect to the kind of prior probabilities that are fed into the analysis.

5. Revisiting Lindley’s Paradox

We now investigate how Bernardo’s approach deals with Lindley’s Paradox and return to the ESP example from the introduction. It turns out that the BRC quantifies the expected loss from using as a proxy for the true value as substantial. Using a reference prior for (Bernardo Reference Bernardo1979), the expected loss under the null hypothesis is calculated as . This establishes that “under the accepted conditions, the precise value is rather incompatible with the data” (Bernardo Reference Bernardo and Bernardo2012, 18). In other words, the predictive loss from taking the null as a proxy for the posterior-corrected alternative will be substantial.

Of course, the rejection of the null hypothesis does not prove the ESP of our subject; a much more plausible explanation is a small bias in the random generator. This is actually substantiated by looking at the posterior distribution of : due to the huge sample size, we find that for any nonextreme prior probability function, we obtain the posterior , which shows that most of the posterior mass is concentrated in a narrow interval that does not contain the null. In this sense, we are justified to reject the null without having to infer to a substantial discrepancy between and .

Although BRC has a sound basis in Bayesian decision theory, the results of a BRC analysis disagree with Jeffrey’s subjective Bayesian analysis. Why is this the case? First, the conventional utility structure is substantially changed in BRC, and the final decision is no longer a simple function of the posterior probability of . Second, a Bayes factor comparison effectively compares the likelihood of the data under to the averaged likelihood of the data under . However, this quantity is strongly influenced by whether there are some extreme hypotheses in that fit the data poorly. Compared to the huge amount of data that we have just collected, the impact of these hypotheses (mediated via the conventional uniform prior) should be minute. These arguments explain why most people would tend to judge the data as incompatible with the precise null but fail to see a scientifically interesting effect. Thus, BRC indeed gives a convincing account of Lindley’s Paradox in the ESP example.

6. Conclusion and Outlook

We have demonstrated how Lindley’s Paradox—the extreme divergence of Bayesian and frequentist inference in tests of a precise null hypothesis with large sample size—challenges the standard methods of both Bayesian and frequentist inference. Neither a classical frequentist nor a subjective Bayesian analysis provides a convincing account of the problem. Therefore, I have presented Bernardo’s BRC as a full Bayesian model of testing point null hypotheses. It turns out that BRC gives a sensible Bayesian treatment of Lindley’s Paradox, due to its focus on predictive performance and likely replication of the effect. Although BRC has sound foundations in subjective expected utility theory, it preserves testing a precise hypothesis as a distinct form of statistical inference and can be motivated from a broadly Popperian perspective.

Of course, the BRC approach is not immune to objections (see the discussion pieces in Bernardo [Reference Bernardo and Bernardo2012]). However, BRC definitely underlines that Bayesian inference in science need not necessarily infer to highly probable models—a misconception that is perpetuated in post-Carnapian primers on Bayesian inference and that has attracted understandable criticism. For instance, Earman (Reference Earman1992, 33) takes, in his exposition of Bayesian reasoning, the liberty of announcing that “issues in Bayesian decision theory will be ignored.” Contrary to Earman, I claim that Bayesian reasoning cannot dispense with the decision-theoretic dimension if it aims at scientific relevance. A purely epistemic approach to theory choice, as exemplified in much of Bayesian confirmation theory, falls short of an appropriate model of scientific reasoning. Therefore, this article is not only a contribution to statistical methodology: it highlights the need to appreciate the subtle interplay of probabilities and (predictive) utilities in Bayesian inference and to change our perspective on the use of Bayesian reasoning in science.