1. Introduction

Bayesianism in philosophy of science is most often associated with the “degree of belief” interpretation of probability or with a particular normative theory of belief revision. In statistics, “Bayesian methods” refers to a collection of statistical tools and procedures that stem from a systematic decision-theoretic foundation for statistical practice. According to the Bayesian viewpoint, these methods are preferable alternatives to the ubiquitous “classical” methods popular among experimentalists; Bayesians think that experimentalists ought to interpret their findings in light of the proper methods of belief revision and that they ought to use certain procedures (the Bayesian ones, not the classical ones) in analyzing their data. The normative justification for using Bayesian methods and eschewing (some) classical methods often takes the form of a so-called Dutch Book argument. There are many variants of the Dutch Book argument that I will not get into here,Footnote ¹ but generally speaking the Dutch Book argument compels rational agents to conform their degrees of belief over some set of possible events to the Kolmogorov probability axioms (nonnegativity, normalization, and finite additivity). The first clear and explicit examples of such arguments can be found in the work of Frank Ramsey (Reference Ramsey and Braithwaite1926/1931) and Bruno de Finetti (Reference de Finetti, Kyburg, Kyburg and Smokier1937/1964). De Finetti shows that if an agent’s previsions on some set of possible events do not conform to the Kolmogorov probability axioms, they are incoherent, that is, there exists a number of bets, each of which is acceptable to the agent, the combination of which guarantees a sure loss for the agent regardless of the state of the world.Footnote ² But this is a result about the rational behavior, or rational degrees of belief, of an idealized gambling agent, which seems to have little direct connection to decisions made by scientists in experimental settings—what relevance does it have to statistical practice? A statistical procedure is incoherent if the statisticians’ implied prior probabilities (i.e., their implied degrees of belief) over the hypothesis space are incoherent. Bayesians sometimes argue that scientists should abandon certain classical statistical procedures, like fixed-level null hypothesis significance testing, on the ground that the procedures are incoherent. In other words, the procedure implies that the scientist’s degrees of belief over the possible outcomes of the experiment are incoherent, and rational agents with incoherent beliefs are liable to be made sure losers by a clever gambler. Thus, the argument goes, the scientist should not use this procedure. But the experimentalists studying psychological response theories, or searching for new fundamental particles, are not entering into any bets, so why should they care that their procedures are incoherent? Should the mere fact that a procedure is judged to be incoherent provide sufficient reason for a rational scientist to abandon the procedure and use a coherent one instead? I am using the generic terms “scientist” and “experimentalist” here to refer to any agent who makes decisions about what methods to use in an experimental setting; this can be a laboratory supervisor, a technician, an evaluator at a research funding agency, and so on. I assume the scientist is not already a Bayesian, meaning that scientists are not already convinced that they ought to act as if their prior probabilities conform to the axioms.

I consider this question by looking at a paper by Schervish, Seidenfeld, and Kadane (Reference Schervish, Seidenfeld and Kadane2002), in which they relate gambling outcomes to the trade of risk functions. In that work, they present a Dutch Book–style mathematical result that could naturally be interpreted as lending normative support to the Bayesian criticism of experimentalists who engage in fixed-level hypothesis significance testing. A Bayesian statistician might use the result Schervish et al. present in their paper as the basis of a Dutch Book argument against the non-Bayesian scientist; I argue that such an argument should fail to convince a non-Bayesian scientist to change her methods because certain features of the theorem are uninterpretable in the context of actual laboratory procedures.Footnote ³ However, I also note that Schervish et al.’s work on the connections between risk and incoherence points to a different criticism of classical statistical practice, one that makes no reference to either gambling or any “nonclassical” statistical concepts. In particular, their work makes it clear that fixed-level hypothesis testing violates the classical statistician’s preference for lower-risk functions, at least when that preference is meant to apply to combinations (or averages) of hypothesis tests that might be done over the course of an experiment. This will be made precise below.

2. Background

A Dutch Book theorem (of the synchronic de Finetti/Ramsey type) generally has the following form: “if an agent’s previsions violate the probability axioms, then there exists a combination of individually acceptable bets such that entering into those bets guarantees a sure loss for the agent, regardless of the state of the world.” A Dutch Book argument, then, which compels an agent to conform to the probability axioms, will generally include as a premise that the agent would actually accept each of the individually acceptable bets that make her a sure loser. There are a number of interesting criticisms of Dutch Book arguments in the literature, many of which center on this premise (or something like it) and consider to what extent the idealized gambling model is appropriate for representing agents in the real world. See Kyburg (Reference Kyburg1978), Glymour (Reference Glymour1980, 71–72), and Hájek (Reference Hájek2005) for examples of such criticisms and related discussion. In contrast with those authors, I am interested in whether the premises of Dutch Book arguments, which are generally interpreted in terms of gambling or forecasting scenarios, have plausible analogues in the context of statistical decisions a scientist might make over the duration of an experiment. If they do, then Dutch Book arguments may provide compelling reasons for a scientist to abandon incoherent methods and adopt Bayesian methods instead. If they do not, then Dutch Book arguments seem to be of little relevance to experimentalists. Of course, in thinking that this is a worthwhile question to ask I am implicitly taking a controversial stance on the issue of what can constitute a “compelling reason.” The mere fact that an agent is incoherent—that is, the fact that there exists a set of gambles with such-and-such properties—does not by itself compel an agent to revise her beliefs or change her methods. There are compelling reasons to change methods if and only if the actual commitments and preferences of the agent, when taken seriously, cause the agent to engage in self-defeating behavior. There are some Bayesians who might disagree; for them, the fact that an agent’s beliefs are incoherent constitutes a compelling reason to revise them (e.g., Christensen Reference Christensen1996). I think this is a misguided view, but for considerations of space I will not elaborate on my reasons here.

The following discussion will focus on a particularly controversial but common statistical practice: fixed-level null hypothesis significance testing. Most of the Bayesian criticisms leveled at the practice of hypothesis testing involve consideration of prior probabilities or mixed tests. I eschew discussion of these concepts here because many practicing scientists who have been trained in classical methods claim they do not make use of these concepts. What principles do scientists trained in the classical paradigm actually endorse, then, when it comes to null hypothesis significance testing?

Suppose the experimentalist is interested in testing a simple null hypothesis H ₀ against a simple alternative H ₁. Let the independent and identically distributed random variables X ₁, … , X_n ∼ N(θ, σ²) with known σ² and consider H ₀ : θ = 0 versus H ₁ : θ = 1. After observing X ₁, … , X_n the experimentalist either decides to reject the null hypothesis or fails to reject the null hypothesis.Footnote ⁴ The negative consequences of making such a decision d can be summarized in the loss function L(θ, d):

with c ₀, c ₁ > 0. Following Schervish et al. (Reference Schervish, Seidenfeld and Kadane2002) one can set c ₁ = 1 and define the risk function for a sequence of decisions δ (also called a decision rule) as the expected loss:

where α is the probability of type I error (or the size of the test) and β is the probability of a type II error (1 − β is the power of the test). In general the experimentalist aims to minimize the risk function by choosing δ appropriately. In a classical hypothesis-testing scenario it is standard to accept the following two principles:Footnote ⁵

P2 is not properly a principle, but rather it is an extension of the Neyman-Pearson theory of hypothesis testing, and the .05 level is a convention adopted by a wide range of practitioners. In some situations α is fixed at a lower conventional level, like .01. For a given sample size, the Neyman-Pearson theory defines an admissible set of universally most powerful (α, β) pairs, where the risk function of a particular (α, β) pair is not dominated by any other; that is, each test is constructed to be the most powerful (lowest β) for a given α. Fixing the α value amounts to preferentially controlling the probability of type I error.

In section 5, I discuss the circumstances under which P1 and P2 might be mutually unsatisfiable. First, let us examine Schervish et al.’s (Reference Schervish, Seidenfeld and Kadane2002) analysis.

3. Gambling and Risk Functions

Schervish et al.’s (Reference Schervish, Seidenfeld and Kadane2002) stated aim is to provide a way of moving beyond the binary judgment of the coherence or incoherence of a particular method; coherence can come in degrees, and it is particularly useful to have a measure of how much bookies stand to lose when they expose themselves to sure loss in de Finetti’s sense.Footnote ⁶ The “rate of incoherence” then quantifies how badly the bookie is sure to do in the worst case, the case in which her incoherent previsions are maximally exploited by some clever gambler (or market of gamblers). With such a measure, the bookie can make a more informed decision about how or whether to become coherent, since becoming coherent is often “expensive” to the bookie. Bookies are assumed to have finite resources, so one way of assessing their incoherence is by examining the fraction of their resources they stand to lose (in the worst case) with their stated previsions. Much of Schervish et al.’s paper is devoted to working out the technical details of this normalization.

Following de Finetti, Schervish et al. (Reference Schervish, Seidenfeld and Kadane2002) consider gambling payoffs of the form α(X − x), where X is a random variable, α is the number of “units” of the bet up for sale, and x is a prevision for the random variable X. To be more precise, the bookie can actually announce upper previsions x for X by indicating that all bets with payoff α(X − y), where α < 0, y > x, are acceptable gambles, or she can announce lower previsions x for X by indicating that all bets with payoff α(X − y), where α > 0, y < x, are acceptable gambles. The bookie can announce only upper previsions or only lower previsions or both. The random variable X might be the indicator function I_A on event A, which is 1 when A occurs and 0 when A does not occur. Schervish et al. (2002, S250) state the following definition of incoherence:

A collection x ₁, … , x_n of upper and/or lower previsions for X ₁, … , X_n respectively is incoherent if there exists ε > 0 and a collection of acceptable gambles

such that

in which case we say that a Dutch Book has been made against the bookie.

The ε in equation (3) is the total guaranteed loss that the bookies face when they accept the set of gambles, but it is not by itself an adequate measure of the degree of incoherence because it is dependent on the α_i (the number of “units” bought or sold). Schervish et al. (Reference Schervish, Seidenfeld and Kadane2002) want a measure that reflects the stated previsions, not the number of gambles undertaken. To accomplish this, they characterize the bookies’ escrow, that is, the quantity that the bookies can be expected to set aside or commit themselves to betting with when entering into a gambling contract. They subsequently define the bookies’ “maximum rate of guaranteed loss” with respect to this escrow. The mathematical details do not affect the discussion in this essay, but the reader should consult Schervish et al. (Reference Schervish, Seidenfeld and Kadane2002) for the full technical presentation.

Schervish et al. (Reference Schervish, Seidenfeld and Kadane2002) apply their measure of incoherence to the case of fixed-level hypothesis testing. They interpret an agent’s preference for a test δ₁ over test δ₂ to mean that the agent would prefer to suffer the loss from δ₁ rather than the loss from δ₂, that is, the agent would rather “pay out” R(θ, δ₁) than R(θ, δ₂). They call R(θ, δ₂) − R(θ, δ₁) an acceptable gamble. Schervish et al. use α_δ(σ) and β_δ(σ) to denote the size and power of test δ conditional on σ (equivalently, conditional on sample size n). The conventional .05 test is δ_CL, with

and

equal to whatever is the highest power possible under the Neyman-Pearson theory for that σ. Schervish et al. show that the following gamble is acceptable to the scientist who prefers the .05-level test:

where

So, the preference for test δ_CL amounts to partaking in a “trade” of risk functions that has the form of a de Finetti gamble on an indicator function taking the value 0 when θ = 1 and 1 when θ = 0. The constants a and b are meant to be uncontroversial combinations of sizes and powers—purely classical statistical concepts with no explicit dependence on subjective prior probabilities.

Schervish et al. (Reference Schervish, Seidenfeld and Kadane2002) prove a theorem (their theorem 2) that applies to arbitrary decision rules. Rather than introduce the notation required to express this more general result, I will state the analogous result for the special case we have been considering: if δ₀ and δ₁ are not Bayes rules with respect to a common prior,Footnote ⁷ then there exist real numbers d ₀ and d ₁ and alternative decision rules

and

such that the two gambles

and

are both acceptable, but

where

,

,

,

are the generalized versions of constants a and b above (with respect to the alternative decision rules

,

). This demonstrates a kind of incoherence, because a combination of acceptable gambles yield a (sure) negative payoff for the bookie. And, since the conventional .05-level testing procedure is not a Bayes rule, that means that using such a rule is equivalent to announcing previsions that expose the scientist to a sure loss. In the next section we ask the question of how this result should be properly interpreted in an experimental setting.

4. The Dutch Book Result

Let us take a closer look at the mathematics, starting with the acceptable gamble in equation (4). In the context of gambling, the constants a and b have unproblematic standard interpretations: b is the prevision on random variable I_A and a is the number of “units” of the gamble bought or sold (in this case, the gamble is presumed to be one-sided, so the sign of a can be fixed appropriately). In the context of statistical decision making, the constants a and b have unproblematic interpretations in terms of combinations of type I and type II error probabilities, as in equation (5). But in the theorem stated by Schervish et al. (Reference Schervish, Seidenfeld and Kadane2002), the acceptable gambles are and . In the gambling context, the real numbers d ₀ and d ₁ are again unproblematic: it is presumed that if a bookie is willing to accept gambles on I_A with prevision b, gambles of the form w(I_A − b) are acceptable for any real-valued w, so long as the total number of “units” does not exceed the bookie’s available resources. In this case, the entire product acts as the number of “units” bought or sold (of that betting contract). But what is the intended interpretation of the real-valued coefficients d ₀ and d ₁ in the context of scientific decision making? Maybe looking at equation (6) can provide a hint.

Equation (6) can be rewritten in terms of risk functions:Footnote ⁸

and can be rearranged to take the form

Again, the theorem states that so long as δ_CL is a non-Bayes rule (which it is), there exist real numbers d ₀, d ₁ and rules

,

such that the inequality in (8) holds. Is this a compelling reason for a practitioner to abandon the method that uses δ_CL? This depends on whether the mathematical statement is interpretable in terms relevant to laboratory practice. A few interpretations seem ostensibly plausible.

One might think, in analogy with taking a real number multiple of “units” of a bet, that the coefficients d ₀ and d ₁ represent performing the respective tests d ₀ or d ₁ many times. But this cannot be the case, because the scientist is interested in performing the test one time for each sample size; assuming she takes “accept or reject” decisions seriously, she has nothing to gain by repeating the same test on the same data. Furthermore, it is not clear what it would mean to perform a test a non–whole number of times. Alternatively, the coefficients could represent the probability of performing the associated test, if d ₀, d ₁ > 0 and (d ₀ + d ₁) = 1 (think of flipping a coin to determine whether test or is performed). This does not seem to be the authors’ intended interpretation because it would make the normative force of their theorem dependent on the normative force of the controversial ancillarity principle (also known as the conditionality principle) in the presence of mixed tests.Footnote ⁹ That fixed-level hypothesis testing violates dominance in the presence of mixed tests has been known at least since Cox (Reference Cox1958). But the appropriateness of the ancillarity principle and the prescriptions for hypothesis testing in the presence of mixed tests is much debated among statisticians and philosophers of statistics, so it would add no new normative force to recast the issue in the form of a Dutch Book theorem. Furthermore, the theorem as stated by Schervish et al. (Reference Schervish, Seidenfeld and Kadane2002) would need to be constrained to reflect the fact that d ₀ and d ₁ are meant to be probabilities and not just some real-valued numbers.

It is possible that the coefficients d ₀ and d ₁ are not meant to have any special interpretation in the context of laboratory procedures that a scientist might actually carry out. The coefficients might simply be mathematical devices necessary for the formalism that calculates rates of incoherence for the implied priors that one can extract from the use of these testing procedures. This is perfectly reasonable for the purpose of demonstrating the rate of incoherence of the procedure, but it has the unfortunate consequence of making the Dutch Book result uninterpretable in the context of laboratory practice and thus not part of a compelling argument to change methods for any scientist who is not already a Bayesian.

I should note that nothing in the above analysis is meant to preclude the possibility that some other Dutch Book–style result will meet the desideratum I outlined, that is, that the book-making strategy has plausible interpretation in terms of statistical procedures an experimentalist might actually carry out in a laboratory setting. In particular, alternative measures of rates of incoherence might suggest strategies that do not require trades of risk functions to be multiplied by real-valued coefficients; this will depend on the mathematical details of the measure of incoherence (e.g., how the escrow function is defined). I am not aware of any measure that identifies de Finetti gambles with trades of risk functions but that does not require the manipulation of real-valued coefficients in order to guarantee a sure loss to the bookie.

5. Combined (or Average) Risk Functions

One corollary to Schervish et al.’s (Reference Schervish, Seidenfeld and Kadane2002) result, which is apparent when their result is stated in the form of equation (8), is that for any non-Bayes decision rule like δ_CL, there exist two alternative decision rules and such that the combined risk function of performing on sample n ₀ and on sample n ₁ will be lower than the combined risk function of performing δ_CL on sample n ₀ and sample n ₁. A combined risk function is just the sum of two risk functions for tests performed on two different samples. So, if we index the risk function associated with running test δ_CL on a sample n ₀ as , then the combined risk function for samples n ₀ and n ₁ is . More generally one can define the combined risk for a sequence of N tests: . The exact same could be said replacing combined risk functions with average risk functions, which might be more natural to consider in certain contexts: . So, if we extend what I called principle P1 in section 2 to cover combined (or average) risk functions, we now see that the α = .05 convention (P2) clashes with another purely classical statistical principle (P1*):

The α = .05 convention, when applied to tests at multiple sample sizes, is tantamount to a preference for dominated combined risk functions.

There is nothing mathematically novel in this claim. In fact, the same can be gleaned from discussions in Seidenfeld, Schervish, and Kadane (Reference Seidenfeld, Schervish, Kadane and Sieg1990) and Berry and Viele (Reference Berry and Viele2008), both of which draw on Cox (Reference Cox1958). The difference is that these discussions employ the terminology of mixed tests, and, as mentioned above, the presence of mixed tests brings to the fore the somewhat controversial ancillarity principle. Since statisticians disagree about the validity of this principle (e.g., Helland Reference Helland1995), it would be beneficial for practitioners to be able to get some insight into the apparent conflict between the Bayesian and classical recommendations with minimal foray into philosophically troubled waters. In another paper, Schervish, Seidenfeld, and Kadane (Reference Seidenfeld and Kadane2009, 218) prove a theorem (theorem 7) about the risk inadmissibility of non-Bayes decision rules when randomized decision rules are available. Randomized decision rules can be thought of as similar to mixed tests, as discussed above (i.e., the decision maker chooses which test to perform by some random process). Whether a randomized decision rule is in fact available or appealing to the practicing scientist depends on a number of considerations I do not have the space to discuss here. The formulation in terms of combined risk is, I think, advantageous because it does not appeal to any concepts or procedures that might be controversial by the lights of the experimentalist trained in the classical paradigm.

Berry and Viele (Reference Berry and Viele2008) show that one can calculate (for the normal distribution) a coherent decision rule in which α is a function of sample size. In summary: a coherent decision rule for null hypothesis testing requires that α decrease as the sample size increases. The precise functional dependence of α on sample size depends on a number of details, but for simple versus simple the prescription is not complicated. The recipe for compound testing is more involved and requires some perhaps controversial decisions on behalf of the scientist. An interesting implication of this result is that the experimentalist who chooses values of α consistent with Berry and Viele’s prescription can arrive at combined risk functions that have both lower α and lower β than δ_CL. In other words, we can find combinations of tests that are strictly better than δ_CL with respect to combined risk.

6. Future Work: Planning Particle Physics Experiments

While journals in fields like psychology have at times enforced something like the α = .05 or .01 convention as a prerequisite for publication, contemporary experimental particle physics has an (unwritten) analogous convention in the so-called 5σ criterion. The 5σ value amounts to an α of approximately 3 × 10⁻⁷ (see James Reference James2006, 321). Physics journals will generally only publish claims of “observation of” (as opposed to “evidence for” or something like it) new particles if the null hypothesis—which coincides with a “no new particle” model—is rejected at this extremely low significance level. For the history of this convention and its use, see Franklin (Reference Franklin2013). The common justification for the 5σ criterion is a high cost associated with false positives in the discovery of new particles; particle discovery claims at the 3σ and 4σ levels have been frequently overturned by more data and repeated experiments.

The long lifetimes of typical experiments in contemporary particle physics provide plausible contexts for evaluation in terms of average or combined risk functions. The ATLAS and CMS experiments at the Large Hadron Collider, for example, began taking data in 2009 and will continue to do so for several years. In July 2012, ATLAS and CMS both announced the observation of a particle resembling the long-sought-after Higgs boson. The null hypothesis, which consisted of an expected distribution of particle masses that did not include the Higgs, was rejected at the 5σ level in both experiments (see ATLAS Collaboration 2012; CMS Collaboration Reference Collaboration2012). Both experiments continue to take data and have presented updated results with much larger data sets, again rejecting the null hypothesis but now (as of mid-2014) at levels well above 5σ. Fortunately, as the experiments collect more data, they report the highest significance level at which they can reject the null hypothesis, and not only the fact that the hypothesis remains rejected. Yet it remains true that many journals in the field do not accept “observation” results below the 5σ level, regardless of sample size. This convention is suboptimal with respect to combined risk. Physicists could do better by adopting a coherent decision rule, and this does not require specifying prior probabilities over competing hypotheses. In future work, I hope to include suggestions for adopting a coherent decision rule, taking the details and complexities of particle physics data into account.

7. Conclusion

Dutch Book arguments in themselves seem to be unconvincing to non-Bayesian experimentalists because they are not relevant (from the perspective of a non-Bayesian) to decisions actually made in laboratories. But luckily, the results of Schervish et al. (Reference Schervish, Seidenfeld and Kadane2002) point to a way to move forward. By changing the language and focus of the discussion, I have shown how two commonly espoused principles of classical hypothesis testing—the preference for lower risk functions and the convention of using a fixed α-level test at all sample sizes—conflict when the former principle is extended to include combined or average risk. No considerations of mixing tests, ancillarity, or prior probabilities played a key role in identifying this problem or in the proposed solution. The context of contemporary particle physics is offered as a motivating example for the plausibility of considering combined (or average) risk functions in planning experiments. Experimentalists have compelling reasons to take steps analogous to those advocated by Berry and Viele (Reference Berry and Viele2008) in order to determine an optimal way to tie the choice of α values to sample size. Thus, we need not find ourselves at an impasse when the experimentalist insists she “doesn’t bet.” We can use the results of Bayesian analysis to illuminate the circumstances in which classical statistical commitments conflict with each other.